BASIC_LEXER Attributes

BASIC_LEXER has the attributes shown in Table 2-20.

Table 20 BASIC_LEXER Attributes

Attribute	Attribute Value
`continuation`	characters
`numgroup`	characters
`numjoin`	characters
`printjoins`	characters
`punctuations`	characters
`skipjoins`	characters
`startjoins`	non alphanumeric characters that occur at the beginning of a token (string)
`endjoins`	non alphanumeric characters that occur at the end of a token (string)
`whitespace`	characters (string)
`newline`	NEWLINE (\n) CARRIAGE_RETURN (\r)
`base_letter`	NO (disabled)
`base_letter`	YES (enabled)
`base_letter_type`	GENERIC (default)
`base_letter_type`	SPECIFIC
`override_base_letter`	TRUE
`override_base_letter`	FALSE (default)
`mixed_case`	NO (disabled)
`mixed_case`	YES (enabled)
`composite`	DEFAULT (no composite word indexing, default)
`composite`	GERMAN (German composite word indexing)
`composite`	DUTCH (Dutch composite word indexing)
`index_stems` Use the numeric value in a string or the string value.	NONE ENGLISH DERIVATIONAL DUTCH FRENCH GERMAN ITALIAN SPANISH CATALAN CZECH GREEK HUNGARIAN POLISH PORTUGUESE ROMANIAN RUSSIAN SWEDISH (see Note) DERIVATIONAL_NEW (see Note) DUTCH_NEW (see Note) ENGLISH_NEW (see Note) FRENCH_NEW (see Note) GERMAN_NEW (see Note) ITALIAN_NEW (see Note) SPANISH_NEW (see Note) TURKISH Note: De-compounding word stemming is automatically performed when `index_stems` is set to SWEDISH, or DUTCH_NEW values. Note: Seven of the `index_stem` attributes that are new for this release have a “_NEW” suffix to enable you to utilize the new stemmer attributes while maintaining backward compatibility with previous releases of Oracle Text.
`index_themes`	YES (enabled)
`index_themes`	NO (disabled, default)
`index_text`	YES (enabled, default)
`index_text`	NO (disabled)
`prove_themes`	YES (enabled, default)
`prove_themes`	NO (disabled)
`theme_language`	AUTO (default)
`theme_language`	(any Globalization Support language)
`alternate_spelling`	GERMAN (German alternate spelling)
`alternate_spelling`	DANISH (Danish alternate spelling)
`alternate_spelling`	SWEDISH (Swedish alternate spelling)
`alternate_spelling`	NONE (No alternate spelling, default)
`new_german_spelling`	YES NO (default)

continuation

Specify the characters that indicate a word continues on the next line and should be indexed as a single token. The most common continuation characters are hyphen ‘-‘ and backslash ‘'.

numgroup

Specify a single character that, when it appears in a string of digits, indicates that the digits are groupings within a larger single unit.

For example, comma ‘,’ might be defined as a numgroup character because it often indicates a grouping of thousands when it appears in a string of digits.

numjoin

Specify the characters that, when they appear in a string of digits, cause Oracle Text to index the string of digits as a single unit or word.

For example, period ‘.’ can be defined as a numjoin character because it often serves as a decimal point when it appears in a string of digits.

Note:

The default values for numjoin and numgroup are determined by the globalization support initialization parameters that are specified for the database.

In general, a value need not be specified for either numjoin or numgroup when creating a lexer preference for BASIC_LEXER.

printjoins

Specify the non alphanumeric characters that, when they appear anywhere in a word (beginning, middle, or end), are processed as alphanumeric and included with the token in the Text index. This includes printjoins that occur consecutively.

For example, if the hyphen ‘-‘ and underscore ‘_’ characters are defined as printjoins, terms such as pseudo-intellectual and _file_ are stored in the Text index as pseudo-intellectual and _file_.

Note: If a printjoins character is also defined as a punctuations character, the character is only processed as an alphanumeric character if the character immediately following it is a standard alphanumeric character or has been defined as a printjoins or skipjoins character.

punctuations

Specify a list of non-alphanumeric characters that, when they appear at the end of a word, indicate the end of a sentence. The defaults are period ‘.’, question mark ‘?’, and exclamation point ‘!’.

Characters that are defined as punctuations are removed from a token before text indexing. However, if a punctuations character is also defined as a printjoins character, then the character is removed only when it is the last character in the token.

For example, if the period (.) is defined as both a printjoins and a punctuations character, then the following transformations take place during indexing and querying as well:

Token	Indexed Token
.doc	.doc
dog.doc	dog.doc
dog..doc	dog..doc
dog.	dog
dog…	dog..

In addition, BASIC_LEXER use punctuations characters in conjunction with newline and whitespace characters to determine sentence and paragraph delimiters for sentence/paragraph searching.

skipjoins

Specify the non-alphanumeric characters that, when they appear within a word, identify the word as a single token; however, the characters are not stored with the token in the Text index.

For example, if the hyphen character ‘-‘ is defined as a skipjoins, then the word pseudo-intellectual is stored in the Text index as pseudointellectual.

Note: Printjoins and skipjoins are mutually exclusive. The same characters cannot be specified for both attributes.

startjoins/endjoins

For startjoins, specify the characters that when encountered as the first character in a token explicitly identify the start of the token. The character, as well as any other startjoins characters that immediately follow it, is included in the Text index entry for the token. In addition, the first startjoins character in a string of startjoins characters implicitly ends the previous token.

For endjoins, specify the characters that when encountered as the last character in a token explicitly identify the end of the token. The character, as well as any other startjoins characters that immediately follow it, is included in the Text index entry for the token.

The following rules apply to both startjoins and endjoins:

The characters specified for startjoins/endjoins cannot occur in any of the other attributes for BASIC_LEXER.
startjoins/endjoins characters can occur only at the beginning or end of tokens

Printjoins differ from endjoins and startjoins in that position does not matter. For example, $35 will be indexed as one token if $ is a startjoin or a printjoin, but as two tokens if it is defined as an endjoin.

whitespace

Specify the characters that are treated as blank spaces between tokens. BASIC_LEXER uses whitespace characters in conjunction with punctuations and newline characters to identify character strings that serve as sentence delimiters for sentence and paragraph searching.

The predefined default values for whitespace are space and tab. These values cannot be changed. Specifying characters as whitespace characters adds to these defaults.

newline

Specify the characters that indicate the end of a line of text. BASIC_LEXER uses newline characters in conjunction with punctuations and whitespace characters to identify character strings that serve as paragraph delimiters for sentence and paragraph searching.

The only valid values for newline are NEWLINE and CARRIAGE_RETURN (for carriage returns). The default is NEWLINE.

base_letter

Specify whether characters that have diacritical marks (umlauts, cedillas, acute accents, and so on) are converted to their base form before being stored in the Text index. The default is NO (base-letter conversion disabled). For more information on base-letter conversions and base_letter_type, see Base-Letter Conversion.

base_letter_type

Specify GENERIC or SPECIFIC.

The GENERIC value is the default and means that base letter transformation uses one transformation table that applies to all languages. For more information on base-letter conversions and base_letter_type, see “Base-Letter Conversion”.

override_base_letter

When base_letter is enabled at the same time as alternate_spelling, it is sometimes necessary to override base_letter to prevent unexpected results from serial transformations. See “Overriding Alternative Spelling Features”. Default is FALSE.

mixed_case

Specify whether the lexer leaves the tokens exactly as they appear in the text or converts the tokens to all uppercase. The default is NO (tokens are converted to all uppercase).

Note: Oracle Text ensures that word queries match the case sensitivity of the index being queried. As a result, if you enable case sensitivity for your Text index, queries against the index are always case sensitive.

composite

Specify whether composite word indexing is disabled or enabled for either GERMAN or DUTCH text. The default is DEFAULT (composite word indexing disabled).

Words that are usually one entry in a German dictionary are not split into composite stems, while words that aren’t dictionary entries are split into composite stems.

To retrieve the indexed composite stems, you must enter a stem query, such as $bahnhof. The language of the wordlist stemmer must match the language of the composite stems.