AUTO_LEXER Language-Independent Attributes

Table 2-16 lists the language-independent attributes available in the AUTO_LEXER.

Table 16 AUTO_LEXER Language-Independent Attributes

Attribute	Attribute Value	Description
language	<characters> (space-delimited string)	Specifies the possible languages of the input documents. If no language is specified, then AUTO_LEXER performs auto detection. If one language is specified, then the language is set manually and AUTO_LEXER does not perform auto detection. If more than one language is specified, then AUTO_LEXER performs auto detection but limits the detected language to be among the language set. Note: The automatic detection of language is statistically based and, thus, inherently imperfect.
deriv_stems	NO (disabled)	Specifies whether the derivational stemming should be used or not. Currently, derivational stemming is only available for English. Hence, the DERIV_STEMS has no effect in other languages. Also, when derivational stemming is performed, tagging and tag stemming is not used. As a result, the tagging and tagged stemming client dictionary has no effect on the stemming result.
deriv_stems	YES (default)	Specifies whether the derivational stemming should be used or not. Currently, derivational stemming is only available for English. Hence, the DERIV_STEMS has no effect in other languages. Also, when derivational stemming is performed, tagging and tag stemming is not used. As a result, the tagging and tagged stemming client dictionary has no effect on the stemming result.
german_decompound	NO (disabled)	Specifies whether German de-compounding should be performed in the stemmer or not.
german_decompound	YES (default, enabled for German only)	Specifies whether German de-compounding should be performed in the stemmer or not.
index_stems	NO (disabled)	Specifies whether an index stemmer should be used. If specified as YES, then the stemmer that corresponds to the document language will be used and the stemmer will always be configured to maximize document recall. Note that this means that the stemmer attribute of BASIC_WORDLIST will be ignored, and the stemmer used by the AUTO_LEXER will be used during query to determine the stem of the given query term. If specified as NO, then queries with stem operators will use the word list stemming to try to stem the tokens. If word list stemming is not available, then the stem operator will be ignored. For documents in Swedish and Dutch, if the index_stems is set to YES, then compound word stemming will automatically be performed, and compounds are always separated into their component stems.
index_stems	YES (default)	Specifies whether an index stemmer should be used. If specified as YES, then the stemmer that corresponds to the document language will be used and the stemmer will always be configured to maximize document recall. Note that this means that the stemmer attribute of BASIC_WORDLIST will be ignored, and the stemmer used by the AUTO_LEXER will be used during query to determine the stem of the given query term. If specified as NO, then queries with stem operators will use the word list stemming to try to stem the tokens. If word list stemming is not available, then the stem operator will be ignored. For documents in Swedish and Dutch, if the index_stems is set to YES, then compound word stemming will automatically be performed, and compounds are always separated into their component stems.
base_letter	NO (disabled)	Specify whether characters that have diacritical marks (umlauts, cedillas, acute accents, and so on) are converted to their base form before being stored in the Text index.
base_letter	YES (enabled)	Specify whether characters that have diacritical marks (umlauts, cedillas, acute accents, and so on) are converted to their base form before being stored in the Text index.
base_letter_type	GENERIC (default)	The GENERIC value is the default and means that base letter transformation uses one transformation table that applies to all languages.
base_letter_type	SPECIFIC	The GENERIC value is the default and means that base letter transformation uses one transformation table that applies to all languages.
override_base_letter	TRUE FALSE (default)	When base_letter is enabled at the same time as alternate_spelling, it is sometimes necessary to override base_letter to prevent unexpected results from serial transformations.
mixed_case	NO (disabled)	Specify whether the lexer leaves the tokens exactly as they appear in the text or converts the tokens to all uppercase. The default is NO (tokens are converted to all uppercase).
mixed_case	YES (enabled)	Specify whether the lexer leaves the tokens exactly as they appear in the text or converts the tokens to all uppercase.
alternate_spelling	GERMAN (German alternate spelling)	Specifies whether German alternate spelling should be used or not.
alternate_spelling	SWEDISH (Swedish alternate spelling)	Specifies whether Swedish alternate spelling should be used or not.
alternate_spelling	NONE (No alternate spelling, default)	The default is NONE. No alternate spelling is specified.
printjoins	characters	Specify the non alphanumeric characters that, when they appear anywhere in a word (beginning, middle, or end), are processed as alphanumeric and included with the token in the Text index. This includes printjoins that occur consecutively. See Basic Lexer “printjoins”.
skipjoins	characters	Specify the non-alphanumeric characters that, when they appear within a word, identify the word as a single token; however, the characters are not stored with the token in the Text index. See Basic Lexer “skipjoins”.