AUTO_LEXER Language-Independent Attributes

These are the language-independent attributes that are supported for the AUTO_LEXER component.

Table 15 AUTO_LEXER Language-Independent Attributes

Attribute Attribute Value Description
language characters (space-delimited string)

Specifies the possible languages of the input documents.

If no language is specified, then AUTO_LEXER performs auto detection.

If one language is specified, then the language is set manually and AUTO_LEXER does not perform auto detection.

If more than one language is specified, then AUTO_LEXER performs auto detection but limits the detected language to be among the language set.

Note: The automatic detection of language is statistically based and, thus, inherently imperfect.

deriv_stems

YES (default)

NO (disabled)

Specifies whether the derivational stemming should be used or not. Currently, derivational stemming is only available for English. Hence, the DERIV_STEMS has no effect in other languages.

Also, when derivational stemming is performed, tagging and tag stemming is not used. As a result, the tagging and tagged stemming client dictionary has no effect on the stemming result.
german_decompound

YES (default, enabled for German only)

NO (disabled)

Specifies whether German de-compounding should be performed in the stemmer or not.
index_stems

YES (default)

NO (disabled)

Specifies whether an index stemmer should be used.

When set to YES, compound word stemming is automatically performed and compounds are always separated into their component stems. The stemmer that corresponds to the document language is used and the stemmer is always configured to maximize document recall. Note that this means that the stemmer attribute of BASIC_WORDLIST is ignored, and the stemmer used by the AUTO_LEXER is used during query to determine the stem of the given query term.

When set to NO, queries with stem operators use the word list stemming to stem the tokens. If word list stemming is not available, then the stem operator is ignored.

base_letter

YES (enabled)

NO (disabled)

Specify whether characters that have diacritical marks (umlauts, cedillas, acute accents, and so on) are converted to their base form before being stored in the Text index.
base_letter_type

SPECIFIC

GENERIC (default)

The GENERIC value is the default and means that base letter transformation uses one transformation table that applies to all languages.
override_base_letter

TRUE

FALSE (default)

When base_letter is enabled at the same time as alternate_spelling, it is sometimes necessary to override base_letter to prevent unexpected results from serial transformations.
mixed_case

YES (enabled)

NO (disabled)

Specify whether the lexer leaves the tokens exactly as they appear in the text or converts the tokens to all uppercase. The default is NO (tokens are converted to all uppercase).
alternate_spelling

GERMAN (German alternate spelling)

SWEDISH (Swedish alternate spelling)

NONE (No alternate spelling, default)

Specifies whether alternate spelling should be used or not.

The default is NONE. No alternate spelling is specified.

printjoins characters Specify the non alphanumeric characters that, when they appear anywhere in a word (beginning, middle, or end), are processed as alphanumeric and included with the token in the Text index. This includes printjoins that occur consecutively. See Basic Lexer ““printjoins””.
skipjoins characters Specify the non-alphanumeric characters that, when they appear within a word, identify the word as a single token; however, the characters are not stored with the token in the Text index. See Basic Lexer ““skipjoins””.
composite

YES (default)

NO

Specify whether compound word stemming is enabled or disabled for the supported languages text. The default value is YES (compound word stemming enabled). You can use this feature for all languages that are supported for AUTO_LEXER.

When set to NO, words that are usually one entry in a dictionary are not split into composite stems, while words that are not dictionary entries are split into composite stems.

To retrieve the indexed composite stems, you must enter a stem query. For example, $bahnhof in German. The language of the wordlist stemmer must match the language of the composite stems.
timeout number

Specify the timeout value in seconds for auto_lexer tokenization.

Use a number between 0 and 600. The default value is 300.

Related Topics