TOKENS

Use this procedure to identify all text tokens in a document. The tokens returned are those tokens that are inserted into the index.

Thesaurus support also enables you to generate synonyms or broader terms of the queried index tokens. This feature is useful for implementing document classification, routing, or clustering.

Stopwords are not returned. Section tags are not returned because they are not text tokens.

Syntax 1: In-Memory Table Storage

CTX_DOC.TOKENS(index_name      IN VARCHAR2,
               textkey         IN VARCHAR2,
               restab          IN OUT NOCOPY TOKEN_TAB,
               thes_name       IN VARCHAR2 DEFAULT NULL,
               thes_toktype    IN VARCHAR2 DEFAULT 'SYN',
               use_saved_copy  IN NUMBER DEFAULT CTX_DOC.SAVE_COPY_FALLBACK);

Syntax 2: Result Table Storage

CTX_DOC.TOKENS(index_name      IN VARCHAR2,
               textkey         IN VARCHAR2,
               restab          IN VARCHAR2,
               thes_name       IN VARCHAR2 DEFAULT NULL,
               thes_toktype    IN VARCHAR2 DEFAULT 'SYN',
               query_id        IN NUMBER DEFAULT 0,
               use_saved_copy  IN NUMBER DEFAULT CTX_DOC.SAVE_COPY_FALLBACK);

index_name

Specify the name of the index for the text column.

textkey

Specify the unique identifier (usually the primary key) for the document.

The textkey parameter can be as follows:

• A single column primary key value

• Encoded specification for a composite (multiple column) primary key. To encode a composite textkey, use the CTX_DOC.PKENCODE procedure.

• The rowid of the row containing the document

Toggle between primary key and rowid identification using CTX_DOC.SET_KEY_TYPE.

restab

You can specify that this procedure store results to either a table or to an in-memory PL/SQL table.

The tokens returned are those tokens that are inserted into the index for the document (or row) named with textkey. Stop words are not returned. Section tags are not returned because they are not text tokens.

thes_name

Specify the thesaurus name. If you do not specify a thesaurus name, then no synonyms or broader terms will be generated. To use the system default thesaurus, specify DEFAULT.

If you specify thes_name, then the token table must include the THES_TOKENS column, otherwise the CTX_DOC.TOKENS procedure fails with an “ORA-00904: THES_TOKENS: Invalid identifier when thes_name parameter is used” error.

thes_toktype

Specify SYN to generate synonyms of index tokens. Alternatively, specify BT to generate broader terms of index tokens. By default, synonyms are generated. To use this parameter, you must first specify a thesaurus name using the thes_name parameter.

Specifying a Token Table

To store results to a table, specify the name of the table. Token tables can be named anything, but must include the columns shown in the following table, with names and datatypes as specified.

Table 2 Required Columns for Token Tables

Column Name	Type	Description
QUERY_ID	NUMBER	The identifier for the results generated by a particular call to CTX_DOC.TOKENS (only populated when table is used to store results from multiple TOKEN calls)
TOKEN	VARCHAR2(255)	The token string in the text.
THES_TOKENS	VARCHAR2(4000)	Synonyms or broader terms generated using a thesaurus for the token in the TOKEN column. These values must be colon-separated. Note: The THES_TOKENS column is required only if you specify the thes_name argument with the CTX_DOC.TOKENS API.
OFFSET	NUMBER	The position of the token in the document, relative to the start of document which has a position of 1.
LENGTH	NUMBER	The character length of the token.

Specifying an In-Memory Table

To store results to an in-memory table, specify the name of the in-memory table of type TOKEN_TAB. The TOKEN_TAB datatype is defined as follows:

type token_rec is record (

token varchar2(255),
offset number,
length number

);

type token_tab is table of token_rec index by binary_integer;

CTX_DOC.TOKENS clears the TOKEN_TAB you specify before the operation.

query_id

Specify the identifier used to identify the row(s) inserted into restab.

use_saved_copy

Specify whether to refer to the $D table to fetch the copy of the document, and what action to take when the copy of the document is not available in the $D table.

You can specify one of the following values for the use_saved_copy parameter:

• CTX_DOC.SAVE_COPY_FALLBACK: Fetch the copy of the document from the $D table. If the copy of the document is not present in the $D table, then fetch the document from the data store.

• CTX_DOC.SAVE_COPY_ERROR: Fetch the copy of the document from the $D table. If the copy of the document is not present in the $D table, then show an error message. Specify this value when you want to implement a specific fallback logic when the copy of the document is not available in the $D table.

• CTX_DOC.SAVE_COPY_IGNORE: Always fetch the document from the data store.

The default value is CTX_DOC.SAVE_COPY_FALLBACK.

Example

In-Memory Tokens

The following example generates the tokens for document 1 and stores them in an in-memory table, declared as the_tokens. The example then loops through the table to display the document tokens.

declare
 the_tokens ctx_doc.token_tab;

begin
 ctx_doc.tokens('myindex','1',the_tokens);
 for i in 1..the_tokens.count loop
  dbms_output.put_line(the_tokens(i).token);
  end loop;
end;