Skip Headers

Oracle® Ultra Search User's Guide
10g Release 1 (10.1)

Part Number B10731-01
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Index
Index
Go to Master Index
Master Index
Go to Feedback page
Feedback

Go to previous page
Previous
Go to next page
Next
View PDF

9 Oracle Ultra Search Developer's Guide and API Reference

This chapter explains the Oracle Ultra Search APIs and related information. This chapter contains the following topics:

9.1 Overview of Oracle Ultra Search APIs

Oracle Ultra Search provides the following APIs:

Oracle Ultra Search also includes highly functional query applications to query and display search results. The query applications are J2EE-compliant Web applications.

9.2 Oracle Ultra Search Query API

Oracle Ultra Search provides a Java API for querying indexed data. The API methods retrieve and display query results. Because it is written in Java, it is compatible with a large spectrum of Web application servers that support any Java-based technology, such as JSP version 1.1 and higher. The API uses JDBC connection pooling for scalability.

The Java API does not impose any HTML rendering elements. The application can completely customize the HTML interface. For example:

You embed Oracle Ultra Search query functionality in your Web application with the supplied Oracle Ultra Search Java query API. The API supports two methods:

  1. Methods that retrieve query result data only.

  2. Methods that retrieve HTML code containing query result data.

The data-only methods do not return any HTML and can be used when you require full control over the HTML code to be rendered. The methods that retrieve HTML code support features such as allowing you to embed query input boxes and result lists in your Web application.

Some features of the Oracle Ultra Search Java query API include the following:

9.3 Customizing the Query Syntax Expansion

Oracle Ultra Search uses the Oracle Text engine to index and search documents. When an end user specifies a certain query string, Oracle Ultra Search takes that string and transforms it into an Oracle Text query expression. This process is called query syntax expansion.

You can customize Oracle Ultra Search to use your own implementation of the query syntax expansion.

The default query expansion lets you specify a query syntax similar to most internet search engines. The syntax boosts scores for documents that match the user's query in the document 'title' string attribute. The syntax for Contains is the same when used on the document content and on string attributes.

The default query syntax expansion is implemented in the oracle.ultrasearch.query.Contains class. To customize query expansion, use the oracle.ultrasearch.query.CtxContains class.

This section describes the default query expansion rules, and how to customize the query syntax expansion to suit your organization's preferences.

9.3.1 Default Query Syntax Expansion Implementation

The default query syntax expansion implementation directly affects the following:

  • The way the end user enters a query string (known as the end user query syntax)

  • The way the documents matching the query are scored (known as scoring)

  • The way the end user's query string is transformed into an Oracle Text query string (known as the expansion rules)

The default query syntax expansion is implemented in the oracle.ultrasearch.query.Contains class. The sample query applications makes use of this syntax expansion for content search as well as string attribute search.

9.3.1.1 End User Query Syntax

The end user query syntax defined by the default query syntax expansion implementation is similar to the standard text query syntax employed by most search engines on the Web.

  • Token: A token is a string enclosed in double-quotes ("). It can be a single word or a phrase.

  • Operators: The default implementation defines three operators. They are the [+], [-] and [*] operators. These operators are defined by the default implementation. Change these operators to whatever you prefer in your own custom implementation.

    The plus operator [+] specifies that the token immediately following it must appear in all documents included in the search result.

    The minus operator [-] specifies that the token immediately following it cannot appear in any document included in the search result.

    The asterisk [*] specifies a wildcard search. It matches zero or more characters. A token starting with the asterisk is ignored. The asterisk can only be specified at the end (right side) or middle of a token. For example, "hel*o" and "hell*" use the asterisk correctly, but "*ello" is unacceptable.

The following table summarizes the rules for the Oracle Ultra Search end user query syntax:


Note:

All end-user query strings are encased in square braces. For example, the end user query string Oracle Applications is notated as [Oracle Applications].

Rule Description
Single word search Entering one word finds documents that contain that word.

For example, searching for [Oracle] finds all documents that contain the word "Oracle" anywhere in that document.

Note: Searching for [Oracle] is not equivalent to [Oracle*].

Multiple word search Entering more than one word finds documents that each contain any of those words in any order.

For example, searching for [Oracle Applications] finds documents that contain "Oracle" or "Applications" or "Oracle Applications."

Compulsory inclusion [+] Attaching a [+] in front of a word requires that the word be found in all matching documents.

For example, searching for [Oracle + Applications] only finds documents that contain the word "Applications." Note: In a multiple word search, you can attach a [+] in front of every token including the very first token.

Compulsory exclusion [-] Attaching a [-] in front of a word requires that the word must not be found in all matching documents.

For example, searching for [Oracle - Applications] only finds documents that do not contain the word "Applications". Note: In a multiple word search, you can attach a [-] in front of every token except the very first token.

Phrase matching ["..."] Putting quotes around a set of words only finds documents that contain that precise phrase.

For example, searching for ["Oracle Applications"] finds only documents that contain the string "Oracle Applications."

Wildcard matching [*] Attaching a [*] to the right-hand side of a word returns left side partial matches.

For example, searching for the string [Ora*] finds documents that contain all words beginning with "Ora," such as "Oracle" and "Orator." You can also insert an asterisk in the middle of a word. For example, searching for the string [A*e] retrieves documents that contain words such as "Apple", "Ate", "Ape", and so on. Wildcard matching requires more computational processing power and is generally slower than other types of queries.


9.3.1.2 Scoring Classes

There are three ways documents are matched against an end user query string. These three ways are known as scoring "classes." Documents are scored and ranked higher if they satisfy the requirements for a higher class. Within each class, documents are also ranked differently depending on how well they match the conditions of that scoring class.

Class 1 is the most heavily weighted class. The score is derived from the number of occurrences of a precise phrase in a document. A document that has more instances of the precise phrase have a higher score than another document that has fewer occurrences of the precise phrase.

Class 2 is the next more heavily weighted class. In this class, the closer the tokens appear in a document, the higher the score becomes. For example, an end user query string [Oracle Applications Financials] can result in three documents found. None of the three documents contain the precise phrase "Oracle Applications Financials." However, document X contains the all three tokens "Oracle", "Applications", and "Financials" in the same sentence separated by other words. Document Y contains the individual tokens in the same paragraph but in different sentences. Document Z contains the same three tokens, but each token resides in different paragraphs. In this scenario, document X has the highest score, because the tokens are closest together. Likewise, Y has a higher score than Z.

Class 3 is the least weighted class. A document that has more tokens gets a higher score. For example, an end user query string [Oracle Applications Financials] can result in three documents found. Document X might contain all three tokens. Document Y might contain the tokens "Oracle" and "Applications" only. Document Z might contain only the token "Oracle." In this scenario, document X has a higher score than Y. Likewise, Y has a higher score than Z.

9.3.1.3 Expansion Rules

As mentioned previously, the end user query is expanded to an Oracle Text query. The expanded query string rules are captured in BNF (Backus Naur Form) notation. Again, these rules are the rules that Oracle Ultra Search uses as a default query syntax expansion implementation.

The rules that define an expanded query:

<expanded query> ::= (<expression> within <title section>)*2, <expression>

<expression> ::= <generic query expression> | <simple query expression>

<generic query expression> ::= (([ <plus expression>*100 & ]) (<main expression>)) [ <minus expression> ]

<simple query expression> ::= (<phrase expression>)*2, (<main expression>)

<main expression> ::= (<near expression>)*2, (<accum expression>)

The following list contains some terms and their meanings, which explain some of the terms used in the preceding rules:

A <plus expression> is an AND expression of all plus tokens.

A <minus expression> is a NOT expression of all minus tokens.

A <phrase expression> is a PHRASE formed by all tokens in the <main expression>

A <near expression> is a NEAR expression of all tokens but minus tokens.

An <accum expression> is an ACCUMULATE expression of all tokens but minus tokens.

A <simple query expression> is used only when the end user query has multiple tokens and does not have any operator or a double quote. Otherwise, a <generic query expression> is used.

If there is no token that is neither plus token nor minus token, then the <plus expression> and the <accum expression> are eliminated.

9.3.1.4 Examples of Applying the Rules

The following table illustrates how the default query syntax expansion implementation converts end user query strings into Oracle Text compatible query strings.

End User Query String Expanded Query String Understandable by Oracle Text
[Oracle]
((({Oracle}) within TITLE__31)*2,({Oracle}))
[Oracle + Applications]
((((({Applications})*10)*10&(({Oracle};{Applications})*2,({Oracle},{Applications
}))) within TITLE__31)*2,((({Applications})*10)*10&(({Oracle};{Applications})*2,
({Oracle},{Applications}))))  
[Oracle - Applications]
(((({Oracle})~{Applications}) within TITLE__31)*2,(({Oracle})~{Applications}))
["Oracle Applications"]
((({Oracle Applications}) within TITLE__31)*2,({Oracle Applications}))
[Ora*]
((((Ora%)) within TITLE__31)*2,((Ora%)))
[Oracle Applications]
(((({Oracle Applications})*2,(({Oracle};{Applications})*2,({Oracle},{Application
s}))) within TITLE__31)*2,(({Oracle Applications})*2,(({Oracle};{Applications})*
2,({Oracle},{Applications}))))   

9.3.2 Customizing the Rules

Customize this expansion to suit your organization's purposes by defining and implementing your own query syntax expansion. You should have detailed understanding of Oracle Text queries using the ctxsys.contains() operator. Oracle Text offers a rich set of linguistic features, such as thesaurus, theme, stemming, and soundex as a part of its query language.

To customize Oracle Ultra Search to use your own implementation of the query syntax expansion, use the oracle.ultrasearch.CtxContains class in your query application instead of the oracle.ultrasearch.query.Contains class. CtxContains lets you use any Oracle Text query as a part of an Oracle Ultra Search query. Use the following steps:

  1. Construct a Oracle Text query based on the user's input. For example, if the user's input is "cat", using the stemming feature, you can construct a Text query "$cat", which will find documents with "cat" or "cats". You can use any tool to construct the Text query, as long as it is a string object. Depending on the complexity of user's query syntax, you might want to leverage some existing lexers in Java.

  2. Construct a CtxContains using the Text query. For example:

    String textQuery = "$cat";
    oracle.ultrasearch.Query query = new oracle.ultrasearch.CtxContains (textQuery);
    
    

    The preceding code constructs a query for documents with "cat" or "cats". You can also limit that query to document titles (not content) as follows:

    String textQuery = "cat";
    StringAttribute titleAttribute = instanceMetaData.getStringAttribute("TITLE");
    oracle.ultrasearch.Query query = new oracle.ultrasearch.CtxContains (textQuery, titleAttribute);
    
    
  3. You can optionally combine the CtxContains with any other Oracle Ultra Search query by joining them with the And/Or query operators.

  4. Run the query by invoking the getResult() method with the constructed query object.


See Also:

Oracle Ultra Search Java API Reference for detailed information on the oracle.ultrasearch.query.CtxContains API

9.4 Oracle Ultra Search Query Tag Library

On top of the Java query API, Oracle Ultra Search provides a JSP tag library as an alternative for developing search applications. Based on the Sun Microsystems JavaServer Pages specification version 1.1, the Oracle Ultra Search tag library better separates the dynamic/Java development effort from the static/HTML development effort, and enables Web developers who are unfamiliar with Java to incorporate search functionality into their applications.

The Oracle Ultra Search tag library provides a subset of the features in the Java Query API. Advanced features, such as custom query expansion and URL submission, are not available as tags. The main features of the tag library are the following: ability to retrieve search attributes, groups, languages, and LOVs for rendering the advance query form; and ability to iterate through the resulting hit set, and retrieve document attributes and properties for rendering the result page.

The tag library is summarized in following table:

Tag Description Attributes
instance This tag establishes a connection to an Oracle Ultra Search instance. instanceId

username

password

URL

dataSourceName

tablePagePath

emailPagePath

filePagePath

showAttributes For an advanced query, use this tag to show the list of attributes available. instance

locale

showGroups For an advanced query, use this tag to show the list of groups. instance

locale

showLanguages For an advanced query, use this tag to show the list of languages defined in the instance. instance
showLOV Show all values defined for a search attribute. instance

locale

attributeName

attributeType

getResult Perform the search. resultId

instance

query

queryLocale

documentLanguage

from

to

boostTerm

withCount

fetchAttribute This is a nested tag within getResult to specify which attributes of each document should be fetched along with the query results. There can be any number of nested fetchAttribute tags. attributeName

attributeType

showHitCount If withCount="true" in the getResult tag, then the result includes a total number of hits, and you can use showHitCount to display this number. result
showResults Renders the results of the search. result

instance

showAttributeValue Renders a document attribute. attributeName

attributeType


Details of these tags are described in the following subsections. Note the following requirements for using Oracle Ultra Search tags:

The Oracle Ultra Search tag library definition (TLD) file can be found in $ORACLE_HOME/ultrasearch/sample/query/WEB-INF/ultrasearch-taglib.tld after sample.ear has been deployed. It is also packaged with ultrasearch_query.jar under the name META-INF/taglib.tld.

9.4.1 Query Tag Descriptions

The following section describes each Oracle Ultra Search tag, its attributes, and action. Examples are shown without any static HTML, which can be inserted to format the output.

9.4.1.1 <instance> Tag: Connecting to the Oracle Ultra Search Instance

This tag establishes a connection to an Oracle Ultra Search instance. Some basic parameters must be established for this tag to work, such as JDBC connection string, schema user name/password, Oracle Ultra Search instance name, and so on.

Attribute Name Description
instanceId="name" Names the instance defined by this tag. This name is then used by other Oracle Ultra Search tags to specify the instance being searched.
username Creates a database connection.
password Creates a database connection.
url Gets the URL used to create a JDBC connection. This attribute is optional if dataSourceName is specified.
dataSourceName The JNDI name that identifies a JDBC data source. Users should set either the URL or data source name properties. This is optional if URL is specified.
instanceName The name of the Oracle Ultra Search instance that is owned by the schema user. If the schema user owns only one Oracle Ultra Search instance, then this is optional.
tablePagePath The URL path of the Web application that renders the contents of a database table.
emailPagePath The URL path of the Web application that renders the contents of an email.
filePagePath The URL path of the Web application that renders the contents of a file.

This tag defines a scripting variable of the name set by the instanceId property. All the other tag properties correspond to a property in the oracle.ultrasearch.query.QueryInstance class. Either the URL or the dataSourceName attribute should be set. They are exclusive of each other.

The following example uses the URL property to connect to the database.

<US:instance 
 instanceId="mybookstore"
 url="oracle:jdbc:thin:@dbhost:1521:inst1"
 username="scott"
 password="tiger"
 tablePage="../display.jsp"
 emailPage="../mail.jsp"
 filePage="../display.jsp"
/>

9.4.1.2 <iterAttributes> Tag: Show All Search Attributes

When a user wants to perform an advanced query, the application needs to show the list of attributes that are available, the list of groups, and the list of languages defined in the instance. This can be done using some iteration tags that define script variables for page rendering.

Each attribute in Oracle Ultra Search has a name, a type, and a display name that is translated depending on the locale that is set for the QueryInstance tag. The attribute type should be used to determine which operators can be used on this attribute and how to parse the user's input.

Attribute Name Description
instance="name" This is a mandatory attribute to refer to the object defined by the instance tag.
locale="locale" This determines the display name fetched using this tag.

This tag is an iteration tag. It loops through all the search attributes in the instance referred to by the instance tag attribute. In each loop, it defines a scripting variable named "attribute", which is an oracle.ultrasearch.query.Attribute object. It also defines a string variable named "displayname", which is the localized name of the attribute.

The following example shows all the attributes in "mybookstore" instance, using their English display names.

<US:iterAttributes instance="mybookstore" locale="<%=Locale.ENGLISH%>" >
<%= attribute %>
<%= displayname %>
</US:iterAttributes>

9.4.1.3 <iterGroups> Tag: Show All Search Groups

Similar to the showAttributes tag, the showGroups tag iterates through all the groups defined in an instance.

Attribute Name Description
instance="name" This is a mandatory attribute to refer to the object defined by the instance tag.
locale="locale" This determines the display name fetched using this tag.

This tag loops through all the search groups in the instance referred to by the instance tag attribute. In each loop, it defines a scripting variable named "group", which is an oracle.ultrasearch.query.Group object. It also defines a string variable named "displayname", which is the localized name of the group.

The following example shows all the groups in "mybookstore" instance, using their English display names.

<US:iterGroups instance="mybookstore" locale="<%=Locale.ENGLISH%>" >
<%= group %>
<%= displayname %>
</US:iterGroups >

9.4.1.4 <iterLanguages> Tag: Show All Search Languages

Similar to the showAttributes tag, the showLanguages tag iterates through all the languages defined in an instance. Because each language is defined by a java.util.Locale object, their display names are not handled by Oracle Ultra Search. Therefore, this tag does not define the displayname scripting variable.

Attribute Name Description
instance="name" This is a mandatory attribute to refer to the object defined by the instance tag.

This tag is an iteration tag. It loops through all the search languages in the instance referred to by the instance tag attribute. In each loop, it defines a scripting variable named "language", which is a java.util.Locale object. The display name for the language is provided by Java as a property of the object itself (through the getDisplayName() method).

The following example shows all the languages in "mybookstore" instance, using their English display names.

<US:iterLanguages instance="mybookstore">
<%= language %>
<%= language.getDisplayName (Locale.ENGLISH) %>
</US:iterLanguages >

9.4.1.5 <iterLOV> Tag: Show All Values Defined for a Search Attribute

Attribute Name Description
instance="name" This is a mandatory attribute to refer to the object defined by the instance tag.
locale="locale" This determines the display name fetched using this tag.
attributeName="attname" The name of the attribute whose LOV is being fetched in this LOV.
attributeType="string | number | date" The type of the attribute whose LOV is being fetched in this LOV. This is needed because attribute name does not uniquely identify an attribute in the instance.

This tag is an iteration tag. It loops through all the values in a search attribute's LOV. In each loop, it defines a scripting variable named "value", which is either a java.lang.String, java.util.Date, or java.math.BigDecimal object, depending on the attribute type. It also defines a string variable named "displayname", which is the localized display name of the value.

The following example shows all the values for a string attribute named "Dept" in "mybookstore" instance, using their English display names.

<US:iterLOV instance="mybookstore" attribute_name="Dept" attribute_type="String" >
<%= value %>
<%= displayname %>
</US:iterLOV >

9.4.1.6 Formulating the Query

Oracle Ultra Search supports a set of classes for building queries. Currently these classes do not have any tag equivalents.

9.4.1.7 <getResult> Tag: Perform Search

This tag performs the search and returns the result by defining a scripting variable of the type oracle.ultrasearch.query.Result.

Attribute Name Description
resultId="name" This names the result generated by this tag. This name is then used by other tags to render the result on the page.
instance="name" This is a mandatory attribute to refer to the object defined by the instance tag.
query="<%= expression %>" This specifies a query object to search with.
queryLocale="locale" This specifies the locale of the query object.
documentLanguage="locale" This specifies the language of the documents for which to search. This is optional. If it is not specified, then all languages are included in the search.
from="number" This specifies the index of the first hit.
to="number" This specifies the index of the last hit.
boostTerm="string" This specifies the search term that is used for relevance boosting. This is optional.
withCount="true | false" This specifies whether the result has an estimate of the total hit count. This is optional. If unspecified, the behavior is same as withCount=false.

The <getResult> tag corresponds to the getResult() method on the oracle.ultrasearch.query.Instance class. The attributes of tag map to the parameters of the method, with the exception that getResult() method can specify the attributes to fetch. The <getResult> tag require the use of the nested <fetchAttribute> tag to accomplish metadata selection.

The following example shows a search for the first 20 documents of a query in English that appears in French documents.

<US:getResult 
 resultId="searchresult"
 instance="mybookstore"
 query=""
 queryLocale=""
 documentLanguage=""
 from="1" to="20">
</US:getResult>

9.4.1.8 <fetchAttribute> Tag: Metadata Selection

This tag is used as nested tag inside <getResult>. It specifies which attributes of each document should be fetched along with the query result. Each <getResult> can have any number of nested <fetchAttribute> tags.

Attribute Name Description
attributeName="attname" The name of the attribute whose LOV is being fetched in this LOV.
attributeType="string | number | date" The type of the attribute whose LOV is being fetched in this LOV. This is needed because attribute name does not uniquely identify an attribute in the instance.

Each occurrence of the <fetchAttribute> adds to the list of attributes passed to the getResult() invoked by the <getResult> tag.

The following example shows the same search in <getResult> tag, but fetching title and publication-date attributes of each book.

<US:getResult 
 resultId="searchresult"
 instance="mybookstore"
 query=""
 queryLocale=""
 documentLanguage=""
 from="1" to="20">
<US:fetchAttribute 
 attributeName="title"
 attributeType="string" />
<US:fetchAttribute 
 attributeName="publication-date"
 attributeType="date" />
</US:getResult>

9.4.1.9 <showHitCount> Tag: Show Estimated Hit Count

After the search is performed, the result must be rendered. If withCount=true is in the <US:getResult> tag, then the result contains a count of total hits, and <showHitCount> tag can be used to display it.

Attribute Name Description
result="name" This refers to the resultId specified in the <US:getResult> tag.

This tag outputs the hit count to the page.

The following shows the hit count of the a search result.

<US:showHitCount result="searchresult" />

9.4.1.10 <iterResult> Tag: Render the Results

This tag is an iteration tag. It loops through all the documents in a search result.

Attribute Name Description
result="name" This refers to the resultId specified in the <US:getResult> tag.
instance="name" This refers to the instanceId specified in the <US:instance> tag.

The tag loops through all the documents in a search result and defines a scripting variable "doc" that is a oracle.ultrasearch.query.Document object. In addition, it can have nested tags of <showAttributeValue>, which helps to render the document's attributes. It is an error if the result specified is not one obtained from search on the instance specified. In other words, the result must come from the instance.

The following example shows the URL of all documents in a search result.

<US:iterResult
result="searchresult" 
instance="mybookstore">
</US:iterResult>

9.4.1.11 <showAttributeValue> Tag: Render a Document Attribute

This tag shows an attribute of a document within the <US:iterResult> tag.

Attribute Name Description
attributeName="attname" The name of the document attribute.
attributeType="string | number | date" The type of the document attribute. This is needed because attribute name does not uniquely identify an attribute in the instance.
default="default string" A value to output when the document has no value for this attribute. This is useful when a document has no title. The string "No Title" can be displayed as the default value.

This tag looks up the document attribute value and renders it on the page. If the attribute was not fetched as part of the search result, then nothing is output to the page.

The following example shows the title and publication dates of all documents in a search result.

<US:iterResult
result="searchresult" 
instance="mybookstore">
<US:showAttributeValue attributeName="title" attributeType="string" default="No Title" />
<US:showAttributeValue attributeName="publication-date" attributeType="date" />
</US:iterResult>

9.5 Oracle Ultra Search Crawler Agent API

You can implement a crawler agent to crawl and index a proprietary document repository, such as Lotus Notes or Documentum. In Oracle Ultra Search, the proprietary repository is called a user-defined data source. The module that enables the crawler to access the data source is called a crawler agent.

The agent collects document URLs and associated metadata from the user-defined data source and returns the information to the Oracle Ultra Search crawler, which enqueues it for later crawling. The crawler agent must be implemented in Java using the Oracle Ultra Search crawler agent API.

Oracle Ultra Search provides a sample implementation of user-defined crawler agents using the Oracle Ultra Search agent API. Upon invocation, this sample agent connects to a specified Oracle database and retrieves the contents of a table for the crawler to collect and index.

The sample agents are fully functional and can be customized to adapt to other database-based data sources. These agents performs the following tasks:

9.5.1 Crawler Agent Overview

A crawler agent does the following:

  • Authenticates the crawler for accessing the data source

  • Provides access to the data source document through a HTTP URL (display URL)

  • Provides the metadata of the document in the form of document attributes

  • Maps each document attribute to a common attribute name used by end users

  • Provides a "flattened" view of the data source, such that documents are retrieved one by one in a streaming fashion

  • Instructs the crawler to parse the URL document for standard metadata, like author and title, if necessary

  • Optionally provides the list of URLs that have changed since a given time stamp

  • Optionally provides an access URL in addition to the display URL for the processing of the document

From the crawler's perspective, the agent retrieves the list of URLs from the target data source and saves it in the crawler queue before processing it.


Note:

If the crawler is interrupted for any reason, then the agent invocation process is repeated with the original last crawl time stamp. If the crawler finished enqueuing URLs fetched from the agent and is half way done crawling, then the crawler only starts the agent, but does not try to fetch URLs from the agent. Instead, it finishes crawling the URLs already enqueued.

There are two kinds of crawler agents:

9.5.1.1 Standard Agent

The standard agent returns the list of URLs currently existing in the data source. It does not know whether any of the URLs had been crawled before, and it relies on the crawler to find any updates to the target data source. The standard agent's interaction with the crawler is the following:

  • Crawler marks all existing URLs of this data source for garbage collection, assuming they no longer exist in the target data source.

  • Crawler calls the agent to get an updated list of URLs. It marks for crawling every URL that already exists. If it is new, it inserts it into the URL table and queue.

  • Crawler deletes the URLs that are still marked for garbage collection.

  • Crawler goes through every URL marked for crawling and checks for updates.

9.5.1.2 Smart Agent

The smart agent uses a modified-since time stamp (provided by the crawler) to return the list of URLs that have been updated, inserted, and deleted. The crawler only crawls URLs returned by the agent and does not recrawl existing ones. For URLs that were deleted, the crawler removes them from the URL table. If the smart agent can only return updated or inserted URLs but not deleted URLs, then deleted URLs are not detected by the crawler. In this case, you must change the schedule crawler recrawl policy to periodically run the schedule in force recrawl mode. Force recrawl mode signals to the agent to return every URL in the data source.

The agent API isDeltaCrawlingCapable() tells the crawler whether the agent it invokes is a standard agent or a smart agent. The agent API startCrawling(boolean forceRecrawl, Date lastCrawlTime) lets the crawler tell the agent the last crawl time and whether the crawler is running in force recrawl mode.

9.5.1.3 Document Attributes and Properties

Document attributes, or metadata, describe document properties. Some attributes can be irrelevant to your application. The crawler agent creator must decide which document attributes should be extracted and saved. The agent also can be created such that the list of collected attributes are configurable. Oracle Ultra Search automatically registers attributes returned by the agent. The agent can decide which attributes to return for a document.

9.5.2 Crawler Agent Functionality

This section describes aspects of the crawler agent.

9.5.2.1 Data Source Type Registration

A data source type is an abstraction of a data source. You can define new data source types with the following attributes:

  • Name of data source type: For example, Lotus Notes. The name cannot be more than 100 bytes.

  • ID of data source type: This is automatically assigned.

  • Description of the data source type: This limit is 4000 bytes.

  • Agent Java class name: For example, WebDbAgent. The location of this class is predefined by Oracle Ultra Search in $ORACLE_HOME/ultrasearch/lib/agent/ and cannot be changed.

  • Agent Java jar file name: The agent class can be stored in a Java jar file. This jar file must be in $ORACLE_HOME/ultrasearch/lib/agent/, where $ORACLE_HOME is the Oracle home directory where the Oracle Ultra Search backend, not the middle tier, is installed.

  • Parameters: Parameters are the properties of a data source; for example, seed URL, inclusion pattern, and robots exclusion for a Web data source. Define a parameter by specifying a parameter name (100 bytes maximum) and a description (4000 bytes maximum). By default, a parameter is not encrypted.

  • Encryption: Should the value of this parameter be encrypted when stored.

Oracle Ultra Search does not enforce the occurrence of parameters. You cannot specify a particular parameter to have 0 or more, at least 1, or only 1 occurrence.

9.5.2.2 Data Source Registration

After a data source type is defined, any instance of that data source type can be defined:

  • Data source name

  • Description of the data source; limit to 4000 bytes

  • Data source type ID

  • Default language; default is 'en' (English)

  • Parameter values; for example, seed - http://www.oracle.com depth - 8

9.5.2.3 Data Source Attribute Registration

You can add new attributes to Oracle Ultra Search by providing the attribute name and the attribute data type. The data type can be string, number, or date. Attributes with the same name but different data type can be added. Attributes returned by an agent are automatically registered if they have not been defined.

9.5.2.4 User-Implemented Crawler Agent

The crawler agent has the following requirements:

  • The agent must be implemented in Java.

  • The agent must support the Java agent APIs defined by Oracle Ultra Search.

  • The agent must return the URL attributes and properties.

  • The agent optionally can authenticate the crawler's access to the data source.

  • The agent must "flatten" the data source such that each document is retrieved one by one in a streaming fashion. This is to encapsulate the crawling logic of a specific data source into the agent.

  • The agent must decide which document attributes Oracle Ultra Search should keep. Any attribute not defined in Oracle Ultra Search is registered automatically.

  • The agent can map attributes to data source properties. For example, if an attribute "ID" is the unique ID of a document, then the agent should return (document_key, 4) where "ID" has been mapped to the property "document_key" and its value is 4 for this particular document.

  • If the attribute LOV is available, then the agent returns them upon request.

9.5.2.5 Interaction Between the Crawler and the Crawler Agent

The crawler crawls data sources defined by the user through the invocation of the user-supplied crawler agent. The crawler can do the following:

  • Invoke the crawler agent of the defined data source

  • Supply data source parameter information to the agent

  • Authenticate itself with the agent if needed

  • Retrieve a list of URLs and associate attributes/properties that must be crawled

  • Use the URL provided by the agent to retrieve the document

  • Detect insert, update, and delete to the data source

  • Retrieve attribute LOV data if available

9.5.2.6 Crawler Agent APIs and Classes

The crawler agent API is a collection of methods used to implement a crawler agent. A sample implementation of a crawler agent SampleAgent.java is provided under $ORACLE_HOME/ultrasearch/extension/.

UrlData: The crawler agent uses this interface to populate document properties and attribute values. Oracle Ultra Search provides a basic implementation of this interface that the agent can use directly or extend if necessary. The class is DocAttributes with a constructor that has no argument. The agent might decide to create a pool of UrlData objects and cycle through them during crawling. In the most simple implementation, the agent creates one DocAttributes object, repeatedly resets and populates the data, and returns this object.

LovInfo: The crawler agent uses this interface to submit attribute LOV definitions.

DataSourceParams: The crawler agent uses this interface to read and write data source parameters.

AgentException: The crawler agent uses this exception class when an error occurs.

CrawlerAgent: This interface lets the crawler communicate with the user-defined data source. The crawler agent must implement this interface.

9.5.3 Sample Agent Files

The sample agent files are located in the $ORACLE_HOME/ultrasearch/extension directory. You can view the sample agent source code using your preferred text editor.

There is a SampleAgent_readme.htm file and a SampleAgent.java file. These are for the sample crawler agent implementation using agent APIs.

9.5.4 Setting up the Sample Crawler Agent

This section describes how to set up the sample crawler agent.

9.5.4.1 Compiling and Building the Agent Jar File

The Java source code for the sample agent first must be compiled into class files and put into a jar file in the $ORACLE_HOME/ultrasearch/lib/agent/ directory, where $ORACLE_HOME is the Oracle home directory where the Oracle Ultra Search backend, not the middle tier, is installed.

The classes needed for compilation are the JDK class (classes.zip), Oracle JDBC Thin Driver (classes12.zip), and ultrasearch.jar. For example:

javac -J-ms16m -J-mx96m -O -classpath /jdk1.2.2 05/lib/classes.zip:/lib/classes12.zip:
   $ORACLE_HOME/ultrasearch/lib/ultrasearch.jar SampleAgent.java

To build the SampleAgent.jar file, enter the following:

/jdk1.2.2_05/bin/jar cv0f /oracle/ultrasearch/lib/agent/SampleAgent.jar 
  SampleAgent.class 'SampleAgent$DocNode.class'

9.5.4.2 Creating a Data Source Type

A data source type that uses the sample agent must be created first.

  • Name: URL table type

  • Description: Table with rows of URLs

  • Agent Name: SampleAgent

  • Agent Jar File: sampleagent

9.5.4.3 Defining Data Source Parameters

Define parameters for a data source type:

  • Database Connect String (DB connection)

  • User Name (schema owner of the URL table)

  • Password (schema owner password, encrypted)

  • Table Name (URL table name)

  • URL Column (Column holding doc URLs)

  • Ignore Flag Column (1 for ignoring, 0 otherwise)

  • Language Column (Document Language)

  • Attribute List (List of column for attributes)

  • It is in the following format: [column name/attribute name] <data type> [column name/attribute name] <data type> ... where <data type> 0 is number, 1 is string, and 2 is date. For example, if the document has 4 attributes: Company Name, Category, Revenue, S&P Rating, then it is specified as: [Company Name/Company/1][Category/Classification/1][Revenue/Revenue/0][Rating/Analyst Rating/1]

  • Log File Name (log file)

  • Log Directory (Location of log file)

9.5.4.4 Defining a Data Source of this Type

A data source is defined, which initializes the data source parameters. For example, the value specified accesses a table whose schema is the following:

TABLE NEWS (
    ARTICLE_NO    NUMBER,
    NEWS_URL      VARCHAR2(740),
    TITLE         VARCHAR2(200),
    AUTHOR        VARCHAR2(100),
    PUB_DATE      DATE default SYSDATE,
    PUBLISHER     VARCHAR2(100),
    PRICE         NUMBER,
    LANG          VARCHAR2(10),
    IGNORE        NUMBER DEFAULT 0,
    PRIMARY KEY (NEWS_URL)
    );

  • Database Connect String: dlsun1710:5521:search

  • User Name: SCOTT

  • Password: TIGER

  • Table Name: NEWS

  • URL Column: NEWS_URL

  • Ignore Flag Column: IGNORE

  • Language Column: LANG

  • Attribute List: [ARTICLE_NO/Article Number/0][TITLE/Article Title/1][AUTHOR/Author/1][PUB_DATE/Report Date/2][PUBLISHER/Newspaper/1][PRICE/Download Cost/0]

  • Log File Name: testagent.log

  • Log Directory: /tmp/ultrasearch/

9.6 Oracle Ultra Search Java Email API

Oracle Ultra Search provides a Java API for accessing archived emails. The Oracle Ultra Search query application uses the API to display emails addressed to mailing lists that have been indexed by the Oracle Ultra Search system. The API can also be used to build your own custom query application.

The application user-interface logic is entirely controlled in the JSP. Therefore, you can customize the look-and-feel to your needs.

Email documents contain valuable information, but they are not structured to find specific relevant information easily. Oracle Ultra Search lets you retrieve and index emails on a server that supports the IMAP4 protocol.

An email source is a data source that derives its content from emails sent to a specific email address. When the Oracle Ultra Search crawler searches an email source, the crawler collects all emails that have the specific email address in any of the "To:" or "Cc:" email header fields.


Note:

Oracle Ultra Search stores copies of all retrieved emails in the local file system of the Oracle Ultra Search server installation.

A possible application of an email source is where an email source represents all emails sent to a mailing list. In such a scenario, multiple email sources are defined where each email source represents an email list.

Oracle Ultra Search email crawling and rendering is built on top of the JavaMail API using Sun Microsystems' reference implementation of JavaMail. This enables Oracle Ultra Search to provide a Java API for accessing indexed emails. The API is known as the Oracle Ultra Search Java Email API. This API lets you retrieve information such as email header information, email body content, and attachments of an email.

Use this API to embed Oracle Ultra Search email browsing functionality into JavaServer Page (JSP) or servlet-based Web applications. Oracle Ultra Search ships a fully functional JSP Web application that directly uses this API to render indexed emails. Because the source code is viewable, you can use it as an example for building your own customized email browser.

9.6.1 JavaMail Implementation

Oracle Ultra Search requires a JavaMail 1.1 compliant implementation. The reference implementation by Sun Microsystems is JavaMail version 1.2. This reference implementation is shipped with Oracle Ultra Search.

9.6.2 Java Email API

The Oracle Ultra Search Java Email API is encapsulated in the oracle.ultrasearch.query package.

9.6.3 Sample Mailing List Browser Application Files

The sample mailing list browser applications files are located in the $ORACLE_HOME/ultrasearch/sample/query directory. You can directly view the sample mailing list browser application source code using your preferred text editor.

The following tables describe all sample mailing list browser application files, README file, and stylesheets:

File Description
SampleAgent_readme.html 
Readme
mail.css
Style sheet for sample email Web application

Sample JavaServer Page Mailing List Browser Applications Files:

File Description
mail.jsp
Mailing list browser applications that selectively include HTML code returned by other JSP files, depending on what the end user wants to view
mailindex.jsp
JSP page that displays all email sources (mailing lists) of an Oracle Ultra Search instance
mailmsgs.jsp
JSP page that displays all emails for an email source (mailing list)
mailreader.jsp
JSP page that displays an email
mailutil.jsp
JSP page that defines various functions that are used by mailreader.jsp

Graphics Files for All Applications:

File Description
images/ultra_mediumbanner.gif
Oracle Ultra Search banner
images/wsd.gif
Background image used in sample query application

9.6.4 Setting up the Sample Mailing List Browser Application

For detailed instructions on setting up the sample JSP mailing list browser application, see "Installing the Oracle Ultra Search Middle Tier on Web Server Hosts".

9.7 Oracle Ultra Search URL Rewriter API

A URL rewriter is a user supplied Java module that implements the Oracle Ultra Search UrlRewriter Java interface. When activated, it is used by the crawler to filter and rewrite extracted URL links before they are inserted into the URL queue.

Web crawling generally consists of the following steps:

  1. Get the next URL from the URL queue. (Web crawling stops when the queue is empty.)

  2. Fetch the contents of the URL.

  3. Extract URL links from the contents.

  4. Insert the links into the URL queue.

The generated new URL link is subject to all existing host, path, and mimetype inclusion and exclusion rules.

There are two possible operations that can be done on the extracted URL link:

9.7.1 URL Link Filtering

Users control what type of URL links are allowed to be inserted into the queue with the following mechanisms supported by the Oracle Ultra Search crawler:

  • robots.txt file on the target Web site; for example, disallow URLs from the /cgi directory

  • Hosts inclusion and exclusion rules; for example, only allow URLs from www.acme.com

  • File path inclusion and exclusion rules; for example, only allow URLs under the /archive directory

  • Mimetype inclusion rules; for example, only allow HTML and PDF files

  • Robots metatag NOFOLLOW; for example, do not extract any link from that page

  • Black list URL; for example, URL explicitly singled out not to be crawled


    Note:

    All URLs must pass domain rules before being checked for path rules. Path rules let you further restrict the crawling space. Path rules are host-specific, but you can specify more than one path rule for each host. For example, on the same host, you can include path files://host/doc and exclude path files://host/doc/unwanted.

With these mechanisms, only URL links that meet the filtering criteria are processed. However, there are other criteria that users might want to use to filter URL links. For example:

  • Allow URLs with certain file name extensions

  • Allow URLs only from a particular port number

  • Disallow any PDF file if it is from a particular directory

The possible criteria could be very large, which is why it is delegated to a user-implemented module that can be used by the crawler when evaluating an extracted URL link.

9.7.2 URL Link Rewriting

For some applications, due to security reasons, the URL crawled is different from the one seen by the end user. For example, crawling is done on an internal Web site behind a firewall without security checking, but when queried by an end user, a corresponding mirror URL outside the firewall must be used.

A display URL is a URL string used for search hit display. This is the URL used when users click the search hit link. An access URL is a URL string used by the crawler for crawling and indexing. An access URL is optional. If it does not exist, then the crawler uses the display URL for crawling and indexing. If it does exist, then it is used by the crawler instead of the display URL for crawling.

For regular Web crawling, there are only display URLs available. But in some situations, the crawler needs an access URL for crawling the internal site while keeping a display URL for the external use. For every internal URL, there is an external mirrored one.

For example:

http://www.acme-qa.us.com:9393/index.html
http://www.acme.com/index.html

When the URL link http://www.acme-qa.us.com:9393/index.html is extracted and before it is inserted into the queue, the crawler generates a new display URL and a new access URL for it:

Access URL:

http://www.acme-qa.us.com:9393/index.html

Display URL:

http://www.acme.com/index.html

The extracted URL link is rewritten, and the crawler crawls the internal Web site without exposing it to the end user.

Another example is when the links that the crawler picks up are generated dynamically and can be different (depending on referencing page or other factor) even though they all point to the same page. For example:

http://compete3.acme.com/rt/rt.wwv_media.show?p_type=text&p_id=4424&p_currcornerid=281&p_textid=4423&p_language=us

http://compete3.acme.com/rt/rt.wwv_media.show?p_type=text&p_id=4424&p_currcornerid=498&p_textid=4423&p_language=us

Because the crawler detects different URLs with the same contents only when there is sufficient number of duplication, the URL queue could grow to a huge number of URLs, causing excessive URL link generation. In this situation, allow "normalization" of the extracted links so that URLs pointing to the same page have the same URL. The algorithm for rewriting these URLs is application dependent and cannot be handled by the crawler in a generic way.

When a URL link goes through a rewriter, there are the following possible outcomes:

  • The link is inserted with no changes made to it.

  • The link is discarded; it is not inserted.

  • A new display URL is returned, replacing the URL link for insertion.

  • A display URL and an access URL are returned. The display URL may or may not be identical to the URL link.

9.7.3 Creating and Using a URL Rewriter

Follow these steps to create and use a URL rewriter:

  1. Create a new Java file implementing the UrlRewriter interface open(), close(), and rewrite() methods. A sample rewriter, SampleRewriter.java, is available for reference under $ORACLE_HOME/ultrasearch/extension/.

  2. Compile the rewriter Java file into a class file. For example:

    /jdk1.3.1/bin/javac -O -classpath $ORACLE_HOME/ultrasearch/lib/ultrasearch.jar SampleRewriter.java 
    
    
  3. Package the rewriter class file into a jar file under the $ORACLE_HOME/ultrasearch/lib/agent/ directory. For example:

    /jdk1.3.1/bin/jar cv0f $ORACLE_HOME/ultrasearch/lib/agent/sample.jar SampleRewriter.class 
    
    
  4. Specify the rewriter class name and jar file name (for example, SampleRewriter and sample.jar) in the administration tool in step 2 of "Creating Web Sources" or in the crawler parameters page of an existing Web data source.

  5. Enable the UrlRewriter option from Web Sources page in the administration tool.

  6. Crawl the target Web data source by launching the corresponding schedule. The crawler log file confirms the use of the URL rewriter with the message Loading URL rewriter "SampleRewriter"...


    Note:

    URL rewriting is available for Web data sources only.


    See Also:


9.8 Oracle Ultra Search Sample Query Applications

Oracle Ultra Search provides several sample query applications and a sample crawler agent. Use the sample query applications as examples for creating your own query application. The query applications are written as J2EE-compliant Web applications. Your query application uses the Oracle Ultra Search query API. You can also use the sample crawler agent to create your own crawler agent.


Note:

Pointers to the sample query applications and the sample crawler agent Java source code, as well as their corresponding readmes, are in the Oracle Ultra Search welcome page: http://hostname.domainname:port/ultrasearch/index.html

The sample query applications are shipped as a deployed J2EE Web application (sample.ear). This component depends on a J2EE container to host the Web pages, a JDBC driver, and Java Mail API for displaying email results. After the sample.ear file is deployed by the Oracle Containers for J2EE (OC4J), you see a set of JSP files that demonstrate the query API usage.

The sample query applications include a sample search portlet. The sample Oracle Ultra Search portlet demonstrates how to write a search portlet for use in Oracle Application Server Portal.

When the user issues a query in any of the query applications, a hit list containing query results is returned. The user can select a document to view from the hit list. A hit list can include HTML documents, files, database table content, archived emails, or Oracle Application Server items. The Oracle Ultra Search sample query applications also incorporate an email browser for reading and browsing emails.

The Oracle Ultra Search administration tool and the Oracle Ultra Search sample query applications are part of the Oracle Ultra Search middle tier. However, the Oracle Ultra Search administration tool is independent from the Oracle Ultra Search sample query applications. Therefore, they can be hosted on different computers to enhance security or scalability.

If you do not want to use the sample query applications, you can build your own query application by directly invoking the Oracle Ultra Search Java Query API. Because the API is coded in Java, you can invoke the API methods from any Java-based application, such as from a Java servlet or a JavaServer Page (as in the case of the provided sample query applications). For rendering emails that have been crawled and indexed, you can also directly invoke the Oracle Ultra Search Java email API methods.

9.8.1 Sample Query Applications

The sample query applications are located in the $ORACLE_HOME/ultrasearch/sample directory.

9.8.2 JavaServer Page Concepts

As mentioned earlier, you can use JSP code and the supplied Java APIs to create your Web application. Typically, your Web application runs in an application server, such as Oracle Application Server. The application server typically runs on a separate computer from the Oracle server for performance and scalability reasons. The Oracle server holds the Oracle Ultra Search indexes.

JSP applications are compiled into Java servlets at runtime. The compiled servlets run in one or more Java Virtual Machine processes. The JSP application communicates with the Oracle server through the Oracle JDBC driver.

As in any Java application, you must include the following files in your servlet engine classpath to use the Java query and email APIs:

  1. $ORACLE_HOME/ultrasearch/lib/ultrasearch_query.jar

  2. $ORACLE_HOME/lib/mail.jar

  3. $ORACLE_HOME/lib/activation.jar

Figure 9-1 shows how your Web query application calls the Oracle Ultra Search Java query API.

Figure 9-1 Calling JavaServer Pages

Description of isrch009.gif follows
Description of the illustration isrch009.gif