| Oracle® Secure Enterprise Search Administrator's Guide 10g Release 1 (10.1.8.2) Part Number E10418-03 |
|
|
View PDF |
| Oracle® Secure Enterprise Search Administrator's Guide 10g Release 1 (10.1.8.2) Part Number E10418-03 |
|
|
View PDF |
This chapter explains the Oracle Secure Enterprise Search (SES) APIs and related information. This chapter contains the following topics:
Note:
Oracle SES only supports the APIs discussed in this chapter.Oracle Secure Enterprise Search provides the following APIs:
Web Services APIs
The Web Services APIs are used to integrate Oracle SES search capabilities into your search application. Oracle SES provides Java proxy libraries. You either can use the Java libraries or create proxies, based on the published Web Services Description Language (WSDL) files, to access Oracle SES Web Services. Oracle SES provides two Web Services APIs:
Query Web Services API
Admin Web Services API
Crawler Plug-in API
Oracle SES includes an extensible crawler plug-in framework that lets you crawl and index proprietary document repositories with the Crawler Plug-in API.
Security APIs
Oracle SES also includes an extensible authentication and authorization framework. You use any identity management system to authorize users with the Identity Plug-in API, and you can define your own security model for each source with the Authorization Plug-in API.
URL Rewriter API
The URL Rewriter API is used by the crawler to filter and rewrite extracted URL links before they are inserted into the URL queue.
Query-time Authorization API
The Query-time Authorization API filters search results and access to document information at search time. Query-time filtering can be used in addition to, or in place of, ACLs.
Oracle SES includes the following Web Services APIs:
Query Web Services API: This lets you perform search queries; for example, search for "oracle benefits" and return all the documents. You can also customize the default Oracle SES ranking to create a more relevant search result list for your enterprise or configure clustering for customized applications.
Admin Web Services API: This lets you perform a subset of administrative actions; for example, start or stop a crawl schedule, check schedule status, get the estimated index fragmentation level, and perform index optimization.
See Also:
Oracle Secure Enterprise Search Java API Reference and the "Web Services Interface" section in the Oracle SES administration tutorial:http://st-curriculum.oracle.com/tutorial/SESAdminTutorial/index.htm
Oracle Secure Enterprise Search Web Services APIs let you write your own application to search and administer Oracle SES over the network. The APIs provide the following benefits:
Applications can be deployed into any computer that connects to Oracle SES server through a standard Internet protocol.
Web Services protocol is XML-based, which makes for easy application integration.
Oracle SES also provides the client-side Java proxies for marshalling and parsing Web Services SOAP messages. Client applications can use the library instead of creating SOAP requests and parsing SOAP responses by themselves to access Oracle SES Web Services.
This section contains the following topics:
Oracle SES Web Services runs on top of the Oracle SES middle tier standalone OC4J server. They are installed and configured as part of the default installation. You can use Oracle SES Web Services out-of-the-box. Follow the same middle tier administration steps to start and stop Oracle SES Web Services.
There is a default Oracle SES Web Services administrator console provided by OC4J. The administrator console URL is the same as the Oracle SES Web Services URL.
The Query Web service is located at the following address for an Oracle SES installation:
http://<host>:<port>/search/query/OracleSearch
For example, if your Oracle SES middle tier is running on host 'myhost' and the port number is 8888, then the Query Web Services URL is the following:
http://myhost:8888/search/query/OracleSearch
You can obtain the following information from the administrator console:
Oracle SES Query WSDL description
List of Web Services messages and operations
Client-side Java proxies and source codes
The Admin Web service is located at the following address for an Oracle SES installation:
http://<host>:<port>/search/ws/admin/SearchAdmin.
You can obtain the following information from the administrator console:
Oracle SES Admin WSDL description
List of Web Services messages and operations
Client-side JavaScript stub
Oracle SES Web Services consists of a remote procedure call (RPC) interface to Oracle SES that enables the client application to invoke operations on Oracle SES over the network. The client application uses WSDL specification published by Oracle SES Web Services URL to send a request message using Simple Object Access Protocol (SOAP). The server then responds to the client application with a SOAP response message.
This section explains the following concepts:
A Web Service is a software application identified by a URI whose interfaces and binding are capable of being defined, described, and discovered by XML artifacts. A Web Service supports direct interactions with other software applications using XML-based messages and internet-based products.
A Web Service does the following:
Exposes and describes itself: A Web Service defines its functionality and attributes so that other applications can understand it. By providing a WSDL file, a Web Service makes its functionality available to other applications.
Allows other services to locate it on the Web: A Web Service can be registered in a UDDI registry so that applications can locate it.
Can be invoked: After a Web Service has been located and examined, the remote application can invoke the service using an Internet standard protocol.
Web Services are of either request and response or one-way style, and they can use either synchronous or asynchronous communication. However, the fundamental unit of exchange between Web Services clients and Web Services, of either style or type of communication, is a message.
The Simple Object Access Protocol (SOAP) is a lightweight XML-based protocol for exchanging information in a decentralized distributed environment. SOAP supports different styles of information exchange, including RPC-oriented and message-oriented exchange. RPC style information exchange allows for request-response processing, where an endpoint receives a procedure-oriented message and replies with a correlated response message. Message-oriented information exchange supports organizations and applications that need to exchange messages or other types of documents where a message is sent, but the sender might not expect or wait for an immediate response. Message-oriented information exchange is also called document style exchange.
SOAP has the following features:
Protocol independence
Language independence
Platform and operating system independence
Support for SOAP XML messages incorporating attachments (using the multipart MIME structure)
The Web Services Description Language (WSDL) is an XML format for describing network services containing RPC-oriented and message-oriented information. Programmers or automated development tools can create WSDL files to describe a service and can make the description available over the Internet. Client-side programmers and development tools can use published WSDL specifications to obtain information about available Web Services and to build and create proxies or program templates that access available services.
Oracle SES Web Services is powered by the Oracle SES middle tier OC4J server. The implementation, configuration, and deployment of Oracle SES Web Services follow the procedures and standards provided by OC4J server.
Oracle SES WSDL defines the operations and messages for Oracle SES Web Services. The message exchange of Oracle SES Web Services is RPC style, in which the contents of the SOAP message body conform to a structure that specifies a procedure and includes set of parameters or a response with a result and any additional parameters.
Oracle SES SOAP messages use HTTP binding where a SOAP message is embedded in the body of a HTTP request and a SOAP message is returned in the HTTP response.
The following diagram illustrates the architecture of Oracle SES Web Services:
You can implement client applications using platforms that support SOAP, such as Oracle JDeveloper, Microsoft .NET, or Apache Axis. These platforms allow you to automatically create code using the Oracle SES WSDL interface. Include the generated code along with the application logic to create a request, invoke the Web Services, and interpret the response.
This section contains the following topics:
Oracle Secure Enterprise Search Web Services uses the following base data types:
The mapping between XML schema data types and Java data types depends on the SOAP development environment. The following table shows mappings for the Oracle JDeveloper environment:
Oracle SES Web Services uses the following complex data types:
The search result container. It has the following elements:
returnCount: A Boolean value indicating whether the result return count estimate for the hitlist
estimatedHitCount: The estimated count of the search result, -1 means the search result does not return estimated hit count
dupRemoved: A Boolean value indicating whether near duplicate documents have been removed from search result
dupMarked: A Boolean value indicating whether near duplicate documents have been marked in search result. If dupRemoved is true, then dupMarked is always false.
resultElements: An array of resultElement, which represents the actual hitlist
suggestedLinks: An array of suggestedLink for the given search
query: The actual search string. The search string should follow Oracle SES query syntax
altKeywords: Alternate keywords (suggestions) for the given search
startIndex: The start index of search results
docsReturned: The number of search hits returned
This is the data type for search result element. It has the following elements:
author: Primary author of the document
description: Description of the document
url: URL of the document
snippet: Keywords in context (KWIC) of the document
title: Title of the document
lastModified: Last modified date of the document
mimetype: Mime type of the document
score: Oracle Text score of the document
docID: Document ID
language: Language of the document
contentLength: Content length of the document
signature: Signature of the document
infoSourceID: InfoSource ID of the document
infoSourcePath: InfoSource path of the document
groups: Array of groups to which the document belongs
isDuplicate: Boolean value indicating whether this document is a near duplicate of another document in the result list
hasDuplicate: Boolean value indicating whether this document has one or more near duplicates in the result list
fedID: Federated instance ID, used to track which federated instance the document is fetched from
customAttributes: Array of custom nondefault attributes extracted from/for the document during crawling
CustomAttribute encapsulates the name and value of the custom (user-defined) attribute. The name of the attribute is represented by actual name and type of the attribute in name_type format. For example, the string attribute foo is represented as foo_STRING. All Date attributes use the format mm/dd/yyyy.
Suggested content from a provider. It has following elements:
name: name of the suggested content provider
content: suggested content from the provider. The content is a byte array of the XML or HTML content
The source group. It has the following elements:
groupID: Source group ID
groupName: Source group name
groupDisplayName: Display name for the source group
The data type for search attribute. It has the following elements:
id: Search attribute ID
name: Internal name of search attribute
displayName: Display name of search attribute
type: The search attribute type. Value is either number, string, or date.
The data type for filter condition (predicate). It has the following elements:
attributeId: Search attribute ID
attributeType: Search attribute type. Value is either number, string, or date.
operator: Operator of the filter condition
If attributeType is string, then it should be either equals or contains.
If attributeType is number or date, then it should be either greaterthan, greaterthanequals, lessthan, lessthanequals, or equals.
attributeValue: Value of the filter condition (predicate)
For string type attribute, the value is simply the string itself.
For number type attribute, the value should be represented by a string consisting of an optional sign, (+) or (-), followed by a sequence of zero or more decimal digits ("the integer"), optionally followed by a fraction. The fraction consists of a decimal point followed by zero or more decimal digits. The string must contain at least one digit in either the integer or the fraction.
For date type attribute, the value should be in the format mm/dd/yyyy, where mm is the month (01~12), dd is the date (01~31), yyyy is the year (for example, 2005)
Examples:
If the filter condition is Title contains 'Oracle Secure Enterprise Search', then the client application needs to lookup the attribute ID of search attribute 'Title' and include the following (element, value) pairs:
attributeID = 1 (assuming the search attribute id of 'Title' is 1)
operator = contains
attributeValue = Oracle Secure Enterprise Search
If the filter condition is Price greater than 1000, then the client application needs to lookup the attribute ID of search attribute 'Price' and include the following (element, value) pairs:
attributeID = 2 (assuming the search attribute id of 'Price' is 2)
operator = greaterthan
attributeValue = 1000
This is the data type for the infosource node. It has the following elements:
id: Infosource node ID
fedId: Federated instance ID, used to track which federated instance the node belongs to
name: Name of the node
docCount: Number of documents under the node. If the value is –1, then there exists documents under the node but the count cannot be shown.
hasChildren: Indicates if the node has any children
fullpath: Full path of the category node
fullpathIds: The IDs of each node in the full path
This is the element of AttributeLOV, the list of search attribute values. It has the following elements:
value: Attribute value (internal value)
displayValue: Display value
This data structure is used to store authentication information for the search user in the form of a name-value pair, which can be used during query-time authorization filtering of the results. It has following elements:
name: Name of the authentication attribute
value: Value of the authentication attribute
Oracle Secure Enterprise Search Web Services uses the following complex array types:
AttributeArray: Array of Attribute
AttributeLOVElementArray: Array of AttributeLOVElement
CustomAttributeArray: Array of CustomAttribute
SCElementArray: Array of SCElement
DataGroupArray: Array of DataGroup
FilterArray: Array of Filter
IntArray: Array of int
LanguageArray: Array of Language
NodeArray: Array of Node
ResultElementArray: Array of ResultElement
SessionContextElementArray: Array of SessionContextElement
StringArray: Array of String
This section contains the following topics:
Oracle Secure Enterprise Search provides the following categories of Web Services operations:
Authentication: Authenticate a user's access to Oracle SES. The operation is only required if the user performs secure search.
Search: Run a search on Oracle SES and obtain a hitlist along with information such as estimated hit count, near duplicate documents in the result list, suggested links, and alternate keywords for the search. Get suggested content from external providers for the given query. You can also customize the default Oracle SES ranking to create a more relevant search result list for your enterprise or configure clustering for customized Oracle SES applications.
Metadata: Obtain the search metadata, such as the list of source groups, the list of supported languages, or the list of search attributes.
Search Hit: Obtain the search result details, such as the cached version of search result and in-links and out-links of the search hit.
User Feedback: Send user feedback to Oracle SES, such as user submitted URL.
See Also:
"Query Web Services Operations"This section describes the following authentication operations:
This message requests Oracle SES to authenticate the search user. It consists of the following parameters:
username: User name for the search user
password: Password for the search user
<message name="loginRequest"> <part name="username" type="xsd:string"/> <part name="password" type="xsd:string"/> </message>
Note:
User name is not case-sensitive.This message contains the return status for the loginRequest message.
<message name="loginResponse"> <part name="return" type="typens:Status"/> </message>
This message is used when the user logs out from the search application.
<message name="logoutRequest"> </message>
This message contains the return status for the logoutRequest message.
<message name="logoutResponse"> <part name="return" type="typens:Status"/> </message>
This message is used to pass authentication information for the search user, which can be used during query-time filtering.
Note:
Login and logout Web Services calls cause Oracle SES to automatically set or reset the
AUTH_USER value in the session context that is passed to the query-time filter. This session context attribute cannot be overwritten explicitly through the setSessionContext call.It consists of the following parameter:
sessionContext: An array of SessionContextElement. This array stores the authentication information needed for the query-time authentication filtering in the form of name-value pairs.
<message name="setSessionContextRequest">
<part name="sessionContext" type="typens:SessionContextElementArray"/>
</message>
This message contains the return status for the setSessionContext message.
<message name="setSessionContextResponse"> <part name="return" type="typens:Status"/> </message>
This message logs in the end user to Oracle SES using proxy authentication. It consists of following parameters:
username: User name of the proxy user
password: Password of the proxy user
searchUser: User name of the end user
<message name="proxyLoginRequest"> <part name="username" type="xsd:string"/> <part name="password" type="xsd:string"/> <part name="searchUser" type="xsd:string"/> </message>
The proxy user must be one of the federation trusted entities created on the Oracle SES instance.
See Also:
"Federation Trusted Entities"This message contains the return status for the proxyLoginRequest message.
<message name="proxyLoginResponse"> <part name="return" type="typens:Status"/> </message>
This section describes the following search operations:
This invokes Oracle SES advanced search and returns search results. It consists of the following parameters:
query: The search string. This should follow Oracle SES query syntax. See "Query Web Services Query Syntax" for details.
startIndex: Index of the first document in the hitlist to be returned. The default is 1 if not set explicitly.
docsRequested: The maximum number of documents in the hitlist to be returned. The default is 10 if not set explicitly.
dupRemoved: Boolean flag to enable or disable duplicate removal. If turned on, then duplicate documents in the hitlist are removed. The default is false if not set explicitly. Note: The dupMarked switch has no effect when dupRemoved is turned on.
dupMarked: Boolean flag to enable or disable duplicate detection. The default is false if not set explicitly. Note: The dupMarked switch has no effect when dupRemoved is turned on.
groups: Data source groups that the search will be restricted to. The default is all groups if not set explicitly.
queryLang: Language of the query. This is equivalent to Locale. The default is English("en") if not set explicitly. This is used in relevancy boosting.
docLang: Language of the documents to restrict the search. If the value is not set explicitly, then search is performed against documents of all the languages.
returnCount: Boolean flag to fetch the total hit count with the result. The default is false if not set explicitly.
filterConnector: Connector between all the filters: "and" indicates that the documents in the hitlist must satisfy all the filters, and "or" indicates that the documents in the hitlist must satisfy at least one of the filters. The default is "and" if not set explicitly.
filters: An array of filters. Each filter is a restriction on on the search result. Filters are connected by the filterConnector. The default is null (no filter applies to the search result) if not set explicitly.
fetchAttributes: Array of integers representing the IDs of custom or nondefault attributes to be fetched with the search result
searchControls: XML string to specify advanced filter conditions and ranking parameters
Note:
The attributefilterLastModifiedDate uses the format mm/dd/yyyy.public OracleSearchResult doOracleAdvancedSearch(String query,
Integer startIndex,
Integer docsRequested,
Boolean dupRemoved,
Boolean dupMarked,
DataGroup[] groups,
String queryLang,
String docLang,
Boolean returnCount,
String filterConnector,
Filter[] filters,
Integer[] fetchAttributes,
String searchControls)
throws Exception
This invokes Oracle SES fetch search and returns fetch results. It consists of the following parameters:
query: The search string. This should follow Oracle SES query syntax. See "Query Web Services Query Syntax" for details.
targetDocIdList: Target document ID list, most likely from a cluster node.
startIndex: Index of the first document in the hitlist to be returned. The default is 1 if not set explicitly.
docsRequested: Maximum number of documents in the hitlist to be returned. The default is 10 if not set explicitly.
queryLang: Language of the query. This is equivalent to Locale. The default is English("en") if not set explicitly. This is used in relevancy boosting.
fetchAttributeNames: Array of names of custom or nondefault attributes to be fetched with the search result.
groupAttr: Attribute used for grouping.
sortAttrList: List of sorting attribute settings.
clusterList: List of cluster configurations.
public OracleResultContainer doOracleFetchSearch(String query,
String[] targetDocIdList,
Integer startIndex,
Integer docsRequested,
String queryLang,
String[] fetchAttributeNames,
GroupAttribute groupAttr,
SortAttribute[] sortAttrList,
ClusterConfig[] clusterList)
throws Exception
This invokes Oracle SES organized search and returns search results. It consists of the following parameters:
query: The search string. This should follow Oracle SES query syntax. See "Query Web Services Query Syntax" for details.
topN: Top N search result for grouping, sorting, and clustering.
startIndex: Index of the first document in the hitlist to be returned. The default is 1 if not set explicitly.
docsRequested: Maximum number of documents in the hitlist to be returned. The default is 10 if not set explicitly.
dupRemoved: Boolean flag to enable or disable duplicate removal. If turned on, duplicate documents in the hitlist will be removed. The default is false if not set explicitly. Note: The dupMarked switch will have no effect when dupRemoved is turned on.
dupMarked: Boolean flag to enable or disable duplicate detection. The default is false if not set explicitly. Note: The dupMarked switch will have no effect when dupRemoved is turned on.
groups: Data source groups that the search will be restricted to. The default is all groups if not set explicitly.
queryLang: Language of the query. This is equivalent to Locale. The default is English("en") if not set explicitly. This is used in relevancy boosting.
docLang: Language of the documents to restrict the search. If the value is not set explicitly, then search is performed against documents of all the languages.
returnCount: Boolean flag to fetch the total hit count with the result. The default is false if not set explicitly.
filterConnector: Connector between all the filters: "and" indicates that the documents in the hitlist must satisfy all filters, "or" indicates that the documents in the hitlist must satisfy at least one of the filters. The default is "and" if not set explicitly.
filters: An array of filters. Each filter is a restriction on on the search result. Filters are connected by the filterConnector. The default is null (no filter applies to the search result) if not set explicitly.
fetchAttributeNames: Array of names of custom or nondefault attributes to be fetched with the search result.
searchControls: XML string to specify advanced filter conditions and ranking parameters.
groupAttr: Attribute used for grouping.
sortAttrList: List of sorting attribute settings.
clusterList: List of cluster configurations.
public OracleResultContainer doOracleOrganizedSearch(String query,
Integer topN,
Integer startIndex,
Integer docsRequested,
Boolean dupRemoved,
Boolean dupMarked,
DataGroup[] groups,
String queryLang,
String docLang,
Boolean returnCount,
String filterConnector,
Filter[] filters,
String[] fetchAttributeNames,
String searchControls,
GroupAttribute groupAttr,
SortAttribute[] sortAttrList,
ClusterConfig[] clusterList)
throws Exception
This is the main message for the search application. It consists of the following parameters:
query: A search string. It must be a valid string and it cannot be null. The search string should follow Oracle SES query syntax. See "Query Web Services Query Syntax" for details.
startIndex: The index of the first result to be returned. For example, if there are 67 results, you might want to start at 20. The default is 1 if not set explicitly.
docsRequested: The maximum number of results to be returned. The default is 10 if not set explicitly.
dupRemoved: Enable or disable duplicate removal. If turned on, then the search result will eliminate all near duplicate documents from the result list. The dupMarked switch will have no effect when dupRemoved is turned on. The default is false if not set explicitly.
dupMarked: Enable or disable duplicate detection. If dupRemoved is turned off and dupMarked is turned on, then the search result will keep all near duplicate documents from the result list and mark them as duplicates. If dupRemoved is turned on, then the dupMarked switch will have no effect. The default is false if not set explicitly.
groups: Limit the search result to the documents from specified source groups. The default is for all groups if not set explicitly.
queryLang: The query language argument should be a valid ISO language code. These codes are the lower case, two-letter codes as defined by ISO-639. Examples: "en" for English and "de" for German. The default is English ("en") if not set explicitly. This is used for relevancy boosting.
docLang: Set the language of the documents to limit the search. If the value is not set explicitly, then search is performed against documents of all the languages.
returnCount: Set to true to return total hit count with the result. The default is false if not set explicitly.
filterConnector: The connector between all filters: "and" indicates the search result must satisfy all filters, "or" indicates the search result just needs to satisfy at least one filter. The default is "and" if not set explicitly.
filters: An array of filters. Each filter is a restriction on search results. Filters are connected by filterConnector. The default is null (no filter applies to the search result) if not set explicitly.
fetchAttributes: Array of integers representing the nondefault attribute IDs to be fetched in the resultElements. The default is null (or set one int value '0'), so no attributes other than default-attributes are fetched in the resultElements.
<message name="doOracleSearch"> <part name="query" type="xsd:string"/> <part name="startIndex" type="xsd:int"/> <part name="docsRequested" type="xsd:int"/> <part name="dupRemoved" type="xsd:boolean"/> <part name="dupMarked" type="xsd:boolean"/> <part name="groups" type="typens:DataGroupArray"/> <part name="queryLang" type="xsd:string"/> <part name="docLang" type="xsd:string"/> <part name="returnCount" type="xsd:boolean"/> <part name="filterConnector" type="xsd:string"/> <part name="filters" type="typens:FilterArray"/> <part name="fetchAttributes" type="typens:IntArray"/> </message>
This message returns the search result in OracleSearchResult data type.
<message name="doOracleSearchResponse">
<part name="return" type="typens:OracleSearchResult"/>
</message>
This message restricts a search to a particular node. It consists of the following parameters:
query: A search string. It must be a valid string, and it cannot be null. The search string should follow Oracle SES query syntax. See "Query Web Services Query Syntax" for more details.
nodeID: The ID of the node to restrict the search to.
fedID: The ID of the federated instance the parent node belongs to (null for local node).
startIndex: The index of the first result to be returned. For example, if there are 67 results, then you might want to start at 20. The default is 1 if not set explicitly.
docsRequested: The maximum number of results to be returned. The default is 10 if not set explicitly.
dupRemoved: Enable or disable duplicate removal. If turned on, then the search result will eliminate all near duplicate documents from the result list, and the dupMarked switch will have no effect when dupRemoved is turned on. The default is false if not set explicitly.
dupMarked: Enable or disable duplicate detection. If dupRemoved is turned off and dupMarked is turned on, then the search result will keep all near duplicate documents from the result list and mark them as duplicates. If dupRemoved is turned on, then the dupMarked switch will have no effect. The default is false if not set explicitly.
queryLang: The query language argument should be a valid ISO language code. These codes are the lower case, two-letter codes as defined by ISO-639. Examples: "en" for English and "de" for German. The default is English ("en") if not set explicitly. This is used for relevancy boosting.
docLang: Set the language of the documents to limit the search. If the value is not set explicitly, then search is performed against documents of all the languages.
returnCount: Set to true to return total hit count with the result. The default is false if not set explicitly.
fetchAttributes: Array of integers representing the nondefault attribute IDs to be fetched in the resultElements. The default is null (or set one int value '0'), so no attributes other than default-attributes are fetched in the resultElements.
<message name="doOracleBrowseSearch">
<part name="query" type="xsd:string"/>
<part name="nodeID" type="xsd:string"/>
<part name="fedID" type="xsd:string"/>
<part name="startIndex" type="xsd:int"/>
<part name="docsRequested" type="xsd:int"/>
<part name="dupRemoved" type="xsd:boolean"/>
<part name="dupMarked" type="xsd:boolean"/>
<part name="queryLang" type="xsd:string"/>
<part name="docLang" type="xsd:string"/>
<part name="returnCount" type="xsd:boolean"/>
<part name="fetchAttributes" type="typens:IntArray"/>
</message>
This message returns the search result in OracleSearchResult data type.
<message name="doOracleBrowseSearchResponse">
<part name="return" type="typens:OracleSearchResult"/>
</message>
This is a simplified form of the doOracleSearch message. In this message you don't need to specify the advanced search parameters that are specified in the doOracleSearch message. It consists of following parameters:
query: A search string. It must be a valid string and it cannot be null. The search string should follow Oracle SES query syntax. See "Query Web Services Query Syntax" for details.
startIndex: The index of the first result to be returned. For example, if there are 67 results, you might want to start at 20. The default is 1, if not set explicitly.
docsRequested: The maximum number of results to be returned. The default is 10, if not set explicitly.
dupRemoved: Enable or disable duplicate removal. If turned on, then the search result will eliminate all near duplicate documents from the result list. The dupMarked switch will have no effect when dupRemoved is turned on. The default is false if not set explicitly.
dupMarked: Enable or disable duplicate detection. If dupRemoved is turned off and dupMarked is turned on, then the search result will keep all near duplicate documents from the result list and mark them as duplicates. If dupRemoved is turned on, then the dupMarked switch will have no effect. The default is false if not set explicitly.
returnCount: Set to true to return total hit count with the result. The default is false if not set explicitly.
<message name="doOracleSimpleSearch"> <part name="query" type="xsd:string"/> <part name="startIndex" type="xsd:int"/> <part name="docsRequested" type="xsd:int"/> <part name="dupRemoved" type="xsd:boolean"/> <part name="dupMarked" type="xsd:boolean"/> <part name="returnCount" type="xsd:boolean"/> </message>
This message returns the search result in OracleSearchResult data type.
<message name="doOracleSimpleSearchResponse"> <part name="return" type="typens:OracleSearchResult"/> </message>
This message returns the suggested content for the given query. It consists of the following parameters:
query: Query string
returnType: Format in which the content is to be returned, either "html" or "xml". If no style sheet is configured for a given provider, then the return type is the return type of the content returned by the provider, regardless of whether "html" or "xml" is specified.
<message name="getSuggestedContent"> <part name="query" type="xsd:string"/> <part name="returnType" type="xsd:string"/> </message>
This message returns the suggested content for the query.
<message name="getSuggestedContentResponse"> <part name="return" type="typens:SCElementArray"/> </message>
This section describes the following browse operations:
This message gets the list of info source nodes given the parent node ID. It consists of the following parameters:
parentNodeID: The node ID for which all children nodes will be returned. If it is not set, then the message will return all the root nodes.
fedID: The ID of the federated instance the parent node belongs to (null for local node).
locale: A two letter representation of locale. The default is English ("en") if not set explicitly.
<message name="getInfoSourceNodesRequest">
<part name="parentNodeID" type="xsd:string"/>
<part name="fedID" type="xsd:string"/>
<part name="locale" type="xsd:string"/>
</message>
This message returns an array of info source nodes.
<message name="getInfoSourceNodesResponse">
<part name="nodes" type="typens:NodeArray"/>
</message>
This message gets the full path of a node, from root to node, given an info source node. It consists of the following parameters:
nodeID: The node ID for which all the nodes in the path from root to node will be returned, nodeID must be set and it cannot be null.
locale: A two letter representation of locale. The default is English ("en") if not set explicitly.
<message name="getInfoSourceAncestorNodesRequest">
<part name="nodeID" type="xsd:string"/>
<part name="locale" type="xsd:string"/>
</message>
Note:
ThegetInfoSourceAncestorNode messages have been deprecated from Oracle SES.This message returns an array of info source ancestor nodes.
<message name="getInfoSourceAncestorNodesResponse">
<part name="nodes" type="typens:NodeArray"/>
</message>
This message retrieves a particular node. It consists of the following parameters:
nodeID: The node ID of the node to get, nodeID must be set and it cannot be null.
fedID: The ID of the federated instance the parent node belongs to (null for local node).
locale: A two letter representation of Locale, the default is English ("en") if not set explicitly.
Message format:
<message name="getInfoSourceNodeRequest"> <part name="nodeID "type="xsd:string"/> <part name="fedID" "type="xsd:string"/> <part name="locale "type="xsd:string"/> </message>
This message returns the node requested.
<message name="getInfoSourceNodeResponse">
<part name="node "type="typens:Node"/>
</message>
This section describes the following metadata operations:
This message gets all the languages supported by Oracle SES. It is used by the client application to display the list of languages. It consists of the following parameter:
locale: A two letter representation of locale. The default is English ("en") if not set explicitly.
<message name="getLanguagesRequest">
<part name="locale" type="xsd:string"/>
</message>
This message returns all supported languages.
<message name="getLanguagesResponse">
<part name="return" type="typens:LanguageArray"/>
</message>
This message requests for all source groups defined in Oracle SES. It is used by the client application to show all source groups in the search page, such that the end user can restrict their search results within one or multiple source groups. It consists of the following parameter:
locale: A two letter representation of locale. The default is English ("en") if not set explicitly.
<message name="getDataGroupsRequest">
<part name="locale" type="xsd:string"/>
</message>
This message returns all source groups defined in Oracle SES.
<message name="getDataGroupsResponse">
<part name="groups" type="typens:DataGroupArray"/>
</message>
This message gets a list of search attributes that applied to the given source groups. It consists of the following parameters:
locale: A two letter representation of locale. The default is English ("en") if not set explicitly.
groups: Limit the request to the attributes from specified source groups. The default is all groups if not set explicitly.
groupConnector: The connector between all groups: "and" indicates the response is the attributes available in the set of source groups by finding the intersection of each group's attributes, "or" indicates the response is the attributes available in the set of source groups by finding the union of each group's attributes. The default is "or" if not set explicitly.
<message name="getAttributesRequest">
<part name="locale" type="xsd:string"/>
<part name="groups" type="typens:DataGroupArray"/>
<part name="groupConnector" type="xsd:string"/>
</message>
This message returns an array of search attributes.
<message name="getAttributesResponse">
<part name="return" type="typens:AttributeArray"/>
</message>
This message gets all search attributes defined in Oracle SES. It consists of the following parameter:
locale: A two letter representation of locale. The default is English ("en") if not set explicitly.
<message name="getAllAttributesRequest"> <part name="locale" type="xsd:string"/> </message>
This message returns all search attributes defined in Oracle SES.
<message name="getAllAttributesResponse">
<part name="return" type="typens:AttributeArray"/>
</message>
This message gets the LOV items given a search attribute. It consists of the following parameters:
attribute: A search attribute for the LOV (list of values) requested.
locale: A two letter representation of locale. The default is English ("en") if not set explicitly.
<message name="getAttributeLOVRequest"> <part name="attribute" type="typens:Attribute"/> <part name="locale" type="xsd:string"/> </message>
This message returns an array of search attribute LOV elements.
<message name="getAttributeLOVResponse"> <part name="return" type="typens:AttributeLOVElementArray"/> </message>
This section describes the following search hit operations:
This message gets the cached version of a document given the document ID and the search string. The search string will be highlighted in the output. It consists of the following parameters:
query: The search string
docID: The document ID to be fetched
fedID: The federated instance ID, used to track which federated instance the document is fetched from
<message name="getCachedPageRequest">
<part name="query" type="xsd:string"/>
<part name="docID" type="xsd:int"/>
<part name="fedID" type="xsd:string"/>
</message>
This message returns the byte array of the cached HTML page.
<message name="getCachedPageResponse">
<part name="return" type="xsd:base64Binary"/>
</message>
This message gets all the incoming links for a given search hit (document). It consists of the following parameters:
docID: The document ID for which the incoming links to be fetched. It must be a valid document ID and it cannot be null.
maxNum: The maximum number of incoming links requested. The default is 25 if not set explicitly.
fedID: The federated instance ID, used to track which federated instance the document is fetched from
<message name="getInLinksRequest">
<part name="docID" type="xsd:int"/>
<part name="maxNum" type="xsd:int"/>
<part name="fedID" type="xsd:string"/>
</message>
This message returns an array of incoming link URL strings.
<message name="getInLinksResponse">
<part name="return" type="typens:StringArray"/>
</message>
This message gets all the outgoing links for a given search hit (document). It consists of the following parameters:
docID: The document ID for which the outgoing links to be fetched. It must be a valid document ID and it cannot be null.
maxNum: The maximum number of outgoing links requested. The default is 25 if not set explicitly.
fedID: The federated instance ID, used to track which federated instance the document is fetched from
<message name="getOutLinksRequest">
<part name="docID" type="xsd:int"/>
<part name="maxNum" type="xsd:int"/>
<part name="fedID" type="xsd:string"/>
</message>
This message returns an array of outgoing link URL strings.
<message name="getOutLinksResponse">
<part name="return" type="typens:StringArray"/>
</message>
This message logs the user's click. It consists of the following parameters:
queryID: ID of the submitted search
urlID: ID of the document that the user clicked on
infosourceID: Infosource ID. If none, then –1 is used as the default value
position: The position of the document in the result list (for example, first hit on the page or 9th hit on the page)
fedID: Federation ID. Specifies the federated instance on which the document resides.
<message name="logUserClickRequest">
<part name="queryID" type="xsd:int"/>
<part name="urlID" type="xsd:int"/>
<part name="infoSourceID" type="xsd:int"/>
<part name="position" type="xsd:int"/>
<part name="fedID" type="xsd:string"/>
</message>
This message returns the URL of the clicked-on document.
<message name="logUserClickResponse"> <part name="url" type="xsd:string"/> </message>
This section describes the following user feedback operations:
This message submits a URL to Oracle SES, such that Oracle SES will crawl and index the URL. It consists of the following parameter:
url: The URL to be submitted to the crawler so it can be crawled next time. It must be a valid URL and it cannot be null.
<message name="submitUrlRequest">
<part name="url" type="xsd:string"/>
</message>
This message returns the status, which consists of two strings: the first one is the submission status, it is either "successful" or "failed"; the second string is the error message in case that submission status is "failed".
<message name="submitUrlResponse">
<part name="return" type="typens:Status"/>
</message>
This section describes the query syntax used in the Oracle Secure Enterprise Search Search API.
A search term can be a single word, a phrase, or a special search term. For example, if the search string is oracle secure enterprise search, then there are four search terms in the search string: oracle, secure, enterprise, and search. If the search string is oracle "secure enterprise search", then there are two search terms in the search string: oracle and "secure enterprise search".
Search terms in different cases are treated the same (case insensitive). For example, searching oracle, Oracle, or ORACLE will return the same search result.
The following operators are defined in the query syntax:
Plus [+]: The plus operator specifies that the search term immediately following it must be found in all matching documents. For example, searching for [Oracle +Applications] only finds documents that contain the word "Oracle" and "Applications". In a multiple word search, you can attach a [+] in front of every token including the very first token. You can also attach a [+] in front of a phrase enclosed in double-quotes ("). But there should be no space between the [+] and the search term.
Minus [-]: The minus operator specifies that the search term immediately following it cannot appear in any document included in the search result. For example, searching for [Oracle -Applications] only finds documents that do not contain the word "Applications". In a multiple word search, you can attach a [-] in front of every token except the very first token. It can be a single word or a phrase, but there should be no space between the [-] and the token.
Asterisk [*]: The asterisk specifies a wildcard search. For example, searching for the string [Ora*] finds documents that contain all words beginning with "Ora" such as "Oracle" and "Orator". You can also insert an asterisk in the middle of a word. For example, searching for the string [A*e] finds documents that contain words such as "Apple" or "Ape".
By default, Oracle SES searches all of your search terms, as well as relevant variations of the terms you have entered. There is no need to include any operators (like 'AND') between terms. The order of the terms in the search will affect the search results.
Oracle SES query syntax only supports 'Site' and 'File type' filter conditions. It does not support any other filter conditions (advanced conditions) such as title, author, last modified date. To restrict your search with other filter conditions, you can specify them in the Web Services API message doOracleSearch.
Oracle SES supports the use of several special search terms that allow the user or search administrator to access additional capabilities of the Oracle SES in front of it. Following is the list of special search terms:
You can exclude a word from your search by putting a minus sign [-] immediately in front of the term you want to exclude from the search results. Exclusion does not work with stop words.
Example: oracle -search
Negative search is not allowed unless there is another positive search term. For example:
-search is an invalid search.
oracle -search is a valid search.
Search for words starting with "ora". The asterisk can only be specified at the end (right side) or middle of a search term. So you cannot search for something like *earch.
Example: Ora*
Search for complete phrases by enclosing them in quotation marks. Words marked in this way will appear together in all results exactly as entered.
Example: "oracle secure enterprise search"
If you know the specific Web site you want to search, but are not sure where the information is located within that site, then search only within the specific Web site. Enter the search followed by the string "site:" followed by the host name.
Example: oracle site:text.us.oracle.com
Notes:
Domain restriction is not supported, because Oracle SES does not support left-truncated wildcard search (such as *.oracle.com)
The exclusion operator (-) can be applied to this search term to remove a Web site from consideration in the search.
Site restricted search term is implicit AND with other search terms.
Only one site restriction is allowed. Also, you cannot have both site inclusion and exclusion in the search string. For example, the following search string is invalid:
oracle search site:www.oracle.com -site:otn.oracle.com
The search prefix "filetype:" filters the results returned to include only documents with the extension specified immediately after. There can be no space between "filetype:"; and the specified extension.
Example: oracle filetype:doc
Notes:
The exclusion operator (-) can be applied to this search term to remove a file type from consideration in the search.
Only one file type can be included. The following extensions are supported: doc, htm, html, xml, ps, pdf, txt, rtf, ppt, and xls. doc, html, pdf, txt, rtf, ppt, xls.
File type restricted search term is implicit AND with other search terms.
Only one file type restriction is allowed. Also, you cannot have both file type inclusion and exclusion in the search string. For example, the following search string is invalid:
oracle search filetype:doc -filetype:pdf
Following is a simple JSP application using Oracle Secure Enterprise Search proxy Java library to provide the basic search functionality:
<%@page contentType="text/html; charset=utf-8" %>
<%@page import = "java.util.Vector" %>
<%@page import = "java.net.URL" %>
<%@page import = "java.util.Properties" %>
<%@page import = "java.util.HashMap" %>
<%@page import = "org.apache.soap.Header" %>
<%@page import = "org.apache.soap.rpc.Call" %>
<%@page import = "org.apache.soap.rpc.Parameter" %>
<%@page import = "org.apache.soap.rpc.Response" %>
<%@page import = "org.apache.soap.Fault" %>
<%@page import = "org.apache.soap.SOAPException" %>
<%@page import = "org.apache.soap.Constants" %>
<%@page import = "org.apache.soap.encoding.SOAPMappingRegistry" %>
<%@page import = "org.apache.soap.encoding.soapenc.BeanSerializer" %>
<%@page import = "org.apache.soap.util.xml.QName" %>
<%@page import = "oracle.soap.transport.http.OracleSOAPHTTPConnection" %>
<%@page import = "oracle.soap.encoding.soapenc.EncUtils" %>
<%@page import = "oracle.search.query.webservice.client.*" %>
<%
//
// Get the search term entered by the user
//
String searchTerm = request.getParameter("searchTerm");
if (searchTerm == null) searchTerm = "";
//
// Define the result element array.
//
ResultElement[] resElemArray = null; // ResultElement is one of the proxy Java
classes
int estimatedHitCount = 0;
if (searchTerm != null && !"".equals(searchTerm))
{
//
// Create the Oracle SES Web Services client stub
//
OracleSearchService stub = new OracleSearchService();
//
// Set the Oracle SES Web Services URL.
// The URL is http://<host>:<port>/search/query/OracleSearch
//
stub.setSoapURL("http://staca19:7777/search/query/OracleSearch");
//
// Get the search result by calling OracleSearchService.doOracleSearch()
//
OracleSearchResult result = stub.doOracleSearch(searchTerm,
new Integer(1),
new Integer(10),
Boolean.TRUE,
Boolean.TRUE,
null,
"en",
"en",
Boolean.TRUE,
null,
null,
null);
//
// Get the estimated hit count by calling
estimatedHitCount = result.getEstimatedHitCount().intValue();
// Get the search results
resElemArray = result.getResultElements();
}
%>
<HTML>
<HEAD>
<TITLE>Oracle SES Web Services Demo </TITLE>
</HEAD>
<BODY>
<FORM name="searchBox" method="post" action="./DemoWS.jsp">
<INPUT id="inputMain" type="text" size="40" name="searchTerm" value="<%=searchTerm%>">
<INPUT type="hidden" name="searchTerm" value="<%= searchTerm %>">
<INPUT type="submit" name="action" value="Search">
</FORM>
<BR><BR><BR>
<%
//
// Render the search results
//
if (resElemArray == null || resElemArray.length == 0)
{
%>
<H3> There are no matches for the search term </H3>
<%
}
else
{
%>
<H3> There are about <%=estimatedHitCount%> matches </H3>
<%
for (int i=0; i<resElemArray.length; i++)
{
String title = resElemArray[i].getTitle();
if (title == null) title = "Untitled Document";
%>
<P>
<B><A HREF="<%=resElemArray[i].getUrl()%>"><%=title%></A> </B>
<BR>
<%=resElemArray[i].getSnippet()%>
<BR>
</P>
<%
}
}
%>
</BODY>
</HTML>
Oracle SES also provides client-side Java proxies for marshalling and parsing Web Services SOAP messages. Client applications can use the library to access Oracle SES Web Services.
The proxy library includes the following Java classes, which are mapped to the corresponding Web Services data types and messages:
oracle.search.query.webservice.client.Attribute
oracle.search.query.webservice.client.AttributeLOVElement
oracle.search.query.webservice.client.ClusterAttribute
oracle.search.query.webservice.client.ClusterConfig
oracle.search.query.webservice.client.ClusterTree
oracle.search.query.webservice.client.CustomAttribute
oracle.search.query.webservice.client.DataGroup
oracle.search.query.webservice.client.Filter
oracle.search.query.webservice.client.GroupAttribute
oracle.search.query.webservice.client.GroupingResult
oracle.search.query.webservice.client.Language
oracle.search.query.webservice.client.Node
oracle.search.query.webservice.client.OracleSearchResult
oracle.search.query.webservice.client.OracleSearchService
oracle.search.query.webservice.client.ResultElement
oracle.search.query.webservice.client.SCElement
oracle.search.query.webservice.client.SessionContextElement
oracle.search.query.webservice.client.SortAttribute
oracle.search.query.webservice.client.Status
oracle.search.query.webservice.client.SuggestedLink
To compile and run your client application using the Oracle SES client-side Java proxy library, you need to include the following files in the Java CLASSPATH. You can obtain these files from Oracle SES server file directory.
$ORACLE_HOME/search/lib/search_client.jar (The proxy Java library)
$ORACLE_HOME/oc4j/webservices/lib/soap.jar
$ORACLE_HOME/oc4j/j2ee/home/lib/http_client.jar
$ORACLE_HOME/lib/xmlparserv2.jar
$ORACLE_HOME/lib/mail.jar
$ORACLE_HOME/lib/activation.jar
The following Web Services messages and operations are intended for Oracle SES internal use only. They are subject to change or removal in future releases.
setSearchUserRequest, setSearchUserResponse, setSearchUser
The Admin Web Services API includes the following administrative operations:
Table 9-3 Admin Web Services Operations
| Class | Description |
|---|---|
|
|
Returns the estimated index fragmentation level, as an integer percentage. |
|
|
Returns information for all crawler schedules.
|
|
|
Returns the current status of a crawler schedule.
|
|
|
Authenticates and creates a session.
|
|
|
Logs out and closes the current session. |
|
|
Optimizes the search index. |
|
|
Starts the specified crawler schedule.
|
|
|
Stops the specified crawler schedule.
|
Oracle SES provides client-side Java proxies for marshalling and parsing Web Services SOAP messages. Client applications can use the library to access the Oracle SES Admin Web Service.
The proxy library includes the following Java classes, which are mapped to the corresponding Web Services data types and messages:
oracle.search.admin.ws.client.Schedule
oracle.search.admin.ws.client.ScheduleStatus
oracle.search.admin.ws.client.SearchAdminClient
To compile and run your client application using the Oracle SES client-side Java proxy stub, include the following files in the Java CLASSPATH:
$ORACLE_HOME/search/lib/search_admin_wsclient.jar
wsclient_extended.jar
The wsclient_extended.jar file is available as a separate download from the Oracle Technology network:
http://www.oracle.com/technology/software/products/ias/htdocs/utilsoft.html
See Also:
"Setting the Classpath for a Web Service Proxy" in the Oracle Application Server Web Services Developer's Guide, 10g Release 3 (10.1.3.1.0)
If an error occurs as a result of an Admin Web Service request, a SOAP fault is returned. When using the provided Java proxy client, a javax.xml.rpc.soap.SOAPFaultException is thrown. To access the computer parseable error code, call the SOAPFaultException.getFaultCode() method.
The following table lists the Admin Web Service error codes:
Table 9-4 Admin Web Services Error Codes
| Error Code | Description | SOAP Fault Code Prefix |
|---|---|---|
|
|
The provided security credentials are not valid. |
Client |
|
|
An internal error occurred. Please try again. |
Server |
|
|
The specified schedule is invalid for the operation performed. |
Client |
|
|
The specified schedule name does not exist. |
Client |
The Oracle Secure Enterprise Search Java SDK contains the following APIs:
See Also:
Building Custom Crawlers whitepaper for detailed information about how to build secure crawlers:http://www.oracle.com/technology/products/oses/pdf/Buidling_Custom_CrawlersJan12_07.doc
You can implement a crawler plug-in to crawl and index a proprietary document repository. In Oracle SES, the proprietary repository is called a user-defined source. The module that enables the crawler to access the source is called a crawler plug-in (or connector).
The plug-in collects document URLs and associated metadata from the user-defined source and returns the information to the Oracle SES crawler. The crawler starts processing each URL as it is collected. The crawler plug-in must be implemented in Java using the Oracle SES Crawler Plug-in API. Crawler plug-ins go in the $ORACLE_HOME/search/lib/plugins directory.
Here are the basic steps to build a crawler plug-in.
Compile and Build the Plug-in Jar File
The Java source code for the plug-in first must be compiled into class files and put into a jar file in the $ORACLE_HOME/search/lib/plugins/ directory. The library needed for compilation is $ORACLE_HOME/search/lib/search_sdk.jar.
Create a Source Type
Before you can create a source for the crawler plug-in, you first must create a source type for it. From the Oracle Secure Enterprise Search Search administration tool, go to the Global Settings - Source Types page and provide the Java class name and jar file name (created in the previous step).
Create the Source
From the Home - Sources page, create a source from the source type you just created. You also must define the parameter for the source type just created.
For example, suppose you want to crawl /scratch/test on a Linux box for the file crawler plug-in. Specify the seed URL like as follows: file://localhost/scratch/test.
Run the Crawler Plug-in
From the Home - Schedules page, start the schedule for the crawler.
This section includes the following topics:
See Also:
Oracle SES developer tutorial for a guide to using the Crawler Plug-in API:http://st-curriculum.oracle.com/tutorial/SESDevTutorial/index.htm
The following diagram illustrates the crawler plug-in architecture.
Two interfaces in the Crawler Plug-in API (CrawlerPluginManager and CrawlerPlugin) need to be implemented to create a crawler plug-in. A crawler plug-in does the following:
Provides the metadata of the document in the form of document attributes.
Provides access control list information (ACL) if the document is protected.
Maps each document attribute to a common attribute name used by end users.
Optionally provides the list of URLs that have changed since a given time stamp.
Optionally provides an access URL in addition to the display URL for the processing of the document.
Provide the document contents in the form of a Java Reader. In other words, the plug-in is responsible for fetching the document.
Can submit "attribute-only" documents to the crawler; that is, a document that has metadata but no document contents.
Document attributes, or metadata, describe document properties. Some attributes can be irrelevant to your application. The crawler plug-in creator must decide which document attributes should be extracted and saved. The plug-in also can be created such that the list of collected attributes are configurable. Oracle SES automatically registers attributes returned by the plug-in. The plug-in can decide which attributes to return for a document.
Any other Java class needed by the plug-in should be included in the plug-in jar file. (You could add the paths for the additional jar files needed by the plug-in into the Class-Path of the MANIFEST.MF file in the plug-in jar file.) This is because Oracle SES automatically adds the plug-in jar file to the crawler Java class path, and Oracle SES does not let you add other class paths from the administration interface.
If the plug-in code also relies on a particular library file (for example, a .dll file on Windows or a .so file on UNIX), then the library must be put under the $ORACLE_HOME/lib directory or the $ORACLE_HOME/search/lib/plugins directory. The Java library path is set explicitly by the crawler to those locations.
The plug-in must handle mimetype rejection and large document rejection itself. For example, the plug-in should reject files it does not want to index based on its type or size, such as zip files. Also, plain text files, such as log files, can grow very large. Because the crawler reads HTML and plain text files into memory, it could run out of memory with very large files.
This section describes aspects of the crawler plug-in.
Source registration is automated. After a source type is defined, any instance of that source type can be defined:
Source name
Description of the source; limit to 4000 bytes
Source type ID
Default language; default is 'en' (English)
Parameter values; for example:
seed - http://www.oracle.com depth – 8
You can add new attributes to Oracle SES by providing the attribute name and the attribute data type. The data type can be string, number, or date. Attributes returned by an plug-in are automatically registered if they have not been defined.
The crawler plug-in has the following requirements:
The plug-in must be implemented in Java.
The plug-in must support the Java plug-in APIs defined by Oracle SES.
The plug-in must return the URL attributes and properties.
The plug-in must decide which document attributes Oracle SES should keep. Any attribute not defined in Oracle SES is registered automatically.
The plug-in can map attributes to source properties. For example, if an attribute "ID" is the unique ID of a document, then the plug-in should return (document_key, 4) where "ID" has been mapped to the property "document_key" and its value is 4 for this particular document.
If the attribute LOV is available, then the plug-in returns them upon request.
The Crawler Plug-in API is a collection of classes and interfaces used to implement a crawler plug-in.
Table 9-5 Crawler Plug-in Interfaces and Classes
| Interface/Class | Description |
|---|---|
|
|
This interface is used by the crawler plug-in to integrate with the Oracle SES crawler. The Oracle SES crawler loads the plug-in manager class and invokes the plug-in manager API to obtain the crawler plug-in instance. Each plug-in instance is run in the context of a thread execution. |
|
|
This interface is used to generate the crawler plug-in instances. It provides general plug-in information for automatic plug-in registration on the administration page for defining user-defined source types. It has the control on which plug-in object (if more than one implementation is available) to return in The |
|
|
This interface is used by a crawler plug-in to perform crawl-related tasks. It has execution context specific to the crawling thread that invokes the plug-in |
|
|
This interface is implemented by the Oracle SES crawler and made available to the plug-in through the This interface is used by a crawler plug-in to manage the current crawled document set. |
|
|
This interface is used by a crawler plug-in to submit access control list (ACL) information for the document. |
|
|
This interface is used by a crawler plug-in to submit or retrieve document information. |
|
|
This interface holds a document's attributes and properties for processing and indexing. This interface is used by a crawler plug-in to submit URL-related data to the crawler. |
|
|
This interface is used by a document service plug-in to submit document attributes and/or document contents to the crawler. |
|
|
This interface is used to register the document service plug-in. It is also used by the crawler to create a DocumentService object. |
|
|
This interface provides Oracle SES service and implemented interface objects to the plug-in. It is implemented by the Oracle SES crawler and made available through plug-in manager initialization. This interface is used by a crawler plug-in to obtain Oracle SES interface objects. |
|
Logger |
This interface is used by a crawler plug-in to output messages to the crawler log file. |
|
LovInfo |
This interface is used by both the crawler and a source plug-in to set and retrieve attribute list of values (LOV) from the source. |
|
|
This interface for a crawler plug-in reads the value of the source parameter. |
|
|
This interface is implemented by the Oracle SES crawler and made available to the plug-in through the This interface is used by the crawler plug-in to submit URL-related data to the crawler. |
|
|
This class describes the general properties of a parameter. |
|
|
This provides a severity code to direct the crawler's response. |
|
|
This class encapsulates information about a data source plugin-specific error. |
|
|
This class encapsulates information about errors from processing plug-in requests. |
|
|
The crawler manager class must implement this interface in order to use a crawler that uses attribute-based security. |
The Document Service API is included as part of the Crawler Plug-in API. It is used to register a document service plug-in. A document service accepts input from documents and performs some operation on it. For example, you could create a document service for auditing or to show custom metatags.
Note:
The Secure Enterprise Search Document Summarizer is a document service included by default for search result clustering. It extracts the most significant phrases (and optionally sentences) for a document.A document service plug-in (or document service instance) is a Java class that implements the document service API. A document service plug-in accepts document content and attributes to come up with revised document content and new attributes. The title, author, and description attribute values are always used for search hit display. A document service plug-in can also set the document language or replace the input document content with a revised or filtered document.
A document service pipeline is a list of document service instances invoked in the order of the list. The same instance can be assigned to different pipelines, but it cannot be assigned twice in the same pipeline. You can have multiple pipeline definitions; for example, one could be used globally and the other used for certain sources. Not every instance needs to be in a pipeline.
In the administration tool, you can set a global pipeline for all sources on the Global Settings - Crawler Configuration page. Set individual sources to use a particular pipeline on the Home - Sources - Crawling Parameters page for each source. If enabled, the global pipeline is used for all sources, unless a local service pipeline is defined.
DocumentServiceManager is an interface used by the Oracle SES administration tool to register the document service plug-in. When you create a document service, you select the type of document service manager. You can either create a new document service manager or select from the list of already created document service managers.
You can create document service managers, instances and pipelines on the Global Settings - Document Service page.
When a document service is invoked, the document parsing, attribute extraction, and language detection has already been done. The crawler only honors the change made by the document service plug-in, and then the document is cached for indexing.
You must perform a force re-crawl on a source if you add or change the document service pipeline for the source.
Follow these steps to create and use a document service plug-in:
Create a new Java file implementing the DocumentService interface init, close, and process methods; for example, DocumentSummarizer.java.
Create a new Java file implementing the DocumentServiceManager interface; for example, DocumentSummarizerManager.java.
Compile all of the related Java files into class files. For example:
$ORACLE_HOME/jdk/bin/javac -classpath $ORACLE_HOME/search/lib/search_sdk.jar DocumentSummarizer.java
Package all the class files into a jar file under the $ORACLE_HOME/search/lib/plugins/doc directory. For example:
$ORACLE_HOME/jdk/bin/jar cv0f $ORACLE_HOME/search/lib/plugins/doc/extractor/extractor.jar DocumentSummarizer.class DocumentSummarizerManager.class
Note:
the document service plug-in jar file must be deployed under the$ORACLE_HOME/search/lib/plugins/doc directory.From the Global Setting - Document Service page, register the jar file as a new document service plug-in where the jar file name is extractor/extractor.jar and the service plug-in manager class name is oracle.search.plugin.doc.extractor.DocumentSummarizerManager.
After a document service plug-in is registered, an instance can be created from it.
A URL rewriter is a user supplied Java module that implements the Oracle SES UrlRewriter Java interface. When activated, it is used by the crawler to filter and rewrite extracted URL links before they are inserted into the URL queue.
Note:
The URL Rewriter API is included as part of the Crawler Plug-in SDK. The URL Rewriter API is used for Web sources.Web crawling generally consists of the following steps:
Get the next URL from the URL queue. (Web crawling stops when the queue is empty.)
Fetch the contents of the URL.
Extract URL links from the contents.
Insert the links into the URL queue.
The generated new URL link is subject to all existing boundary rules.
There are two possible operations that can be done on the extracted URL link:
Filtering: removes the unwanted URL link
Rewriting: transforms the URL link
Users control what type of URL links are allowed to be inserted into the queue with the following mechanisms supported by the Oracle SES crawler:
robots.txt file on the target Web site; for example, disallow URLs from the /cgi directory
Hosts inclusion and exclusion rules; for example, only allow URLs from www.example.com
File path inclusion and exclusion rules; for example, only allow URLs under the /archive directory
Mimetype inclusion rules; for example, only allow HTML and PDF files
Robots metatag NOFOLLOW; for example, do not extract any link from that page
Black list URL; for example, URL explicitly singled out not to be crawled
With these mechanisms, only URL links that meet the filtering criteria are processed. However, there are other criteria that users might want to use to filter URL links. For example:
Allow URLs with certain file name extensions
Allow URLs only from a particular port number
Disallow any PDF file if it is from a particular directory
The possible criteria could be very large, which is why it is delegated to a user-implemented module that can be used by the crawler when evaluating an extracted URL link.
For some applications, due to security reasons, the URL crawled is different from the one seen by the end user. For example, crawling is done on an internal Web site behind a firewall without security checking, but when queried by an end user, a corresponding mirror URL outside the firewall must be used.
A display URL is a URL string used for search result display. This is the URL used when users click the search result link. An access URL is a URL string used by the crawler for crawling and indexing. An access URL is optional. If it does not exist, then the crawler uses the display URL for crawling and indexing. If it does exist, then it is used by the crawler instead of the display URL for crawling.
For regular Web crawling, there are only display URLs available. But in some situations, the crawler needs an access URL for crawling the internal site while keeping a display URL for the external use. For every internal URL, there is an external mirrored one.
For example:
http://www.example-qa.us.com:9393/index.html http://www.example.com/index.html
When the URL link http://www.example-qa.us.com:9393/index.html is extracted and before it is inserted into the queue, the crawler generates a new display URL and a new access URL for it:
Access URL:
http://www.example-qa.us.com:9393/index.html
Display URL:
http://www.example.com/index.html
The extracted URL link is rewritten, and the crawler crawls the internal Web site without exposing it to the end user.
Another example is when the links that the crawler picks up are generated dynamically and can be different (depending on referencing page or other factor) even though they all point to the same page. For example:
http://compete3.example.com/rt/rt.wwv_media.show?p_type=text&p_id=4424&p_currcornerid=281&p_textid=4423&p_language=us http://compete3.example.com/rt/rt.wwv_media.show?p_type=text&p_id=4424&p_currcornerid=498&p_textid=4423&p_language=us
Because the crawler detects different URLs with the same contents only when there is sufficient number of duplication, the URL queue could grow to a huge number of URLs, causing excessive URL link generation. In this situation, allow "normalization" of the extracted links so that URLs pointing to the same page have the same URL. The algorithm for rewriting these URLs is application dependent and cannot be handled by the crawler in a generic way.
When a URL link goes through a rewriter, there are the following possible outcomes:
The link is inserted with no changes made to it.
The link is discarded; it is not inserted.
A new display URL is returned, replacing the URL link for insertion.
A display URL and an access URL are returned. The display URL might or might not be identical to the URL link.
Follow these steps to create and use a URL rewriter:
Create a new Java file implementing the UrlRewriter interface open, close, and rewrite methods.
Compile the rewriter Java file into a class file. For example:
$ORACLE_HOME/jdk/bin/javac -classpath $ORACLE_HOME/search/lib/search_sdk.jar SampleRewriter.java
Package the rewriter class file into a jar file under the $ORACLE_HOME/search/lib/plugins/ directory. For example:
$ORACLE_HOME/jdk/bin/jar cv0f $ORACLE_HOME/search/lib/plugins/sample.jar SampleRewriter.class
Enable the UrlRewriter option and specify the rewriter class name and jar file name (for example, SampleRewriter and sample.jar) in the administration tool Home - Sources - Crawling Parameters page of an existing Web source
Crawl the target Web source by launching the corresponding schedule. The crawler log file confirms the use of the URL rewriter with the message Loading URL rewriter "SampleRewriter"...
Note:
URL rewriting is available for Web sources only.See Also:
Oracle Secure Enterprise Search Java API Reference for the API (oracle.search.sdk.crawler package)In addition to the extensible crawler plug-in framework that lets you crawl and index proprietary document repositories (Crawler Plug-in API), Oracle SES also includes an extensible authentication and authorization framework. This lets you use any identity management system to authorize users (Identity Plug-in API). You can also define your own security model for each source (Authorization Plug-in API).
The Identity Plug-in API communicates with the identity management system to authenticate a user at login with a user name and password. It also provides a list of groups (or roles) for a specified user.
The identity plug-in manager manages initialization parameters and returns the IdentityPlugin object.
To add an identity plug-in, click Register New Identity Plug-in on the Global Settings - Identity Management Setup page, and enter the class name and jar file name for the identity plug-in manager.
For sources with authorization requirements that do not fit the user/group model, an authorization plug-in provides a more flexible security model. (Authentication is still handled by an identity plug-in.)
With an authorization plug-in, a crawler plug-in can add security attributes similar to document attributes. The authorization plug-in is invoked at login time to build security filters onto the query string. The security filters are applied against the values of the security attributes for each document. Only documents whose security attributes' values match the security filter are returned to the user. (All security attributes have string values.)
The authorization plug-in can contain one or both of the following components:
QueryFilterPlugin: This gets a list of security attributes for the current user. It can return a list of groups (or security roles) of which the user is a member. For example, if resp is a GRANT security attribute for responsibilities and if User1 is logged in, then QueryFilterPlugin.getSecurityValues("resp") should return an array of values corresponding to the responsibilities of User1. These values can be used to build a filter to return the documents authorized for User1 and her responsibilities.
ResultFilterPlugin: This implements query-time authorization (QTA). When building the hitlist, Oracle SES calls a result filter plug-in to check if the user is authorized to view to each document. Only documents the user is authorized to view will be listed in the hitlist. The result filter can be used as the only security device., or it can be used together with other security. The result filter can also be used to modify the title or display URL.
Note:
ResultFilter is the preferred method of implementing QTA (as opposed to QueryTimeFilter).With the user-defined security model, Oracle SES takes the administrator to an Authorization page before a new user-defined source can be defined. The UserDefinedSecurityModel interface provides a method that returns the name of the class implementing the AuthorizationManager interface and the names and types (GRANT or DENY) of the security attributes used to build the security filter for a given user.
If the AuthorizationManager plug-in class name or jar file name must be changed, then security for that source must be turned off to allow the change. After the ACL setting is changed to "No Access Control List" and applied, the AuthorizationManager details can be edited. Warning: While security is turned off, any user can access documents in the affected source. In addition, the new AuthorizationManager should share the same security attribute model as the previous one.
See Also:
Oracle Secure Enterprise Search Java API Reference for the API (oracle.search.sdk.security package)Query-time authorization allows an Oracle SES administrator to associate a Java class with a source that will, at search time, validate every document fetched out of the Oracle SES repository belonging to the protected source. This result filter class can dynamically check access rights to make sure that the current search user has the credentials to view each document.
This authorization model can be applied to any source other than self service or federated sources. Besides acting as the sole provider of access control for a source, it can also be used as a post-filter. For example, a source can be stamped with a more generic ACL, while query-time authorization can be used to fine tune the results.
Query-time authorization has the following characteristics:
It allows dynamic access control at search time compared to more static ACL stamping.
It filters documents returned to a search user.
It controls the Browse functionality to determine whether a folder is visible to a search user.
Optionally, it allows pruning of an entire source from the results to reduce performance costs of filtering each document individually.
It is applicable to all source types except self service and federated sources.
The result filter can modify the Title or Display URL for the result returned to the search user.
Query-time filtering is handled by class implementations of the ResultFilterPlugin interface.
Filtering document access is handled by the filterDocuments method of the ResultFilterPlugin interface. The most common situation for filtering will occur with a search request, in which this method will be invoked with batches of documents from the result list. Based on the values returned by this method, all, some, or none of the documents might be removed from the results returned to the search user.
Access of individual documents is also controlled. For example, viewing a cached copy of a document or accessing the in-links and out-links will require a call into filterDocuments to determine the authorization for the search user.
The ResultFilterPlugin implementation is also responsible for controlling the access to, and visibility of folders in, the Browse application. If a folder belongs to a source protected by a query-time filter, then the folder name in the Browse page will not have a document count listed next to it. Instead, the folder will show a view_all link.
For performance reasons, it could be costly to determine the exact number of documents visible to the current search user for every query-time filtered folder displayed on a Browse page. This task would require that every document in every folder be processed by the filter in order to calculate the total number of documents available for each folder. To prevent this comprehensive and potentially time-consuming operation, document counts are not used. Instead, folder visibility is explicitly determined by the query-time filter.
Based on the results from the filterBrowseFolders method, a folder might be hidden or shown in the Browse page. This result also controls access to the single folder browsing page, which displays the documents contained in a folder.
If the security of folder names is not a concern for a particular source, then the filterBrowseFolders method can blindly authorize all folders to be visible in the Browse application. After a folder is selected, the document list is still filtered through the filterDocuments method. This strategy should not be employed if folder names could reveal sensitive information.
If security is very critical, then it might be easiest to hide all folders for browsing. The documents from the source will still be available for search queries from the Basic and Advanced Search boxes, but a user will not be able to browse the source in the Browse pages of the search application.
Limitations of folder filtering:
The filterBrowseFolders method does not implicitly restrict access to subfolders. For example, if folder /Miscellaneous/www.example.com/private is hidden for a search user, then it is still possible for that user to view any subfolder, such as /Miscellaneous/www.example.com/private/a/b, if that subfolder is not also explicitly filtered out by this method. It would be possible to view this subfolder if the user followed a bookmark or outside link directly to the authorized subfolder in the Browse application.
This method does not affect functionality outside of the Browse application. This is not a generic folder pruning method. Search queries and document retrieval outside of the Browse application are only affected by the filterDocuments and pruneSource methods.
The ResultFilterPlugin interface provides the ability to determine access privileges at the source level. This is achieved through calls to the pruneSource method. This method can be called in situations where there are a large number of documents or folders to be filtered. Authorizing or unauthorizing the entire source for a given user could provide a large performance gain over filtering each document individually.
The implementation of ResultFilterPlugin must not rely on this method to secure access to documents or folders. This method is strictly an optimization feature. There is no guarantee that this will ever be invoked for any particular search request or document access. For example, when performing authorization for a single document, Oracle SES may call the filterDocuments method directly without invoking this method at all. Therefore, the filterDocuments and filterBrowseFolders methods must be implemented to provide full security in the absence of pruning.
A query-time filter is free to define a search user's access privileges to sources and documents based on any criteria available. For example, a filter could be written to deny access to a source depending on the time of day.
In most cases, however, a filter will impose restrictions based on the authenticated user for that search request. The Oracle SES authenticated user name for a request is contained in the RequestInfo object. The steps for accessing this user name value depend on whether the request originated from the JSP search application or the Oracle SES Query Web Services interface. For either type of request, the key used to access the authenticated user name is the string value AUTH_USER.
Note:
User name is not case-sensitive.This sample implementation of the ResultFilterPlugin.getCurrentUserName method illustrates how to retrieve the current authenticated user from either a JSP or Web Services request:
public String getCurrentUserName( RequestInfo req )
throws PluginException
{
HttpServletRequest servReq = req.getHttpRequest();
Map sessCtx = req.getSessionContext();
String user = null;
if( servReq != null )
{
HttpSession session = servReq.getSession();
if( session != null )
user = ( String ) session.getAttribute( "AUTH_USER" );
}
else if( sessCtx != null )
{
// Web Service request
user = ( String ) sessCtx.get( "AUTH_USER" );
}
if( user == null )
user = "unknown";
return user;
}
See Also:
"Authentication Methods"The oracle.search.sdk package contains all interfaces and exceptions for the Query-time Authorization API.
To write a query-time authorization filter, implement the ResultFilterPlugin interface. The methods in this interface may throw instances of PluginException.
Objects that implement the RequestInfo, DocumentInfo, and FolderInfo interfaces are passed in as arguments for filtering, but these interfaces do not need to be implemented by the filter writer.
The API contains the following interfaces and exceptions:
Table 9-6 Query-time Authorization Interfaces and Exceptions
| Interface/Exception | Description |
|---|---|
|
|
This interface filters search results and access to document information at search time. If an object implementing this interface has been assigned to a source, then any search results or other retrieval of documents belonging to the source are passed through this filter before being presented to the end user. |
|
|
This exception is thrown by methods in the |
|
|
This interface represents information about a document that can be passed to a |
|
|
This interface represents information about a folder that can be passed to a |
|
|
This interface represents information about a request that can be passed to a |
Classes that implement the ResultFilterPlugin interface should be designed to persist for the lifetime of a running Oracle SES search application. A single instance of ResultFilterPlugin will generally handle multiple concurrent requests from different search end users. Therefore, the filterDocuments, pruneSource, filterBrowseFolders, and getCurrentUserName methods in this class must be both reentrant and thread-safe.
To compile your query-time filter class, you will need to include at least the two following files in the Java CLASSPATH. These files can be found in the Oracle SES server directory.
$ORACLE_HOME/search/lib/search_query.jar
$ORACLE_HOME/lib/servlet.jar
It is recommended to build a jar file containing your ResultFilterPlugin class (or classes) and any supporting Java classes. This jar file should be placed in a secure location for access by the Oracle SES server. If this jar file is compromised, then the security of document access in the search server can be compromised.
Your query-time filter might require other class or jar files that are not included in the jar file you build and are not located in the Oracle SES class path. If so, these files should be added to the Class-Path attribute of the JAR file manifest. This manifest file should be included in the jar file you build.
If Oracle SES cannot locate a class used by a ResultFilterPlugin during run-time, then an error message will be written to the log file and all documents from that source will be filtered out for the search request being processed.
See Also:
http://java.sun.com/j2se/1.4.2/docs/guide/jar/jar.html for more information about jar file manifests