Search Robot

The robot examines a set of selected URLs and searches for documents. For each found document, the robot then creates a resource description (RD) of the document using a predefined schema. The schema defines what pieces of information about the document are put in the RD. For example, the RD could contain a date, the author, the title, the URL, and an abstract about the document. These RDs can be grouped together or classified according to a given hierarchical taxonomy.

You can configure the robot through the Portal Server administration console.

The robot has many customizeable parameters, including the following configuration parameters:

• The URLs that it starts crawling from

• Server access delays

• Passwords

• User agent string

• Certificates for SSL

• Proxy setup

In addition, the robot API enables you to write custom content parsers and summarizers for special URL handling requirements. You can also use the robot API to remove advertisements, generate alerts when certain pages are found, and perform specialized logging.