Search Attributes

Appendix F
Search Attributes

This appendix describes attributes that you can configure for the search engine through the Sun ONE Identity Server administration console.

When you select Search Properties from the Service Configuration view, a two-toned menu bar is displayed. This appendix is organized according to the topics on the upper or dark blue portion of the menu bar.

Server

Robot

Database

Table F-1 Search Select Server Attribute
Attribute	Default Value	Description
Select Server	http://servername:80/portal	Fully qualified server name of your Search server.

Server

The Server section is where you configure the preferences for your server. You select what directory to use for temporary files, what information to log and how much detail should be in the logs. The Server attributes are displayed on two pages:

Settings

Advanced

Settings

This page contains the basic settings for the administration and operation of the search server.

Table F-2 Server Settings Attributes
Attribute	Default Value	Description
Server Root	/var/opt/SUNWps/https-servernamefull/portal	Houses the configuration, log, database, and robot information files. Also it is the root directory for all of the search files that are generated and updated when conducting a search. This is not configurable.
Temporary Files	/var/opt/SUNWps/https-servernamefull/portal/tmp	Contains all temporary files used to manage a search during the search. It includes newly generated resource descriptions that have not yet been added to the main database. These are removed when the search is completed.
Document level security	Off	Controls who can access documents. When this setting is changed, the server must be restarted. Values: Off (default) means all users have access to the RDs. On means that the ReadACL field in an RD is checked to see if the user asking for the RD has permission because the user is in an acceptable organization or role, or is an acceptable individual user. The ReadACL field is set in the Database, Edit page.

Advanced

This page contains the advanced settings for the administration and operation of the search server. Here is where you configure the log files for user queries, index maintenance, resource description management, and debugging.

Table F-3 Server Advanced Settings Attributes
Attribute	Default Value	Description
Search (rdm)	/var/opt/SUNWps/https-servername/portal/logs/rdm.log	Logs the queries end users make of the database. You can check the Disable Search Log checkbox to suppress this logging. If you do, you cannot view the User Queries (rdm) report.
Disable Search Log	False (unchecked) - enabled	Controls use of query log. In the report section, you can generate a report the lists the most popular queries based on this log. Values: Checked—disabled Unchecked—enabled. Every user query is entered in this log.
Index Maintenance	/var/opt/SUNWps/https-servername/portal/logs/searchengine.log	Logs the transactions involving the search engine, except for not registration of resource descriptions.
RD Manager	/var/opt/SUNWps/https-servername/portal/logs/rdmgr.log	Logs the registration of resource descriptions from the robot or import agents into the database. You can view this log as a RD Manager (rdmgr) report.
RDM Server	/var/opt/SUNWps/https-servername/portal/logs/rdmserver.log	Logs debugging information on RDM transactions. The level of detail is controlled by the Log Level. You can view this log as a RDM Server (rdmsvr) report.
Log Level	1	Controls the amount of detail the RDM Server log file contains. The possible levels are 2, 10, 20, 50, 100, and 999. A setting of 1 (default) logs only severe errors. The higher the number, the more detail the RDM Server log file contains.

Robot

The properties for the robot are quite complex. You can select the sites to be searched or crawled, check to see if a site is valid, define what types of documents should be picked up, and schedule when the searches take place.

This section is organized as follows:

Control

The Robot Control panel is where you can see what the robot is doing: if it is Off, Idle, Running, or Paused; and if it is Running, what progress it is making in the search since the panel is refreshed about every 30 seconds. The refresh rate is defined using the robot-refresh parameter in the search.conf file.

The two buttons on the top right are appropriate for its state. If the robot is Off, the buttons are Start and Remove Status. If it is Running or Idle, the two buttons are Stop and Pause. If it is Paused, the two buttons are Stop and Resume. By selecting on any of the Attributes, you go to the Reports section where you can get a detailed up-to-the-minute report of that Attribute.

Table F-4 Robot Control Attributes
Attribute	Default Value	Description
The Robot is	Current activity	The Robot’s state. Value can be Idle, Running, Paused, or Off
Updated at date	Date and time last refreshed.	This page is refreshed to keep you aware of what progress the robot is making.
Starting Points	Number defined	Number of sites you have selected to be searched. A site is disabled (not included in a search) on the Robot, Site page.
URL Pool	Number URLs waiting	Number of URLs yet to be investigated. When you begin a search, the starting point URLs are entered into the URL pool. As the search progresses, the robot discovers links to other URLs. These URLs get added to the pool. After all the URLs in the pool have been processed, the URL pool is empty and the robot is idle.
Extracting	Number connections per second	Number of resources looked at in a second. Extracting is the process of discovering or locating resources, documents or hyperlinks to be included in the database and filtering out unwanted items.
Filtering	Number URLs rejected	Total number of URLs that are excluded.
Indexing	Number URLs per second	Number of resources or documents turned into a resource description in a second. Indexing is the phase when all the information that has been gathered on a document is turned into a resource description for inclusion in the search database.
Excluded URLs	Number URLs excluded by filters	Number of URLs that did not meet the filtering criteria.
	Number URLs excluded by errors	Number of URLS where the robot encountered errors as file not found.
Resource Descriptions	Number RDs contributed	Number of resource descriptions added to the database.
	Number Bytes of RDs contributed	Number of bytes added to the database.
General Stats	Number URLs retrieved	Number of URLs retrieved during run.
	Number Bytes average size of RDs	Average number of bytes per resource description.
	Time in days, hours, minutes, and seconds running	The amount of time the robot has been running.

Sites

The initial page in this section shows what sites are available to be searched.

A site can be enabled (On) and disabled (Off) by using the radio buttons. A disabled site is not searched when the robot is run. The Edit link displays a page where you can change how a search site is defined.

To delete a site, check the checkbox and select Delete.

To add a new site, select New. Add a URL or Domain in the text box and select a depth for the search. Select Create to use the default values. Otherwise, select Create and Edit to select non-default values and go to the Edit page to define the search site.

Table F-5 Robot Manage Sites Attributes
Attribute	Default Value	Description
Checkbox to select site	Unchecked	Unchecked—not selected Checked—selected
Lock or cluster graphic	Status of site	Lock open means that the URL is accessible. The closed lock means that the site is a secure web server and uses SSL. The cluster means that the site is a domain.
On/Off	On	Choose to search this site or not when the robot is run.

The New Site page allows you to set up an entire site for indexing.

Table F-6 Robot New Site Attributes
Attribute	Default Value	Description
New site	URL	URL - format: http://www.sesta.com Domain - format: *.sesta.com
Depth	10	You have a choice of 1 for this URL only, 2 for this URL and first links, 3 - 10 or unlimited. The default value is set on the Robot, Crawling page.

The edit page is where you can define the search site more completely. You can specify what type of server it is, redefine the depth of the search, and select what type of files to add to the search database. The attributes for URL and Domain sites are mostly the same. The additional column in this table shows which attributes are shared and which are unique.

A number of actions are performed on this page. You can verify the server name for the search site you entered. You can add more servers to the server group by selecting Add in the Server Group section. You can add more starting points by selecting Add in the Starting Points section. In the Filter Definition section, you can add or delete, exclude or include certain types of files as well as change the order the filters for these files are applied. For more detail on these actions, see Chapter 9, ""” in the Sun ONE Portal Server 6.1 Administrator’s Guide.

Table F-7 Robot Sites Edit Attributes
Attribute	URL/ Domain	Default Value	Description
Site Nickname	URL/D	Site entered - www.sesta.com	Name that is displayed on the initial page. The default is the URL or domain you entered. You can change this name here.
Checkbox to select site for deletion or verification	URL/D	Unchecked	Unchecked—not selected Checked—selected
Server Group - Name	URL	URL - www.sesta.com	Is either a single server or a part of a single server. The entry must include the full host name. If you specify just a host name, the site is limited to that host. If you provide directory information in addition to the host name, the site is defined as only that directory and any of its subdirectories.
Domain Suffix	D	Domain entered - *.sesta.com	Includes all servers within a domain, such as *.sesta.com.
Port	URL/D	80 for URL; blank for Domain	If the site you are searching uses a different port, enter it here.
Type	URL	Web Server	Web Server, File Server, FTP Server, Secure Web Server
Allowed Protocols	D	Checkboxes all checked	Checkboxes for http, file, ftp, https
Starting Points- Checkbox to select site for deletion	URL/D	Unchecked	Unchecked—not selected Checked—selected
Starting Points- URL	URL/D	http:// URL:80	URL or domain
Starting Points - Depth	URL/D	10	1 - this URL only 2 - this URL and first links 3-10 unlimited
Filter Definition - Checkbox to select file type for deletion	URL/D	Unchecked	Unchecked - not selected Checked - selected
Filter Definitions	URL/D	In this order, the defaults are Archive Files; Audio Files; Backup Files; Binary Files; CGI Files; Image Files; Java, Javascript, Style Sheet Files; Log Files; Revision Control Files; Source Code Files; Temporary Files; Video Files.	The possible choices are Archive Files; Audio Files; Backup Files; Binary Files; CGI Files; Image Files; Java, Javascript, Style Sheet Files; Log Files; Power Point Files; Revision Control Files; Source Code Files; Temporary Files; Video Files; Spreadsheet Files; Plug-in Files; Lotus Domino Documents; Lotus Domino OpenViews; System Directories (UNIX); System Directories (NT).
Comment	URL/D	Blank	Text field that describes the site to you. It is not used by the robot.
DNS Translation	URL	Blank	The DNS translation modifies the URL and the way it is crawled by replacing a domain name or alias with a cname. Format: alias1->cname1,alias2->cname1

All of the filters available are listed. This is the same list that is found in the Robot Sites Edit page under Filter Definitions. Each filter name is proceeded by a checkbox to select that document type and two radio buttons to turn the Filter Rule On and Off.

If a checkbox is checked, the filter is selected and can be deleted. You can add a new filter by selecting New. The new filter page is an abbreviated Edit page, requiring only a Nick Name and one rule. Another option is selecting the Edit link, which takes you to a page where you define the rules for that file type or what that filter does. Each rule is made up of a drop down list of Filter Sources, a drop down list to Filter By, and a text box to enter specifics in.

Table F-8 Robot Filter Edit Attributes
Attribute	Default Value	Description
Nick Name	Prompts for new name. File name of the file type you choose to edit.	A descriptive name that reflects the type of file the filter applies to.
Drop down list of Filter Sources	URL for new filter. Displays previously chosen information for that particular file type.	URL, protocol, host, path, MIME type
Drop down list of positions	is for new filter. Displays previously chosen information for that particular file type. For example, Binary Files ends with exe.	is, contains, begins with, ends with, regular expression
Text box for type (directory, protocol, file extensions) specifics	Blank for new filter. Displays previously entered information for that particular file type. For example, Temporary Files contains /tmp/.	In this text box, list what you want to match. What would match in this example - http://docs.sesta.com/manual.html protocol is http; host contains sesta; file ends with html.
Description	Prompts for new description. Displays previously entered description for that particular file type.	Describe the filter rule for yourself. The robot does not use it.
New Site	True (checked) for new filter. Displays previously chosen value for that particular file type.	Use this as one of the default filters when creating new sites. If you do not check this, you can still add this filter to a new site by editing the site on the Robot, Sites page.
By Default	Nothing selected for a new filter. Default selected previously for a defined file type.	Exclude documents matching this filter. Include documents matching this filter. Selection for a new filter does not affect existing site definitions. To use your new filter on an existing site, you must add it by editing the site on the Robot, Sites page.
Deployment	Lists the sites that use this filter.

Crawling

The settings on this page control the robot’s operational parameters and defaults. It is divided into these sections: Speed, Completion Actions, Logfile Settings, Standards Compliance, Authentication Parameters, Proxying, Advanced Settings and Link Extraction. For information on the tasks to set these attributes, see Chapter 9, "Administering the Search Engine Service"” of the Sun ONE Portal Server 6.1 Administrator’s Guide.

Table F-9 Robot Crawling Attributes
Attribute	Default Value	Description
Server Delay	No Delay	No Delay (default), 1 second, 2 seconds, 5 seconds, 10 seconds, 30 seconds, 1 minute, 5 minutes.
Maximum Connections - Max concurrent retrieval URLs	8	1, 2, 4, 8 (default), 10, 12, 16, 20.
Maximum Connections per Site	2	(no limit), 1, 2, 4, 8, 10, 12, 16, 20.
Send RDs to Indexing every	30 minutes	3 minutes, 5 minutes, 10 minutes, 15 minutes, 30 minutes (default), 1 hour, 2 hours, 4 hours, 8 hours.
Script to Launch	nothing (default)	nothing (default). For sample files, see the cmdHook files in the /opt/SUNWps/samples/robot directory (for the default installation).
After Processing all URLs	go idle (default)	go idle (default), shut down, start over.
Contact Email	user@domain	Enter your own.
Log Level	1 - Generation	0 Errors only; 1 Generation (default); 2 Enumeration, Conversion; 3 Filtering; 4 Spawning; 5 Retrieval
User Agent	SunONERobot/6.0	Version of the search server.
Ignore robots.txt protocol	False (unchecked)	Some servers have a robot.txt file that says robots do not come here. If your search robot encounters this file on a site and this attribute is false, it does not search the site. If this attribute is true, the robot ignores the file and searches the site.
Perform Authentication	Yes	Yes No
Robot Username	anonymous	Robot uses the anonymous user name to gain access to a site.
Password	user@domain	Frequently a site that allows anonymous users requires a email address as a password. This address is in plain text.
Proxy Username	anonymous	Robot uses the anonymous user name to gain access to a site.
Password	user@domain	Frequently a site that allows anonymous users requires an email address as a password. This address is in plain text.
Proxy Connection Type	Direct Internet Connection	Direct Internet Connection, Proxy—Auto Configuration, Proxy—Manual Configuration
Auto Proxy Configuration Type	Local Proxy File	Local Proxy File, Remote Proxy File
Auto Proxy Configuration Location	Blank	The auto proxy has a file that lists all the proxy information needed. An example of a local proxy file is robot.pac. An example of a emote proxy file is http://proxy.sesta.com:8080/proxy.pac
Manual Configuration HTTP Proxy	Blank	Format: server1.sesta.com:8080 These three manual configuration values are put in the robot.pac file in the /var/opt/SUNWps/https-servername/portal/config directory.
Manual Configuration HTTPS Proxy	Blank	This manual configuration value is put in the robot.pac file. Format: server1.sesta.com:8080
Manual Configuration FTP Proxy	Blank	This manual configuration value is put in the robot.pac file. Format: server1.sesta.com:8080
Follow links in HTML	True (checked)	Extract hyperlinks from HTML
maximum links	1024	Limits the number of links the robot can extract from any one HTML resource. As the robot searches sites and discovers links to other resources, it could conceivably end up following huge numbers of links a great distance from its original starting point.
Follow links in plain text	False (unchecked)	Extract hyperlinks from plain text.
maximum links	1024	Limits the number of links the robot can extract from any one text resource.
Use Cookies	False (unchecked)	If checked, the robot uses cookies when it crawls. Some sites require the use of cookies in order for them to be navigated correctly. The robot keeps its cookies in a file called cookies.txt in the robot state directory. The format of cookies.txt is the same format as used by the Netscape™ Communicator browser.
Use IP as Source	True (checked)	In most cases, the robot operates only on the domain name of a resource. In some cases, you might want to be able to filter or classify resources based on subnets by Internet Protocol (IP) address. In that case, you must explicitly allow the robot to retrieve the IP address in addition to the domain name. Retrieving IP addresses requires an extra DNS lookup, which can slow the operation of the robot. If you do not need this option, you can turn it off to improve performance.
Smart Host Heuristics	False (unchecked)	If checked, the robot converts common alternate host names used by a server to a single name. This is most useful in cases where a site has a number of servers all aliased to the same address, such as www.sesta.com, which often have names such as www1.sesta.com, www2.sesta.com, and so on. When you select this option, the robot will internally translate all host names starting with wwwn to www, where n is any integer. This attribute only operates on host names starting with wwwn. This attribute cannot be used when CNAME resolution is OFF (false).
Resolve hostnames to CNAMEs	False (unchecked)	If checked, the robot validates and resolves any host name it encounters into a canonical host name. This allows the robot to accurately track unique RDs. If unchecked, the robot validates host names without converting them to the canonical form. So you may get duplicate RDs listed with the different host names found by the robot. For example, devedge.sesta.com is an alias for developer.sesta.com. With CNAME resolution on, a URL referenced as devedge.sesta.com is listed as being found on developer.sesta.com. With CNAME resolution off, the RD retains the original reference to devedge.sesta.com. Smart Host Heuristics cannot be enabled when CNAME resolution is OFF (false).
Accepts commands from ANY host	False (unchecked)	Most robot control functions operate through a TCP/IP port. This attribute controls whether commands to the robot must come from the local host system (false), or whether they can come from anywhere on the network (true). It is recommended that you restrict direct robot control to the local host (false). You can still administer the robot remotely through the Administration Console.
Default Starting Point Depth	10	1- starting points only, 2- bookmark style, 3-10, unlimited. Default value for the levels of hyperlinks the robot traverses from any starting point. You can set the depth for any given starting point by editing the site on the Robot, Sites page.
Work Directory	/var/opt/SUNWps/https-servernamefull/portal/tmp	Full pathname of a temporary working directory the robot can use to store data. The robot retrieves the entire contents of documents into this directory, often many at a time, so this space should be large enough to handle all of those at once.
State Directory	/var/opt/SUNWps/https-servernamefull/portal/robot	Full pathname of a temporary directory the robot uses to store its state information, including the list of URLs it has visited, the URL pool, and so on. This database can be quite large, so you might want to locate it in a separate partition from the Work Directory.

Indexing

The robot searches sites and collects documents based on the filters you have selected. The documents collected are in many different formats. To make them uniform and easily readable they need to be in one format, which is HTML. This page controls some of the parts that go into each resource description.

Table F-10 Robot Index Attributes
Attribute	Default Value	Description
Full Text or Partial Text	Partial Text	Full uses the complete document in the resource description. Partial text only uses the specified number of bytes in the resource description.
extract first # bytes	4096	Enter the number of bytes.
Extract Table Of Contents	True (checked)	True includes the Table of Contents in the resource description.
Extract data in META tags	True (checked)	True includes the META tags in the resource description.
Document Converters	All checked (true); if false, that type of document cannot be indexed.	Adobe PDF Corel Presentations Corel Quattro Pro FrameMaker Lotus Ami Pro Lotus Freelance Lotus Word Pro Lotus 1-2-3 Microsoft Excel Microsoft Powerpoint Microsoft RTF Microsoft Word Microsoft Works Microsoft Write WordPerfect StarOffice™ Calc StarOffice™ Impress StarOffice™ Writer XyWrite
Converter Timeout	600	Time in seconds allowed for one document to be converted to HTML. If this time is exceeded, the URL is excluded.

Simulator

This page is a debugging tool that performs a partial simulation of robot filtering on a URL. You can type in a new URL to check. It checks the URL, DNS translations (including Smart Host Heuristics), and site redirections. It does not check the contents of the document specified by the URL, so it does not detect duplications, MIME types, network errors, permissions, and the like. The simulator indicates whether the listed sites would be accepted by the robot (ACCEPTED) or not (WARNING).

Table F-11 Robot Simulator Properties
Attribute	Default Value	Description
URL	URLs you have already defined and one blank text box.	You can check access to a new site by typing its URL in the blank text box. This checks to see if the new site accepts crawling. Format http://www.sesta.com:80/
Check for DNS aliases	True (checked)	True (checked) checks for number of servers aliased to the same address.
Check for Server Redirects (302)	True (checked)	True (checked) checks for any server redirects.

Site Probe

This page is a debugging tool that checks for DNS aliases, server redirects, and virtual servers. This tool returns information about site but does not test its acceptance of crawling.

Table F-12 Robot Site Probe Attributes
Attribute	Default Value	Description
Site	Blank	Type in URL in format http://www.sesta.com:80
Show advanced DNS information	False (unchecked)	True (checked) displays more information about the site including IP addresses.

Schedule

This page is where you set up the automatic search schedule for the robot and activate the cron job.

Table F-13 Robot Schedule Attributes
Attribute	Default Value	Description
Start Robot Time in hours and minutes	00:00	This is the time that the robot starts to search.
Days	none selected	Sun, Mon, Tue, Wed, Thu, Fri, or Sat Check at least one day.
Stop Robot Time in hours and minutes	00:00	If you plan to run the robot continuously, it is recommended that you stop and restart it at least once per day. This gives the robot a chance to release resources and reinitialize itself.
Days	none selected	Sun, Mon, Tue, Wed, Thu, Fri, or Sat
Cron Status	Lists what its current status is.	Off On

Database

The Database attributes are divided as follows:



Note	To partition the database, you must use the command line function because stopping the search server is required.

Import

Import agents are the processes that bring resource descriptions from other servers or databases and merge them into your search database.

The initial Import page lists the available import agents. You can create a new one, or run, edit or delete an existing one. Use the checkbox to select an agent to delete. Use the small icons above the checkbox to select or deselect all import agents. Use the radio buttons to turn an Agent Action On or Off. To schedule the import agents, select Schedule on the lower menu bar.

If you choose to edit or modify an existing import agent or create a new one, the following attributes are displayed.

Table F-14 Database Import Attributes
Attribute	Default Value	Description
Import From	None selected	Select either Local File or Search Server.
Local File	Blank for new	Gives the full path name of local file that contains valid resource descriptions in SOIF (Summary Object Interchange Format). This can be a file on another server, as long as the path is addressable as if it were locally mounted.
Remote Server	Blank for new	Gives the URL of the search server to retrieve resource descriptions from; format http://www.sesta.com:80
Instance Name	Blank for new	Server instance name used by the search server. You can find this instance name in the Server Preferences for the server you are importing from. Value must be 3.01C or 3.01C SP1.
Search URI	blank for new	Enter full path and file names. Use /portal/search.
Is Compass Server 3.01X?	False (unchecked)	Is the server you are importing from a Compass Server 3.01X?
Enable SSL	False (unchecked)	If this is a server-to-server transaction, select if the servers should use the SSL (Secure Sockets Layer) protocol.
Authentication	None (default)	None (default) or Use User/Password This specifies how the import agent should identify itself to the system it imports from. By default, no authentication is used. If the server you want to import from requires authentication, you can specify a user name and password for the import agent to use. Importing from 3.01C does not require authentication. Importing data from 3.01C SP1 requires authentication.
User	Blank for new or none	If you selected Use User/Password, enter a user.
Password	Blank for new or none	If you selected Use User/Password, enter a password (shown as *).
Content Transfer	Use Incremental Gathering of Full Contents (default)	Choice of Use Incremental Gathering of Full Contents (default) or Use Search Query These specify which resource descriptions to import from the source. By default, an import agent asks for all resource descriptions added or changed since its last import from the same source. The search query specifies that the import agent should request only certain resource descriptions from the source. This is much the same way that users request listings of resources from the search database. Use Scope, View-Attributes and View-Hits fields to specify the query.
Scope	Blank for new	The text of the query. The query syntax is identical to that used for end-user queries from the server. For the full syntax available, see Chapter 9, ""” in the Sun ONE Portal Server 6.1 Administrator’s Guide.
View-Attributes	Blank for new	Lists which fields (not case sensitive) you want to import in each resource description. For example, title and author. The default is all.
View-Hits	Blank for new	The maximum number of matching resource descriptions to import. If no hits are specified, it defaults to 20.
Agent Description	Blank for new	Appears in the list of available import agents on the initial Import page. It is ignored by the program. If this field is blank, the Resource Description Source file name or server name is used to identify the import agent. Note here if user name and password are needed.
Newest Resource Description	Blank for new	The date of the creation of the newest resource description previously imported by this import agent. This date is used by the Use Incremental Gathering of Full Contents option to determine which resources are new and should be imported.
Network Timeout in seconds	Blank for new	Specifies the number of seconds the import agent will allow before timing out the connection over the network. You can adjust this to allow for varying network traffic and quality.
Charset	Blank for new	Specifies the character set of the input SOIF stream. For example, ISO8859-1, UTF-8, UTF-16. Character sets ISO8859-1 through ISO8859-15 are supported.

Edit

The edit function allows you to edit the Resource Descriptions in the database. For example, you can correct a typographical error in an RD or manually assign RDs discovered by the robot to categories.

Table F-15 Database Edit Attributes
Attribute	Default Value	Description
Search For	All RDs	All RDs, Uncategorized RDs, Categorized RDs, RDs by category, Specific RD by URL, RDs that contain
Text box	Blank	Enter a unique text string to identify the RDs searched for. Use with the RDs by category, Specific RD by URL, and RDs that contain attribute values.

A successful search displays the Number of RDs found and a text box listing the RDs found. After selecting one, the following attributes, which you can edit, and partial text of the RD are displayed. All these attributes except Classification are set to editable in the Database/ Schema page.

Table F-16 Database RD Editable Attributes
Attribute	Default Value	Description
Classification	Category name of selected RD.	Category name if classified; No Classification if not classified.
ReadACL	Blank	Related to document level security.
Description	Description from the selected RD.	Description from RD.
Keywords	Keywords, if any, from the selected RD.	Keywords taken from meta- tags.
Title	Title of the selected RD.	Title of RD

Schema

The schema determines what information is in a resource description and what form that information is in. You can add new attributes or fields to an RD and set which ones can be edited and which ones can be indexed. When importing new RDs, you can convert schemas embedded in new RDs into your own schema.

Table F-17 Database Schema Attributes
Attribute	Default Value	Description
Schema	Author	List of non-internal schemas: Author, Author-EMail, Content-Charset, Content-Encoding, Content-Language, Content-Length, Content-Type, Description, Expires, Full-Text, Keywords, Last-modified, Partial-Text, Phone, ReadACL, Title, URL
Name	Author	Names the Schema chosen: Author, Author-EMail, Content-Charset, Content-Encoding, Content-Language, Content-Length, Content-Type, Description, Expires, Full-Text, Keywords, Last-modified, Partial-Text, Phone,ReadACL, Title, URL
Description	Author(s) of the document	These descriptions match with the schemas listed above. This text box is for your comments. It is not used by the Search server. Author(s) of the document. Email address to contact the Author(s) of the document. Content-Charset information from HTTP Server. Content-Encoding information from HTTP Server. Content-Language information from HTTP Server. Content-Length information from HTTP Server. Content-Type information from HTTP Server. Brief one-line description for document. Date on which resource description is no longer valid. Entire contents of the document. Keywords that best describe the document. Date when the document was last modified. Partial selection of text from the document. Phone number for Author contact. Used by Search servers to enforce security. Title of the document, Uniform Resource Locator or Web address for the document.
Aliases	Blank	When you import new RDs, you can convert schemas embedded in new RDs into your own schema. You would use this conversion when there are discrepancies between the names used for fields in the import database schema and the schema used for RDs in your database. An example would be if you imported RDs that used Writer as a field for the author and you used Author in your RDs as the field for the author. The conversion would be Writer to Author, so you would enter Writer in this text box.
Editable	False (unchecked)	If true (checked), the selected attribute (field) appears in the Database RD Editor, so you can change its values. Description, Keywords, Title and ReadACL are editable.
Indexable	True (checked)	If true (checked), the selected attribute (field) can be used as a basis for indexing. Author, Title and URL appear in the menu in the Advanced Search screen for the end user. This allows end users to search for values in those particular fields. Author, Expires, Keywords, Last Modified, Title, URL and ReadACL can be used as the basis for indexing.

Reindex

You should reindex the database if you have edited the schema to add or remove an indexed field (as author), if a disk error has corrupted the index or after adding a large number of new resource descriptions.

Because the time required to reindex the database is proportional to the number of RDs in the database, a large database should be reindexed when the server is not in high demand.

The robot must not be running.

Table F-18 Database Reindex Attribute
Attribute	Default Value	Description
Reindex the database?	False (unchecked)	True (checked) reindexes the database.

Purge

This page allows you to purge the contents of the database. Disk space used for indexes will be recovered, but disk space used by the main database will not be recovered; instead, it is reused as new data is added to the database.

Table F-19 Database Purge Attribute
Attribute	Default Value	Description
Purge the database?	False (unchecked)	True (checked) removes all of the data in the database.

Expire

This option deletes all RDs that are deemed out-of-date. It does not decrease the size of the database. By default, an RD is scheduled to expire in 90 days from the time of creation.

Table F-20 Database Expire Attributes
Attribute	Default Value	Description
Expire RDs	False (unchecked)	True (checked) removes from the database only the RDs that have expired.

Analysis

Select Save to show a sorted list of all sites and the number of resources from that site currently in the search database.

Table F-21 Database Analysis Attributes
Attribute	Default Value	Description
Total number of RDs	Current number of RDs in database.	Lists current total number of resource descriptions in the database.
Number of servers	Current number of servers that the database is partitioned across.	The database can be partitioned and placed on a number of servers.
Site	URL or domain that the robot has successfully searched.	A URL or domain that has added resource descriptions to the database.
Number of RDs	Current number of RDs from that site.	Lists current number of RDs from that site.
Type	Type of RD	Resource descriptions can be of many different types, for example, http.
Percentage	Type of RD/ Total number of RDs	Percentage of this type of document compared to the total number of resource descriptions.

Schedule

This page is where you set up the cron job to schedule the running of the import agents.

Table F-22 Database Import Schedule Attributes
Attribute	Default Value	Description
Start Import Time in hours and minutes	00:00	Time that the import agent starts to import.
Days	none selected	Sun -Sat Check at least one day.
Cron Status	Lists what its current status is.	Off On

Table F-23 Categories Editor Attributes
Attribute	Default Value	Description
Name	Name of new or selected category	Note Search (name of your search database) can only have a Child category.
Description	Blank	Enter a description.

Table F-24 Categories Classification Rules Editor Attributes
Attribute	Default Value	Description
Rules	Blank	List rules that have already been defined.
Drop down 1	Author	Author, Author-EMail, Content-Charset, Content-Encoding, Content-Language, Content-Length, Content-Type, Description, Expires, Full-Text, Keywords, Last-modified, Partial-Text, Phone, ReadACL, Title, URL, host, protocol, uri, ip, path, type.
Drop down 2	is	is, contains, begins with, ends with, regular expression
Textbox 1	Blank	Enter text to support Drop down selections.
then Classify as	List of available categories.	Select one of the categories to complete the rule.
Rules	Blank	List rules that have already been defined.

Table F-25 Categories Reindex Attribute
Attribute	Default Value	Description
Index the Category Tree?	False (unchecked)	True (checked) indexes the database based on the categories you have defined.

Reports

The Reports section allows you to monitor your search server. You can see a summary of its activity: what sites were searched, what URLs were excluded and why, detailed information about URLs visited by the robot, and what your end users are interested in.

The Reports topic is divided into the following subtopics:

Starting Points

Excluded URLs

Robot Advanced Reports

View Log Files

Starting Points

The robot will visit all the enabled sites every time it starts.

Table F-26 Reports Starting Points Attributes
Attribute	Default Value	Description
Enabled	Current value of site.	Yes or No. This is set on the Robot/ Sites page.
Starting Point	Chosen URL:80	Link brings up chosen URL.
in site definition	Chosen URL	Links to Robot/ Sites edit page.
Depth	Lists selected level of search.	1-n Set on the Robot/ Sites edit page.

Excluded URLs

This page shows a list of robot runs. To display a list of reasons URLs were excluded, select a robot run to examine, select View Selected, then select one of the Reasons for Exclusion. Displayed is a list of the excluded URLs for that reason. Duplicate and warning exclusions have been removed.

Table F-27 Reports Excluded URLs Attributes
Attribute	Default Value	Description
Log	Lists log from most recent run.	Lists all run logs available.
Count	Numbers	List of numbers with reasons for exclusion.
Reason for Exclusion	List of reasons sites have not been allowed. Each reason is linked to a list of all the URLs that were excluded for that reason.	Filter rules, file not found, site not allowed, protocol not allowed, errors, duplication are some of the reasons URLs were excluded.

Robot Advanced Reports

This page gives you access to a number of different reports from the robot. Select from a drop down list to get information for chosen report to show up. The Refresh button gets the current information.

Table F-28 Reports Robot Advance Reports Attribute
Attribute	Default Value	Description
Advanced Robot Reports	Version	Version, DNS Cache Dump, Performance, Servers Found-All, Server Found-RDM, Status-Current Configuration, Status -Database (internal), Status-Libnet, Status -Modules, Status-Overview, URLs-ready for extraction, URLs-ready for indexing, URLs- waiting for filtering (URL pool), URLs- waiting for indexing, all reports.

View Log Files

View the entries or specific lines from a log file. Drop down list of log files. Enter the number of lines you want to be displayed when you select View button.

Table F-29 Reports View Log Files Attributes
Attribute	Default Value	Description
View this logfile	Excluded URLs (filter)	Excluded URLs (filter), RD Manager (rdmgr), RDM Server (rdmsvr), Robot Activities (robot), Search Engine (searchengine), User Queries (rdm).
Number of lines	25	A number you can enter to display the most current entries in the log file.

Appendix F
Search Attributes

Server

Settings

Advanced

Robot

Control

Sites

Crawling

Indexing

Simulator

Site Probe

Schedule

Database

Import

Edit

Schema

Reindex

Purge

Expire

Analysis

Schedule

Categories

Category Editor

Classification Rules Editor

Reindex

Reports

Starting Points

Excluded URLs

Robot Advanced Reports

View Log Files

Popular Searches

Appendix F Search Attributes

Server

Settings

Advanced

Robot

Control

Sites

Crawling

Indexing

Simulator

Site Probe

Schedule

Database

Import

Edit

Schema

Reindex

Purge

Expire

Analysis

Schedule

Categories

Category Editor

Classification Rules Editor

Reindex

Reports

Starting Points

Excluded URLs

Robot Advanced Reports

View Log Files

Popular Searches

Appendix F
Search Attributes