iPlanet Compass Server provides a searchable database of documents and other network resources that enables users to find information in a distributed network and also lets them define topics they want to stay up-to-date on.

This chapter describes the elements that make up a iPlanet Compass Server system, how they work together, and how you as system administrator can determine which features you want to use and deploy.

Specifically, it covers the following topics:

These topics introduce you to the concepts important to a Compass Server system. The remainder of the manual explains the tasks involved in deploying, administering, and maintaining such a system.

A Compass Server system is a group of software programs that enable users to locate resources on a network, much as they might consult a card catalog in a library or a catalog of merchandise in a store. The Compass system consists of three main parts:

The rest of this chapter looks at these parts of the Compass Server system in more detail, including how they interrelate, so that you can decide how you want to deploy them in your system. The rest of this manual explains the specific tasks involved in designing, maintaining, and administering the system in depth.

Networks contain vast amounts of information, spread out among multiple servers, stored in many formats, and maintained by different individuals. Even a carefully designed network can be confusing to those looking for specific information, and when the domain of the search extends to a company-wide collection of networks (an "intranet") or to the global Internet, where standards and coordination are nearly impossible, the search for the proper information can be nearly impossible.

iPlanet Compass Server therefore addresses one of the most common issues network users face: finding information that might exist anywhere in a distributed environment. For example, a user might want to locate a specific report on a network that has dozens or even hundreds of servers.

Network indexes have two general purposes:

In addition to these standard features of a network index, iPlanet Compass Server can also alert network users to new resources of interest to them.

In most cases, a user looking for information has no alternative except to browse (or "surf") through known locations and hyperlinks, hoping to find the desired resource.

Unless the user knows where to look, browsing is generally the least effective method of finding information. Users should not have to keep a mental map of the network to be able to find resources.

The solution is to create an index, which is a database at a specific location that contains information about network resources in a form that users can search in a number of ways.

Instead of having to know about and visit a number of sites, the user now visits only a single site, using that site's tools to pinpoint and retrieve the desired resources.

The user can

The index reduces the number of locations the user visits to find a particular resource, thereby reducing searching time and network traffic. In addition, it gives network and server administrators greater control over what resources are available. Most importantly, it greatly increases the chances of actually locating the desired resource.

If you can pinpoint it with a URL, you can index it.

By default, the robot installed with the Compass Server handles the largest subset of the resources you can index: those that can appear in a web browser such as Netscape Navigator. Those common resources include

You can also add other kinds of resources to your database, either manually or by customizing your robot to handle them for you.

This section describes the available features of your iPlanet Compass Server, primarily from the viewpoint of the users of the server. Depending on what your users will want to do with the server, you can choose different features or different configurations of hardware and software in the system.

Specifically, this section describes the following topics:

Once you determine which features you want to implement, the next section deals with how to deploy the system.

The term index generally denotes a list of items, usually with pictures or summary information for each item. The index often groups related items together by categories. Physically, catalogs can range from a typed list of items to one or more printed volumes to extensive online databases.

People use indexes to locate particular items based on their needs. Take several different examples:

These examples demonstrate the three qualities that users tend to expect from an index: searching by general keywords, searching by specific criteria, and browsing by categories.

All of the features described in this section are explained in much more detail in the iPlanet Compass Server User's Guide. Users of the Compass Server can access the appropriate pages in the User's Guide by pressing the Help button on each form. For your convenience, the User's Guide is also available in the Manuals section of the Server Manager for the Compass Server.

The main Compass Server screen shown in the figure below provides a box in which users can type one or more keywords. When the user clicks the Search button, the Compass Server returns a list of items matching those keywords, with the best matches first. The user can then choose items from the list, which includes hypertext links to each item. The results list also provides links to similarly categorized items, if any.

In addition to the basic keyword search, Compass Server also provides for the construction of complex queries. In addition to keywords, users can specify date ranges and other information about the items to search for. They can also combine search criteria with logical ANDs. This type of search is especially useful for locating specific items, rather than just a general type of item. The figure below shows the Advanced Search tool and highlights some of its main features.

Another useful way to locate items in an index is to navigate through a hierarchy of categories, narrowing down the included items until you find the specific item or items desired. Where a retail catalog might categorize its items by product types, you can customize your network index categories to best meet your users' needs.

Depending on the type of items on your network, you might set up categories that follow the organizational structure of a company, product lines, geographical regions, and so on. In fact, you could set up categories for all those sorts of items, because items can fit in more than one category. Thus, a document might fit into several categories, such as Sales Department, Widgets, and North America, so a user could find it in any of several ways.

Categories and their subcategories appear on the main search screen, so users can navigate to more specific or more general categories. There is also a Category Browser, shown in the figure below, that shows the full hierarchy of categories in an expandable outline.

A computerized, networked resource database has several advantages over more traditional methods of indexing. Among these are

As you plan and implement your Compass system, keep in mind the tasks your users will want to perform, and make sure you facilitate those tasks.

Just as users have a set of expectations when using an online index, the administrator of that system brings certain expectations and requirements for setting up and maintaining the system.

With any server system, the administrator will have a number of requirements for features. This section describes both the more general server-administration features of iPlanet Compass Server and those features particular to this kind of distributed indexing system.

As with all iPlanet server products, the Compass Server provides full access control and security. For description of these features, see the manual Managing iPlanet Servers.

Many administrators need to be able to schedule administrative and maintenance tasks to run at regular intervals, often when the system is physically unattended.

It is often useful to be able to perform routine tasks such as restarting the server from a remote location through the network. This enables the administrator to perform tasks without physical access to the server, either because the server is at another site or because it is in a computer room.

For an online index, the administrator needs to be able to control what materials go into the index (and of course, which are excluded). The administrator also needs to be able to set up categories, assign documents to categories, rearrange the data, and maintain the database.

There are several aspects of network and computer load balancing that affect all servers, and some specific to indexes. These include partitioning databases across physical devices, distributing workloads across different computers, and scheduling tasks based on network traffic and resource availability.

Most of the features of iPlanet Compass Server are active by default on installation. That is, you designate which site or sites on a network you want to index, and the Compass robot locates the resources at those sites, generates descriptions of them, and places them in the database, where they become immediately searchable.

Of course, you have to allow enough time for the robot to find, describe, and register all the items on the network. Depending on the size of the network, that can take anywhere from a few minutes to several days. However, the database is usable all the time, even while being built initially or updated periodically.

The Compass Server robot is a process that runs continuously. Using a list of sites as starting points, the robot follows hypertext links to locate as many resources on the site and linked sites as possible (within parameters you set). For each resource it locates, it generates a resource description, which it then stores in its database.

Once the robot places resource descriptions in the database, users can search for resources based on keywords or more complex criteria.

In order for users to locate items in the database by browsing categories, the administrator must first set up the categories.

Most organizations will probably need to define their own categories, depending on the structure and needs of the organization. Compass Server comes with a number of sample category sets, however, that can serve as starting points for custom sets.

Once a Compass Server system has content, there are a few maintenance tasks needed to keep the database up-to-date and running smoothly. Most of these tasks can run on an automated schedule.

Although the Compass robot is a continuous process that can always search for resources and generate descriptions of them for the database, you need to determine how often the robot should revist all its sites to look for new items and update descriptions of changed ones.

There are a few periodic tasks you can schedule to clean up unneeded files and improve system performance.

You also need to monitor and maintain the operating environment in general, to ensure that the Compass Server has enough disk space, memory, swap space, CPU cycles, and so on.

The primary concern at the planning stage, however, involves how to get resource descriptions into the database. There are basically two approaches you can use to getting resource descriptions:

One excellent way to visualize a resource index is to refer to some common, physical situations that are similar in function.

In many ways, a computer network is like a library: information is stored in many separate physical locations (books) that can be inconvenient to locate and browse through. A book's contents might not be immediately obvious from the title.

The classic solution to locating particular books is to provide a card catalog (although most have been replaced by online indexes). The books in the library are the resources, and the individual cards are resource descriptions that contain summary information such as title, author, date, content summary, and shelving information. These resource description cards are generally organized alphabetically, but they are often dually indexed, by author and by title/subject.

The user can use the catalog to pinpoint which of the indexed resources are most likely to be useful.

Suppose you want to order new seat covers for your car from a mail-order company. To order the correct seat covers, you need to be able to specify the correct product number.

Mail-order companies generally provide catalogs of the products they sell. The products in this case are resources, and the resource descriptions are the entries in the catalog booklet, which might be as simple as single-line entries in a list or pictures with descriptions, but each contains a description of the resource: name, price, stock number, and so on. The products are generally grouped with similar products, often with an index for easy location.

By first locating the appropriate part of the catalog, then narrowing down choices until you find the seat covers you want, you can locate part numbers and other information needed to place your order.

In each of the preceding examples, the abstract situation is the same:

Similarly, an index on a network involves some resources, a standard way of describing those resources, a database of resource descriptions, a way for users to find information on certain resources, and a way to access the desired resources once located.

A Compass Server most often relies on a robot to enumerate the resources it is responsible for. As the administrator, part of your job is to configure the robot to locate resources and have it generate resource descriptions for them.

The most important part of that process is refining the robot's instructions to ensure that it finds all the resources you want included in the database without including extraneous resources or overburdening the network.

Most of the material in this chapter deals with configuring and controlling the Compass Server robot.

A very efficient way to get resource descriptions for your database is to import them from another iPlanet Compass Server, Netscape Compass Server or a Netscape Enterprise Server. In cases where a number of systems all want to index the same material, it is much more efficient to generate the resource descriptions once and import them into other systems. Similarly, if network access is costly, you can minimize traffic by importing completed data, rather than making repeated robot inquiries.

Once you have determined what you want your Compass Server system to do, you can proceed with deploying the system. This section focuses on what you have to do to set up an effective Compass Server system for the features you need.

As administrator, you have complete control over the user interface presented to your users. All the screens are controlled by JavaScript templates. The standard distribution provides a number of default screens you can use, but you can also modify those or create your own. Using templates, you can control both the layout and presentation of search screens and result lists as well as add or remove selected features from your system.

This section describes the following choices of default user interface features:

There are two main ways for users to use the Compass Server: searching and browsing categories. As the administrator, you have a lot of control over the ease and usefulness of these features. The details of customizing the search and browse features appear elsewhere, but this section provides a brief overview of the issues you should consider when planning your Compass system so that you can maximize both its searchability and browsability.

One thing to remember is that the default configuration of the iPlanet Compass Server provides for a reasonable, useful database for searching. Implementing browsing takes more planning. However, browsing is not a required element of a system. You can disable the browsing tools and use only the search capabilities of the Compass Server. It's a question of what your users need and want, and the usefulness of your browsing categories.

The most common way for users to find resources in a Compass database is to type a query containing a few keywords that describe the desired documents. The Compass Server then provides a list of documents matching those keywords by searchig its database of indexed resource descriptions. The search allows for simple matches as well as more complex searches that combine terms with AND or OR and other operators.

Generation of keywords for documents is provided by the Compass robot. By default, the robot generates keywords for each document based on words appearing in titles and headers. You can change the default keyword generation by customizing your robots through the robot application program interface (API), as described in the iPlanet Compass Server Programmer's Guide.

Document authors can also provide specific keywords by embedding them with an editor such as Netscape Composer. Guidelines for authors appear in the iPlanet Compass Server User's Guide.

With the iPlanet Compass Server you can assign your indexed resources to categories. By designing your hierarchy of categories, you can make it very easy for users to find the types of information they want.

For example, suppose your Compass database covers a web site that contains only three kinds of documents: home pages, product announcements, and sales reports. You could create three categories within the global category of documents. Users looking only for sales reports could choose that category and then have the server display a list of only those documents.

Well-defined categories can make resources accessible in ways that ordinary searches might miss. For example, you could set up a category called "Competitors," containing information about companies that yours competes against. By directing your robot to categorize all items pertaining to competing companies in that category, you can ensure that users of your index don't have to search for each and every competitor individually.

One of the most confusing things for users of a search system is knowing just what the system includes. iPlanet Compass Server provides a standard way for you to provide that information through the About Compass button across the bottom of all the standard user screens.

The simplest case is a database that covers the resources of a single web server. In that case, the Compass Server need only direct its robot to the root of the web site. The robot will then traverse the links it finds on the home page.

For example, a large company might have a Compass database for each of its departments and a master Compass system for the entire company that covers all the departments by importing resource descriptions from each of the departmental Compass Servers.

There are nearly as many different Compass Server configurations as there are network sites, and the considerations will be different for each. However, you can group the types of considerations into several categories:

A Compass Server that indexes remote sites over the global Internet, of course, does not have much control over what it finds, but you can still customize the way your system indexes those remote sites.

The format for robots.txt is straightforward: it is a plain ANSI text file. The file consists of one or more groups of lines with name-value pairs that instruct the robots.

Each group of lines should describe the User-agent type, which is a name that robots call themselves. The iPlanet Compass Server robot is called iPlanet-Compass-Robot/3.0. After you specify which User-agents you want to configure, you include a Disallow line that lists the directories you want to restrict. You can include one or more groups in your robots.txt file.

Each line in the group has the format

field:value

The field name is case insensitive, but value is case-sensitive. You can include comment lines by beginning the comment with the # character. The following example shows one group that configures all robots and tells them not to go into the directory called /usr:

# This is a sample robots.txt file

User-agent: * 

Disallow: /usr 

The following example robots.txt file specifies that no robots should visit any URL starting with /usr or /tmp:

# robots.txt for http://www.mysite.com/

User-agent: *

Disallow: /usr

Disallow: /tmp 

The next example restricts all robots from your web site except the Compass Server robot:

# robots.txt for http://www.site.com/

User-agent: *

Disallow: *

# iPlanet robot is a good robot

User-agent: iPlanet-Compass-Robot/3.0

Disallow: 

The following example tells all robots, including the Compass Server robot, not to traverse your web site.

# No robots allowed!

User-agent: *

Disallow: / 

Another way to control the behavior of robots at the individual document level is to create metainformation in the document that indicates whether the robot should index that document and whether it should follow links found in the document.

This allows users who do not have access to the site's robots.txt file to define how their documents should be indexed. Note that, unlike robots.txt, users cannot specify permissions for any particular robot, just for robots in general.

In HTML documents, you use META tags to create this metainformation. In other kinds of documents, you can use whatever document information will be converted into the appropriate HTML META tags.

The META tag that controls robot behavior uses the name ROBOTS. It's content tells a visiting robot whether it should include the document itself in its index and whether to follow hyperlinks found in the document to index the linked documents.

The general format for the ROBOTS tag is as follows:

<META NAME="ROBOTS" CONTENT="terms">

The terms in the CONTENT portion can be any of the following, separated by commas:

Content String	Meaning
ALL	The robot is welcome to include this document in its index and to follow any links found in it. This is the default value. You can get the same result by leaving the CONTENT portion empty, by omitting the ROBOTS tag entirely, or by using the contents "INDEX, FOLLOW".
NONE	The robot should ignore the page. This is the equivalent of "NOINDEX, NOFOLLOW".
INDEX	The robot is welcome to include the document in its index for searching.
NOINDEX	The robot should not include the document in its index. The robot can still follow links, unless you also include the NOFOLLOW string.
FOLLOW	The robot is welcome to follow any hyperlinks in the document to locate other documents for its index.
NOFOLLOW	The robot should not follow any hyperlinks in the document to locate other documents. This enables you to index just the entry point of a complex document, for example, or to index the open access point to an otherwise restricted site.

In addition to the standard ROBOTS META tag, you can define other tags that have special meanings to your Compass Server robot. By default, the robot converts any user-defined META tags into corresponding fields in the document's resource description. You can then program the robot (as described in the iPlanet Compass Server Programmer's Guide) to ignore documents containing certain tags, restrict its following of links, and so on.

For example, consider a document that includes the following META tag:

<META NAME="Importance" CONTENT="Trivial">

The Compass Server robot will automatically create a document field called "Importance" and assign it the value "Trivial" for this document. If you program the robot to ignore documents with an Importance of "Trivial," you can filter information for your index based on this self-rating.