This chapter describes the elements that make up a Netscape Compass Server system, how they work together, and how you as system administrator can determine which features you want to use and deploy.

Specifically, it covers the following topics:

• What Is a Compass Server?
• What Is a Compass Server For?
• Choosing Compass Server Features
• About Resource Descriptions
• Deploying a Compass Server System

What Is a Compass Server?

• The Compass Server itself--This is the part the users actually interact with. Users can either type in search queries using keywords or browse through a set of categories, and the Compass Server responds with a list of items that correspond to the specified keywords or category.

  

From the user perspective, the Compass Server is a stand-alone index service, much like commercial web indexes they are probably familiar with, but customized for your work environment both in content and organization.
• The Compass database--The heart of the Compass Server is a database of information about resources available on the network, such as documents, text files, or spreadsheets. The administrator uses a program called a robot to find all the available resources on the network and generate information about them for the database. The robot works behind the scenes, invisible to end users.

  

The administrator can also submit descriptions of resources manually. Most of the work of the Compass Server administrator involves setting up and customizing the robot and its settings.
• The My Compass subscription system--Users who want to be notified when new material of interest comes into the Compass Server database can create subscriptions that describe what topics they want to hear about, using the Interest Profile Editor. The system can then notify them by email or by updating a web page.

  

These My Compass newsletters provide both a customized notification of updates and a way for users to rate new materials so that other users have an idea of their relevance.

What Is a Compass Server For?

One way to bring some order to this chaotic jumble of networks, servers, documents, and resources is indexing. An index is a centralized, searchable database that brings together enough information about a set of resources so that users can pinpoint and retrieve the resources they want.

Netscape Compass Server therefore addresses one of the most common issues network users face: finding information that might exist anywhere in a distributed environment. For example, a user might want to locate a specific report on a network that has dozens or even hundreds of servers.

Network indexes have two general purposes:

1. Discover and gather resource information from diverse sources in the network into a single, searchable location.

2. Provide a means of locating and retrieving the indexed resources.

Browsing for Information

Unless the user knows where to look, browsing is generally the least effective method of finding information. Users should not have to keep a mental map of the network to be able to find resources.

Generating Navigational Aids

Netscape Compass Server uses a program called a robot to visit specified sites on the network, locating all the resources at those sites and generating descriptions of them. These descriptions go into the Compass database, where users can search for the resources in several ways.

Using the Index

The user can

• Search by keywords
• Browse by categories
• Combine searching and browsing
• Request notification of future updates of interest

What Can You Index?

The general rule is: : If you can pinpoint it with a URL, you can index it. By default, the robot installed with the Compass Server handles the largest subset of the resources you can index: those that can appear in a web browser such as Netscape Navigator. Those common resources include

• Web pages (HTML documents)
• Plain text files
• Word-processing documents
• Other documents that can be converted into HTML
• FTP directories

You can also add other kinds of resources to your database, either manually or by customizing your robot to handle them for you.

Choosing Compass Server Features

Specifically, this section describes the following topics:

• What Users Expect from an Index
• Enabling Compass Server Features

What Users Expect from an Index

People use indexes to locate particular items based on their needs. Take several different examples:

• To locate a book in a library, you can search a card catalog, with books listed alphabetically by title, author, or subject. Of course, computerized indexes have made most card catalogs obsolete, but they operate on the same principle.
• To locate the proper part to repair a car, you can look in a parts catalog based on the make, model, and year of the car. These indexes, too, are frequently online, allowing for easier, more direct access to information.
• To order a shirt from a mail-order catalog, you can browse until you find a section with clothing, then locate shirts, then choose the one you prefer. Or perhaps you could look at an index or table of contents for shirts and go directly to that section.

User Features in Compass Server

All of the features described in this section are explained in much more detail in the Netscape Compass Server User's Guide. Users of the Compass Server can access the appropriate pages in the User's Guide by pressing the Help button on each form. For your convenience, the User's Guide is also available in the Manuals section of the Server Manager for the Compass Server.

Keyword Searches: The main Compass Server screen shown in the figure below provides a box in which users can type one or more keywords. When the user clicks the Search button, the Compass Server returns a list of items matching those keywords, with the best matches first. The user can then choose items from the list, which includes hypertext links to each item. The results list also provides links to similarly categorized items, if any.

Advanced Searches: In addition to the basic keyword search, Compass Server also provides for the construction of complex queries. In addition to keywords, users can specify date ranges and other information about the items to search for. They can also combine search criteria with logical ANDs. This type of search is especially useful for locating specific items, rather than just a general type of item. The figure below shows the Advanced Search tool and highlights some of its main features.

Category Browsing: Another useful way to locate items in an index is to navigate through a hierarchy of categories, narrowing down the included items until you find the specific item or items desired. Where a retail catalog might categorize its items by product types, you can customize your network index categories to best meet your users' needs.

Depending on the type of items on your network, you might set up categories that follow the organizational structure of a company, product lines, geographical regions, and so on. In fact, you could set up categories for all those sorts of items, because items can fit in more than one category. Thus, a document might fit into several categories, such as Sales Department, Widgets, and North America, so a user could find it in any of several ways.

Categories and their subcategories appear on the main search screen, so users can navigate to more specific or more general categories. There is also a Category Browser, shown in the figure below, that shows the full hierarchy of categories in an expandable outline.

Subscribing to Information

These My Compass notifications are based on subscriptions. Each user can specify a set of criteria that indicate what sorts of information the personal report should contain. My Compass then either sends electronic mail or updates a personal web page when new information arrives.

In addition to receiving information about new and changed items, users can provide feedback on the usefulness of the items reported.

The following figure shows a sample of a personal news page.

Advantages of the Compass Server

• Multiple views of the same resources
• Easier reorganization of resources
• A variety of indexing methods that don't involve changing the data
• Customizable lookups

What Administrators Need in an Index

Administrator Features in Compass Server

Access Control and Security: As with all Netscape server products, the Compass Server provides full access control and security. For description of these features, see the manual Managing Netscape Servers.

Scheduling: Many administrators need to be able to schedule administrative and maintenance tasks to run at regular intervals, often when the system is physically unattended.

Remote Administration: It is often useful to be able to perform routine tasks such as restarting the server from a remote location through the network. This enables the administrator to perform tasks without physical access to the server, either because the server is at another site or because it is in a computer room.

Indexing control: For an online index, the administrator needs to be able to control what materials go into the index (and of course, which are excluded). The administrator also needs to be able to set up categories, assign documents to categories, rearrange the data, and maintain the database.

Load balancing: There are several aspects of network and computer load balancing that affect all servers, and some specific to indexes. These include partitioning databases across physical devices, distributing workloads across different computers, and scheduling tasks based on network traffic and resource availability.

Enabling Compass Server Features

Of course, you have to allow enough time for the robot to find, describe, and register all the items on the network. Depending on the size of the network, that can take anywhere from a few minutes to several days. However, the database is usable all the time, even while being built initially or updated periodically.

Indexing Documents

The Compass Server robot is a process that runs continuously. Using a list of sites as starting points, the robot follows hypertext links to locate as many resources on the site and linked sites as possible (within parameters you set). For each resource it locates, it generates a resource description, which it then stores in its database.

The Compass Server robot is described in much more detail in Filling the Database.

Once the robot places resource descriptions in the database, users can search for resources based on keywords or more complex criteria.

Defining Categories

Most organizations will probably need to define their own categories, depending on the structure and needs of the organization. Compass Server comes with a number of sample category sets, however, that can serve as starting points for custom sets.

Creating and maintaining a set of categories is the subject of Setting Up Categories.

Maintaining the System

Refreshing Content: Although the Compass robot is a continuous process that can always search for resources and generate descriptions of them for the database, you need to determine how often the robot should revisit all its sites to look for new items and update descriptions of changed ones.

How often this happens depends on how long the entire process takes, how often resources actually change, and how much load the Compass Server and network can handle. Filling the Database discusses these issues in greater detail.

Generating My Compass Newsletters: You should set up the My Compass profiler to generate timely newsletters for all users who subscribe to the service.

There are at least two ways to handle this:

1. Schedule the My Compass profiler to run at a specified day and time.

2. Run a script at the completion of each robot run to run the profiler for the newly updated items.

System Maintenance: There are a few periodic tasks you can schedule to clean up unneeded files and improve system performance.

• Optimizing the Database
• Reindexing the Database
• Purging Expired Resource Descriptions
• Archiving Log Files

About Resource Descriptions

The fields of each resource description are determined by the database schema. The database itself and the database schema are discussed in greater detail in Managing the Compass Database.

The primary concern at the planning stage, however, involves how to get resource descriptions into the database. There are basically two approaches you can use to getting resource descriptions:

• Creating Resource Descriptions--This is by far the most common method, using a robot process to locate resources and generate their descriptions.
• Exchanging Resource Descriptions--This method is appropriate for large, distributed network indexes. A remote system generates resource descriptions, and the Compass Server imports those into its database.

NOTE: A special case of exchanging is migrating, which happens when you upgrade from Netscape Compass Server 1.0 to Netscape Compass Server. Migration is explained in Migrating from Catalog Server 1.0.

Examples

Scenario: Finding a book in the library: In many ways, a computer network is like a library: information is stored in many separate physical locations (books) that can be inconvenient to locate and browse through. A book's contents might not be immediately obvious from the title.

The classic solution to locating particular books is to provide a card catalog (although most have been replaced by online indexes). The books in the library are the resources, and the individual cards are resource descriptions that contain summary information such as title, author, date, content summary, and shelving information. These resource description cards are generally organized alphabetically, but they are often dually indexed, by author and by title/subject.

The user can use the catalog to pinpoint which of the indexed resources are most likely to be useful.

Scenario: Ordering from a mail-order catalog: Suppose you want to order new seat covers for your car from a mail-order company. To order the correct seat covers, you need to be able to specify the correct product number.

Mail-order companies generally provide catalogs of the products they sell. The products in this case are resources, and the resource descriptions are the entries in the catalog booklet, which might be as simple as single-line entries in a list or pictures with descriptions, but each contains a description of the resource: name, price, stock number, and so on. The products are generally grouped with similar products, often with an index for easy location.

By first locating the appropriate part of the catalog, then narrowing down choices until you find the seat covers you want, you can locate part numbers and other information needed to place your order.

Scenario: Finding out what's new: Suppose you need to keep current on industry trends or the latest information on competing companies. Many people subscribe to news clipping services or specialized newsletters to keep them informed of the latest developments.

If your Compass Server is kept up-to-date, users will come to rely on it for such information, too. However, remembering to search for the same kinds of information every day is tedious and unreliable.

By creating customized subscriptions through the My Catalog Subscription Editor, users can have the system notify them of new or updated items in categories they choose, keyword searches they define, or documents others rate as important.

The Common Thread

• There is a large number of resources to be represented.
• There is a common format for information describing each resource.
• A central repository holds all the resource descriptions.
• The resource descriptions are arranged, ordered, or grouped for easy access.
• The index provides direction on how to retrieve the desired resource.

Creating Resource Descriptions

The most important part of that process is refining the robot's instructions to ensure that it finds all the resources you want included in the database without including extraneous resources or overburdening the network.

Most of the material in this chapter deals with configuring and controlling the Compass Server robot.

Exchanging Resource Descriptions

Much as a user requests certain kinds of information from the Compass Server, a Compass Server can make requests of other Compass Servers. This allows what is called distributed indexing, spreading the load for discovering and describing resources among different robots. A central Compass Server can gather resource descriptions from a number of different systems by importing the resource descriptions they generate. Importing and distributed indexing are explained in Managing the Compass Database.

The dialog between Compass Servers uses resource description messages (RDMs). RDM is an open protocol built on top of the standard HTTP protocol designed expressly for exchanging resource descriptions.

Filtering of resource descriptions in response to an RDM request is called scoping and is defined as part of the request. That is, a Compass Server need not import all the resource descriptions from another server's database. It can request only those that match certain criteria, just as a user does.

Deploying a Compass Server System

Choosing User Features

This section describes the following choices of default user interface features:

• Searching and Browsing
• Deploying My Compass
• Providing Information About the Server

Searching and Browsing

One thing to remember is that the default configuration of the Netscape Compass Server provides for a reasonable, useful database for searching. Implementing browsing takes more planning. However, browsing is not a required element of a system. You can disable the browsing tools and use only the search capabilities of the Compass Server. It's a question of what your users need and want, and the usefulness of your browsing categories.

Factors Affecting Searching

Generation of keywords for documents is provided by the Compass robot. By default, the robot generates keywords for each document based on words appearing in titles and headers. You can change the default keyword generation by customizing your robots through the robot application program interface (API), as described in the Netscape Compass Server Programmer's Guide.

Document authors can also provide specific keywords by embedding them with an editor such as Netscape Composer. Guidelines for authors appear in the Netscape Compass Server User's Guide.

In addition to keywords, the Compass Server can search for documents based on metainformation, such as author's name, creation date, or other attributes chosen by the administrator. The structure and content of the metainformation stored in the database is determined by the database schema. Managing the Compass Database describes how to adjust the schema for your particular needs.

Factors Affecting Browsing

For example, suppose your Compass database covers a web site that contains only three kinds of documents: home pages, product announcements, and sales reports. You could create three categories within the global category of documents. Users looking only for sales reports could choose that category and then have the server display a list of only those documents.

Well-defined categories can make resources accessible in ways that ordinary searches might miss. For example, you could set up a category called "Competitors," containing information about companies that yours competes against. By directing your robot to categorize all items pertaining to competing companies in that category, you can ensure that users of your index don't have to search for each and every competitor individually.

The details of creating and maintaining a hierarchy of resource categories are explained in Setting Up Categories.

Deploying My Compass

These issues are discussed in greater detail in Administering My Compass.

The My Compass user and group definitions can either come from a local database or from an LDAP-based directory service, such as Netscape Directory Server. Keep in mind that once you choose your user database, you won't be able to easily change it later. That is, if you start out using a local database, you cannot easily change the configuration later to get the same information from a directory service.

For more about user and group management, see Managing Netscape Servers and the README file and release notes for Netscape Compass Server.

Scheduling the profiler is mostly a matter of balancing how often your users want updated newsletters with how often you update the information in the Compass Server database by running its robot. As you schedule the various parts of the system that can be automated, you will determine when best to include the My Compass profiler.

Providing Information About the Server

For more on the About page, see Customizing the User Interface.

Choosing Sites to Index

The simplest case is a database that covers the resources of a single web server. In that case, the Compass Server need only direct its robot to the root of the web site. The robot will then traverse the links it finds on the home page.

On the other hand, a Compass database could cover multiple web sites, portions of one or more web sites, and so on. The administrator controls the scope of the robot by choosing the starting points and refining its site definition.

It's also possible to have separate databases for, say, several groups of servers, and other databases that combine the contents of those. This is done by importing resource descriptions into the Compass database.

For example, a large company might have a Compass database for each of its departments and a master Compass system for the entire company that covers all the departments by importing resource descriptions from each of the departmental Compass Servers.

There are nearly as many different Compass Server configurations as there are network sites, and the considerations will be different for each. However, you can group the types of considerations into several categories:

• Usability--Your first consideration needs to be how easily users will be able to find information they need. This will affect issues such as scheduling the availability of the server, scheduling system maintenance, choosing sites to index, and designing a set of categories for browsing. All these topics are discussed elsewhere in this manual.
• Performance--Some of your configuration choices will affect the overall performance of the system. For example, when you design a set of categories, the number of documents that fit into each category will affect the performance of the system as users browse. Categories that contain many documents will tend to slow down browsing, so if possible, you'll want to divide such categories into subcategories.
• Frequency of changes--If the sites you index don't change often, you might be able to reduce the frequency of your robot runs. Similarly, if you index sites that change frequently, you will need to send the robot to find new or changed information often. It's quite possible that you'll encounter both kinds of sites, and depending on issues such as network load, you can either compromise on the robot timing or split the duty among several robots.
• Administration and configuration--Simply stated, the more complex and distributed your Compass Server system, the more work it takes to configure and administer it. The more systems involved, the more maintenance tasks, and so on.

A Compass Server that indexes remote sites over the global Internet, of course, does not have much control over what it finds, but you can still customize the way your system indexes those remote sites.

Controlling Robot Access

The format for robots.txt is straightforward: it is a plain ANSI text file. The file consists of one or more groups of lines with name-value pairs that instruct the robots.

Each group of lines should describe the User-agent type, which is a name that robots call themselves. The Netscape Compass Server robot is called Netscape-Compass-Robot/3.0. After you specify which User-agents you want to configure, you include a Disallow line that lists the directories you want to restrict. You can include one or more groups in your robots.txt file.

Each line in the group has the format

field:value

# This is a sample robots.txt file
User-agent: * 
Disallow: /usr 

# robots.txt for http://www.mysite.com/
User-agent: *
Disallow: /usr
Disallow: /tmp 

# robots.txt for http://www.site.com/
User-agent: *
Disallow: *

# Netscape robot is a good robot
User-agent: Netscape-Compass-Robot/3.0
Disallow: 

# No robots allowed!
User-agent: *
Disallow: / 

This allows users who do not have access to the site's robots.txt file to define how their documents should be indexed. Note that, unlike robots.txt, users cannot specify permissions for any particular robot, just for robots in general.

In HTML documents, you use META tags to create this metainformation. In other kinds of documents, you can use whatever document information will be converted into the appropriate HTML META tags.

Using the ROBOTS Tag

The general format for the ROBOTS tag is as follows:

<META NAME="ROBOTS" CONTENT="terms">

Using Other META Tags

For example, consider a document that includes the following META tag:

<META NAME="Importance" CONTENT="Trivial">

Last Updated: 02/12/98 13:30:51