Integrating Search

     Previous  Next    Open TOC in new window    View as PDF - New Window  Get Adobe Reader - New Window
Content starts here

Staging Search Capabilities

When you move to a production environment, you must configure Autonomy to match the portal environment you use.

This involves editing the configuration files for the search components you are using, deploying the Autonomy Service Dashboard, and configuring Autonomy fetches to search for information according to parameters you set.

The tasks covered in this chapter assume a typical Autonomy configuration running in a WebLogic Portal cluster. Consult the Autonomy documentation if you want to create a more complex configuration, such as running HTTPFetch on a separate server.

Figure 4-1 provides an example of a typical production environment.

Figure 4-1 Example of WebLogic Portal Cluster Using Autonomy

Example of WebLogic Portal Cluster Using Autonomy

This chapter discusses the following topics:

 

Installing Autonomy on Your Target Server

You need to install the appropriate Autonomy engines for your server’s operating system.

When you installed WebLogic Portal, the Autonomy engine for the target operating system was included. If you need a version of Autonomy for a different operating system than the operating system on which you installed WebLogic Portal, you will need to download and install WebLogic Portal onto the operating system for which you need Autonomy. You can then retrieve the respective operating system files for Autonomy.

For example, if you downloaded and installed WebLogic Portal on a Windows server, the Windows version of Autonomy was included in the download. If you want to install Autonomy on a Linux server, you need to download and install the Linux version of WebLogic Portal in order to have the correct version of Autonomy for a Linux machine.

Note: Remember that due to licensing restrictions, you can only run one Autonomy server.

To install Autonomy:

  1. Create a directory on your target server for the Autonomy components.
  2. From your WebLogic Portal installation, navigate to the Autonomy distribution for your target operating system. For example, WebLogic_HOME/weblogic92/portal/thirdparty/autonomy-wlp92/operatingsystem
  3. Copy the entire directory from your installation directory to the target directory on your target server.

 

Updating the Autonomy License

This section explains how to update the default Autonomy license with a production license. The default license allows 10,000 documents to be indexed, while the production license allows 500,000 documents to be indexed.

  1. Stop all Autonomy Services. To do this, call the autonomy.cmd script with the stop parameter. See Starting the Autonomy Services.
  2. In the IDOLserver/DiSH directory, do the following:
    1. Delete the uid and license directories.
    2. Run the command: AutonomyDiSH.exe -revokelicense
  3. In the IDOLserver/IDOL directory, do the following:
    1. Delete the uid directory.
    2. In the content, agentstore, category, and community directories under IDOLserver/IDOL, delete the uid and license directories.
  4. Create a backup of the existing IDOLserver/DiSH/licensekey.dat file to prevent overwriting it.
  5. Copy the production license file (licensekey.dat) into IDOLserver/DiSH.
  6. Restart the Autonomy Services. See Starting the Autonomy Services.
  7. Verify that the new license is accepted by reviewing the license.log files in the above directories and verify that the license now allows 500,000 documents, rather than the evaluation license limit of 10,000 documents. To do this, search in IDOLserver\IDOL\logs\content_application.log for the string “This license allows 500000 documents to be indexed”.

 

Configuring Autonomy on Your Target Server

You need to modify your Autonomy configuration to match your production environment and the parameters of your cluster. This includes modifying the respective configurations of the search tools you are using to account for security concerns and their network location.

You also need to configure the types of information you want to include in your searches.

Table 4-1 lists the location of the configuration files you need to modify.

Table 4-1 Autonomy Components and Their Respective Configuration Files
Autonomy Component
Configuration File You Need to Modify
DiSH Server
//IDOLserver/DiSH/AutonomyDiSH.cfg
IDOL Server
//IDOLserver/IDOL/AutonomyIDOLServer.cfg
Agent Stores for the IDOL Server
//IDOLserver/IDOL/agentstore/agentstore.cfg
HTTP Fetch
//HTTPFetch/HTTPFetch.cfg
File System Fetch
//FileSystemFetch/FileSystemFetch.cfg

This section includes the following topics:

Configuring the Autonomy IDOL Server

The Intelligent Data Operating Layer (IDOL) server is responsible for indexing content as well as processing queries. For more information about the IDOL server, see the Autonomy IDOL Server documentation.

To configure the IDOL server for your production environment modify the AutonomyIDOLServer.cfg file. The file is located //autonomyhome/IDOLserver/IDOL/AutonomyIDOLServer.cfg.

To configure the Autonomy IDOL server:

  1. Open the AutonomyIDOLServer.cfg file in a text editor.
  2. In the [License] section, edit the LicenseServerACIPort to match the port on which DiSH is running, if you changed this port.
  3. In the [Service] section, edit the ServicePort if needed. This setting must match the corresponding setting in the autonomyDiSH.cfg.
  4. In the [Server] section,
    • Edit the client list settings (IndexClients, AdminClients) for security as required.
    • Edit the IndexPort and Port (ACI) settings as needed.
  5. In the [Paths] section, edit the Modules and TemplateDirectory to point to the location of these directories on the target system. These must be absolute paths.
  6. Locate and edit all other directory or file path settings and adjust to point to the new location (for example, the [NT_V4] Library). These must be absolute paths.
  7. In the [Database] section, create and remove Autonomy databases as required for your needs. Consult the Autonomy documentation for managing databases.
  8. When finished, save your changes.

Configuring the Autonomy DiSH

The Distributed Services Handler (DiSH) is used to manage Autonomy components. You can access DiSH functions through the Autonomy Service Dashboard or use Autonomy’s ACI interface. For more information about the Autonomy DiSH, see the Autonomy DiSH documentation.

To configure DiSH, you can use a text editor to modify the autonomyDiSH.cfg file. The autonomyDiSH.cfg file is located in the //autonomyhome/IDOLserver/IDOL/AutonomyIDOLDiSH.cfg directory.

To configure Autonomy DiSH:

  1. Open the autonomyDiSH.cfg file in a text editor.
  2. In the [Service] section, edit the ServicePort setting if needed to avoid port conflicts.
  3. In the [Server] section, edit the following:
    • Modify the AdminClients as required for establishing security as needed.
    • Modify the Port setting if needed to avoid port conflicts.
  4. In the [Email] section, make modifications as defined by your company’s SMTP setup.
  5. In the [ChildServices] section, remove the setting for the BEACMRepoFetch service.
  6. Remove the [BEACMRepoFetch] section.
  7. In the [IDOLServer], [HTTPFetch] and [FileSystemFetch] sections, modify each path to ensure the executable files use the location on the target server. These paths must be absolute.
  8. If you changed the Service Port or ACI port (or plan on doing so in the agentstore.cfg file), you need to adjust these settings to match.
  9. When finished, save your changes.

Configuring Agentstore

Agents provide the facilities to find and monitor information from a configurable list of internet and intranet sites, news feeds, chat streams and internal repositories that you want to enable your portal users to search.

For more information about using agents, see the Autonomy IDOL Server Guide.

To configure the Agentstore for your cluster, edit the agentstore.cfg file. The file is located //IDOLserver/IDOL/agentstore/agentstore.cfg.

To configure agentstore:

  1. Open the agentstore.cfg file in a text editor.
  2. Modify [License], [Service] and [Server] settings as required for port conflicts and security.
  3. Locate and replace all file and directory settings and adjust to point to the new location. The paths must be absolute.
    • In the [Paths] section, change the TemplateDirectory to point to the new location.
    • In the [Logging] section, change the LogDirectory and the LogArchiveDirectory to point to the new location.
    • In the [LanguageTypes] section, change the LanguageDirectory to point to the new location.
  4. When finished, save your changes.

Configuring HTTP Fetch

HTTP Fetch is responsible for crawling specified websites and passes the content to the IDOLServer for indexing.You need configure this fetch and create HTTP fetch jobs that you need.

You do this by editing the HTTPFetch.cfg file. It is located //AUTONOMYHOME/HTTPFetch/HTTPFetch.cfg

To configure the HTTP Fetch:

  1. Open the HTTPFetch.cfg in a text editor.
  2. The [Service] section determines which machines are permitted to use and control the HTTPFetch service via the service port. Modify the port and client security control as required.
    3. Note: If you modify the port settings in this file, you need to update the HTTPFetch port settings in the AutonomyDiSH.cfg file.
  3. The [Default] section contains the default settings that apply to all the jobs that you define in [Spider]section. If you changed the IndexPort in the AutonomyIDOLserver.cfg file, you need to modify the IndexPort setting to match.
  4. When finished, save your changes.
  5. Create HTTP Fetch jobs as required (to spider and index the websites you want to search). For information about creating fetch jobs, see the Autonomy HTTPFetch documentation.

Configuring File System Fetch

File System Fetch polls specified areas of a filesystem and, when content changes are found, imports the content and passes the content to the IDOLServer for indexing.

To control how files are imported from an internal location (for example, from a computer on your network), you need to configure File System Fetch and then create the fetch jobs you need.

The File System Fetch configuration file is located: //autonomyhome/FileSystemFetch/FileSystemFetch.cfg

Use a text editor to edit the FileSystemFetch.cfg file to match your production environment.

To configure File System Fetch:

  1. Modify the [Server] and [Service] sections to change ports, if needed, and to control security. If you modify the port information in this file, you need to also update the settings related to File System Fetch in the AutonomyDiSH.cfg file.
  2. In the [Default] section, modify the IndexPort to match the IndexPort set in AutonomyIDOLserver.cfg, if necessary.
  3. When finished, save your changes.
  4. Create File System Fetch jobs as required (to spider and index certain file system locations) for your needs. For information about creating fetch jobs, see the Autonomy File System Fetch documentation.

If deploying WebLogic Portal in a cluster environment, you need to ensure that each machine in your cluster can access the content indexed by File System Fetch, see Staging File System Fetch within a WebLogic Cluster.

 

Setting up BEA Content Management Search

To set up full-text search for your BEA repositories, you must configure the BEA Content Management fetch and then enable your BEA repositories for full-text search.

This section includes the following topics:

Configuring the BEA Content Management Fetch

The BEA content management fetch enables full-text search for BEA repositories. For each managed server in your WebLogic Portal cluster, you need to configure the BEA content management fetch.

To configure the content management fetch:

  1. Set an environment variable called WLP_SEARCH_OPTION and assign it a value of minimal.
  2. Edit the BEACMRepoFetch.cfg file located in //operating_system_directory/internal/BEACMRepoFetch/BEACMRepoFetch.cfg. This file configures the settings for the full-text search of BEA repositories.
    • Modify [Server] and [Default] settings to change port numbers and client security as required.
    • Modify [Default] DreHost settings to point to the hostname or IP address of the server which is running the IDOLServer.
    • Modify [Default] IndexPort to match the IndexPort setting in the AutonomyIDOLServer.cfg file on the remote server.
  3. When finished, save your changes.
WARNING: Do not modify any other settings within this file.

Configuring BEA Repositories for Full-Text Search

After you have configured the BEA Content Management Fetch, you need to enable your BEA repositories to take advantage of full-text search. This ensures your BEA repositories can locate the Autonomy IDOL server.

This section includes the following topics:

Adding Autonomy Properties to your BEA Repository

You need to define Autonomy properties for your BEA repositories within the Virtual Content Repository using the Portal Administration Console. These properties ensure that your BEA repositories can locate the Autonomy services.

Table 4-2 lists the Autonomy properties you need to add.

Table 4-2 Autonomy Properties for BEA Repositories
Property
Definition
search.staging.area
This property is used to define the name of the destination directory to which to export content so it can be indexed.
The default directory is:
//BEA_HOME/weblogic/portal/third-party/autonomy/internal/
search.engine.host
This is the hostname for the machine on which the IDOL server resides.
search.index.port
This is the Autonomy index port.
This value needs to match the [Server]IndexPort setting in the autonomyIDOLServer.cfg file. See Configuring the Autonomy IDOL Server for information about this file.
search.query.port
This is the port setting that is used by the IDOL server.
This value needs to match the [Server]Port setting in the autonomyIDOLServer.cfg file. See Configuring the Autonomy IDOL Server for information about this file.
search.urlconnection.timeout
When Autonomy database commands are issued using HTTP to the search indexing port and the search engine port, this time-out setting is specifies the HTTP connection time-out, in milliseconds. The default time-out is 180000 (180 seconds).

Note: After you make any changes to repository properties, Portal Administration Console users must log out and log back in to view the changes.

To add a property to a repository:

  1. From the main menu of the Portal Administration Console, select Content > Content Management.
  2. In the resource tree, click Repositories to view the Manage | Repositories tree.
  3. In the Manage | Repositories resource tree, select the BEA Repository to which you want to add a property.
  4. In the Properties section on the Summary tab, click Add Property.
  5. In the Add Property dialog, enter the name and value for your property. Enter each property included in Table 4-2.
  6. Click Save.

A summary of the new repository information is displayed in the Summary tab.

Editing Full-Text Search Properties

You need to ensure that all full-text functions are enabled for each BEA repository you want to enable to use full-text search.

Table 4-3 lists the advanced full-text search repository properties and how they are used.

Table 4-3 Required Settings for Full-Text Search
Advanced Property
What it does:
Search Enabled
Enables users to search the repository using metadata.
Search Indexing Enabled
Allows content to be indexed for portal search. This enables portal developers to include full-text content search or metadata search in any portlets that they develop.
Full-Text Search Enabled
Enables users to search the repository using the full-text of the content within the repository.

To edit full-text search repository properties:

  1. Select Content > Content Management from the navigation menu at the top of the console.
  2. Select Manage | Repositories.
  3. In the resource tree, click the repository you want to modify to view its Summary tab.
  4. In the Advanced section, click Advanced to view the Edit Advanced Properties for Repository dialog.
  5. In the Edit Advanced Properties for Repository dialog, ensure that each property in Table 4-3 is enabled.
  6. When finished making changes, click Save.

Your modifications display in the Advanced section of the Summary page.

Note: After you disconnect a repository or make any changes to repository properties, Portal Administration Console users must log out and log back in to view the changes.

 

Staging File System Fetch within a WebLogic Cluster

When you deploy WebLogic Portal in a cluster environment, each machine in the cluster must be able to access information and content that is indexed by Autonomy fetches. For example, both BEA repositories and Autonomy’s File System Fetch use filesystems to store indexed content. You should configure each machine in your cluster to be able to access these filesystems.

Note: The BEA Content Management Fetch does not require these steps, see Setting up BEA Content Management Search

File System Fetch is used to index content that resides in a filesystem. When indexing, unless otherwise configured, the DREREFERENCE property is set to the complete path of the file. Therefore, with default queries, the link to return the actual content (file) will be the path to the file. Within a server cluster, each node in the cluster must have access to the filesystem on which the document resides.

  1. Create a shared filesystem that is accessible by both the host machine upon which File System Fetch resides and also accessible by each node in your WebLogic Portal cluster. The mapping to the path where the files reside must be the same for each node in the cluster and the FileSystemFetch host.
  2. Place the files to be imported/indexed into the shared drive as required.
  3. Configure the FileSystemFetch job to import/index the contents of the shared drive using the mapping from the above step. For more information about configuring File System Fetch, see the Autonomy File System Fetch documentation.
Note: When returning query results to the browser and displaying a link to access/download the file, pass the DREREFERENCE property (which will contain the fully qualified path/file name) through a servlet which will stream the file to the browser. For more information about indexing and queries, see the Autonomy IDOL Server Guide and the Autonomy JavaDoc.

 

Starting the Autonomy Services

You must configure the start script that is used to start Autonomy services on your server. You can either copy these to your target server and modify as required, based on your target directory or you can create similar scripts to meet your needs.

The Autonomy start script depends on two environment variables that should be set on your portal domain server: WL_HOME and WLP_SEARCH_OPTION.

You call this script with either start or stop as a parameter.

  1. Review the autonomy.cmd/sh files that reside in the //WebLogic_Home/Portal/third-party/autonomy-wlp92 directory.
  2. Modify as necessary. Be sure to map the Autonomy shared library directories and ensure that the AutonomyDiSH.exe is started.
  3. Run your script.
  4. Verify that the services are running. On Windows, use the TaskManager application and view the Processes tab. If using Unix, use the ps command to view a list of services that are currently running. The following services should be running:
    • content.exe
    • category.exe
    • community.exe
    • agentstore.exe
    • AutonomyIDOLserver.exe,
    • AutonomyDiSH.exe
    • HTTPFetch.exe
    • FileSystemFetch.exe.
  5. Inspect the log files for each of the services to verify that there were no errors such as port conflicts, license restrictions, and so on.

 

Installing Autonomy Service Dashboard

The Autonomy Service Dashboard is an stand-alone front-end web application that allows administrators to manage all Autonomy modules and child services running locally or remotely.

The Dashboard communicates with one or more Autonomy Distributed Service Handler (DiSH) modules that provide the back-end process for monitoring and controlling all the Autonomy child services, such as fetches.

You deploy the Autonomy Service Dashboard as a portal application within your enterprise application using the WebLogic Server Console. Before deploying the dashboard, you must modify the configuration to match your production environment.

For complete documentation on how to use the Autonomy Service Dashboard, see the Autonomy DiSH documentation.

This section includes the following topics:

Prepare the Dashboard for Installation

Before you deploy the Autonomy Service Dashboard, you need to edit the location configuration to match the new deployment location. The default location that is used is: \\weblogic92\portal\thirdparty\autonomy-wlp92\common\lib\.

To prepare the Autonomy Service Dashboard for installation:

  1. Copy the autonomyservicedashboard.cfg file to its new location.
  2. Edit the configuration information to match the new location by editing the web.xml file for the Autonomy Service Dashboard.
    1. Create a temporary directory to use when completing your edits. For example, c:/temp/working
    2. Copy the autonomyservicedashboard.war to your working directory.
    3. Using a compression utility such as WinZip or JavaJar, unzip the autonomyservicedashboard.war.
    4. Configure the //WEB-INF/web.xml file by editing the <context-param> value to match the new location of the autonomyservicedashboard.cfg file. The default location is C:/bea/weblogic92/portal/thirdparty/autonomy-wlp92/common/lib.
    5. Save the C:/working/META-INF/web.xml file.
  3. In the C:/temp/working directory, re-zip or re-compress the autonomyservicedashboard.war file. Overwrite the existing autonomyservicedashboard.war file with the new file of the same name that contains your modified web.xml file. Be sure to keep the files in their original directory structure, including the META-INF directory.
  4. Copy the autonomyservicedashboard.war file you just created back to the location where you want to locate the Autonomy Service Dashboard. For example, C:\bea\weblogic92\portal\thirdparty\autonomy-wlp92\common\lib\

Deploy the Autonomy Service Dashboard

You should deploy the Autonomy Service Dashboard in the same domain as is used by your portal cluster. However it needs to be deployed as a stand-alone application rather than part of your portal application.

Note: You can also deploy the Autonomy Service Dashboard into another web application container, such as Tomcat.

After you deploy the Autonomy Service Dashboard, you can use the default username of admin and the default password of admin to log in.

To deploy the Autonomy Service Dashboard:

  1. From the WebLogic Portal domain where you wish to deploy the Autonomy Service Dashboard, run the WebLogic Server Console.
  2. In the Domain Structure section, select Deployments. This displays a list of the deployed components.
  3. In the Change Center section, click Lock & Edit.
  4. In the Summary of Deployments section, click Install.
  5. Navigate to the location of the autonomyservicedashboard.war file and select it.
  6. Click Next.
  7. Select Install this deployment as an application and click Next.
  8. Make changes as required
  9. Click Finish.
  10. In the Summary of Deployments section, review any messages or errors displayed.
  11. In the Change Center section, click Activate Changes.
  12. Verify that the autonomyservicedashboard deployment is prepared.
  13. Mark the check box next to the autonomyservicedashboard deployment and click Start > Servicing all requests.
  14. In the Start Application Assistant, click Yes.
  15. Navigate to the Autonomy Service Dashboard to verify that is it deployed. The default URL is: http://localhost:7001/autonomyservicedashboard.
  16. Using the default login (admin) and password (admin), log in to the Autonomy Service Dashboard.
  17. Configure the Autonomy Service Dashboard to point to your DiSH implementation (use the same server and port settings that you used when you edited the autonomyDiSH.cfg file in Configuring the Autonomy DiSH.

    18. For complete documentation on how to use the Autonomy Service Dashboard, see the Autonomy DiSH documentation.

  18. Ensure that the IP address of the computer(s) running the Autonomy Service Dashboard are configured in the AdminClient settings of the services. For information on how to configure these settings, see the Autonomy IDOL Server documentation.

  Back to Top       Previous  Next