Enabling Search Logging

In addition to the application server logs, Search includes two additional log file types:

Search activity logs. Search receives requests from and returns responses to the user interface in XML form; these activity files include detailed information about the search.
Search engine logs. The engine logs can help determine whether an engine is functioning properly.

Note: Log files can quickly consume a great deal of space. Either clean out the logging directory often, or turn logging off when it is not required.

Enabling Search Activity Logs

The search activity files record the request input and response output for an engine. Errors in the response files can help with troubleshooting, and a missing response might indicate a timeout or other communication problem.

By default, the engine activity files log only request and response pairs corresponding to errors. You can use Dynamo Administration to enable logging for other activities. To enable logging:

Go to http://host:port/nucleus/atg/search/routing/ SearchEngineService/ (see the ATG Installation and Configuration Guide for information on Dynamo Administration).
Change the loggingAllRequests property value to true to log all requests made to the Search engine. You can also set the following individual properties to true to log only certain request types:
- LoggingDeploy
- LoggingIndex
- LoggingLoad
- LoggingSearch
- LoggingStart
- LoggingStop
The indexing server writes out request and response XML to the following directory:
<ATG9dir>\home\logs\searchEngineActivity

The names of the log files provide information about the contents. For example, the log file name 0006320004_m_2011-04-21-101254_26200058_mymachine_6100_search_request.xml includes the following components:

0006320004—Request ID
m—Indicates that the file is a merge file, representing the final results returned to the user. The merge files do not include a partition, host, or port number, since they include data from multiple partitions, and potentially multiple hosts.
2011-04-21-101254—Timestamp in YYYY-MM-DD-HHMMSS format
26200058—Partition ID
mymachine—Host name
6100—Port number
search—Request type
request—File type

A further option is to use verbose logging. When used, in addition to requests, responses, and merges, the log files include two additional types: engine_info and stack_trace. To use verbose logging, set the loggingVerboseStyle property to true.

To change the directory where activity files are stored, edit the engineActivityPath property value and set it to the directory you want to use.

You can view the most recent activity logs directly in Dynamo Administration. To view activity logs, navigate to http://host:port/nucleus/atg/search/routing/SearchEngineService. Note that the log history visible in Dynamo Administration is reset when the application server or Search engine is restarted; however, files in the SearchEngineActivity directory are not affected by restarts.

Enabling Search Engine Logs

Each Search engine has an associated engine log file. Errors in the log files can help with troubleshooting, and a missing log file can indicate a problem with starting that engine.

The engine log contains platform, build, and system information at the beginning of the file:

2011/05/04 15:56:14 : LOGGING : Server : 0 :      Platform: x86-Windows32
2011/05/04 15:56:14 : LOGGING : Server : 0 :      Build: ATGSearch_9.3_090504-
002806
2011/05/04 15:56:14 : LOGGING : Server : 0 :      Unicode Support: utf8
2011/05/04 15:56:14 : LOGGING : Server : 0 :      Processors: 2
2011/05/04 15:56:14 : LOGGING : Server : 0 :      Processor speed (MHz): 2993
2011/05/04 15:56:14 : LOGGING : Server : 0 :      Total physical memory (MB): 4029
2011/05/04 15:56:14 : LOGGING : Server : 0 :      Avail physical memory (MB): 1399
2011/05/04 15:56:14 : LOGGING : Server : 0 :      Total swap space (MB): 6238
2011/05/04 15:56:14 : LOGGING : Server : 0 :      Avail swap space (MB): 3744

The platform and build information can help you determine whether the correct version of the engine is installed. The system information can show whether the system has sufficient processors, physical memory, and swap space.

A search engine log also contains the index filename, partition name, and the Content ID (also known as Partition GUID) associated with it, as shown in the following example:

2011/05/04 15:56:19 : INFO : Server : 0 : Starting copy of
'C:\ATG\ATG9.3\Search9.3\SearchEngine\x86-win32\data\initial.index' to
'../indexFiles\400001\initial.index'

2011/05/04 15:56:25 : INFO : Server : 0 : Content ID: 0fd62287-c697-4fe4-
8639-716e2c0e2f07

This information can show whether the path to the deployment share directory is correct, and can be used to match the partition listed in the Search activity logs.

Each step in the indexing process generates a separate log; a full index task generates a minimum of three logs, corresponding to the estimation, indexing, and deployment steps. If the index spans multiple physical partitions, each additional partition generates additional logs for the indexing and deployment steps.

For an estimation or indexing step, the search engine is launched with an initial partition. This is reflected in the logs by the index filename initial.index. For a deployment step, the engine log starts with an existing partition. This is reflected in the logs by filenames with idx and stg extensions. The content ID from the deployment step matches that of the prior indexing step; however the estimation step uses a different content ID. See the following example:

2011/05/04 15:57:14 : INFO : Server : 0 : Starting copy of 'C:\ATG\ATG9.3\
Search9.3\bolt.100002.ContentSet1.400020.idx' to '../indexFiles\400003\
bolt.100002.ContentSet1.400020.idx'

2011/05/04 15:57:14 : INFO : Server : 0 : Starting copy of 'C:\ATG\ATG9.3\
Search9.3\bolt.100002.ContentSet1.400020.stg' to '../indexFiles\400003\
bolt.100002.ContentSet1.400020.stg'

2011/05/04 15:57:15 : INFO : Server : 0 : Content ID: 9b6342d1-bfb2-4010-
992c-9793c7ff0cae

The default logging level is I (information). This level includes warnings and errors, but does not include requests and responses from the engine. Set the logging level to d (debug) to include the request and response XML in the output, as in the following example:

2011/05/04 18:11:09 : DEBUG : Server : 0 : request RequestXML: <query
minScore='300' RQText='ANSWER' exclusion='' sorting='SCORE' suggestion=
'TERMSET' version='2.0.1' autoRefine='false' docSetSort='None' debug='0'>

2011/05/04 18:11:09 : DEBUG : Server : 0 : response RequestXML: <?xml
version="1.0" encoding="utf-8"?>

<answer contentID="d2127cff-6209-4e3a-8ed4-
3800461c33ad" sorting="score" docSetSort="None"

To enable engine logging:

Go to http://host:port/nucleus/atg/search/routing/ LaunchingService (see the ATG Installation and Configuration Guide for information on Dynamo Administration).
Change the engineLogLevel property value to D to enable debug logging. This will log all requests and takes effect upon the next engine restart.
The search engine writes out files named atgsearch_*.log to the following directory:
<ATG9dir>\Search9.3\SearchEngine\platform\bin

To change the directory where engine log files are stored, create a filelog_basepath environment variable and set it to the directory you want to use.

By default, engine log files are retained for two days. To configure how long the files are retained, edit the params property value of the /atg/search/routing/LaunchingService.properties file and add a –t flag as shown:

params=atgsearch,-p,$port$,-s,AESoapWaspConfig.xml,-a,$AEConfigXml$,-
m,Messages.xml,-l,$loglevel$,-t,NN

ATG Search Administration Guide