Web spiders (also called robots) crawl the Web and create indexes for Web search services such as Google. This activity is generally benign, but it can skew reporting results. For example, if a spider issues a search query, that query will be reflected in search reports you run. This is generally undesirable, since you typically want the reports to reflect only queries issued by actual site visitors. Therefore, the SearchMessageService has a mechanism for determining whether a search query is being issued by a spider, and if it is, suppressing the firing of search events.

To enable this mechanism, you set the SearchMessageService component’s spiderlikeTypes property to an array of the atg.servlet.BrowserType components that you consider to be spiders. When a search query is issued, the SearchMessageService examines the userAgent property of the QueryRequest component. SearchMessageService compares the value of the userAgent property with the values of the patterns properties of the spiderlikeTypes components, and if it finds a match, suppresses the events.

Typically, the search request’s userAgent property is set to the value of the User-Agent property of the HTTP request. You can override this value by explicitly setting the userAgent property in the properties file of the search request component. This is what happens in the Oracle ATG Web Commerce Merchandising Search Testing environment. When you use Search Testing, the userAgent property of the QueryRequest component is set to SearchTesting. By default, one of the spiderlikeTypes components is /atg/dynamo/servletpipeline/BrowserTypes/Robot, and one of the entries in this component’s patterns array is SearchTesting, so events are not fired.