Web spiders (also called robots) crawl the Web and create indexes for Web search services such as Google. This activity is generally benign, but it can skew reporting results. For example, if a spider issues a search query, that query will be reflected in search reports you run. This is generally undesirable, since you typically want the reports to reflect only queries issued by actual site visitors. Therefore, the SearchMessageService
has a mechanism for determining whether a search query is being issued by a spider, and if it is, suppressing the firing of search events.
To enable this mechanism, you set the SearchMessageService
component’s spiderlikeTypes
property to an array of the atg.servlet.BrowserType
components that you consider to be spiders. When a search query is issued, the SearchMessageService
examines the userAgent
property of the QueryRequest
component. SearchMessageService
compares the value of the userAgent
property with the values of the patterns
properties of the spiderlikeTypes
components, and if it finds a match, suppresses the events.
Typically, the search request’s userAgent
property is set to the value of the User-Agent
property of the HTTP request. You can override this value by explicitly setting the userAgent
property in the properties file of the search request component. This is what happens in the Oracle ATG Web Commerce Merchandising Search Testing environment. When you use Search Testing, the userAgent
property of the QueryRequest
component is set to SearchTesting
. By default, one of the spiderlikeTypes
components is /atg/dynamo/servletpipeline/BrowserTypes/Robot
, and one of the entries in this component’s patterns
array is SearchTesting
, so events are not fired.