The URLs generated by web applications can create problems for web spiders. These URLs typically include query parameters that the spider may not know how to interpret. In some cases, a spider will simply ignore a page whose URL includes query parameters.

Even if the spider does index the page, it may give the page lower ranking than desired, because the URL may not contain search terms that could increase the ranking. For example, consider a typical URL for an ATG Commerce site:


This type of URL is sometimes referred to as “dynamic,” because the content of the page is dynamically generated based on the values of the query parameters.

Now consider a static URL for the same page:


A spider is more likely to index the page with the static URL, and when it does, it is likely to mark “Mountain Bikes” and “Q33 UltraMountain” as key search terms and weight the page heavily for them. As a result, when a user searches for one of these terms, this page appears near the top of the search results. The dynamic URL may return the same page and content when it’s clicked, but it is less likely to be ranked highly for these searches, and in some cases may not be indexed at all.

To address this concern, the Oracle ATG Web Commerce platform includes a URL recoding feature that enables you to optimize your pages for indexing by web spiders, without compromising the human usability of the site. The key to this feature is the ability to render URLs in different formats, depending on whether a page is accessed by a human visitor or a web spider. This is handled through the atg.repository.seo.ItemLink servlet bean, which uses the User-Agent property of the HTTP request to determine the type of visitor. If the visitor is a spider, the servlet bean renders a static URL that the spider can use for indexing; otherwise, it renders a standard Oracle ATG Web Commerce dynamic URL.

Of course, the Oracle ATG Web Commerce request-handling components cannot actually interpret these static URLs. Therefore, URL recoding also requires a servlet (atg.repository.seo.JumpServlet) that reads incoming static URLs (for example, if a user clicks a link returned by a Google search), and translates these URLs into their dynamic equivalents.

This section describes: