How Does Crawler Work?

Just like a web crawler, a set of site collection URLs are defined through the admin portlet as starting points. The crawler, however, is using the web service provided in Sharepoint service for retrieving data. It keeps track of timestamps of each site, list and item to avoid unnecessary data retrieving and indexing. The data collected are indexed into Web Space Server as search index. ACL and Membership are part of metadada indexed, so it is import that all Sharepoint services should run in a single windows domain; the same domain defined in the user mapping described in the next section.