Introduction

Web Crawler overview
The Endeca Web Crawler is installed by default as part of the IAS installation. The Web Crawler gathers source data by crawling HTTP and HTTPS Web sites and writes the data in a format that is accessible to Endeca Information Discovery Integrator (either XML or a Record Store instance).

Running a sample Web crawl of oracle.com
You can examine the configuration and operation of the Web Crawler by running a sample Web crawl. The sample is located in the <install path>\IAS\workspace\conf\web-crawler\polite-crawl directory.

Running a sample Web crawl that writes to a Record Store
In this topic, you run a sample Web crawl that writes output to a Record Store instance instead of to a file on disk. This sample is stored in <install path>\IAS\<version>\sample\webcrawler-to-recordstore. The run-sample script runs the sample Web Crawler.