You can use the free WGet utility to retrieve Web pages for indexing. To use:
Download WGet from
http://www.christopherlewis.com/WGet/wget_SVN.zipUnzip the file. See the
wget.hlpfile for information on using the utility.Run WGet against the Web site you want to retrieve. For example:
wget –P atg -r -l 2 http://www.mycorp.comThis example downloads the URL to an
/atgdirectory, and specifies two levels of recursion; the maximum supported is five levels.Add the directory that WGet created (in this example, the
atgdirectory) as file system content for your project. Be sure to specify the URL of the Web site you are retrieving as the external access URL in the content Advanced Settings. In this example, you would specify http://www.mycorp.com.Index the project.
