You can use the free WGet utility to retrieve Web pages for indexing. To use:
Download WGet from
http://www.christopherlewis.com/WGet/wget_SVN.zip
Unzip the file. See the
wget.hlp
file for information on using the utility.Run WGet against the Web site you want to retrieve. For example:
wget –P atg -r -l 2 http://www.mycorp.com
This example downloads the URL to an
/atg
directory, and specifies two levels of recursion; the maximum supported is five levels.Add the directory that WGet created (in this example, the
atg
directory) as file system content for your project. Be sure to specify the URL of the Web site you are retrieving as the external access URL in the content Advanced Settings. In this example, you would specify http://www.mycorp.com.Index the project.