Appendix B.  Using WGet for Web Site Indexing

You can use the free WGet utility to retrieve Web pages for indexing. To use:

  1. Download WGet from

    http://www.christopherlewis.com/WGet/wget_SVN.zip

  2. Unzip the file. See the wget.hlp file for information on using the utility.

  3. Run WGet against the Web site you want to retrieve. For example:

    wget –P atg -r -l 2 http://www.mycorp.com

    This example downloads the URL to an /atg directory, and specifies two levels of recursion; the maximum supported is five levels.

  4. Add the directory that WGet created (in this example, the atg directory) as file system content for your project. Be sure to specify the URL of the Web site you are retrieving as the external access URL in the content Advanced Settings. In this example, you would specify http://www.mycorp.com.

  5. Index the project.