The crawl depth (as set by the -d
flag) specifies how many levels of page links will be followed. Each URL in the seed has a level of 0 and each link from a seed URL has a level of 1. The links from a level 1 URL have a level of 2 and so on.
For example, if the seed is www.endeca.com
, the levels are as follows:
Level 0: www.endeca.com is level 0 and has a link to about.html. Level 1: about.html is level 1 and its links are level 2. Level 2: contacts.html is level 2 and its links are level 3.
Therefore, if you want to crawl all the level 2 pages, specify -d 2
as the flag argument.