Differences among the site.xml and default.xml files

The following table lists the differences between the site.xml files in the non-polite-crawl and the polite-crawl directories, as well as the differences between those files and the global default.xml file.

config property

default.xml

polite site.xml

non-polite site.xml

http.robots.ignore

false

false

true

fetcher.delay

2.0

1.0

0.0

fetcher.threads.total

100

not used

52

fetcher.threads.per-host

1

1

52

output.file.directory

workspace

polite-crawl-workspace

non-polite-crawl-workspace

output.file.name

webcrawler-output

polite-crawl

non-polite-crawl

output.file.is-xml

false

true

true

output.file.is-compressed

true

false

false

Copyright © Legal Notices