Example syntax of URL filters

This section includes examples of common URL filter syntax.

To crawl only file systems (not HTTP or HTTPS hosts), use a URL inclusion filter with a regular expression pattern of: /^file/i
To crawl only documents with an .htm or .html extension, use a URL inclusion filter with a regular expression pattern of: /\.html?$/i
To crawl the development branch of the Example corporate Web site, use a URL inclusion filter with a regular expression pattern of:

/example\.com\/dev\/.*/i

This pattern confines the crawler to URLs of the form:

example.com/dev/
To restrict a crawler so that it does not crawl URLs on a corporate intranet (for example, those located on host intranet.foo.com/dev), use a Host exclusion filter with a regular expression pattern of:

/intranet\.example\.com/