This section includes examples of common URL filter syntax.
To crawl the development branch of the Example corporate Web site, use a URL inclusion filter with a regular expression pattern of:
/example\.com\/dev\/.*/i
This pattern confines the crawler to URLs of the form:
example.com/dev/
To restrict a crawler so that it does not crawl URLs on a corporate intranet (for example, those located on host intranet.foo.com/dev), use a Host exclusion filter with a regular expression pattern of:
/intranet\.example\.com/