Knowing how the Web Crawler processes URLs helps you understand where a new plug-in fits in, because the URL processing is accomplished by a series of plug-ins.

Each URL is processed by a thread in the following manner:

The processing flow is as follows:

In the processing flow, the sample htmlmetatags plug-in would be part of step 5, because it does additional processing of the parsed content.


Copyright © Legal Notices