Running the Sample Web Crawler Plugin

This section provides instructions for running the sample Web Crawler plugin, a custom parse filter plugin that adds HTML meta tags as additional properties to the output records.

About the Web Crawler plugin framework
The Endeca Web Crawler is based on the Apache Nutch open-source project. As a result, its major functionality is implemented as plugins. Its framework allows you to write your own plugins, such as plugins that extract additional content from Web pages.
About the sample custom filter plugin
Custom filters (ParseFilter) implement content extensions. These filters can examine the contents of a page (either the raw page contents or the parsed DOM) and add additional properties to records that are produced.
Adding a custom plugin to the Endeca Web Crawler
This topic provides an overview of how to add a custom plugin to the Endeca Web Crawler.
Opening the sample plugin project
For the purpose of this sample, you load the sample parse filter plugin project. If you were creating your own plugin, you would create your own Eclipse project.
Overview of the sample HTMLMetatagFilter plugin
For the purpose of this sample, we use the source for the HTMLMetatagFilter class that is in the HTMLMetatagFilter.java source file (in the IAS\<version>\sample\custom-web-crawler-plugin\src directory). If you were writing your own plugin, you would write the code for your custom plugin.
Overview of the plugin.xml file
The plugin.xml file describes the plugin to the Web Crawler. The file resides in the plugin directory along with the JAR file.
Building the sample plugin
For the purpose of this sample, use Eclipse to build a JAR of the sample Web Crawler parse plugin.
Adding the plugin to the IAS lib directory
After you build the Jar for your custom plugin, create a directory for the plugin and copy this to the Web Crawler's plugin directory.
Activating the plugin for the Web Crawler
Oracle recommends that you modify the crawl-specific site.xml file, rather than the global default.xml file (this is because the site.xml settings override the default.xml global settings).
Running the Web Crawler with a new plugin
After you activate the new plugin, you can run new crawls exactly as before.