  
 |
To debug custom content crawlers, use logging.
Logging is an important component of any successful content crawler. Logging allows you to track progress and find problems.
In most implementations, using Log4J or Log4Net for logging is the best approach. The IDK ICrawlerLog object is more efficient and useful than ALI Logging Spy or a SOAP trace, but it only includes standard exceptions and messages from ContainerProvider.AttachToContainer.
If you are viewing the ICrawlerLog, do not assume that the every card was imported if the job is successful. Successful means no catastrophic failures, such as ALI Search not started, or unable to attach to the start node. Individual document failures will not fail a job.
If you are viewing logs created by Log4net or Log4j, see the associated documentation for logging configuration options. Both products allow you to specify a file location and a rollover log with a specified file size. If you know the location of the file, it is not difficult to create a servlet/aspx page that streams the file from the log to the browser.
   |