> AquaLogic User Interaction Development Guide > About Content Service Development > About Content Crawlers > About Content Crawler Indexing > Indexing Streaming Content

AquaLogic User Interaction Development Guide

     Previous Next  Open TOC in new window   View as PDF - New Window  Get Adobe Reader - New Window
Content starts here

Indexing Streaming Content

If the content being crawled is in a file, the file can be streamed directly from the source directory.

The following steps describe a typical custom mechanism to return files in a indexable format and set the content type and file name using the appropriate headers.

  1. In IDocument, get all the variables needed to access the document and add them to the query string of the indexing servlet. This could be as simple as a UNC path for a file crawler or as complicated as server name, database name, schema, table, primary key(s) and primary key value(s) for a database record. It depends entirely on the content crawler and the document being crawled. Make sure all values are URLEncoded.
  2. Add the content type to the query string.
  3. In IDocument, add URLEncoded credentials to the query string. Keep in mind that URLEncoding the credentials will turn a '+' to a space, which must be turned back into a space in the indexing servlet.
  4. Pass back URLs via the IDK's DocumentMetadata class that point to the servlet(s).
    • UseDocFetch: Set UseDocFetch to False.
    • IndexingURL: Set the IndexingURL to the endpoint/servlet that provides the indexable version of the file, including the query string arguments defined in steps 1-3 above.
    • ClickThroughURL: Set the ClickThroughURL to the endpoint/servlet that provides the path to be used when a user clicks through to view the file. During the crawl, the ClickThroughURL value is stored in the associated Knowledge Directory document.
  5. In the indexing servlet, get the location string and content type from the query string and parse the location string to get the path to the resource.
  6. Obtain the resource.
  7. Set the ContentType header and the Content-Disposition header.
  8. Stream the file (binary or text) or write out the file (text) in a try-catch block.
Parent topic: About Content Crawler Indexing

  Back to Top      Previous Next