When the ALI Automation Service initiates a crawl, it issues a SOAP request to return a list of folders. It iterates over the list of folders and retrieves lists of documents with metadata. In general, the portal calls IDK interfaces in the following order. See the definitions that follow for more information.

The sections below provide helpful information on the interfaces used to implement a Content Crawler. For a complete listing of interfaces, classes, and methods, see the IDK API documentation.

IContainerProvider

IContainer

The portal uses the IContainer interface to query information about back-end resource directories. This interface provides the following methods:

• GetGroups and GetUsers return a list of the portal groups or users that have read access to the container. These calls are made only if the Web Service and Content Crawler objects are configured to import security. The portal batches these calls; the Content Crawler code should return all groups or users at once.
• GetChildContainers returns the containers inside the current container (i.e., subfolders of a folder). The value stored in the key CrawlerContants.TAG_DEPTH is used to determine how many times GetChildContainers is called (crawl depth). This value must be set via a Service Configuration page. If no value is stored with this key, GetChildContainers is never called; only the documents in the folder specified for the start location are crawled into the portal. Note: Setting CrawlerConstants.TAG_DEPTH to -1 could result in an infinite loop.
• GetChildDocuments returns the documents inside the current container (folder). The portal batches this call; the Content Crawler code should return all documents at once. The TypeNamespace and TypeID parameters define the Content Type for the document. TypeNamespace associates the document with a row in the Global Content Type Map, and the TypeID associates it with a particular Content Type. The value in ChildDocument.GetLocation is used in IDocumentProvider.AttachToDocument, so any information required by AttachToDocument must be included in the location string. You can describe the document using file or MIME, as shown in the example below.

  ChildDocument doc=new ChildDocument();
  String filename = WordDoc.doc;
  
  //Location is a crawler-specific string to retrieve doc, e.g., file name 
  doc.setLocation(filename);
  
  //TypeNameSpace is either FILE or MIME unless using a custom namespace (Notes, Exchange)
  //NOTE: example uses getCode because setTypeNameSpace expects a String
  doc.setTypeNameSpace(TypeNamespace.MIME.getCode()):
  
  //For file descriptions, TypeID is simply the document name with extension (i.e., filename)
  //For MIME descriptions, set the document type or map multiple file extensions to MIME types 
  doc.setTypeID("application/msword");
  
  //DisplayName is the name to display in the KD, usually overridden in IDocument.getMetaData();
  doc.setDisplayName(filename);

• GetMetaData (DEPRECATED) returns all metadata available in the repository about the container. The name and location are used in mirrored crawls to mirror the structure of the source repository. In most cases, the container metadata is only the name and description.

IDocumentProvider

IDocument

The IDocument interface allows the portal to query information about and retrieve documents. This interface provides the following methods:

• GetDocumentSignature allows the portal to determine if the document has changed and should be re-indexed and flagged as updated. It can be a version number, a last-modified date, or the CRC of the document. The IDK does not enforce any restrictions on what to use for the document signature, or provide any utilities to get the CRC of the document. This is always the first call made to IDocument; on re-crawls, if the documentSignature has not changed, no additional calls will be made.
• GetMetadata returns all metadata available in the repository about the document. The portal maps this data to properties based on the mappings defined for the appropriate Content Type, along with metadata returned by the associated accessor. The following field names are reserved. Additional properties can be added using the portal's Global Document Property Map; for details, see Configuring Custom Content Crawlers: Properties and Metadata. (Any properties that are not in the Global Document Property Map will be discarded.)

  Field Name	Description
  Name	REQUIRED. The name of the link to be displayed in the portal Knowledge Directory. Note: By default, the portal uses the name from the crawled file properties as the name of the card. To set the portal to use the Name property returned by GetMetadata, you must set the CrawlerConstants.TAG_PROPERTIES to REMOTE using the Service Configuration Interface.
  Description	The description of the link to be displayed in the portal Knowledge Directory.
  UseDocFetch	Whether or not to use DocFetch to retrieve the file. The default is False. If you use DocFetch, the value in the File Name field is used to retrieve the file during both indexing and click-through. If you do not use DocFetch, you must provide values for Indexing URL and Click-Through URL.
  File Name (required for DocFetch)	The name of the click-through file, used for DocFetch.
  Content Type (required for DocFetch)	The content type of the click-through file, used to associated the crawled document with the Global Content Type Map.
  Indexing URL (public URL)	(Required if not using DocFetch.) The URL to the file that can be indexed in the portal. URLs can be relative to the Remote Server. If a file is publicly accessible via a URL, that URL can be used to access the document for both indexing and click-through. Documents that cannot be indexed must provide an additional URL at crawl-time for indexing purposes. For details on crawling secured content, see Accessing Secured Content .
  Click-Through URL (public URL)	(Required if not using DocFetch.) The URL to the click-through file. URLs can be relative to the Remote Server. For details on crawling secured content, see Accessing Secured Content.
  Image UUID (optional)	This parameter is only required for custom Content Types. For standard Content Types, the accessor will assign the correct image UUID.