This chapter contains the following sections:
The functions discussed in this chapter are used during filtering to manipulate or generate information on the resource. The robot can then process the resource by calling filtering functions. These functions can be used in Enumeration and Generation filters in the filter.conf file.
The assign-source function assigns a new value to a given information source. This permits editing during the filtering process. The function can assign an explicit new value, or it can copy a value from another information source.
The parameters used with the assign-source function and their description are:
Name of the source whose value is to be changed.
Specifies an explicit value.
Information source to copy to dst
You must specify either a value parameter or a src parameter, but not both.
Data fn=assign-source dst=type src=content-type |
The assign-type-by-extension function uses the resource’s file name to determine its type and assigns this type to the resource for further processing.
The setup-type-by-extension function must be called during setup before assign-type-by-extension function can be used.
The parameter used with the assign-type-by-extension function and its description is:
Source of file name to compare. If you do not specify a source, the default is the resource’s path.
MetaData fn=assign-type-by-extclear-source |
The clear-source function deletes the specified data source. You typically do not need to perform this function. You can create or replace a source by using the assign-source function.
The parameter used with the clear-source function and its description is:
Name of source to delete.
The following example deletes the path source:
MetaData fn=clear-source src=path |
The convert-to-html function converts the current resource into an HTML file for further processing, if its type matches a specified MIME type. The conversion filter automatically detects the type of the file it is converting.
The parameter used with the convert-to-html function and its description is:
MIME type from which to convert.
The following sequence of function calls causes the filter to convert all Adobe Acrobat PDF files, Microsoft RTF files, and FrameMaker MIF files to HTML, as well as any files whose type was not specified by the server that delivered it.
Data fn=convert-to-html type=application/pdf Data fn=convert-to-html type=application/rtf Data fn=convert-to-html type=application/x-mif Data fn=convert-to-html type=unknown |
The copy-attribute function copies the value from one field in the resource description into another.
The parameters used with the copy-attribute function and their description are:
Field in the resource description from which to copy.
Item in the resource description into which to copy the source.
Maximum length of the source to copy.
Boolean parameter indicating whether to fix truncated text (such as not leaving partial words). This parameter is false by default.
Generate fn=copy-attribute \\ src=partial-text dst=description truncate=200 clean=true |
The generate-by-exact function generates a source with a specified value, but only if an existing source exactly matches another value.
The parameters used with the generate-by-exact function and their description are:
Name of source to generate.
Value to assign dst.
Source against which to match.
The following example sets the classification to Siroe if the host is www.siroe.com.
Generate fn="generate-by-exact" match="www.siroe.com:80" src="host" value="Siroe" dst="classification" |
This generate-by-prefix function generates a source with a specified value, but only if the prefix of an existing source matches another value.
The parameters used with the generate-by-prefix function and their description are:
Name of the source to generate.
Value to assign to dst.
Source against which to match.
Value to compare to src.
The following example sets the classification to Search if the protocol prefix is HTTP:
Generate fn="generate-by-prefix" match="http" src="protocol" value="World Wide Web" dst="classification" |
The generate-by-regex function generates a source with a specified value, but only if an existing source matches a regular expression.
The parameters used with the generate-by-regex function and their description are:
Name of the source to generate.
Value to assign to dst.
Source against which to match.
Regular expression string to compare to src.
The following example sets the classification to Siroe if the host name matches the regular expression *.siroe.com. For example, resources at both developer.siroe.com and home.siroe.com will be classified as Siroe:
Generate fn="generate-by-regex" match="\\\\*.siroe.com" src="host" value="Siroe" dst="classification" |
The generate-md5 function generates an MD5 checksum and adds it to the resource. You can then use the filter-by-md5 function to deny resources with duplicate MD5 checksums.
none
Data fn=generate-md5 |
The generate-rd-expires function generates an expiration date and adds it to the specified source. The function uses metadata such as the HTTP header and HTML <META> tags to obtain any expiration data from the resource. If none exists, it generates an expiration date three months from the current date.
The parameter used with the generate-rd-expires function and its description is:
Name of the source. If you omit it, it defaults to rd-expires.
Generate fn=generate-rd-expires |
The generate-rd-last-modified function adds the current time to the specified source.
The parameter used with the generate-rd-last-modified function and its description is:
Name of the source. If you omit it, it defaults to rd-last-modified.
Generate fn=generate-last-modified |
The rename-attribute function changes the name of a field in the resource description. It is most useful in cases where, for example, extract-html-meta copies information from a <META> tag into a field, and you want to change the name of the field.
The parameter used with the generate-rd-last-modified function and its description is:
String containing a mapping from one name to another.
The following example renames an attribute from author to author-name:
Generate fn=rename-attribute src="author->author-name" |