Output properties

You can set output properties in the default.xml file. You can configure output to either an output file (the default) or to a Record Store instance.

The properties in the table below allow you to specify the attributes of a crawl output file, such as its name, location, and output type. The default name of the output file is endecaOut and it is a compressed binary file by default.

Note

By default, the Web Crawler writes output to a file on disk. If desired, you can configure the Web Crawler to write output to a Record Store instance. Oracle recommends this approach.

Property Name

Description

output.file.directory

Directory name (default is workspace). Specifies the directory for the output file. The name is case-sensitive and is relative to where you run the crawl from. You can specify a multi-level path. Note that this setting can be overridden with the -w command-line flag.

output.file.name

File name (default is webcrawler-output). Specifies the filename of the output file. The name is case-sensitive.

output.file.is-xml

Boolean value (default is false). Specifies whether the output type is XML (true) or binary (false). XML is useful if you want to visually inspect the Endeca records after crawling.

output.file.is- compressed

Boolean value (default is true). Specifies whether to compress the Endeca records in a .gz file. Setting this property to true is useful when storing and transferring large files.

output.file.binary.file-size- max

Integer value (default is -1). Sets the maximum file size for binary output files. Output is written to a new file once the maximum size is reached. If the value is set to -1, no limits are imposed on the file size.

output.dom.include

Boolean value (default is false). Specifies whether to include the DOM for the Web page in the output Endeca records.

output.records.properties. excludes

Space-delimited list of output record properties (default is empty). Specifies the properties that should be excluded from the records. The names can be specified in a case-insensitive format. Note that wildcard names are not supported.

log.interval

Integer value in seconds (default is 60). Outputs crawl metrics information to the log every time this number of seconds has elapsed, per depth.

log.interval.summary

Integer value in seconds (default is 300). Outputs detailed crawl progress information (organized by host) every time this number of seconds has elapsed.

Related links

If the output.dom.include property is set to true, the Web Crawler normalizes the content of HTML documents into XHTML and stores it in the Endeca.Document.XHTML property in the record.

  1. In a text editor, open default.xml.

  2. Set the output.dom.include to true.

    You can now extract information from the XHTML using XSLT or any other XML processing system.

  3. Note that the Endeca.Document.Text property will also have extracted text, except that the XML header and the HTML tags are removed. Therefore, if you do not need the XHTML version of the content, set the output.dom.include property to false.

  4. Save and close the file.

The output.records.properties.excludes property allows you to specify a list of record properties that you want excluded from the records.

The list of the excluded property names is space delimited.

Note

Wildcards are not supported for the property names.

  1. In a text editor, open default.xml.

  2. Within the <configuration> element, add the following lines of code: 

    <property>
   <name>output.records.properties.excludes</name>
   <value>excludedProperties</value>
</property>

    Where excludedProperties is a space delimited list of the properties you wish to exclude.

  3. Save and close the file.

Example 3. Example of excluding record properties

For example, assume you want to exclude both Outlink properties from the output. You would add this entry to the site.xml configuration file:

<property>
   <name>output.records.properties.excludes</name>
   <value>Endeca.Document.Outlink Endeca.Document.OutlinkCount</value>
</property>

On the next crawl, the Endeca.Document.Outlink and the Endeca.Document.OutlinkCount properties will not appear in the output.

Note

You can add the exclusion list to the default.xml file, but the site.xml file is recommended because you can then specify different property exclusions for different crawl configurations.


For the output.file.binary.file-size-max property, if output has to be written to more than one output, the name pattern of the new files is similar to this example: 

endecaOut-sgmt000.bin
endecaOut-sgmt001.bin
endecaOut-sgmt002.bin

That is, if the output.file.name value is set to endecaOut, then the suffix -sgmt000 is used for the first file and the number is increased for subsequent files.

The site.xml files in the workspace/conf/web-crawler/polite-crawl and workspace/conf/web-crawler/non-polite-crawl directories contain these output file overrides.

config property

default.xml

polite site.xml

non-polite site.xml

output.file.directory

workspace

polite-crawl-workspace

non-polite-crawl-workspace

output.file.name

webcrawler-output

polite-crawl

non-polite-crawl

output.file.is-xml

false

true

true

output.file.is-compressed

true

false

false

Copyright © Legal Notices