About the output properties for crawls

The OutputConfig class specifies whether the output from a crawl is stored in a Record Store instance, an output file, or in an MDEX compatible format (Dgidx files).

The OutputConfig class uses two methods to set the properties: setModuleId() and setModuleProperties().

The setModuleId() method sets the module ID of the output type. You specify a string value to indicate the type of output. The string can be set to either:

You can set one output option per crawl configuration.

Each ModuleProperty is a key/value pair or a key/multi-value pair that provides configuration information about this an output type.

You specify a ModuleProperty by calling setKey() to specify a string representing the key and by calling setValues() to set one or more corresponding values.

You then set eachModuleProperty on the SourceConfig object by calling addModuleProperty().

The OutputConfig class allows a client to write the crawl output to a Record Store instance.

Record Store Property Key Name

Key Value

host

The name of the host on which the Record Store is running. The default is localhost.

port

The port number on which the Record Store is listening. The default is 8500.

isPortSsl

Specify how to interpret the port setting.

A value of true means that port is an SSL port and the API uses HTTPS for connections.

A value of false means that port is a non-SSL port and the API uses HTTP for connections. The default is false.

Specify false if you enabled HTTPS redirects.

instanceName

The name of the Record Store instance that you want to write output to. The default is <crawlID>.

isManaged

A Boolean value that indicates whether the Record Store instance is managed or not. Managment ties a Record Store instance to its corresponding crawl configuration. Specifying true indicates that a Record Store instance is created if you run a crawl and a Record Store instance does not already exist. Specifying true also indicates that a Record Store instance is deleted if you delete the corresponding crawl configuration. The default is true (is managed).


Here is an example of the output properties for a crawl writing to a Record Store instance. 

// Create the output configuration.
OutputConfig outputConfig = new OutputConfig();

// Create a Record Store module ID.
ModuleId moduleId = new ModuleId("Record Store");

// Set the module ID in the output configuration.
outputConfig.setModuleId(moduleId);

// Create a module property object.
ModuleProperty host = new ModuleProperty();
// Set the key for specifying the host name.
host.setKey("host");
host.setValues("localhost");

// create a module property object.
ModuleProperty port = new ModuleProperty();
// set the key for specifying the port number
port.setKey("port");
port.setValues("8500");

// Create a module property object.
ModuleProperty instanceName = new ModuleProperty();
// set the key for specifying the instance name of the Record Store
instanceName.setKey("instanceName");
instanceName.setValues("RS1");

// Create a module property object.
ModuleProperty isManaged = new ModuleProperty();
// Set the key for specifying whether the Record Store is managed.
isManaged.setKey("isManaged");
isManaged.setValues("true");

// Create a list for the module property objects.
List<ModuleProperty> outputPropsList = new ArrayList<ModuleProperty>();

// Set the module property objects in the list.
outputPropsList.add(host);
outputPropsList.add(port);
outputPropsList.add(instanceName);
outputPropsList.add(isManaged);

// Set the module property in the output config (if not already done).
outputConfig.setModuleProperties(outputPropsList);

// Set the output configuration in the main crawl configuration.
crawlConfig.setOutputConfig(outputConfig);

// Create the crawl.
crawler.createCrawl(crawlConfig);

The OutputConfig class allows a client to write the crawl output in an MDEX compatible format (Dgidx input files).

MDEX Compatible Property Key Name

Key Value

inputDirectory

The path to the directory containing Developer Studio instance configuration files.

outputDirectory

The path to the directory where CAS writes output in an MDEX compatible format (i.e. as Dgidx input files). This CAS output is consumed by Dgidx.

dimensionValueIdManagerInstanceName

The name of the Dimension Value Id Manager for the application.


Here is an example of the output properties for a crawl writing to an MDEX compatible format (Dgidx files). 

// Create the output configuration.
OutputConfig outputConfig = new OutputConfig();

// Create an MDEX module ID.
ModuleId moduleId = new ModuleId("com.endeca.cas.output.Mdex");

// Set the module ID in the output configuration.
outputConfig.setModuleId(moduleId);

// Create a module property object.
ModuleProperty inputDir = new ModuleProperty();
// Set the key for specifying Developer Studio instance configuration files.
inputDir.setKey("inputDirectory");
inputDir.setValues("C:/Endeca/apps/ebizsampleapp/data/complete_index_config");

// create a module property object.
ModuleProperty outputDir = new ModuleProperty();
// Set the key for specifying the directory to store
// CAS output in an MDEX compatible format. 
outputDir.setKey("outputDirectory");
outputDir.setValues("C:/Endeca/apps/ebizsampleapp/data/dgidx_input");

// Create a module property object.
ModuleProperty dvalMgr = new ModuleProperty();
// set the key for specifying the instance name of the Record Store
dvalMgr.setKey("dimensionValueIdManagerInstanceName");
dvalMgr.setValues("ebizsampleapp-dimension-value-id-manager");

// Create a list for the module property objects.
List<ModuleProperty> outputPropsList = new ArrayList<ModuleProperty>();

// Set the module property objects in the list.
outputPropsList.add(inputDir);
outputPropsList.add(outputDir);
outputPropsList.add(dvalMgr);

// Set the module property in the output config (if not already done).
outputConfig.setModuleProperties(outputPropsList);

// Set the output configuration in the main crawl configuration.
crawlConfig.setOutputConfig(outputConfig);

// Create the crawl.
crawler.createCrawl(crawlConfig);

The OutputConfig class allows a client to write the crawl output to a record output file (i.e. file system output).

File System Property Key Name

Key Value

outputPrefix

The prefix of the output file (CrawlerOutput is the default prefix). Optional.

outputDirectory

The name and path of the output directory under the CAS Server's workspace directory. The default name of outputDirectory is output and the default name of <crawlID> is used to create a subdirectory for each crawl. This ensures each crawl has a unique subdirectory for its output. For example, if you use the default value for outputDirectory and have a <crawlID> of FileSystemCrawl, the resulting directory structure is CAS\workspace\output\FileSystemCrawl\.

outputXml

A Boolean value that sets the output format to either XML or binary. Specifying true sets the output to XML. Specifying false sets the output to binary. The default is false.

outputCompressed

A Boolean value that indicates whether the output file should be compressed. Specifying true compresses the output. The default is false (not compressed). Optional.


Here is an example of the output properties for a file system crawl. 

// Create the output configuration.
OutputConfig outputConfig = new OutputConfig();

// Create a file system module ID.
ModuleId moduleId = new ModuleId("File System");

// Set the module ID in the output configuration.
outputConfig.setModuleId(moduleId);

// Create a module property object.
ModuleProperty outputPrefix = new ModuleProperty();
// set the key for the output prefix
outputPrefix.setKey("outputPrefix");
outputPrefix.getValue().add("newPrefix");

// Set the outputPrefix module property on the output config.
outputConfig.addModuleProperty(outputPrefix);

// Create a module property object.
ModuleProperty outputDirectory = new ModuleProperty();
// Set the key for the output directory.
outputDirectory.setKey("outputDirectory");
outputDirectory.setValues("output");

// Set the outputDirectory module property on the output config.
outputConfig.addModuleProperty(outputDirectory);

// Create a module property object.
ModuleProperty outputXml = new ModuleProperty();
// Set the key for specifying whether output is in XML format.
outputXml.setKey("outputXml");
outputXml.setValues("true");

// Set the outputXml module property on the output config.
outputConfig.addModuleProperty(outputXml);

// Create a module property object.
ModuleProperty outputCompressed = new ModuleProperty();
// Set the key for specifying whether output is compressed.
outputCompressed.setKey("outputCompressed");
outputCompressed.setValues("true");

// Set the outputCompressed module property on the output config.
outputConfig.addModuleProperty(outputCompressed);

// Set the output config in the main crawl configuration.
crawlConfig.setOutputConfig(outputConfig);

// Create the crawl.
crawler.createCrawl(crawlConfig);

Copyright © Legal Notices