com.endeca.eac.toolkit.component.cas

Class ContentAcquisitionServerComponent

java.lang.Object

com.endeca.soleng.eac.toolkit.base.EacElement

com.endeca.soleng.eac.toolkit.base.Provisionable

com.endeca.soleng.eac.toolkit.component.CustomComponent

com.endeca.eac.toolkit.component.cas.ContentAcquisitionServerComponent

public class ContentAcquisitionServerComponent
extends com.endeca.soleng.eac.toolkit.component.CustomComponent

This custom component represents a Content Acquisition System Server. It exposes methods for executing CAS crawls using the CAS Server. Crawls must be configured externally, and this component does not provide utilities for provisioning new crawls and changing the configuration of an existing crawl. Execution of commands against a non-existent crawl will fail with an exception. It is expected that developers will use an external tool or the CAS console to create and manage configuration of crawls. Though this is "provisioned" like a component in the app configuration, this object's provisioning is stored as a set of custom properties (key value pairs) on the provisioned host on which this component runs.

Field Summary

Fields

Modifier and Type	Field and Description
static String	CAS_HOST Property that specifies the CAS hostname.
static String	CAS_PORT Property that specifies the CAS port.

Fields inherited from class com.endeca.soleng.eac.toolkit.base.EacElement

appName, dataPrefix, eacHost, eacPort, elementId, lockManager, logDir, sslEnabled, workingDir

Constructor Summary

Constructors

Constructor and Description
ContentAcquisitionServerComponent() Constructor for the Config Manager custom component.

Method Summary

Methods

Modifier and Type	Method and Description
void	archiveCumulativePartials() Backs up the cumulative partials directory.
void	archiveDvalIdMappingsForCrawlIfChanged(String crawlName) Backs up dimension value Id mappings from a Dimension Value Id Manager to a CSV file at the path specified on the CAS Server.
void	checkCasVersion() This method checks the version of the CAS server (as reported by a service call to the CAS web service) against the expected version defined when this class was built.
void	checkCrawlCompletedSuccessfully(String crawlName) Checks whether or not the given crawl has completed successfully.
void	cleanCumulativePartials() Removes all accumulated partials files from the cumulative partials directory.
void	cleanOutputDir(String crawlName) Cleans the crawl's output directory (specified in the crawl configuration, usually through the CAS Console), removing any files in that directory that match the naming convention used by this component.
void	clearCrawlOutputFromDir(String crawlName, boolean isIncrementalCrawl, String hostId, String targetDir) Deletes files associated with the specified crawl from the specified directory.
void	clearFullCrawlOutputFromDestinationDir(String crawlName) Convenience method to invoke clearCrawlOutputFromDestinationDir method with the isIncrementalCrawl argument set to false.
void	clearIncrementalCrawlOutputFromDestinationDir(String crawlName) Convenience method to invoke clearCrawlOutputFromDestinationDir method with the isIncrementalCrawl argument set to true.
void	copyBaselineCrawlOutputToDestinationDir(String crawlName) Convenience method to invoke copyCrawlOutputToDestinationDir method with the isIncrementalCrawl argument set to false.
void	copyCrawlOutputToDestinationDir(String crawlName, boolean isIncrementalCrawl) This method is expected to be used in conjunction with the renameCrawlOutput method, as it assumes the naming convention created by that method.
void	copyIncrementalCrawlOutputToDestinationDir(String crawlName) Convenience method to invoke copyCrawlOutputToDestinationDir method with the isIncrementalCrawl argument set to true.
void	copyOutputDimensionsFile(String crawlName, String destDirectory) Copy (prefix).dimensions.xml file generated by the output of an MDEX-outputter crawl to a temporary directory for subsequent intake by Workbench.
void	exportDimensionValueIdMappings(String dimensionValueIdManagerInstanceName, String exportFilePath) Exports dimension value Id mappings from a Dimension Value Id Manager to a CSV file at the path specified on the CAS Server.
void	fetchPartialsToCumulativeDir(String crawlName) Copies the partial MDEX output for the given crawl to the cumulative partials directory.
String	getCasCrawlFullOutputDestDir()
String	getCasCrawlIncrementalOutputDestDir()
String	getCasCrawlOutputDestHost()
String	getCasHost()
int	getCasHttpSocketTimeout()
String	getCasPort()
String	getCrawlOutputDir(String crawlName) Queries the CAS to retrieve the crawl output directory from the crawl config.
String	getCrawlOutputPrefix(String crawlName) Queries the CAS to retrieve the crawl output file prefix from the crawl config.
String	getCrawlOutputSuffix(String crawlName) Queries the CAS to retrieve the crawl output file suffix from the crawl config.
com.endeca.cas.wsdl.CrawlerState	getCrawlStatus(String crawlName) Retrieves the status of the specified crawl.
String	getCumulativePartialsDir() Returns the directory on the CAS host where partial MDEX output should be accumulated.
String	getDvalIdMappingsArchiveDir() Returns the directory on the CAS host where DvalIdManager mappings should be written.
long	getLongPropertyValueOrDefault(String propName, long defaultValue)
long	getMaxMissedStatusChecksAllowed()
long	getMaxWaitTimeSeconds()
long	getMinWaitTimeSeconds()
int	getNumDvalIdMappingsBackups() Returns the number of backups of the DvalIdManager mappings to keep.
int	getNumMissedStatusChecks()
int	getNumPartialsBackups() Returns the number of backups of the cumulative partials directory to keep.
long	getSlowPollingIntervalMs()
long	getStandardPollingIntervalMs()
void	importDimensionValueIdMappings(String dimensionValueIdManagerInstanceName, String importFilePath) Imports dimension value Id mappings from a CSV file located at the specified path to a specified Dimension Value Id Manager.
void	incrementMissedStatusQueries()
boolean	isCrawlActive(String crawlName) Test whether the crawl is active (running or stopping).
boolean	isCrawlDefined(String crawlName) Requests a list of crawls defined in the CAS and returns true if one of them matches the specified crawl name.
boolean	isCrawlFileOutput(String crawlName) Returns true iff the crawl outputs Endeca record files.
boolean	isCrawlMdexOutput(String crawlName) Returns true iff the crawl outputs MDEX configuration files.
boolean	isCrawlRunning(String crawlName) Test whether the crawl is running.
boolean	isCrawlStopped(String crawlName) Test whether the crawl is stopped.
boolean	isCrawlStopping(String crawlName) Test whether the crawl is stopping.
List<String>	listCasCrawls() Returns a list of the crawls defined in the CAS.
void	renameBaselineCrawlOutput(String crawlName) Convenience method to invoke renameCrawlOutput method with the isIncrementalCrawl argument set to false.
void	renameCrawlOutput(String crawlName, boolean isIncrementalCrawl) Renames the crawl output file for a specified crawl to include the crawl name as a prefix as well as a timestamp if it's an incremental crawl.
void	renameIncrementalCrawlOutput(String crawlName) Convenience method to invoke renameCrawlOutput method with the isIncrementalCrawl argument set to true.
void	runBaselineCasCrawl(String crawlName) Runs to completion a full crawl of the CAS crawl with the specified name.
void	runIncrementalCasCrawl(String crawlName) Runs to completion an incremental crawl of the CAS crawl with the specified name.
void	setDefinition()
void	setNumMissedStatusChecks(int numMissedStatusQueries)
void	startBaselineCasCrawl(String crawlName) Executes a baseline crawl for the specified crawl name.
void	startIncrementalCasCrawl(String crawlName) Executes an incremental crawl for the specified crawl name.
boolean	waitForFinished(String crawlName) Waits for the specified crawl to finish (to be in a non-active state) or for the maximum amount of wait time to elapse.

Methods inherited from class com.endeca.soleng.eac.toolkit.component.CustomComponent

buildEacStub, getCustomDirectories, getHost, getHostId, getProperties, getProperty, isDefined, isDefinitionChanged, removeDefinition, setCustomDirectories, setHostId, setProperties, updateDefinition

Methods inherited from class com.endeca.soleng.eac.toolkit.base.Provisionable

getAppDefinitionFromEac, getCachedAppDefinition, getProvisioningPort, invalidateCachedAppDefinition, setCachedAppDefinition

Methods inherited from class com.endeca.soleng.eac.toolkit.base.EacElement

getAppName, getDataPrefix, getEacHost, getEacPort, getElementId, getLockManager, getLogDir, getWorkingDir, isSslEnabled, setAppName, setDataPrefix, setEacHost, setEacPort, setElementId, setLockManager, setLogDir, setSslEnabled, setWorkingDir

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CAS_HOST

public static final String CAS_HOST

Property that specifies the CAS hostname.

See Also:

Constant Field Values

CAS_PORT

public static final String CAS_PORT

Property that specifies the CAS port.

See Also:

Constant Field Values

Constructor Detail

ContentAcquisitionServerComponent

public ContentAcquisitionServerComponent()

Constructor for the Config Manager custom component.

Method Detail

cleanOutputDir

public void cleanOutputDir(String crawlName)
                    throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException,
                           com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                           com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                           com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

Cleans the crawl's output directory (specified in the crawl configuration, usually through the CAS Console), removing any files in that directory that match the naming convention used by this component. Both incremental and full crawl files (or, or specifically, files that match the naming pattern of both full and incremental crawl files) are removed from the crawl output directory. For details, refer to the clearCrawlOutputFromDir method.

Parameters:

crawlName - Crawl whose output directory should be cleaned.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

setDefinition

public void setDefinition()
                   throws com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

Overrides:

setDefinition in class com.endeca.soleng.eac.toolkit.component.CustomComponent

Throws:

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

isCrawlDefined

public boolean isCrawlDefined(String crawlName)
                       throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Requests a list of crawls defined in the CAS and returns true if one of them matches the specified crawl name. This method should not be used to ensure that crawl execution doesn't fail, as even if this method returns true, the crawl may be removed by another process before execution begins.

Parameters:

crawlName - Crawl to test.

Returns:

True if the crawl is defined.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

listCasCrawls

public List<String> listCasCrawls()
                           throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Returns a list of the crawls defined in the CAS.

Returns:

List of the crawls defined in the CAS.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

runBaselineCasCrawl

public void runBaselineCasCrawl(String crawlName)
                         throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException,
                                com.endeca.soleng.eac.toolkit.exception.CasControlException

Runs to completion a full crawl of the CAS crawl with the specified name.

Parameters:

crawlName - Name of the crawl to run.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

com.endeca.soleng.eac.toolkit.exception.CasControlException - Thrown if the crawl does not complete successfully.

runIncrementalCasCrawl

public void runIncrementalCasCrawl(String crawlName)
                            throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException,
                                   com.endeca.soleng.eac.toolkit.exception.CasControlException

Runs to completion an incremental crawl of the CAS crawl with the specified name.

Parameters:

crawlName - Name of the crawl to run.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

com.endeca.soleng.eac.toolkit.exception.CasControlException - CasControlException Thrown if the crawl does not complete successfully.

waitForFinished

public boolean waitForFinished(String crawlName)
                        throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Waits for the specified crawl to finish (to be in a non-active state) or for the maximum amount of wait time to elapse. While waiting, the method will sleep its thread, swallowing any InterruptedExceptions that are caught.

Parameters:

crawlName - Name of the crawl to poll for status.

Returns:

Returns true if the crawl completes or false if the maximum time elapses before the crawl finishes.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

startBaselineCasCrawl

public void startBaselineCasCrawl(String crawlName)
                           throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Executes a baseline crawl for the specified crawl name. If the crawl is not already provisioned, or if the CAS can't be reached, this method throws an exception.

Parameters:

crawlName - Name of the crawl to execute.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

startIncrementalCasCrawl

public void startIncrementalCasCrawl(String crawlName)
                              throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Executes an incremental crawl for the specified crawl name. If the crawl is not already provisioned, or if the CAS can't be reached, this method throws an exception.

Parameters:

crawlName - Name of the crawl to execute.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

getCrawlStatus

public com.endeca.cas.wsdl.CrawlerState getCrawlStatus(String crawlName)
                                                throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Retrieves the status of the specified crawl.

Parameters:

crawlName - Name of the crawl to query.

Returns:

Returns the state of the crawl.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

isCrawlActive

public boolean isCrawlActive(String crawlName)
                      throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Test whether the crawl is active (running or stopping).

Parameters:

crawlName - Crawl to test.

Returns:

Returns true if the crawl is active (running or stopping).

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

isCrawlFileOutput

public boolean isCrawlFileOutput(String crawlName)
                          throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Returns true iff the crawl outputs Endeca record files.

Parameters:

crawlName -

Returns:

true iff the crawl outputs Endeca record files.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

isCrawlMdexOutput

public boolean isCrawlMdexOutput(String crawlName)
                          throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Returns true iff the crawl outputs MDEX configuration files.

Parameters:

crawlName -

Returns:

true iff the crawl outputs MDEX configuration files.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

isCrawlRunning

public boolean isCrawlRunning(String crawlName)
                       throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Test whether the crawl is running.

Parameters:

crawlName - Crawl to test.

Returns:

Returns true if the crawl is running.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

isCrawlStopping

public boolean isCrawlStopping(String crawlName)
                        throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Test whether the crawl is stopping.

Parameters:

crawlName - Crawl to test.

Returns:

Returns true if the crawl is stopping.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

isCrawlStopped

public boolean isCrawlStopped(String crawlName)
                       throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Test whether the crawl is stopped.

Parameters:

crawlName - Crawl to test.

Returns:

Returns true if the crawl is stopped.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

clearIncrementalCrawlOutputFromDestinationDir

public void clearIncrementalCrawlOutputFromDestinationDir(String crawlName)
                                                   throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException,
                                                          com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                                                          com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                                                          com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

Convenience method to invoke clearCrawlOutputFromDestinationDir method with the isIncrementalCrawl argument set to true.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

clearFullCrawlOutputFromDestinationDir

public void clearFullCrawlOutputFromDestinationDir(String crawlName)
                                            throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException,
                                                   com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                                                   com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                                                   com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

Convenience method to invoke clearCrawlOutputFromDestinationDir method with the isIncrementalCrawl argument set to false.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

clearCrawlOutputFromDir

public void clearCrawlOutputFromDir(String crawlName,
                           boolean isIncrementalCrawl,
                           String hostId,
                           String targetDir)
                             throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException,
                                    com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                                    com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                                    com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

Deletes files associated with the specified crawl from the specified directory. This method expects files to follow the naming convention created by this object's renameCrawlOutput method. Specifically, when deleting incremental crawl files, the method searches for files that begin with "<crawlName>_" and contain "<crawlPrefix>-INCR-sgmt". And, when deleting full crawl files, the method searches for files in the that begin with "<crawlName>_" and contain "<crawlPrefix>-FULL-sgmt". All files that match the pattern are deleted from the file system.

Parameters:

crawlName - Name of the crawl whose output files are to be removed.

isIncrementalCrawl - True if files are to be deleted for an incremental crawl.

hostId - ID of host on which directory to be cleaned resides.

targetDir - Directory from which target files should be removed.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

copyIncrementalCrawlOutputToDestinationDir

public void copyIncrementalCrawlOutputToDestinationDir(String crawlName)
                                                throws com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                                                       com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                                                       com.endeca.soleng.eac.toolkit.exception.EacComponentControlException,
                                                       com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Convenience method to invoke copyCrawlOutputToDestinationDir method with the isIncrementalCrawl argument set to true.

Throws:

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

copyBaselineCrawlOutputToDestinationDir

public void copyBaselineCrawlOutputToDestinationDir(String crawlName)
                                             throws com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                                                    com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                                                    com.endeca.soleng.eac.toolkit.exception.EacComponentControlException,
                                                    com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Convenience method to invoke copyCrawlOutputToDestinationDir method with the isIncrementalCrawl argument set to false.

Throws:

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

copyCrawlOutputToDestinationDir

public void copyCrawlOutputToDestinationDir(String crawlName,
                                   boolean isIncrementalCrawl)
                                     throws com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                                            com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                                            com.endeca.soleng.eac.toolkit.exception.EacComponentControlException,
                                            com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

This method is expected to be used in conjunction with the renameCrawlOutput method, as it assumes the naming convention created by that method. This method identifies crawl output files in the crawl output directory with the naming convention created by the renameCrawlOutput method. Once identified, files are copied to the configured destination directory. For an incremental crawl, the file is copied into the "incremental" subdirectory of the destination dir. For a full crawl, the file is copied into the "full" subdirectory of the destination dir.

Parameters:

crawlName - Name of the crawl whose output is to be copied.

isIncrementalCrawl - True if the output is that of an incremental crawl.

Throws:

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

renameIncrementalCrawlOutput

public void renameIncrementalCrawlOutput(String crawlName)
                                  throws com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                                         com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                                         com.endeca.soleng.eac.toolkit.exception.EacComponentControlException,
                                         com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Convenience method to invoke renameCrawlOutput method with the isIncrementalCrawl argument set to true.

Throws:

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

renameBaselineCrawlOutput

public void renameBaselineCrawlOutput(String crawlName)
                               throws com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                                      com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                                      com.endeca.soleng.eac.toolkit.exception.EacComponentControlException,
                                      com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Convenience method to invoke renameCrawlOutput method with the isIncrementalCrawl argument set to false.

Throws:

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

renameCrawlOutput

public void renameCrawlOutput(String crawlName,
                     boolean isIncrementalCrawl)
                       throws com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                              com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                              com.endeca.soleng.eac.toolkit.exception.EacComponentControlException,
                              com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Renames the crawl output file for a specified crawl to include the crawl name as a prefix as well as a timestamp if it's an incremental crawl. The method starts by identifying the crawl output file by looking for <crawlPrefix>-FULL-sgmt*<crawlSuffix> or <crawlPrefix>-INCR-sgmt*<crawlSuffix>, where the prefix and suffix are configured in the CAS as part of the crawl definition. Once the file is identified, the file is renamed to include the crawl name as a prefix, allowing it to be uniquely identified when files are merged with the output of other crawls. For incremental crawls, a timestamp is inserted after the crawl name in the form yyyy.MM.dd.HH.mm.ss. The renamed form of the file will be: <crawlName> ;_<crawlPrefix>-FULL[-sgmt<num>]<crawlSuffix> or <crawlName >_<timestamp>_<crawlPrefix>-INCR[-sgmt<num& gt;]<crawlSuffix> See the description of the getCrawlOutputSuffix method on this object for details about how the suffix is identified.

Parameters:

crawlName - Name of the crawl whose output is to be renamed.

isIncrementalCrawl - True if an incremental crawl's output is to be renamed.

Throws:

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

getCrawlOutputDir

public String getCrawlOutputDir(String crawlName)
                         throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Queries the CAS to retrieve the crawl output directory from the crawl config.

Parameters:

crawlName - Name of the crawl to be queried.

Returns:

Returns the String representation of the crawl output dir.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

getCrawlOutputPrefix

public String getCrawlOutputPrefix(String crawlName)
                            throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Queries the CAS to retrieve the crawl output file prefix from the crawl config.

Parameters:

crawlName - Name of the crawl to be queried.

Returns:

Returns the String representation of the crawl output file prefix.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

getCrawlOutputSuffix

public String getCrawlOutputSuffix(String crawlName)
                            throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Queries the CAS to retrieve the crawl output file suffix from the crawl config. If the crawl is configured to output XML, the suffix is ".xml"; otherwise, it's assumed to be ".bin". If the crawl output is configured to be compressed, ".gz" is appended to the suffix. So, four options might be returned: - .bin - .xml - .bin.gz - .xml.gz

Parameters:

crawlName - Name of the crawl to be queried.

Returns:

Returns the String representation of the crawl output file suffix.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

copyOutputDimensionsFile

public void copyOutputDimensionsFile(String crawlName,
                            String destDirectory)
                              throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException,
                                     com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                                     com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                                     com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

Copy (prefix).dimensions.xml file generated by the output of an MDEX-outputter crawl to a temporary directory for subsequent intake by Workbench.

Parameters:

crawlName - name of crawl which should have produced the file

destDirectory - directory to which to copy the file, relative to working directory. In AppConfig.xml, this is likely configured as webStudioTempDir in ConfigManager, and accessed via ConfigManager.getWsTempDir().

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

checkCasVersion

public void checkCasVersion()

This method checks the version of the CAS server (as reported by a service call to the CAS web service) against the expected version defined when this class was built. If the version (excluding build number) doesn't match, or if the service can't be reached, this method logs a warning. However, the method will not throw any exceptions and doesn't provide any information to be used automatically. It's intended to provide messaging/warning in the case that there is a version mismatch that may cause problems.

checkCrawlCompletedSuccessfully

public void checkCrawlCompletedSuccessfully(String crawlName)
                                     throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException,
                                            com.endeca.soleng.eac.toolkit.exception.CasControlException

Checks whether or not the given crawl has completed successfully. A CasControlException is thrown if the crawl did not complete successfully.

Parameters:

crawlName - Name of the crawl to be queried.

Throws:

com.endeca.soleng.eac.toolkit.exception.CasControlException - Thrown if the crawl has not completed successfully.

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

getCasPort

public String getCasPort()

getCasHost

public String getCasHost()

getCasHttpSocketTimeout

public int getCasHttpSocketTimeout()

getCasCrawlFullOutputDestDir

public String getCasCrawlFullOutputDestDir()

getCasCrawlIncrementalOutputDestDir

public String getCasCrawlIncrementalOutputDestDir()

getCasCrawlOutputDestHost

public String getCasCrawlOutputDestHost()

getLongPropertyValueOrDefault

public long getLongPropertyValueOrDefault(String propName,
                                 long defaultValue)

getMaxMissedStatusChecksAllowed

public long getMaxMissedStatusChecksAllowed()

getStandardPollingIntervalMs

public long getStandardPollingIntervalMs()

getSlowPollingIntervalMs

public long getSlowPollingIntervalMs()

getMinWaitTimeSeconds

public long getMinWaitTimeSeconds()

getMaxWaitTimeSeconds

public long getMaxWaitTimeSeconds()

getNumMissedStatusChecks

public int getNumMissedStatusChecks()

setNumMissedStatusChecks

public void setNumMissedStatusChecks(int numMissedStatusQueries)

incrementMissedStatusQueries

public void incrementMissedStatusQueries()

getNumDvalIdMappingsBackups

public int getNumDvalIdMappingsBackups()

Returns the number of backups of the DvalIdManager mappings to keep. This is configured via the numDvalIdMappingsBackups configuration property. If this property is not configured, then five backups will be retained. If set to zero, then no backing up will be performed. This should only be used when running crawls configured to write MDEX output.

Returns:

The number of backups of the DvalIdManager mappings to keep

getDvalIdMappingsArchiveDir

public String getDvalIdMappingsArchiveDir()

Returns the directory on the CAS host where DvalIdManager mappings should be written. This allows DvalIdManager exports to be imported in the event of a failure. This is configured via the dvalIdMappingsArchiveDir configuration property. If this property is not configured, then mappings will be written to ./data/dvalid_mappings_archive. However, to provide protection against failure, this should be configured to point to a network drive accessible to but separate from the CAS server. This should only be used when running crawls configured to write MDEX output.

Returns:

the directory on the CAS host where DvalIdManager mappings should be written

archiveDvalIdMappingsForCrawlIfChanged

public void archiveDvalIdMappingsForCrawlIfChanged(String crawlName)
                                            throws com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                                                   com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                                                   com.endeca.soleng.eac.toolkit.exception.EacComponentControlException,
                                                   InterruptedException,
                                                   com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Backs up dimension value Id mappings from a Dimension Value Id Manager to a CSV file at the path specified on the CAS Server. This will keep a configurable number of the most recent backups. It will make a timestamped copy of the existing directory (if any) as a sibling. If that results in more than the number of backups to keep, the oldest backup directory is deleted. It then creates the directory, if necessary This should only be used when running crawls configured to write MDEX output. Information about the location of the DValIdManager to back up will be take from the configuration of a particular crawl. This method would typically be called after running that particular crawl, which may have introduced new mappings to the DvalIdManager. The method writes out a new "dvalIdMappings.csv" file to the directory. It also writes out an empty file with an all-numeric filename and an extension of ".vsn". The name of this file represents the "version" of the mappings stored in that directory. This file is used to determine if mappings have changed, and therefore if a new archive needs to be written. If either (or both) of these files are missing, then the mappings are considered changed and an archive is performed.

Parameters:

crawlName - Name of the crawl that is configured to use a Dimension Value ID Manager.

Throws:

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

InterruptedException

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

See Also:

getDvalIdMappingsArchiveDir(), getNumDvalIdMappingsBackups()

importDimensionValueIdMappings

public void importDimensionValueIdMappings(String dimensionValueIdManagerInstanceName,
                                  String importFilePath)
                                    throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Imports dimension value Id mappings from a CSV file located at the specified path to a specified Dimension Value Id Manager. The file must be accessible to the CAS server. The imported mappings replace any existing mappings.

Parameters:

dimensionValueIdManagerInstanceName - the name of the DimensionValueIdManager

importFilePath - path to the file that contains the mappings to import

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

exportDimensionValueIdMappings

public void exportDimensionValueIdMappings(String dimensionValueIdManagerInstanceName,
                                  String exportFilePath)
                                    throws com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Exports dimension value Id mappings from a Dimension Value Id Manager to a CSV file at the path specified on the CAS Server.

Parameters:

dimensionValueIdManagerInstanceName - the name of the DimensionValueIdManager

exportFilePath - path to the file that mappings will be exported to

Throws:

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

getCumulativePartialsDir

public String getCumulativePartialsDir()

Returns the directory on the CAS host where partial MDEX output should be accumulated. This allows partial updates to be reapplied in the event of a failure while applying partial updates.

Returns:

the directory on the CAS host where partial MDEX output should be accumulated

cleanCumulativePartials

public void cleanCumulativePartials()
                             throws com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                                    com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                                    com.endeca.soleng.eac.toolkit.exception.EacComponentControlException,
                                    InterruptedException

Removes all accumulated partials files from the cumulative partials directory. This should only be used when running crawls with MDEX output.

Throws:

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

InterruptedException

See Also:

getCumulativePartialsDir()

fetchPartialsToCumulativeDir

public void fetchPartialsToCumulativeDir(String crawlName)
                                  throws com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                                         com.endeca.soleng.eac.toolkit.exception.EacComponentControlException,
                                         InterruptedException,
                                         com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                                         com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

Copies the partial MDEX output for the given crawl to the cumulative partials directory.

Parameters:

crawlName -

Throws:

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

InterruptedException

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.CasCommunicationException

See Also:

getCumulativePartialsDir()

archiveCumulativePartials

public void archiveCumulativePartials()
                               throws com.endeca.soleng.eac.toolkit.exception.AppConfigurationException,
                                      com.endeca.soleng.eac.toolkit.exception.EacCommunicationException,
                                      com.endeca.soleng.eac.toolkit.exception.EacComponentControlException,
                                      InterruptedException

Backs up the cumulative partials directory. This will keep a configurable number of the most recent backups. This should only be used when running crawls configured to write MDEX output.

Throws:

com.endeca.soleng.eac.toolkit.exception.AppConfigurationException

com.endeca.soleng.eac.toolkit.exception.EacCommunicationException

com.endeca.soleng.eac.toolkit.exception.EacComponentControlException

InterruptedException

See Also:

getCumulativePartialsDir(), getNumPartialsBackups()

getNumPartialsBackups

public int getNumPartialsBackups()

Returns the number of backups of the cumulative partials directory to keep. This is configured via the numPartialsBackups configuration property. If this property is not configured, then no backups will be retained. This should only be used when running crawls configured to write MDEX output.

Returns:

The number of backups of the cumulative partials directory to keep