After running an incremental CAS crawl, you can run a partial update that incorporates the records from a Record Store instance.
To create this sequential workflow of incremental CAS crawl and then partial update, you can do the following:
Remove the default
PartialForge.isPartialDataReadycheck from the partial update script. This call handles concurrency control around Forge input files. The Record Store has built-in logic to handle concurrency between read and write operations, so no external concurrency control is required. Removing this call means that the lock manager does not check the flag or wait on the flag to be cleared before running a CAS crawl.Add a call
runIncrementalCasCrawl()to run the incremental CAS crawl.If the pipeline does not read from sources in the Forge incoming directory, remove the call to
PartialForge.getPartialIncomingData()that fetches extracted data files.
For example, this partial update script calls
CAS.runIncrementalCasCrawl("MyCrawl") which runs an
incremental CAS crawl named
MyCrawl. Then the script continues with partial
update processing.
<!--
########################################################################
# Partial update script
#
-->
<script id="PartialUpdate">
<log-dir>./logs/provisioned_scripts</log-dir>
<provisioned-script-command>./control/partial_update.bat</provisioned-script-command>
<bean-shell-script>
<![CDATA[
log.info("Starting partial update script.");
// obtain lock
if (LockManager.acquireLock("update_lock")) {
// call the partial crawl script to run an incremental
// CAS crawl.
CAS.runIncrementalCasCrawl("MyCrawl");
// archive logs
PartialForge.archiveLogDir();
// clean directories
PartialForge.cleanDirs();
// fetch config files to forge input
PartialForge.getConfig();
// run ITL
PartialForge.run();
// timestamp partial, save to cumulative partials dir
PartialForge.timestampPartials();
PartialForge.fetchPartialsToCumulativeDir();
// distribute partial update, update Dgraphs
DgraphCluster.cleanLocalPartialsDirs();
DgraphCluster.copyPartialUpdateToDgraphServers();
DgraphCluster.applyPartialUpdates();
// archive partials
PartialForge.archiveCumulativePartials();
// release lock
LockManager.releaseLock("update_lock");
log.info("Partial update script finished.");
} else {
log.warning("Failed to obtain lock.");
}
]]>
</bean-shell-script>
</script>
You run the partial update by running
partial_update in the
apps/<app dir>/control directory. For example:
C:\Endeca\apps\DocApp\control>partial_update.bat
