After running an incremental CAS crawl, you can run a partial update that incorporates the records from a Record Store instance.
To create this sequential workflow of incremental CAS crawl and then partial update, you can do the following:
Remove the default
PartialForge.isPartialDataReady
check from the partial update script. This call handles concurrency control around Forge input files. The Record Store has built-in logic to handle concurrency between read and write operations, so no external concurrency control is required. Removing this call means that the lock manager does not check the flag or wait on the flag to be cleared before running a CAS crawl.Add a call
runIncrementalCasCrawl()
to run the incremental CAS crawl.If the pipeline does not read from sources in the Forge incoming directory, remove the call to
PartialForge.getPartialIncomingData()
that fetches extracted data files.
For example, this partial update script calls
CAS.runIncrementalCasCrawl("MyCrawl")
which runs an
incremental CAS crawl named
MyCrawl
. Then the script continues with partial
update processing.
<!-- ######################################################################## # Partial update script # --> <script id="PartialUpdate"> <log-dir>./logs/provisioned_scripts</log-dir> <provisioned-script-command>./control/partial_update.bat</provisioned-script-command> <bean-shell-script> <![CDATA[ log.info("Starting partial update script."); // obtain lock if (LockManager.acquireLock("update_lock")) { // call the partial crawl script to run an incremental // CAS crawl. CAS.runIncrementalCasCrawl("MyCrawl"); // archive logs PartialForge.archiveLogDir(); // clean directories PartialForge.cleanDirs(); // fetch config files to forge input PartialForge.getConfig(); // run ITL PartialForge.run(); // timestamp partial, save to cumulative partials dir PartialForge.timestampPartials(); PartialForge.fetchPartialsToCumulativeDir(); // distribute partial update, update Dgraphs DgraphCluster.cleanLocalPartialsDirs(); DgraphCluster.copyPartialUpdateToDgraphServers(); DgraphCluster.applyPartialUpdates(); // archive partials PartialForge.archiveCumulativePartials(); // release lock LockManager.releaseLock("update_lock"); log.info("Partial update script finished."); } else { log.warning("Failed to obtain lock."); } ]]> </bean-shell-script> </script>
You run the partial update by running
partial_update
in the
apps/<app dir>/control
directory. For example:
C:\Endeca\apps\DocApp\control>partial_update.bat