====================================================================== Endeca Deployment Template GENERAL (3.2.2 Revision A) - Updated: March 14, 2012 - Revision A consists of an update to this release note file. The software has not changed since the 3.2.2 release. KNOWN ISSUES (3.2.2 Revision A) Documentation: - The instructions in the Installation Guide do not reflect the correct installation file name. For this release, the installation package is named according to the Oracle part number. ====================================================================== GENERAL (3.2.2) - Release Date: March 2012 - The release notes may have been updated since the release date. Contact Oracle Endeca Customer Support for the most recent version of the release notes. MIGRATION NOTES (3.2.2) - If you are upgrading from a version of the Deployment Template earlier than 3.2.1, refer to the MIGRATION NOTES (3.2.1) below. ====================================================================== GENERAL (3.2.1) - Release Date: November 2011 - The release notes may have been updated since the release date. Access the Product Downloads section of the Endeca Developer Network (https://eden.endeca.com) for the most recent version of the release notes. KNOWN ISSUES (3.2.1) - The documentation for this release has not been updated. Updated documentation may be provided in a future release if necessary. Please see the EDeN knowledge base for the most recent documentation. MIGRATION NOTES (3.2.1) - If you are upgrading to the 3.2.1 release of the Deployment Template, you must also upgrade your Platform Services to version 6.1.1. - The Deployment Template relies on a version of the Wget utility being installed in Platform Services. This executable is not provided with Platform Services 6.1.1. You must download and install it separately. For Windows installations, download and deploy a Windows implementation of GNU Wget 1.9.1 to: %ENDECA_ROOT%\bin\wget.exe For Linux and Solaris installations, download and deploy GNU Wget 1.9.1 to: $ENDECA_ROOT/bin/wget You can obtain the utility freely from the GNU project, or a Windows implementation such as MSYS, a component of MinGW (Minimalist GNU for Windows). ====================================================================== GENERAL (3.2.0) - Release Date: March 2010 - This release is compatible with IAP 6.1.x, MDEX 6.1.x, Workbench 2.1.x, and CAS Server 2.2.x. - The release notes may have been updated since the release date. Access the Product Downloads section of the Endeca Developer Network (https://eden.endeca.com) for the most recent version of the release notes. This release introduces the following bug fixes and enhancements: * ENH: LocalMoveUtility will now throw an error if you tell it to move a file or directory that doesn’t exist. * BUG1270: CleanDirUtility will now throw an error if it fails to re-create the cleaned directory. * ENH: CAS crawls will now throw an error if they fail to complete successfully. ====================================================================== GENERAL (3.1.0 Revision A) - Release Date: May 2009 - This release is compatible with IAP 6.0.x, MDEX 6.1.x, Workbench 2.0.x, and CAS Server 2.0.x. - The release notes may have been updated since the release date. Access the Product Downloads section of the Endeca Developer Network (https://eden.endeca.com) for the most recent version of the release notes. This release introduces the following documentation bug fixes: * BUG15819: The Deployment Template Usage Guide has been revised with better procedures in "Specifying a CAS Server host" and "Specifying a CAS Server as a custom component for Record Store output". * BUG15818: The Deployment Template Usage Guide has been revised with better procedures in "Editing a baseline CAS crawl script for a Record Store instance" and in "Editing an incremental CAS crawl script for a Record Store instance". ========================================================================= GENERAL (3.1.0) - Release Date: May 2009 - This release is compatible with IAP 6.0.x, MDEX 6.1.x, Workbench 2.0.x, and CAS Server 2.0.x. - The release notes may have been updated since the release date. Access the Product Downloads section of the Endeca Developer Network (https://eden.endeca.com) for the most recent version of the release notes. This release introduces the following bug fixes and enhancements: * ENH: The Deployment Template Usage Guide has been revised to describe working with CAS 2.0 and Record Store instances. * ENH: Upgraded CAS integration to support CAS 2.0. * ENH: Added new "updateGroup" property for DgraphComponent. Partial updates will respect this new property, instead of the "restartGroup" property, when applying updates. Index updates, configuration updates, provisioning updates, and xquery updates will continue to respect the "restartGroup" property. This allows users to, for example, apply partial updates to all dgraphs simultaneously (because MDEX 6.1 supports query processing while applying updates), while applying other updates to one dgraph at a time (allowing a load balancer to distribute traffic while dgraphs are temporarily unavailable). * ENH: Improved parallelization of utility invocations during DgraphCluster operations. Now, the DgraphCluster's applyIndex, applyPartialUpdates, applyConfigUpdate, and reloadXqueryModules will execute operations in parallel for all dgraphs in the same group. For example, during a partial update, all dgraphs in the same update group will invoke the call to the admin?op=update administrative url simultaneously. * ENH: Previously, each server that hosted a Dgraph would maintain a copy of all partial updates applied since the last baseline update. This can improve recovery speed when restoring a failed dgraph or rolling back an unwanted update, but can cause update performance to degrade over time, as updates accumulate and all are applied to the Dgraph, instead of the new update. The new default behavior is to only maintain the full copy of cumulative updates on the ITL server (and just the latest update on each MDEX server), improving update performance. * ENH 1168: Added config/lib/xquery directory for users to store XQuery modules. Added new LoadXQueryModules script and supporting DgraphComponent and DgraphCluster methods to distribute XQuery modules to Dgraph servers and instruct Dgraphs to load the modules. * ENH 1175: Upgrade to Spring Framework 2.5.6. * ENH 1192: Added new "--version" argument for runcommand.bat (and .sh) script, which prints the version number of the Deployment Template's EAC Development Toolkit. * ENH 3436: Updated sample wine pipeline and sample CAS pipeline to remove the primary dimension and the manipulator that associated records with the primary dimension. These are no longer necessary, as MDEX 6.1 removed the requirement for the primary dimension. * ENH 3470: Removed support for the "warmupseconds" argument when invoking the admin?op=update administrative URL on the Dgraph, as this parameter is no longer supported by the MDEX Engine. The argument is now ignored and a warning is logged. * BUG 1157: baseline_update.bat now returns a non-zero status code when the script fails. This change removes the use of tee.pl, which used to route standard error and output to the console as well as to a file. Logging configuration (managed in logging.properties) provides equivalent functionality. * BUG 1159: Removed unused --unix argument from deploy.pl script. * BUG 1195: Resolved issue that prevented scripts from executing on Solaris. Scripts now resolve the absolute path of the deployed application on supported platforms when using common shells (e.g., bash). * BUG 1198: CleanDirUtility's default behavior includes an optimization to (quickly) test directory contents and skip cleanup if the directory is empty. Added ability to skip this test and always execute the cleanup. Added new "skipTestingForFilesDuringCleanup" property that can be set to "true" for components to have them skip the directory contents test when cleaning their directories. * BUG 1204: Absolute path resolution failed on Linux, when scripts were invoked in a new shell (e.g., sh runcommand.sh). Scripts now resolve the application's absolute path relative the runcommand.sh without requiring runcommand.sh to be on the user's PATH. * BUG 2127: BeanShell script interpreters are now set up prior to updating definitions, so that Agraph/Dgraph components with pre-shutdown and post-startup scripts can execute those scripts if they need to be restarted to pick up a provisioning update. * BUG 2278: A DgraphCluster configured with copyInParellel set to "true" could fail if multiple Dgraphs in the same cluster specify a different localIndexDir, as both copies to the same server are constructed with the same token (identifier). Tokens now include the name of the Dgraph, and utilities invoked with Utility.runInParallel will automatically attempt to avoid conflicts by appending a timestamp to avoid invoking utilities with the same identifier at the same time. * BUG 3004: load_partial_test_data.sh script updated to use "=" instead of "==" operator, allowing the script to run on Solaris. * BUG 3162: deploy.sh script updated to remove references to undefined "usage" function. * BUG 3573: Sample wine pipeline and sample CAS pipeline updated to use recommended wilcard configurations. Both pipelines now specify a value of 3 for DICTIONARY_MAX_NGRAM_LENGTH and MAX_NGRAM_LENGTH. * BUG 4199: Enhanced ForgeComponent.timestampPartials method to match a broader set of output file names when applying timestamps. Now, a timestamp is applied to any file in the Forge output directory that matches the following pattern: [dataPrefix](-partN)-sgmtX.records.xml(.gz). The method now supports a compression-enabled update adapter, Agraph deployments (with and without parallel Forge), as well as multiple file segments when output exceeds Forge's configurable maximum file size. * BUG 4342: Enhanced WgetUtility and its subclasses to support communication with a secure MDEX Engine by specifying client and CA certificates. DgraphComponent will now invoke administrative URLs (admin?op=update, config?op=update, admin?op=flush, and admin?op=reload-services) with the appropriate certificates when the Dgraph is enabled for SSL. ========================================================================= (3.0.0) * Updated for use with IAP 6.0. This release introduces the following enhancements: - Updated installer to ask the user if they want to integrate with IAP Workbench (formerly Web Studio) and, if so, what port to use. Updated default install_config.xml file to include settings to enable Workbench integration and specify the default Workbench port (8006). - ConfigManagerComponent includes new "webStudioEnabled" property. All interaction with IAP Workbench hinges on this property, taking no action if the property is not set to "true." For backward compatibility, applications that don't specify this setting have this property set to "true" by default. - Default BaselineUpdate and ConfigUpdate scripts interact with IAP Workbench only when Workbench integration is enabled. - initialize_services.[bat|sh] now calls a new InitialSetup script, rather than uploading configuration to IAP Workbench. If Workbench integration is enabled, the InitialSetup script uploads configuration as before. - runcommand.[bat|sh] no longer finds and specifies eac.ks and ca.ks Java keystores and truststores for use when communicating with an SSL-enabled EAC Central Server. These default keystores are no longer provided by Endeca. For information about enabling SSL and using Endeca's certificate utilities to generate custom SSL certificates, see the Endeca Security Guide. - Removed CrawlerComponent and references to EAC type CrawlerComponentType. This deprecated component has been removed from the EAC. - Updated report generation scripts to specify the directory from which Web Studio should read its reports. Reports are now delivered to a location specified in a "webstudio-report-dir" property on a host that uses the reserved Workbench host name "webstudio." - Updated DgraphComponent and DgraphUpdateUtility to discontinue support for the "offlineUpdate" property and the "&offline=[true|false]" parameter it used to create on the admin?op=update URL. MDEX 6.0 no longer supports online updates and returns an error if "offline=false" is specified, so the Deployment Template now logs a warning and ignores the unsupported behavior. KNOWN ISSUES: - BUG 1195: On Solaris, logic added in 2.2 to support path independence causes EAC provisioning to fail. This occurs because, on this OS, set_environment.sh does not resolve ENDECA_PROJECT_DIR to an absolute path. A simple workaround is to edit AppConfig.xml and set the element under to specify the absolute path to the deployed project directory, rather than the variable ${ENDECA_PROJECT_DIR}. This workaround effectively reverts the feature described in ENH 5616 below. - BUG 14341: On Linux and Solaris, provisioning may fail under rare circumstances when attempting to create custom directories. For example, when provisioning the "webstudio" host, attempting to create the "webstudio-report-dir" custom directory may return an error like "Directory webstudio-report-dir (/localdisk/apps/myApp/control/.././reports) could not be created." Users can work around this issue by editing the AppConfig.xml file and setting the element under to specify the absolute path to the deployed project directory, rather than the variable ${ENDECA_PROJECT_DIR}. ========================================================================= (2.3.0) * Migration Notes from 2.2 to 2.3: Due to the flexible nature of the DT and the opportunities for customization, specifying a universal migration path from DT 2.2 to DT 2.3 is impossible. It is the job of the DT user to know how they have customized their deployment so that the appropriate modifications can be retained. The updates in DT 2.3 include support for CAS 1.2. These updates have affected numerous files through renaming and as actual code changes. The following is a full list of the affected files and can act as a launching point for migration. Below, the left side of --> is the 2.2 filename and the right side is the 2.3 filename. The paths are defined relative to an unmodified application directory. Changed only: ./config/lib/java/casStubs.jar ./config/lib/java/eacComponents.jar ./config/crawl_script_templates/README Renamed only: ./data/complete_filesys_crawl_output/ --> ./data/complete_cas_crawl_output/ Renamed and changed: ./config/crawl_script_templates/@CRAWL_NAME@FsCrawlConfig --> ./config/crawl_script_templates/@CRAWL_NAME@CasCrawlConfig ./config/fs_crawl_pipeline/ --> ./config/cas_crawl_pipeline/ ./config/fs_crawl_pipeline/pipeline.epx --> ./config/cas_crawl_pipeline/pipeline.epx ./config/fs_crawl_pipeline/pipeline.lyt --> ./config/cas_crawl_pipeline/pipeline.lyt ./config/lib/perl/make_fs_crawl_scripts.pl --> ./config/lib/perl/make_cas_crawl_scripts.pl ./config/script/endecaFsCrawlConfig.xml --> ./config/script/endecaCasCrawlConfig.xml ./config/script/fetchFsCrawlDataConfig.xml --> ./config/script/fetchCasCrawlDataConfig.xml ./control/cas/*_fs_*.bat --> ./control/cas/*_cas_*.bat ./control/cas/*_fs_*.sh --> ./control/cas/*_cas_*.sh In your old ./config/script/, any instantiations of @CRAWL_NAME@FsCrawlConfig should be remade using ./control/cas/make_cas_crawl_scripts. * ENH: Updated to support CAS 1.2 servers. * BUG 1606: Updated to handle all CAS output file formats, including compressed and uncompressed XML. ========================================================================= (2.2.0) * ENH: Scripts, shell/batch wrappers, sample pipeline and documentation added to integrate (and document implementation practices for) file system crawls powered by the CAS. * ENH 5817: Enhanced deploy.pl installer to support custom token substitution. Users can now specify in app descriptor XML files the names of tokens to substitute when filter-copying files, and the question to prompt the user (or the option to parse in an install config file) to retrieve the value for that token. By default, the app descriptors for the dgraph and agraph deployments now use this functionality to prompt the user for agraph, dgraph, logserver and forge server ports. * ENH 5616: Deployed application enhanced to be "path independent," by locating deployed location relative to the invoking scripts. App can now be moved post-deployment and will re-provision itself to the new location. Custom log handler implemented to support relative file paths. * BUG 5358: Documented sample EndecaMDEXUpdateScript fixed to use isSslEnabled boolean argument when instantiating GetWSDgraphFilesUtility. * BUG 5314: Update default report_settings.xml to inclued A/B test report. * ENH 4879: Adding default java argument of -Xmx1G to all report generator components. * BUG 3875: Documentation updated to use isSslEnabled boolean argument when instantiating CopyUtility. * BUG 3755: AppConfig.xml now properly references "numStateBackups" for Forge component, instead of "numStateArchives." * BUG 3750: AppConfig.xml fixed to determine ".bat" or ".sh" script suffix for provisioned report generator scripts at deployment time. * EDT ENH: New custom component ContentAcquisitionServerComponent implemented to support CAS-based file system crawls. * EDT ENH: New property "numIdleSecondsAfterStop" allows configurable pause after a component stops, ensuring file locks are released before proceeding. * EDT BUG: Fixed init() method for RollbackUtility to set directory specified as init argument. * EDT ENH 5830: Component object now holds a reference to the Host object on which the component runs, providing easier access to hostname/port information. * EDT BUG 5829: ConfigManager.updateWsConfig method no longer filters inactive merch rules when retrieving Web Studio files. New methods downloadWsConfig and downloadWsDgraphConfig now accept a boolean arg to specify whether they should filter merch rules. No-arg methods continue to filter inactive merch rules when invoked. * EDT BUG 5822: ConfigManager.updateWsConfig method now uploads config files to Web Studio, even if no files are specified as being maintained by Web Studio (i.e. the method will upload dev studio-maintained files). * EDT ENH 5818: Introduced caching of application definition during provisioning check, to require fewer requests of the app definition from the EAC. * EDT ENH 5701: Components now allow a configurable number of failed status queries before assuming a component has failed. By default, 5 consecutive failed status queries are allowed before a component is assumed failed. * EDT ENH 5639: Upgraded Spring version used to 2.0.8 (latest 2.0.x. release). * EDT ENH 5560: Enhancing eacToolkit.xsd to allow ${token} values to be specified for boolean, int and long values. * EDT ENH 5513: Dgraphs now support "offlineUpdate" property (defaults to "true") to specify whether dgraph should remain online while processing partial updates. Also supports "numCacheWarmupSeconds" (incompatible with online updates) to specify number of seconds to spend warming cache before going back online. * EDT BUG 4939: Utility tokens (IDs) will be automatically truncated to ensure the token name and the associated log directory (i.e. ENDECA_CONF/logs/.../app.token.log) are shorter than 255 characters, to avoid Windows path length limitations and pre-5.1.3 EAC data store field length limits. * EDT ENH 4889: Components can now specify numLogBackups=-1 to clear the log directory, rather than archiving it. If numLogBackups=0, log dir is not archived, but most components will append, so the new -1 setting may be useful for dev environments, to save disk space. * EDT BUG 4282: Fixed SSL-argument handling for agraph, dgraph, forge to treat "cipher" is an optional argument. * EDT ENH 3974: Agraph and dgraph components allow scripts to be executed before the component is stopped and/or after the component is started. New "pre-shutdown-script" and "post-startup-script" attributes can specify the names of Scripts defined in AppConfig.xml. Component will invoke the BeanShell script before shutdown or after startup, allowing common actions like signaling a load balancer that the dgraph should be added or removed. BeanShell scripts used for this purpose have a new variable, "invokingObject," which references the dgraph or agraph that invoked the script, allowing the same script to be executed for multiple dgraphs. * EDT ENH 3957: ConfigManager's webStudioMaintainedFile now detects when names include data prefix, adding it only when not specified (e.g. wine.thesaurus.xml and thesaurus.xml can now be specified). * EDT ENH 3837: Added wildcard support for specifying webStudioMaintainedFiles in ConfigManager custom component. * EDT BUG 3821: Provisioning must always set/update hosts before components, as the EAC will error when components are added before their associated host is defined. * EDT ENH 3691: Enhanced logging to include host IDs in "starting" messages for all components and utilities (e.g. [02.24.08 21:10:25] INFO: [ITLHost] Starting component 'Dgidx'.) * ENH 3505: Updated javadoc to clarify that Host ID (not hostname) must be specified to FileUtility method invocations. * EDT ENH 2952: Server components will now poll for status stopped or failed after sending asynchronous "stop" command. New property "shutdownTimeout" to specify non-default time to wait. ========================================================================= (2.1.0) * Implemented EAC agraph deployment powered by Java EAC Development Toolkit, with support for baseline index updates and Web Studio integration. * ENH: Added support for agraph and parallel forge. * ENH: Implemented AgidxComponent. * ENH: Implemented AgraphComponent. * ENH: Implemented AgraphCluster. * ENH: Implemented ForgeCluster. * ENH: Implemented IndexingCluster. ========================================================================= (2.0.1) * BUG: Fixed a bug in the installer that caused it to stop copying if any copy pattern matched no files. * ENH: Added report generation scripts. * EDT BUG: LocalMoveUtility updated to strip trailing slash from source path. * EDT BUG: Updated Controller JVM args and Utility invocations to support SSL- enabled communication with the EAC. * EDT ENH: Added acquireLockBlocking and waitForFlagSet methods to allow blocking while waiting to acquire a lock. * EDT ENH: Added support for multiple --app-config files. * EDT ENH: Added support for --config-override properties files. * EDT ENH: Exposed all ForgeComponent methods as public. * EDT ENH: Implemented CrawlerComponent. * EDT ENH: Added support for an auto-executed BeanShell script to run for each BeanShell Interpreter, allowing common imports to be defined centrally and made available in all scripts. * EDT ENH: Implemented isPartialDataReady, getPartialIncomingData and timestampPartials methods on ForgeComponent, removing the need to shell out to Perl scripts for these methods. * EDT ENH: Enhanced LogServerComponent with methods for identifying logs by data range, rolling the log and delivering logs for a requested date range for report generation. * EDT ENH: Enhanced ReportGeneratorComponent with methods for updating RG provisioning to output files/charts named with a timestamp (to conform to Web Studio's naming convention) and for delivering output reports/charts to Web Studio. ========================================================================= (2.0.0) * Removed legacy Perl EAC and Control Interpreter deployments from the 2.0 release. * Implemented EAC dgraph deployment powered by Java EAC Development Toolkit, with support for baseline and partial index updates and Web Studio integration. ========================================================================= (1.8.0) * Added support for partial updates to Perl EAC dgraph deployment. * (BUG2688) Updated agraph control script to remove duplicate agraph and dgraph process/request logs. All logs are now written to logs/dgraphs/[CLUSTER]_[DGRAPH] and logs/agraphs/[CLUSTER]_[AGRAPH]. Initial setup and archiving bricks updated to match. * (BUG2613) Updated set_environment.[bat|sh] scripts to avoid setting irrelevant ENDECA_PROJECT_OVERRIDE environment variable for EAC deployments. ========================================================================= (1.7.0) * Updated EAC dgraph deployment to support changes in the EAC WSDL introduced in IAP 5.0.1. Refer to the Migration Guide (Chapter 5: Incremental Changes from 5.0.0 to 5.0.1) for details about the changes. * (BUG2526) Updated .sh/.bat wrapper scripts that execute baseline and partial updates to redirect both STDERR and STDOUT to screen and to a log file. * (BUG2454) Updated installer to replace indexer adapter's output prefix with the name of the installed project. This removes a warning when opening the pipeline in Dev Studio. ========================================================================= (1.6.0) * Added agraph deployment type, including support for parallel forge and baseline index updates. Sample pipelines are provided for both parallel and non-parallel forge deployments. * Updated EAC deployment to conform to IAP 5.0.1 component/utility status message formats (e.g. "NotRunning" instead of "NOT_RUNNING". ========================================================================= (1.5.0) * Added support for partial updates, including control script enhancements and test data. * Added support for file-based configuration of the deployment template installer. New install_config.xml file may be used instead of interactively configuring the installer at runtime. * Enhanced installer to use a config file to define the application being installed. External modules and additional features are installed by providing new/additional application descriptor files. The installer now also supports use for installing/deploying custom or project-specific deployments by allowing developers to write their own installer configuration, including directory structure and files to distribute. * Added EAC-powered deployment template for dgraph, supporting baseline index updates. * Added documentation to describe the procedure for installing a pipeline into a deployment template, including required changes to pipeline paths and file prefix names. * (ENH2193) Updated installer to validate application name against regex /^[a-zA-Z0-9]+$/, to avoid special characters that may cause problems for some processes. * (ENH1880) Updated installer to update filenames for the sample wine config files, changing the file prefix during installation to match the project name. * (ENH2191) Added config/lib directory with perl and java subdirectories. * (ENH2189) Added --logDir flag for forge, to redirect logs generated by forge components to the baseline logs directory. * (ENH2253) Added --inputDir flag for forge, to specify where forge should read data and dimensions. This will enable cleaner integration with Manager, as the --inputDir flag is always specified by Manager. * (BUG1945) Enhanced shell/batch wrapper scripts to reference Perl relative to the ENDECA_ROOT environment variable and to validate paths and executables before proceeding. ========================================================================= (1.1.0) * (ENH1829) Updated documentation to describe (1) configuring a Unix deployment to run SSL-enabled JCD, (2) updating the deployment control script to use SSL for communication with the JCD (especially when deploying on Windows, where SSL is enabled by default). * (ENH1836) The 4.8 Report Generator has been updated to use a report_settings.xml configuration file (replacing the old report_config.xml) and the control script brick has been updated to use a "settings" parameter (replacing the old "config" parameter). Updated the control script and deployment scripts to support use of the template on IAP 4.7 and 4.8. ========================================================================= (1.0.0) * Initial release