Data domain operations

These commands operate on Endeca data domains. For example, they allow you to create data domains and return information about their status.

You can use the data domain operations only after you have used endeca-cmd to configure Endeca Server node profile and data domain profiles.

The commands for managing data domains are:

create-dd

The create-dd command creates, registers, and starts a new data domain with the specified name, using the configuration settings from either the default data domain profile or the specified data domain profile.

The syntax for this command is: 
endeca-cmd create-dd <new-data-domain> [global-options] [create-options]
where new-data-domain is mandatory and is the name of the new Endeca data domain. The name follows these rules:
  • The name must be unique among any other Endeca data domains in this Endeca Server instance (or Endeca Server cluster).
  • The name cannot contain these characters: & (ampersand), | (pipe), ; (semicolon), \ (back slash), / (forward slash).
  • The name cannot start with "." (a period).
  • The name must be enclosed in double quotes if it contains spaces. Note that after being created, the name must be referenced within double quotes in subsequent commands.

The index for the data domain is created in the $DOMAIN_HOME/EndecaServer/data directory by default. You can change this location by specifying a different directory for the endeca-data-dir parameter in the Endeca Server configuration file EndecaServer.properties. Inside this index, an internal .profile file is created. This file is then used when you export the data domain (by exporting its index). This file is also used as the new data domain's profile, when you create a new data domain by importing a previously exported index. If you update a data domain profile, Endeca Server automatically updates the .profile in the data domain's index.

Note: You cannot use create-dd command to restore a previously created data domain based on its index. Use import-dd and export-dd for this purpose.
The following additional command options can be used to change the created data domain's configuration (defaults are used otherwise):
Create Option Description
--dd-profile-name name Specifies the name of the data domain profile name to be used. If not specified, defaults to using the default data domain profile.
--is-enabled boolean If set to true, indicates that the new data domain is enabled. If this option is not specified, the data domain is enabled.

If you want to create and register but not start the data domain, specify --is-enabled false.

Example 1: 
endeca-cmd create-dd MyDD
creates an Endeca data domain named MyDD using the default data domain profile. The data domain will be enabled.
Example 2: 
endeca-cmd create-dd MyDD2 --dd-profile-name MyProfile --is-enabled false
creates a MyDD2 data domain using the data domain profile named MyProfile. The data domain is created and registered, but not started.

clone-dd

The clone-dd command creates a data domain based on the existing data domain. The syntax for this command is: 
endeca-cmd clone-dd <cloned-domain> --source-name <source-domain> [global-options] [clone-option]

where cloned-domain is the unique name of the new Endeca data domain, as copied from source-domain. The source data domain must be enabled before you run this command. The new data domain name follows the same naming rules as the create-dd command.

The --is-enabled option, if set to true, indicates that the new data domain is enabled. If this option is not specified, the data domain is enabled. If you want to clone but not start the data domain, specify --is-enabled false.

This example: 
endeca-cmd clone-dd MyDD3 --source-name MyDDMaster --is-enabled false
creates a MyDD3 data domain by cloning the MyDDMaster data domain. The data domain is created and registered, but not started.

delete-dd

The delete-dd command deletes a data domain with the specified name. This de-registers the data domain from the Endeca Server cluster, shuts down the Dgraph nodes serving this data domain, and deletes the index files for this data domain (including the .profile file for this data domain).

The syntax for this command is: 
endeca-cmd delete-dd <data-domain> [global-options]

disable-dd

The disable-dd command stops the specified enabled data domain but continues to allocate resources to the Dgraph nodes.

The syntax for this command is: 
endeca-cmd disable-dd <data-domain> [global-options]
where data-domain is the name of the disabled data domain that will be stopped.

The data domain must be enabled before it can answer queries.

enable-dd

The enable-dd command starts the specified disabled data domain. The syntax for this command is: 
endeca-cmd enable-dd <data-domain> [global-options]
where data-domain is the name of the disabled data domain that will be started.

export-dd

The export-dd command exports the index of the specified data domain by taking a snapshot of the index files and copying them into the offline directory under another name. A snapshot represents a copy of the index files only, and does not capture any other characteristics of the data domain.

The syntax for this command is: 
endeca-cmd export-dd <data-domain> [--offline-name <exported-domain>] [global-options]
where data-domain is the name of the existing Endeca data domain to be exported.
The --offline-name option specifies the name to give to the exported data domain. This name must be unique. If this option is not specified, the exported name is assigned automatically, by appending the date to the original data domain name, using this format: 
name_MMMMM-dd-yyyy-hh-mm

The command returns the resulting name used for the exported index.

Important: Keep track of the value of --export-name, because you will need it later for importing this index.

The location of the offline directory is specified by the endeca-offline-dir parameter in the Endeca Server configuration file. By default, this is the $DOMAIN_HOME/EndecaServer/offline directory.

This example: 
endeca-cmd export-dd MyDD --offline-name MyDD_offline
exports the MyDD data domain to the offline directory under the name MyDD_offline.

import-dd

The import-dd command creates a new data domain with the specified name using previously exported index files. The syntax for this command is: 
endeca-cmd import-dd <new-data-domain> --offline-name <exported-domain> [global-options] [import-options]
where new-data-domain is the name of the new data domain that will be created from the exported data domain specified by the --offline-name flag.

Using this operation assumes that you have previously created a data domain whose index is currently exported. It also assumes that you kept track of the --export-name you used in the export-dd command, or know the name that was automatically assigned.

The command has these options:
Import Option Description
--dd-profile-name (Optional). The name of the data domain profile to use. Here is what happens if it is not specified:

If you are importing a data domain that was exported from an Endeca Server before version 7.6.1.13, and do not specify a data domain profile, or if it does not exist in your system, Endeca Server uses the default data domain profile when creating a new data domain. If you are importing an index from the data domain created in Endeca Server 7.6.1.13 or greater, then you do not need to specify the data domain profile because the system automatically uses the .profile stored in the imported index.

--is-enabled If set to true, indicates that the new data domain is enabled after it is imported. If this option is not specified, the imported data domain is enabled (this is the default). If you want to import but not start the data domain, specify --is-enabled false.
This example: 
endeca-cmd import-dd NewDD --offline-name MyDD --dd-profile-name MyDDProfile --is-enabled false
imports the index MyDD from the offline directory into the new data domain NewDD, which is created with the MyDDProfile data domain profile and is not enabled.

get-dd

The get-dd command returns the characteristics of the specified data domain.

The syntax for this command is: 
endeca-cmd get-dd <data-domain> [global-options]
where data-domain is the name of the data domain for which to return information.

The returned information includes whether the data domain is enabled, the number of follower nodes, the number of query processing threads, and the list of arguments sent to the Dgraph processes for this data domain.

get-dd-health

The get-dd-health command returns information about a data domain's health in the Endeca Server cluster, with the specified name.

The syntax for this command is: 
endeca-cmd get-dd-health <data-domain> [global-options]
where data-domain is the name of the data domain for which to return health information. The data domain can be enabled or disabled.

The returned information lists the status of the Dgraph nodes, including the leader node and follower nodes.

get-dd-status

The get-dd-status command returns runtime statistics about the specified data domain.

The syntax for this command is: 
endeca-cmd get-dd-status <data-domain> [global-options]
where data-domain is the name of the data domain for which to return statistics. The data domain must be enabled for statistics to be returned.
The response includes the following information:
  • The size of the index records (in MB)
  • The number of source records in the data domain (this number excludes non-data, or system records).
  • The Dgraph node statistics for each running Dgraph node in the data domain, including the Dgraph startup time, last index creation time, and path. (This information is intended only for Oracle Endeca Support.)

rescale-dd

The rescale-dd command adds a specified number of follower Dgraph nodes to the data domain and starts these nodes.

The syntax for this command is: 
endeca-cmd rescale-dd <data-domain> [--num-followers <integer>] [global-options]
where data-domain is the name of the existing Endeca data domain to export.

The --num-followers option specifies the number of nodes to add. This name must be unique. If this option is not specified, the number defaults to 1. Note that the specified number cannot exceed the configured maximum allowable number of over-subscribed nodes.

This example: 
endeca-cmd rescale-dd MyDD --num-followers 4
adds four follower Dgraph notes to the MyDD data domain and starts them.

allocate-bulk-load-port

The allocate-bulk-load-port command returns a host name for the leader node and the port used for Bulk Load Interface, for a specified data domain. The syntax for this command is: 
endeca-cmd allocate-bulk-load-port <data-domain> [global-options]

This is a read-write operation — if the current leader node is available, it verifies the current Dgraph leader node and reports it along with the port used for Bulk Load. If the current leader node is not available, it appoints a new leader node and a new bulk load port and reports them.

Note: If a data domain is idle, issuing this command activates the data domain (by restarting its Dgraph process).

list-dd

The list-dd command lists all existing data domains and shows their status (enabled or disabled).

The syntax for this command is: 
endeca-cmd list-dd [--verbose] [global-options]

The --verbose option displays additional status information for each data domain: the name, description, the number of nodes, the number of query processing threads, and the list of arguments sent to the Dgraph processes for this data domain.

update-dd

The update-dd command updates the specified data domain with a new data domain profile. You can run this command only on a disabled data domain. The data domain remains disabled after running this command, and you must enable it.

The syntax for this command is: 
endeca-cmd update-dd <data-domain> --dd-profile-name <profile-name> [global-options]

The --dd-profile-name option is required. It specifies the name of the data domain profile you want to use for this data domain. Thus before using this command, ensure that the data domain profile you want to use is already created.

This command changes all parameters of the data domain that are specified in the data domain profile, and automatically reallocates the data domain on the nodes in the Endeca Server cluster.

warm-cache-dd

The warm-cache-dd command warms the underlying Dgraph process cache for the specified data domain.

The syntax for this command is: 
endeca-cmd warm-cache-dd <data-domain> [--time-limit-sec <sec>] [global-options]

The --time-limit-sec parameter is optional. It specifies the time limit during which you would like the warming script to run. If you do not specify the timeout, the default value of 1800 seconds (30 minutes) is used.

This command causes the Dgraph process to warm up its cache without the need to create a custom warm-up script. The command takes into account current Dgraph process usage pattern for selecting the set of queries for replay for a specified period of time. It allows the Dgraph to reuse its computation results across queries, and thus helps reduce the user-observable latencies in query processing and improves performance.

version

The version command lists the version of the Oracle Endeca Server and the version of the Dgraph process powering the data domains (if the Dgraph processes are currently running).

The syntax for this command is: 
endeca-cmd version [global-options]

get-logs

The get-logs command obtains WebLogic, Dgraph, and Cluster Coordinator (ZooKeeper) log files and compresses them to a single zip file. A configuration file specifies the settings needed by the command, such as the destination path for the zip file and whether the installation is a single-node or clustered environment.

Windows prerequisites:
  • PowerShell 2.0 (or later) must be installed.
  • The Java SDK jar command must be a recognized CMD command'
  • In a clustered environment, a credential that can be used on all nodes is required. The get-logs will ask for a the Remote Server username and password once. The input will be used as the credential for all nodes.

There are no special prerequisites for Linux.

Command syntax: The get-logs syntax is: 
endeca-cmd get-logs <conf-file>
where conf-file is the name (including the path) of the configuration file for harvesting the logs.

Configuration file: The $ENDECA_HOME/endeca-cmd/log-harvest directory contains two versions of the get-logs configuration file: getLogs_linux.prop for Linux and getLogs_windows.prop for Windows.

Property Platform Meaning
CLUSTER_MODE Both Whether this is a cluster deployment:
  • If set to TRUE, it is a cluster deployment.
  • If set to FALSE, it is a single-node deployment.
ORACLE_HOME Both The path to the WebLogic root directory. Examples: 
// Windows:
C:\WebLogic\Oracle\Middleware

// Linux:
/localdisk/WebLogic/Oracle/Middleware
ENDECA_SERVER_DOMAIN_NAME Both The name of the WebLogic domain for Endeca Server. Example: 
ENDECA_SERVER_DOMAIN_NAME=endeca_domain
DEST_PATH_SINGLE Windows For single-node environments, specifies the name of the folder where the log packages will be stored. Example: 
C:\tmp

This property is ignored if CLUSTER_MODE=TRUE.

TARGET_SERVERS Both For cluster environments, specifies a comma-separated list of servers to get logs from. The servers must be specified by short names. For example, if the server names are web01.example.com, web02.example.com, and web03.example.com, you use: 
TARGET_SERVERS=web01,web02,web03

This property is ignored if CLUSTER_MODE=FALSE, in which case get-logs will run only on localhost.

DEST_PATH Linux The name of the directory where the log packages will be stored:
  • For cluster environments, the directory must be a shared directory.
  • For single-node environments, a local path can be used.
SHARE_FOLDER Windows For cluster environments, specifies the name of a directory shared by all the machines. For example: 
\\server\share

This property is ignored if CLUSTER_MODE=FALSE.

DRIVE_NAME Windows For cluster environments, specifies the drive mapped to the share folder (the SHARE_FOLDER property). For example: 
Z:

This property is ignored if CLUSTER_MODE=FALSE.

DEST_PATH_CLUSTER Windows For cluster environments, specifies the name of the folder where the log packages will be stored. This will be created under the share folder (the SHARE_FOLDER property). Example: 
endecaserver\log-harvest

This property is ignored if CLUSTER_MODE=FALSE.

Returned log files: The command produces a ZIP file containing the gathered logs. The package name is: 
ES7.7-log-package_<timestamp>.zip
where the format of timestamp is: YYYYMMDDHHMMSSSS
When unzipped on a single-node environment, two directories are extracted:
  • One directory contains the Dgraph logs and the Cluster Coordinator log.
  • The other directory contains the WebLogic logs, plus the EndecaServer.properties configuration file.

The cluster version will include a set of Dgraph and WebLogic logs for each node.

Note that unzipping any log package creates a directory named ES7.7-log-packages. the ES7.7-log-packages directory is used as a temp folder. Therefore, unzipping subsequent log package ZIPs may overwrite the ES7.7-log-packages directory, which should not matter as the original ZIP file contents are not changed.

Windows example of running get-logs on a single-node environment: The command line is: 
endeca-cmd get-logs .\log-harvest\getLogs_windows.prop
The command response is: 
[fcal-pc] Creating temp folder ES7.7-log-packages under C:\tmp......Success
[fcal-pc] Collecting WebLogic logs.......Success
[fcal-pc] Collecting Dgraph and ZooKeeper logs.......Success
[fcal-pc] Compressing all logs under C:\tmp\ES7.7-log-packages......Success

****************************************
Generated log package C:\tmp\ES7.7-log-package-2015111813383322.zip
****************************************