You can create a data domain profile that automatically turns a
data domain to idle if it receives no queries for the specified timeout period.
Automatic idling of data domains is useful as it helps control the
proliferation of data domains in the Endeca Server deployment, thus improving
hardware utilization by the Endeca Server cluster. This topic discusses how
Endeca Server manages automatic idling of data domains.
A data domain profile contains these settings for idling of a data
domain:
- autoIdle
indicates whether to turn a data domain to idle after a timeout, by stopping
its Dgraph processes. If set to
false, Endeca Server never makes this data domain
idle. If set to
true, the data domain is made idle after the optional
timeout specified in the
idleTimeoutMinutes, if during this timeout the data
domain does not receive queries. The default for
autoIdle is
false.
- idleTimeoutMinutes
indicates the timeout to use for idling, in minutes. The timeout is counted
from the time a data domain has been created and enabled. This parameter is
optional. If not specified, the default of 10 minutes is used. To change the
idling timeout for a data domain, disable a data domain, create a new data
domain profile with new timeout, update the data domain to use the new data
domain profile, and enable the data domain.
A data domain is
idle when the Dgraph process is stopped, on all Endeca
Server nodes hosting this data domain. A data domain that is idle continues to
be registered with the Endeca Server, however the Endeca Server stops
allocating resources (memory and processing threads), to this data domain. An
idle data domain is activated automatically if it receives a query. In other
words, once an end user of a front-end application powered by the Endeca
Server, (such as Studio), issues a request, an idle data domain is activated,
and its Dgraph processes are restarted automatically. The following statements
describe which operations activate (or do not activate) an idle data domain:
- Queries from the
Conversation Web Service, Configuration Web Service, and Data Ingest Web
Service activate an idle data domain and restart the timeout timer.
- Queries from Administration
and Configuration URL commands also activate and restart the Dgraph processes
for the idle data domain (and restart the timer), with the exception of
http://<host>:<port>/endeca-server/admin/<data_domain>?op=ping,
which does not activate an idle data domain.
- Bulk load interaction. The
process of loading data through the Bulk Ingest interface starts with the
allocateBulkLoadPort operation from the Manage Web
Service, which activates an idle data domain. If a bulk ingest operation is
issued and is in progress for the data domain that is set to auto-idle, it is
possible that bulk load may take longer than the idling timeout. If the timeout
expires while bulk data load is in progress, the Endeca Server refreshes the
idling timeout and the data domain does not shut down, so that bulk load can
complete successfully.
- The
endeca-cmd commands do not activate an idle data
domain with the exception of
endeca-cmd allocate-bulk-load-port, which activates an
idle data domain.
Note: It is important to distinguish an "idle data domain" from a "data
domain that is configured to auto-idle": an idle data domain is the one that is
already idle (its processes are stopped); while a data domain that is
configured to auto-idle is a data domain that uses the data domain profile with
autoIdle set to
true. Such a data domain can be active but set to
auto-idle after a timeout, or, it can already idle be, if it received no
queries during the timeout.
Comparing with an ability to manually disable a data domain, if a data
domain is configured to auto-idle, stopping and restarting its Dgraph process
is handled automatically by the Endeca Server, without requiring you to call
these operations explicitly.
The following statements describe how an idle data domain behaves in
various situations:
- Interaction with
long-running operations in the Endeca Server: If an operation is long-running
in the data domain (such as, data is being loaded, or a cache warming operation
is running), the Endeca Server does not turn the data domain idle until the
operation finishes. The timeout for idling is restarted as the operation
starts, and is restarted as many times as needed for the operation to finish.
If the operation finishes, but the timeout does not expire, the data domain
remains active for the remaining duration of the timeout.
- If a data domain is set to
automatically idle, but is currently active, and you disable and then enable
it, it will be active.
- If a data domain is idle and
you disable it, it will be idle when you re-enable it, until a query arrives
that activates the data domain by restarting its Dgraph process.
- If a data domain is idle,
its idling timer is stopped. Once the data domain is activated, the clock for
the timeout is restarted.
- Updating a data domain to
use a new data domain profile does not interfere with idling, because updating
is done on a disabled data domain.
- Cloning, exporting, and
importing an idle data domain behaves in the same way as performing these
operations for a disabled data domain.
- If you idle a data domain,
and then restart the WebLogic Server hosting the Endeca Server, a data domain
will continue to be idle after the Endeca Server restarts.
Limitations for the idling of data domains are as follows:
- Auto-idling is guaranteed on
a data domain that has a single Dgraph process, and is ignored for data domains
deployed in a cluster. Specifically, if you create a data domain based on the
data domain profile with
autoIdle set to
true in a cluster, the data domain can be created
successfully, but the
autoIdle setting is ignored. Similarly, if you use a
specific data domain profile with
autoIdle set to
true for a data domain that has one Dgraph node, and
then rescale the data domain by adding nodes, the
autoIdle setting is ignored by the Endeca Server.
- Enabling a data domain and
activating an idle data domain operations succeed only if they meet all
resources constraints. In other words, if a data domain is idle and a query is
issued for it (which would normally activate it), but other data domains were
configured and enabled in the Endeca Server in the meantime, it is possible
that the Endeca Server may not have sufficient resources to activate this data
domain. In this case, the Endeca Server issues an error and the data domain
remains idle until you manually free up space in the Endeca Server cluster by
disabling other data domains. Alternatively, you can also add nodes to the
Endeca Server cluster to increase its capacity, and update data domains to
utilize it.
