How idling affects data domain behavior

You can create a data domain profile that automatically turns a data domain to idle if it receives no queries for the specified timeout period. Automatic idling of data domains is useful as it helps control the proliferation of data domains in the Endeca Server deployment, thus improving hardware utilization by the Endeca Server. This topic discusses how Endeca Server manages automatic idling of data domains.

A data domain is idle when the Dgraph process is stopped, on the Endeca Server node hosting this data domain. A data domain that is idle continues to be registered with the Endeca Server, however the Endeca Server stops allocating resources (memory and processing threads), to this data domain. An idle data domain is activated automatically if it receives a query. In other words, once an end user of a front-end application powered by the Endeca Server, (such as Studio), issues a request, an idle data domain is activated, and its Dgraph process is restarted automatically.

Limitations for the idling of data domains are as follows:

Auto-idling is guaranteed on a data domain that has a single Dgraph process, and is ignored for data domains deployed in a cluster of more than one node. Specifically, if you create a data domain based on the data domain profile with autoIdle set to true in a cluster of more than one Endeca Server nodes, the data domain can be created successfully, but the autoIdle setting is ignored. Similarly, if you use a specific data domain profile with autoIdle set to true for a data domain that has one Dgraph node, and then rescale the data domain by adding nodes, the autoIdle setting is ignored by the Endeca Server.
Enabling a data domain and activating an idle data domain operations succeed only if they meet all resources constraints. In other words, if a data domain is idle and a query is issued for it (which would normally activate it), but other data domains were configured and enabled in the Endeca Server in the meantime, it is possible that the Endeca Server may not have sufficient resources to activate this data domain. In this case, the Endeca Server issues an error and the data domain remains idle until you manually free up space in the Endeca Server by disabling other data domains.

A data domain profile contains these settings (in the Manage Web Service) for idling of a data domain:

autoIdle indicates whether to turn a data domain to idle after a timeout, by stopping its Dgraph processes. If set to false, Endeca Server never makes this data domain idle. If set to true, the data domain is made idle after the optional timeout specified in the idleTimeoutMinutes, if during this timeout the data domain does not receive queries. The default for autoIdle is false.
idleTimeoutMinutes indicates the timeout to use for idling, in minutes. The timeout is counted from the time a data domain has been created and enabled. This parameter is optional. If not specified, the default of 15 minutes is used. 15 minutes is the lowest idle timeout value you can set. This timeout allows for long-running queries to complete successfully before the data domain is turned idle. If you have any queries running longer than 15 minutes, increase the idle timeout. To change the idling timeout for a data domain, disable a data domain, create a new data domain profile with new timeout, update the data domain to use the new data domain profile, and enable the data domain.

The following statements describe which operations activate (or do not activate) an idle data domain:

Queries from the Conversation Web Service, Configuration Web Service, and Data Ingest Web Service activate an idle data domain and restart the timeout timer.
Queries from Administration and Configuration URL commands also activate and restart the Dgraph processes for the idle data domain (and restart the timer), with the exception of http://<host>:<port>/endeca-server/admin/<data_domain>?op=ping, which does not activate an idle data domain.
Bulk load interaction. The process of loading data through the Bulk Ingest interface starts with the allocateBulkLoadPort operation from the Manage Web Service, which activates an idle data domain. If a bulk ingest operation is issued and is in progress for the data domain that is set to auto-idle, it is possible that bulk load may take longer than the idling timeout. If the timeout expires while bulk data load is in progress, the Endeca Server refreshes the idling timeout and the data domain does not shut down, so that bulk load can complete successfully.
The endeca-cmd commands do not activate an idle data domain with the exception of endeca-cmd allocate-bulk-load-port, which activates an idle data domain.

Note: It is important to distinguish an "idle data domain" from a "data domain that is configured to auto-idle": an idle data domain is the one that is already idle (its processes are stopped); while a data domain configured to auto-idle is the one that uses the data domain profile with autoIdle set to true. It can be active but set to auto-idle after a timeout, or, it can already be idle, if it received no queries during the timeout.

If a data domain is configured to auto-idle, stopping and restarting its Dgraph process is handled automatically by the Endeca Server, without requiring you to call these operations explicitly, as you would do when manually starting and stopping a data domain.

The following statements describe how an idle data domain behaves in various situations:

If a data domain is set to automatically idle, but is currently active, and you disable and then enable it, it will be active.
If a data domain is set to automatically idle, but is currently active and an outer transaction is open, it will not turn to idle until the transaction closes or is canceled (even if the auto-idling timeout expires and no queries are sent to the data domain during the timeout).
If a data domain is idle and you disable it, it will be idle when you re-enable it, until a query arrives that activates the data domain by restarting its Dgraph process.
If a data domain is idle, its idling timer is stopped. Once the data domain is activated, the clock for the timeout is restarted.
Updating a data domain to use a new data domain profile does not interfere with idling, because updating is done on a disabled data domain.
Cloning, exporting, and importing an idle data domain behaves in the same way as performing these operations for a disabled data domain.
If you idle a data domain, and then restart the WebLogic Server hosting the Endeca Server, a data domain will continue to be idle after the Endeca Server restarts.