Avoiding defunct EAC processes

On UNIX systems, the ps command may report a number of defunct EAC-originated processes. This is known and expected EAC behavior and it does not necessarily indicate a problem.

For example, you might see the following output from the ps command:

> ps -ef | grep endeca
endeca 1924 1875 0 - ? 2:00 <defunct>
[...]

Additionally, warning messages of this form appear in the $ENDECA_CONF/logs/process.0.log file on the affected server:

Apr 17, 2009 11:24:17 AM
com.endeca.esf.delegate.procctrl.ExecutableProcessHandle
tryCleanShutdown
WARNING: Process 1924 did not shutdown cleanly after 30 seconds.
Terminating forcefully.

The cause of these warning messages is as follows. When the EAC shuts down a child process like a Dgraph, it initially sends the correct exit command for the process (admin?op=exit in the case of the Dgraph) and waits 30 seconds for the process to exit. However, if the Dgraph is processing a long-running query, or if its request queue is long, it may not be able to shut down within 30 seconds.

If the process does not exit after 30 seconds, the EAC logs the warning message shown above and then kills the process with the operating system's kill command. When this occurs, the affected process is reported by ps as being in a <defunct> state. In this state, it does not use memory, disk space, or ports and should not be a problem for the system.

Alternatively, this can happen if you kill the EAC process directly rather than by using the shutdown.sh script. In this case the EAC process terminates immediately, leaving any chlid processes in a <defunct> state.

To avoid defunct EAC processing, consider the following recommendations:

For a Dgraph or Agraph, the request log shows whether queuing or long processing times are preventing the Dgraph from responding in time to the admin?op=exit command. If this is the case, spreading traffic over a larger number of MDEX Engine mirrors (for queuing) or reducing query complexity (for long processing times) should allow the Dgraph to respond more quickly to the exit command.
Another option may be to override the default 30-second timeout period for EAC shutdowns by modifying the value of the com.endeca.eac.process.shutdownTimeoutSecs setting in your server's $ENDECA_CONF/conf/eac.properties file.

This value shows the length of time in seconds that the EAC Agent on that server waits for a process to exit. Specifying a higher value for this setting may help prevent creation of <defunct> EAC child processes, but may also make EAC updates slower, because the EAC Agent will wait longer for all processes to exit.

Note: Modifications to this setting will not take effect until the Endeca HTTP service (which contains the EAC) is restarted on the server.