Recommended threading strategies and OS platform

The size of the thread pool and the host operating system impact performance and processor utilization.

In general, Endeca recommends using one thread per processor or core for good performance in most cases. The actual optimal number of threads for a given application depends on many factors, and is best determined through experimental performance measurements using expected query load on production data.

If high performance is required, enable more than one thread. Determine the optimal number of threads through load testing of different configurations.

As a starting point, enable the following number of threads:

For example, consider a server with two hyperthreaded processors and sufficient disk resources and RAM, on which a high-performance application will be deployed. The appropriate starting point for such an architecture would be one MDEX Engine running multithreaded with 4 threads.

Multithreaded MDEX Engine on Linux and Solaris

On Linux and Solaris, the MDEX Engine uses the POSIX Thread Library, Pthreads. You can examine the thread count using standard tools, such as top.

Multithreaded MDEX Engine on Windows

On Windows, the MDEX Engine uses native Windows threads. The thread count for an MDEX Engine can be examined in the Windows Task Manager in the Threads column.

Note: The number of threads listed may be greater than the value specified for the --threads flag; the additional threads that could be listed are those that are used infrequently by processes that are not CPU-intensive and represent internal maintenance tasks. All the CPU-intensive, query processing-related threads are controlled by the --threads flag.

Multithreaded MDEX Engine on VMware

On VMware, use the following configuration:
  • Be aware of the limitation of four virtual CPUs per virtual machine.
  • Specify four threads for each Dgraph. Overall, the number of threads should not exceed the number of VCPUs.