15.3 Tuning the File Descriptor Limit

The operating system uses file descriptors to handle file-system files as well as pseudo files, such as connections and listener sockets.

When an Oracle Traffic Director instance starts, the following parameters are taken into consideration when auto-configuring values related to file descriptors:

  • HTTP processing threads (<thread-pool>)

  • Access log counts for all virtual servers (<access-log>)

  • Listeners (<http-listener>, <tcp-listener>)

  • Keep-alive connections (<keep-alive>)

  • Number of origin server pools (<origin-server-pool>)

  • Number of origin servers (<origin-server>)

  • Origin server connections (<origin-server>/<max-connections>)

  • TCP processing threads (<tcp-thread-pool>)

The key Oracle Traffic Director objects that require file descriptors are keep-alive connections, queued connections, and connections to origin servers. If you do not explicitly specify limits for these objects, then when the Oracle Traffic Director instance starts, it configures the limits—maximum keep-alive connections, connection queue size, and maximum connections for each origin server—automatically based on the total number of available file descriptors in the system.

For instance, the maximum number of HTTP processing threads * 4 should ideally be less than the maximum number of file descriptors available to the process. If you are going to increase the Maximum number of HTTP processing threads, then you should also correspondingly increase the total number of File Descriptor available to the Traffic Director. (For example, if the file descriptor limit is set to 65536, then setting the maximum number of HTTP processing threads to 20000 will cause sub-optimal tuning as 80000 (20000*4=80000) will exhaust/reserve file descriptors for the worker threads, which does not leave much for other subsystems). Hence you should set a high value for the maximum number of HTTP processing threads only after some experimentation.

Figure 15-1 depicts increasing the maximum number of HTTP processing threads in the advanced settings for the configuration.

Figure 15-2 Maximum Number of HTTP Processing Threads

Description of Figure 15-2 follows
Description of ''Figure 15-2 Maximum Number of HTTP Processing Threads''

The number of allocated file descriptors cannot exceed the limit that the system can support. To find out the current system limit for file descriptors, run the following command:

$ cat /proc/sys/fs/file-max
2048

To find out how many of the available file descriptors are being currently used, run the following command:

$ cat /proc/sys/fs/file-nr

The command returns an output that resembles the following:

625 52 2048

In this example, 625 is the number of allocated file descriptors, 52 is the number of free allocated file descriptors, and 2048 is the maximum number of file descriptors that the system supports.

Note:

In Solaris, system wide file descriptors in use can be found by using the following command: 
# echo ::kmastat | mdb -k | grep file_cache

This command returns an output that resembles the following:

file_cache        56    1154   1305      73728B   659529     0

In this example, 1154 is the number of file descriptors in use and 1305 the number of allocated file descriptors. Note that in Solaris, there is no maximum open file descriptors setting. They are allocated on demand as long as there is free RAM available.

When the number of allocated file descriptors reaches the limit for the system, the following error message is displayed in the system console when you try to open a file:

Too many open files in system.

The following message is written to the server log:

[ERROR:16] [OTD-10546] Insufficient file descriptors for optimum configuration.

This is a serious problem, indicating that the system is unable to open any more files. To avoid this problem, consider increasing the file descriptor limit to a reasonable number.

To change the number of file descriptors in Linux, do the following as the root user:

  1. Edit the following line in the /etc/sysctl.conf file:

    fs.file-max = value

    value is the new file descriptor limit that you want to set.

  2. Apply the change by running the following command:

    # /sbin/sysctl -p

    Note:

    In Solaris, change the value of rlim_fd_max in the /etc/system file to specify the ”hard” limit on file descriptors that a single process might have open. Overriding this limit requires superuser privilege. Similarly, rlim_fd_cur defines the ”soft” limit on file descriptors that a single process can have open. A process might adjust its file descriptor limit to any value up to the ”hard” limit defined by rlim_fd_max by using the setrlimit() call or by issuing the limit command in whatever shell it is running. You do not require superuser privilege to adjust the limit to any value less than or equal to the hard limit.

    For example, to increase the hard limit, add the following command to /etc/system and reboot it once:

    set rlim_fd_max = 65536

    For more information about Solaris file descriptor settings, see Section 15.10.1, "Files Open in a Single Process (File Descriptor Limits)".

Most Operating Systems allow system administrators to configure a limit on the maximum number of file descriptors available to a process (such as Oracle Traffic Director). On Linux machines, this limit is typically set via ulimit -n – either at the time of logging into the system or simply while starting an application. System administrators will need to ensure that this limit (typically via ulimit -n) is set to least above 32276 file descriptors (or 65536 for high throughput systems) before starting an Oracle Traffic Director Server instance.

As a rough rule of thumb, the thread-pool element, max-threads * 4 should be less than the maximum number of file descriptors available to the process. That is, max-threads should be less than 1/5th of the maximum number of file descriptors.

For example, if the file descriptor limit is set to 65536, then setting max-threads to 20000 will cause sub-optimal tuning as 20000*4=80000 will exhaust/reserve file descriptors for the worker threads, leaving little else for other subsystems.

High values of max-threads should be used only after experimentation. Having tens of thousands of threads in a process may hurt performance.