15.2 Tuning the File Descriptor Limit

The operating system uses file descriptors to handle file-system files as well as pseudo files, such as connections and listener sockets.

When an Oracle Traffic Director instance starts, the following parameters are taken into consideration when auto-configuring values related to file descriptors:

HTTP processing threads (<thread-pool>)
Access log counts for all virtual servers (<access-log>)
Listeners (<http-listener>, <tcp-listener>)
Keep-alive connections (<keep-alive>)
Number of origin server pools (<origin-server-pool>)
Number of origin servers (<origin-server>)
Origin server connections (<origin-server>/<max-connections>)
TCP processing threads (<tcp-thread-pool>)

The key Oracle Traffic Director objects that require file descriptors are keep-alive connections, queued connections, and connections to origin servers. If you do not explicitly specify limits for these objects, then when the Oracle Traffic Director instance starts, it configures the limits—maximum keep-alive connections, connection queue size, and maximum connections for each origin server—automatically based on the total number of available file descriptors in the system.

When the file descriptor limit is set to a very high value, auto-configuration of unspecified parameters can cause Oracle Traffic Director instances to consume excessive amount of memory or can result in sub-optimal configurations. To avoid these issues, specify values for these parameters explicitly on systems that have a high file-descriptor limit.

For instance, max-threads * 4 should ideally be less than the maximum number of file descriptors available to the process. For example, if the file descriptor limit is set to 65536, then setting max-threads to 20000 will cause sub-optimal tuning as 80000 (20000*4=80000) will exhaust/reserve file descriptors for the worker threads, which does not leave much for other subsystems. Hence a high value should be set for max-threads only after some experimentation.

The number of allocated file descriptors cannot exceed the limit that the system can support. To find out the current system limit for file descriptors, run the following command:

$ cat /proc/sys/fs/file-max
2048

To find out how many of the available file descriptors are being currently used, run the following command:

$ cat /proc/sys/fs/file-nr

The command returns an output that resembles the following:

625 52 2048

In this example, 625 is the number of allocated file descriptors, 52 is the number of free allocated file descriptors, and 2048 is the maximum number of file descriptors that the system supports.

Note:

In Solaris, system wide file descriptors in use can be found by using the following command:

# echo ::kmastat | mdb -k | grep file_cache

This command returns an output that resembles the following:

file_cache        56    1154   1305      73728B   659529     0

In this example, 1154 is the number of file descriptors in use and 1305 the number of allocated file descriptors. Note that in Solaris, there is no maximum open file descriptors setting. They are allocated on demand as long as there is free RAM available.

When the number of allocated file descriptors reaches the limit for the system, the following error message is displayed in the system console when you try to open a file:

Too many open files in system.

The following message is written to the server log:

[ERROR:16] [OTD-10546] Insufficient file descriptors for optimum configuration.

This is a serious problem, indicating that the system is unable to open any more files. To avoid this problem, consider increasing the file descriptor limit to a reasonable number.

To change the number of file descriptors in Linux, do the following as the root user:

Edit the following line in the /etc/sysctl.conf file:
```
fs.file-max = value
```
value is the new file descriptor limit that you want to set.
Apply the change by running the following command:
```
# /sbin/sysctl -p
```
Note:

In Solaris, change the value of rlim_fd_max in the /etc/system file to specify the “hard” limit on file descriptors that a single process might have open. Overriding this limit requires superuser privilege. Similarly, rlim_fd_cur defines the “soft” limit on file descriptors that a single process can have open. A process might adjust its file descriptor limit to any value up to the “hard” limit defined by rlim_fd_max by using the setrlimit() call or by issuing the limit command in whatever shell it is running. You do not require superuser privilege to adjust the limit to any value less than or equal to the hard limit.

For example, to increase the hard limit, add the following command to /etc/system and reboot it once:
```
set rlim_fd_max = 65536
```
For more information about Solaris file descriptor settings, see Section 15.13.1, "Files Open in a Single Process (File Descriptor Limits)".

As a rough rule of thumb, the thread-pool element, max-threads * 4 should be less than the maximum number of file descriptors available to the process. That is, max-threads should be less than 1/5th of the maximum number of file descriptors.

For example, if the file descriptor limit is set to 65536, then setting max-threads to 20000 will cause sub-optimal tuning as 20000*4=80000 will exhaust/reserve file descriptors for the worker threads, leaving little else for other subsystems.

High values of max-threads should be used only after experimentation. Having tens of thousands of threads in a process may hurt performance.