Sun Java System Web Server 6.1 SP7 Performance Tuning, Sizing, and Scaling Guide

Chapter 5 Platform-specific Issues and Tips

This chapter provides platform-specific tuning tips, and includes the following topics:

Solaris-specific Issues

This section discusses miscellaneous Solaris-specific issues and tuning tips, and includes the following topics:

Files Open in a Single Process

Different platforms have different limits on the number of files that can be open in a single process at one time. For busy sites, increase that number to 8192. To do so on Solaris, make the change in the file /etc/system by setting rlim_fd_max, and then reboot.

File Descriptor Limits

Append the following line to the file /etc/system to increase the number of file descriptors for Sun Java System Web Server:

set rlim_fd_max=65536

After making this or any change in the /etc/system file, reboot Solaris to put the new settings into effect. In addition, if you upgrade to a new version of Solaris, any line added to /etc/system should be removed and added again only after verifying that it is still valid.

Failure to Connect to HTTP Server

If users are experiencing connection timeouts from a browser to Sun Java System Web Server when the server is heavily loaded, you can increase the size of the HTTP listener backlog queue. To increase this setting, edit the file ListenQ parameter in the magnus.conf file:

ListenQ  8192

In addition to this setting, you must also increase the limits within the Solaris TCP/IP networking code. There are two parameters that are changed by executing the following commands:

/usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q 8192
/usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q0 8192

These two settings increase the maximum number of two Solaris listen queues that can fill up with waiting connections. tcp_conn_req_max_q increases the number of completed connections waiting to return from an accept() call. tcp_conn_req_max_q0 increases the maximum number of connections with the handshake incomplete. The default values are 128 and 1024 respectively. To automatically have these ndd commands executed after each system reboot, place them in a file called /etc/init.d/network-tuning and create a link to that file named /etc/rc2.d/S99network-tuning.

You can monitor the effect of these changes by using the netstat -s command and looking at the tcpListenDrop, tcpListenDropQ0, and tcpHalfOpenDrop values. Review them before adjusting these values. If they are not zero, adjust the value to 2048 initially, and continue to monitor the netstat output.

The Sun Java System Web Server ListenQ setting and the related Solaris tcp_conn_req_max_q and tcp_conn_req_max_q0 settings should match the throughput of the Sun Java System Web Server HTTP server. These queues act as a "buffer" to manage the irregular rate of connections coming from web users. These queues allow Solaris to accept the connections and hold them until they are processed by the Sun Java System Web Server HTTP server application.

You do not want to accept more connections than the Sun Java System Web Server HTTP server will be able to process. It is better to limit the size of these queues and reject further connections than to accept excess connections and fail to service them. The value of 2048 for these three parameters will typically reduce connection request failures, and improvement has been seen with values as high as 4096.

This adjustment is not expected to have any adverse impact in any web hosting environment, so you can consider this suggestion even if your system is not showing the symptoms mentioned.

Connection Refused Errors

If users are experiencing connection refused errors on a heavily loaded server, you can tune the use of network resources on the server.

When a TCP/IP connection is closed, the port is not reused for the duration of tcp_time_wait_interval (default value of 240000 milliseconds). This is to ensure that there are no leftover segments. The shorter the tcp_time_wait_interval, the faster precious network resources are again available. This parameter is changed by executing the following command (do not reduce it below 60000):

usr/sbin/ndd -set /dev/tcp tcp_time_wait_interval 60000

To automatically have this ndd command executed after each system reboot, place it in a file called /etc/init.d/network-tuning and create a link to that file named /etc/rc2.d/S99network-tuning.

If your system is not exhibiting the symptoms mentioned, and if you are not well-versed in tuning the TCP protocol, it is suggested that you do not change the above parameter.

Tuning TCP Buffering

If you are seeing unpredictable intermittent slowdowns in network response from a consistently loaded server, you might investigate setting the sq_max_size parameter by adding the following line to the /etc/system file:

set sq_max_size=512

This setting adjusts the size of the sync queue, which transfers packets from the hardware driver to the TCP/IP protocol driver. Using the value of 512 allows the queue to accommodate high volumes of network traffic without overflowing.

Using the Solaris Network Cache and Accelerator (SNCA)

The Solaris Network Cache and Accelerator (SNCA) is a caching server that provides improved web performance to the Solaris operating system. It is available on Solaris 8 Update 5 and higher.

It is assumed that SNCA has been configured for the system on which the Web Server is running. For more information about SNCA and its configuration and tuning, refer to the following man pages on your system:

Enable SNCA to work with Sun Java System Web Server (assuming SNCA configuration, as discussed above)

ProcedureTo enable SNCA to work with Sun Java System Web Server

  1. Edit the Sun Java System Web Server server.xml file so the listen socket on port 80 includes family="nca" as shown below (the server must be listening on port 80 for this to work):

    <LS id="ls1" ip="0.0.0.0" port="80" family="nca" security="off" acceptorthreads="1">

  2. Edit the Sun Java System Web Server nsfc.conf file and set the following:

    CacheFileContent=falseTransmitFile=true

  3. Restart the Web Server for changes to take effect.

RqThrottle and ConnQueueSize

When configuring Sun Java System Web Server to be used with SNCA, setting the RqThrottle and ConnQueueSize parameters in magnus.conf to 0 provides better performance. Because SNCA manages the client connections, it is not necessary to set these parameters. These parameters can also be set to 0 with non-SNCA configurations, especially for cases in which short latency responses with no keep-alives must be delivered..


Note –

RqThrottle and ConnQueueSize must both be set to 0.


For more information about RqThrottle and ConnQueueSize, see the chapter pertaining to magnus.conf in the Sun Java System Web Server 6.1 SP7 Administrator’s Configuration File Reference. Also consult the RqThrottle and ConnQueueSize entries in the index in this book.

Solaris File System Tuning

This section discusses changes that can be made for file system tuning, and includes topics that address the following issues:

High File System Page-in Rate

If you are seeing high file system page-in rates on Solaris 8 or 9, you may benefit from increasing the value of segmap_percent. This parameter is set by adding the following line to the /etc/system file:

set segmap_percent=25

segmap_percent adjusts the percentage of memory that the kernel will map into its address space for the file system cache. The default value is 12; that is, the kernel will reserve enough space to map at most 12% of memory for the file system cache. On a heavily loaded machine with 4 GB of physical memory, improvements have been seen with values as high as 60. You should experiment with this value, starting with values around 25. On systems with large amounts of physical memory, you should raise this value in small increments, as it can significantly increase kernel memory requirements.

Reduce File System Housekeeping

UNIX file system (UFS) volumes maintain the time that each file was accessed. Note that the following change does not turn off the access time updates when the file is modified, but only when the file is accessed. If the file access time updates are not important in your environment, you could turn off the same by adding the noatime parameter to the data volume's mount point in /etc/vfstab. For example:

/dev/dsk/c0t5d0s6 /dev/rdsk/c0t5d0s6 /data0 ufs 1 yes noatime

Long Service Times on Busy Disks or Volumes

Sun Java System Web Server's responsiveness depends greatly on the performance of the disk subsystem. Use the iostat utility to monitor how busy the disks are and how rapidly they complete I/O requests (the %b and svc_t columns, respectively). Service times are unimportant for disks that are less than about 30% busy, but for busier disks service times should not exceed about 20 milliseconds. If your busy disks have slower service times, improving disk performance may help Sun Java System Web Server performance substantially.

Your first step should be to balance the load: if some disks are busy while others are lightly loaded, move some files off of the busy disks and onto the idle disks. If there is an imbalance, correcting it will usually give a far greater payoff than trying to tune the overloaded disks.

Solaris-specific Performance Monitoring

This section describes some of the Solaris-specific tools and utilities you can use to monitor your system's behavior, and includes the following topics:

Short-term System Monitoring

Solaris offers several tools for taking "snapshots" of system behavior. Although you can capture their output in files for later analysis, the tools listed below are primarily intended for monitoring system behavior in real time:

Long-term System Monitoring

It is important not only to "spot-check" system performance with the tools mentioned above, but to collect longer-term performance histories so you can detect trends. If nothing else, a baseline record of a system performing well may help you figure out what has changed if the system starts behaving poorly. We recommend you enable the system activity reporting package by doing the following:

"Intelligent" Monitoring

The "SE toolkit" is a freely downloadable software package developed by Sun performance experts. In addition to collecting and monitoring raw performance statistics, the toolkit can apply heuristics to characterize the overall health of the system and highlight areas that may need adjustment. You can download the toolkit and its documentation from the following location:

http://www.setoolkit.com/

Tuning Solaris for Performance Benchmarking

The following table shows the operating system tuning for Solaris used when benchmarking for performance and scalability. These values are an example of how you might tune your system to achieve the desired result.

Table 5–1 Tuning Solaris for performance benchmarking

Parameter  

Scope  

Default Value  

Tuned Value  

Comments  

rlim_fd_max

/etc/system

1024 

8192 

Process open file descriptors limit; should account for the expected load (for the associated sockets, files, pipes if any). 

rlim_fd_cur

/etc/system

64 

8192 

 

sq_max_size

/etc/system

Controls streams driver queue size; setting to 0 makes it infinity so the performance runs won’t be hit by lack of buffer space. Set on clients too. 

tcp_close_wait_interval

ndd /dev/tcp

240000 

60000 

Set on clients too. 

tcp_time_wait_interval

ndd /dev/tcp

240000 

60000 

For Solaris 7 only. Set on clients too. 

tcp_conn_req_max_q

ndd /dev/tcp

128 

1024 

 

tcp_conn_req_max_q0

ndd /dev/tcp

1024 

4096 

 

tcp_ip_abort_interval

ndd /dev/tcp

480000 

60000 

 

tcp_keepalive_interval

ndd /dev/tcp

7200000 

900000 

For high traffic web sites, lower this value. 

tcp_rexmit_interval_initial

ndd /dev/tcp

3000 

3000 

If retransmission is greater than 30-40%, you should increase this value. 

tcp_rexmit_interval_max

ndd /dev/tcp

240000 

10000 

 

tcp_rexmit_interval_min

ndd /dev/tcp

200 

3000 

 

tcp_smallest_anon_port

ndd /dev/tcp

32768 

1024 

Set on clients too. 

tcp_slow_start_initial

ndd /dev/tcp

Slightly faster transmission of small amounts of data. 

tcp_xmit_hiwat

ndd /dev/tcp

8129 

32768 

To increase the transmit buffer. 

tcp_recv_hiwat

ndd /dev/tcp

8129 

32768 

To increase the receive buffer.