This chapter provides platform-specific tuning tips, and includes the following topics:
This section discusses miscellaneous Solaris-specific issues and tuning tips, and includes the following topics:
Different platforms each have limits on the number of files that can be open in a single process at one time. For busy sites, increase that number. On Solaris systems, control this limit by setting rlim_fd_max in the /etc/system file. For Solaris 8, the default is 1024, which you can increase to 65536. For Solaris 9 and 10, the default is 65536, which doesn't need to be increased.
After making this or any change in the /etc/system file, reboot Solaris to put the new settings into effect. In addition, if you upgrade to a new version of Solaris, remove any line added to /etc/system and add it again only after verifying that it is still valid.
An alternative way to make this change is using the ulimit –n "value" command. Using this command does not require a system restart. However, this command only changes the login shell, while editing the etc/system file affects all shells.
If users are experiencing connection timeouts from a browser to Web Server when the server is heavily loaded, you can increase the size of the HTTP listener backlog queue. To increase this setting, edit the HTTP listener's listen queue value.
In addition to this setting, you must also increase the limits within the Solaris TCP/IP networking code. There are two parameters that are changed by executing the following commands:
/usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q 8192 /usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q0 8192
These two settings increase the maximum number of two Solaris listen queues that can fill up with waiting connections. tcp_conn_req_max_q increases the number of completed connections waiting to return from an accept() call. tcp_conn_req_max_q0 increases the maximum number of connections with the handshake incomplete. The default values are 128 and 1024, respectively. To automatically have these ndd commands executed after each system reboot, place them in a file called /etc/init.d/network-tuning and create a link to that file named /etc/rc2.d/S99network-tuning.
You can monitor the effect of these changes by using the netstat -s command and looking at the tcpListenDrop, tcpListenDropQ0, and tcpHalfOpenDrop values. Review them before adjusting these values. If the parameters are not set to zero, adjust the value to 2048 initially, and continue to monitor the netstat output.
The Web Server HTTP listener's listen queue setting and the related Solaris tcp_conn_req_max_q and tcp_conn_req_max_q0 settings are meant to match the throughput of the Web Server. These queues act as a "buffer" to manage the irregular rate of connections coming from web users. These queues allow Solaris to accept the connections and hold them until they are processed by the Web Server.
Do not accept more connections than the Web Server is able to process. Instead, limit the size of these queues and reject further connections than to accept excess connections and fail to service them. The value of 2048 for these three parameters typically reduces connection request failures, and improvement has been seen with values as high as 4096.
This adjustment is not expected to have any adverse impact in any web hosting environment, so you can consider this suggestion even if your system is not showing the symptoms mentioned.
If users are experiencing connection refused errors on a heavily loaded server, you can tune the use of network resources on the server.
When a TCP/IP connection is closed, the port is not reused for the duration of tcp_time_wait_interval (default value of 240000 milliseconds). This is to ensure that there are no leftover segments. The shorter the tcp_time_wait_interval, the faster precious network resources are again available. This parameter is changed by executing the following command. Do not reduce the parameter below 60000
usr/sbin/ndd -set /dev/tcp tcp_time_wait_interval 60000
To automatically have this ndd command executed after each system reboot, place it in a file called /etc/init.d/network-tuning and create a link to that file named /etc/rc2.d/S99network-tuning.
If your system is not exhibiting the symptoms mentioned, and if you are not well-versed in tuning the TCP protocol, do not change the above parameter.
If you are seeing unpredictable intermittent slowdowns in network response from a consistently loaded server, investigate setting the sq_max_size parameter by adding the following line to the /etc/system file:
set sq_max_size=512
This setting adjusts the size of the sync queue, which transfers packets from the hardware driver to the TCP/IP protocol driver. Using the value of 512 allows the queue to accommodate high volumes of network traffic without overflowing.
The Solaris Network Cache and Accelerator (SNCA) is a caching server that provides improved web performance to the Solaris operating system.
It is assumed that SNCA has been configured for the system on which the Web Server is running. For more information about SNCA and its configuration and tuning, refer to the following man pages on your system:
ncab2clf(1)
ncakmod(1)
nca(1)
snca(1)
nca.if(4)
ncakmod.conf(4)
ncalogd.conf(4)
This procedure assumes that SNCA has been configured, as discussed above.
From the Common Tasks page, choose a configuration and click Edit Configuration.
Click the HTTP Listeners tab and select the HTTP listener to edit.
On the Edit HTTP Listener page, set the Protocol Family to nca.
The HTTP listener must be listening on port 80 for this to work.
Save your changes.
Click the Performance tab.
Click the Cache sub tab.
On the Cache Settings page, make sure the file cache is enabled and enable Use Sendfile.
Save your changes.
Redeploy the configuration for your changes to take effect.
When configuring Web Server to be used with SNCA, disabling the thread pool provides better performance. These settings are on the configuration's Performance tab ⇒ HTTP sub tab, under Thread Pool Settings. To disable the thread pool, deselect the Thread Pool Enabled checkbox. You can also disable the thread pool using the wadm set-thread-pool-prop command's enabled property.
The thread pool can also be disabled with non-SNCA configurations, especially for cases in which short latency responses with no keep-alives must be delivered.
This section discusses changes that can be made for file system tuning, and includes topics that address the following issues:
Read the descriptions of the following parameters carefully. If the description matches your situation, consider making the adjustment.
If you are seeing high file system page-in rates on Solaris 8 or 9, increase the value of segmap_percent. This parameter is set by adding the following line to the /etc/system file:
set segmap_percent=25
segmap_percent adjusts the percentage of memory that the kernel maps into its address space for the file system cache. The default value is 12; that is, the kernel reserves enough space to map at most 12% of memory for the file system cache. On a heavily loaded machine with 4 GB of physical memory, improvements have been seen with values as high as 60. You can experiment with this value, starting with values around 25. On systems with large amounts of physical memory, you can raise this value in small increments, as it can significantly increase kernel memory requirements.
UNIX file system (UFS) volumes maintain the time that each file was accessed. Note that the following change does not turn off the access time updates when the file is modified, but only when the file is accessed. If the file access time updates are not important in your environment, you can turn them off by adding the noatime parameter to the data volume's mount point in /etc/vfstab. For example:
/dev/dsk/c0t5d0s6 /dev/rdsk/c0t5d0s6 /data0 ufs 1 yes noatime
Web Server's responsiveness depends greatly on the performance of the disk subsystem. Use the iostat utility to monitor how busy the disks are and how rapidly they complete I/O requests (the %b and svc_t columns, respectively). Service times are unimportant for disks that are less than about 30% busy, but for busier disks, service times should not exceed about 20 milliseconds. If your busy disks have slower service times, improving disk performance can help Web Server performance substantially.
Your first step is to balance the load: if some disks are busy while others are lightly loaded, move some files off of the busy disks and onto the idle disks. If there is an imbalance, correcting it usually gives a far greater payoff than trying to tune the overloaded disks.
This section describes some of the Solaris-specific tools and utilities you can use to monitor your system's behavior, and includes the following topics:
The tools described in this section monitor performance from the standpoint of how the system responds to the load that Web Server generates. For information about using Web Server's own capabilities to track the demands that users place on the Web Server itself, see Monitoring Server Performance.
Solaris offers several tools for taking “snapshots” of system behavior. Although you can capture their output in files for later analysis, the tools listed below are primarily intended for monitoring system behavior in real time:
The iostat -x 60 command reports disk performance statistics at 60-second intervals.
Watch the %b column to see how much of the time each disk is busy. For any disk busy more than 20% of the time, pay attention to the service time as reported in the svct column. Other columns report the I/O operation rates, the amount of data transferred, and so on.
The vmstat 60 command summarizes virtual memory activity and some CPU statistics at 60-second intervals.
Monitor the sr column to keep track of the page scan rate and take action if it's too high. Note that "too high" is very different for Solaris 8 and 9 than for earlier releases. Watch the us, sy, and id columns to see how heavily the CPUs are being used; remember that you need to keep plenty of CPU power in reserve to handle sudden bursts of activity. Also keep track of the r column to see how many threads are contending for CPU time; if this remains higher than about four times the number of CPUs, reduce the server's concurrency.
The mpstat 60 command gives a detailed look at CPU statistics, while the netstat -i 60 command summarizes network activity.
It is important not only to "spot-check" system performance with the tools mentioned above, but to collect longer-term performance histories so you can detect trends. If nothing else, a baseline record of a system performing well will help you figure out what has changed if the system starts behaving poorly. Enable the system activity reporting package by doing the following:
Edit the file /etc/init.d/perf and remove the # comment characters from the lines near the end of the file. For Solaris 10, run the following command:
svcadm enable system/sar
Run the command crontab -e sys and remove the # comment characters from the lines with the sa1 and sa2 commands. You can adjust how often the commands run and at what times of day depending on your site's activity profile. See the crontab man page for an explanation of the format of this file.
This command causes the system to store performance data in files in the /var/adm/sa directory, where by default they are retained for one month. You can then use the sar command to examine the statistics for time periods of interest.
The SE toolkit is a freely downloadable software package developed by Sun performance experts. In addition to collecting and monitoring raw performance statistics, the toolkit can apply heuristics to characterize the overall health of the system and highlight areas that need adjustment. You can download the toolkit and its documentation from the following location:
http://www.sunfreeware.com/setoolkit.html
DTrace is a comprehensive dynamic tracing framework for the Solaris Operating Environment. You can use the DTrace Toolkit to monitor the system. It is available from the following URL:
http://www.opensolaris.org/os/community/dtrace/dtracetoolkit/
The following table shows the operating system tuning for Solaris used when benchmarking for performance and scalability. These values are an example of how you can tune your system to achieve the desired result.
Table 4–1 Tuning Solaris for Performance Benchmarking
Parameter |
Scope |
Default Value |
Tuned Value |
Comments |
---|---|---|---|---|
/etc/system |
65536 |
65536 |
Process open file descriptors limit; accounts for the expected load (for the associated sockets, files, and pipes if any). |
|
/etc/system |
2 |
0 |
Controls streams driver queue size; setting to 0 makes it infinite so the performance runs are not hit by lack of buffer space. Set on clients too. Note that setting sq_max_size to 0 is not be optimal for production systems with high network traffic. |
|
ndd /dev/tcp |
240000 |
60000 |
Set on clients too. |
|
ndd /dev/tcp |
128 |
1024 | ||
ndd /dev/tcp |
1024 |
4096 | ||
ndd /dev/tcp |
480000 |
60000 | ||
ndd /dev/tcp |
7200000 |
900000 |
For high traffic web sites, lower this value. |
|
ndd /dev/tcp |
3000 |
3000 |
If retransmission is greater than 30-40%, increase this value. |
|
ndd /dev/tcp |
240000 |
10000 | ||
ndd /dev/tcp |
200 |
3000 | ||
ndd /dev/tcp |
32768 |
1024 |
Set on clients too. |
|
ndd /dev/tcp |
1 |
2 |
Slightly faster transmission of small amounts of data. |
|
ndd /dev/tcp |
8129 |
32768 |
To increase the transmit buffer. |
|
ndd /dev/tcp |
8129 |
32768 |
To increase the receive buffer. |
Use a combination of tunable parameters and other parameters to tune your system for performance benchmarking. These values are an example of how you can tune your system to achieve the desired result.
The following table shows the operating system tuning for Solaris 10 used when benchmarking for performance and scalability on UtraSPARC T1–based systems (64 bit systems).
Table 4–2 Tuning 64–bit Systems for Performance Benchmarking
Parameter |
Scope |
Default Value |
Tuned Value |
Comments |
---|---|---|---|---|
/etc/system |
65536 |
260000 |
Process open file descriptors limit; accounts for the expected load (for the associated sockets, files, pipes if any). |
|
/etc/system |
1 | |||
/etc/system |
2 |
0 |
Controls streams driver queue size; setting to 0 makes it infinite so the performance runs are not hit by lack of buffer space. Set on clients too. Note that setting sq_max_size to 0 is not optimal for production systems with high network traffic. |
|
0 | ||||
1 | ||||
/etc/system |
0 | |||
/etc/system |
2048 | |||
/etc/system |
2048 | |||
/etc/system |
384 | |||
ipge:ipge_dvma_thresh |
/etc/system |
384 | ||
/etc/system |
1 | |||
ndd /dev/tcp |
128 |
3000 | ||
ndd /dev/tcp |
1024 |
3000 | ||
ndd /dev/tcp |
4194304 | |||
ndd/dev/tcp |
2097152 | |||
ndd /dev/tcp |
8129 |
400000 |
To increase the transmit buffer. |
|
ndd /dev/tcp |
8129 |
400000 |
To increase the receive buffer. |
Note that the IPGE driver version is 1.25.25.
If HTTP access is logged, follow these guidelines for the disk:
Write access logs on faster disks or attached storage.
If running multiple instances, move the logs for each instance onto separate disks as much as possible.
Enable the disk read/write cache. Note that if you enable write cache on the disk, some writes can be lost if the disk fails.
Consider mounting the disks with the following options, which can yield better disk performance: nologging, directio, noatime.
If more than one network interface card is used, make sure the network interrupts are not all going to the same core. Run the following script to disable interrupts:
allpsr=`/usr/sbin/psrinfo | grep -v off-line | awk '{ print $1 }'` set $allpsr numpsr=$# while [ $numpsr -gt 0 ]; do shift numpsr=`expr $numpsr - 1` tmp=1 while [ $tmp -ne 4 ]; do /usr/sbin/psradm -i $1 shift numpsr=`expr $numpsr - 1` tmp=`expr $tmp + 1` done done |
Put all network interfaces into a single group. For example:
$ifconfig ipge0 group webserver $ifconfig ipge1 group webserver |
In some cases, performance can be improved by using large page sizes. To start the 32–bit Web Server with 4 MB pages:
LD_PRELOAD_32=/usr/lib/mpss.so.1 ; export LD_PRELOAD_32; export MPSSHEAP=4M; ./bin/startserv; unset LD_PRELOAD_32; unset MPSSHEAP
For 64–bit servers:
LD_PRELOAD_64=/usr/lib/64/mpss.so.1; export LD_PRELOAD_64; export MPSSHEAP=4M; ./bin/startserv; unset LD_PRELOAD_64; unset MPSSHEAP