test

Sun Java System Access Manager 7.1 Performance Tuning and Troubleshooting Guide

Chapter 5 Tuning Considerations

Note –

The following tuning considerations are based on the tuning of various test deployments. Because each deployment is unique, you might need further customization and interactive testing to satisfy your specific requirements.

Operating System (OS) Considerations

Solaris OS

Sun Fire T1000 and T2000 Servers

If Access Manager is installed on a Sun Fire T1000 or T2000 server, the tuning scripts for Web Server 7.0 and Application Server 8.2 set the JVM GC ParallelGCThreads parameter to 8:

-XX:ParallelGCThreads=8

This parameter reduces the number of garbage collection threads, which could be unnecessarily high on a 32-thread capable system. However, you can increase the value to 16 or even 20 for a 32 virtual CPU machine such as a Sun Fire T1000 server, if it minimizes full garbage collection activities.

Solaris SPARC Systems with CMT Processor with CoolThreads Technology

For Solaris SPARC systems with CMT processor with CoolThreads technology, in the /etc/opt/SUNWam/config/AMConfig.properties file, it is recommended that you add the following properties at the end of the file:

com.sun.identity.log.resolveHostName=false
com.sun.am.concurrencyRate=value

where value depends on the number of cores in a Sun Fire T1000 or T2000 server. For example, for 8 cores, set value to 8, or for 6 cores, set value to 6.

Linux OS

To tune for maximum performance on Linux systems, you need to make tuning adjustments to the following items:

Note –

If you are running Application Server 8.1 on Red Hat Linux, the stack size of the threads created by the Red Hat OS for Application Server is 10 Mbytes, which can cause JVM resource problems (CR 6223676). To prevent these problems, set the Red Hat OS operating stack size to a lesser value such as 2048 or even 256 Kbytes, by executing the ulimit command before you start Application Server. Execute the ulimit command on the same console that you will use to start Application Server. For example:

ulimit -s 256

File Descriptors

You might need to increase the number of file descriptors from the default. Having a higher number of file descriptors ensures that the server can open sockets under high load and not abort requests coming in from clients. Start by checking system limits for file descriptors with this command: 

cat /proc/sys/fs/file-max
8192

The current limit shown is 8192. To increase it to 65535, use the following command (as root):

echo "65535" > /proc/sys/fs/file-max

To make this value to survive a system reboot, add it to /etc/sysctl.conf and specify the maximum number of open files permitted: 

fs.file-max = 65535

Note: The parameter is not proc.sys.fs.file-max, as you might expect.

To list the available parameters that can be modified using sysctl:

sysctl -a

To load new values from the sysctl.conf file:

sysctl -p /etc/sysctl.conf

To check and modify limits per shell, use the following command:

limit

The output will look something like this:

cputime         unlimited
filesize        unlimit
datasize        unlimited
stacksize       8192 kbytes
coredumpsize    0 kbytes
memoryuse       unlimited
descriptors     1024
memorylocked    unlimited
maxproc         8146
openfiles       1024

The openfiles and descriptors show a limit of 1024. To increase the limit to 65535 for all users, edit /etc/security/limits.conf as root, and modify or add the nofile setting (number of file) entries:

*         soft    nofile                     65535
*         hard    nofile                     65535

The asterisk (*) is a wildcard that identifies all users. You could also specify a user ID instead.

Then edit /etc/pam.d/login and add the line:

session required /lib/security/pam_limits.so

On Red Hat Linux , you also need to edit /etc/pam.d/sshd and add the following line:

session required /lib/security/pam_limits.so

On many systems, this procedure will be sufficient. Log in as a regular user and try it before doing the remaining steps. The remaining steps might not be required, depending on how pluggable authentication modules (PAM) and secure shell (SSH) are configured.

Virtual Memory

To change virtual memory settings, add the following to /etc/rc.local:

echo 100 1200 128 512 15 5000 500 1884 2 > /proc/sys/vm/bdflush

For more information, view the man pages for bdflush.

Network Interface

To ensure that the network interface is operating in full duplex mode, add the following entry into /etc/rc.local:

mii-tool -F 100baseTx-FD eth0

where eth0 is the name of the network interface card (NIC).

Disk I/O Settings

To tune disk I/O performance for a non-SCSI disk, follow these steps:

  1. Test the disk speed with this command:

     /sbin/hdparm -t /dev/hdX

  2. Enable direct memory access (DMA) with this command: 

    /sbin/hdparm -d1 /dev/hdX

  3. Check the speed again using the hdparm command. Given that DMA is not enabled by default, the transfer rate might have improved considerably. In order to do this at every reboot, add the /sbin/hdparm -d1 /dev/hdX line to /etc/conf.d/local.start, /etc/init.d/rc.local, or whatever the startup script is called.

TCP/IP Settings

To tune the TCP/IP settings, follow these steps:

  1. Add the following entry to /etc/rc.local: 

    echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 60000 > /proc/sys/net/ipv4/tcp_keepalive_time
echo 15000 > /proc/sys/net/ipv4/tcp_keepalive_intvl
echo 0 > /proc/sys/net/ipv4/tcp_window_scaling

  2. Add the following to /etc/sysctl.conf:

    # Disables packet forwarding
net.ipv4.ip_forward = 0
# Enables source route verification
net.ipv4.conf.default.rp_filter = 1
# Disables the magic-sysrq key
kernel.sysrq = 0
net.ipv4.ip_local_port_range = 1204 65000
net.core.rmem_max = 262140
net.core.rmem_default = 262140
net.ipv4.tcp_rmem = 4096 131072 262140
net.ipv4.tcp_wmem = 4096 131072 262140
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_keepalive_time = 60000
net.ipv4.tcp_keepalive_intvl = 15000
net.ipv4.tcp_fin_timeout = 30

  3. Add the following as the last entry in /etc/rc.local:

    sysctl -p /etc/sysctl.conf

  4. Reboot the system.

  5. Use this command to increase the size of the transmit buffer:

    tcp_recv_hiwat ndd /dev/tcp 8129 32768

Third-Party Web Containers

IBM WebSphere Application Server

Consider making the following changes in the WebSphere Administrative Console:

For more information, see the “IBM WebSphere V5.1 Performance, Scalability, and High Availability WebSphere Handbook Series” at:

http://www.redbooks.ibm.com//Redbooks.nsf/RedbookAbstracts/sg246198.html?OpenDocument

JVM Tuning Parameters

Add the JVM tuning parameters for JVM 1.4.2 shown below, by following these links in the console: 

Servers>Application Servers>server1>Process Definition>Java Virtual Machine

Add “-server” as the first parameter in the “Generic JVM arguments” box. Then, add the following entries after the other existing parameters: 

-XX:NewSize=336M -XX:MaxNewSize=336M
-XX:+DisableExplicitGC
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSPermGenSweepingEnabled
-XX:+UseCMSCompactAtFullCollection
-XX:CMSFullGCsBeforeCompaction=0
-XX:+CMSClassUnloadingEnabled
-XX:SoftRefLRUPolicyMSPerMB=0
-XX:+PrintClassHistogram 
-XX:+PrintGCTimeStamps 
-Xloggc:/opt/WebSphere/AppServer/logs/server1/gc.log
-XX:-CMSParallelRemarkEnabled

If you use WebSphere 6.x with Sun JVM 1.5 or later, then some of the garbage collection (GC) algorithms can be safely removed. The following is a list of JVM options that can be used with Sun JVM 1.5 or later. 

-XX:NewSize=336M -XX:MaxNewSize=336M
-XX:+DisableExplicitGC
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+PrintClassHistogram
-XX:+PrintGCTimeStamps
-Xloggc:/opt/WebSphere/AppServer/logs/server1/gc.log
-XX:-CMSParallelRemarkEnabled

Servlet Caching

Make sure that servlet caching is enabled by checking the checkbox next to “Enable servlet caching” by following these links in the console:

Application Servers>server1>Web Container>Configuration: Servlet caching

Thread Pool Size

Allow the thread pool to grow beyond the maximum thread pool size set by checking the checkbox next to “Allow thread allocation beyond maximum thread size” by following these links: 

Application Servers>server1>Web Container>Thread Pool Is Growable

BEA WebLogic Server

Consider making the following changes:

JVM GC Parameter

For BEA WebLogic Server 8.1 SP4, to avoid the java.lang.OutofMemoryError reported by the WebLogic JVM 1.4.2_05, add the following JVM GC (garbage collection) parameter in the startWebLogic.sh JAVA_OPTIONS: 

-XX:-CMSParallelRemarkEnabled

Set this parameter in addition to the other heap size and GC parameters that have been added for JVM 1.4.2 for Application Server 8.1 and Web Server 6.1.

For example, if Access Manager is installed in the default user_projects location (/usr/local/bea/user_projects/domains/mydomain/startWebLogic.sh): 

JAVA_OPTIONS="-XX:+DisableExplicitGC -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled
-XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0
-XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled
-XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram
-XX:+PrintGCTimeStamps
-Xloggc:/usr/local/bea/user_projects/domains/mydomain/myserver/gc.log"

If you use WebLogic 9.x with Sun JVM 1.5 or later, then some of the GC algorithms can be safely removed. The following is a list of JVM options that can be used with Sun JVM 1.5 or later. 

-XX:NewSize=336M -XX:MaxNewSize=336M
-XX:+DisableExplicitGC
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+PrintClassHistogram
-XX:+PrintGCTimeStamps
-Xloggc:/opt/WebSphere/AppServer/logs/server1/gc.log
-XX:-CMSParallelRemarkEnabled

Heap Size

Modify the commonEnv.sh script in the /usr/local/bea/weblogic81/common/bin directory for heap size increases in the section where $PRODUCTION_MODE" = "true" (which should be set to true, before running Access Manager in /usr/local/bea/user_projects/domains/mydomain/startWebLogic.sh):

# Set up JVM options base on value of JAVA_VENDOR
if [ "$PRODUCTION_MODE" = "true" ]; then
  case $JAVA_VENDOR in
  BEA)
    JAVA_VM=-jrockit
    MEM_ARGS="-Xms128m -Xmx256m"
  ;;
  HP)
    JAVA_VM=-server
    MEM_ARGS="-Xms32m -Xmx200m -XX:MaxPermSize=128m"
  ;;
  IBM)
    JAVA_VM=
    MEM_ARGS="-Xms32m -Xmx200m"
  ;;
  Sun)
    JAVA_VM=-server
    MEM_ARGS="-Xms2688M -Xmx2688M -XX:NewSize=336M -XX:MaxNewSize=336M"
   # MEM_ARGS="-Xms32m -Xmx200m -XX:MaxPermSize=128m"

Execute Queue Thread Count

Set the Execute Queue Thread count to be more than the number of CPUs. For example, consider using a value that is twice the number of CPUs. Set this value in either the console or in the /usr/local/bea/user_projects/domains/mydomain/config.xml file:

<ExecuteQueueName="MyExecute Queue" ThreadCount="8" ThreadsIncrease="4"/>

For more information, see “Modifying the Default Thread Count” in “WebLogic Server Performance and Tuning” at:

http://e-docs.bea.com/wls/docs81/perform/WLSTuning.html#1142218

Connection Backlog Buffering

A guideline for setting Connection Backlog Buffering is 8192 for a server with 4 Gbytes of physical memory (which is equivalent to the ConnectionQueue size tuning set in the Sun Java System Web Server 6.1 magnus.conf file).

For more information, see “Tuning Connection Backlog Buffering” in the “WebLogic Server Performance and Tuning” document at:

http://e-docs.bea.com/wls/docs81/perform/WLSTuning.html#1136287