test

Sun Java System Access Manager 7.1 Performance Tuning and Troubleshooting Guide

Chapter 7 Advanced Performance Tuning

After conducting basic performance tuning and following the best practices recommendations described in previous chapters, you may still encounter performance issues. This chapter helps you troubleshoot the most common Access Manager performance issue. Topics in this chapter include:

Tuning the LDAP Connection Pool and LDAP Configurations

The amtune script provided by AccessManager recommends parameter values for the following three LDAP connection pools:

But the script does not actually tune the LDAP connection pools for you. You need to make the changes manually. In addition, in deployments with a subrealm, you must also manually tune the subrealm's connection pools. Just like the root realm, each sub-realm can have its own user authentication LDAP connection pool and data store LDAP connection pool. You must manually tune these as well.

You can modify one or more of the three LDAP connection pool configurations . In each configuration, the recommended values are MIN=8 and MAX=32. Under some conditions, you can increase the MAX value up to 64. The following sections describe how to manually tune the connection pools:

To Tune the User Authentication LDAP Configuration

You can modify the settings on one of the following depending upon the module you use for user authentication.

LDAP Authentication Module

This module is used only to authenticate the user. In the Access Manager console, under Configuration, click Authentication > Core.

Data Store Authentication Module

When the Data Store is as the authentication module, the Data Store LDAP connection pool settings are used. No additional Authentication connection pool settings are used.

To Tune the Data Store LDAP Configuration

The Data Store LDAP Configuration is used for retrieving user profiles and can also be used for authentication. By default, Access Manager 7.1 supports two types of Data Store plug-ins: AMSDK and LDAPv3. If the Data Store Authentication module is used for authentication (see above), then the recommended Data Store LDAP configuration settings are MIN=8 and MAX=64. You can modify the settings on one of the following depending upon the Data Store plug-in you use:

AMSDK Configuration

The AMSDK LDAP configuration is stored in the serverconfig.xml file under the Access Manager config directory. The server group name is default.

LDAPv3 Configuration

To modify the LDAPv3 Configuration, in the Access Manager console, under Access Control, click Realm > DataStore.

To Tune the Access Manager Configuration Store and SMS LDAP Configuration

The Service Management (SMS) LDAP Configuration is used for storing and retrieving all Access Manager configuration and Policy Service configuration. The SMS LDAP Configuration is stored in the serverconfig.xml file under the Access Manager config directory. The server group is sms.

  1. Start by setting all the connection pool configurations with MIN=8 and MAX=32.

  2. If you must make adjustments based on performance test results, adhere to the following requirements:

    • The MIN value should be at least 8.

    • The MAX value for any pool should not be greater than 64. The MAX value of 32 is enough for most typical deployments.

    Special requirements are outside the scope of this document.

  3. After following steps 1 and 2, if low throughput or low response times persist, then try the following solutions:

    • Verify that the Directory Server instance is not at 100% CPU usage. If the Directory Server instance is at 100% and the throughput is still low, revisit the indexing on the Directory Server entries. Be sure that Directory Server indexing is configured properly.

    • Run load tests to verify that Access Manager login is not causing performance to slow down. First run the tests with logging enabled, and then run the tests with logging disabled. If you find that logging is causing low response time, then you can tune the logging service through the Access Manager console. See Logging in Sun Java System Access Manager 7.1 Administration Reference.

Resolving Memory Issues

The amtune script automatically tunes all memory related parameters. In most deployments, this is sufficient. However, occasionally the amtune tuning may not be sufficient and you may run into memory issues. Memory issues manifest themselves through excessively frequent garbage collection (GC) operations or frequent “Out of Memory” errors.

To resolve memory related issues, tune the following parameters:

All the parameters listed above can be tuned by editing the AMconfig.properties file which is located in under /etc/opt/SUNWam/config if installed using the JES installer. If the Access Manger is installed using the single WAR, than AMConfig.properties is located in directory you specified when you configured the WAR file.

The minimum required JVM heap size for Access Manager is 1024 mb.

Tuning com.iplanet.am.session.maxSessions

The tuning of this property entire depnds on the JVM Heap size configured in the web container where the Access Manager is deployed. The minimum required JVM heap size for Access Manager is 1024 mb and the # of sessions supported for 1024mb is 12000 and every additional 512mb can support upto 18000 sessions.

Tuning com.iplanet.am.sdk.cache.maxSize

The sdk cache size should be same as the value set for com.iplanet.am.session.maxSessions.

Tuning com.iplanet.am.notification.threadpool.threshold

This is the Notification Queue size. The Notification Queue size should be less than or equal to 30% of the Max Sessions.

The following table lists sample settings for the parameters listed above based on the rules described above.

Maximum JVM Heap Size 

Maximum Active Sessions 

SDK Cache Size 

Notification Queue Size 

1024mb 

12000 

12000 

4000 

1536mb 

30000 

30000 

10000 

2048mb 

48000 

48000 

16000 

2560mb 

66000 

66000 

22000 

3136mb 

90000 

90000 

30000 

The above settings may not be suitable for certain deployments. When the number of user attributes retrieved is large, the SDK cache size will increase. Similarly, if the Extra Session properties are set, the Session size will increase.

In these cases, use one of the following options to solve the memory related issues:

To Tune the Notification Threadpool Size

Set the value of com.iplanet.am.notification.threadpool.size based on number of CPUs and based on the purgedelay value. See To Tune the Purge Delay Settingsfor related information.

To Tune the Purge Delay Settings

The purgedelay property is used to keep the session in memory in a timed-out state after the session has timed out. If the value is set to 0, then the session is removed from memory immediately. If the value is greater than zero, then the session is maintained in the memory until the purgedelay time elapses.

The property com.iplanet.am.session.maxSessions describes the maximum number of active sessions that the system will allow. When the purgedelay is set to 0, the total number of sessions (active sessions and timed-out sessions) in memory will be equal to the value set for com.iplanet.am.session.maxSessions. If purgedelay is greater than 0, then the total number of sessions (active and timed-out sessions) in memory can be greater than active sessions. The difference will be based on three factors: the purgedelay time , the percentage of timed-out sessions, and the authentication rate. Therefore, when purgedelay is greater than zero, the maximum active sessions value should be reduced accordingly.

The simple way to do this is to look in the AccessManager 7.1sp1 session stats file. The amMasterSessionTable shows the current and peak values for maxSessions (active sessions + timed-out sessions) and maxActive (only active sessions) sessions in memory . Based on this information, the maxSessions value in the stats file limit should not exceed the 90000 limit for a JVM heap size of 3136mb. When the purgedelay is set to 0, only one notification is sent when a session is removed from memory. When the purgedelay is greater than 0, then there will be two notifications for each timed-out session. The number of notifications for timed-out sessions are increased, and now more notification threads are needed. So the notification thread pool size should also be increased.

Performance Tuning Questions and Answers

Question:

How can I improve authentication performance against any LDAP v3 data repository?

Answer:

If the Profile Ignored option is selected in the Access Manager console (go to Realm > Authentication >Advanced Properties), performance may improve. However, improved performance is not guaranteed because the Profile Ignored option prevents applications and policy agents from retrieving the user's profile attributes. The amtune script automatically tunes the LDAP connection pool for the Access Manager root realm which points to the configuration Directory Server instance. But the amtune script will not tune the subrealm you created, the subrealm where the LDAP v3 data repository is configured. You may need to manually tune the LDAP connection pool. After tuning the LDAP connection pool, if poor performance persists, troubleshoot the LDAP v3 repository itself.

Consider limiting the time you spend troubleshooting authentication performance issues. Authentication usually contributes only a small portion of the overall system overhead. Authorizations tend to be slower and a lot more frequent than other processes. But each user session involves only one authentication and multiple authorizations.

Question:

How do I set the JVM heap sizes and other JVM option tuning parameters for a Distributed Authentication UI web application?

Answer:

The web container that will load the Access Manager Distributed Authentication UI web application should have the same heap sizes and the same JVM tuning settings as the web container that runs Access Manager. You can use amtune-ws7, amtune-ws61 or amtune-as8 which come with Access Manager 7.1. You don't need much CPU usage as for Access Manager server machines. It is hard to tell by what ratio one can reduce the number of CPU usage on a Distributed Authentication UI machine. The ratio can be 1:4 or less. Run some load tests for a specific scenario to determine a good ratio.

The reason why a Distributed Authentication UI web container needs the same JVM heap sizes and garbage collection tuning parameters as those for the Access Manager server web containers is that amclientsdk maintains the same number of Access Manager sessions on the client side as on the Access Manager server itself. A Request for Enhancement (CR 6465831) has been filed for removal of the Access Manager sessions in Distributed Authentication UI amclientsdk deployments.

Question:

What is the impact of checking notenforced_list for a set of URIs or URLs on J2EE policy agents?

Answer:

The performance impact of checking notenforced_list is negligible. In general, having a notenforced_list of commonly requested and static content improves the overall system performance.

Question:

What is the impact of using the SSL , for example the NSS library version that comes with JES 4 installer, on the performance of Access Manager 7.0 deployed on Niagara boxes such as T1000 or T2000?

Answer:

If Access Manager 7.0 was installed using JES 4 installer and its default SSL libraries, then the markedly improved performance that comes with NSS 3.11 may not be present and won't be used. Use the NSS libraries version 3.11 or higher when Access Manager 7.0 is deployed on Niagra T1000 or T2000 systems. Go to the Sunsolve web site for downloading the NSS libraries. Note that starting with JES 5 and Access Manager 7.1, the NSS libraries have been upgraded to a version higher than 3.11.

Question:

Why is it so slow to create or delete users if I use a program based on amsdk, but much faster if I use the ldapmodify command?

Answer:

If the same policy is modified for each user, the XML parsing and processing must occur for every user. So you should group as many users as possible with the same one policy, and then add the users to that policy. You should use the same LDAP group or role for as many users as possible in an organization.

Be sure that a policy is not modified or updated for each user. Modifying a policy is an expensive operation since the policy is stored as XML data.

Question:

Is Sun Java Message Queue tuning necessary when session failover is configured for Access Manager?

Answer:

In most deployments using Access Manager session failover, Java Message Queue tuning only requires setting adequate JVM heap and stack sizes. See the Sun Java System Message Queue 2005Q1 Administration Guide at http://docs.sun.com/app/docs/coll/MessageQueue_2005Q1 for further information.

Question:

When the amtune script tunes the Directory Server with the recommended values, an onscreen message says the tuning parameters such as minConnPool and maxConnPool in serverconfig.xml are dependent on the number of Access Manager instances and other factors. How exactly should I tune the Directory Server with these factors taken into account?

Answer:

Values recommended by the amtune script for minConnPool and maxConnPool are per Access Manager server instance. The parameters are stored in /etc/opt/SUNWam/config/serverconfig.xml. The recommended values are based on the following assumptions:

When multiple Access Manager instances exist, the total number of persistent LDAP connections may be too high for the Directory Server to handle. Each Access Manager instance establishes its own pool of the same size. Memory allocation is also on the high side if the user data is not stored there. The amtune script assumes the user data is stored together with Access Manager configuration data in the Directory Server.

For example, consider the typical real-world deployment scenario illustrated in the document Deployment Example 1: Access Manager 7.1 Load Balancing, Distributed Authentication UI, and Session Failover. This deployment differs from the amtune script assumptions in the following ways:

First, if you have a large number of Access Manager instances, you can reduce the recommended pool size for the configuration Directory Server. This only applies when you have large number of Access Manager instances. When you have only two or three Access Manager instances, it may not be necessary to reduce the pool size.

Secondly, you can significantly reduce the memory allocation to the configuration Directory Server . The configuration data is minimal with usually only a few thousands entries. Reducing the memory allocation is particularly important if the configuration Directory Server runs on the same host as the user data Directory Server. You do not want the smaller configuration Directory Server to compete with the larger user data Directory Server for the system memory.

Thirdly, be sure to tune the user data Directory Server. This directory contains a large data set. You can use the amtune recommended Directory Server tuning changes as a starting point. For more information, see step 3 of Related Systems Tuning.

Ultimately, you have to look at your directory data and tune it specifically. This is the standard Directory Server tuning procedure. See the DS Performance Tuning Guide.

Finally, the amtune script does not tune the LDAPv3 data store connection pool which is used by Access Manager to access the user data Directory Server. You have to manually tune the data store connection pool. See step 4 of Related Systems Tuning.

Question:

Where do I find specific performance tuning guidelines for Access Manager implementations on the T2000 platform?

Answer:

The Access Manager amtune script does the automatic tuning specifically for the T2000 platform. No manual tuning is necessary. The following is the tuning specific to T2000, done automatically by the amtune script.

Sun Fire CoolThreads technology servers, specifically Sun Fire T1000 and Sun Fire T2000 servers, contain a single Ultrasparc T1 chip or processor. The T1 processor is a unique design of 8 individual processing units, called cores, sharing one on-chip interconnection. It is somewhat like an 8-way system on a single chip.

Each core supports 4 hardware threads of execution. These hardware threads are scheduled on the core processing unit in round-robin order. A different software thread can run on each one of these hardware threads. Thirty-two software threads can run in parallel on a single T1 processor.

You can determine the number of cores by dividing the number of hardware threads (run psrinfo -v) by 4. The T1000 and T2000 can have a maximum of 4 hardware threads per core. So the number of cores is usually 6 (a 24 thread system) or 8 (a 32-thread system).

The only JVM parameter that would be different for Chip Multi-threading (CMT) servers is the following parameter 


-XX:ParallelGCThreads=N

By default, if the parameter is not set, the value of ParallelGCThreads would be the same number as the number of hardware threads (either 24 or 32) on the T1000 and 2000. This is unnecessarily high. The amtune script today automatically sets the number of these parallel GC threads to be equal the number of cores in a T1000 or T2000 box.

For more information, see the document Java Tuning Whitepaper at http://java.sun.com/performance/reference/whitepapers/tuning.html#section4.2.1 .

More Resources

For more information on performance tuning and troubleshooting, see the following resources: