Troubleshooting Low Memory Conditions

This section describes several methods you can use to investigate low memory conditions, including:

• Monitoring the Admin Cache

• Monitoring the Waveset Debug Pages

• Using JConsole

Monitoring the Admin Cache

If you start experiencing slow response times when accessing users or when logging into accounts, you might have exceeded the admin cache size, which results in the admin cache being disabled.

The admin cache is a cache maintained on each server that contains all WSUser objects that are considered to be administrators. Waveset loads this cache when the server starts up, and automatically maintains this cache by watching for WSUser object changes.

Users are generally considered administrators when they have at least one capability and control an object group. If the number of administrator users exceeds the number specified for the cache, Waveset disables the cache, which causes poor system performance.

The default size of the admin cache is 10K. If you have more than 10K administrators, Waveset does not initialize the admin cache. You can adjust this limit by changing the admincache.rowlimit property in Waveset.properties. The cost of increasing admin cache size is the additional use of heap memory. The WSUser objects in the cache consume memory for the entire lifecycle of the server.

Waveset reads the admincache.rowlimit property at server startup, so any change to this property requires the server to be restarted.

You must set this property on each server because Waveset.properties is not shared across servers.

Waveset no longer requires approvers to be administrators, as in earlier versions of the product, which potentially reduces the number of administrators in the cache and improves system performance.

Troubleshooting

Out-of-memory conditions can be difficult to troubleshoot, but you can start by ensuring that sufficient memory was initialized at start up and by checking that Waveset is tracing memory usage levels and alerts.

If you are debugging problems with the admin cache, tracing com.waveset.server.InitAdminCacheThread at level 4 can show the following information:

• What were the start-up timings

• How long did it take to load the admin and auth caches

• What data was loaded

• What happened when you added or edited a user from the Waveset Administrator interface

Tracing Methodology

You can use com.waveset.server.InitAdminCacheThread to trace start-up timings. For example,

Example 5–7 Sample Trace Output

20091124 12:02:56.203 InitAdminCacheThread(0x010dae16)
InitAdminCacheThread#initAdminCache() Info Getting 9 admins from admin cache 
took 0m0.266s 20091124 12:02:56.437 InitAdminCacheThread(0x010dae16) 
InitAdminCacheThread#initAdminCache() Info Loading auth cache for 
9 admins took 0m0.234s

Solution

Changing admincache.rowlimit in Waveset.properties only takes affect after you restart the server. Increasing the cache size to accommodate all of the administrators in the repository may increase server performance if there is enough memory to hold the cache without resulting in excessive JVM garbage collection.

Be aware that this solution does not scale.

Available JVM heap memory is a limitation to consider when using the admincache.rowlimit property to configure cache size, particularly when implementing and tuning an x64 system.

Testing

To test your configuration,

1. Set admincache.rowlimit=xxxxx on VM (where xxxxx denotes the cache size setting).

2. Trace com.waveset.server.InitAdminCacheThread at level 4.

3. Review the trace file output.

   • Verify that the cache was loaded.

   • Confirm that you can disable the admincache.rowlimit setting.

   • Compare the machine's memory usage before and after you disable the admincache.rowlimit setting.

     Ensure that the operating system has sufficient memory available to run effectively. For example, check for excess paging, CPU activity, and so forth.

   • Check the JVM memory usage.

   • Verify how long it takes the cache to load and compare this time with that of a smaller cache, such as 1000.

Monitoring the Waveset Debug Pages

You must have the Debug capability to access and execute operations from the Waveset Debug pages. Administrators and the Configurator are assigned this capability by default.

If you do not have the Debug capability, an error message results.

You can open the following Waveset Debug pages from the Administrator interface to monitor how much memory is being used by your system:

• Host Connection Pool page (debug/Show_ConnectionPools.jsp). View a summary of connection pool statistics (if you are not using a data source), including the pool version, how many connections were created, how many are active, how many connections are in the pool, how many requests were serviced from the pool, and how many connections were destroyed.

  You can also use the Host Connection Pool page to view a summary of the connection pools used to manage connections to the Gateway. You can use this information to investigate low-memory conditions.

• List Cache Cleared (debug/Clear_XMLParser_Cache.jsp ). Clear the cache of recently used XML parsers.

• Private collection pool (debug/Show_JDBC.jsp). View a summary of the cache of Java DataBase Connectivity (JDBC) connections being used by the repository and some resource adapters.

• System Memory Summary page (debug/Show_Memory.jsp). View the used and total memory in the system. You must click the Garbage Collect button to get the most current used memory value.

• System Memory Summary page (debug/Show_Memory2.jsp). View an updated Show_Memory.jsp page that allow you to clear all unused memory in the JVM so you can investigate heap usage.

• User Session Pool Cleared (Clear_User_Cache.jsp). Clear the cached sessions for recently logged in users.

• XML Resource Adapter Caches Flushed and Cleared (Clear_XMLResourceAdapter_Cache.jsp). Clear the cache of the test XML resource adapter.

Using JConsole

Use the Java Monitoring and Management Console (JConsole) to detect low memory and deadlocks. JConsole is a Java Management Extension (JMX) technology-compliant graphical management tool that is co-packaged with JDK 5 (and later).

JConsole accesses the memory system, memory pools, and MBeans garbage collector to provide information about memory use such as memory consumption, memory pools, and garbage collection statistics. In addition, You can use JConsole to monitor MBeans for information about current heap memory use and non-heap memory use.

For information about using JConsole to monitor applications that run on the Java platform, see Using JConsole to Monitor Applications. This document is available from the following URL:

http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html