3858: Out of memory exceptions occur under heavy load with JDK 1.5 and 1.6 SunPKCS11 provider

JDK 1.5 and 1.6 contain a list of PKCS11 providers. The default is sun.security.pkcs11.SunPKCS11 (see the provider list below). Under a heavy load, this provider will generate an Out of Memory Exception (OOME) for the web container and cause the container to crash. At minimum, the following scenarios are impacted:

• SSL on these web containers: GlassFish Application Server V2 UR2, WebLogic Server 9.2, and JBoss Application Server 4.2.2 (but not on Sun Java System Web Server 7.0, which uses a different JSS implementation for SSL)

• SAML2 signing on Sun Java System Web Server 7 U3

The issue is currently under investigation and might impact other web container platforms not listed above.

Workaround. Remove the SunPKCS11 provider from the provider list in the java.security file for the JVM. For example, if the security provider section in your java.security file (found in JDK_Path/jre/lib/security/) looks like:

security.provider.1=sun.security.pkcs11.SunPKCS11 \
   ${java.home}/lib/security/sunpkcs11-solaris.cfg
security.provider.2=sun.security.provider.Sun
security.provider.3=sun.security.rsa.SunRsaSign
security.provider.4=com.sun.net.ssl.internal.ssl.Provider
security.provider.5=com.sun.crypto.provider.SunJCE
security.provider.6=sun.security.jgss.SunProvider
security.provider.7=com.sun.security.sasl.Provider

Change it to:

security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=com.sun.net.ssl.internal.ssl.Provider
security.provider.4=com.sun.crypto.provider.SunJCE
security.provider.5=sun.security.jgss.SunProvider
security.provider.6=com.sun.security.sasl.Provider

Note. This workaround can lower your performance because the provider used now is not as optimized as the SunPKCS11 provider. It also prevents you from using hardware security tokens if the SunPKCS11 provider is required.