JDK 1.5 and 1.6 contain a list of PKCS11 providers. The default is sun.security.pkcs11.SunPKCS11 (see the provider list below). Under a heavy load, this provider will generate an Out of Memory Exception (OOME) for the web container and cause the container to crash. At minimum, the following scenarios are impacted:
SSL on these web containers: GlassFish Application Server V2 UR2, WebLogic Server 9.2, and JBoss Application Server 4.2.2 (but not on Sun Java System Web Server 7.0, which uses a different JSS implementation for SSL)
SAML2 signing on Sun Java System Web Server 7 U3
The issue is currently under investigation and might impact other web container platforms not listed above.
Workaround. Remove the SunPKCS11 provider from the provider list in the java.security file for the JVM. For example, if the security provider section in your java.security file (found in JDK_Path/jre/lib/security/) looks like:
security.provider.1=sun.security.pkcs11.SunPKCS11 \ ${java.home}/lib/security/sunpkcs11-solaris.cfg security.provider.2=sun.security.provider.Sun security.provider.3=sun.security.rsa.SunRsaSign security.provider.4=com.sun.net.ssl.internal.ssl.Provider security.provider.5=com.sun.crypto.provider.SunJCE security.provider.6=sun.security.jgss.SunProvider security.provider.7=com.sun.security.sasl.Provider
Change it to:
security.provider.1=sun.security.provider.Sun security.provider.2=sun.security.rsa.SunRsaSign security.provider.3=com.sun.net.ssl.internal.ssl.Provider security.provider.4=com.sun.crypto.provider.SunJCE security.provider.5=sun.security.jgss.SunProvider security.provider.6=com.sun.security.sasl.Provider
Note. This workaround can lower your performance because the provider used now is not as optimized as the SunPKCS11 provider. It also prevents you from using hardware security tokens if the SunPKCS11 provider is required.