J Troubleshooting OPSS

This appendix describes common problems that you may encounter when you configure or use OPSS and explains how to solve them.

It includes the following sections:

The OPSS Diagnostic Framework

OPSS includes a framework that helps you reduce the resolution time of problems. This framework allows you to extract internal states of a domain and dump that information to a file. The framework provides a number of tests to generate dumps that record the characteristics of domain servers and services in that domain.

The OPSS diagnostic framework provides the following tests:

  • Configuration test

    • oracle.security.jps.diag.test.JpsConfigTest

  • Connectivity tests

    • oracle.security.jps.diag.test.JpsContainerAuthenticationLdapConnectivityTest

    • oracle.security.jps.diag.test.JpsIdentityStoreUserRoleApiLdapConnectivityTest

  • Credential store tests

    • oracle.security.jps.diag.test.JpsCredentialStoreConfigTest

    • oracle.security.jps.diag.test.JpsCredentialStoreTest

  • Identity store tests

    • oracle.security.jps.diag.test.JpsIdentityStoreIDSSearchTest

    • oracle.security.jps.diag.test.JpsIdentityStoreLibOvdConfigTest

    • oracle.security.jps.diag.test.JpsIdentityStoreUserRoleApiSearchTest

  • Keystore tests

    • oracle.security.jps.diag.test.JpsKeyStoreTest

    • oracle.security.jps.diag.test.JpsKeyStoreConfigTest

Running a Test

To call a test, use the executeDump command with the following syntax:

executeDump(name='opss.diagTest', outputFile='dumpLocation', args={'testtname':'testName'})

where:

  • The first argument, name='opss.diagTest', is fixed.

  • outputFile specifies the location where the dump is generated relative to where the command is run.

  • All character strings in the third argument are fixed except for the testName string, which you set to one of the tests. If you use the wild card *, then all the tests are run in the following order:

    oracle.security.jps.diag.test.JpsConfigTest
    oracle.security.jps.diag.test.JpsContainerAuthenticationLdapConnectivityTest
    oracle.security.jps.diag.test.JpsCredentialStoreConfigTest
    oracle.security.jps.diag.test.JpsCredentialStoreTest
    oracle.security.jps.diag.test.JpsIdentityStoreIDSSearchTest
    oracle.security.jps.diag.test.JpsIdentityStoreLibOvdConfigTest
    oracle.security.jps.diag.test.JpsIdentityStoreUserRoleApiLdapConnectivityTest
    oracle.security.jps.diag.test.JpsIdentityStoreUserRoleApiSearchTest
    oracle.security.jps.diag.test.JpsKeyStoreConfigTest
    oracle.security.jps.diag.test.JpsKeyStoreTest

Example of Use

The following sequence illustrates a situation where you use the diagnose framework:

  1. You receive a customer problem.

  2. From the logged failure, you determine that the problem has to do with a connection (for example).

  3. You run one or more tests in the domain in question, such as oracle.security.jps.diag.test.JpsContainerAuthenticationLdapConnectivityTest:

    1. Connect to Oracle WebLogic Server:

      wls:/offline> connect('adminuser', 'adminpass', 't3://localhost:7001')
      
    2. Call a test, such as the following:

      >executeDump(name='opss.diagTest', outputFile='myDumpTest', 
      args={'testname': 'oracle.security.jps.diag.test.JpsContainerAuthenticationLdapConnectivityTest'})
      

      The test generates a dump at the specified location. The dump includes the following lines:

      LDAP authenticator : OID
      	host : myHost.com
      	port : 7066
      	principal : cn=orcladmin
      	password : ********
      	Is SSL? false
      	Testing LDAP connection to ldap://myHost.com:7066 with principal cn=orcladmin.
      	JNDI settings: 
      	java.naming.provider.url = ldap://myHost.com:7066
      	java.naming.factory.initial = com.sun.jndi.ldap.LdapCtxFactory
      	com.sun.jndi.ldap.connect.timeout = 5000
      	java.naming.security.principal = cn=orcladmin
      	java.naming.security.authentication = simple
      	java.naming.security.credentials = ********
      	Failed to establish LDAP connection to ldap://myHost.com:7066. 
      The stack trace: 
      	javax.naming.CommunicationException: myHost.com:7066 [Root exception is java.net.ConnectException: Connection refused]
      	at com.sun.jndi.ldap.Connection.<init>(Connection.java:214) …
      
    3. Inspect the dump for errors and failures. In this example, the text in bold indicates that the connection to the LDAP provider could not be established.

Diagnosing Security Errors

In addition to the OPSS diagnostic framework, use loggers to diagnose and solve a variety of security issues as described in the following sections:

About OPSS Loggers

The following sections describe the log files and loggers that OPSS supports and explains how to configure, set logger levels, and view log files with Fusion Middleware Control and WebLogic Scripting Tool (WLST):

About Diagnostic Log Files

Each server instance in a domain writes all OPSS exceptions raised by applications to a server log file in the file system of the local host computer.

By default, this log file is located in the logs directory below the server instance root directory. The names of these log files have the following format: ServerName-diagnostic.logxxxxx, where xxxxx denotes an integer between 1 and 99999.

Servers write security-related errors to diagnostic files. Server-related security errors, such as exceptions raised by issues with a subject or principal, or errors that may occur while migrating or reassociating domain security data, are written in the Administration Server diagnostic log. Application-related security errors, such as exceptions raised by application-specific policies or credentials, are written in the diagnostic log of the Managed Server where the application is running.

By default and similar to diagnostic log files, server log files are located in the logs directory below the server instance root directory. Domain log files are located in the logs directory below the Administration Server root directory. The names of these log files have the format ServerName.logxxxxx and domain.logxxxxx, where xxxxx denotes an integer between 1 and 99999.

The domain logs duplicate some of the messages in server logs, and they help determine the server on which a fault has occurred in domains with a large number of servers.

The generation of a new log file is determined by file size: when a log file exceeds a specified size, the system generates a new one with a name whose integer suffix is increased by 1.

See also:

Server Log Files and Domain Log Files in Configuring Log Files and Filtering Log Messages for Oracle WebLogic Server.

What Is the WebLogic Diagnostics Framework? in Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.

Adding WebLogic Logging Services to Applications Deployed on Oracle WebLogic Server.

Managing Log Files and Diagnostic Data in Administering Oracle Fusion Middleware.

Offline WLST Loggers

Logging for online WLST commands is automatic, but it is not for offline commands. When you use an offline command such as the migrateSecurityStore command, enable logging by starting the Java Virtual Machine (JVM) with the following system properties:

  • wlst.offline.log, with one of the following values: <filename>, stdout, sterr, or disable. If unspecified, then the log files are located in the $MW_HOME/logs directory.

  • wlst.offline.log.priority, with one of the following values: OFF, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, ALL, debug, info, warn, error, fatal.

Loggers by Service

The following sections list the loggers available per service:

Logging Authorization

You can enable and disable loggers without having to stop and restart servers. Typically, you set the level of loggers to TRACE:32.

OPSS provides several loggers that help troubleshooting runtime authorization failures:

  • oracle.security.jps.util.JpsAuth - Logs the start and return of checkPermission.

  • oracle.security.jps.trace.logger - Logs information about application roles, permissions, targets, principals, and granted and denied policies. This logger generates a large output. Use it to debug a single use case only.

  • oracle.jps.authorization - Logs messages during runtime authorization.

  • oracle.jps.common - Logs messages in OPSS common functional areas, such as service management and user management.

  • oracle.security.jps.dbg.logger - Logs messages that are used to debug the JpsFilter filter permission checks.

  • oracle.security.jps.az.internal.runtime.policy.AbstractPolicyImpl - Logs messages for the oracle.security.jps.az.internal.runtime.policy.AbstractPolicyImpl class.

  • oracle.security.jps.internal.policystore.JavaPolicyProvider - Logs messages for the oracle.security.jps.internal.policystore.JavaPolicyProvider class.

Logging Audit

This section explains how to interpret audit log messages and how to use them to diagnose component failures. Log files are located at:

DomainName/servers/$SERVER_NAME/logs/$SERVER_NAME-diagnostic.log

Table J-1 lists the available diagnostic log files.

Table J-1 Log Files for Audit Diagnostics

Component Log Location Configuring Loggers

Java EE Components

DomainName/servers/$SERVER_NAME/logs/$SERVER_NAME-diagnostic.log

oracle.security.audit.logger (See instructions below)

OPMN Components

See Logging Audit.

See Logging Audit.

Startup Class Audit Loader

DomainName/servers/$SERVER_NAME/logs/$SERVER_NAME-diagnostic.log

oracle.security.audit.logger (See instructions following this table)

OPMN Audit Loader

$ORACLE_INSTANCE/diagnostics/logs/OPMN/opmn/rmd.out

java.util.logging.config.file system property can be set to the file that contains the log level for OPMN Audit Loader

Config/Proxy Mbeans

DomainName/servers/$SERVER_NAME/logs/$SERVER_NAME-diagnostic.log

oracle.security.audit.logger (See instructions below)

Audit Schema Support

Oracle Fusion Middleware Repository Creation Utility log location (Default is $ORACLE_HOME/rcu/log/)RCU_LOG_LOCATION can be set to change this location

Repository Creation Utility log level. Default is ERROR.

Configuring Audit Loggers

Use Fusion Middleware Control to configure the oracle.security.audit.logger logger, and to view loggers. For information about log files, see Managing Log Files and Diagnostic Data in Administering Oracle Fusion Middleware.

Audit error messages are numbered IAU-XXX.

To configure audit logging for system components managed with OPMN, edit the auditconfig.xml file to specify the log directory location to which the component's audit logs should be written:

<LogsDirectory>
   <MaxFileSize></MaxFileSize>
   <Location>/tmp/audit/auditlogs</Location>
</LogsDirectory>
Logging the User and Role API

To trace OPSS User and Role API calls, set the oracle.idm.userroleapi logger to TRACE:32.

Logging Other Components

Additionally, OPSS provides the following loggers:

oracle.jps.deployment logs issues with OPSS artifacts packed with the application when you deploy the application.

oracle.jps.openaz logs issues with PEP API calls. Setting oracle.jps.openaz.level to FINEST, logs information about submitted requests - identity, resource, action, context - and authorization results.

oracle.jps.attribute logs issues with the OPSS Attribute service.

System Properties

Use the following properties to configure debugging at server startup:

  • jps.auth.debug, logs permission checks that fail only. To disable permission check messages, set this property to false. By default, it is true.

  • jps.auth.debug.verbose, logs permission checks that fail. Logs more information than jps.auth.debub. By default, it is false.

  • DebugOPSSPolicyLoading, a flag that monitors the progress and setting of the policy provider.

  • java.security.debug=policy, the standard Java security debug flag that produces print information about policy files as they are parsed, including their location in the file system, the permissions they grant, and the certificates they use.

Edit the setDomainEnv.sh script and add the desired property to the EXTRA_JAVA_PROPERTIES system property. This change requires that you restart the server.

Caution:

Setting a high logging output may cause stuck threads, especially when file loading takes place. To avoid this situation, change the timeout value that WebLogic Server uses to mark a thread as stuck to a higher value.

Other System Properties

Additional system properties that may further help you debugging are the following:

  • oracle.security.jps.log.for.approle.substring, logs the name of an application role that contains a specified substring. If the substring to match is unspecified, then it logs all application role names.

  • oracle.security.jps.log.for.permeffect, logs a grant that was granted or denied. If the value is unspecified, then it logs all grants (regardless whether they were granted or denied).

  • oracle.security.jps.log.for.permclassname, logs the name of the permission class that matches exactly a specified name. If the name to match is unspecified, then it logs all permission class names.

  • oracle.security.jps.log.for.permtarget.substring, logs the name of a permission target that contains a specified substring. If the substring to match is unspecified, then it logs all permission targets.

  • oracle.security.jps.log.for.enterprise.principalname, logs the name of the principal (enterprise user or role) that matches exactly a specified name. If the name to match is unspecified, then it logs all principal names.

Examples of Use

The following examples illustrate typical settings of system properties.

  • To log all application role names that contain the myAppRole string:

    -Doracle.security.jps.log.for.approle.substring=myAppRole
    
  • To log all denied permission checks:

    -Doracle.security.jps.log.for.permeffect=deny
    
  • To log all granted permission checks:

    -Doracle.security.jps.log.for.permeffect=grant
    
  • To log all granted or denied permission checks, do not set oracle.security.jps.log.for.permeffect.

  • To log all permission checks that match exactly java.util.PropertyPermission:

    -Doracle.security.jps.log.for.permclassname=java.util.PropertyPermission
    
  • To log all target names that contain the p.mon string:

    -Doracle.security.jps.log.for.permtarget.substring=p.mon
    
  • To log all authorizations involving the manager principal name:

    -Doracle.security.jps.log.for.enterprise.principalname=manager
    
  • To log application role names that match a substring or principal names that match a string, use both oracle.security.jps.log.for.approle.substring and oracle.security.jps.log.for.enterprise.principalname.

Understanding Log Entries

Your understanding of log errors is crucial to isolate and solve an error. This section explains how to interpret the contents of a diagnostic log file, such as the following:

[2009-01-07T09:15:02.393-08:00] [AdminServer] [ERROR] [JPS-00004] [oracle.jps.admin] 
[tid: [ACTIVE].ExecuteThread: '3' for queue: 'weblogic.kernel.Default
(self-tuning)'] [userId: weblogic] [ecid: 0000Hum5kxw7MAn54nU4Ui19PD8S000005,0]
Unable to add principal to the application role. Reason: Principal
"abc.xxx@myComp.com" is already a member of the application role
"BPMWorkflowAdmin"[[
java.security.PrivilegedActionException:
oracle.security.jps.service.policystore.PolicyObjectAlreadyExistsException:
Unable to add principal to the application role. Reason: Principal "abc.xxx@myComp.com" is already a member of the application role
"BPMWorkflowAdmin"
        at java.security.AccessController.doPrivileged(Native Method)
        at oracle.security.jps.mas.mgmt.jmx.policy.JpsApplicationPolicyStoreImpl.
addRemoveMembersToRole(JpsApplicationPolicyStoreImpl.java:408)
        at oracle.security.jps.mas.mgmt.jmx.policy.JpsApplicationPolicyStoreImpl.
addMembersToApplicationRole(JpsApplicationPolicyStoreImpl.java:385)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

The meaning of the fields in the preceding message is:

  • [2009-01-07T09:15:02.393-08:00]

    Identifies the date and time when the error was logged.

  • [AdminServer]

    Identifies the name of the server where the error occurred.

  • [JPS-00004]

    Identifies the error code and hints to the kind of error that occurred. For a complete list of JPS error codes, see Error Messages.

  • [oracle.jps.admin]

    Identifies the logger category. The subcategories of oracle.jps (such as admin) indicate the kind of error that occurred. They include the following:

    • common - generic errors

    • upgrade - upgrade errors

    • config - configuration errors

    • deployment - deployment errors

    • authentication - login module errors (Java SE applications only)

    • idmgmt - identity store errors

    • credstore - credential store errors

    • authorization - policy store errors at runtime

    • policymgmt - policy store management errors

    • admin - JMX and WLST errors

  • [tid: [ACTIVE].ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)']

    Identifies the thread where the error occurred.

  • [userId: weblogic]

    Identifies the user that performed the operation that generated the error.

  • [ecid: 0000Hum5kxw7MAn54nU4Ui19PD8S000005,0]

    Identifies the execution context ID. Correlates and traces sequence of events. The execution context ID (ECID) provides information about the flow across processes, such as, from a request, to WebLogic Server, to an Oracle Internet Directory server.

  • Unable to add principal to the application role. Reason: Principal abc.xxx@myComp.com is already a member of the application role BPMWorkflowAdmin

    Identifies the reason why the error was logged.

  • java.security.PrivilegedActionException: oracle.security.jps.service.policystore.PolicyObjectAlreadyExistsException: Unable to add principal to the application role. Reason: Principal abc.xxx@myComp.com is already a member of the application role BPMWorkflowAdmin

    Identifies the exception that was raised and the reason for it.

Troubleshooting Reassociation and Migration

The following sections describe issues with data security operations:

Reassociation Failure

This section explains three reasons why reassociating from a file to an LDAP store may fail.

Symptom 1- Error Code 32

Reassociation fails and an error like the following is logged in the MyServerName.diagnostic.log file:

[LDAP: error code 32 - No Such Object]
Authentication to LDAP server ldap://myServer.com:3060 is unsuccessful. 

Diagnosis 1

This error means that the specified node does not exist in the LDAP server.

It is required that the root node you specified exist in the LDAP before you start reassociating the store.

Solution 1

Verify that the data you enter in the JPS Root DN text field matches the name of a node in the target LDAP directory, and then rerun the reassociation.

Symptom 2- Error Code 68

Reassociation fails and an error like the following is logged in the serverName.diagnostic.log file:

Authentication to LDAP server ldap://myServer.com:3060 is successful.
Starting to migrate policy store...
Set up security provider reassociation successfully.
Checked and seeded security store schema successfully.
null
[LDAP: error code 68 - Object already exists]:cn=SystemPolicy,cn=domain1,cn=JPSContext,cn=nb_policy
Error occurred while migrating LDAP based policy store. 

Diagnosis 2

This error indicates that the name specified in the WebLogic Domain Name text field is a grandchild of the JPS Root DN node in the target LDAP directory.

It is required that the cn not be a descendant of the root node.

Solution 2

Verify that the name you enter in the WebLogic Domain Name text field does not match the name of a grandchild of the specified JPS Root DN node, and rerun the reassociation.

Symptom 3

Reassociation, carried out with Fusion Middleware Control, fails and an error like the following is logged in the serverName.diagnostic.log file:

[2009-01-21T10:09:24.326-08:00] [AdminServer] [ERROR] [] [oracle.jps.admin] [tid
: [ACTIVE].ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)
'] [userId: weblogic] [ecid: 0000HvuOTpe7q2T6uBADUH19Tpyb000006,0] Unable to rem
ove the principal from the application role. Reason: Principal "Managers" is not
a member of the application role "test-role"[[
java.security.PrivilegedActionException: oracle.security.jps.service.policystore
.PolicyObjectNotFoundException: Unable to remove the principal from the applicat
ion role. Reason: Principal "Managers" is not a member of the application role "
test-role"
       at oracle.security.jps.mas.mgmt.jmx.policy.JpsApplicationPolicyStoreImpl
.addRemoveMembersToRole(JpsApplicationPolicyStoreImpl.java:408)... 

Diagnosis 3

This error points to some problem with the test-role application role.

Ensure that when you enter data to perform reassociation with Fusion Middleware Control, you use the button Test LDAP Authentication immediately after you have completed entering all required values to connect to the target LDAP server. This test catches any problems with those values before reassociation begins.

Solution 3

In our example, a quick inspection of system-jazn-data.xml reveals that the test-role role is used by a policy, but it is not defined. The following example illustrates where the required data is missing:

<application>
    <name>myApp</name>
        <app-roles>
 <--! test-role should have been defined here -->
        </app-roles>
        <jazn-policy>
            <grant>
                <grantee>
                    <principals>
                       <principal>
                         <class>
oracle.security.jps.service.policystore.ApplicationRole</class>
                         <name>test-role</name>
                         <guid>66368900E7E511DD9F62F9ADA4233FE2</guid>
                       </principal>
                     </principals>...

To solve this particular error, (a) fix system-jazn-data.xml by inserting the definition of the application test-role, (b) revert to file stores with the fixed file, and (c) rerun the reassociation.

Symptom 4 - Audit Store is Not Reassociated

This note applies to release 11.1.1.6.0. Reassociation with Oracle Enterprise Manager Fusion Middleware Control (Fusion Middleware Control) completes successfully, but you do not see the audit store in the target store. This is because in that release the audit store could be reassociated only with the reassociateSecurityStore WLST command.

Solution 4

In the original environment, run the reassociateSecurityStore command with a different jpsroot node. This effects an LDAP-to-LDAP directory reassociation and data (including audit data) gets migrated to the new node.

Unsupported Schema

This section explains a reason why reassociation to an LDAP server may fail.

Symptom

Reassociating the security store to an LDAP repository fails and the Administration Server log reports an error like the following:

[2011-02-09T07:01:13.884-05:00] [AdminServer] [ERROR] [] [oracle.jps.admin] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 41050d66ef2ec40b:-4c1fb689:12e06cc7b6c:-8000-00000000000001e1,0] Schema seeding failed, check the server type of the given ldap url.[[
oracle.security.jps.JpsException: Error Modifying JPS Schema,  Record: dn: cn=schema
changetype: modify
delete: objectclasses
objectclasses: ( 2.16.840.1.113894.7.2.2 NAME 'orclContainer' SUP ( top ) MUS
 T ( cn ) MAY ( orclVersion $ orclServiceType ) )
-
: [LDAP: error code 32 - No Such Object]:cn=schema

Diagnosis

The error indicates that the schema of the target LDAP repository is not supported.

Solution

Update the target LDAP repository to one supported and then try reassociating again. The version of Oracle Internet Directory must be 10.1.4.3 or later. For a list of supported versions, see Using an LDAP Security Store.

Missing Policies in Reassociated Security Store

Symptom

You successfully reassociated a file store to an LDAP store, but codesource policies are missing in the target store.

Diagnosis

At runtime, the server reports a stack trace like the following:

<BEA-000000> <JspServlet: initialization complete>
####<May 4, 2009 8:32:50 AM PDT> <Error> <HTTP> <ap626atg> <WLS_Spaces>
<[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default
(self-tuning)'> <<WLS Kernel>> <> <> <1241451170341> <BEA-101020>
<[ServletContext@20193148[app:webcenter module:/webcenter path:/webcenter
spec-version:2.5]] Servlet failed with Exception
java.security.AccessControlException: access denied
(oracle.security.jps.service.policystore.PolicyStoreAccessPermission
context=APPLICATION,name=webcenter getApplicationPolicy)
        at
java.security.AccessControlContext.checkPermission(AccessControlContext.java:323)
        at
java.security.AccessController.checkPermission(AccessController.java:546)
        at
oracle.security.jps.util.JpsAuth$AuthorizationMechanism$3.checkPermission(JpsAuth.java:348)
        at
oracle.security.jps.util.JpsAuth$Diagnostic.checkPermission(JpsAuth.java:268)
        at
oracle.security.jps.util.JpsAuth$AuthorizationMechanism$6.checkPermission(JpsAuth.java:372)
        at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:408)
        at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:431)
        at
oracle.security.jps.internal.policystore.AbstractPolicyStore.checkPolicyStoreAccessPermission(AbstractPolicyStore.java:246)
        at
oracle.security.jps.internal.policystore.ldap.LdapPolicyStore.getApplicationPolicy(LdapPolicyStore.java:281)
        at
oracle.security.jps.internal.policystore.PolicyUtil.getGrantedAppRoles(PolicyUtil.java:898)
        at
oracle.security.jps.internal.policystore.PolicyUtil.getJpsAppRoles(PolicyUtil.java:1354)
        at
oracle.security.jps.wls.JpsWlsSubjectResolver$1.run(JpsWlsSubjectResolver.java:273)
        at
oracle.security.jps.wls.JpsWlsSubjectResolver$1.run(JpsWlsSubjectResolver.java:270)
        at java.security.AccessController.doPrivileged(Native Method)

Here the permission

oracle.security.jps.service.policystore.PolicyStoreAccessPermission
context=APPLICATION,name=webcenter getApplicationPolicy

is granted to a codesource, and the authorization is not allowed because it evaluates to false.

Solution

Check the Administration Server logs for messages like the following, which suggests that the schema was never seeded during the reassociation:

AdminServer-diagnostic.log:[2009-05-28T02:27:52.249-07:00] [AdminServer] [NOTIFICATION] [JPS-00072] [oracle.jps.config] [tid: Thread-39] [ecid: 0000I66Z0KH0fplp4sm3Ui1A7_Rl00002s,1:5001] [arg: 11.1.1.1.0] [arg: 11.1.1.0.0] Policy schema upgrade not required. Store Schema version 11.1.1.1.0 is compatible to the seed schema version 11.1.1.0.0
AdminServer-diagnostic.log:[2009-05-28T02:28:58.012-07:00] [AdminServer] [NOTIFICATION] [JPS-00078] [oracle.jps.config] [tid: Thread-39] [ecid: 0000I66Z0KH0fplp4sm3Ui1A7_Rl00002s,1:5001] [arg: 11.1.1.1.0] [arg: 11.1.1.0.0] Credential store schema upgrade not required. Store Schema version 11.1.1.1.0 is compatible to the seed schema version 11.1.1.0.0

To ensure that the schema is seeded during reassociation:

  1. Remove the cn=OPSS container under the cn=OracleSchemaVersion container in LDAP.

  2. Start with a clean working instance of a file policy store.

  3. Reassociate the file store to a target LDAP store.

Check the Administration Server logs to confirm that the OPSS schema was seeded in the LDAP server by looking for messages like the following:

AdminServer-diagnostic.log:[2009-05-29T07:18:18.002-07:00] [AdminServer] [NOTIFICATION] [JPS-00078] [oracle.jps.config] [tid: Thread-12] [ecid: 0000I61Z0MH0fplp4sm3Ui1A7_Ll00002s,1:5001] [arg: 11.1.1.0.0]  Policy schema version set to 11.1.1.0.0

If reassociating to a Release 11g Oracle Internet Directory server, then the schema version should read: 11.1.1.1.0

If reassociating to a Release 10.1.4.3 Oracle Internet Directory server, then the schema version should read: 11.1.1.0.0

The policy store version is set in LDAP under:

cn=PolicyStore,cn=OPSS,cn=OracleSchemaVersion

The credential store version is set in LDAP under:

cn=CredentialStore,cn=OPSS,cn=OracleSchemaVersion

Migration Failure

This section describes a reason why policy migration fails when you deploy the application. Note that an application deployment may succeed even though the migration has failed. For information about version issues, see also Incompatible Versions of Security Stores.

Symptom

You configured your application to migrate policies at deployment. The application deployment succeeds, but the server diagnostic file has a message like the following:

[2009-01-21T13:34:48.144-08:00] [server_soa] [NOTIFICATION] []
[oracle.jps.deployment] [tid: [ACTIVE].ExecuteThread: '2' for queue:
'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] 
[ecid: 0000Hvv7U_H7q2T6uBADUH19Tq0B00002I,0] [APP: JpsJdev#V2.0] 
Application [JpsJdev#V2.0] is being deployed, start policy migration.

[2009-01-21T13:34:48.770-08:00] [server_soa] [WARNING] [] 
[oracle.jps.deployment] [tid: [ACTIVE].ExecuteThread: '2' for queue:
'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] 
[ecid: 0000Hvv7U_H7q2T6uBADUH19Tq0B00002I,0] [APP: JpsJdev#V2.0] 
Exception in application policy migration.[[
oracle.security.jps.JpsException: appplication Role: 
test_role not found for the application in the destination policy store
at oracle.utility.destination.apibased.JpsDstPolicy.convertAppPolicyPrincipal
(JpsDstPolicy.java:815)
at oracle.utility.destination.apibased.JpsDstPolicy.clone
(JpsDstPolicy.java:691)...

In this example, the JpsJdev application was deployed to the server_soa Managed Server. The key phrase to look for to locate such error is highlighted in the output example. This error also describes the artifact that raised the exception, the test_role application role.

Diagnosis

A look at jazn-data.xml reveals that the test_role role is referenced in a grantee:

<grantee>
   <display-name>myPolicy</display-name>
   <principals>
     <principal>
       <class>oracle.security.jps.service.policystore.ApplicationRole</class>
       <name>test_role</name>
     </principal>
   </principals>
</grantee> ...

But the name is misspelled to test_rolle:

<application>
   <name>JpsJdev</name>
   <app-roles>
     <app-role>
       <name>test_rolle</name>
       <class>oracle.security.jps.service.policystore.ApplicationRole</class>
        <members> ...

Solution

Ensure that you have defined all application roles referenced in application policies. If a referenced role name cannot be matched, then the migration fails.

Troubleshooting Server Startup

This section explains several reasons why Oracle WebLogic Server may fail to start in the following sections:

Missing Required LDAP Authentication Provider

This section explains a reason why WebLogic Server may fail to start after modifying the list of authentication providers in a domain.

Symptom

After modifying the list of providers in a domain, WebLogic Server fails to start, and the error messages include the following:

java.lang.IllegalArgumentException: null KeyStore name

Diagnosis

One cause of this problem is that the list in your domain does not include an LDAP provider. An LDAP provider is required in this list in any domain using OPSS.

Solution

Add a provider:

  1. Open DOMAIN_NAME/config/config.xml.

  2. Edit to include within the element <realm> an LDAP provider:

    <realm>
     ...
     <sec:authentication-provider xsi:type="wls:default-authenticatorType"> 
     </sec:authentication-provider>
     ...
    </realm>
    
  3. Restart the server.

After the server is up and running, modify the list of providers to include the provider of your choice with Oracle WebLogic Server Administration Console, and ensure that at least one of them is an LDAP authentication provider.

Use WebLogic Server Administration Console to:

  1. Go to the page Create a new Authentication Provider.

  2. Enter a name and choose a type:

    • ActiveDirectoryAuthenticator

    • WebLogic Default Authenticator (this is the one inserted manually in the example)

    • LDAPAuthenticator

    • LDAPX509IdentityAsserter

    • OpenLDAPAuthenticator

    • OracleInternetDirectoryAuthenticator

    • OracleVirtualDirectoryAuthenticator

Missing Administrator Account

This section explains a reason why WebLogic Server may fail to start.

Symptom

After removing the configured authentication provider and adding the Oracle Internet Directory authentication provider, the server fails to start.

Diagnosis

Most likely, you have forgotten to enter an account member of the Administrators group in the provider you added. The server requires that such an account be present in one authentication provider. This account is always present in the default one.

Solution

Add the deleted LDAP provider manually:

  1. Open DOMAIN_NAME/config/config.xml.

  2. Edit to include within the <realm> element the default provider:

    <realm>
     ...
     <sec:authentication-provider xsi:type="wls:default-authenticatorType"> 
     </sec:authentication-provider>
     ...
    </realm>
    
  3. Restart the server.

After the server is up and running:

  1. Create an account for a member of the Administrators group.

  2. Set the flag to SUFFICIENT.

  3. Restart the server, which it should start without problems, because it is using the account in the Administrators group provided in the default provider.

  4. Reset the flag to REQUIRED and remove the default provider. The server should now start using the account in the Administrators group that you created.

Missing Permission

This section explains a reason why WebLogic Server may fail to start.

Symptom

The server fails to start when you try to start it with the security manager enabled (with the -Djava.security.manager system property).

Diagnosis

The grant that allow access to public key methods in oraclepki.jar when the security manager is enabled at server startup is not found.

Solution

Ensure that a grant like the following is present in the weblogic.policy file, or add it if it is not present:

grant codeBase "file:${oracle.home}/modules/oracle.pki_${opss.version}/*" { 
 permission java.security.AllPermission; 
};

This grant is provided by default. Note that when security manager is enabled, the access to all system resources requires codesource permission grants.

See Using the Java Security Manager to Protect WebLogic Resources in Developing Applications with the WebLogic Security Service.

Server Fails to Start

This section explains two reasons why WebLogic Server fails to start.

Symptom 1

The ${domain.home}/config/fmwconfig domain directory is on an NFS-mounted partition, and an error message like the following is logged when you start the server:

JPS-01050: Opening of wallet based credential store failed. Reason 
java.io.IOException: PKI-02002: Unable to open the wallet. Check password.

Furthermore, when orapki debugging is turned on and you start the server again, the following message is logged:

java.io.IOException: No locks available.

Diagnosis 1

Because OPSS requires file locking to manage security data in file stores, the No locks available error message indicates that the file system on which the domain directory is NFS-mounted does not support file locking.

Solution 1

Perform either of the following and restart the server:

  • Upgrade from NFS v3 to NFS v4.

  • Mount the remote file system with the nolock option enabled.

  • Move files in ${domain.home}/config/fmwconfig to a local storage

Symptom 2

The server fails to start because a credential operation run into an exception. Furthermore, when orapki debugging is turned on and you start the server again, a file permission error is logged.

Diagnosis 2

Credential operations create and make use of temporary files in the /tmp directory. All files matching the pattern /tmp/*pki* must be owned by the user that started the server. A file permission error message indicates that some files matching that pattern are not owned by the appropriate user.

Solution 2

Remove any files matching the pattern /tmp/*pki* not owned by the user starting the server, and restart the server.

Symptom 3

This symptom is same as symptom 2, but it has a different resolution. The server fails to start because a credential operation run into an exception. Furthermore, when orapki debugging is turned on and you start the server again, a file permission error is logged.

Diagnosis 3

The server must be started by the same OS user as the one who installed the domain. The server fails to start even if started by any other member of the OS group to which the installer belongs.

Solution 3

Restart the server using the OS user who installed the domain.

Other Server Start Issues

This section explains several reasons why WebLogic Server may fail to start.

Symptom

When attempting to load and set the policy provider, WebLogic Server fails to start and logs an exception similar to the following:

<Mar 30, 2010 3:15:54 PM EDT> <Error> <Security> <BEA-090892> <The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider.
...
<Mar 30, 2010 3:15:54 PM EDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider.
...
weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider.
...

Diagnosis

The server startup includes loading and setting the policy provider as defined in the jps-config.xml configuration file. If this task is not completed successfully, then WebLogic Server fails to start. This type of failure is identified in the server's log by the string

Exception was thrown when loading or setting the JPSS policy provider.

To determine the root cause of a server startup failure, check the server's log file and inspect the logged stack trace. For information about identifying errors, see Diagnosing Security Errors.

Here are some reasons why the server fails to start:

  1. The path to the configuration file is incorrectly specified.

  2. The default context is missing in the configuration file.

  3. The XML parser is not available.

  4. A codesource URL is incorrectly specified in a system policy. This situation is identified by a logged exception that includes the string

    java.net.MalformedURLException: unknown protocol.

Solution

The following steps describe a solution to each of these reasons:

  1. Ensure that the correct path is specified by the oracle.security.jps.config system parameter:

    -Doracle.security.jps.config=<full-path-to-jps-config.xml>
    

    Note that special characters (such as backlashes or white space characters) in the full path specification must be properly escaped. One way to verify correctness is to test with the full path specified in a command line.

  2. The configuration file must include a default context. For an example of a default context configuration, see <jpsContext>.

  3. Make sure that the XML parser is available in your system and that the XML parser JAR file is included in the classpath.

  4. The following examples illustrate incorrect and corrected codesource URLs:

    Example 1 - Incorrect URL
    <grantee>
      <codesource>
         <url>${my.oracle.home}/jlib/webcacheua.jar</url> 
       </codesource>
    </grantee>
     
    Example 1 - Corrected URL (in bold)
    <grantee>
      <codesource>
         <url>file:/${my.oracle.home}/jlib/webcacheua.jar</url> 
       </codesource>
    </grantee>
     
    Example 2 - Incorrect URL
    <grantee>
      <codesource>
         <url>c:/myfolder/jlib/webcacheua.jar</url> 
       </codesource>
    </grantee>
     
    Example 2 - The corrected URL (in bold) is either one of the following three:
    <grantee>
      <codesource>
         <url>file:///c:/myfolder/jlib/webcacheua.jar</url> 
       </codesource>
    </grantee>
     
    <grantee>
      <codesource>
         <url>file:c:/myfolder/jlib/webcacheua.jar</url> 
       </codesource>
    </grantee>
     
    <grantee>
      <codesource>
         <url>file:/c:/myfolder/jlib/webcacheua.jar</url> 
       </codesource>
    </grantee>
    

    For information about the syntax of URL specifications in a codesource, see <url>.

Permission Failure Before Server Starts

This section describes a reason why a permission check may fail before the server has completed its starting phase.

Symptom

An authorization check fails before the server has started. The server has started up, but the server diagnostic file contains the following:

<WebLogicServer> <BEA-000365> <Server state changed to STARTING>

Diagnosis

A permission check error before the server has changed status to STARTING usually indicates that the service required to check that permission was not fully initialized at the time of the request.

Solution

To workaround this issue:

  1. Edit the weblogic.policy file and add the appropriate grant(s).

  2. Start WebLogic Server with the following system properties:

    • java.security.policy set to the location of the weblogic.policy file.

    • jps.policystore.hybrid.mode set to true.

Troubleshooting Permissions

This section describes the following issues:

Troubleshooting System Policy Failures

A system policy is a policy that specifies a set of permissions that a principal or a codesource is allowed to perform, and it holds for an entire domain. Codesource grants are required, for example, when an application code must perform a management operation on a policy, a credential, a key, or audit.A runtime authorization failure on any of these operations throws the java.security.AccessControlException exception. This section describes the process that you follow to diagnose the issue.

Symptom

A runtime authorization failure occurs:

java.security.AccessControlException: access denied
(oracle.security.jps.service.credstore.CredentialAccessPermission
context=SYSTEM,mapName=oracle.patching,keyName=FUSION_APPS_PATCH_WLS_ADMIN-KEY read)

The first step is to get information about the failure by enabling the following loggers and then reproducing the failure:

setLogLevel(target="serverName", logger="oracle.security.jps.util.JpsAuth", level="TRACE:32", persist=1);
 
setLogLevel(target="serverName", logger="oracle.security.jps.trace.logger", level="TRACE:32", persist=1);
 
setLogLevel(target="serverName", logger="oracle.security.jps.dbg.logger", level="TRACE:32", persist=1);
 
setLogLevel(target="serverName", logger="oracle.security.jps.internal.policystore.JavaPolicyProvider", level="TRACE:32", persist=1);
 
setLogLevel(target="serverName", logger="oracle.jps.common", level="TRACE:32", persist=1);

In the JpsAuth logger output, look for the string “Failed ProtectionDomain." The following output example illustrates the relevant lines around that string:

Failed ProtectionDomain:ClassLoader=sun.misc.Launcher$AppClassLoader@1823ab20                      CodeSource=file:/scratch/idmprov/idmtop/products/iam/patch_wls1036/patch_jars/BUG14331527_1036.jar
Principals=total 0 of principals<no principals>
Permissions=((java.io.FilePermission /scratch/idmprov/idmtop/products/iam/patch_wls1036/patch_jars/BUG14331527_1036.jar read)
…
Call Stack:    java.security.AccessControlException: access denied (oracle.security.jps.service.credstore.CredentialAccessPermission context=SYSTEM, mapName=OAM_STORE, keyName=jks write)
e] java.security.AccessControlContext.checkPermission(AccessControlContext.java:374)
e] java.security.AccessController.checkPermission(AccessController.java:546)
e] oracle.security.jps.util.JpsAuth$AuthorizationMechanism$3.checkPermission (JpsAuth.java:463)

Diagnosis

This failure indicates a mismatch between the provisioned codesource grant and the runtime expanded evaluation of the grant.

Solution

To resolve this issue, one must update the provisioned codesource grant so that it matches the permission, target, and action(s) that the runtime evaluates to. To this end:

  1. Inspect the provisioned codesource grant with either Fusion Middleware Control as explained in Managing Policies with Fusion Middleware Control or the listCodeSourcePermissions WLST command. See listCodeSourcePermissions in WLST Command Reference for Infrastructure Security.

  2. If the provisioned codesource grant does not match the data in the runtime error, then modify the grant with Fusion Middleware Control or the grantPermission WLST command. See grantPermission in WLST Command Reference for Infrastructure Security.

  3. Ensure that domain and other variables expand to correctly specified the grant. Note that, in loggers, codesources are written using the absolute path to the JAR file, but URLs are written using environment variables.

  4. If you specified the grant in the system-jazn-data.xml file, then review the grant in that file so that the fix will take effect when you deploy the application to a new environment.

  5. If your code is required to run in a privileged block, then make sure that it is doing so.

Failure to Get Permissions - Case Mismatch

This section explains some reasons why an enterprise user or role (group) fails to get permissions.

Symptom

An enterprise user or group, properly entered in an authentication provider, is not granted the permissions defined by a grant.

Diagnosis

This problem is likely to occur when there is a case mismatch between the stored name and the supplied name. For example, this mismatch would occur when the stored user name is jDoe and the supplied user name is jdoe.

Solution 1

The first solution involves setting the appropriate property in the authentication provider being used in your domain. As long as both strings (the supplied and the stored) contain identical sequence of characters (irrespective of case), this setting guarantees that the user name populated in the subject matches the user name present in an authentication provider, even when the corresponding characters differ in case.

To set your provider property:

  1. Use WebLogic Server Administration Console to go to the page where your provider is configured. For example, if you are using the default provider, then go to the Default Authenticator page at Realms.myrealm.Providers.Default Authenticator.

  2. Select the tab Provider Specific.

  3. Set the property userRetrievedUserNameAsPrincipal to true.

  4. Restart the server.

Solution 2

The second solution considers the case where one must produce a principal from a user name.

When you create a user or role principal, use:

Principal userPrincipal = new WLSUserImpl(user.getUserProfile()getName());
Principal rolePrincipal = new WLSGroupImpl(role.getRoleProfile().getName());

instead of:

Principal userPrincipal = new WLSUserImpl(user.getName());
Principal rolePrincipal = new WLSGroupImpl(role.getName());

To obtain the correct user or group name, either pass the name exactly as it is stored in the authentication provider or use the sequence of calls:

import weblogic.security.principal.WLSGroupImpl;
import weblogic.security.principal.WLSUserImpl;
 
// Set the context
JpsContextFactory ctxFact = JpsContextFactory.getContextFactory();
ServerContextFactory scf = (ServerContextFactory) ctxFact;
JpsContext ctx = scf.getContext(ServerContextFactory.Scope.SYSTEM);
ctx = ctxFact.getContext();

// Set the identity store
IdentityStore identityStore = ctx.getServiceInstance(IdentityStoreService.class).getIdmStore();
 
// In case of a user name, search the user that matches the supplied name
User user = idStore.searchUser(IdentityStore.SEARCH_BY_NAME, suppliedUserName);

// Use the obtained object (user) to obtain the stored user name and create 
// the Principal
String storedUserName = user.getName();
Principal userPrincipal = new WLSUserImpl(storedUserName);
 
// Similarily, in case of a role name, search the role that matches 
// the supplied role name
Role role = identityStore.searchRole(IdentityStore.SEARCH_BY_NAME, suppliedRoleName);
 
// Use the obtained object (role) to obtain the stored role name and create 
// the Principal
String storedRoleName = role.getName();
Principal rolePrincipal = new WLSGroupImpl(storedRoleName);

Authorization Check Failure

This section explains a reason why an authorization check has failed.

Symptom

An attempt to authorize a user by your application fails, and the system logs an error containing a line like the following:

Servlet failed with Exception oracle.adf.controller.security.AuthorizationException:ADFC-0619:
Authorization check failed: '/StartHere.jspx' 'VIEW'. 

Diagnosis

One reason that can lead to this an authorization failure is a mismatch between the context and the stripe that you application is using.

On the one hand, the application stripe that an application uses is specified in the web.xml file with the application.name parameter in the configuration of the JpsFilter filter or the JpsInterceptor interceptor. If the application stripe is unspecified, then the system picks an application stripe based on the application name.

On the other hand, the runtime policies that your application uses are specified in the system-jazn-data.xml file with the <application.name> element.

If those two names do not match or you have not explicitly specified the stripe to use, then, most likely, your application is accessing the wrong policy stripe and is not able to authorized your application's users as expected.

Solution

Ensure that you specify explicitly your application stripe, and that stripe is the one that your application is supposed to use. In most cases, the two names specified in those two different files match. However, in cases where several applications share the same policy stripe, they may differ.

User Gets Unexpected Permissions

This section explains the likely reasons why a user gets permissions other than those anticipated.

Symptom

A new user or a modified user gets unexpected permissions.

Diagnosis

This issue is likely to come up in cases where a user is added with the name of previously removed user, or an old user gets its name or uid changed. The common reason why the user may get more or less permissions than expected is that the security store has not been properly updated before you remove users or change user data.

Solution

Before deleting a user, revoke all the permissions, application roles, and enterprise groups that had been granted to that user. If you fail to remove all security data referencing the user, they are left dangling and, potentially, inherited if another user with the same name is created at a later time.

Similar considerations apply to when a user name is changed: all policies (grants, permissions, roles) referring to the old data must be updated so that they work as expected with the new data.

Granting Permissions in Java SE Applications

This section describes the correct way to code a grant in Java SE applications. Even though the problem described is not an issue in Java EE applications, for maximum portability, it is recommended that this solution be used in Java EE applications too.

Symptom

The application code includes lines like the following:

Permission p = new FilePermission(resourceName, "write");
PrincipalEntry myPrincipal2 = InfoFactory.newPrincipalEntry("weblogic.security.principal.WLSGroupImpl", enterpriseRoleName2);
ap.grant(new Principal[]{myPrincipal2.getPrincipal()}, null, new Permission[]{p});

At runtime, however, the grant is not taking effect as expected.

Diagnosis

A little inspection indicates that the security store includes the following attribute:

orcljaznjavaclass=oracle.security.jps.internal.policystore.UnresolvedPrincipal+cn=enterpriseRoleName2

Solution

The code should be replaced by the following lines:

Permission p = new FilePermission (resourceName, "write");
PermissionEntry permEntry  = InfoFactory.newPermissionEntry(p.getClassName(), p.getName(),  p.getActions());
ap.grant (new PrincipalEntry[] {myPrincipal2}, null, new PermissionEntry[] {permEntry});

The solution uses the PrincipalEntry array instead of the Principal array and the PermissionEntry array instead of the Permission array.

Application Policies Not Seen in 12c High Availability (HA) Domain

This section describes a sequence that results in application policies not taking effect in the 12c high availability (HA) domain, and how to work around it.

Symptom

The following sequence throws an exception:

  1. Deploy a custom application (packed with application policies) in a 12c HA domain, either to the Administration Server or to a Managed Server (but not to both).
  2. Undeploy the application.
  3. Redeploy the application to a server different from the server in step 1.

Step 3 results in an exception and the application policies do not take effect. This issue is observed in 12c HA domains only.

Example J-1 Diagnosis

This issue is seen because the domain servers do not have their caches synchronized. Note that the various servers run in distinct JVM's (Java Virtual Machine).

Suppose, for example, that the HA domain has three servers: server A (the Administration Server), server B (a Managed Server), and server C (a Managed Server). Suppose further, that the application is first deployed to server A, the Administration Server, and then all three servers are restarted. The caches in servers A, B, and C will then have been initialized (with policies read from the security store) and are synchronized.

If the application is then undeployed (from server A), then the server A's cache will be cleared, but the caches in servers B and C will not. Further, if the application is redeployed, say, to server B, then the caches become out of synch, and an exception is thrown (because policy objects already exist).

Example J-2 Solution

Undeploy the application and, before deploy it, restart all servers in the HA domain. This way, the modified procedure leads to synchronized caches in all servers in the HA domain, and the application policies are seen as expected.

Troubleshooting Connections and Access

This section describes the following issues:

Database Connection Exception

This section explains why a database connectivity exception is thrown for the OPSS security store.

Symptom

OPSS throws the oracle.security.jps.service.policystore.PolicyStoreConnectivityException when it fails to connect to the database to read or update security store data.

Diagnosis

Check log files to see if there is oracle.security.jps.service.policystore.PolicyStoreConnectivityException printed.

Solution

Check the database status and connectivity from the host where the exception above was thrown.

Other Database Exceptions

This section explains how to identify and solve other database exceptions for OPSS database security store access.

Symptom

OPSS throws the oracle.security.jps.service.policystore.PolicyStoreException with org.eclipse.persistence.exceptions.DatabaseException and java.sql.SQLException as the cause of the exception.

Diagnosis

Get error code from java.sql.SQLException such as java.sql.SQLException: ORA-28000: the account is locked.

Solution

Contact your database Administrator to solve the SQL exception.

JNDI Connection Exception

This section explains why Java Naming and Directory Interface (JNDI) connections may throw a time out exception.

Symptom

JNDI Connections throw the javax.naming.NamingException: LDAP response read timed out, timeout used:-1ms exception.

Diagnosis

This issue is found in domains configured to use an Oracle Identity Directory security store, or when using the User Role API or IGF/IDS against an LDAP identity store on any of the following JDK versions: Java SE 6u85, 7u72, or 8u20.

Solution

Update the JDK to a version supported in this release. For certified JDK versions, see Oracle Fusion Middleware Supported System Configurations.

Failure to Connect to the Embedded LDAP Server

This section explains why a connection to the embedded LDAP server fails.

Symptom

The connections that client applications use to request queries to the embedded LDAP server with the User and Role API, are stored and maintained in a connection pool. By default this pool is the JNDI pool as specified in the jps-config.xml file.

If the number of current connections in the pool exceeds the maximum allowed by the LDAP service, then client applications will not be able to connect to the service or, even when they are already connected, receive a “socket closed" exception. The server log would indicate, in this case, that the number of concurrent connections allowed has been exceeded.

Diagnosis

To avoid going over the limit, you must adjust the maximum number of concurrent connections allowed by the LDAP service as appropriate to the application's needs. This threshold must be finely tuned up: a too small maximum may not be sufficient (and cause an exception). A too large maximum may risk a denial of service (DOS) attack. The correct maximum depends on your application and the particular LDAP service the application uses.

Solution

There are two alternative ways that resolve this issue:

  • Increase the maximum number of concurrent connections allowed by the provider:

    • If the provider that your application uses is the embedded LDAP server, then edit the file DomainName/servers/MyServerName/data/ldap/conf/vde.prop, and increase the value of the vde.quota.max.conpersubject property from the default 100 to, for example, 200, or any other value.

    • Otherwise, if your application is using any other provider, then consult the provider's documentation to learn how to modify the maximum.

  • Edit the file DomainName/config/fmwconfig/jps-config.xml and remove the CONNECTION_POOL_CLASS property from the provider server instance (by default, this property has the value oracle.security.idm.providers.stdldap.JNDIPool.

Note that these settings do not exclude each other and, in any case, you must restart the server for the changes to take effect.

Failure to Connect to LDAP Server

This section explains the likely reasons why a connection to an LDAP server may fail. This failure can also happen during reassociation.

Symptom

The migration of data from a source repository to a target LDAP server fails.

Diagnosis

This kind of problem is due to incorrect parameter values in the target LDAP server.

For further probing into WebLogic Server log files, search any of the log files in the DomainName/servers/AdminServer or DomainName/servers/ManagedServers directories for the following strings: <Error>, <Critical>, and <Warning>.

For more information about identifying and solving errors, see Diagnosing Security Errors.

Solution

Verify that all the target server data provided for the migration is valid. If you use Fusion Middleware Control to reassociate to an LDAP server, then ensure that you use the button Test LDAP Authentication before initiating the operation. This test catches incorrect parameters.

Failure to Access Data in the Credential Store

This section explains a likely reason why an application fails to access data in the domain's credential store.

Symptom

An application fails to retrieve credential data from the domain's credential store, and an error message containing lines like the following is logged:

07/07/26 18:22:22 [JpsAuth] For permisson ( CredentialAccessPermission [target] [actions]), domain that failed: ProtectionDomain
 cs(file:somePath/aWarFile.war/WEB-INF/classes/), []

Diagnosis

If an application is to access the credential store to perform an operation (such as retrieving a user password, for example), then its code must be granted the appropriate permission to perform the secured operation. Otherwise, the application runs into an error.

Solution

To grant the permission that an application requires to access the credential store, include the appropriate CredentialAccessPermission permission in the application's jazn-data.xml. This grant takes effect when you deploy or redeploy the application.

To add a permission with Fusion Middleware Control, see Managing Policies with Fusion Middleware Control.

To add a permission with a WLST command, see Managing Policies with WLST.

The following example illustrates how to grant all code in the myApp application permission to read all credentials in the myAlias folder:

<jazn-data>
    <!--  codesource policy -->
    <jazn-policy>
        <grant>
            <grantee>
                <codesource>
 <!-- This grants applies to all code in the following directory -->
                    <url>${domain.home}/tmp/_WL_user/myApp/-</url>
                </codesource>
            </grantee>
            <permissions>
                <permission>
                    <class>oracle.security.jps.service.credstore.CredentialAccessPermission</class>
<!-- Allow read permission to all credentials under folder MY_MAP -->
                    <name>context=SYSTEM,mapName=MY_MAP,keyName=*</name>
                    <actions>read</actions>
                </permission>
            </permissions>
        </grant>
    </jazn-policy>
</jazn-data>

Security Access Control Exception

This section explains a reason why your code may run into a security access control exception.

Symptom

At runtime, your application prints an error like the following one (only the first few lines are shown):

<Jan 20, 2009 5:45:33 PM PST> <Error> <HTTP> <BEA-101020>
<[weblogic.servlet.internal.WebAppServletContext@140cf52 
- appName: 'Application2', 
name: 'Application2.war', 
context-path: '/Application2', 
spec-version: '2.5'] 
Servlet failed with Exceptionjava.lang.RuntimeException:java.security.AccessControlException:access denied
...

Diagnosis

This error means that a call in your code does not have sufficient permissions to execute a secured operation.

Solution

Your code must be granted the appropriate permissions to execute the secured operation. Depending on the scope of the permission you would like to set, you have two alternatives.

The first alternative is to grant permission to all application code in the application Enterprise ARchive (EAR) or web application archive (WAR) files. In this case, the call to the operation can be inserted anywhere in the application code.

The second alternative is to grant permission to just a JAR file. In this case, the call to the operation must be inside a privileged block.

Each of these solutions is next illustrated by an application attempting to access the credential store.

The following example illustrates how to set permission to read any key within the map MY_MAP in the credential store to any code within the BasicAuth directory:

<jazn-policy>
   <grant>
       <grantee>
           <codesource>
              <url>file:${domain.home}/servers/_WL_user/BasicAuth/-</url>
           </codesource>
       </grantee>
       <permissions>
           <permission>
             <class>
                 oracle.security.jps.service.credstore.CredentialAccessPermission
             </class>
             <name>context=SYSTEM,mapName=MY_MAP,keyName=*</name>
             <actions>read</actions>
          </permission>
      </permissions>
   </grant>
</jazn-policy>

When granting permissions to code in a particular EAR or WAR file, then change the url specification to one like the following:

<url>file:${domain.home}/servers/_WL_user/jpsBasicAuth/.../BasicAuth.ear</url>

When granting permissions to just the code in a particular JAR file, the change the url specification to one like the following:

<url>file:${domain.home}/servers/_WL_user/jpsBasicAuth/myJars/Foo.jar</url>

In this last case, the code in the Foo.jar file that calls a read operation on the credential store must be placed in an AccessController.doPrivileged block:

import oracle.security.jps.*;
import oracle.security.jps.service.credstore.*;
 
JpsContextFactory factory = JpsContextFactory.getContextFactory();
JpsContext jpsContext = factory.getContext();
final CredentialStore store = jpsContext.getServiceInstance(CredentialStore.class);
Credential cred = AccessController.doPrivileged(new PrivilegedExceptionAction<PasswordCredential>() {
    public PasswordCredential run() throws JpsException {
        return store.getCredential("MY_MAP", "anyKey");
    }
});
           
PasswordCredential pwdCred = (PasswordCredential)cred;
...

Note that because our example grant allows a read permission only, none of the set or reset operations work, even inside a privileged block.

Failure to Establish an Anonymous SSL Connection

This section explains the likely reasons why you are not able to establish an anonymous Secure Sockets Layer (SSL) connection while reassociating policies and credentials.

Symptom

Reassociation of a file security store to an LDAP server with Fusion Middleware Control involves testing the anonymous SSL connection to the LDAP server. When you click Test LDAP, the test fails.

Diagnosis

Your target LDAP server must be trusted by WebLogic Server and the port number you are using to connect to the LDAP server must be an SSL port.

Solution

Establishing SSL connections to an LDAP server requires additional LDAP server configuration. For information about this configuration, see Prerequisites to Using the LDAP Security Store.

In addition, to use an anonymous SSL connection, you must enter a port that has been set for receiving secure data. If your LDAP server has not been configured with such a port, then the connection fails.

Ensure that the supplied LDAP server port is an SSL port configured to listen in anonymous SSL mode, and that the supplied server name reachable.

Oracle Business Intelligence Publisher Time Zone

You may see some problems with audit reports if BI Publisher and the database are installed in sites with different time zones. Ensure that BI Publisher and the database are installed in the same time zone.

Troubleshooting Searching

The following sections describe query issues:

Search Failure When Matching Attribute in Security Store

This section describes a reason why you must catalog attributes.

Symptom

When querying the security store, you run into the following exception:

oracle.security.jps.service.policystore.PolicyStoreOperationNotAllowedException
javax.naming.OperationNotSupportedException:
[LDAP: error code 53 - Function Not Implemented, search filter attribute orcljpsresourcetypename is not indexed/cataloged];
remaining name 'cn=Permissions,cn=JAASPolicy,cn=IDCCS, cn=sprint6_policy_domain,cn=JPSContext,cn=FusionAppsPolicies'

Diagnosis

You must catalog orcljpsresourcetypename before you use it in a filter to search the security store.

Solution

An LDAP attribute used in a search filter must be indexed and cataloged. Indexing and cataloging are optional operations, in general, but required for OPSS-related attributes. Attribute indexing and cataloging is automatically performed by the reassociateSecurityStore WLST command.

To catalog attributes manually use the ldapmodify command:

>ldapmodify -h <host> -p <port> -D <bind DN> -w <bind password> -v -f <catalogue modify ldif file name>

To catalog the createtimestamp and modifytimestamp attributes, for example, use an LDAP Data Interchange format (LDIF) file like the following:

dn: cn=catalogs
changetype: modify
add: orclindexedattribute
orclindexedattribute: modifytimestamp
orclindexedattribute: createtimestamp

The list of LDAP attributes that must be indexed follows:

OrclJpsAllResourcKeyword
OrclJpsAllResourceActionKeyword
OrclJpsEncodedAttributes
OrclJpsExtensionType
OrclJpsResourceConverter
OrclJpsResourceMatchingAlg
OrclJpsResourceMatchingAlgorithm
OrclJpsResourceNameExpression
OrclJpsRoleType
orcOesAppAttributes
orclASInstanceName
orclFarmName
orclJPSObjGUID
orclJavaApplicationEntityRef
orclJpsPolicyDomainName
orclJpsResourceActionsetMembers
orclJpsResourceExpression
orclJpsResourceLocalityRef
orclJpsResourceMatcherJavaclass
orclJpsResourceName
orclJpsResourceTypeActionAttrs
orclJpsResourceTypeActionNames
orclJpsResourceTypeName
orclJpsResourceTypeProviderName
orclJpsResourceTypeResourceAttrs
orclJpsRoleCategory
orclJpsRoleMemberExpression
orclJpsSuperResourceType
orclOESActCollectionName
orclOESActCollectionRfs
orclOESActionAttributes
orclOESActionConstraint
orclOESAlgorithmJavaClass
orclOESAllResourceType
orclOESAllowAdviceRef
orclOESAllowObligationRef
orclOESAttributeCategory
orclOESAttributeCollectionHandlerFunctionName
orclOESAttributeCollectionHandlerPackageName
orclOESAttributeCollectionHandlerSchemaName
orclOESAttributeCollectionName
orclOESAttributeDataType
orclOESAttributeIssuer
orclOESAttributeNamespace
orclOESAttributeType
orclOESCombinerParameter
orclOESConditionExpression
orclOESDSColumnAttrs
orclOESDSPrimKey
orclOESDataSourceCtrnt
orclOESDataSourceName
orclOESDataSourceType
orclOESDefaultPolSetRef
orclOESDenyAdviceRef
orclOESDenyObligationRef
orclOESDistributionEndTime
orclOESDistributionID
orclOESDistributionIssuer
orclOESDistributionMessage
orclOESDistributionPercentComplete
orclOESDistributionStartTime
orclOESEffect
orclOESEnvAttributes
orclOESEnvConstraint
orclOESExecutionFrequency
orclOESExpression
orclOESFunctionCategory
orclOESFunctionClass
orclOESFunctionParameters
orclOESFunctionReturnType
orclOESIsSensitive
orclOESIsSingleValued
orclOESMatchInfo
orclOESMaxDelegationDepth
orclOESObligationFulfillOn
orclOESPDPAddress
orclOESPDPConfigurationID
orclOESPDPInstanceName
orclOESPDPStatusSuccess
orclOESPIPType
orclOESPolicyCategory
orclOESPolicyCombinerParameter
orclOESPolicyCombiningAlgorithmRef
orclOESPolicyDefaults
orclOESPolicyExtension
orclOESPolicyIssuer
orclOESPolicyRef
orclOESPolicyRuleOrder
orclOESPolicyRuleRef
orclOESPolicySetCategory
orclOESPolicySetDefaults
orclOESPolicySetRef
orclOESPolicySetType
orclOESPolicyType
orclOESPolicyVersion
orclOESPresenceRequired
orclOESPrincConstraint
orclOESPrincipalAttributes
orclOESResConstraint
orclOESResTypeCategory
orclOESResourceAttributes
orclOESResourceHirchyType
orclOESResourceNameDelim
orclOESResourceParentName
orclOESRoleMapping
orclOESRuleCombinerParameter
orclOESRuleCombiningAlgorithmRef
orclOESSQLExpression
orclOESSetCombinerParameter
orclOESSetMemberOrder
orclOESTargetExpression
orclOESXMLExpression
orclassignedpermissions
orclassignedroles
orcldistributionversion
orcljazncodebase
orcljaznjavaclass
orcljaznpermissionactions
orcljaznpermissionresourceref
orcljaznpermissionsigner
orcljaznpermissiontarget
orcljaznpermissiontargetexpr
orcljaznprincipal
orcljaznsigner
orcljpsRuleCombiningAlgorithmRef
orcljpsactionsdelim
orcljpsassignee
orclrolescope

Search Failure with an Unknown Host Exception

When searching Microsoft Active Directory (configured for LDAP referrals), the referrals fail if the host being referred to is in a different domain than the Active Directory server.

Symptom

When a user requests a resource, at times verification of the user's identity can fail due to an inability to validate the user's identity in the directory. This error can occur with Microsoft Active Directory when the browser runs on a system different from Microsoft Windows, or the browser runs on Microsoft Windows but not in the server where Microsoft Active Directory is running.

Diagnosis

This problem can arise due to LDAP referral chasing. An LDAP referral occurs when a domain controller does not have the section of the directory tree where a requested object resides. The domain controller refers the client to another target so that the client can conduct a DNS search for another domain controller. If the client is configured to chase referrals, then the search can continue.

For the scenario where the user has a Windows-based computer, an issue can occur with LDAP referrals if trust is not established between the client's domain controller and the Active Directory domain controller.

Solution

Add the entry for the Active Directory host's address in the following list:

WINDOWS_HOME_DIRECTORY\system32\drivers\etc\hosts

On Windows XP, the list is located at:

C:\WINDOWS\system32\drivers\etc\host

On a Unix-based system, add this entry to the /etc/hosts file, using the format:

ADhost_IPaddress ADhost_name

where AD_host_name is the host name specified in the referral, such as 123.123.123.123 my2003ad.com.

Troubleshooting Versions

The following sections describe the version issues:

Incompatible Versions of Binaries and Security Store

This section describes the reason why the server would throw the PolicyStoreIncompatibleVersionException exception.

Symptom

You encounter an error like the following:

Oracle.security.jps.service.policystore. PolicyStoreIncompatibleVersionException
JPS-06100: Policy Store version 11.1.1.5.0 and Oracle Platform Security Services
version 11.1.1.4.0 are not compatible.

Diagnosis

The exception indicates that the domain OPSS binaries version (11.1.1.4.0) and the security store version (11.1.1.5.0) used by that domain are incompatible. The version of the security store is established during reassociation and that version is used until you upgrade the security store to a newer version.

OPSS domain binary versions are backward compatible with security store versions, but they are not forward compatible. So, the error indicates that the security store has version newer that the version of the OPSS binaries.

Here are three scenarios where OPSS binaries ends up being incompatible:

  • Scenario 1

    • Domain1 and Domain2 point to the same security store. Domain1, Domain2, and that security store have all version 11.1.1.3.0.

    • The binaries in Domain1 are upgraded to 11.1.1.4.0.

    • The security store is upgraded to 11.1.1.4.0 (using the upgradeOPSS command).

    • When the Domain2 is brought up again, the binary and security store versions are incompatible.

  • Scenario 2

    • Domain1 points to a security store, and both binaries and the security store have version 11.1.1.3.0.

    • An attempt to migrate the security store to a 11.1.1.4.0 fails because the migration would render a scenario with incompatible versions.

      Migration is supported only when the OPSS binaries and the security store versions are same.

  • Scenario 3

    • A 11.1.1.3.0 Domain1 attempts to join a 11.1.1.4.0 security store (in some other domain), using the reassociateSecurityStore WLST command with the join argument.

    • The operation fails because the sharing would render a scenario with incompatible versions.

      Reassociation is supported only when the OPSS binaries and the security store versions are same.

Solution

The solution to all three scenarios is either one of the following:

  • Update OPSS binaries to match the version of the security store the domain is using.

  • Reassociate the security store to a security store that has version not newer than the version of OPSS binaries.

Incompatible Versions of Security Stores

This section describes the reason why you may encounter the PolicyStoreIncompatibleVersionException exception when migrating the security store.

The PolicyStoreIncompatibleVersionException exception indicates that the version of the source store is higher than the version of the target store. Migration carries on only if the version of the source is not higher than the version of the target.

The workaround is to upgrade the target store to a version compatible with the version of the source store.

Troubleshooting Other Errors

The following sections describe a variety of issues:

Runtime Permission Check Failure

This section explains a reason why a permission may fail to pass a permission check.

Symptom

Your application prints an error like the following one:

[JpsAuth] Check Permission
          PolicyContext:        [null]
          Resource/Target:      [test]
          Action:               [null]
          Permission Class:     [com.oracle.permission.SimplePermission]
          Evaluator:            [ACC]
          Result:               [FAILED]
          Failed ProtectionDomain:ClassLoader=weblogic.utils.classloaders.ChangeAwareClassLoader@14061a8 
finder: weblogic.utils.classloaders.CodeGenClassFinder@2dce7a8 
annotation: Application2@Application2.war
CodeSource=file:/scratch/servers/AdminServer/tmp/permission/TestServlet$1.class
Principals=total 0 of principals<no principals>
Permissions=(
(oracle.security.jps.service.credstore.CredentialAccessPermission 
context=SYSTEM,mapName=default,keyName=* read,write)
(java.net.SocketPermission localhost:1024- listen,resolve)
(oracle.security.jps.service.policystore.PolicyStoreAccessPermission
context=APPLICATION,name=* getApplicationPolicy)
(oracle.security.jps.service.policystore.PolicyStoreAccessPermission
context=SYSTEM getConfiguredApplications)
(com.oracle.permission.SimplePermission *)
...
java.security.AccessControlException: access denied
(com.oracle.permission.SimplePermission test)...

Diagnosis

When two or more applications share a permission class, that permission class must be set in the system class path so the class is loaded just once. Otherwise, only the first application loading the class passes the permission check. Other applications loading the same class thereafter fail the permission check and print a similar error.

Note that even though the permission class is in the permission collection, the check fails and the access is denied, because the environment contains several instances of a permission class with the same name.

Solution

Ensure that if two or more applications running in the same domain share a permission class, then you include that class in the system class path.

Tablespace Needs Resizing

This section describes a reason why SQL operations fail in a DB security store.

Symptom

While performing policy operations, the database issues a the following error:

"ORA-1652: unable to extend temp segment by 128 in tablespace" 

Diagnosis

The reason is that the temporary table used by the operations has no more free blocks available.

Solution

To solve this issue:

  1. Extend the size of the temporary table to 4096 M:

    ALTER TABLESPACE "<TEMP_TABLESPACE_NAME>" ADD TEMPFILE '<data_file_name>' size 4096M
    
  2. Restart the Administration Server.

Oracle Internet Directory Exception

A domain using Oracle Internet Directory 11.1.1.6.0 as the security store runs into the following exception:

[LDAP: error code 53 - OID-5018: Cataloging for attr orcljpsextensiontype 
is already in progress.]:cn=catalogs

You must apply a patch to fix bug 13782459 in Oracle Internet Directory 11.1.1.6.0. For a list of Oracle Internet Directory patches, see Using an LDAP Security Store.

User and Role API Failure

This section explains reasons why you may fail to access data in an authentication provider with the User and Role API.

Symptom

The User and Role API fails to access data.

Diagnosis 1

The User and Role API can access data only in the first LDAP provider configured in a domain. At least one such provider must be present in a domain. The API access to that first LDAP provider fails if the target user is not present in that provider, even though that user is present in some other authentication provider.

Solution 1

Enter the missing user in the first LDAP authentication provider or reorder the list of LDAP providers in your domain.

Diagnosis 2

Assume that the target user on which the API that fails is present in the first LDAP provider configured in your domain.

By default, the User and Role API uses the uid attribute to perform user search in an LDAP provider. If for some reason, a user entered in the LDAP is lacking this attribute, then the User and Role API fails.

Solution 2

Ensure that all users in the first LDAP authentication provider have the uid attribute set.

If you are developing a Java SE application and you want the User and Role API to use an attribute other than uid to search users, say mail for example, then you must configure the username.attr and user.login.attr attributes in the LDAP provider instance of the identity store (in the jps-config-jse.xml file):

<serviceInstance provider="idstore.ldap.provider" name="idstore.ldap">
   ...
   <property name="username.attr" value="mail"/>
   <property name="user.login.attr" value="mail"/>
   ...
</serviceInstance>

For information about adding properties to a provider instance with a prescribed script, see Configuring Services with Scripts.

Characters in Policies

Special Characters in Oracle Internet Directory 10.1.4.3

When the security store is Oracle Internet Directory 10.1.4.3, then using the characters '*', '(', ')', or '\' in the RFC 2252/2253 filter results in error 53 (DSA unwilling to perform). To resolve this error, apply the patch for bug number 7711351 to Oracle Internet Directory 10.1.4.3.

Characters in File Security Stores

The issue explained in this section applies to file stores only.

The following characters:

 < "  &  $ ? * , / \ ` :    ( ) ^ ' % +  { } 

are not recommended in role names within file stores.

If you use one of those characters to create a role, then ensure that you escape such characters in arguments to methods like ApplicationPolicy.searchAppRoles so they return correct results. For example, if you have an application role named appRole^$, then pass the argument as in appRole\\^\\$ to find the match in the security store.

Alternatively, you could use a wild card in the search expression without including these escaped special characters, as in appRole*.

Characters in Application Role Names

An application role name is a string of alphanumeric characters (ASCII or Unicode) and other printable characters (such as underscore or square brackets) that contains no white space. This rule applies to roles names in any type of storage.

Missing Newline Characters in File Store

In a file store, a new-line character is required between the closing of a <permission> or <principal> tag and the opening of the following one.

The following are examples of strings illustrating incorrect and correct formats.

Incorrect:

<permission>
    <class>java.lang.RuntimePermission</class>
    <name>getClassLoader</name>
 </permission><permission>
      <class>java.io.FilePermission</class>
      <name>/foo</name>
      <actions>read,write</actions>
 </permission> 

Corrected:

<permission>
    <class>java.lang.RuntimePermission</class>
    <name>getClassLoader</name>
</permission>
<permission>
      <class>java.io.FilePermission</class>
      <name>/foo</name>
      <actions>read,write</actions>
</permission>

Invalid Key Size

This section explains why an invalid key size exception occurs.

Symptom

You encounter the following exception:

java.security.InvalidKeyException: Illegal key size

Diagnosis 1

During domain creation, OPSS uses the keystore service to create an Advanced Encryption Standard (AES) symmetric key used to encrypt security store data. OPSS tries to generate a 256-bit key first. If it is not supported, then it tries to generate a192-bit key, and if not supported either, then it creates a 128-bit key. If 128-bit keys are not supported, the invalidKeyException exception is thrown.

Diagnosis 2

During OPSS startup, if the OPSS encryption key cannot be read because the runtime JDK does not support the AES algorithm or if the domain provisioned key size differs from the size that the JDK supports, the invalidKeyException exception is thrown.

Solution

Install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in the JDK, and then retry the domain creation or OPSS startup. You can download the JCE files at http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html

Need Further Help?

To find more information about Oracle Support, visit http://myoraclesupport.oracle.com. If you do not find a solution to your problem, log a service request.