I Troubleshooting Security in Oracle Fusion Middleware

This appendix describes common problems that you may encounter when configuring and using Oracle Enterprise Manager Fusion Middleware security, and explains how to solve them. It contains the following sections:

I.1 Diagnosing Security Errors

This section describes how to detect and solve security errors and it contains the following topics:

The logging support with Fusion Middleware Control is explicitly stated whenever the tool can help managing, isolating, or interpreting faults when they occur.

I.1.1 Log Files

This section describes the various log files supported by Oracle WebLogic Server and how to configure, set log levels, and locate and view log files with Fusion Middleware Control, in the following sections:

I.1.1.1 Diagnostic Log Files

Each server instance in a domain writes all OPSS-based exceptions raised by its subsystems and applications to a server log file in the file system of the local host computer.

By default, this log file is located in the logs directory below the server instance root directory. The names of these log files have the following format: ServerName-diagnostic.logxxxxx, where xxxxx denotes an integer between 1 and 99999.

Here are some examples of diagnostic file full names: DomainName/servers/AdminServer/logs/AdminServer-diagnostic.log00001 (administration server log), DomainName/servers/soa/logs/soa-diagnostic.log00013 (managed server log).

All server instances output security-related errors to diagnostic files. Server-related security errors, such as exceptions raised by issues with a subject or principal, and errors that may occur while migrating or reassociating domain security data, get written in the administration server diagnostic log. Application-related security errors, such as exceptions raised by application-specific policies or credentials, get written in the corresponding managed server diagnostic log.

I.1.1.2 Generic Log Files

In addition to diagnostic log files, Oracle WebLogic Server supports other log files for each server in a domain and for each domain in a topology.

By default and similar to diagnostic log files, server log files are located in the logs directory below the server instance root directory. Domain log files are located in the logs directory below the administration server root directory. The names of these log files have the format ServerName.logxxxxx and domain.logxxxxx, where xxxxx denotes an integer between 1 and 99999.

Here are some examples of server and domain log files full names: DomainName/servers/AdminServer/logs/AdminServer.log00001, DomainName/servers/AdminServer/logs/domain1.log00033.

Server and domain logs are files where one should look for generic errors, such as exception raised by authenticators or other domain service providers.

The domain logs duplicate some messages written to server logs (for servers in the domain), and they help determine the server where a fault has occurred in a domain that contains a large number of servers.

Note:

The generation of a new log file is determined by its rotation policy; typically, the rotation is determined by file size, so when a log file exceeds a specified size, the system generates a new one with a name whose integer suffix is increased by 1.

Related Documentation

For information about server log files and domain log files, see section Server Log Files and Domain Log Files in Oracle Fusion Middleware Configuring Log Files and Filtering Log Messages for Oracle WebLogic Server.

For information about the Oracle WebLogic Framework, see Oracle Fusion Middleware Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.

For additional information about logging services, see Oracle Fusion Middleware Using Logging Services for Application Logging for Oracle WebLogic Server.

For complete details about logging in Oracle Fusion Middleware, see chapter 10, Managing Log Files and Diagnostic Data, in Oracle Fusion Middleware Administrator's Guide.

I.1.1.3 Audit Diagnostic Log Files

There are several run-time components in the Fusion Middleware Audit Framework. This section helps you navigate the diagnostic log files for these components and explains how to interpret diagnostic messages.

The log files are located at:

DomainName/servers/$SERVER_NAME/logs/$SERVER_NAME-diagnostic.log

Table I-1 lists the various diagnostic log files.

Table I-1 Log Files for Audit Diagnostics

Component Log Location Configuring Loggers

Java EE Components using Audit APIs

DomainName/servers/$SERVER_NAME/logs/$SERVER_NAME-diagnostic.log

oracle.security.audit.logger (See instructions below)

OPMN Component Using Audit APIs

See the Administration Guide for the component to locate its log files.

Loggers are based on the OPMN Components's Location. Please see the corresponding component guide.

Startup Class Audit Loader

DomainName/servers/$SERVER_NAME/logs/$SERVER_NAME-diagnostic.log

oracle.security.audit.logger (See instructions following this table)

OPMN Audit Loader

$ORACLE_INSTANCE/diagnostics/logs/OPMN/opmn/rmd.out

java.util.logging.config.file system property can be set to the file that contains the log level for OPMN Audit Loader

Config/Proxy Mbeans

DomainName/servers/$SERVER_NAME/logs/$SERVER_NAME-diagnostic.log

oracle.security.audit.logger (See instructions below)

Audit Schema Support

RCU log location (Default is $ORACLE_HOME/rcu/log/)RCU_LOG_LOCATION can be set to change this location

RCU log level (Default is ERROR) RCU_LOG_LEVEL - [SEVERE; ERROR; NOTIFICATION; TRACE


I.1.1.3.1 Configuring the Loggers

You can configure oracle.security.audit.logger using Fusion Middleware Control.

oracle.security.audit.logger can take any log level from ERROR to TRACE allowing control over the amount of information that gets logged.

You can also view these diagnostic files with Fusion Middleware Control.

See Also:

For more information about the following topics, see chapter 10, Managing Log Files and Diagnostic Data, in Oracle Fusion Middleware Administrator's Guide:
  • instructions for configuring the loggers

  • details on viewing logs from domain, server, and each application

I.1.1.3.2 Interpreting Audit Diagnostics

The Audit diagnostic messages can be categorized into two types - errors and trace messages.

All error messages are numbered IAU-XXX. These messages are found in the Error Message Guide with a proper cause and an action that can be taken to rectify the error.

The trace messages, however, are meant to provide more information about the running components. Depending on their nature, the message may or may not require any action on your part.

I.1.1.4 Using Fusion Middleware Control Logging Support

Fusion Middleware Control provides several pages to manage log information. Using this tool you can:

  • Configure several attributes of a log file, including the log level and rotation.

  • Search the contents of all log files in a domain and group the results of a query by message ID or type.

  • Correlate a given error with others by context or time span.

  • Download a portion of a log file or the results of a query in one of several formats.

This section explains briefly how to configure a log file. The other three functions above are explained, also briefly, in section Section I.1.3, "Solving Security Errors."

For full details about these topics, see section Managing Log Files and Diagnostic Data, in the Oracle Fusion Middleware Administrator's Guide.

To configure a log file with Fusion Middleware Control, proceed as follows:

  1. Navigate to Server > Logs > Log Configuration, to display the Log Configuration page for the selected server. This page allows you to configure the log level for both persistent loggers and active run-time loggers.

  2. Click the Log File entry for the desired logger, to display the page showing the current parameter settings for that file.

  3. In this page, select a row and then click the button Edit Configuration, to display the Edit Log File dialog, where you can set various parameters, including the log level and the rotation policy.

I.1.2 System Properties

To increase the logging output with Java options at server start, add the following Java options to the script that starts your Oracle WebLogic Server and restart the server:

Two other system properties that can be passed at server start and that can help debugging security issues are the following:

  • -DDebugOPSSPolicyLoading, a flag that monitors the progress and setting of the OPSS policy provider.

  • -Djava.security.debug=policy, the standard Java security debug flag that produces print information about policy files as they are parsed, including their location in the file system, the permissions they grant, and the certificates they use for signed code.

Note:

A consequence of setting a high logging output is that many threads may be reported in a stuck state, specially when file loading takes place. To avoid this situation, change the time out value that Oracle WebLogic Server uses to mark a thread as stuck to a higher value.

A system property cannot be set without restarting the server. In order to set a system property the administrator must edit the setDomainEnv.sh shell script and add the property to the environment variable EXTRA_JAVA_PROPERTIES in that script.

I.1.2.1 jps.auth.debug

Assume that just this system property is set to true:

-Djps.auth.debug=true 

Then, a permission check that fails generates an output with details illustrated in the following sample:

[JpsAuth] Check Permission
          PolicyContext:        [jps-wls-Demo]
          Resource/Target:      [app.monitor]
          Action:               [read,write]
          Permission Class:     [java.util.PropertyPermission]
          Evaluator:            [ACC]
          Result:               [FAILED] 
Failed ProtectionDomain:ClassLoader=weblogic.servlet.jsp.JspClassLoader@fb111c finder: weblogic.utils.classloaders.CodeGenClassFinder@106bb21 annotation:
CodeSource=file:/C:/MyOracle/domains/base_domain/servers/AdminServer/tmp/_WL_user/jps-wls-Demo/kebqfo/jsp_servlet/test.class
Principals=total 5 of principals(
 1. weblogic.security.principal.WLSUserImpl "duane"
 2. weblogic.security.principal.WLSGroupImpl "employee"
 3. JpsPrincipal: oracle.security.jps.principals.JpsAuthenticatedRoleImpl "authenticated-role"
 4. JpsPrincipal: oracle.security.jps.principals.JpsAnonymousRoleImpl "anonymous-role"
 5. JpsPrincipal: oracle.security.jps.service.policystore.ApplicationRole "appRoleEmployee")
 Permissions=(
 (java.util.PropertyPermission line.separator read)
 ...
 (oracle.security.jps.service.credstore.CredentialAccessPermission context=SYSTEM,mapName=default,keyName=* read,write))

And a permission check that succeeds generates no output.

I.1.2.2 jps.auth.debug.verbose

Assume that jps.auth.debug and jps.auth.debug.verbose are both set to true:

-Djps.auth.debug=true 
-Djps.auth.debug.verbose=true

Then, a permission check that succeeds generates an output with details illustrated in the following sample:

[JpsAuth] Check Permission
          PolicyContext:        [jps-wls-Demo]
          Resource/Target:      [app.monitor]
          Action:               [read,write]
          Permission Class:     [java.util.PropertyPermission]
          Result:               [SUCCEEDED]
          Subject:              [total 5 of principals(
 1. weblogic.security.principal.WLSGroupImpl "manager"
 2. weblogic.security.principal.WLSUserImpl "shawn"
 3. JpsPrincipal: oracle.security.jps.internal.core.principals.JpsAuthenticatedRoleImpl "authenticated-role" GUID=null DN=null
 4. JpsPrincipal: oracle.security.jps.internal.core.principals.JpsAnonymousRoleImpl "anonymous-role" GUID=null DN=null
 5. JpsPrincipal: oracle.security.jps.service.policystore.ApplicationRole "appRoleManager" GUID=null DN=null)]
          Evaluator:            [ACC] 

And a permission check that fails generates an output with details illustrated in the following sample:

JpsAuth] Check Permission
          PolicyContext:        [jps-wls-Demo]
          Resource/Target:      [app.monitor]
          Action:               [read,write]
          Permission Class:     [java.util.PropertyPermission]
          Evaluator:            [ACC]
          Result:               [FAILED]
          Failed ProtectionDomain:ClassLoader=weblogic.servlet.jsp.JspClassLoader@1b7682d finder: weblogic.utils.classloaders.CodeGenClassFinder@7d32cf annotation:
CodeSource=file:/C:/Mydom/domains/domain/servers/AdminServer/jspservlet/test.class
 Principals=total 5 of principals(
 1. weblogic.security.principal.WLSUserImpl "duane"
 2. weblogic.security.principal.WLSGroupImpl "employee"
 3. JpsPrincipal: oracle.security.principals.JpsAuthenticatedRoleImpl "authenticated-role" GUID=null DN=null
 4. JpsPrincipal: oracle.security.principals.JpsAnonymousRoleImpl "anonymous-role" GUID=null DN=null
 5. JpsPrincipal: oracle.security.jps.service.policystore.ApplicationRole "appRoleEmployee" GUID=null DN=null)
 Permissions=(
 (java.util.PropertyPermission line.separator read)
 ... 
 (java.lang.RuntimePermission stopThread))
 Call Stack: java.security.AccessControlException: access denied (java.util.PropertyPermission app.monitor read,write)
 java.security.AccessControlContext.checkPermission(AccessControlContext.java:323)
 ...
 weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
          ProtectionDomains for class stack:
 Class[0]: class oracle.security.jps.util.JpsAuth$Diagnostic$SMSupport
 ProtectionDomain: ClassLoader=sun.misc.Launcher$AppClassLoader@360be0
 CodeSource=file:/C:/MyOracle/jdeveloper/modules/oracle.jps_11.1.1/jps-api.jar
 Principals=total 0 of principals<no principals>
 Permissions=(
 (java.io.FilePermission \C:\MyOracle\jdeveloper\modules\jps-api.jar read)
 ...
 )
 Class[1]: class oracle.security.jps.util.JpsAuth$Diagnostic$SMSupport 

I.1.3 Solving Security Errors

There is no generic way to resolve errors when they occur. One must search for hints and frequently follow multiple hypotheses until, hopefully, the source of the error is isolated and understood. To this end, this section describes how to search and interpret log information to resolve most common security errors. These topics are addressed in the following sections:

I.1.3.1 Understanding Sample Log Entries

Understanding log error output is crucial to isolate and solve an error. Let's take a closer look at a diagnostic log file to describe the information you find for an error logged in such a file. This description is best illustrated with a real-life example.

The following is an excerpt of an error in the file AdminServer-diagnostic.log:

[2009-01-07T09:15:02.393-08:00] [AdminServer] [ERROR] [JPS-00004] [oracle.jps.admin] 
[tid: [ACTIVE].ExecuteThread: '3' for queue: 'weblogic.kernel.Default
(self-tuning)'] [userId: weblogic] [ecid: 0000Hum5kxw7MAn54nU4Ui19PD8S000005,0]
Unable to add principal to the application role. Reason: Principal
"abc.xxx@myComp.com" is already a member of the application role
"BPMWorkflowAdmin"[[
java.security.PrivilegedActionException:
oracle.security.jps.service.policystore.PolicyObjectAlreadyExistsException:
Unable to add principal to the application role. Reason: Principal "abc.xxx@myComp.com" is already a member of the application role
"BPMWorkflowAdmin"
        at java.security.AccessController.doPrivileged(Native Method)
        at oracle.security.jps.mas.mgmt.jmx.policy.JpsApplicationPolicyStoreImpl.
addRemoveMembersToRole(JpsApplicationPolicyStoreImpl.java:408)
        at oracle.security.jps.mas.mgmt.jmx.policy.JpsApplicationPolicyStoreImpl.
addMembersToApplicationRole(JpsApplicationPolicyStoreImpl.java:385)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
…

The meaning of the fields in the preceding message is as follows:

  • [2009-01-07T09:15:02.393-08:00]

    Identifies the date and time when the error was logged.

  • [AdminServer]

    Identifies the name of the server where the error occurred.

  • [JPS-00004]

    Identifies the error code and hints to the kind of error that occurred. For a complete list of JPS error codes, see chapter 41 in Oracle Fusion Middleware Error Messages Reference.

  • [oracle.jps.admin]

    Identifies the category of the logger. The subcategories of oracle.jps (such as admin above) hint to the kind of error that occurred. For the complete list of categories under oracle.jps, see Subcategories of oracle.jps.

  • [tid: [ACTIVE].ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)']

    Identifies the thread where the error occurred.

  • [userId: weblogic]

    Identifies the user that performed the operation that generated the error.

  • [ecid: 0000Hum5kxw7MAn54nU4Ui19PD8S000005,0]

    Identifies the execution context id. Typically used to correlate and trace sequence of events. Ecids provide information about the flow across processes, such as, from a request, to the WebLogic server, to an Oracle Internet Directory server.

  • Unable to add principal to the application role. Reason: Principal abc.xxx@myComp.com is already a member of the application role BPMWorkflowAdmin

    Identifies the reason why the error was logged.

  • java.security.PrivilegedActionException: oracle.security.jps.service.policystore.PolicyObjectAlreadyExistsException: Unable to add principal to the application role. Reason: Principal abc.xxx@myComp.com is already a member of the application role BPMWorkflowAdmin

    Identifies the exception that was raised and the reason for it.

Subcategories of oracle.jps

Here is the list of subcategories under oracle.jps and the kind of errors logged in the category:

  • common - generic errors.

  • config - configuration errors.

  • deployment - deployment errors.

  • authentication - login module errors in JavaSE applications only.

  • idmgmt - identity store errors.

  • credstore - credential store errors.

  • authorization - policy store errors at run time.

  • policymgmt - policy store management errors.

  • admin - JMX and WLST errors.

I.1.3.2 Searching Logs with Fusion Middleware Control

To initiate a search in the contents of all log files in a domain, select Domain > Logs > View Log Messages, to display the Log Messages page.

In this page you have several parameters that you can choose from to specify your search query; specifically, you can:

  • Choose a time interval in which a message was issue, by selecting the appropriate Date Range.

  • Display messages with a given severity error, by checking any of the Message Types boxes.

  • Display messages satisfying further constrains, by choosing an item from the menu Message and entering a string in the box next to it. For example, you could query for just messages that contain the string exception in it.

  • Add extra query fields, by clicking the button Add Fields and checking any of the available choices. For example, you could add the field Host, and then enter the appropriate query, such as starts with a particular string.

Once these parameters are set, click Search and the result of the query is displayed in the page. The result of a query can be further redisplayed by message type, message ID, or simple list of messages, by selecting an item from the menu Show. Moreover, the result can be automatically refreshed by choosing an item from the menu at the top right of the page (by default set to Manual Refresh).

To broaden a search to log files beyond a domain, use the button Broaden Target Scope at the top right of the page.

I.1.3.3 Identifying a Message Context with Fusion Middleware Control

In some situations, it is necessary to know the context in which a message has occurred. For example, it may be useful to know messages that have preceded or followed a given error message by, say, 2 minutes.

The tab View Related Messages provides this functionality, and you can use it as follows:

  1. Display the results of a query with Show set to Messages.

  2. Select a message within the result table. Note that the tabs View Related Messages and Export Messages to File become then available. Let's assume, for example, that the selected message has the time stamp Jan 21, 2009 4:05:00 PM PST.

  3. Select Time Interval from the Date Range menu, and enter a Start Date and an End Date. For example, you could enter Jan 21, 2009 4:02:00 PM, as a start date, and Jan 21, 2009 4:07:00 PM.

  4. Select by Time from the menu View Related Messages, to display the page with all the messages related to the selected one in the specified time span.

  5. In the Related Messages by Time page, you can modify the time window around the time of the selected message by choosing an item from the menu Scope, at the right of the page.

I.1.3.4 Generating Error Listing Files with Fusion Middleware Control

In some situations, you may want to download the list of errors displayed into a separate file to forward it, for example, to a support center, or just to keep it for your records.

Whenever available, the tab Export Messages allows you to generate a file containing just the displayed results by choosing an item from the menu. The format of the generated file can be plain text, XML, or CSV.

The following sample, showing only the first of 29 messages, is an excerpt of a text file generated this way:

#
#Search Criteria
#        Start Time: 2009-01-21T16:34:41.381-08:00
#        End Time:  2009-01-21T16:39:41.381-08:00
#        Message Types: ERROR, WARNING
 
#Selected Targets List
#        /Farm_base_domain/base_domain/AdminServer:Oracle WebLogic Server
#        /Farm_base_domain/base_domain/AdminServer/DMS Application(11.1.1.1.0):Application Deployment
#        /Farm_base_domain/base_domain/AdminServer/em:Application Deployment
#        /Farm_base_domain/base_domain/AdminServer/wsil-wls:Application Deployment
#        /Farm_base_domain/base_domain/AdminServer/wsm-pm:Application Deployment
#
[2009-01-21T16:34:54.045-08:00] [AdminServer] [WARNING] [] [org.apache.myfaces.trinidad.bean.PropertyKey] [host: stacz39] [nwaddr: 140.87.5.40] [tid: 13] [userId: <anonymous>] [ecid: 0000HvvkgjVE^MT6uBj8EH19TvXj000008,0] [APP: em] [Target: /Farm_base_domain/base_domain/AdminServer/em] [Target Type: Application Deployment] Unserializable value:oracle.sysman.core.view.tgtctls.common.DefaultTreeModel@1fcadd2 for key:UINodePropertyKey[value,17]
…
#
#Number of messages exported: 29
#

I.2 Reassociation Failure

Policy and credential reassociation from an XML-based store to an LDAP-based store may fail for several reasons. This section explains three reasons why this operation may fail.

Symptom 1- Error Code 32

Reassociation fails and an error like the following is logged in the administration server diagnostic file serverName.diagnostic.log:

[LDAP: error code 32 - No Such Object]
Authentication to LDAP server ldap://myServer.com:3060 is unsuccessful. 

Diagnosis 1

The error above identifies a problem with the target node in the LDAP server, namely, that the node specified does not exist.

It is required that the root node specified in the text box JPS Root DN (of the page Set Security Provider) be present in the LDAP directory before invoking the reassociation.

Solution 1

Verify that the data you enter in the box JPS Root DN matches the name of a node in the target LDAP directory, and then rerun the reassociation.

Symptom 2- Error Code 68

Reassociation fails and an error like the following is logged in the administration server diagnostic file serverName.diagnostic.log:

Authentication to LDAP server ldap://myServer.com:3060 is successful.
Starting to migrate policy store...
Set up security provider reassociation successfully.
Checked and seeded security store schema successfully.
null
[LDAP: error code 68 - Object already exists]:cn=SystemPolicy,cn=domain1,cn=JPSContext,cn=nb_policy
Error occurred while migrating LDAP based policy store. 

Diagnosis 2

The error above indicates that the name specified in the box WebLogic Domain Name is a descendant (more precisely, a grandchild) of the JPS Root DN node in the target LDAP directory.

It is required that the domain specified do not be a descendant of the root node.

Solution 2

Verify that the name you enter in the box WebLogic Domain Name does not match the name of a grandchild of the specified JPS Root DN node, and rerun the reassociation.

Symptom 3

Reassociation, carried out with Fusion Middleware Control, fails and an error like the following is logged in the administration server diagnostic file serverName.diagnostic.log:

[2009-01-21T10:09:24.326-08:00] [AdminServer] [ERROR] [] [oracle.jps.admin] [tid
: [ACTIVE].ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)
'] [userId: weblogic] [ecid: 0000HvuOTpe7q2T6uBADUH19Tpyb000006,0] Unable to rem
ove the principal from the application role. Reason: Principal "Managers" is not
a member of the application role "test-role"[[
java.security.PrivilegedActionException: oracle.security.jps.service.policystore
.PolicyObjectNotFoundException: Unable to remove the principal from the applicat
ion role. Reason: Principal "Managers" is not a member of the application role "
test-role"
       at oracle.security.jps.mas.mgmt.jmx.policy.JpsApplicationPolicyStoreImpl
.addRemoveMembersToRole(JpsApplicationPolicyStoreImpl.java:408)... 

Diagnosis 3

The error above points to some problem with the application role test-role, which is, in this case, the root of the problem.

Ensure that when entering data to perform reassociation with Fusion Middleware Control, you use the button Test LDAP Authentication immediately after you have completed entering all required values to connect to the target LDAP server. This test catches any problems with those values before reassociation begins.

Solution 3

In our example, a quick inspection of the file system-jazn-data.xml reveals that the application test-role is used by an application policy, but it was not defined. Here is an excerpt of that file illustrating where the required data is missing:

<application>
    <name>myApp</name>
        <app-roles>
 <--! test-role should have been defined here -->
        </app-roles>
        <jazn-policy>
            <grant>
                <grantee>
                    <principals>
                       <principal>
                         <class>
oracle.security.jps.service.policystore.ApplicationRole</class>
                         <name>test-role</name>
                         <guid>66368900E7E511DD9F62F9ADA4233FE2</guid>
                       </principal>
                     </principals>...

To solve this particular error, (a) fix the system-jazn-data.xml by inserting the definition of the application test-role; (b) revert to file-based domain stores with the fixed file; and (c) rerun the reassociation.

I.3 Server Fails to Start - Missing Required LDAP Authenticator

This section explains a reason why the Oracle WebLogic Server may fail to start after modifying the list of authenticators in a domain.

Symptom

After modifying the list of authenticator providers in a domain, the Oracle WebLogic Server fails to start, and the error messages output include the following:

java.lang.IllegalArgumentException: null KeyStore name

Diagnosis

One cause of this problem is that the list of authenticators in your domain does not include an LDAP authenticator.

Important:

An LDAP authenticator is required in this list for any domain using OPSS.

Solution

Since the server cannot start, you must add one LDAP authenticator manually, as follows:

  1. Open the file DOMAIN_NAME/config/config.xml.

  2. Edit config.xml and include, within the element <realm>, an LDAP authenticator, such as the default authenticator illustrated in the following sample:

    <realm>
     ...
     <sec:authentication-provider xsi:type="wls:default-authenticatorType"> 
     </sec:authentication-provider>
     ...
    </realm>
    
  3. Restart the server.

Once the server is back up and running, you can modify the list of providers to include the provider of your choice using the WebLogic Administration Console, but ensure that at least one of them is an LDAP authenticator provider.

To this end, use the WebLogic Administration Console as follows:

  1. Navigate to the page Create a new Authenticator Provider.

  2. Enter the authenticator name and select an authenticator type, all of which are LDAP-based:

    • ActiveDirectoryAuthenticator

    • DefaultAuthenticator (this is the one inserted manually in the sample above)

    • LDAPAuthenticator

    • LDAPX509IdentityAsserter

    • OpenLDAPAuthenticator

    • OracleInternetDirectoryAuthenticator

    • OracleVirtualDirectoryAuthenticator

I.4 Server Fails to Start - Missing Administrator Account

This section explains a reason why the Oracle WebLogic Server may fail to start.

Symptom

After removing the out-of-box default authenticator and adding, say an Oracle Internet Directory authenticator, the server fails to start.

Diagnosis

Most likely, you have forgotten to enter an account member of the Administrators group in your added authenticator. The server requires that such an account be present in one domain authenticator. This account is always present in the default authenticator.

Solution

Since the server cannot start, you must add the deleted one LDAP authenticator manually, as follows:

  1. Open the file DOMAIN_NAME/config/config.xml.

  2. Edit config.xml and include, within the element <realm>, the default authenticator, as illustrated in the following sample:

    <realm>
     ...
     <sec:authentication-provider xsi:type="wls:default-authenticatorType"> 
     </sec:authentication-provider>
     ...
    </realm>
    
  3. Restart the server.

Once the server is back up and running, proceed as follows:

  1. Use the WebLogic Administration Console to create in the Oracle Internet Directory authenticator an account that is member of the Administrators group.

  2. Set the Oracle Internet Directory authenticator flag to SUFFICIENT.

  3. Restart the server, which it should start without problems, since it is using the account in the Administrators group provided in the default authenticator.

  4. Reset the Oracle Internet Directory authenticator flag to REQUIRED and remove the default authenticator. The server should now start using the account in the Administrators group that you created in the Oracle Internet Directory authenticator.

I.5 Server Fails to Start - Missing Permission

This section explains a reason why the Oracle WebLogic Server may fail to start.

Symptom

The server fails to start when it started with security manager is enabled (with the system property -Djava.security.manager).

Diagnosis

One reason why you may run into this issue is the lack of permission grants to PKI APIs in oraclepki.jar when the security manager is enabled at server startup.

Solution

Ensure that a grant like the following is present in the file weblogic.policy, or add it if it is not:

grant codeBase "file:${oracle.home}/modules/oracle.pki_${jrf.version}/*" { 
 permission java.security.AllPermission; 
};

The above grant is provided by default. Note that when security manager is enabled, the access to all system resources requires codebase permission grants.

For complete details about using the Java Security Manager to protect WebLogic resources, see Oracle Fusion Middleware Programming Security for Oracle WebLogic Server.

Note:

Printing Security Manager is a WebLogic server enhancement to the Java Security Manager. Use Printing Security Manager to identify all of the required permissions for a Java application running under Java Security Manager. Unlike the Java Security Manager, which identifies needed permissions one at a time, the Printing Security Manager identifies all the needed permissions without intervention.

I.6 Failure to Grant or Revoke Permissions - Case Mismatch

This section explains the likely reasons why an enterprise user or role (group) may fail to be granted or revoked permissions.

Symptom

An enterprise user or group, properly entered in a domain authenticator, is not granted or revoked the permissions defined by a grant.

Diagnosis

This problem is likely to occur when there is a case mismatch between the stored name (in a domain authenticator) and the supplied name (either actively entered by a user or obtained programmatically). For example, this mismatch would occur when the stored user name is JdOE and the supplied user name is jdoe.

Solution

There are two ways to resolve this issue.

The first solution involves setting the appropriate property in the authenticator being used in your domain. As long as both strings (the supplied and the stored) contain identical sequence of characters (irrespective of case), this setting guarantees that the user name populated in the Subject matches the user name present in a domain authenticator, even when the corresponding characters differ in case. Thus, when this setting is in place, the user names JdOE and jdoe match.

To set your domain authenticator property, proceed as follows:

  1. Use the Administration Console to navigate to the page where your authenticator is configured. For example, if you are using the default authenticator, navigate to the DefaultAuthenticator page by choosing Realms > myrealm > Providers > DefaultAuthenticator.

  2. Choose the tab Provider Specific.

  3. Set the property userRetrievedUserNameAsPrincipal to true.

  4. Restart the server.

The second solution considers the case where the supplied name is obtained programmatically, that is, where one must produce a principal from a user name.

To obtained the correct user or group name, either pass the name exactly as it is stored in the authenticator or use the sequence of calls illustrated in the following code snippet:

import weblogic.security.principal.WLSGroupImpl;
import weblogic.security.principal.WLSUserImpl;
 
// Set the context
JpsContextFactory ctxFact = JpsContextFactory.getContextFactory();
ServerContextFactory scf = (ServerContextFactory) ctxFact;
JpsContext ctx = scf.getContext(ServerContextFactory.Scope.SYSTEM);
ctx = ctxFact.getContext();

// Set the identity store
IdentityStore identityStore = ctx.getServiceInstance(IdentityStoreService.class).getIdmStore();
 
// In case of a user name, search the user that matches the supplied name
User user = idStore.searchUser(IdentityStore.SEARCH_BY_NAME, suppliedUserName);

// Use the obtained object (user) to obtain the stored user name and create 
// the Principal
String storedUserName = user.getName();
Principal userPrincipal = new WLSUserImpl(storedUserName);
 
// Similarily, in case of a role name, search the role that matches 
// the supplied role name
Role role = identityStore.searchRole(IdentityStore.SEARCH_BY_NAME, suppliedRoleName);
 
// Use the obtained object (role) to obtain the stored role name and create 
// the Principal
String storedRoleName = role.getName();
Principal rolePrincipal = new WLSGroupImpl(storedRoleName);

Important:

When creating a user or role principal, you must use the calls:
Principal userPrincipal = new WLSUserImpl(user.getUserProfile()getName());
Principal rolePrincipal = new WLSGroupImpl(role.getRoleProfile().getName());

Instead of the calls:

Principal userPrincipal = new WLSUserImpl(user.getName());
Principal rolePrincipal = new WLSGroupImpl(role.getName());

I.7 Failure to Connect to an LDAP Server

This section explains the likely reasons why a connection to an Oracle Virtual Directory or Oracle Internet Directory LDAP server can fail. This failure can also happen during reassociation.

Symptom

The migration of data from a source repository to a target LDAP server repository fails.

Diagnosis

Typically, this kind of problem is due to an incorrect set up of parameters in the target LDAP server.

For further probing into Oracle WebLogic Server log files, search any of the log files in the directories DomainName/servers/AdminServer or DomainName/servers/ManagedServers for the following strings: <Error>, <Critical>, and <Warning>.

For more information about identifying and solving errors, see Section I.1, "Diagnosing Security Errors."

Solution

Verify that all the target server data provided for the migration is valid. You may require the assistance of your LDAP server administrator to perform this validation.

Note:

If you are using Fusion Middleware Control to reassociate to an LDAP server, ensure that you use the button Test LDAP Authorization before initiating the operation. Typically, this test catches incorrect supplied parameters.

I.8 User and Role API Failure

This section explains some reasons why you may fail to access data in a domain authenticator with the User and Role API.

Symptom

The User and Role API fails to access data in a configured authenticator.

Diagnosis 1

The OPSS User and Role API can access data only in the first LDAP authenticator configured in a domain. At least one such authenticator must be present in a domain. The API access to that first LDAP authenticator fails if the target user is not present in that authenticator, even though that user is present in some other domain authenticator.

Solution 1

Enter the missing user in the first LDAP authenticator, or reorder the list of LDAP authenticators in your domain.

Diagnosis 2

Let's assume that the target user on which the API that fails is present in the first LDAP authenticator configured in your domain.

By default, the User and Role API uses the attribute uid to perform user search in an LDAP authenticator. If for some reason, a user entered in the LDAP is lacking this attribute, then the User and Role API fails.

Solution 2

Ensure that all users in the first LDAP authenticator have the attribute uid set.

Note:

If you want the User and Role API to employ an attribute other than the default one to search users, say mail for example, then the LDAP authenticator instance must be configured to contain the properties username.attr and login.name.attr, as illustrated in the following code snippet:
<serviceInstance provider="idstore.ldap.provider" name="idstore.ldap">
   ...
   <property name="username.attr" value="mail"/>
   <property name="login.name.attr" value="mail"/>
   ...
</serviceInstance>

To add properties to a provider instance with a prescribed script, see Section E.1, "Configuring OPSS Service Provider Instances with a WLST Script."

I.9 Failure to Access Data in the Domain Credential Store

This section explains a likely reason why an application fails to access data in the domain's credential store.

Symptom

An application fails to retrieve credential data from the domain's credential store, and an error message (containing lines like the one illustrated below) is logged (text in between brackets should describe information specific to the particular failure):

07/07/26 18:22:22 [JpsAuth] For permisson ( CredentialAccessPermission [target] [actions]), domain that failed: ProtectionDomain
 cs(file:somePath/aWarFile.war/WEB-INF/classes/), []

Diagnosis

If an application is to access the credential store to perform an operation (such as retrieving a user password, for example), then its code must be granted the appropriate permission to perform the secured operation; otherwise, the application runs into an error like the one described above.

Solution

To grant the permission that an application requires to access the credential store, include the appropriate CredentialAccessPermission in the application's jazn-data.xml; this grant takes effect when the application is deployed or redeployed.

To include a permission using Fusion Middleware Control, see Section 8.4.1, "Managing Policies with Fusion Middleware Control."

To include a permission using a WLST command, see Section 8.4.2, "Managing Policies with WLST Commands."

The following fragment of the file jazn-data.xml illustrates how to grant all code in the application myApp permission to read all credentials in the folder myAlias:

<jazn-data>
    <!--  codebase policy -->
    <jazn-policy>
        <grant>
            <grantee>
                <codesource>
 <!-- This grants applies to all code in the following directory -->
                    <url>${domain.home}/tmp/_WL_user/myApp/-</url>
                </codesource>
            </grantee>
            <permissions>
                <permission>
                    <class>oracle.security.jps.service.credstore.CredentialAccessPermission</class>
<!-- Allow read permission to all credentials under folder MY_MAP -->
                    <name>context=SYSTEM,mapName=MY_MAP,keyName=*</name>
                    <actions>read</actions>
                </permission>
            </permissions>
        </grant>
    </jazn-policy>
</jazn-data>

I.10 Failure to Establish an Anonymous SSL Connection

This section explains the likely reasons why you are not able to establish an anonymous SSL connection while reassociating policies and credentials.

Symptom

A step in the reassociation of file-based policies and credentials to an LDAP-base storage using an Oracle Internet Directory or an Oracle Virtual Directory server with Fusion Middleware Control, involves testing the anonymous SSL connection to the LDAP server (specifically with the button Test LDAP). This test fails.

Diagnosis

Your target LDAP server must be trusted by the Oracle WebLogic Server and the port number you are using to the LDAP server must be an SSL port.

Solution

Establishing a connection to an LDAP server requires some previous configuration on the LDAP server. For details, see Section 8.1.2, "Prerequisites to Using an LDAP-Based Policy Store."

In addition, to use an anonymous SSL connection, you must enter a port that has been set for receiving secure data. If your LDAP server has not been configured with such a port, the connection fails.

Ensure that the supplied LDAP server port is an SSL port configured to listen in anonymous SSL mode, and that the supplied server name reachable. Typically, the setting of this port involves an LDAP server administrator.

I.11 Authorization Check Failure

This section explains a reason why an authorization check has failed.

Symptom

An attempt to authorize a user by your application fails, and the system logs an error containing a line like the following:

Servlet failed with Exception oracle.adf.controller.security.AuthorizationException:ADFC-0619:
Authorization check failed: '/StartHere.jspx' 'VIEW'. 

Diagnosis

One reason that can lead to such an authorization failure is a mismatch between the run-time policy context and the policy store stripe that you application is using.

On the one hand, the application stripe (or subset of policies in the domain policy store) that an application uses is specified in the file web.xml with the parameter application.name within the filter configuring the JpsFilter (for a servlet) or the interceptor configuring the JpsInterceptor (for an EJB). For details and samples, see Application Name (Stripe). If the application stripe is not specified (or left blank), then the system picks up an application stripe based on the application name.

On the other hand, the run-time policies that your application uses are specified in the file system-jazn-data.xml with the element <application.name>.

If those two names do not match or if you have not explicitly specified the stripe to use, then, most likely, your application is accessing the wrong policy stripe and, therefore, not able to authorized your application users as expected.

Solution

Ensure that you specify explicitly your application stripe, and that stripe is the one that your application is supposed to use. In most cases, the two names specified in those two different files (as explained above) match; however, in cases where several applications share the same policy stripe, they may differ.

I.12 User Gets Unexpected Permissions

This section explains the likely reasons why a user gets permissions other than those anticipated.

Symptom

A new user or a modified user gets unexpected permissions.

Diagnosis

This issue is likely to come up in cases where a user is added with the name of previously removed user, or an old user gets its name or uid changed. The common reason why the user may get more or less permissions than expected is that the policy store has not been properly updated before a user is removed or a user data is changed.

Solution

Before deleting a user, revoke all permissions, application roles, and enterprise groups that had been granted to the user. If you fail to remove all security artifacts referencing a user to be deleted, they are left dangling and, potentially, inherited if another user with the same name or uid is created at a later time.

Similar considerations apply to when a user name or uid is changed: all policies (grants, permissions, roles) referring to the old data must be updated so that they work as expected with the new data.

I.13 Security Access Control Exception

This section explains a reason why your code may run into a security access control exception.

Symptom

At run time, your application outputs an error like the following one (only the first few lines are shown):

<Jan 20, 2009 5:45:33 PM PST> <Error> <HTTP> <BEA-101020>
<[weblogic.servlet.internal.WebAppServletContext@140cf52 
- appName: 'Application2', 
name: 'Application2.war', 
context-path: '/Application2', 
spec-version: '2.5'] 
Servlet failed with Exceptionjava.lang.RuntimeException:java.security.AccessControlException:access denied
...

Diagnosis

The above error means that a call in your code does not have sufficient permissions to execute a secured operation.

Solution

Your code must be granted the appropriate permissions to execute the secured operation. Depending on the scope of the permission you would like to set, you have two alternatives.

The first one is to grant permission to all application code in the application's EAR or WAR files; in this case, the call to the secured operation can be inserted anywhere in the application code.

The second one is to grant permission to just a JAR file; in this case, the call to the secured operation must be inside a privileged block.

Each of these solutions is next illustrated by an application attempting to access the credential store.

The following fragment of an applicationjazn-data.xml illustrates how to set permission to read any key within the map MY_MAP in the credential store to any code within the directory BasicAuth:

<jazn-policy>
   <grant>
       <grantee>
           <codesource>
              <url>file:${domain.home}/servers/_WL_user/BasicAuth/-</url>
           </codesource>
       </grantee>
       <permissions>
           <permission>
             <class>
                 oracle.security.jps.service.credstore.CredentialAccessPermission
             </class>
             <name>context=SYSTEM,mapName=MY_MAP,keyName=*</name>
             <actions>read</actions>
          </permission>
      </permissions>
   </grant>
</jazn-policy>

If the permission is to be granted to the code in a particular EAR or WAR file, the url specification above would have to be changed to one like the following:

<url>file:${domain.home}/servers/_WL_user/jpsBasicAuth/.../BasicAuth.ear</url>

In both above cases, the call to read the credential store can be placed anywhere in the application code.

If, however, the permission is to be granted to just the code in a particular JAR file, the url specification above would have to be changed to one like the following:

<url>file:${domain.home}/servers/_WL_user/jpsBasicAuth/myJars/Foo.jar</url>

In this last case, the code in the file Foo.jar that calls a read operation on the credential store must be placed in an AccessController.doPrivileged block, as illustrated in the following code snippet:

import oracle.security.jps.*;
import oracle.security.jps.service.credstore.*;
 
JpsContextFactory factory = JpsContextFactory.getContextFactory();
JpsContext jpsContext = factory.getContext();
final CredentialStore store = jpsContext.getServiceInstance(CredentialStore.class);
Credential cred = AccessController.doPrivileged(new PrivilegedExceptionAction<PasswordCredential>() {
    public PasswordCredential run() throws JpsException {
        return store.getCredential("MY_MAP", "anyKey");
    }
});
           
PasswordCredential pwdCred = (PasswordCredential)cred;
...

Note that since our sample grant above allows only read permission, none of the set or reset operations work, even inside a privileged block.

I.14 Permission Check Failure

This section explains a reason why a permission may fail to pass a permission check.

Symptom

At run time, your application outputs an error like the following one (only the first few lines are shown):

[JpsAuth] Check Permission
          PolicyContext:        [null]
          Resource/Target:      [test]
          Action:               [null]
          Permission Class:     [com.oracle.permission.SimplePermission]
          Evaluator:            [ACC]
          Result:               [FAILED]
          Failed ProtectionDomain:ClassLoader=weblogic.utils.classloaders.ChangeAwareClassLoader@14061a8 
finder: weblogic.utils.classloaders.CodeGenClassFinder@2dce7a8 
annotation: Application2@Application2.war
CodeSource=file:/scratch/servers/AdminServer/tmp/permission/TestServlet$1.class
Principals=total 0 of principals<no principals>
Permissions=(
(oracle.security.jps.service.credstore.CredentialAccessPermission 
context=SYSTEM,mapName=default,keyName=* read,write)
(java.net.SocketPermission localhost:1024- listen,resolve)
(oracle.security.jps.service.policystore.PolicyStoreAccessPermission
context=APPLICATION,name=* getApplicationPolicy)
(oracle.security.jps.service.policystore.PolicyStoreAccessPermission
context=SYSTEM getConfiguredApplications)
(com.oracle.permission.SimplePermission *)
...
java.security.AccessControlException: access denied
(com.oracle.permission.SimplePermission test)...

Diagnosis

When two or more applications share a permission class, that permission class must be set in the system class path so the class is loaded just once. Otherwise, only the first application loading the class passes the permission check; other ones loading the same class thereafter may fail the permission check and output an error like the one illustrated above.

Note that even though the permission class is in the permission collection (see bold text in sample output above), the check fails and the access is denied. This is because, at that point, the environment contains several instances of a permission class with the same name.

Solution

Ensure that if two or more applications to be run in the same domain share a permission class, then include that class in the system class path.

I.15 Policy Migration Failure

This section describes a reason why the automatic migration of policies at application deployment may fail. Note that the deployment of an application may succeed even though the migration of policies failed.

Note:

The reason why the automatic migration can fail, as explained in this section, can also lead to similar failures when reassociating domain stores.

Symptom

The application is configured to migrate policies automatically at deployment. The application deployment succeeds, but the diagnostic file corresponding to the server where it has been deployed outputs a message like the following:

[2009-01-21T13:34:48.144-08:00] [server_soa] [NOTIFICATION] []
[oracle.jps.deployment] [tid: [ACTIVE].ExecuteThread: '2' for queue:
'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] 
[ecid: 0000Hvv7U_H7q2T6uBADUH19Tq0B00002I,0] [APP: JpsJdev#V2.0] 
Application [JpsJdev#V2.0] is being deployed, start policy migration.

[2009-01-21T13:34:48.770-08:00] [server_soa] [WARNING] [] 
[oracle.jps.deployment] [tid: [ACTIVE].ExecuteThread: '2' for queue:
'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] 
[ecid: 0000Hvv7U_H7q2T6uBADUH19Tq0B00002I,0] [APP: JpsJdev#V2.0] 
Exception in application policy migration.[[
oracle.security.jps.JpsException: appplication Role: 
test_role not found for the application in the destination policy store
at oracle.utility.destination.apibased.JpsDstPolicy.convertAppPolicyPrincipal
(JpsDstPolicy.java:815)
at oracle.utility.destination.apibased.JpsDstPolicy.clone
(JpsDstPolicy.java:691)...

The above excerpt was extracted from the file server_soa-diagnostic.log, and the application JpsJdev was deployed to the managed server server_soa. Note that the key phrase to look for to locate such error is highlighted in the sample above. In addition, the error describes the artifact that raised the exception, the application role test_role.

Diagnosis

Something is wrong with the definition of this role in the application file jazn-data.xml. In fact, a quick look at this file reveals that the role test_role is referenced in a grantee, as illustrated in the following excerpt:

<grantee>
   <display-name>myPolicy</display-name>
   <principals>
     <principal>
       <class>oracle.security.jps.service.policystore.ApplicationRole</class>
       <name>test_role</name>
     </principal>
   </principals>
</grantee> ...

But the name of what is supposed to be the application role named test_role, however, was inadvertently misspelled to test_rolle:

<application>
   <name>JpsJdev</name>
   <app-roles>
     <app-role>
       <name>test_rolle</name>
       <class>oracle.security.jps.service.policystore.ApplicationRole</class>
        <members> ...

Solution

Ensure that all application roles referenced in application policies have been properly defined in the jazn-data.xml file. If a referenced role name cannot be matched, as in the samples above, the migration fails.

I.16 Troubleshooting Oracle Business Intelligence Reporting

This section describes common problems and solutions for Oracle Business Intelligence when used as a reporting tool for Oracle Fusion Middleware security. It contains the following topics:

I.16.1 Audit Templates for Oracle Business Intelligence Publisher

To view Oracle Fusion Middleware Audit Framework reports in Oracle Business Intelligence, you must use the appropriate audit templates.

For details, see Section 13.1.3, "Set Up Oracle Reports in Oracle Business Intelligence Publisher".

I.16.2 Oracle Business Intelligence Publisher Time Zone

You may see problems with Oracle Fusion Middleware Audit Framework reports if Oracle Business Intelligence Publisher and the database are installed in sites with different time zones.

To avoid this issue, ensure that Oracle Business Intelligence Publisher and the database are installed in the same time zone.

I.17 Need Further Help?

You can find more solutions on My Oracle Support (formerly MetaLink) at http://myoraclesupport.oracle.com. If you do not find a solution to your problem, log a service request.