L Troubleshooting Security in Oracle Fusion Middleware

This appendix describes common problems that you may encounter when configuring and using Oracle Enterprise Manager Fusion Middleware security, and explains how to solve them. It contains the following sections:

L.1 Diagnosing Security Errors

This section the tools available to diagnose and solve a variety of security errors. It contains the following sections:

The logging support with Fusion Middleware Control is explicitly stated whenever the tool can help managing, isolating, or interpreting faults when they occur.

L.1.1 Log Files and OPSS Loggers

This section describes the various log files and OPSS loggers supported by Oracle WebLogic Server and how to configure, set logger levels, and locate and view log files with Fusion Middleware Control, in the following sections:

L.1.1.1 Diagnostic Log Files

Each server instance in a domain writes all OPSS-based exceptions raised by its subsystems and applications to a server log file in the file system of the local host computer.

By default, this log file is located in the logs directory below the server instance root directory. The names of these log files have the following format: ServerName-diagnostic.logxxxxx, where xxxxx denotes an integer between 1 and 99999.

Here are some examples of diagnostic file full names: DomainName/servers/AdminServer/logs/AdminServer-diagnostic.log00001 (administration server log), DomainName/servers/soa/logs/soa-diagnostic.log00013 (managed server log).

All server instances output security-related errors to diagnostic files. Server-related security errors, such as exceptions raised by issues with a subject or principal, and errors that may occur while migrating or reassociating domain security data, get written in the administration server diagnostic log. Application-related security errors, such as exceptions raised by application-specific policies or credentials, get written in the corresponding managed server diagnostic log.

L.1.1.2 Generic Log Files

In addition to diagnostic log files, Oracle WebLogic Server supports other log files for each server in a domain and for each domain in a topology.

By default and similar to diagnostic log files, server log files are located in the logs directory below the server instance root directory. Domain log files are located in the logs directory below the administration server root directory. The names of these log files have the format ServerName.logxxxxx and domain.logxxxxx, where xxxxx denotes an integer between 1 and 99999.

Here are some examples of server and domain log files full names: DomainName/servers/AdminServer/logs/AdminServer.log00001, DomainName/servers/AdminServer/logs/domain1.log00033.

Server and domain logs are files where one should look for generic errors, such as exception raised by authenticators or other domain service providers.

The domain logs duplicate some messages written to server logs (for servers in the domain), and they help determine the server where a fault has occurred in a domain that contains a large number of servers.

Note:

The generation of a new log file is determined by its rotation policy; typically, the rotation is determined by file size, so when a log file exceeds a specified size, the system generates a new one with a name whose integer suffix is increased by 1.

For details about particular loggers, see Authorization Loggers and Audit Loggers.

Related Documentation

For information about server log files and domain log files, see section Server Log Files and Domain Log Files in Oracle Fusion Middleware Configuring Log Files and Filtering Log Messages for Oracle WebLogic Server.

For information about the Oracle WebLogic Framework, see Oracle Fusion Middleware Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.

For additional information about logging services, see Oracle Fusion Middleware Using Logging Services for Application Logging for Oracle WebLogic Server.

For complete details about logging in Oracle Fusion Middleware, see chapter Managing Log Files and Diagnostic Data in Oracle Fusion Middleware Administrator's Guide.

L.1.1.3 Authorization Loggers

OPSS provides two loggers that help troubleshooting runtime authorization failures:

These two loggers, as any other OPSS logger, can be enabled and disabled dynamically, that is, without having to stop and restart the Oracle WebLogic Application Server; for details about setting the properties of a logger, see Managing Loggers with Fusion Middleware Control. The level of the above two loggers must be set to TRACE:32.

For information about additional loggers, see Other OPSS Loggers.

L.1.1.3.1 oracle.security.jps.util.JpsAuth

The logger oracle.security.jps.util.JpsAuth logs the start and return of the method checkPermission; the following snippets of a log file illustrate the entry and exit demarcations to this method in the log file:

[SRC_CLASS: oracle.security.jps.util.JpsAuth] [APP: JeeScenarioApp] 
[SRC_METHOD: Entering checkPermission] ENTRY
(oracle.security.jps.ResourcePermission
resourceType=TaskFlowResourceType,resourceName=ResourceNameX read)
 
[SRC_CLASS: oracle.security.jps.util.JpsAuth] [APP: JeeScenarioApp] 
[SRC_METHOD: Exiting checkPermission] RETURN 
java.security.AccessControlException: access denied (oracle.security.jps.ResourcePermission
resourceType=TaskFlowResourceType,resourceName=ResourceNameX read)

The following snippet illustrates a successful authorization log:

[JpsAuth] Check Permission
PolicyContext: [JeeScenarioApp]
Resource/Target: [getSubjectFromDomainCombiner]
Action:[null]
Permission Class: [javax.security.auth.AuthPermission]
       Result:               [SUCCEEDED]
       Subject:              [null]
       Evaluator:            [SM]

The following snippet illustrates an unsuccessful authorization log:

[JpsAuth] Check Permission
PolicyContext: [JeeScenarioApp]
Resource/Target: [resourceType=TaskFlowResourceType,resourceName=ResourceNameX]
Action:[read]
Permission Class: [oracle.security.jps.ResourcePermission]
       Result:               [FAILED]
       Evaluator:            [ACC]
       Failed
L.1.1.3.2 oracle.security.jps.trace.logger

The logger oracle.security.jps.trace.logger logs information about application roles, permissions, targets, principals, and granted and denied policies. Since enabling this logger can lead to a large output, it is recommended that it be used to debug a single use-case only. Specifically, this logger records:

  • The following information about an authorization request: the application roles granted to an enterprise role, all deny's and grant's, the permission class names, the permission targets, and the principal names.

  • Member cache updates, such as when a principal is added to a role; keywords: "Principal:", "Inserted Roles:".

  • Role managing information; keywords: "In App:", "Query Store for Principal:", "Number of direct app roles:".

  • Calls to the method getPermissions.

L.1.1.4 Other OPSS Loggers

In addition to authorization loggers, OPSS provides the following loggers:

oracle.jps.common enables diagnosing issues with the OPSS JpsFilter and the OPSS JpsInterceptor.

oracle.jps.deployment enables diagnosing issues with OPSS artifacts packed with the application, when the application is deployed; keyword:"migration".

oracle.jps.openaz enables diagnosing issues with PEP API calls.

L.1.1.5 Audit Loggers

There are several run-time components in the Fusion Middleware Audit Framework. This section helps you navigate the diagnostic log files for these components and explains how to interpret diagnostic messages.

The log files are located at:

DomainName/servers/$SERVER_NAME/logs/$SERVER_NAME-diagnostic.log

Table L-1 lists the various diagnostic log files.

Table L-1 Log Files for Audit Diagnostics

Component Log Location Configuring Loggers

Java EE Components using Audit APIs

DomainName/servers/$SERVER_NAME/logs/$SERVER_NAME-diagnostic.log

oracle.security.audit.logger (See instructions below)

OPMN Component Using Audit APIs

See the Administration Guide for the component to locate its log files.

Loggers are based on the OPMN Components's Location. Please see the corresponding component guide.

Startup Class Audit Loader

DomainName/servers/$SERVER_NAME/logs/$SERVER_NAME-diagnostic.log

oracle.security.audit.logger (See instructions following this table)

OPMN Audit Loader

$ORACLE_INSTANCE/diagnostics/logs/OPMN/opmn/rmd.out

java.util.logging.config.file system property can be set to the file that contains the log level for OPMN Audit Loader

Config/Proxy Mbeans

DomainName/servers/$SERVER_NAME/logs/$SERVER_NAME-diagnostic.log

oracle.security.audit.logger (See instructions below)

Audit Schema Support

RCU log location (Default is $ORACLE_HOME/rcu/log/)RCU_LOG_LOCATION can be set to change this location

RCU log level (Default is ERROR) RCU_LOG_LEVEL - [SEVERE; ERROR; NOTIFICATION; TRACE


L.1.1.5.1 Configuring the Audit Loggers

You can configure oracle.security.audit.logger using Fusion Middleware Control.

oracle.security.audit.logger can take any log level from ERROR to TRACE allowing control over the amount of information that gets logged.

You can also view these diagnostic files with Fusion Middleware Control.

See Also:

For more information about the following topics, see chapter 10, Managing Log Files and Diagnostic Data, in Oracle Fusion Middleware Administrator's Guide:
  • instructions for configuring the loggers

  • details on viewing logs from domain, server, and each application

L.1.1.5.2 Interpreting Audit Diagnostics

The Audit diagnostic messages can be categorized into two types - errors and trace messages.

All error messages are numbered IAU-XXX. These messages are found in the Error Message Guide with a proper cause and an action that can be taken to rectify the error.

The trace messages, however, are meant to provide more information about the running components. Depending on its nature, a message may require some action on your part.

L.1.1.6 Managing Loggers with Fusion Middleware Control

Fusion Middleware Control provides several pages to manage log information. Using this tool you can:

  • Configure several attributes of a log file, including the log level and rotation.

  • Search the contents of all log files in a domain and group the results of a query by message ID or type.

  • Correlate a given error with others by context or time span.

  • Download a portion of a log file or the results of a query in one of several formats.

This section explains briefly how to configure a log file. The other three functions above are explained, also briefly, in section Section L.1.3, "Solving Security Errors."

For full details about these topics, see section Managing Log Files and Diagnostic Data, in the Oracle Fusion Middleware Administrator's Guide.

To configure a log file with Fusion Middleware Control, proceed as follows:

  1. Navigate to Server > Logs > Log Configuration, to display the Log Configuration page for the selected server. This page allows you to configure the log level for both persistent loggers and active run-time loggers.

  2. Click the Log File entry for the desired logger, to display the page showing the current parameter settings for that file.

  3. In this page, select a row and then click the button Edit Configuration, to display the Edit Log File dialog, where you can set various parameters, including the log level and the rotation policy; typically, the logger level is set to TRACE:32.

L.1.2 System Properties

To increase the debug output, set one the following system properties to the script that starts your Oracle WebLogic Server and restart the server:

To get debug output during the authorization process, set any of the system properties described in section Debugging the Authorization Process.

Two other system properties that can be passed at server start and that can help debugging security issues are the following:

  • -DDebugOPSSPolicyLoading, a flag that monitors the progress and setting of the OPSS policy provider.

  • -Djava.security.debug=policy, the standard Java security debug flag that produces print information about policy files as they are parsed, including their location in the file system, the permissions they grant, and the certificates they use for signed code.

Note:

A consequence of setting a high logging output is that many threads may be reported in a stuck state, specially when file loading takes place. To avoid this situation, change the time out value that Oracle WebLogic Server uses to mark a thread as stuck to a higher value.

A system property cannot be set without restarting the server. In order to set a system property the administrator must edit the setDomainEnv.sh shell script and add the property to the environment variable EXTRA_JAVA_PROPERTIES in that script.

L.1.2.1 jps.auth.debug

Assume that just this system property is set to true:

-Djps.auth.debug=true 

Then, a permission check that fails generates an output with details illustrated in the following sample:

[JpsAuth] Check Permission
          PolicyContext:        [jps-wls-Demo]
          Resource/Target:      [app.monitor]
          Action:               [read,write]
          Permission Class:     [java.util.PropertyPermission]
          Evaluator:            [ACC]
          Result:               [FAILED] 
Failed ProtectionDomain:ClassLoader=weblogic.servlet.jsp.JspClassLoader@fb111c finder: weblogic.utils.classloaders.CodeGenClassFinder@106bb21 annotation:
CodeSource=file:/C:/MyOracle/domains/base_domain/servers/AdminServer/tmp/_WL_user/jps-wls-Demo/kebqfo/jsp_servlet/test.class
Principals=total 5 of principals(
 1. weblogic.security.principal.WLSUserImpl "duane"
 2. weblogic.security.principal.WLSGroupImpl "employee"
 3. JpsPrincipal: oracle.security.jps.principals.JpsAuthenticatedRoleImpl "authenticated-role"
 4. JpsPrincipal: oracle.security.jps.principals.JpsAnonymousRoleImpl "anonymous-role"
 5. JpsPrincipal: oracle.security.jps.service.policystore.ApplicationRole "appRoleEmployee")
 Permissions=(
 (java.util.PropertyPermission line.separator read)
 ...
 (oracle.security.jps.service.credstore.CredentialAccessPermission context=SYSTEM,mapName=default,keyName=* read,write))

A permission check that succeeds generates no output. To disable permission check messages, set this property to false; by default, it is set to true. Disabling persmission check messages is not recommended in production environments.

L.1.2.2 jps.auth.debug.verbose

Assume that jps.auth.debug and jps.auth.debug.verbose are both set to true:

-Djps.auth.debug=true 
-Djps.auth.debug.verbose=true

Then, a permission check that succeeds generates an output with details illustrated in the following sample:

[JpsAuth] Check Permission
          PolicyContext:        [jps-wls-Demo]
          Resource/Target:      [app.monitor]
          Action:               [read,write]
          Permission Class:     [java.util.PropertyPermission]
          Result:               [SUCCEEDED]
          Subject:              [total 5 of principals(
 1. weblogic.security.principal.WLSGroupImpl "manager"
 2. weblogic.security.principal.WLSUserImpl "shawn"
 3. JpsPrincipal: oracle.security.jps.internal.core.principals.JpsAuthenticatedRoleImpl "authenticated-role" GUID=null DN=null
 4. JpsPrincipal: oracle.security.jps.internal.core.principals.JpsAnonymousRoleImpl "anonymous-role" GUID=null DN=null
 5. JpsPrincipal: oracle.security.jps.service.policystore.ApplicationRole "appRoleManager" GUID=null DN=null)]
          Evaluator:            [ACC] 

A permission check that fails generates an output with details illustrated in the following sample:

JpsAuth] Check Permission
          PolicyContext:        [jps-wls-Demo]
          Resource/Target:      [app.monitor]
          Action:               [read,write]
          Permission Class:     [java.util.PropertyPermission]
          Evaluator:            [ACC]
          Result:               [FAILED]
          Failed ProtectionDomain:ClassLoader=weblogic.servlet.jsp.JspClassLoader@1b7682d finder: weblogic.utils.classloaders.CodeGenClassFinder@7d32cf annotation:
CodeSource=file:/C:/Mydom/domains/domain/servers/AdminServer/jspservlet/test.class
 Principals=total 5 of principals(
 1. weblogic.security.principal.WLSUserImpl "duane"
 2. weblogic.security.principal.WLSGroupImpl "employee"
 3. JpsPrincipal: oracle.security.principals.JpsAuthenticatedRoleImpl "authenticated-role" GUID=null DN=null
 4. JpsPrincipal: oracle.security.principals.JpsAnonymousRoleImpl "anonymous-role" GUID=null DN=null
 5. JpsPrincipal: oracle.security.jps.service.policystore.ApplicationRole "appRoleEmployee" GUID=null DN=null)
 Permissions=(
 (java.util.PropertyPermission line.separator read)
 ... 
 (java.lang.RuntimePermission stopThread))
 Call Stack: java.security.AccessControlException: access denied (java.util.PropertyPermission app.monitor read,write)
 java.security.AccessControlContext.checkPermission(AccessControlContext.java:323)
 ...
 weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
          ProtectionDomains for class stack:
 Class[0]: class oracle.security.jps.util.JpsAuth$Diagnostic$SMSupport
 ProtectionDomain: ClassLoader=sun.misc.Launcher$AppClassLoader@360be0
 CodeSource=file:/C:/MyOracle/jdeveloper/modules/oracle.jps_11.1.1/jps-api.jar
 Principals=total 0 of principals<no principals>
 Permissions=(
 (java.io.FilePermission \C:\MyOracle\jdeveloper\modules\jps-api.jar read)
 ...
 )
 Class[1]: class oracle.security.jps.util.JpsAuth$Diagnostic$SMSupport 

To disable permission check messages, set both jps.auth.debug and jps.auth.debug.verbose to false; by default, jps.auth.debug.vebose is set to false.

L.1.2.3 Debugging the Authorization Process

This section describes the use of several other system properties that help debugging the authorization process based on several criteria. Specifically, the following system properties:

oracle.security.jps.log.for.approle.substring
oracle.security.jps.log.for.permeffect
oracle.security.jps.log.for.permclassname
oracle.security.jps.log.for.permtarget.substring
oracle.security.jps.log.for.enterprise.principalname

generate logging messages during the following authorization phases:

  • Phase 1 - The application roles that were granted to an enterprise user or to an enterprise role during the OPSS Subject computation.

  • Phase 2 - The permission instances that were granted to a grantee.

  • Phase 3 - The outcome of a permission check, that is, whether the grant was granted or denied.

Each of the above properties and the phases they apply are described next.

oracle.security.jps.log.for.approle.substring - During phases 1, 2, and 3, it logs the name of an application role that contains a specified substring; if the substring to match is unspecified, it logs all application role names.

oracle.security.jps.log.for.permeffect - During phase 3 and according to a specified value, it logs a grant that was granted or denied; if the value is unspecified, it logs all grants (regardless whether they were granted or denied).

oracle.security.jps.log.for.permclassname - During phases 2 and 3, it logs the name of the permission class that matches exactly a specified name; if the name to match is unspecified, it logs all permission class names.

oracle.security.jps.log.for.permtarget.substring - During phases 2 and 3, it logs the name of a permission target that contains a specified substring; if the substring to match is unspecified, it logs all permission targets.

oracle.security.jps.log.for.enterprise.principalname - During phases 1, 2, and 3, it logs the name of the principal (enterprise user or enterprise role) that matches exactly a specified name; if the name to match is unspecified, it logs all principal names.

The following characteristics apply to all of the above system properties:

  • They are optional.

  • They can be set at most once.

  • The matchings (where they apply) are case insensetive

To enable the logging of any of the above system properties, proceed as follows:

  1. Set the desired system properties.

  2. Stop the JVM.

  3. Restart the JVM.

  4. Set the logger oracle.security.jps.dbg.logger to TRACE:32. For details on how to set a logger, see Managing Loggers with Fusion Middleware Control.

  5. Run the scenario to be debugged.

  6. Examine the log output; to locate the messages output by the settings of any of the above properties, search the log file for the key word [oracle.security.jps.dbg.logger].

L.1.2.3.1 Examples of Use

The following examples illustrate typical settings of the above system properties.

  • To log all application role names that contain the substring myAppRole, include the following setting:

    -Doracle.security.jps.log.for.approle.substring=myAppRole
    
  • To log all denied permission checks, include the following setting:

    -Doracle.security.jps.log.for.permeffect=deny
    
  • To log all granted permission checks, include the following setting:

    -Doracle.security.jps.log.for.permeffect=grant
    
  • To log all granted or denied permission checks, do not set oracle.security.jps.log.for.permeffect.

  • To log all permission checks that match exactly the class name java.util.PropertyPermission, include the following setting:

    -Doracle.security.jps.log.for.permclassname=java.util.PropertyPermission
    
  • To log all target names that contain the substring p.mon, include the following setting:

    -Doracle.security.jps.log.for.permtarget.substring=p.mon
    
  • To log all authorizations involving the principal name manager, include the following setting:

    -Doracle.security.jps.log.for.enterprise.principalname=manager
    
  • To log application role names that match a substring or principal names that match a string, set both oracle.security.jps.log.for.approle.substring and oracle.security.jps.log.for.enterprise.principalname as indicated above.

  • To log all application roles names and all principal names, set neither oracle.security.jps.log.for.approle.substring nor oracle.security.jps.log.for.enterprise.principalname.

L.1.3 Solving Security Errors

There is no generic way to resolve errors when they occur. One must search for hints and frequently follow multiple hypotheses until, hopefully, the source of the error is isolated and understood. To this end, this section describes how to search and interpret log information to resolve most common security errors. These topics are addressed in the following sections:

L.1.3.1 Understanding Sample Log Entries

Understanding log error output is crucial to isolate and solve an error. Let's take a closer look at a diagnostic log file to describe the information you find for an error logged in such a file. This description is best illustrated with a real-life example.

The following is an excerpt of an error in the file AdminServer-diagnostic.log:

[2009-01-07T09:15:02.393-08:00] [AdminServer] [ERROR] [JPS-00004] [oracle.jps.admin] 
[tid: [ACTIVE].ExecuteThread: '3' for queue: 'weblogic.kernel.Default
(self-tuning)'] [userId: weblogic] [ecid: 0000Hum5kxw7MAn54nU4Ui19PD8S000005,0]
Unable to add principal to the application role. Reason: Principal
"abc.xxx@myComp.com" is already a member of the application role
"BPMWorkflowAdmin"[[
java.security.PrivilegedActionException:
oracle.security.jps.service.policystore.PolicyObjectAlreadyExistsException:
Unable to add principal to the application role. Reason: Principal "abc.xxx@myComp.com" is already a member of the application role
"BPMWorkflowAdmin"
        at java.security.AccessController.doPrivileged(Native Method)
        at oracle.security.jps.mas.mgmt.jmx.policy.JpsApplicationPolicyStoreImpl.
addRemoveMembersToRole(JpsApplicationPolicyStoreImpl.java:408)
        at oracle.security.jps.mas.mgmt.jmx.policy.JpsApplicationPolicyStoreImpl.
addMembersToApplicationRole(JpsApplicationPolicyStoreImpl.java:385)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
…

The meaning of the fields in the preceding message is as follows:

  • [2009-01-07T09:15:02.393-08:00]

    Identifies the date and time when the error was logged.

  • [AdminServer]

    Identifies the name of the server where the error occurred.

  • [JPS-00004]

    Identifies the error code and hints to the kind of error that occurred. For a complete list of JPS error codes, see chapter 41 in Oracle Fusion Middleware Error Messages Reference.

  • [oracle.jps.admin]

    Identifies the category of the logger. The subcategories of oracle.jps (such as admin above) hint to the kind of error that occurred. For the complete list of categories under oracle.jps, see Subcategories of oracle.jps.

  • [tid: [ACTIVE].ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)']

    Identifies the thread where the error occurred.

  • [userId: weblogic]

    Identifies the user that performed the operation that generated the error.

  • [ecid: 0000Hum5kxw7MAn54nU4Ui19PD8S000005,0]

    Identifies the execution context id. Typically used to correlate and trace sequence of events. Ecids provide information about the flow across processes, such as, from a request, to the WebLogic server, to an Oracle Internet Directory server.

  • Unable to add principal to the application role. Reason: Principal abc.xxx@myComp.com is already a member of the application role BPMWorkflowAdmin

    Identifies the reason why the error was logged.

  • java.security.PrivilegedActionException: oracle.security.jps.service.policystore.PolicyObjectAlreadyExistsException: Unable to add principal to the application role. Reason: Principal abc.xxx@myComp.com is already a member of the application role BPMWorkflowAdmin

    Identifies the exception that was raised and the reason for it.

Subcategories of oracle.jps

Here is the list of subcategories under oracle.jps and the kind of errors logged in the category:

  • common - generic errors.

  • config - configuration errors.

  • deployment - deployment errors.

  • authentication - login module errors in Java SE applications only.

  • idmgmt - identity store errors.

  • credstore - credential store errors.

  • authorization - policy store errors at run time.

  • policymgmt - policy store management errors.

  • admin - JMX and WLST errors.

L.1.3.2 Searching Logs with Fusion Middleware Control

To initiate a search in the contents of all log files in a domain, select Domain > Logs > View Log Messages, to display the Log Messages page.

In this page you have several parameters that you can choose from to specify your search query; specifically, you can:

  • Choose a time interval in which a message was issue, by selecting the appropriate Date Range.

  • Display messages with a given severity error, by checking any of the Message Types boxes.

  • Display messages satisfying further constrains, by choosing an item from the menu Message and entering a string in the box next to it. For example, you could query for just messages that contain the string exception in it.

  • Add extra query fields, by clicking the button Add Fields and checking any of the available choices. For example, you could add the field Host, and then enter the appropriate query, such as starts with a particular string.

Once these parameters are set, click Search and the result of the query is displayed in the page. The result of a query can be further redisplayed by message type, message ID, or simple list of messages, by selecting an item from the menu Show. Moreover, the result can be automatically refreshed by choosing an item from the menu at the top right of the page (by default set to Manual Refresh).

To broaden a search to log files beyond a domain, use the button Broaden Target Scope at the top right of the page.

L.1.3.3 Identifying a Message Context with Fusion Middleware Control

In some situations, it is necessary to know the context in which a message has occurred. For example, it may be useful to know messages that have preceded or followed a given error message by, say, 2 minutes.

The tab View Related Messages provides this functionality, and you can use it as follows:

  1. Display the results of a query with Show set to Messages.

  2. Select a message within the result table. Note that the tabs View Related Messages and Export Messages to File become then available. Let's assume, for example, that the selected message has the time stamp Jan 21, 2009 4:05:00 PM PST.

  3. Select Time Interval from the Date Range menu, and enter a Start Date and an End Date. For example, you could enter Jan 21, 2009 4:02:00 PM, as a start date, and Jan 21, 2009 4:07:00 PM.

  4. Select by Time from the menu View Related Messages, to display the page with all the messages related to the selected one in the specified time span.

  5. In the Related Messages by Time page, you can modify the time window around the time of the selected message by choosing an item from the menu Scope, at the right of the page.

L.1.3.4 Generating Error Listing Files with Fusion Middleware Control

In some situations, you may want to download the list of errors displayed into a separate file to forward it, for example, to a support center, or just to keep it for your records.

Whenever available, the tab Export Messages allows you to generate a file containing just the displayed results by choosing an item from the menu. The format of the generated file can be plain text, XML, or CSV.

The following sample, showing only the first of 29 messages, is an excerpt of a text file generated this way:

#
#Search Criteria
#        Start Time: 2009-01-21T16:34:41.381-08:00
#        End Time:  2009-01-21T16:39:41.381-08:00
#        Message Types: ERROR, WARNING
 
#Selected Targets List
#        /Farm_base_domain/base_domain/AdminServer:Oracle WebLogic Server
#        /Farm_base_domain/base_domain/AdminServer/DMS Application(11.1.1.1.0):Application Deployment
#        /Farm_base_domain/base_domain/AdminServer/em:Application Deployment
#        /Farm_base_domain/base_domain/AdminServer/wsil-wls:Application Deployment
#        /Farm_base_domain/base_domain/AdminServer/wsm-pm:Application Deployment
#
[2009-01-21T16:34:54.045-08:00] [AdminServer] [WARNING] [] [org.apache.myfaces.trinidad.bean.PropertyKey] [host: stacz39] [nwaddr: 140.87.5.40] [tid: 13] [userId: <anonymous>] [ecid: 0000HvvkgjVE^MT6uBj8EH19TvXj000008,0] [APP: em] [Target: /Farm_base_domain/base_domain/AdminServer/em] [Target Type: Application Deployment] Unserializable value:oracle.sysman.core.view.tgtctls.common.DefaultTreeModel@1fcadd2 for key:UINodePropertyKey[value,17]
…
#
#Number of messages exported: 29
#

L.2 Reassociation Failure

Policy and credential reassociation from an file-based store to an LDAP-based store may fail for several reasons. This section explains three reasons why this operation may fail.

Symptom 1- Error Code 32

Reassociation fails and an error like the following is logged in the administration server diagnostic file serverName.diagnostic.log:

[LDAP: error code 32 - No Such Object]
Authentication to LDAP server ldap://myServer.com:3060 is unsuccessful. 

Diagnosis 1

The error above identifies a problem with the target node in the LDAP server, namely, that the node specified does not exist.

It is required that the root node specified in the text box JPS Root DN (of the page Set Security Provider) be present in the LDAP directory before invoking the reassociation.

Solution 1

Verify that the data you enter in the box JPS Root DN matches the name of a node in the target LDAP directory, and then rerun the reassociation.

Symptom 2- Error Code 68

Reassociation fails and an error like the following is logged in the administration server diagnostic file serverName.diagnostic.log:

Authentication to LDAP server ldap://myServer.com:3060 is successful.
Starting to migrate policy store...
Set up security provider reassociation successfully.
Checked and seeded security store schema successfully.
null
[LDAP: error code 68 - Object already exists]:cn=SystemPolicy,cn=domain1,cn=JPSContext,cn=nb_policy
Error occurred while migrating LDAP based policy store. 

Diagnosis 2

The error above indicates that the name specified in the box WebLogic Domain Name is a descendant (more precisely, a grandchild) of the JPS Root DN node in the target LDAP directory.

It is required that the domain specified do not be a descendant of the root node.

Solution 2

Verify that the name you enter in the box WebLogic Domain Name does not match the name of a grandchild of the specified JPS Root DN node, and rerun the reassociation.

Symptom 3

Reassociation, carried out with Fusion Middleware Control, fails and an error like the following is logged in the administration server diagnostic file serverName.diagnostic.log:

[2009-01-21T10:09:24.326-08:00] [AdminServer] [ERROR] [] [oracle.jps.admin] [tid
: [ACTIVE].ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)
'] [userId: weblogic] [ecid: 0000HvuOTpe7q2T6uBADUH19Tpyb000006,0] Unable to rem
ove the principal from the application role. Reason: Principal "Managers" is not
a member of the application role "test-role"[[
java.security.PrivilegedActionException: oracle.security.jps.service.policystore
.PolicyObjectNotFoundException: Unable to remove the principal from the applicat
ion role. Reason: Principal "Managers" is not a member of the application role "
test-role"
       at oracle.security.jps.mas.mgmt.jmx.policy.JpsApplicationPolicyStoreImpl
.addRemoveMembersToRole(JpsApplicationPolicyStoreImpl.java:408)... 

Diagnosis 3

The error above points to some problem with the application role test-role, which is, in this case, the root of the problem.

Ensure that when entering data to perform reassociation with Fusion Middleware Control, you use the button Test LDAP Authentication immediately after you have completed entering all required values to connect to the target LDAP server. This test catches any problems with those values before reassociation begins.

Solution 3

In our example, a quick inspection of the file system-jazn-data.xml reveals that the application test-role is used by an application policy, but it was not defined. Here is an excerpt of that file illustrating where the required data is missing:

<application>
    <name>myApp</name>
        <app-roles>
 <--! test-role should have been defined here -->
        </app-roles>
        <jazn-policy>
            <grant>
                <grantee>
                    <principals>
                       <principal>
                         <class>
oracle.security.jps.service.policystore.ApplicationRole</class>
                         <name>test-role</name>
                         <guid>66368900E7E511DD9F62F9ADA4233FE2</guid>
                       </principal>
                     </principals>...

To solve this particular error, (a) fix system-jazn-data.xml by inserting the definition of the application test-role; (b) revert to file-based domain stores with the fixed file; and (c) rerun the reassociation.

L.2.1 Missing Policies in Reassociated Policy Store

Symptom

When an file-based policy store is reassociated to use an LDAP-based Oracle Internet Directory policy store, the reassociation may report that it completed successfully.

At runtime, however, the system does not behave as expected. Granted code-based policies, that are supposed to be present in the system policy after migration, are missing.

Diagnosis

At runtime, the server reports a stack trace that resembles the following:

<BEA-000000> <JspServlet: initialization complete>
####<May 4, 2009 8:32:50 AM PDT> <Error> <HTTP> <ap626atg> <WLS_Spaces>
<[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default
(self-tuning)'> <<WLS Kernel>> <> <> <1241451170341> <BEA-101020>
<[ServletContext@20193148[app:webcenter module:/webcenter path:/webcenter
spec-version:2.5]] Servlet failed with Exception
java.security.AccessControlException: access denied
(oracle.security.jps.service.policystore.PolicyStoreAccessPermission
context=APPLICATION,name=webcenter getApplicationPolicy)
        at
java.security.AccessControlContext.checkPermission(AccessControlContext.java:323)
        at
java.security.AccessController.checkPermission(AccessController.java:546)
        at
oracle.security.jps.util.JpsAuth$AuthorizationMechanism$3.checkPermission(JpsAuth.java:348)
        at
oracle.security.jps.util.JpsAuth$Diagnostic.checkPermission(JpsAuth.java:268)
        at
oracle.security.jps.util.JpsAuth$AuthorizationMechanism$6.checkPermission(JpsAuth.java:372)
        at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:408)
        at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:431)
        at
oracle.security.jps.internal.policystore.AbstractPolicyStore.checkPolicyStoreAccessPermission(AbstractPolicyStore.java:246)
        at
oracle.security.jps.internal.policystore.ldap.LdapPolicyStore.getApplicationPolicy(LdapPolicyStore.java:281)
        at
oracle.security.jps.internal.policystore.PolicyUtil.getGrantedAppRoles(PolicyUtil.java:898)
        at
oracle.security.jps.internal.policystore.PolicyUtil.getJpsAppRoles(PolicyUtil.java:1354)
        at
oracle.security.jps.wls.JpsWlsSubjectResolver$1.run(JpsWlsSubjectResolver.java:273)
        at
oracle.security.jps.wls.JpsWlsSubjectResolver$1.run(JpsWlsSubjectResolver.java:270)
        at java.security.AccessController.doPrivileged(Native Method)

Here the permission:

oracle.security.jps.service.policystore.PolicyStoreAccessPermission
context=APPLICATION,name=webcenter getApplicationPolicy

is granted to a code base, and the authorization is not allowed since it evaluates to false.

Solution

Check the AdminServer diagnostic logs for messages like these:

AdminServer-diagnostic.log:[2009-05-28T02:27:52.249-07:00] [AdminServer] [NOTIFICATION] [JPS-00072] [oracle.jps.config] [tid: Thread-39] [ecid: 0000I66Z0KH0fplp4sm3Ui1A7_Rl00002s,1:5001] [arg: 11.1.1.1.0] [arg: 11.1.1.0.0] Policy schema upgrade not required. Store Schema version 11.1.1.1.0 is compatible to the seed schema version 11.1.1.0.0
AdminServer-diagnostic.log:[2009-05-28T02:28:58.012-07:00] [AdminServer] [NOTIFICATION] [JPS-00078] [oracle.jps.config] [tid: Thread-39] [ecid: 0000I66Z0KH0fplp4sm3Ui1A7_Rl00002s,1:5001] [arg: 11.1.1.1.0] [arg: 11.1.1.0.0] Credential store schema upgrade not required. Store Schema version 11.1.1.1.0 is compatible to the seed schema version 11.1.1.0.0

A message of this type suggests that the schema was never seeded during the re-association. If the correct schema is not seeded in the Oracle Internet Directory server, the system will not work as expected.

To ensure that the schema is seeded during re-association, proceed as follows:

  1. Remove the cn=OPSS container under the cn=OracleSchemaVersion container in the Oracle Internet Directory server.

  2. Start with a clean working instance of an OPSS policy store using the file-based store.

  3. Re-associate this file-based store to the Oracle Internet Directory server.

Check the AdminServer diagnostic logs to confirm that the OPSS LDAP schema was seeded in the LDAP server by looking for this message:

AdminServer-diagnostic.log:[2009-05-29T07:18:18.002-07:00] [AdminServer] [NOTIFICATION] [JPS-00078] [oracle.jps.config] [tid: Thread-12] [ecid: 0000I61Z0MH0fplp4sm3Ui1A7_Ll00002s,1:5001] [arg: 11.1.1.0.0]  Policy schema version set to 11.1.1.0.0

If re-associating to a Release 11g Oracle Internet Directory server, the schema version should read: 11.1.1.1.0

If re-associating to a Release 10.1.4.3 Oracle Internet Directory server, the schema version should read: 11.1.1.0.0

The Policy Store schema version is set in the Oracle Internet Directory server under this container:

cn=PolicyStore,cn=OPSS,cn=OracleSchemaVersion

Similarly, the Credential Store schema version is set in the Oracle Internet Directory server under this container:

cn=CredentialStore,cn=OPSS,cn=OracleSchemaVersion

L.2.2 Unsupported Schema

This section explains a reason why reassociation to an LDAP server may fail.

Symptom

Reassociating the security store to an LDAP repository fails and the AdminServer log reports an error like the following:

[2011-02-09T07:01:13.884-05:00] [AdminServer] [ERROR] [] [oracle.jps.admin] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 41050d66ef2ec40b:-4c1fb689:12e06cc7b6c:-8000-00000000000001e1,0] Schema seeding failed, check the server type of the given ldap url.[[
oracle.security.jps.JpsException: Error Modifying JPS Schema,  Record: dn: cn=schema
changetype: modify
delete: objectclasses
objectclasses: ( 2.16.840.1.113894.7.2.2 NAME 'orclContainer' SUP ( top ) MUS
 T ( cn ) MAY ( orclVersion $ orclServiceType ) )
-
: [LDAP: error code 32 - No Such Object]:cn=schema

Diagnosis

The error LDAP: error code 32 indicates that the schema of the reassociation target LDAP repository is not supported, that is, the version of the target LDAP repository is not one of the OPSS supported LDAP stores.

Solution

Update the target LDAP repository to one of the supported LDAP stores and then try reassociating again. The version of an LDAP OID store must be 10.1.4.3 or later. For a list of supported versions, see Section 8.2, "Using an LDAP-Based OPSS Security Store."

L.3 Server Fails to Start

This section explains several reasons why the Oracle WebLogic Server may fail to start in the following sections:

L.3.1 Missing Required LDAP Authenticator

This section explains a reason why the Oracle WebLogic Server may fail to start after modifying the list of authenticators in a domain.

Symptom

After modifying the list of authenticator providers in a domain, the Oracle WebLogic Server fails to start, and the error messages output include the following:

java.lang.IllegalArgumentException: null KeyStore name

Diagnosis

One cause of this problem is that the list of authenticators in your domain does not include an LDAP authenticator.

Important:

An LDAP authenticator is required in this list for any domain using OPSS.

Solution

Since the server cannot start, you must add one LDAP authenticator manually, as follows:

  1. Open the file DOMAIN_NAME/config/config.xml.

  2. Edit config.xml and include, within the element <realm>, an LDAP authenticator, such as the default authenticator illustrated in the following sample:

    <realm>
     ...
     <sec:authentication-provider xsi:type="wls:default-authenticatorType"> 
     </sec:authentication-provider>
     ...
    </realm>
    
  3. Restart the server.

Once the server is back up and running, you can modify the list of providers to include the provider of your choice using the WebLogic Administration Console, but ensure that at least one of them is an LDAP authenticator provider.

To this end, use the WebLogic Administration Console as follows:

  1. Navigate to the page Create a new Authenticator Provider.

  2. Enter the authenticator name and select an authenticator type, all of which are LDAP-based:

    • ActiveDirectoryAuthenticator

    • DefaultAuthenticator (this is the one inserted manually in the sample above)

    • LDAPAuthenticator

    • LDAPX509IdentityAsserter

    • OpenLDAPAuthenticator

    • OracleInternetDirectoryAuthenticator

    • OracleVirtualDirectoryAuthenticator

L.3.2 Missing Administrator Account

This section explains a reason why the Oracle WebLogic Server may fail to start.

Symptom

After removing the out-of-box default authenticator and adding, say an Oracle Internet Directory authenticator, the server fails to start.

Diagnosis

Most likely, you have forgotten to enter an account member of the Administrators group in your added authenticator. The server requires that such an account be present in one domain authenticator. This account is always present in the default authenticator.

Solution

Since the server cannot start, you must add the deleted one LDAP authenticator manually, as follows:

  1. Open the file DOMAIN_NAME/config/config.xml.

  2. Edit config.xml and include, within the element <realm>, the default authenticator, as illustrated in the following sample:

    <realm>
     ...
     <sec:authentication-provider xsi:type="wls:default-authenticatorType"> 
     </sec:authentication-provider>
     ...
    </realm>
    
  3. Restart the server.

Once the server is back up and running, proceed as follows:

  1. Use the WebLogic Administration Console to create in the Oracle Internet Directory authenticator an account that is member of the Administrators group.

  2. Set the Oracle Internet Directory authenticator flag to SUFFICIENT.

  3. Restart the server, which it should start without problems, since it is using the account in the Administrators group provided in the default authenticator.

  4. Reset the Oracle Internet Directory authenticator flag to REQUIRED and remove the default authenticator. The server should now start using the account in the Administrators group that you created in the Oracle Internet Directory authenticator.

L.3.3 Missing Permission

This section explains a reason why the Oracle WebLogic Server may fail to start.

Symptom

The server fails to start when it started with security manager is enabled (with the system property -Djava.security.manager).

Diagnosis

One reason why you may run into this issue is the lack of permission grants to PKI APIs in oraclepki.jar when the security manager is enabled at server startup.

Solution

Ensure that a grant like the following is present in the file weblogic.policy, or add it if it is not:

grant codeBase "file:${oracle.home}/modules/oracle.pki_${jrf.version}/*" { 
 permission java.security.AllPermission; 
};

The above grant is provided by default. Note that when security manager is enabled, the access to all system resources requires codebase permission grants.

For complete details about using the Java Security Manager to protect WebLogic resources, see Oracle Fusion Middleware Programming Security for Oracle WebLogic Server.

Note:

Printing Security Manager is a WebLogic server enhancement to the Java Security Manager. Use Printing Security Manager to identify all of the required permissions for a Java application running under Java Security Manager. Unlike the Java Security Manager, which identifies needed permissions one at a time, the Printing Security Manager identifies all the needed permissions without intervention.

L.3.4 Other Causes

This section explains several reasons why the Oracle WebLogic Server may fail to start.

Symptom

When attempting to load and set the policy provider, the Oracle WebLogic Server fails to start and logs an exception similar to the one illustrated in the following snippet:

<Mar 30, 2010 3:15:54 PM EDT> <Error> <Security> <BEA-090892> <The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider.
...
<Mar 30, 2010 3:15:54 PM EDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider.
...
weblogic.security.SecurityInitializationException: The dynamic loading of the OPSS java security policy provider class oracle.security.jps.internal.policystore.JavaPolicyProvider failed due to problem inside OPSS java security policy provider. Exception was thrown when loading or setting the JPSS policy provider.
...

Diagnosis

The server startup includes loading and setting the policy provider as defined in the configuration file jps-config.xml; if this task is not completed successfully, the Oracle WebLogic Server fails to start. As illustrated in the sample above, this type of failure is identified in the server's log by the string

Exception was thrown when loading or setting the JPSS policy provider.

To determine the root cause of a particular failure server startup, check the server's log file and inspect the logged stack trace. For details about identifying errors, see Diagnosing Security Errors.

Here are some reasons why the server fails to start:

  1. The path to the configuration file is incorrectly specified.

  2. The default context is missing in the configuration file.

  3. The XML parser is not available.

  4. A code source URL is incorrectly specified in a system policy. This situation is identified by a logged exception that includes the string

    java.net.MalformedURLException: unknown protocol.
    

Solution

A solution for each of the above cases above is explained next.

  1. Ensure that the correct path is specified by the system parameter oracle.security.jps.config:

    -Doracle.security.jps.config=<full-path-to-jps-config.xml>
    

    Note that special characters (such as backlashes or white space characters) in the full path specification must be properly escaped. One way to verify correctness is to test using the specified full path in a command line.

  2. The configuration file must include a default context. For an example of a default context configuration, see <jpsContext>.

  3. Make sure that the XML parser is available in your system and that the XML parser JAR file is included in the classpath.

  4. Typical incorrect and corrected code source URLs are illustrated in the following two samples.

    Sample 1 - Incorrect URL
    <grantee>
      <codesource>
         <url>${my.oracle.home}/jlib/webcacheua.jar</url> 
       </codesource>
    </grantee>
     
    Sample 1 - Corrected URL (in bold)
    <grantee>
      <codesource>
         <url>file:/${my.oracle.home}/jlib/webcacheua.jar</url> 
       </codesource>
    </grantee>
     
    Sample 2 - Incorrect URL
    <grantee>
      <codesource>
         <url>c:/myfolder/jlib/webcacheua.jar</url> 
       </codesource>
    </grantee>
     
    Sample 2 - The corrected URL (in bold) is either one of the following three:
    <grantee>
      <codesource>
         <url>file:///c:/myfolder/jlib/webcacheua.jar</url> 
       </codesource>
    </grantee>
     
    <grantee>
      <codesource>
         <url>file:c:/myfolder/jlib/webcacheua.jar</url> 
       </codesource>
    </grantee>
     
    <grantee>
      <codesource>
         <url>file:/c:/myfolder/jlib/webcacheua.jar</url> 
       </codesource>
    </grantee>
    

    For details about the syntax of URL specifications in a code source (including the use of system variables), see <url>.

L.4 Failure to Grant or Revoke Permissions - Case Mismatch

This section explains the likely reasons why an enterprise user or role (group) may fail to be granted or revoked permissions.

Symptom

An enterprise user or group, properly entered in a domain authenticator, is not granted or revoked the permissions defined by a grant.

Diagnosis

This problem is likely to occur when there is a case mismatch between the stored name (in a domain authenticator) and the supplied name (either actively entered by a user or obtained programmatically). For example, this mismatch would occur when the stored user name is JdOE and the supplied user name is jdoe.

Solution

There are two ways to resolve this issue.

The first solution involves setting the appropriate property in the authenticator being used in your domain. As long as both strings (the supplied and the stored) contain identical sequence of characters (irrespective of case), this setting guarantees that the user name populated in the Subject matches the user name present in a domain authenticator, even when the corresponding characters differ in case. Thus, when this setting is in place, the user names JdOE and jdoe match.

To set your domain authenticator property, proceed as follows:

  1. Use the Administration Console to navigate to the page where your authenticator is configured. For example, if you are using the default authenticator, navigate to the DefaultAuthenticator page by choosing Realms > myrealm > Providers > DefaultAuthenticator.

  2. Choose the tab Provider Specific.

  3. Set the property userRetrievedUserNameAsPrincipal to true.

  4. Restart the server.

The second solution considers the case where the supplied name is obtained programmatically, that is, where one must produce a principal from a user name.

To obtained the correct user or group name, either pass the name exactly as it is stored in the authenticator or use the sequence of calls illustrated in the following code snippet:

import weblogic.security.principal.WLSGroupImpl;
import weblogic.security.principal.WLSUserImpl;
 
// Set the context
JpsContextFactory ctxFact = JpsContextFactory.getContextFactory();
ServerContextFactory scf = (ServerContextFactory) ctxFact;
JpsContext ctx = scf.getContext(ServerContextFactory.Scope.SYSTEM);
ctx = ctxFact.getContext();

// Set the identity store
IdentityStore identityStore = ctx.getServiceInstance(IdentityStoreService.class).getIdmStore();
 
// In case of a user name, search the user that matches the supplied name
User user = idStore.searchUser(IdentityStore.SEARCH_BY_NAME, suppliedUserName);

// Use the obtained object (user) to obtain the stored user name and create 
// the Principal
String storedUserName = user.getName();
Principal userPrincipal = new WLSUserImpl(storedUserName);
 
// Similarily, in case of a role name, search the role that matches 
// the supplied role name
Role role = identityStore.searchRole(IdentityStore.SEARCH_BY_NAME, suppliedRoleName);
 
// Use the obtained object (role) to obtain the stored role name and create 
// the Principal
String storedRoleName = role.getName();
Principal rolePrincipal = new WLSGroupImpl(storedRoleName);

Important:

When creating a user or role principal, you must use the calls:
Principal userPrincipal = new WLSUserImpl(user.getUserProfile()getName());
Principal rolePrincipal = new WLSGroupImpl(role.getRoleProfile().getName());

Instead of the calls:

Principal userPrincipal = new WLSUserImpl(user.getName());
Principal rolePrincipal = new WLSGroupImpl(role.getName());

L.5 Failure to Connect to an LDAP Server

This section explains the likely reasons why a connection to an Oracle Internet Directory LDAP server can fail. This failure can also happen during reassociation.

Symptom

The migration of data from a source repository to a target LDAP server repository fails.

Diagnosis

Typically, this kind of problem is due to an incorrect set up of parameters in the target LDAP server.

For further probing into Oracle WebLogic Server log files, search any of the log files in the directories DomainName/servers/AdminServer or DomainName/servers/ManagedServers for the following strings: <Error>, <Critical>, and <Warning>.

For more information about identifying and solving errors, see Section L.1, "Diagnosing Security Errors."

Solution

Verify that all the target server data provided for the migration is valid. You may require the assistance of your LDAP server administrator to perform this validation.

Note:

If you are using Fusion Middleware Control to reassociate to an LDAP server, ensure that you use the button Test LDAP Authorization before initiating the operation. Typically, this test catches incorrect supplied parameters.

L.6 Failure to Connect to the Embedded LDAP Authenticator

This section explains the likely reasons why a connection to the embedded LDAP authenticator can fail.

Symptom

The connections that client applications use to request queries to the embedded LDAP authenticator, via the User and Role API, are stored and maintained in a connection pool. By default, and out-of-the-box, this pool is the JNDI pool, as specified in the file jps-config.xml.

If the number of current connections in the pool exceeds the maximum allowed by the LDAP service, client applications will not be able to connect to the service or, even when they are already connected, receive a “socket closed” exception. The server log would indicate, in this case, that the number of concurrent connections allowed has been exceeded.

Diagnosis

To avoid going over the limit, one needs to adjust the maximum number of concurrent connections allowed by the LDAP service as appropriate to the application's needs. This threshold needs to be finely tuned up: a too small maximum may not be sufficient (and cause the exception mentioned above); a too large maximum may risk a denial of service (DOS) attack. The correct maximum depends on your application and the particular LDAP service the application uses.

Solution

There are two alternative ways that resolve this issue:

  • Increase the maximum number of concurrent connections allowed by the authenticator:

    • If the authenticator your application is using is the WebLogic Embedded LDAP authenticator, then edit the file DomainName/servers/MyServerName/data/ldap/conf/vde.prop, and increase the value of the property vde.quota.max.conpersubject from the default 100 to, for example, 200, or any other value.

    • Otherwise, if your application is using any other authenticator, consult the authenticator's documentation to learn how to modify the maximum.

  • Edit the file DomainName/config/fmwconfig/jps-config.xml and remove the property CONNECTION_POOL_CLASS from the authenticator server instance (by default, this property has the value oracle.security.idm.providers.stdldap.JNDIPool.

Note that (a) these settings do not exclude each other, that is, you can carry out both of them; and (b) in any case, you must restart the server for the changes to take effect.

L.7 User and Role API Failure

This section explains some reasons why you may fail to access data in a domain authenticator with the User and Role API.

Symptom

The User and Role API fails to access data in a configured authenticator.

Diagnosis 1

The OPSS User and Role API can access data only in the first LDAP authenticator configured in a domain. At least one such authenticator must be present in a domain. The API access to that first LDAP authenticator fails if the target user is not present in that authenticator, even though that user is present in some other domain authenticator.

Solution 1

Enter the missing user in the first LDAP authenticator, or reorder the list of LDAP authenticators in your domain.

Diagnosis 2

Let's assume that the target user on which the API that fails is present in the first LDAP authenticator configured in your domain.

By default, the User and Role API uses the attribute uid to perform user search in an LDAP authenticator. If for some reason, a user entered in the LDAP is lacking this attribute, then the User and Role API fails.

Solution 2

Ensure that all users in the first LDAP authenticator have the attribute uid set.

Note:

If you are developing a Java SE application (and only in this case) and want the User and Role API to employ an attribute other than the default one (uid) to search users, say mail for example, then the properties username.attr and user.login.attr must be configured in the LDAP provider instance of the identity store (in the file jps-config-jse.xml) as illustrated in the following code snippet:
<serviceInstance provider="idstore.ldap.provider" name="idstore.ldap">
   ...
   <property name="username.attr" value="mail"/>
   <property name="user.login.attr" value="mail"/>
   ...
</serviceInstance>

To add properties to a provider instance with a prescribed script, see Section E.1, "Configuring OPSS Service Provider Instances with a WLST Script."

L.8 Failure to Access Data in the Credential Store

This section explains a likely reason why an application fails to access data in the domain's credential store.

Symptom

An application fails to retrieve credential data from the domain's credential store, and an error message (containing lines like the one illustrated below) is logged (text in between brackets should describe information specific to the particular failure):

07/07/26 18:22:22 [JpsAuth] For permisson ( CredentialAccessPermission [target] [actions]), domain that failed: ProtectionDomain
 cs(file:somePath/aWarFile.war/WEB-INF/classes/), []

Diagnosis

If an application is to access the credential store to perform an operation (such as retrieving a user password, for example), then its code must be granted the appropriate permission to perform the secured operation; otherwise, the application runs into an error like the one described above.

Solution

To grant the permission that an application requires to access the credential store, include the appropriate CredentialAccessPermission in the application's jazn-data.xml; this grant takes effect when the application is deployed or redeployed.

To include a permission using Fusion Middleware Control, see Section 9.2, "Managing Policies with Fusion Middleware Control."

To include a permission using an OPSS script, see Section 9.3, "Managing Application Policies with OPSS Scripts."

The following fragment of the file jazn-data.xml illustrates how to grant all code in the application myApp permission to read all credentials in the folder myAlias:

<jazn-data>
    <!--  codebase policy -->
    <jazn-policy>
        <grant>
            <grantee>
                <codesource>
 <!-- This grants applies to all code in the following directory -->
                    <url>${domain.home}/tmp/_WL_user/myApp/-</url>
                </codesource>
            </grantee>
            <permissions>
                <permission>
                    <class>oracle.security.jps.service.credstore.CredentialAccessPermission</class>
<!-- Allow read permission to all credentials under folder MY_MAP -->
                    <name>context=SYSTEM,mapName=MY_MAP,keyName=*</name>
                    <actions>read</actions>
                </permission>
            </permissions>
        </grant>
    </jazn-policy>
</jazn-data>

L.9 Failure to Establish an Anonymous SSL Connection

This section explains the likely reasons why you are not able to establish an anonymous SSL connection while reassociating policies and credentials.

Symptom

A step in the reassociation of file-based policies and credentials to an LDAP-base storage using an Oracle Internet Directory server with Fusion Middleware Control involves testing the anonymous SSL connection to the LDAP server (specifically with the button Test LDAP). This test fails.

Diagnosis

Your target LDAP server must be trusted by the Oracle WebLogic Server and the port number you are using to the LDAP server must be an SSL port.

Solution

Establishing a connection to an LDAP server requires some previous configuration on the LDAP server. For details, see Section 8.2.2, "Prerequisites to Using an LDAP-Based Security Store."

In addition, to use an anonymous SSL connection, you must enter a port that has been set for receiving secure data. If your LDAP server has not been configured with such a port, the connection fails.

Ensure that the supplied LDAP server port is an SSL port configured to listen in anonymous SSL mode, and that the supplied server name reachable. Typically, the setting of this port involves an LDAP server administrator.

L.10 Authorization Check Failure

This section explains a reason why an authorization check has failed.

Symptom

An attempt to authorize a user by your application fails, and the system logs an error containing a line like the following:

Servlet failed with Exception oracle.adf.controller.security.AuthorizationException:ADFC-0619:
Authorization check failed: '/StartHere.jspx' 'VIEW'. 

Diagnosis

One reason that can lead to such an authorization failure is a mismatch between the run-time policy context and the policy store stripe that you application is using.

On the one hand, the application stripe (or subset of policies in the policy store) that an application uses is specified in the file web.xml with the parameter application.name within the filter configuring the JpsFilter (for a servlet) or the interceptor configuring the JpsInterceptor (for an EJB). For details and samples, see Application Name (Stripe). If the application stripe is not specified (or left blank), then the system picks up an application stripe based on the application name.

On the other hand, the run-time policies that your application uses are specified in the file system-jazn-data.xml with the element <application.name>.

If those two names do not match or if you have not explicitly specified the stripe to use, then, most likely, your application is accessing the wrong policy stripe and, therefore, not able to authorized your application users as expected.

Solution

Ensure that you specify explicitly your application stripe, and that stripe is the one that your application is supposed to use. In most cases, the two names specified in those two different files (as explained above) match; however, in cases where several applications share the same policy stripe, they may differ.

L.11 User Gets Unexpected Permissions

This section explains the likely reasons why a user gets permissions other than those anticipated.

Symptom

A new user or a modified user gets unexpected permissions.

Diagnosis

This issue is likely to come up in cases where a user is added with the name of previously removed user, or an old user gets its name or uid changed. The common reason why the user may get more or less permissions than expected is that the policy store has not been properly updated before a user is removed or a user data is changed.

Solution

Before deleting a user, revoke all permissions, application roles, and enterprise groups that had been granted to the user. If you fail to remove all security artifacts referencing a user to be deleted, they are left dangling and, potentially, inherited if another user with the same name or uid is created at a later time.

Similar considerations apply to when a user name or uid is changed: all policies (grants, permissions, roles) referring to the old data must be updated so that they work as expected with the new data.

L.12 Security Access Control Exception

This section explains a reason why your code may run into a security access control exception.

Symptom

At run time, your application outputs an error like the following one (only the first few lines are shown):

<Jan 20, 2009 5:45:33 PM PST> <Error> <HTTP> <BEA-101020>
<[weblogic.servlet.internal.WebAppServletContext@140cf52 
- appName: 'Application2', 
name: 'Application2.war', 
context-path: '/Application2', 
spec-version: '2.5'] 
Servlet failed with Exceptionjava.lang.RuntimeException:java.security.AccessControlException:access denied
...

Diagnosis

The above error means that a call in your code does not have sufficient permissions to execute a secured operation.

Solution

Your code must be granted the appropriate permissions to execute the secured operation. Depending on the scope of the permission you would like to set, you have two alternatives.

The first one is to grant permission to all application code in the application's EAR or WAR files; in this case, the call to the secured operation can be inserted anywhere in the application code.

The second one is to grant permission to just a JAR file; in this case, the call to the secured operation must be inside a privileged block.

Each of these solutions is next illustrated by an application attempting to access the credential store.

The following fragment of an applicationjazn-data.xml illustrates how to set permission to read any key within the map MY_MAP in the credential store to any code within the directory BasicAuth:

<jazn-policy>
   <grant>
       <grantee>
           <codesource>
              <url>file:${domain.home}/servers/_WL_user/BasicAuth/-</url>
           </codesource>
       </grantee>
       <permissions>
           <permission>
             <class>
                 oracle.security.jps.service.credstore.CredentialAccessPermission
             </class>
             <name>context=SYSTEM,mapName=MY_MAP,keyName=*</name>
             <actions>read</actions>
          </permission>
      </permissions>
   </grant>
</jazn-policy>

If the permission is to be granted to the code in a particular EAR or WAR file, the url specification above would have to be changed to one like the following:

<url>file:${domain.home}/servers/_WL_user/jpsBasicAuth/.../BasicAuth.ear</url>

In both above cases, the call to read the credential store can be placed anywhere in the application code.

If, however, the permission is to be granted to just the code in a particular JAR file, the url specification above would have to be changed to one like the following:

<url>file:${domain.home}/servers/_WL_user/jpsBasicAuth/myJars/Foo.jar</url>

In this last case, the code in the file Foo.jar that calls a read operation on the credential store must be placed in an AccessController.doPrivileged block, as illustrated in the following code snippet:

import oracle.security.jps.*;
import oracle.security.jps.service.credstore.*;
 
JpsContextFactory factory = JpsContextFactory.getContextFactory();
JpsContext jpsContext = factory.getContext();
final CredentialStore store = jpsContext.getServiceInstance(CredentialStore.class);
Credential cred = AccessController.doPrivileged(new PrivilegedExceptionAction<PasswordCredential>() {
    public PasswordCredential run() throws JpsException {
        return store.getCredential("MY_MAP", "anyKey");
    }
});
           
PasswordCredential pwdCred = (PasswordCredential)cred;
...

Note that since our sample grant above allows only read permission, none of the set or reset operations work, even inside a privileged block.

L.13 Permission Check Failure

This section explains a reason why a permission may fail to pass a permission check.

Symptom

At run time, your application outputs an error like the following one (only the first few lines are shown):

[JpsAuth] Check Permission
          PolicyContext:        [null]
          Resource/Target:      [test]
          Action:               [null]
          Permission Class:     [com.oracle.permission.SimplePermission]
          Evaluator:            [ACC]
          Result:               [FAILED]
          Failed ProtectionDomain:ClassLoader=weblogic.utils.classloaders.ChangeAwareClassLoader@14061a8 
finder: weblogic.utils.classloaders.CodeGenClassFinder@2dce7a8 
annotation: Application2@Application2.war
CodeSource=file:/scratch/servers/AdminServer/tmp/permission/TestServlet$1.class
Principals=total 0 of principals<no principals>
Permissions=(
(oracle.security.jps.service.credstore.CredentialAccessPermission 
context=SYSTEM,mapName=default,keyName=* read,write)
(java.net.SocketPermission localhost:1024- listen,resolve)
(oracle.security.jps.service.policystore.PolicyStoreAccessPermission
context=APPLICATION,name=* getApplicationPolicy)
(oracle.security.jps.service.policystore.PolicyStoreAccessPermission
context=SYSTEM getConfiguredApplications)
(com.oracle.permission.SimplePermission *)
...
java.security.AccessControlException: access denied
(com.oracle.permission.SimplePermission test)...

Diagnosis

When two or more applications share a permission class, that permission class must be set in the system class path so the class is loaded just once. Otherwise, only the first application loading the class passes the permission check; other ones loading the same class thereafter may fail the permission check and output an error like the one illustrated above.

Note that even though the permission class is in the permission collection (see bold text in sample output above), the check fails and the access is denied. This is because, at that point, the environment contains several instances of a permission class with the same name.

Solution

Ensure that if two or more applications to be run in the same domain share a permission class, then include that class in the system class path.

L.14 Policy Migration Failure

This section describes a reason why the automatic migration of policies at application deployment may fail. Note that the deployment of an application may succeed even though the migration of policies failed.

Note:

The reason why the automatic migration can fail, as explained in this section, can also lead to similar failures when reassociating domain stores.

Symptom

The application is configured to migrate policies automatically at deployment. The application deployment succeeds, but the diagnostic file corresponding to the server where it has been deployed outputs a message like the following:

[2009-01-21T13:34:48.144-08:00] [server_soa] [NOTIFICATION] []
[oracle.jps.deployment] [tid: [ACTIVE].ExecuteThread: '2' for queue:
'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] 
[ecid: 0000Hvv7U_H7q2T6uBADUH19Tq0B00002I,0] [APP: JpsJdev#V2.0] 
Application [JpsJdev#V2.0] is being deployed, start policy migration.

[2009-01-21T13:34:48.770-08:00] [server_soa] [WARNING] [] 
[oracle.jps.deployment] [tid: [ACTIVE].ExecuteThread: '2' for queue:
'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] 
[ecid: 0000Hvv7U_H7q2T6uBADUH19Tq0B00002I,0] [APP: JpsJdev#V2.0] 
Exception in application policy migration.[[
oracle.security.jps.JpsException: appplication Role: 
test_role not found for the application in the destination policy store
at oracle.utility.destination.apibased.JpsDstPolicy.convertAppPolicyPrincipal
(JpsDstPolicy.java:815)
at oracle.utility.destination.apibased.JpsDstPolicy.clone
(JpsDstPolicy.java:691)...

The above excerpt was extracted from the file server_soa-diagnostic.log, and the application JpsJdev was deployed to the managed server server_soa. Note that the key phrase to look for to locate such error is highlighted in the sample above. In addition, the error describes the artifact that raised the exception, the application role test_role.

Diagnosis

Something is wrong with the definition of this role in the application file jazn-data.xml. In fact, a quick look at this file reveals that the role test_role is referenced in a grantee, as illustrated in the following excerpt:

<grantee>
   <display-name>myPolicy</display-name>
   <principals>
     <principal>
       <class>oracle.security.jps.service.policystore.ApplicationRole</class>
       <name>test_role</name>
     </principal>
   </principals>
</grantee> ...

But the name of what is supposed to be the application role named test_role, however, was inadvertently misspelled to test_rolle:

<application>
   <name>JpsJdev</name>
   <app-roles>
     <app-role>
       <name>test_rolle</name>
       <class>oracle.security.jps.service.policystore.ApplicationRole</class>
        <members> ...

Solution

Ensure that all application roles referenced in application policies have been properly defined in the jazn-data.xml file. If a referenced role name cannot be matched, as in the samples above, the migration fails.

L.15 Characters in Policies

This section explains several issues related to characters used in policies, in the following sections:

L.15.1 Use of Special Characters in Oracle Internet Directory 10.1.4.3

When the policy store is an LDAP-based Oracle Internet Directory 10.1.4.3 repository, then using the characters '*', '(', ')', or '\' in the RFC 2252/2253 filter results in error 53 (DSA unwilling to perform). To resolve this error, apply the patch for bug number 7711351 to Oracle Internet Directory 10.1.4.3.

L.15.2 XML Policy Store that Contains Certain Characters

The issue explained in this section is relevant to XML Policy Stores only, that is, it does not apply to LDAP-based Policy Stores.

The following characters:

 < "  &  $ ? * , / \ ` :    ( ) ^ ' % +  { } 

are not recommended as part of an Application Role name when using an file-based policy store.

If it becomes necessary to use one of those characters to create a role, for example, then ensure that such characters are escaped in the input to API Policy Store methods like ApplicationPolicy.searchAppRoles(), so they return correct results.

For example, if you have an application role named "appRole^$" it will need to be input as ApplicationPolicy.searchAppRoles("appRole\\^\\$") to find the match in the policy store.

Alternatively, you could use a wild card in the search expression without including these escaped special characters, and it will also match that application role:

ApplicationPolicy.searchAppRoles("appRole*")

L.15.3 Characters in Application Role Names

An application role name is a string of printable characters other than white space, that is, it can contain alpha-numeric characters (ASCII or Unicode) and other printable characters (such as underscore or square brackets) except for white space. This rule applies to all three kinds of supported storage: XML, LDAP, and DB.

L.15.4 Missing Newline Characters in XML Policy Store

In an file-based policy store, a new-line character is required between the closing of a <permission> or <principal> tag and the opening of the following one.

Following are examples of strings illustrating incorrect and correct formats.

Incorrect example fragment of policy store:

<permission>
    <class>java.lang.RuntimePermission</class>
    <name>getClassLoader</name>
 </permission> <permission>
      <class>java.io.FilePermission</class>
      <name>/foo</name>
      <actions>read,write</actions>
 </permission> 

Correct example fragment of policy store:

<permission>
    <class>java.lang.RuntimePermission</class>
    <name>getClassLoader</name>
</permission>
<permission>
      <class>java.io.FilePermission</class>
      <name>/foo</name>
      <actions>read,write</actions>
</permission>

L.16 Granting Permissions in Java SE Applications

This section describes the correct way to code a grant in Java SE applications. Even though the problem described is not an issue in Java EE applications, for maximum portability, it is recommended that this solution be used in Java EE applications too.

Symptom

The application code includes a fragment like the following, by an application creates a grant:

Permission p = new FilePermission(resourceName, "write");
PrincipalEntry myPrincipal2 = InfoFactory.newPrincipalEntry("weblogic.security.principal.WLSGroupImpl", enterpriseRoleName2);
ap.grant(new Principal[]{myPrincipal2.getPrincipal()}, null, new Permission[]{p});

At runtime, however, the grant is not taking effect as expected.

Diagnosis

A bit of inspection indicates that the policy store repository includes the following attribute:

orcljaznjavaclass=oracle.security.jps.internal.policystore.UnresolvedPrincipal+cn=enterpriseRoleName2

Solution

The lines of code above should be replaced by the following:

Permission p = new FilePermission (resourceName, "write");
PermissionEntry permEntry  = InfoFactory.newPermissionEntry(p.getClassName(), p.getName(),  p.getActions());
ap.grant (new PrincipalEntry[] {myPrincipal2}, null, new PermissionEntry[] {permEntry});

The solution uses the array PrincipalEntry instead of the array Principal and the array PermissionEntry instead of the array Permission.

Note:

This same issue applies to the method revoke, which also has overloaded variants that accept Principal[] or PrincipalEntry[]

L.17 Troubleshooting Oracle Business Intelligence Reporting

This section describes common problems and solutions for Oracle Business Intelligence when used as a reporting tool for Oracle Fusion Middleware security. It contains the following topics:

L.17.1 Audit Templates for Oracle Business Intelligence Publisher

To view Oracle Fusion Middleware Audit Framework reports in Oracle Business Intelligence, you must use the appropriate audit templates.

For details, see Section 13.1.3, "Set Up Oracle Reports in Oracle Business Intelligence Publisher".

L.17.2 Oracle Business Intelligence Publisher Time Zone

You may see problems with Oracle Fusion Middleware Audit Framework reports if Oracle Business Intelligence Publisher and the database are installed in sites with different time zones.

To avoid this issue, ensure that Oracle Business Intelligence Publisher and the database are installed in the same time zone.

L.18 Search Failure when Matching Attribute in Policy Store

This section describes a reason why cataloging of an attribute is needed.

Symptom

While searching the policy store, an exception similar to the following is encountered:

oracle.security.jps.service.policystore.PolicyStoreOperationNotAllowedException
javax.naming.OperationNotSupportedException:
[LDAP: error code 53 - Function Not Implemented, search filter attribute orcljpsresourcetypename is not indexed/cataloged];
remaining name 'cn=Permissions,cn=JAASPolicy,cn=IDCCS, cn=sprint6_policy_domain,cn=JPSContext,cn=FusionAppsPolicies'

Diagnosis

The error above indicates that the attribute orcljpsresourcetypename must be cataloged before it is used in a filter to search the policy store.

Solution

To catalog an attribute, such as orcljpsresourcetypename, see section Section 8.8, "Cataloging Oracle Internet Directory Attributes."

L.19 Search Failure with an Unknown Host Exception

When searching for information in an Active Directory environment that is configured for LDAP referrals, the referrals fail if the host being referred to is in a different domain than the Active Directory server.

Symptom

When a user requests a resource, at times verification of the user's identity can fail due to an inability to validate the user's identity in the directory. This error can occur in an Active Directory environment when the user's browser runs on a non-Windows computer, or if the user's browser runs on a Windows computer that is not in the Active Directory server domain.

Diagnosis

This problem can arise due to LDAP referral chasing. An LDAP referral occurs when a domain controller does not have the section of the directory tree where a requested object resides. The domain controller refers the client to another destination so that the client can conduct a DNS search for another domain controller. If the client is configured to chase referrals, the search can continue.

For the scenario where the user has a Windows-based computer, an issue can occur with LDAP referrals if the client's domain controller does not have a trust relationship with the Active Directory domain controller.

Solution

If you encounter this issue, add the entry for the Active Directory host's address in the following list:

WINDOWS_HOME_DIRECTORY\system32\drivers\etc\hosts

On Windows XP, the list is located here:

C:\WINDOWS\system32\drivers\etc\host

On a Unix-based system, add this entry to the /etc/hosts file, using the format:

IP_address_of_AD_host AD_host_name

where AD_host_name is the host name specified in the referral, for example:

123.123.123.123 my2003ad.com

L.20 Incompatible Versions of Binaries and Policy Store

This section describes the reason why the server would throw the exception PolicyStoreIncompatibleVersionException.

Symptom

An error similar to the following is logged or issued by the server:

Oracle.security.jps.service.policystore. PolicyStoreIncompatibleVersionException
JPS-06100: Policy Store version 11.1.1.5.0 and Oracle Platform Security Services
version 11.1.1.4.0 are not compatible.

Diagnosis

The above exception indicates that the domain OPSS binaries version (11.1.1.4.0) and the policy store version (11.1.1.5.0) used by that domain have incompatible versions. The version of the policy store is established during reassociation and that version is used until the policy store is upgraded to a newer version.

OPSS domain binary versions are backward compatible with policy store versions used by that domain, but they are not forward compatible. Thus, the error above indicates that the policy store has version newer that the version of the OPSS binaries. PS3 OPSS binaries cannot use a newer version of the policy store.

Here are three scenarios where OPSS binaries ends up running into this incompatibility.

  • Scenario 1

    • Domain1 and Domain2 point to the same policy store; Domain1, Domain2, and that policy store are all three version PS2.

    • The binaries in Domain1 are upgraded to PS3.

    • The policy store is upgraded to PS3 (using the command upgradeOPSS).

    • When the Domain2 is brought up again, its version and the policy store version are incompatible.

  • Scenario 2

    • Domain1 points to a policy store and both are version PS2.

    • An attempt to migrate the policy store to a PS3 policy store fails because the migration would render a scenario with incompatible versions.

      Migration is supported only when the versions of the OPSS binaries and the policy store are same.

  • Scenario 3

    • A PS2 Domain1 attempts to join a PS3 policy store (in some other domain), using the command reassociateSecurityStore with the join argument.

    • The operation fails because the sharing would render a scenario with incompatible versions.

      Reassociation is supported only when the versions of the OPSS binaries and the policy store are same.

Solution

The solution, common to all three scenarios above, is either one of the following:

  • Update the domain OPSS binaries to match the version of the policy store the domain is pointing to.

  • Reassociate the domain policy store to a policy store that has version not older than the version of the domain OPSS binaries.

L.21 Need Further Help?

You can find more solutions on My Oracle Support (formerly MetaLink) at http://myoraclesupport.oracle.com. If you do not find a solution to your problem, log a service request.