1. Enterprise Deployment Guide for Oracle Identity and Access Management in a Kubernetes Cluster
  2. Common Configuration and Management Procedures for an Enterprise Deployment
  3. Troubleshooting

30 Troubleshooting

You can troubleshoot the common issues that may arise with the Identity and Access Management enterprise deployment. The solutions provided for the common problems help you resolve them quickly.

This chapter includes the following topics:

Parent topic: Common Configuration and Management Procedures for an Enterprise Deployment

Troubleshooting Oracle Access Management Access Manager

Learn about some of the common problems that you may encounter with Oracle Access Manager and the actions you can take to resolve them.

Parent topic: Troubleshooting

Access Manager Runs out of Memory

Problem

After Access Manager has been running for a while, you see the following error message in the output: 

Attempting to allocate 1G bytes
There is insufficient native memory for the Java Runtime Environment to continue.

Possible reasons

  • The system is out of physical RAM or swap space.

  • In 32 bit mode, the process size limit was reached.

Solutions

  • Reduce memory load on the system.

  • Increase physical memory or swap space.

  • Check if swap backing store is full.

  • Use 64 bit Java on a 64 bit OS.

  • Decrease Java heap size (-Xmx/-Xms).

  • Decrease number of Java threads.

  • Decrease Java thread stack sizes (-Xss).

  • Disable compressed references (-XXcompressedRefs=false).

  • Ensure that command line tool adrci can be executed from the command line.

    • at oracle.dfw.impl.incident.ADRHelper.invoke(ADRHelper.java:1309)

    • at oracle.dfw.impl.incident.ADRHelper.createIncident(ADRHelper.java:929

    • at oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl.createADRIncident(DiagnosticsDataExtractorImpl.java:1116)

  • On both OAMHOST1 and OAMHOST2, edit the file setSOADomainEnv.sh, which is located in IAD_MSERVER_HOME/bin and locate the line which begins: 

    PORT_MEM_ARGS=

    Change this line so that it reads:

    PORT_MEM_ARGS="-Xms768m -Xmx2560m"

Parent topic: Troubleshooting Oracle Access Management Access Manager

Access Domain Creation Times Out

Problem

When creating the Access domain, you will see an error in the log file, similar to the following:

[ERROR] Exiting due to failure - the job status is not Completed!

Possible reasons

There is a performance issue in your setup.

Solution

Increase the timeout value when running the command to create the domain. For example:
./create-domain.sh -i $WORKDIR/create-domain-inputs.yaml -t 1200 -o output

Where, 1200 is the number of seconds to wait before timing out. The default value is 600.

Parent topic: Troubleshooting Oracle Access Management Access Manager

User Reaches the Maximum Allowed Number of Sessions

Problem

The Access Manager Server displays an error message similar to this:
The user has already reached the maximum allowed number of sessions. Please close one of the existing sessions before trying to login again.

Solution

If users log in multiple times without logging out, they might overshoot the maximum number of configured sessions. You can modify the maximum number of configured sessions by using the Access Management Administration Console.

To modify the configuration by using the Access Management Administration Console, proceed as follows:

  1. Go to System Configuration -> Common Settings -> Session
  2. Increase the value in the Maximum Number of Sessions per User field to cover all concurrent login sessions expected for any user. The range of values for this field is from 1 to any number.

Parent topic: Troubleshooting Oracle Access Management Access Manager

Policies Do Not Get Created When Oracle Access Management Access Manager is First Installed

Problem

The Administration Server takes a long time to start after configuring Access Manager.

Solution

Tune the Access Manager database. When the Administration Server first starts after configuring Access Manager, it creates a number of default policies in the database. If the database is distant or in need of tuning, this can take a significant amount of time.
Resources
Authentication Policies
   Protected Higher Level Policy
   Protected Lower Level Policy
   Publicl Policy
Authorization Policies
   Authorization Policies

If you do not see these items, the initial population has failed. Check the Administration Server log file for details.

Parent topic: Troubleshooting Oracle Access Management Access Manager

You Are Not Prompted for Credentials After Accessing a Protected Resource

Problem

When you access a protected resource, Access Manager should prompt you for your user name and password. For example, after creating a simple HTML page and adding it as a resource, you should see credential entry screen.

Solution

If you do not see the Credential Entry screen, perform the following steps:

  1. Verify that host aliases for IAMAccessDomain have been set. You should have aliases for IAMAccessDomain:80, IAMAccessDomain:Null, IADADMIN.example.com:80, and login.example.com:443, where Port 80 is HTTP_PORT and Port 443 is HTTP_SSL_PORT.
  2. Verify that WebGate is installed.
  3. Verify that ObAccessClient.xml was copied from IAD_ASERVER_HOME/output to the WebGate Lib directory and that OHS was restarted.
  4. When you first created the ObAccessClient.xml file, it was not formatted. When you restart OHS, re-examine the file to ensure that it is formatted. OHS gets a new version of the file from Access Manager when it first starts.
  5. Shut down the Access Manager servers and access the protected resource. If you do not see an error saying Access Manager servers are not available, re-install WebGate.

Parent topic: Troubleshooting Oracle Access Management Access Manager

Cannot Log In to Access Management Console

Problem

You cannot log in to the Access Management Console. The Administration Server diagnostic log might contain an error message similar to this:
Caused by: oracle.security.idm.OperationFailureException:
oracle.security.am.common.jndi.ldap.PoolingException [Root exception is oracle.ucp.UniversalConnectionPoolException:
Invalid life cycle state.
 Check the status of the Universal Connection Pool]
         at
oracle.security.idm.providers.stdldap.UCPool.acquireConnection(UCPool.java:112)

Solution

Remove the /tmp/UCP* files and restart the Administration Server.

Parent topic: Troubleshooting Oracle Access Management Access Manager

Oracle Coherence Cluster Startup Errors in oam_policy_mgr Server Logs

Problem

The oam_policy_mgr2 server has oam application deployment in failed state. The oam_policy_mgr2 server logs report request timeout exceptions while starting the cluster service, similar to following logs: 

Oracle Coherence GE 3.7.1.13 <Warning> (thread=Cluster, member=n/a): Delaying 
formation of a new cluster; IpMonitor failed to verify the reachability of senior 
Member(Id=1, Timestamp=, Address=, MachineId=,
Location=site:,machine:IADADMINVHN,process:8499, Role=WeblogicServer); if this 
persists it is likely the result of a local or remote firewall rule blocking
either ICMP pings, or connections to TCP port 7>

Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout 
during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=MasterMemberSet(
ThisMember=null
OldestMember=null
ActualMemberSet=MemberSet(Size=0
)
MemberId|ServiceVersion|ServiceJoined|MemberState
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
)
)
)
at
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)

at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)

at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)

Solution

This is a known issue. In some of the environments, the Access Policy Manager Server that is not running on the same host as the WebLogic Administration Server is unable to start the coherence cluster service, which results in the oam application deployment to be in failed state. To solve this issue, you must create a server instance for the effected Access Policy Manager Server by completing the following steps:

  1. Log in to the OAM Console using the following URL:

    http://iadadmin.example.com/oamconsole

    Log in as the Access Manager administration user you created when you prepared the ID Store. For example, oamadmin.

  2. Click Configuration.
  3. Click Server Instances from the configuration launch pad.
  4. Click a new server instance for the Access Policy Manager WebLogic Managed Server, that is not running on the same machine as the IAMAccessDomain Admin Server. For example:

    • Name: oam_policy_mgr2

    • Port: 14150

    • Host: OAMHOST2 (For consolidated topology, the host will be IAMHOST2)

    Note:

    Provide the OAM Proxy details similar to the server instance for oam_server.

  5. Click Apply.

Parent topic: Troubleshooting Oracle Access Management Access Manager

Errors in Log File When Starting OAM Servers

Problem

When you start the OAM Servers, errors similar to the following are seen in the log files which causes LCM heath check module to fail:

[oam_server1] [TRACE:16] [] [oracle.oam.config] [tid: DistributedCacheWorker:4] [userId: <anonymous>] [ecid: 
0000LGmRJqxB9DE5N7P5ie1N5mOd000004,1:16514] [APP: oam_server#11.1.2.0.0] [SRC_CLASS: oracle.security.am.admin.config.util.MapUtil] [SRC_METHOD: 
getDefaultedStringValue] property not found at path:[Ljava.lang.String;@43537067 Defaulting to value:,
[2016-04-20T06:55:39.982+00:00] [oam_server1] [TRACE:16] [] [oracle.oam.config] [tid: DistributedCacheWorker:4] [userId: <anonymous>] [ecid: 
0000LGmRJqxB9DE5N7P5ie1N5mOd000004,1:16514] [APP: oam_server#11.1.2.0.0] [SRC_CLASS: oracle.security.am.admin.config.util.MapUtil] [SRC_METHOD: getStringValue] THROW[[
oracle.security.am.admin.config.ConfigurationException: Cannot get java.lang.String value from configuration for key ResponseEscapeChar. Object null found.
at oracle.security.am.admin.config.util.MapUtil.handleFailedAttributeAccess(MapUtil.java:447)
at oracle.security.am.admin.config.util.MapUtil.getStringValue(MapUtil.java:130)
at oracle.security.am.admin.config.util.MapUtil.getDefaultedStringValue(MapUtil.java:147)
at oracle.security.am.engines.common.identity.provider.util.IdStoreConfig.initializeConfig(IdStoreConfig.java:76)
at oracle.security.am.engines.common.identity.provider.util.IdStoreConfig.<init>(IdStoreConfig.java:69)
at oracle.security.am.engines.common.identity.provider.util.IdStoreConfig.getConfig(IdStoreConfig.java:128)
at oracle.security.am.engines.common.identity.util.OAMUserAttribute.getStringValue(OAMUserAttribute.java:76)
at oracle.security.am.engines.common.identity.util.OAMUserAttribute.toString(OAMUserAttribute.java:114)
at java.lang.String.valueOf(String.java:2849)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at java.util.AbstractMap.toString(AbstractMap.java:523)
at java.lang.String.valueOf(String.java:2849)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at oracle.security.am.engines.common.identity.util.OAMIdentity.toString(OAMIdentity.java:678)
at java.lang.String.valueOf(String.java:2849)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at oracle.security.am.engines.sso.SSOSubject.toString(SSOSubject.java:238)
at java.lang.String.valueOf(String.java:2849)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at oracle.security.am.engines.sme.impl.SessionImpl.toString(SessionImpl.java:629)
at java.lang.String.valueOf(String.java:2849)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at oracle.security.am.engines.sme.mapimpl.db.DbOraSmeStore.loadSession(DbOraSmeStore.java:1705)
at oracle.security.am.engines.sme.mapimpl.db.DbOraSmeStore.loadSession(DbOraSmeStore.java:1691)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at oracle.security.am.foundation.mapimpl.coherence.store.DataConnectionUtility.invokeSqlOperationWithRetries(DataConnectionUtility.java:275)
at oracle.security.am.engines.sme.mapimpl.db.DbOraSmeStore.load(DbOraSmeStore.java:1284)
at com.tangosol.net.cache.ReadWriteBackingMap$CacheStoreWrapper.loadInternal(ReadWriteBackingMap.java:5676)
at com.tangosol.net.cache.ReadWriteBackingMap$StoreWrapper.load(ReadWriteBackingMap.java:4754)
at com.tangosol.net.cache.ReadWriteBackingMap.get(ReadWriteBackingMap.java:717)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.get(PartitionedCache.CDB:10)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onGetRequest(PartitionedCache.CDB:23)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$GetRequest.run(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:66)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:745)
]]

Solution

This occurs when OAM servers cannot communicate with each other using the coherence port. This is often caused by iptables. The workaround for this issue is as follows:

  1. Edit the file /etc/sysconfig/iptables on both OAMHOST1 and OAMHOST2 and add the following line:
    # Generated by iptables-save v1.4.7 on Tue Apr 19 10:02:45 2016
*filter
:INPUT ACCEPT [593:243587]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [614:423013]
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9095 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9097 -j ACCEPT
COMMIT

    In the above set of lines, 9095 and 9097 are the coherence ports being used.

  2. Save the file and restart the servers.

Parent topic: Troubleshooting Oracle Access Management Access Manager

Too Many Redirects Error in Browser

Problem

When navigating from one application to another that uses the same OAM for SSO, you get a redirection error in the web browser. There are two different configurations to validate.

Solution 1:

  1. Log in to the OAM Console at iadadmin.example.com/oamconsole.
  2. From the Launch Pad, click the Agents icon.
  3. In the resulting window > Webgates tab, click search. No search parameters need to be input.
  4. In the search results, click the IAMSuiteAgent link.
  5. Ensure that the Primary Cookie Domain is set to the domain that is used for the login.example.com domain. For example: example.com.
  6. Restart all WebGate OHS instances.

Solution 2:

Ensure that the date and time on all OHS and OAM servers are within 60 seconds of each other. If they are not:

  1. Ensure that the NTP setting are the same and valid on all OHS and OAM hosts.
  2. Start or restart the ntpd service on all hosts.
  3. Restart all WebGate OHS instances, the OAM domain AdminServer, and all Managed Servers.

Parent topic: Troubleshooting Oracle Access Management Access Manager

Troubleshooting Oracle Identity Governance

Learn about some of the common problems that may arise with Oracle Identity Manager and the actions you can take to resolve the problem.

Parent topic: Troubleshooting

OIM Bootstrap Process Fails

Problem

The OIM Bootstrap process fails after deploying composites. The error appears as follows:
Deployment of SOA Composites :-/<INSTALL_LOCATION>/Oracle_Home/idm/server/workflows/composites/scajars/sca_DefaultRequestApproval_rev6.0.jar is successful>
<Jun 12, 2018 4:20:26,136 PM CEST> <Info> <oracle.iam.OIMPostConfigManager> <BEA-000000> <updating feature:DEPLOYSOACOMPOSITESwith state :COMPLETEwith executionTime190108>
java.sql.SQLException: Connection closed

This is caused by a performance issue.

Solution

To resolve the issue temporarily, increase the inactivity timeouts on the following data sources:

  • oimJMSStoreDS
  • oimOperationsDB

The settings can be restored to their original values after the upgrade is complete.

  1. Log in to the WebLogic Server Administration Console.
  2. Click Lock and Edit.
  3. Click Services, Data Sources, and then select the <Data source name>.
  4. Click the Connection Pool tab.
  5. Under the Advanced section, increase the value of Inactive Connection Timeout.
  6. Save and activate the changes.
  7. Restart the OIM Managed Server.

Parent topic: Troubleshooting Oracle Identity Governance

java.io.FileNotFoundException When Running Oracle Identity Governance Configuration

Problem

The following content was added to address bug 12390838

When you run Oracle Identity Manager configuration, the error java.io.FileNotFoundException: soaconfigplan.xml (Permission denied) may appear and Oracle Identity Manager configuration might fail.

Solution

To workaround this issue:

  1. Delete the file /tmp/soaconfigplan.xml.
  2. Start the configuration again (IGD_ORACLE_HOME/bin/config.sh).

Parent topic: Troubleshooting Oracle Identity Governance

ResourceConnectionValidationxception When Creating User in Oracle Identity Governance

Problem

The following content was added to address bug 9816870

If you are creating a user in Oracle Identity Manager (by logging into Oracle Identity Manager System Administration Console, clicking the Administration tab, clicking the Create User link, entering the required information in the fields, and clicking Save) in an active-active Oracle Identity Manager configuration, and the Oracle Identity Manager server that is handling the request fails, you may see a "ResourceConnectionValidationxception" in the Oracle Identity Manager log file, similar to: 

[2010-06-14T15:14:48.738-07:00] [oim_server2] [ERROR] [] [XELLERATE.SERVER]
[tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default
(self-tuning)'] [userId: xelsysadm] [ecid:
004YGJGmYrtEkJV6u3M6UH00073A0005EI,0:1] [APP: oim#11.1.1.3.0] [dcid:
12eb0f9c6e8796f4:-785b18b3:12938857792:-7ffd-0000000000000037] [URI:
/admin/faces/pages/Admin.jspx] Class/Method:
PooledResourceConnection/heartbeat encounter some problems: Operation timed
out[[
com.oracle.oim.gcp.exceptions.ResourceConnectionValidationxception: Operation
timed out
        at
oracle.iam.ldapsync.impl.repository.LDAPConnection.heartbeat(LDAPConnection.ja
va:162)
        at
com.oracle.oim.gcp.ucp.PooledResourceConnection.heartbeat(PooledResourceConnec
tion.java:52)
         .
         .
         .

Solution

Despite this exception, the user is created correctly.

Parent topic: Troubleshooting Oracle Identity Governance

OIG Managed Servers Fail to Join Coherence Cluster

Problem

One or more Managed Servers in the domain fail to start. Examining the log files shows that they are unable to join the Coherence cluster.

Solution 1: Check Firewall (iptables) Requirements

Some Kubernetes distributions create iptables rules that block some types of traffic that Coherence requires to form clusters. If you are not able to form clusters, then you can check for this issue using the following command:
iptables -t nat -v  -L POST_public_allow -n

You should output similar to the following:

Chain POST_public_allow (1 references)
pkts bytes target     prot opt in     out     source               destination
164K   11M MASQUERADE  all  --  *      !lo     0.0.0.0/0            0.0.0.0/0
   0     0 MASQUERADE  all  --  *      !lo     0.0.0.0/0            0.0.0.0/0
If you see a similar output, for example, if you see any entries in this chain, then you need to remove them. You can remove the entries using this command:
iptables -t nat -v -D POST_public_allow 1

Note that you will need to run that command for each line. So, in this example, you would need to run it twice.

After you are done, you can run the previous command again and verify that the output is now an empty list.

After making this change, restart your domains and the Coherence cluster should now form correctly.

Solution 2: Make iptables Updates Permanent Across Reboots

The recommended way to make iptables updates permanent across reboots is to create a systemd service that applies the necessary updates during the startup process.

Here is an example; you may need to adjust this to suit your own environment:

  • Create a systemd service:
    echo 'Set up systemd service to fix iptables nat chain at each reboot (so Coherence will work)...'
    mkdir -p /etc/systemd/system/
    cat > /etc/systemd/system/fix-iptables.service << EOF
[Unit]
Description=Fix iptables
After=firewalld.service
After=docker.service

[Service]
ExecStart=/sbin/fix-iptables.sh

[Install]
WantedBy=multi-user.target
EOF
  • Create the script to update iptables:
    cat > /sbin/fix-iptables.sh << EOF
#!/bin/bash
echo 'Fixing iptables rules for Coherence issue...'
TIMES=$((`iptables -t nat -v -L POST_public_allow -n --line-number | wc -l` - 2))
COUNTER=1
while [ $COUNTER -le $TIMES ]; do
  iptables -t nat -v -D POST_public_allow 1
  ((COUNTER++))
done
EOF
  • Start the service (or just reboot):
    echo 'Start the systemd service to fix iptables nat chain...'
    systemctl enable --now fix-iptables

Parent topic: Troubleshooting Oracle Identity Governance

Oracle Identity Manager Reconciliation Jobs Fail

Problem

Oracle Identity Manager reconciliation jobs fail, or one of the following messages is seen in the log files:

  • Error-1

    LDAP Error 53 : [LDAP: error code 53 - Full resync required. Reason: The provided cookie is older than the start of historical in the server for the replicated domain : dc=example,dc=com]

  • Error-2

    LDAP: error code 53 - Invalid syntax of the provided cookie

This error is caused by the data in the Oracle Unified Directory change log cookie expiring because Oracle Unified Directory has not been written to for a certain amount of time.

Solution

  1. Open a browser and go to the following location:

    http://igdadmin.example.com/sysadmin

  2. Log in a as xelsysadm using the COMMON_IDM_PASSWORD.

  3. Under System Management, click Scheduler.

  4. Under Search Scheduled Jobs, enter LDAP * (there is a space before *) and hit Enter.

  5. For each job in the search results, click on the job name on the left, then click Disable on the right.

    Do this for all jobs. If the job is already disabled do nothing.

  6. Run the following commands on LDAPHOST1:

    cd LDAP_ORACLE_INSTANCE/OUD/bin
./ldapsearch -h LDAPHOST1 -p 1389 -D "cn=oudadmin" -b "" -s base "objectclass=*" lastExternalChangelogCookie

Password for user 'cn=oudadmin': <OudAdminPwd>
dn: lastExternalChangelogCookie: dc=example,dc=com:00000140c682473c263600000862;

    Copy the output string that follows lastExternalChangelogCookie:. This value is required in the next step. For example, 

    dc=example,dc=com:00000140c682473c263600000862;

    The Hex portion must be 28 characters long. If this value has more than one Hex portion then separate the 28char portions with spaces. For example:

    dc=example,dc=com:00000140c4ceb0c07a8d00000043 00000140c52bd0b9104200000042 00000140c52bd0ba17b9000002ac 00000140c3b290b076040000012c;

  7. Run each of the following LDAP reconciliation jobs once to reset the last change number.:

    • LDAP Role Delete Reconciliation

    • LDAP User Delete Reconciliation

    • LDAP Role Create and Update Reconciliation

    • LDAP User Create and Update Reconciliation

    • LDAP Role Hierarchy Reconciliation

    • LDAP Role Membership Reconciliation

    To run the jobs:

    1. Login to the OIM System Administration Console as the user xelsysadm.

    2. Under System Configuration, click Scheduler.

    3. Under Search Scheduled Jobs, enter LDAP * (there is a space before *) and hit Enter.

    4. Click on the job to be run.

    5. Set the parameter Last Change Number to the value obtained in step 6.

      For example:

      dc=example,dc=com:00000140c4ceb0c07a8d00000043 00000140c52bd0b9104200000042 00000140c52bd0ba17b9000002ac 00000140c3b290b076040000012c;

    6. Click Run Now.

    7. Repeat for each of the jobs in the list at the beginning of this step.

  8. For each incremental recon job whose last changelog number has been reset, execute the job and check that the job now completes successfully.

  9. After the job runs successfully, re-enable periodic running of the jobs according to your requirements.

If the error appears again after the incremental jobs have been re-enabled and run successfully ("Full resync required. Reason: The provided cookie is older..."), then increase the OUD cookie retention time. Although there is no hard and fast rule as to what this value should be, it should be long enough to avoid the issue, but small enough to avoid unnecessary resource consumption on OUD. One or two weeks should suffice. Run the following command on each OUD instance to increase the retention time to two weeks:

cd OUD_ORACLE_INSTANCE/bin

./dsconfig set-replication-server-prop --provider-name "Multimaster Synchronization" --set replication-purge-delay:2w -D cn=oudadmin --trustAll -p 4444 -h LDAPHOSTn

Password for user 'cn=oudadmin':  <OudAdminPswd>
Enter choice [f]: f

Parent topic: Troubleshooting Oracle Identity Governance

OIM Reconciliation Jobs Fail When Running Against Oracle Unified Directory

Problem

Reconciliation jobs fail when running against Oracle Unified Directory (OUD). The following error is seen in the OIM WebLogic Server logs:

LDAP: error code 53 - Invalid syntax of the provided cookie

Solution

Perform the workaround described in Oracle Identity Manager Reconciliation Jobs Fail. If this workaround does not resolve the issue, try the following solution:

On each OIMHOST, update the DOMAIN_HOME/config/fmwconfig/ovd/oim/adapters.os_xml file with the following parameter: 

<param name="eclCookie" value="false"/>

Restart the OIM and SOA Managed Servers.

Parent topic: Troubleshooting Oracle Identity Governance

Cannot Open Reports from OIM Self Service Console

Problem

The reports cannot be opened from OIM Self Service Console.

Solution

When you enable the Identity Auditor feature in OIM, do the following configuration changes for the OIM-BI Publisher integration to work fine:

  1. Log in to the IAMGovernanceDomain Enterprise Management Console.
  2. Open the system MBean browser and update the MBean "oracle.iam:Location=wls_oim1,name=Discovery,type=XMLConfig.DiscoveryConfig,XMLConfig=Config,Application=oim,ApplicationVersion=11.1.2.0.0" with Value as http://igdadmin.example.com/.

    Here, igdadmin.example.com is the Governance Domain admin Load balancer URL.

Parent topic: Troubleshooting Oracle Identity Governance

Pending Violations Not Displaying the Correct List

Problem

When viewing the pending violations list, you may see entries that are missing or entries that do not belong to the list.

Solution

If you encounter this issue, a restart of the OIG domain usually resolves it. If the issue is not resolved, raise a Service Request (SR) with Oracle Support.

Parent topic: Troubleshooting Oracle Identity Governance

Domain Patching Failure

Problem

The OIG domain patching fails when you run the patch_oig_domain.sh script.

Solution

If you encounter a patching failure, run the following command to diagnose the issue:
$ kubectl describe domain <OIG_DOMAIN_NAME> -n <OIGNS>
For example:
kubectl describe domain governancedomain -n oigns

Use the output to diagnose the problem and resolve the issue. Also, check the log directory (by default under $WORKDIR/kubernetes/domain-lifecycle) for more details.

Parent topic: Troubleshooting Oracle Identity Governance

Troubleshooting Oracle SOA Suite

Learn about the transaction timeout error that may arise with Oracle SOA Suite and the action you can take to resolve the problem.

Parent topic: Troubleshooting

Transaction Timeout Error

Problem

The following transaction timeout error appears in the log: 

Internal Exception: java.sql.SQLException: Unexpected exception while enlisting
 XAConnection java.sql.SQLException: XA error: XAResource.XAER_NOTA start()
failed on resource 'SOADataSource_soaedg_domain': XAER_NOTA : The XID
is not valid

Solution

Check your transaction timeout settings, and be sure that the JTA transaction time out is less than the DataSource XA Transaction Timeout, which is less than the distributed_lock_timeout (at the database).

With the out of the box configuration, the SOA data sources do not set XA timeout to any value. The Set XA Transaction Timeout configuration parameter is unchecked in the WebLogic Server Administration Console. In this case, the data sources use the domain level JTA timeout which is set to 30. Also, the default distributed_lock_timeout value for the database is 60. As a result, the SOA configuration works correctly for any system where transactions are expected to have lower life expectancy than such values. Adjust these values according to the transaction times your specific operations are expected to take.

Parent topic: Troubleshooting Oracle SOA Suite

Troubleshooting OAM/OIG Integration

Learn about the error you may encounter during the integration process and the solution to fix this error.

Problem

The following content was added to address bug 27567130

Whilst running configureLDAPConnector, you see the following error message:

2018-02-19 06:54:05] LDAPConnectorConfigTool.configureLDAPConnector:  exception: java.lang.reflect.UndeclaredThrowableException  [2018-02-19 06:54:05] javax.management.InstanceNotFoundException: Unable to  contact MBeanServer for  oracle.iam:Location=oim_server1,name=SSOIntegrationMXBean,type=IAMAppRuntimeMB  ean,Application=oim  at weblogic.utils.StackTraceDisabled.unknownMethod()

Solution

This is caused by the OIM Managed Server being called something other than oim_server1. This can be recovered by executing the following workaround. 

Ensure that your OIM Managed Server is running.

  1. Log in to Oracle Fusion Middleware control using the following URL: http://igdadmin.example.com/em.
  2. Start the System Mbean Browser by selecting Weblogic Domain and then clicking on System MBean browser.
  3. Click on find and enter the Mbean name SSOIntegrationMXBean .
  4. Click Search.
  5. When the MBean is found, click Operations > addContainerRules .
  6. Enter the following information:
    Oracle_Home set to the value of IGD_ORACLE_HOME dirType. set to OUD   
userContainer set to 
cn=users,
dc=example,
dc=com    
roleContatiner set to cn=groups,
dc=example,dc=com
  7. Click Invoke button.

Parent topic: Troubleshooting

Troubleshooting Oracle Advanced Authentication

Learn about some of the common problems that may arise with the Oracle Advanced Authentication and the actions you can take to resolve the problem.

Parent topic: Troubleshooting

Creating the Oracle Database Schema Causes an Error

Problem

When you create the Oracle Database schema, an error similar to the following is shown:

ORA-12521: TNS:listener does not currently know of instance requested in connect descriptor

Solution

Ensure that the database.name parameter is empty. That is, no value should appear after the "=" sign.

Parent topic: Troubleshooting Oracle Advanced Authentication

OAA Deployment Results in an Error

Problem

When you deploy OAA, the following message is shown:

OAUTH validation failed
Oauth validation failed..
command terminated with exit code 1

Solution

Run the following command inside the OAA Management container to get more information:

/u01/oracle/scripts/validateOauthForOAA.sh -f
/u01/oracle/scripts/settings/installOAA.properties -d true

Parent topic: Troubleshooting Oracle Advanced Authentication

General Troubleshooting

Learn about the error you may encounter when starting the Managed Server from the WebLogic Console and the resolution to fix the error.

Parent topic: Troubleshooting

Cannot Start Managed Server from WebLogic Console

Problem

When you start a Managed Server from the WebLogic Console, the following error is shown:

. For server WLS_BI1, the Node Manager associated with machine OIMHOST1 is not reachable.
. All of the servers selected are currently in a state which is incompatible with this operation or are not associated with a running Node Manager or you are not authorized to perform the action requested. No action will be performed.

Solution 1

Check if the Node Manager is started on the target host. If not, start it.

Solution 2

Verify that the domain is listed in the file nodemanager.domains, which is located in the directory SHARED_CONFIG_DIR/nodemanger/hostname. If not, do the following:

  1. Start the WebLogic Scripting Tool (WLST) by running the following command from the location ORACLE_HOME/oracle_common/common/bin/:

    ./wlst.sh

  2. Connect to the domain you wish to add by running the following command:

    connect('weblogic_user','password','t3://ADMINVHN:AdminPort')

    In this command:

    weblogic_user is the WebLogic Administration user. For example, weblogic or weblogic_idmw.

    password is the password of the WebLogic Administration user.

    ADMINVHN is the Virtual host name of the Administration Server. For example, IGDADMINVHN or IADADMINVHN.

    adminPort is the port on which the Administration Server is running. For example, 7101.

    Sample Command:

    connect('weblogic_idm','<password>','t3://IGDADMINVHN.example.com:7001')

  3. Enrol the domain using the following command:

    nmEnroll(domainDir=absolute_path_to_the_domain,nm_Home=absolute_path_to_the_nodemanager_home)

    For example:

    nmEnroll(domainDir='/u02/private/oracle/config/domains/IAMGovernanceDomain/',nmHome='/u01/oracle/config/nodemanger/hostname)')

    Note:

    For Managed Servers, the domain home should always be specified as the local Managed Server directory.

Parent topic: General Troubleshooting

Troubleshooting Kubernetes Domains

Learn about some of the common problems you may encounter with Kubernetes domains and the actions you can take to resolve these problems.

Parent topic: Troubleshooting

WebLogic Domain Creation Fails

Problem

The WebLogic domain creation fails when you run the create-domain.sh command.

Solution

To diagnose the issue:
  1. Run the following command to diagnose the create domain job:
    $ kubectl logs <domain_job> -n <domain_namespace>
    For example:
    $ kubectl logs accessinfra-create-fmw-infra-sample-domain-job-c6vfb -n accessns
    Also run:
    $ kubectl describe pod <domain_job> -n <domain_namespace>
    For example:
    $ kubectl describe pod accessinfra-create-fmw-infra-sample-domain-job-c6vfb -n accessns

    Using the output you should be able to diagnose the problem and resolve the issue.

  2. If any of the above commands return the following error, it indicates that there is a permissions error on the directory for the PV and PVC:
    Failed to start container "create-fmw-infra-sample-domain-job": Error response from daemon: error while creating mount source path
'/exports/IAMPVS/accessdomainpv ': mkdir /exports/IAMPVS/accessdomainpv : permission denied

    Check the following:

    1. The directory has 777 permissions: chmod -R 777 <work directory>/accessdomainpv.
    2. If it does have the permissions, check if an oracle user exists and the uid and gid equal 1000.

      Create the oracle user if it does not exist and set the uid and gid to 1000.

    3. Edit the <work directory>/weblogic-kubernetes-operator/kubernetes/samples/scripts/create-access-domain-pv-pvc/create-pv-pvc-inputs.yaml file and add a slash to the end of the directory for the weblogicDomainStoragePath parameter:
      weblogicDomainStoragePath: /exports/IAMPVS/accessdomainpv/

After you have resolved the issue, delete the job and try again.

To delete the job, use the command:
kubectl -n <NAMESPACE> get all -o wide
This will list the name of the job, as follows:
accessdomain-create-oam-infra-domain-job-b6kfd
Now, use the following command to delete the job listed:
kubectl delete job -n <NAMESPACE> <JOBNAME>
For example:
kubectl delete job -n oamns accessdomain-create-oam-infra-domain-job

Parent topic: Troubleshooting Kubernetes Domains

Domain Fails to Start

Problem

Domain does not start.

Solution

If you see that the Administration server pod has started, log in to the Administration Server using the following command and check the server log files:
kubectl exec -n <NAMESPACE> -ti <DOMAIN_NAME>-adminserver -- /bin/bash
If the Administration Server and Managed Servers fail to appear, check the WLS Operator logs:
kubectl logs -n opns weblogic-operator-<ID>

Parent topic: Troubleshooting Kubernetes Domains

WebLogic Operator Fails to Manage Namespace

Problem

Operator log shows message
\":\"configmaps is forbidden: User \\\"system:serviceaccount:opns:op-sa\\\" cannot watch resource

Solution 1

Check the namespaces that the operator can manage by using the command:
kubectl get ns --selector="weblogic-operator=enabled"

If your namespace is not listed, ensure that the namespace is tagged with weblogic-operator=enabled. See the instructions for creating a namespace for the product you are configuring.

If all the namespaces are listed, ensure that you have enabled label-based namespace management by checking the values provided to the WebLogic Operator installation. Use the following command to check:
helm get values --namespace opns weblogic-kubernetes-operator

Solution 2

If you are using named namespaces as in previous releases of the WebLogic Operator, it is not allowed to manage the namespace. Rerun the following command:
helm upgrade --reuse-values --namespace <operator namspace> --set "domainNamespaces={<namespace>}" --wait weblogic-kubernetes-operator kubernetes/charts/weblogic-operator

Parent topic: Troubleshooting Kubernetes Domains