Troubleshooting Clustered OIM and Eclipselink Cache Coordination

E Troubleshooting Clustered OIM and Eclipselink Cache Coordination

Troubleshooting a clustered deployment of Oracle Identity Manager and Eclipselink cache coordination involves starting Oracle Identity Manager, SOA, and WebLogic services in a clustered deployment, setting the clustered deployment mode, configuring unique multicast address for each cluster, testing the multicast network, enabling additional logging for Eclipselink, and testing multicast connectivity between the nodes.

This appendix can be used by Oracle Identity Manager administrators and developers and WebLogic administrators dealing with clustered deployments of Oracle Identity Manager and SOA. It provides some pointers to verify, test, and correct startup procedure for a clustered installation of Oracle Identity Manager and its required components if eclipselink/toplink cache coordination issues are suspected, for example, if the following exception is seen in the logs:

oracle.iam.platform.kernel.ProcessNotInPrePostStageException

This appendix contains the following topics:

E.1 Startup Procedure for Clustered Installation of Oracle Identity Governance

For eclipselink cache co-ordination to happen successfully in a cluster environment, the clustered nodes must be started one after another.

If you start all the nodes at the same time, then cache co-ordination initialization might not happen properly. In addition, the initial startup of the various managed servers is also critical, and starting servers in an incorrect order can cause some data seeding to fail.

Perform the following when starting Oracle Identity Manager, SOA, and WebLogic services:

Start the WebLogic Admin Server first and wait until it is in a RUNNING state.
Start one node of the SOA cluster and wait until it is in RUNNING state.
Start the next SOA Managed Server and wait until it is in RUNNING state.
Repeat step 3 for all SOA Managed Servers.
Start the first Oracle Identity Manager Managed Server and wait until it is in RUNNING state.
Start the next Oracle Identity Manager Managed Server and wait until it is in RUNNING state.
Repeat step 6 for each remaining Oracle Identity Manaager Managed Server, one at a time.
After all services are in RUNNING state, wait for another two minutes before permitting end user operation and other business use of the system.

E.2 Setting Deployment Mode to Cluster

In a clustered deployment, you need to set the deployment mode to cluster.

Make sure deploymentMode is set to cluster in the oim-config.xml file, and MDS is updated with the changes to the oim-config.xml file. Perform an export of /db/oim-config.xml and verify to make sure it is similar to the following:

<deploymentConfig>
<appServerName>weblogic</appServerName>
<initialContextFactory>weblogic.jndi.WLInitialContextFactory</initialContextFactory>
<dataBaseType>oracle</dataBaseType>
<deploymentMode>cluster</deploymentMode>
</deploymentConfig>

E.3 Configuring Multicast Addressing for Oracle Identity Governance

In a clustered deployment, each cluster must have a unique multicast address.

Ensure that each cluster has its own unique multicast address configured in the oim-config.xml file within MDS. Production and test environments must not share the same multicast IP configurations. For example, the following must not share the same multicastAddress value:

<xLCacheProviderProps multicastAddress="IP_ADDRESS" size="5000">
<properties></properties>
</xLCacheProviderProps>

Do not share the same multicastAddress value as other Oracle Identity Manager deployments. If the value is used on a test domain, then do not specify the same address in the production oim-config.xml file.

E.4 Multicast Addressing for Eclipselink

Eclipselink also makes use of multicast networking for its cache coordination.

Ecliplseink cache coordination uses multicast port 3121 and a Time To Live (TTL) setting of 15 hops. This is not configurable. Firewalls must take into account all multicast networking requirements of the environment.

E.5 Testing Multicast Network Testing

Some simple tests on your multicast network can be testing multicast connectivity between the nodes and running a multicast monitor test in a WebLogic cluster.

Refer to the following Tech notes in the My Oracle Support web site for information about how to run some simple tests on your multicast network:

E.6 Enabling Additional Logging for Eclipselink

Add the logging.xml file with the appropriate level and restart the domain to enable additional logging for Eclipselink.

Add the following to the logging.xml file to enable additional logging for elcipselink/toplink:

<logger name='org.eclipse.persistence.session.oim.propagation' level='TRACE:32' useParentHandlers='false'>
<handler name='odl-handler'/>
</logger>

Restart the domain for the changes to take effect.

E.7 Testing Multicast Connectivity Between Oracle Identity Governance Nodes

Oracle Identity Manager makes use of multicast IP network communications for normal operations. It is used by the Design Console as well as for application-level caching within Oracle Identity Manager. Sometimes, a host or router might have multicast networking disabled.

This section describes how to test and verify if multicast communications between two or more nodes of an Oracle Identity Manager cluster is working.

To verify if multicast packets can be sent and received between different nodes of a WebLogic clustered Oracle Identity Manager environment:

Obtain the multicast IP address used by Oracle Identity Manager. To do so:
1. Expand Identity and Access, OIM, oim(OIM_SERVER), and from the drop-down menu on the right pane, view the System MBean Browser.
2. View the IP Address defined in the MulticastAddress attribute within Application Defined MBeans, oracle-iam, MANAGED_SERVER_NAME, Application: oim, XMLConfig, config, XMLConfig.CacheConfig.XLCacheProvider, XLCacheProvider.
Open a command window on each host involved.
Go to the MIDDLEWARE_HOME directory.
Go to the coherence directory, for example coherence_3.6 or coherence_3.7, depending on which version you have.
If the scripts in the bin directory do not have execute permissions, then use the chmod command to enable execute permissions, as shown:
```
chmod u+x bin/*.sh
```
Run the following command to start sending and receiving multicast packets:
```
bin/multicast-test.sh -group MULTICAST_IP_ADDRESS:PORT
```
Here, MULTICAST_IP_ADDRESS is the IP address obtained in Step 1.

The following is a sample output:
```
Starting test on ip=iam.example.com/10.10.10.10, group=/IP_ADDRESS:12345, ttl=4
Configuring multicast socket...
Starting listener...
Wed Feb 21 21:49:59 UTC 2015: Sent packet 1 containing 1468 bytes.
Wed Feb 21 21:49:59 UTC 2015: Received test packet 5 from ip=idmsun.example.com/10.20.20.20, group=/IP_ADDRESS, ttl=4 containing 1468 bytes.
Wed Feb 21 21:50:00 UTC 2015: Received test packet 1 from self (sent 1628ms ago).
Wed Feb 21 21:50:01 UTC 2015: Received test packet 6 from ip=idmsun.example.com/10.20.20.20, group=/IP_ADDRESS:12345, ttl=4 containing 1468 bytes.
Wed Feb 21 21:50:02 UTC 2015: Sent packet 2 containing 1468 bytes.
Wed Feb 21 21:50:02 UTC 2015: Received test packet 2 from self
Wed Feb 21 21:50:03 UTC 2015: Received test packet 7 from ip=idmsun.example.com/10.20.20.20, group=/IP_ADDRESS:12345, ttl=4 containing 1468 bytes.
```
Here, the multicast-test script is run on both the iam.example.com server and idmsun.example.com server using the same IP address and port number. The iam.example.com host is able to send the multicast packets and also receive the packets from idmsun.example.com host. Observing the output of the test script for a short while on each host is necessary to ensure that each host is receiving packets from all of the other hosts within the cluster. If only packets from self are observed, then this host is not receiving the test packets from any other systems running the multicast-test script.

Note:

If the hosts are not within the number of network hops specified in Time To Live (TTL), then you can change the ttl by adding -ttl 10 to the command.

Eclipselink also makes use of multicast networking for its cache coordination. ecliplseink cache coordination uses multicast port 3121 and a TTL setting of 15 hops. Test both the default Oracle Identity Manager multicast port as well as the eclipselink port.

Note:

If a second NIC is used for multicast, then specify the interface with the -local attribute, such as:

multicast-test.sh -local UNICAST_ADDRESS -group IP_ADDRESS:12345

Where UNICAST_ADDRESS is the unicast address on the interface used for the multicast network. For more information see Tech note How To Verify that Multicast Communication Works Correctly Between Machines the Coherence Cluster Members Are Running On (Doc ID 1936452.1) in the My Oracle Support web site.

If WebLogic is not used, and there is no equivalent multicast test script, then using a couple of small Java command-line applications can also be used for testing. For more information, see Tech note How to Test Whether Multicast is Enabled on the Network (Doc ID 413783.1) in the My Oracle Support web site.