Skip Headers
Oracle® Fusion Middleware Enterprise Deployment Guide for Oracle Identity Management (Oracle Fusion Applications Edition)
11g Release 7 (11.1.7)

Part Number E21032-21
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Index
Index
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
PDF · Mobi · ePub

16 Managing the Topology for an Enterprise Deployment

This chapter describes some operations that you can perform after you have set up the Identity Management topology. These operations include monitoring, scaling, backing up your topology, and troubleshooting.

This chapter includes the following topics:

16.1 Starting and Stopping Components

This section describes how to start, stop and restart the various components of the Oracle Enterprise Deployment for Identity Management.

This section contains the following topics:

16.1.1 Startup Order

When starting up your entire infrastructure, start the components in the following order, (ignoring those not in your topology):

  1. Database(s)

  2. Database Listener(s)

  3. Oracle Internet Directory

  4. Oracle Virtual Directory

  5. Node Manager

  6. Oracle Access Manager Server(s)

  7. WebLogic Administration Server

  8. Oracle HTTP Server(s)

  9. SOA Server(s)

  10. Oracle Identity Manager Server(s)

16.1.2 Starting and Stopping Servers

During provisioning, scripts were created in the SHARED_ROOT/config/scripts directory to start and stop all the servers in the environment. Two of the scripts are available for you to use from the command line to start and stop all Identity Management servers. The remaining scripts are used internally and must not be invoked from the command line.

Note:

These scripts do NOT stop or start the database.

16.1.2.1 Starting All Servers

Provisioning created a file called startall.sh, which is used to start all of the components on a particular server. To start everything in the correct order run the command on hosts in the following order:

  • LDAPHOST1

  • LDAPHOST2

  • IDMHOST1

  • IDMHOST2

  • WEBHOST1

  • WEBHOST2

If you want to start the services on a single host, execute the command on that host.

Before invoking this script, set JAVA_HOME to JAVA_HOME.

During exectution you will be prompted to enter the Weblogic and Node Manager administrator passwords.

The script starts the servers in the following order:

  1. Node Manager1

  2. AdminServer

  3. wls_ods1

  4. wls_soa1

  5. wls_oim1

  6. wls_oam1

  7. wls_oif1

  8. ohs1

  9. oid1

  10. oid2

  11. ohs2

  12. Node Manager 2

  13. wls_ods2

  14. wls_soa2

  15. wls_oim2

  16. wls_oam2

  17. wls_oif2

16.1.2.2 Stopping All Servers:

The script to stop all servers is stopall.sh.

Before invoking this script, set JAVA_HOME to JAVA_HOME.

During exectution you will be prompted to enter the Weblogic and Node Manager administrator passwords.

16.2 About Identity Management Console URLs

Table 16-1 lists the administration consoles used in this guide and their URLs.

Table 16-1 Console URLs

Domain Console URL

IDMDomain

WebLogic Administration Console

http://ADMIN.mycompany.com/console

IDMDomain

Enterprise Manager FMW Control

http://ADMIN.mycompany.com/em

IDMDomain

OAM Console

http://ADMIN.mycompany.com/oamconsole

IDMDomain

ODSM

http://ADMIN.mycompany.com/odsm


16.3 Monitoring Enterprise Deployments

This section provides information about monitoring the Identity Management enterprise deployment described in this manual.

This section contains the following topics:

16.3.1 Monitoring Oracle Internet Directory

You can use the Oracle Enterprise Manager Fusion Middleware Control to monitor Oracle Internet Directory, as follows:

  1. On the Farm home page for an Identity Management domain, view the Fusion Middleware pie chart. This chart shows the status of Oracle Fusion Middleware components. Green sections of the chart indicate components that are up and running properly, and red sections indicate components that are down.

  2. The Identity and Access section below the chart includes the name of each individual Oracle Internet Directory instance (for example: oid1, oid2), its status, host name, and CPU usage percentage. A green arrow in the Status column indicates that the instance is up and running properly, and a red arrow indicates the instance is down. Click the name of an individual Oracle Internet Directory instance to view the home page for that instance.

  3. The home page for an instance displays metrics for the instance such as performance, load, security, response, CPU utilization %, and memory utilization %.

16.3.1.1 Oracle Internet Directory Component Names Assigned by Oracle Identity Manager Installer

When you perform an Oracle Internet Directory installation using Oracle Identity Management 11g Installer, the default component name that the installer assigns to the Oracle Internet Directory instance is oid1. You cannot change this component name.

The instance specific configuration entry for this Oracle Internet Directory instance is cn=oid1, cn=osdldapd, cn=subconfigsubentry.

If you perform a second Oracle Internet Directory installation on another computer and that Oracle Internet Directory instance uses the same database as the first instance, the installer detects the previously installed Oracle Internet Directory instance on the other computer using the same Oracle database, so it gives the second Oracle Internet Directory instance a component name of oid2.

The instance specific configuration entry for the second Oracle Internet Directory instance is cn=oid2, cn=osdldapd, cn=subconfigsubentry. A change of properties in the entry cn=oid2, cn=osdldapd, cn=subconfigsubentry does not affect the first instance (oid1).

If a third Oracle Internet Directory installation is performed on another computer and that instance uses the same database as the first two instances, the installer gives the third Oracle Internet Directory instance a component name of oid3, and so on for additional instances on different hosts that use the same database.

Note that the shared configuration for all Oracle Internet Directory instances is cn=dsaconfig, cn=configsets,cn=oracle internet directory. A change in this entry affects all the instances of Oracle Internet Directory.

This naming scheme helps alleviate confusion when you view your domain using Oracle Enterprise Manager by giving different component names to your Oracle Internet Directory instances.

16.3.2 Monitoring Oracle Virtual Directory

You can use the Oracle Enterprise Manager Fusion Middleware Control to monitor Oracle Virtual Directory, as follows:

  1. On the Farm home page for an Identity Management domain, view the Fusion Middleware pie chart. This chart shows the status of Oracle Fusion Middleware components. Green sections of the chart indicate components that are up and running properly, and red sections indicate components that are down.

  2. The Identity and Access section below the chart includes the name of each instance of the Oracle Virtual Directory application (for example, ovd1, ovd2), its status, and host name. A green arrow in the Status column indicates that the Oracle Virtual Directory instance is up and running properly, and a red arrow indicates the instance is down. Click the name of an individual Oracle Virtual Directory instance to view the home page for that instance.

  3. The home page for an instance displays metrics and configurations for the instance such as:

    • Oracle Virtual Directory status - A green arrow next to the Oracle Virtual Directory instance name at the top of the page indicates that the instance is up and running properly and a red arrow indicates that the instance is down.

    • Current Load - This indicates the current work load of this Oracle Virtual Directory instance. It includes three metrics: Open Connections, Distinct Connected Users, and Distinct Connected IP Addresses.

    • Average Response Time Metric - This displays the average time (in milliseconds) to complete an LDAP search request.

    • Operations Metric - This displays the average number of LDAP search requests finished per millisecond.

    • Listeners - This table lists the listeners configured for this Oracle Virtual Directory instance to provide services to clients.

    • Adapters - This table lists existing adapters configured with the Oracle Virtual Directory instance. Oracle Virtual Directory uses adapters to connect to different underlying data repositories.

    • Resource Usage - On the right hand side of the page, the CPU and memory utilization metrics are displayed to indicate the system resources consumed by the Oracle Virtual Directory instance.

16.3.3 Monitoring WebLogic Managed Servers

You can use Oracle Enterprise Manager Fusion Middleware Control to monitor Managed Servers and other Fusion Middleware components, such as Oracle Access Manager, Oracle Identity Manager, Oracle Identity Federation, and SOA. For more information, see the administrator guides listed in the Preface under "Related Documents".

16.4 Auditing Identity Management

Oracle Fusion Middleware Audit Framework is a new service in Oracle Fusion Middleware 11g, designed to provide a centralized audit framework for the middleware family of products. The framework provides audit service for platform components such as Oracle Platform Security Services (OPSS) and Oracle Web Services. It also provides a framework for JavaEE applications, starting with Oracle's own JavaEE components. JavaEE applications are able to create application-specific audit events. For non-JavaEE Oracle components in the middleware such as C or JavaSE components, the audit framework also provides an end-to-end structure similar to that for JavaEE applications.

Figure 16-1 is a high-level architectural diagram of the Oracle Fusion Middleware Audit Framework. For more information, see Oracle Fusion Middleware Application Security Guide.

Figure 16-1 Audit Event Flow

Surrounding text describes Figure 16-1 .

The Oracle Fusion Middleware Audit Framework consists of the following key components:

For more introductory information for the Oracle Fusion Middleware Audit Framework, see the "Introduction to Oracle Fusion Middleware Audit Framework" chapter in the Oracle Fusion Middleware Application Security Guide.

For information on how to configure the repository for Oracle Fusion Middleware Audit Framework, see the "Configuring and Managing Auditing" chapter in the Oracle Fusion Middleware Application Security Guide.

The EDG topology does not include Oracle Fusion Middleware Audit Framework configuration. The ability to generate audit data to the bus-stop files and the configuration of the audit loader are available once the products are installed. The main consideration is the audit database repository where the audit data is stored. Because of the volume and the historical nature of the audit data, it is strongly recommended that customers use a separate database from the operational store or stores being used for other middleware components.

16.5 Performing Backups and Recoveries

You can use the UNIX tar command for most backups. Typical usage is:

tar -cvpf BACKUP_LOCATION/backup_file.tar directories

You can use the UNIX tar command for recovery. Typical usage is:

tar -xvf BACKUP_LOCATION/backup_file.tar 

For database backup and recovery, you can use the database utility RMAN. See the Oracle Database Backup and Recovery Reference for more information on using this command.

This section contains the following topics:

16.5.1 Peforming Baseline Backups

Perform baseline backups when building a system and when applying patches that update static artifacts, such as the Oracle binaries.

After performing a baseline backup, also perform a runtime backup.

Table 16-2 Static Artifacts to Back Up in the Identity Management Enterprise Deployment

Type Host Location Tier

Oracle Home (database)

Oracle RAC database hosts:

OIDDBHOST1

OIDDBHOST2

User Defined

Directory Tier

Oracle Identity and Access Management Binaries

IDMHOST1

IDMHOST2

Middleware Home: IAM_MW_HOME

Application Tier

Oracle Identity Management Binaries

IDMHOST1

IDMHOST2

Middleware Home: IDM_MW_HOME

Application Tier

Web Tier Binaries

WEBHOST1

WEBHOST2

Middleware Oracle home, WEB_ORACLE_HOME:

Web Tier

Install-Related Files

Each host

OraInventory:

ORACLE_BASE/orainventory

/etc/oratab, /etc/oraInst.loc

user_home/bea/beahomelist (on hosts where WebLogic Server is installed)

Not applicable.


Note:

It is also recommended that you back up your load balancer configuration. Refer to your vendor documentation on how to do this.

For more information on backup and recovery of Oracle Fusion Middleware components, see Oracle Fusion Middleware Administrator's Guide.

16.5.2 Performing Runtime Backups

Perform runtime backups on an ongoing basis. These backups contain information on items that can change frequently, such as data in the database, domain configuration information, and identity information in LDAP directories.

Table 16-3 Run-Time Artifacts to Back Up in the Identity Management Enterprise Deployments

Type Host Location Tier

Domain Home

IDMHOST1

IDMHOST2

Admin Server and Shared Files: ASERVER_HOME

Managed Servers: MSERVER_HOME

Application Tier

Application Artifacts (ear and war files)

IDMHOST1

IDMHOST2

Look at all the deployments, including Oracle Directory Services Manager, through the WebLogic Server Administration Console to identity all the application artifacts.

Application Tier

OID Instance Home

LDAPHOST1

LDAPHOST2

OID_ORACLE_INSTANCE

Directory Tier

OVD Instance Home

LDAPHOST1

LDAPHOST2

OVD_ORACLE_INSTANCE

Directory Tier

Oracle HTTP Server

WEBHOST1

WEBHOST2

WEB_ORACLE_INSTANCE

Web Tier

Oracle RAC Databases

OIDDBHOST1

OIDDBHOST2

User defined

Directory Tier

OAM

OAMHOST1

OAMHOST2

All the configurations are within the respective home directories described in this table. There are no instance homes.

Application Tier


16.5.3 Performing Backups During Installation and Configuration

It is an Oracle best practices recommendation to create a backup after successfully completing the installation and configuration of each tier, or at another logical point. Create a backup after verifying that the installation so far is successful. This is a quick backup for the express purpose of immediate restoration in case of problems in later steps. The backup destination is the local disk. You can discard this backup when the enterprise deployment setup is complete. After the enterprise deployment setup is complete, you can initiate the regular deployment-specific Backup and Recovery process.

For more details, see the Oracle Fusion Middleware Administrator's Guide.

For information on database backups, refer to the Oracle Database Backup and Recovery User's Guide.

This section contains the following topics:

16.5.3.1 Backing Up Middleware Home

Back up the Middleware homes whenever you create a new one or add components to it. The Middleware homes used in this guide are Oracle Identity Management and Oracle Identity and Access Management, as listed in Table 16-2.

16.5.3.2 Backing Up LDAP Directories

Whenever you perform an action which updates the data in LDAP, back up the directory contents.

This section contains the following topics:

16.5.3.2.1 Backing up Oracle Internet Directory

To back up an Oracle Internet Directory instance:

  1. Shut down the instance using opmnctl located under the OID_ORACLE_INSTANCE/bin directory:

    OID_ORACLE_INSTANCE/bin/opmnctl stopall
    
  2. Back up the Database hosting the Oracle Internet Directory data and the Oracle Internet Directory instance home on each host.

  3. Start up the instance using opmnctl located under the OID_ORACLE_INSTANCE/bin directory:

    OID_ORACLE_INSTANCE/bin/opmnctl startall
    
16.5.3.2.2 Backing up Oracle Virtual Directory

To back up an Oracle Virtual Directory instance:

  1. Shut down the instance using opmnctl located under the OVD_ORACLE_INSTANCE/bin directory:

    OVD_ORACLE_INSTANCE/bin/opmnctl stopall
    
  2. Back up the Oracle Virtual Directory Instance home on each LDAP host.

  3. Start up the instance using opmnctl located under the OVD_ORACLE_INSTANCE/bin directory:

    OVD_ORACLE_INSTANCE/bin/opmnctl startall
    
16.5.3.2.3 Backing Up Third-Party Directories

Refer to your operating system vendor's documentation for information about backing up directories.

16.5.3.3 Backing Up the Database

Whenever you create add a component to the configuration, back up the IDMDB database. Perform this backup after creating domains or adding components such as Access Manager or Oracle Identity Manager.

16.5.3.4 Backing Up the WebLogic Domain

To back up the WebLogic domain, perform these steps:

  1. Shut down the WebLogic administration server and any managed servers running in the domain as described in Section 16.1, "Starting and Stopping Components."

  2. Back up the ASERVER_HOME directory from shared storage.

  3. Back up the MSERVER_HOME directory from each host.

  4. Restart the WebLogic Administration Server and managed servers.

16.5.3.5 Backing Up the Web Tier

To back up the Web Tier, perform these steps:

  1. Shut down the Oracle HTTP Server as described in Section 16.1, "Starting and Stopping Components."

  2. Back up the Oracle HTTP Server.

  3. Start the Oracle HTTP Server as described in Section 16.1, "Starting and Stopping Components."

16.6 Patching Enterprise Deployments

It is recommended that you patch enterprise deployments by using the IDM Patching Framework provided as part of this release.

The topology data used by the tools is located in the topology store, topology.xml. This file is generated by the Oracle Identity Management Provisioning Wizard and contains most of the environment details used by the tools to apply patches. The Patching Framework uses this file to determine how to apply patches.

In topology.xml, wo instance groups were created during provisioning. An instance group contains a list of components which are required to keep a system available.

For example, in a typical enterprise deployment, two instance groups are created:

The process of applying patches can be summarized as follows:

  1. Create a patch top. A patch top directory contains patches, classified by each product to which patches apply.

  2. Run Patch Manager to generate a patch plan. The plan, based on how the Manager is run, holds steps which apply to the entire deployment topology, or to just a specified instance group.

  3. Run the Patcher against all hosts which are affected by the plan. You might need to execute the Patcher on a given host multiple times if required by a given plan. As each Patcher invocation completes, it directs you where to run the Patcher next.

When Patcher runs, it stops and starts components as necessary, and ensures that patches are applied in the correct order to satisfy dependencies.

To keep downtime to a minimum, whenever possible, use Patcher to apply patches to each instance group in turn. Doing this ensures that you need not stop the entire deployment during patching.

Full details on how to use the IDM Patching Framework can be found in the chapter "Patching Oracle Identity Management Artifacts" in Oracle Fusion Applications Patching Guide.

This section contains the following topics:

16.6.1 Patching an Oracle Fusion Middleware Source File

For information on patching an Oracle Fusion Middleware source file, see the Oracle Fusion Middleware Administrator's Guide.

16.6.2 Patching Identity Management Components

To patch Oracle Identity Management components with minimal down time, it is recommended that you follow these guidelines:

  1. Route LDAP traffic to LDAPHOST2 only. Same steps for other servers in instance group 1.

  2. Create a patch plan for instance group 1 and execute it.

  3. Route all LBR traffic to instance group 1 servers only.

  4. Verify applications are working properly.

  5. Create a patch plan for instance group 2 and execute it.

  6. Route all Load Balancer traffic to instance group 2 servers only.

  7. Verify applications are working properly.

  8. Route all traffic to servers of both instance groups.

16.7 Preventing Timeouts for SQL

Most of the production deployment involves firewalls. Because database connections are made across firewalls, Oracle recommends that the firewall be configured so that the database connection is not timed out. For Oracle Real Application Clusters (Oracle RAC), the database connections are made on Oracle RAC VIPs and the database listener port. You must configure the firewall so it does not time out these connections. If such a configuration is not possible, set the SQLNET.EXPIRE_TIME=n parameter in the ORACLE_HOME/network/admin/sqlnet.ora file on the database server, where n is the time in minutes. Set this value to less than the known value of the timeout for the network device (that is, a firewall). For Oracle RAC, set this parameter in all of the Oracle home directories.

16.8 Manually Failing Over the WebLogic Administration Server

This section discusses how to fail over the Administration Server to IDMHOST2 and how to fail it back to IDMHOST1.

This section contains the following topics:

16.8.1 Failing Over the Administration Server to IDMHOST2

If a node fails, you can fail over the Administration Server to another node. This section describes how to fail over the Administration Server from IDMHOST1 to IDMHOST2.

Assumptions:

  • The Administration Server is configured to listen on ADMINVHN.mycompany.com, and not on ANY address.

  • The Administration Server is failed over from IDMHOST1 to IDMHOST2, and the two nodes have these IP addresses:

    • IDMHOST1: 100.200.140.165

    • IDMHOST2: 100.200.140.205

    • ADMINVIP: 100.200.140.206

      This is the Virtual IP address where the Administration Server is running, assigned to interface:index (for example, eth1:2), available in IDMHOST1 and IDMHOST2.

  • The domain directory where the Administration Server is running in IDMHOST1 is on a shared storage and is mounted also from IDMHOST2.

    Note:

    NM in IDMHOST2 does not control the domain at this point, since unpack/nmEnroll has not been run yet on IDMHOST2. But for the purpose of AdminServer failover and control of the AdminServer itself, Node Manager is fully functional

  • Oracle WebLogic Server and Oracle Fusion Middleware Components have been installed inIDMHOST2 as described in previous chapters. That is, the same path for IDM_ORACLE_HOME and MW_HOME that exists in IDMHOST1 is available in IDMHOST2.

The following procedure shows how to fail over the Administration Server to a different node, IDMHOST2.

  1. Stop the Administration Server as described in Section 16.1, "Starting and Stopping Components."

  2. Migrate the IP address to the second node.

    1. Run the following command as root on IDMHOST1 (where x:y is the current interface used by ADMINVHN.mycompany.com):

      /sbin/ifconfig x:y down
      

      For example:

      /sbin/ifconfig eth0:1 down
      
    2. Run the following command on IDMHOST2:

      /sbin/ifconfig interface:index IP_Address netmask netmask
      

      For example:

      /sbin/ifconfig eth0:1 100.200.140.206 netmask 255.255.255.0
      

    Note:

    Ensure that the netmask and interface to be used match the available network configuration in IDMHOST2.

  3. Update routing tables by using arping, for example:

    /sbin/arping -q -U -c 3 -I eth0 100.200.140.206
    

16.8.2 Starting the Administration Server on IDMHOST2

Perform the following steps to start Node Manager on IDMHOST2.

  1. On IDMHOST2, mount the Administration Server domain directory if it is not already mounted. For example:

    mount /u01/oracle
    
  2. Start Node Manager by using the following commands:

    cd WL_HOME/server/bin
    ./startNodeManager.sh
    
  3. Stop the Node Manager by killing the Node Manager process.

    Note:

    Starting and stopping Node Manager at this point is only necessary the first time you run Node Manager. Starting and stopping it creates a property file from a predefined template. The next step adds properties to that property file.

  4. Run the setNMProps.sh script to set the StartScriptEnabled property to true before starting Node Manager:

    cd ORACLE_COMMON_HOME/common/bin
    ./setNMProps.sh
    

    Note:

    You must use the StartScriptEnabled property to avoid class loading failures and other problems.

  5. Start the Node Manager as described in Section 16.1, "Starting and Stopping Components."

  6. Start the Administration Server on IDMHOST2.

    cd ORACLE_COMMON_HOME/common/bin
    ./wlst.sh
    

    Once in the WLST shell, execute the following commands:

    nmConnect('admin','Admin_Password', 'IDMHOST2','5556', 'IDMDomain','/u1/oracle/config/domains/IDMDomain')
    nmStart('AdminServer')
    
  7. Test that you can access the Administration Server on IDMHOST2 as follows:

    1. Ensure that you can access the Oracle WebLogic Server Administration Console at:

      http://ADMINVHN.mycompany.com:7001/console.

    2. Check that you can access and verify the status of components in the Oracle Enterprise Manager at: http://ADMINVHN.mycompany.com:7001/em.

16.8.3 Validating Access to IDMHOST2 Through Oracle HTTP Server

Perform the same steps as in Section 14.1.1, "Verify Connectivity" This is to check that you can access the Administration Server when it is running on IDMHOST2.

16.8.4 Failing the Administration Server Back to IDMHOST1

This step checks that you can fail back the Administration Server, that is, stop it on IDMHOST2 and run it on IDMHOST1. To do this, migrate ADMINVHN back to IDMHOST1 node as described in the following steps.

  1. Ensure that the Administration Server is not running. If it is, stop it from the WebLogic console, or by running the command stopWeblogic.sh from ASERVER_HOME/bin.

  2. On IDMHOST2, unmount the Administration server domain directory. For example:

    umount /u01/oracle
    
  3. On IDMHOST1, mount the Administration server domain directory. For example:

    mount /u01/oracle
    
  4. Disable the ADMINVHN.mycompany.com virtual IP address on IDMHOST2 and run the following command as root on IDMHOST2:

    /sbin/ifconfig x:y down
    

    where x:y is the current interface used by ADMINVHN.mycompany.com.

  5. Run the following command on IDMHOST1:

    /sbin/ifconfig interface:index 100.200.140.206 netmask 255.255.255.0
    

    Note:

    Ensure that the netmask and interface to be used match the available network configuration in IDMHOST1

  6. Update routing tables by using arping. Run the following command from IDMHOST1.

    /sbin/arping -q -U -c 3 -I interface 100.200.140.206
    
  7. If Node Manager is not already started on IDMHOST1, start it, as described in Section 16.1, "Starting and Stopping Components."

  8. Start the Administration Server again on IDMHOST1.

    cd ORACLE_COMMON_HOME/common/bin
    ./wlst.sh
    

    Once in the WLST shell, execute

    nmConnect(admin,'Admin_Pasword, IDMHOST1,'5556',
         'IDMDomain','/u01/oracle/config/domains/IDMDomain'
    nmStart('AdminServer')
    
  9. Test that you can access the Oracle WebLogic Server Administration Console at:

    http://ADMINVHN.mycompany.com:7001/console

    where 7001 is WLS_ADMIN_PORT in Section 3.7, "Fixed Ports Used by the Provisioning Wizard."

  10. Check that you can access and verify the status of components in the Oracle Enterprise Manager at:

    http://ADMINVHN.mycompany.com:7001/em

16.9 Changing Startup Location

When the environment was provisioned, start and stop scripts were generated to start and stop components in the topology. At the time of provisioning, the Admin server was configured to start on IDMHOST1. If you want to permanently change this to start on IDMHOST2, perform the following steps.

Edit the file serverInstancesInfo.txt, which is located in the directory: SHARED_CONFIG_DIR/scripts

Locate the line which looks like this:

IDMHOST1.mycompany.com AS AdminServer

Change IDMHOST1 to IDMHOST2 and save the file.

16.10 Troubleshooting

This section describes how to troubleshoot common issues that can arise with the Identity Management enterprise deployment described in this manual.

This section contains the following topics:

16.10.1 Troubleshooting Identity Management Provisioning

This section describes some common problems related to provisioning. It contains the following topics:

16.10.1.1 Provisioning Fails

Problem

Provisioning fails.

Solution

Check the provisioning logs located in the directory:

SHARED_ROOT/config/provisioning/logs/hostname

where hostname is the host where the provisioning step failed.

16.10.1.2 OID Account is Locked

Problem

Investigation into the OID logs shows that the OID account is locked.

This is generally caused by the load balancer. The load balancer is continually polling OID to see if it is available using the given credentials. During setup, this can cause the account to become locked.

Solution

Disable the OID load balancer monitor during preconfiguration. Then enable it when provisioning is complete. Another alternative is to reduce the check frequency.

16.10.2 Troubleshooting Start/Stop Scripts

Problem

Problem: Start/Stop scripts fail to start or stop a managed server.

The start/stop logs in the directory SHARED_CONFIG_DIR/scripts/logs contain an error similar to this:

weblogic.utils.AssertionError: ***** ASSERTION FAILED *****
        at weblogic.server.ServerLifeCycleRuntime.getStateRemote(ServerLifeCycleRuntime.java:734)
        at weblogic.server.ServerLifeCycleRuntime.getState(ServerLifeCycleRuntime.java:581)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 

Solution

  1. Shut down the failing managed server. You might have to kill the process.

  2. Back up the managed server's LDAP data, then remove it. For example:

    rm –rf LOCAL_CONFIG_DIR/domains/IDMDomain/servers/server_name/data/ldap
    

    where server_name is the name of the failing managed server.

  3. Restart the managed server

16.10.3 Troubleshooting Oracle Internet Directory

This section describes some common problems that can arise with Oracle Internet Directory and the actions you can take to resolve the problem. It contains the following topics:

16.10.3.1 Oracle Internet Directory Server is Not Responsive.

Problem

The Oracle Internet Directory server is not responsive. When the load balancing router is configured to send an ICMP message to the LDAP SSL port for monitoring, the Oracle Internet Directory server starting SSL negotiation sometimes hangs, and thus it is required that the load balancing router not use ICMP messages for monitoring the LDAP SSL port.

Solution

Use an alternative such as TCP or the LDAP protocol itself.

Also, monitoring the LDAP non-SSL port is sufficient to detect LDAP availability.

16.10.3.2 SSO/LDAP Application Connection Times Out

Problem

The SSO/LDAP Application connection is lost to Oracle Internet Directory server

Solution

Verify the load balancing router timeout and SSO/Application timeout configuration parameter. The SSO/LDAP application timeout value should be less than LBR IDLE time out.

16.10.3.3 LDAP Application Receives LDAP Error 53 (DSA Unwilling to Perform)

Problem

The LDAP application is receiving LDAP Error 53 (DSA Unwilling to Perform). When one of the database nodes goes down during the middle of the LDAP transaction, the Oracle Internet Directory server sends error 53 to the LDAP client

Solution

To see why the Oracle Internet Directory database node went down, see the Oracle Internet Directory logs in this location:

ORACLE_INSTANCE/diagnostics/logs/OID/oidldapd01s*.log

16.10.3.4 TNSNAMES.ORA, TAF Configuration, and Related Issues

Problem

Issues involving TNSNAMES.ORA, TAF configuration, and related issues.

Solution

See the Oracle Database High Availability Overview manual.

16.10.4 Troubleshooting Oracle Virtual Directory

This section describes some common problems that can arise with Oracle Virtual Directory and the actions you can take to resolve the problem. It contains the following topics:

16.10.4.1 Command Not Found Error When Running SSLServerConfig.sh

Problem

You get a command not found error when you run SSLServerConfig.sh, for example:

./SSLServerConfig.sh: line 169: 20110520125611: command not found 

Solution

Edit the file orapki.sh and remove any blank lines at the end of the file. Save the file and run SSLServerConfig.sh again.

16.10.4.2 Oracle Virtual Directory is Not Responsive

Problem

Oracle Virtual Directory is not responsive. When the load balancing router is configured to send an ICMP message to the LDAP SSL port for monitoring, the Oracle Virtual Directory server starting SSL negotiation sometimes hangs, and thus it is required that the load balancing router not use ICMP messages for monitoring the LDAP SSL port.

Solution

Use an alternative such as TCP or the LDAP protocol itself.

Also, monitoring the LDAP non-SSL port is sufficient to detect LDAP availability.

16.10.4.3 SSO/LDAP Application Connection Times Out

Problem

The SSO/LDAP Application connection is lost to the Oracle Virtual Directory server.

Solution

Verify the load balancing router timeout and SSO/Application timeout configuration parameter. The SSO/LDAP application timeout value should be less than LBR IDLE time out.

16.10.4.4 TNSNAMES.ORA, TAF Configuration, and Related Issues

Problem

Issues involving TNSNAMES.ORA, TAF configuration, and related issues.

Solution

See the Oracle Database High Availability Overview manual.

16.10.4.5 SSLServerConfig.sh Fails with Error

Problem

When you run SSLServerConfig.sh for component OVD, sometime it fails with an error similar to this:

>>>Enter password for weblogic:
>>>Enter your keystore name [ovdks1.jks]:
Checking the existence of ovdks1.jks in the OVD...

>>>Failed to configure your SSL server wallet
>>>Please check /scratch/aime1/edgfa/idm//rootCA/keystores/ovd/ks_check.log for more information

In the log file, you see an error message like this:

Problem invoking WLST - Traceback (innermost last):
File "/scratch/aime1/edgfa/idm/rootCA/keystores/ovd/ovdssl-check.py", line 8, in ?
File "<iostream>", line 182, in cd
File "<iostream>", line 1848, in raiseWLSTExceptionWLSTException: Error occured while performing cd : Attributeoracle.as.ovd:type=component.listenersconfig.sslconfig,name=LDAP SSLEndpoint,instance=ovd1,component=ovd1 not found. Use ls(a) to view theattributes

Solution

The problem is intermittent. To work around the issue, re-run the script.

16.10.5 Troubleshooting Oracle Directory Services Manager

This section describes some common problems that can arise with Oracle Directory Services Manager and the actions you can take to resolve the problem. It contains the following topics:

16.10.5.1 ODSM Browser Window and Session Issues

After you have logged into Oracle Directory Services Manager, you can connect to multiple directory instances from the same browser window.

Avoid using multiple windows of the same browser program to connect to different directories at the same time. Doing so can cause a Target unreachable error.

You can log in to the same Oracle Directory Services Manager instance from different browser programs, such as Internet Explorer and Firefox, and connect each to a different directory instance.

If you change the browser language setting, you must update the session to use the new setting. To update the session, either disconnect the current server connection, refresh the browser page (either reenter the Oracle Directory Services Manager URL in the URL filed and press enter or press F5) and reconnect to the same server, or quit and restart the browser.

16.10.5.2 ODSM Does not Open When Invoked from Fusion Middleware Control

Problem

Oracle Directory Services Manager does not open after you attempt to invoke it from Oracle Enterprise Manager Fusion Middleware Control by selecting one of the options from the Directory Services Manager entry in the Oracle Virtual Directory menu in the Oracle Virtual Directory target or the Oracle Internet Directory menu in the Oracle Internet Directory target.

Solution

This is probably an installation problem. See the Oracle Fusion Middleware Installation Guide for Oracle Identity Management.

16.10.5.3 ODSM Failover is Not Transparent

Problem

When you perform an Oracle Directory Services Manager failover using Oracle HTTP Server, the failover is not transparent. You see this behavior when you perform the following steps:

  1. Oracle Directory Services Manager is deployed in a High Availability active-active configuration using Oracle HTTP Server.

  2. Display an Oracle Directory Services Manager page using the Oracle HTTP Server name and port number.

  3. Make a connection to an Oracle Internet Directory or Oracle Virtual Directory server.

  4. Work with the Oracle Internet Directory or Oracle Virtual Directory server using the current Oracle Directory Services Manager Oracle HTTP Server host and port.

  5. Shut down one Managed Server at a time using the WebLogic Server Administration Console.

  6. Go back to the Oracle Directory Services Manager page and port, and the connection which was established earlier with Oracle Internet Directory or Oracle Virtual Directory. When you do, a message is displayed advising you to re-establish a new connection to the Oracle Directory Services Manager page.

Solution

If you encounter this problem, perform the following steps:

  1. In your web browser, exit the current Oracle Directory Services Manager page.

  2. Launch a new web browser page and specify the same Oracle Directory Services Manager Oracle HTTP Server name and port.

  3. Re-establish a new connection to the Oracle Internet Directory or Oracle Virtual Directory server you were working with earlier.

16.10.5.4 ODSM Loses Connection and Displays Message that LDAP Server is Down

Problem

Oracle Directory Services Manager temporarily loses its connection to Oracle Internet Directory and displays the message LDAP Server is down.

Solution

In a High Availability configuration where Oracle Directory Services Manager is connected to Oracle Internet Directory through a load balancer, Oracle Directory Services Manager reports that the server is down during failover from one instance of Oracle Internet Directory to another. In other configurations, this message might indicate that Oracle Internet Directory has been shut down and restarted. In either case, the connection is reestablished in less than a minute, and you are able to continue without logging in again.

16.10.5.5 ODSM Loses Connection to Instance Using ORAC Database

Problem

Oracle Directory Services Manager temporarily loses its connection to an Oracle Internet Directory or Oracle Virtual Directory instance that is using an Oracle RAC Database. Oracle Directory Services Manager might display the message LDAP error code 53 - Function not implemented.

Solution

This error can occur during failover of the Oracle Database that the Oracle Internet Directory or Oracle Virtual Directory instance is using. The connection is reestablished in less than a minute, and you are able to continue without logging in again.

16.10.5.6 OHS Must Be Configured to Route ODSM Requests to Multiple Oracle WebLogic Servers

Problem

You must perform the following steps to configure Oracle HTTP Server to route Oracle Directory Services Manager requests to multiple Oracle WebLogic Servers in a clustered Oracle WebLogic Server environment.

Solution

Perform these steps:

  1. Create a backup copy of the Oracle HTTP Server's admin.conf file, which is located in ORACLE_INSTANCE/config. The backup copy provides a source to revert to if you encounter problems after performing this procedure.

  2. Add the following text to the end of the Oracle HTTP Server's admin.conf file and replace the variable placeholder values with the host names and Managed Server port numbers specific to your environment. Be sure to use the <Location /odsm/ > as the first line in the entry. Using <Location /odsm/faces > or <Location /odsm/faces/odsm.jspx > can distort the appearance of the Oracle Directory Services Manager interface.

    <Location /odsm/ >
       SetHandler weblogic-handler
       WebLogicCluster host-name-1:managed-server-port,host-name_2:managed_server_port
    </Location>
    
  3. Restart the Oracle HTTP Server, as described in Section 16.1, "Starting and Stopping Components," to activate the configuration change.

    Note:

    Oracle Directory Services Manager loses its connection and displays a session time-out message if the Oracle WebLogic Server in the cluster that it is connected to fails. Oracle Directory Services Manager requests is routed to the secondary Oracle WebLogic Server in the cluster that you identified in the httpd.conf file after you log back in to Oracle Directory Services Manager.

16.10.5.7 ODSM is Not Accessible

Problem

Attempting to access Oracle Directory Services Manager using a web browser fails.

Solution

  • Verify the Oracle Virtual Directory server is running. The Oracle Virtual Directory server must be running to connect to it from Oracle Directory Services Manager.

  • Verify you entered the correct credentials in the Server, Port, User Name and Password fields. You can execute an ldapbind command against the target Oracle Virtual Directory server to verify the server, user name, and password credentials.

  • Verify you are using a supported browser. Oracle Directory Services Manager supports the following browsers:

    • Internet Explorer 7

    • Firefox 2.0.0.2 and 3.0

    • Safari 3.1.2 (desktop)

    • Google Chrome 0.2.149.30

      Note:

      While Oracle Directory Services Manager supports all of the preceding browsers, only Internet Explorer 7 and Firefox 2.0.0.2 are certified.

16.10.6 Troubleshooting Oracle Access Manager 11g

This section describes some common problems that can arise with Oracle Access Manager and the actions you can take to resolve the problem. It contains the following topics:

16.10.6.1 OAM Fails to Connect to the Identity Store at First Start

Problem

After IDM Provisioning is complete, attempts to log in to consoles fail. The OAM log contains messages similar to this:

could not be initialized due to
oracle.security.am.engines.common.identity.provider.exceptions.IdentityProvide
rException: OAMSSA-20005: Error initializing User/Role API : null..> 

Solution

The startup script uses opmnctl startall, which starts OID and OVD at the same time. In some cases, OVD attempts to connect with OID before OID is ready, resulting in a failed connection. As a workaround, perform the following steps on LDAPHOST1:

  1. Stop all opmnctl processes.

    OID_ORACLE_INSTANCE/bin/opmnctl stopall
    
  2. Start OPMN.

    OID_ORACLE_INSTANCE/bin/opmnctl start
    
  3. Start each OID instance. For example:

    OID_ORACLE_INSTANCE/bin/opmnctl startproc ias-component=oid1
    OID_ORACLE_INSTANCE/bin/opmnctl startproc ias-component=oid2
    
  4. Wait one minute.

  5. Start OVD

    OID_ORACLE_INSTANCE/bin/opmnctl startproc ias-component=ovd1
    

Repeat Steps 1-5 on LDAPHOST2.

Restart all WebLogic servers.

16.10.6.2 OAM Runs out of Memory

Problem

After Oracle Access Manager has been running for a while, you see the following error message in the output:

Attempting to allocate 1G bytes
There is insufficient native memory for the Java Runtime Environment to continue. 

Possible reasons:

  • The system is out of physical RAM or swap space.

  • In 32 bit mode, the process size limit was reached.

Solutions

  • Reduce memory load on the system.

  • Increase physical memory or swap space.

  • Check if swap backing store is full.

  • Use 64 bit Java on a 64 bit OS.

  • Decrease Java heap size (-Xmx/-Xms).

  • Decrease number of Java threads.

  • Decrease Java thread stack sizes (-Xss).

  • Disable compressed references (-XXcompressedRefs=false).

  • Ensure that command line tool adrci can be executed from the command line.

    • at oracle.dfw.impl.incident.ADRHelper.invoke(ADRHelper.java:1309)

    • at oracle.dfw.impl.incident.ADRHelper.createIncident(ADRHelper.java:929

    • at oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl.createADRIncident(DiagnosticsDataExtractorImpl.java:1116)

  • On both IDMHOST1 and IDMHOST2, edit the file setSOADomainEnv.sh, which is located in MSERVER_HOME/bin and locate the line which begins:

    PORT_MEM_ARGS=
    

    Change this line so that it reads:

    PORT_MEM_ARGS="-Xms768m -Xmx2560m"
    

16.10.6.3 Fusion Applications Preverify Fails to Validate OAM Admin Users

Problem

The Fusion Applications preverify step, described in the task "Run the pre-verify phase" in Oracle Fusion Applications Enterprise Deployment Guide for Customer Relationship Management, fails to validate OAM Admin users. In the OAM diagnostic file, you see an error similar to this:

Caused by: oracle.security.idm.OperationFailureException:@ oracle.security.am.common.jndi.ldap.PoolingException [Root exception is@ oracle.ucp.UniversalConnectionPoolException: Invalid life cycle state. Check@ the status of the Universal Connection Pool]

Solution

  1. Shut down the administration server and all managed servers in the domain, as described in Section 16.1, "Starting and Stopping Components."

  2. Delete the files with names of the form:

    /tmp/UCP*

  3. Restart the administration server and managed servers.

16.10.6.4 User Reaches the Maximum Allowed Number of Sessions

Problem

The Oracle Access Manager server displays an error message similar to this:

The user has already reached the maximum allowed number of sessions. Please close one of the existing sessions before trying to login again.

Solution

If users log in multiple times without logging out, they might overshoot the maximum number of configured sessions. You can modify the maximum number of configured sessions by using the OAM Administration Console.

To modify the configuration by using the OAM Administration Console, proceed as follows:

  1. Go to System Configuration -> Common Settings -> Session

  2. Increase the value in the Maximum Number of Sessions per User field to cover all concurrent login sessions expected for any user. The range of values for this field is from 1 to any number.

16.10.6.5 Policies Do Not Get Created When Oracle Access Manager is First Installed

Problem

The Administration Server takes a long time to start after configuring Oracle Access Manager.

Solution

Tune the OAM database. When the Administration server first starts after configuring Oracle Access Manager, it creates a number of default policies in the database. If the database is distant or in need of tuning, this can take a significant amount of time.

Resources
Authentication Policies
   Protected Higher Level Policy
   Protected Lower Level Policy
   Publicl Policy
Authorization Policies
   Authorization Policies

If you do not see these items, the initial population has failed. Check the Administration Server log file for details.

16.10.6.6 You Are Not Prompted for Credentials After Accessing a Protected Resource

Problem

When you access a protected resource, Oracle Access Manager should prompt you for your user name and password. For example, after creating a simple HTML page and adding it as a resource, you should see credential entry screen.

Solution

If you do not see the credential entry screen, perform the following steps:

  1. Verify that Host Aliases for IDMDomain have been set. You should have aliases for IDMDomain:80, IDMDomain:Null, ADMIN.mycompany.com:80, and SSO.mycompany.com:443, where Port 80 is HTTP_PORT and Port 443 is HTTP_SSL_PORT.

  2. Verify that WebGate is installed.

  3. Verify that OBAccessClient.xml was copied from ASERVER_HOME/output to the WebGate Lib directory and that OHS was restarted.

  4. When OBAccessClient.xml was first created, the file was not formatted. When the OHS is restarted, reexamine the file to ensure that it is now formatted. OHS gets a new version of the file from Oracle Access Manager when it first starts.

  5. Shut down the Oracle Access Manager servers and try to access the protected resource. You should see an error saying Oracle Access Manager servers are not available. If you do not see this error, re-install WebGate.

16.10.6.7 Cannot Log In to OAM Console

Problem

You cannot log in to the OAM Console. The Administration Server diagnostic log might contain an error message similar to this:

Caused by: oracle.security.idm.OperationFailureException:
oracle.security.am.common.jndi.ldap.PoolingException [Root exception is oracle.ucp.UniversalConnectionPoolException:
Invalid life cycle state.
 Check the status of the Universal Connection Pool]
         at
oracle.security.idm.providers.stdldap.UCPool.acquireConnection(UCPool.java:112)

Solution

Remove the /tmp/UCP* files and restart the Administration Server.

16.10.7 Troubleshooting Oracle Identity Manager

This section describes some common problems that can arise with Oracle Identity Manager and the actions you can take to resolve the problem. It contains the following topics:

16.10.7.1 java.io.FileNotFoundException When Running Oracle Identity Manager Configuration

Problem

When you run Oracle Identity Manager configuration, the error java.io.FileNotFoundException: soaconfigplan.xml (Permission denied) may appear and Oracle Identity Manager configuration might fail.

Solution

To workaround this issue:

  1. Delete the file /tmp/oaconfigplan.xml.

  2. Start the configuration again (OH/bin/config.sh).

16.10.7.2 ResourceConnectionValidationxception When Creating User in Oracle Identity Manager

Problem

If you are creating a user in Oracle Identity Manager (by logging into Oracle Identity Manager, clicking the Administration tab, clicking the Create User link, entering the required information in the fields, and clicking Save) in an active-active Oracle Identity Manager configuration, and the Oracle Identity Manager server that is handling the request fails, you may see a "ResourceConnectionValidationxception" in the Oracle Identity Manager log file, similar to:

[2010-06-14T15:14:48.738-07:00] [oim_server2] [ERROR] [] [XELLERATE.SERVER]
[tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default
(self-tuning)'] [userId: xelsysadm] [ecid:
004YGJGmYrtEkJV6u3M6UH00073A0005EI,0:1] [APP: oim#11.1.1.3.0] [dcid:
12eb0f9c6e8796f4:-785b18b3:12938857792:-7ffd-0000000000000037] [URI:
/admin/faces/pages/Admin.jspx] Class/Method:
PooledResourceConnection/heartbeat encounter some problems: Operation timed
out[[
com.oracle.oim.gcp.exceptions.ResourceConnectionValidationxception: Operation
timed out
        at
oracle.iam.ldapsync.impl.repository.LDAPConnection.heartbeat(LDAPConnection.ja
va:162)
        at
com.oracle.oim.gcp.ucp.PooledResourceConnection.heartbeat(PooledResourceConnec
tion.java:52)
         .
         .
         .

Solution

Despite this exception, the user is created correctly.

16.10.8 Troubleshooting Oracle SOA Suite

This section describes some common problems that can arise with Oracle SOA Suite and the actions you can take to resolve the problem. It contains the following topics:

16.10.8.1 Transaction Timeout Error

Problem: The following transaction timeout error appears in the log:

Internal Exception: java.sql.SQLException: Unexpected exception while enlisting
 XAConnection java.sql.SQLException: XA error: XAResource.XAER_NOTA start()
failed on resource 'SOADataSource_soaedg_domain': XAER_NOTA : The XID
is not valid

Solution: Check your transaction timeout settings, and be sure that the JTA transaction time out is less than the DataSource XA Transaction Timeout, which is less than the distributed_lock_timeout (at the database).

With the out of the box configuration, the SOA data sources do not set XA timeout to any value. The Set XA Transaction Timeout configuration parameter is unchecked in the WebLogic Server Administration Console. In this case, the data sources use the domain level JTA timeout which is set to 30. Also, the default distributed_lock_timeout value for the database is 60. As a result, the SOA configuration works correctly for any system where transactions are expected to have lower life expectancy than such values. Adjust these values according to the transaction times your specific operations are expected to take.

16.10.9 Troubleshooting Oracle Identity Federation

This section describes some common problems that can arise with Oracle Identity Federation and the actions you can take to resolve the problem. It contains the following topics:

16.10.9.1 Extending the Domain with Oracle Identity Federation Fails

Problem

Extending the domain with Oracle Identity Federation fails when Oracle Identity Manager is installed at the Create Managed Server step.

Solution

Copy the file setDomainEnv.sh from ASERVER_HOME/bin on IDMHOST1 to OIFHOST1.

Retry the operation.

16.10.9.2 Cannot Change Oracle Identity Federation Parameters by Using Fusion Middleware Control

Problem

You cannot change Oracle Identity Federation parameters by using Oracle Enterprise Manager Fusion Middleware Control. You see the message:

Configuration settings are unavailable because ..... OIF ...... is down

even though Oracle Identity Federation is up and running.

Solution

Here are the common causes and resolutions:

  1. Oracle Identity Federation is up but the EM agent is down.

    1. Check the EM agent status by running:

      ORACLE_INSTANCE/EMAGENT/EMAGENT/bin/emctl status agent
      
    2. Start the EM agent, if it is down, by running:

      ORACLE_INSTANCE/bin/opmnctl startproc ias-component=EMAGENT
      
    3. Log in to Fusion Middleware Control again.

  2. Oracle Identity Federation and EM agent are up, but the OIF home page and configuration pages in Fusion Middleware Control still show: OIF is down.

    1. Check if the EM agent points to the correct Fusion Middleware Control by running:

      ORACLE_INSTANCE/EMAGENT/EMAGENT/bin/emctl status agent
      

      Verify that the host and port for property Repository URL are the same as the Fusion Middleware Control's host and port.

    2. If the host and port are mismatched, change the Repository URL in EM agent to the correct Fusion Middleware Control by running:

      ORACLE_INSTANCE/EMAGENT/EMAGENT/bin/emctl switchOMS  http(s)://Host:Port/em/upload'
      
    3. Log in to Fusion Middleware Control again.

  3. If the issue still exists, once logged in to Fusion Middleware Control, navigate to Farm->Agent-Monitored Targets (Top Left corner of the page) and click the Configure icon of the row that refers to Oracle Identity Federation. On the next page, ensure that all the information is correct and complete. Click OK to confirm.

    Check that the WebLogic user name and password are present.

    Check the host value. It might have been specified with an IPv6 address format.

  4. If the issue still exists, restart the EM agent.

    1. Stop the EM agent by running:

      INST_HOME/bin/opmnctl stopproc ias-component=EMAGENT
      
    2. Start the EM agent by running:

      INST_HOME/bin/opmnctl startproc ias-component=EMAGENT
      
    3. Log in to Oracle Enterprise Manager Fusion Middleware Control again.