15 Managing the Topology for an Enterprise Deployment

This chapter describes some operations that you can perform after you have set up the Identity and Access Management topology. These operations include monitoring, scaling, backing up your topology, and troubleshooting.

This chapter includes the following topics:

15.1 Starting and Stopping Components

This section describes how to start, stop and restart the various components of the Oracle Enterprise Deployment.

This section contains the following topics:

15.1.1 Startup Order

When starting up your entire infrastructure, start the components in the following order, (ignoring those not in your topology):

  1. Database(s)

  2. Database Listener(s)

  3. LDAP hosts

  4. OAM hosts

  5. OIM hosts

  6. Web hosts

  7. Oracle Identity Manager Managed servers

  8. Identity Access Domain WebLogic Administration Server

  9. Oracle Access Management Access Manager Server(s)

  10. OAAM Administration Server

  11. OAAM Managed Server (if OAAM is part of the topology)

  12. Oracle HTTP Server(s)

15.1.2 Starting and Stopping All Servers by Using a Script

During Deployment, scripts were created in the SHARED_CONFIG_DIR/config/scripts directory to start and stop all the servers in the environment. Two of the scripts are available for you to use from the command line to start and stop all Identity and Access Management servers. The remaining scripts are used internally and must not be invoked from the command line.

Note:

These scripts do NOT stop or start the database.

15.1.2.1 Starting All Servers

Deployment created a file called startall.sh, which is used to start all of the components on a particular server. To start everything in the correct order run the command on hosts in the following order:

Consolidated Topology

  • LDAPHOST1

  • LDAPHOST2

  • IAMHOST1

  • IAMHOST2

  • WEBHOST1

  • WEBHOST2

Distributed Topology

  • LDAPHOST1

  • LDAPHOST2

  • OAMHOST1

  • OIMHOST1

  • OAMHOST2

  • OIMHOST2

  • WEBHOST1

  • WEBHOST2

If you want to start the services on a single host, execute the command on that host.

During exectution you will be prompted to enter the Weblogic and Node Manager administrator passwords.

The script starts the components which are installed on a given host in the following order. What is started depends on what is installed on the host on which the script is running:

  1. Oracle Unified Directory

  2. Node Manager

  3. Administration Server(s)

  4. SOA Managed Server

  5. OIM Managed Server

  6. OAM Managed Server

  7. Oracle HTTP Server

  8. OAAM Managed Server

15.1.2.2 Stopping All Servers:

The script to stop all servers is stopall.sh.

Run the command on hosts in the reverse of the order used to start all servers.

During exectution you will be prompted to enter the Weblogic and Node Manager administrator passwords.

15.1.3 Manually Starting and Stopping Identity and Access Management Components

Start and Stop individual Identity and Access Management components as described in the following subsections:

15.1.3.1 Starting and Stopping Oracle Unified Directory

Start and stop Oracle Unified Directory as follows:

15.1.3.1.1 Starting Oracle Unified Directory

To start Oracle Unified Directory issue the following command:

OUD_ORACLE_INSTANCE/OUD/bin/start-ds
15.1.3.1.2 Stopping Oracle Unified Directory

To stop Oracle Unified Directory issue the command:

OUD_ORACLE_INSTANCE/OUD/bin/stop-ds

15.1.3.2 Starting an Oracle Access Manager Managed Servers When None is Running

Normally, you start Access Manager managed servers by using the WebLogic console. After you have enabled Single Sign-On for the administration consoles, however, you must have at least one Access Manager Server running in order to access a console. If no Access Manager server is running, you can start one by using WLST.

To invoke WLST on Linux or UNIX, type:

cd ORACLE_COMMON_HOME/common/bin
./wlst.sh

Once you are in the WLST shell, execute the following commands:

nmConnect('Admin_User','Admin_Password', 'OAMHOST','Port', 'domain_name','IAD_MSERVER_HOME')
nmStart('wls_oam1')

where Port is NMGR_PORT, domain_name is the name of the domain and Admin_User and Admin_Password are the Node Manager username and password. For example:

nmConnect('weblogic','password', 'OAMHOST1','5556', 'IAMAccessDomain','IAD_MSERVER_HOME')

If an Access Manager Managed server is already running then you can start the Access Manager Managed server as you would any other web logic managed server via the WebLogic Administration console.

15.1.3.3 Starting and Stopping a WebLogic Administration Server

Start and stop a WebLogic Administration Server as described in the following sections.

Notes:

  • Admin_User and Admin_Password are only used to authenticate connections between Node Manager and clients. They are independent from the server administration ID and password and are stored in the file: IAD_ASERVER_HOME/config/nodemanager/nm_password.properties

  • If you are starting the IAMAccessDomain Administration server, ASERVER_HOME is IAD_ASERVER_HOME. If you are starting the IAMGovernanceDomain Administration server, ASERVER_HOME is IGD_ASERVER_HOME

15.1.3.3.1 Starting a WebLogic Administration Server

The recommended way to start the Administration server is to use WLST and connect to Node Manager:

cd ORACLE_COMMON_HOME/common/bin
./wlst.sh

Where ORACLE_COMMON_HOME is from the MW_HOME associated with the domain you are starting or stopping.

To start the Administration Server in the Access Domain, use the following command:

nmConnect('Admin_User','Admin_Password','IADADMINVHN','5556', 'IAMAccessDomain','IAD_ASERVER_HOME')
nmStart('AdminServer')

For example:

nmConnect('Admin_User','Admin_Password','IADADMINVHN','5556', 'IAMAccessDomain','/u01/oracle/config/domains/IAMAccessDomain')
nmStart('AdminServer')

Execute the following command to start the Administration Server in the Governance Domain:

nmConnect('Admin_User','Admin_Password','IGDADMINVHN','5556', 'IAMGovernanceDomain','IGD_ASERVER_HOME')
nmStart('AdminServer')

For example:

nmConnect('weblogic','password', 'IADADMINVHN','5556', 'IAMAccessDomain','IAD_MSERVER_HOME')

Alternatively, you can start the Administration server by using the command:

ASERVER_HOME/bin/startWebLogic.sh
15.1.3.3.2 Stopping a WebLogic Administration Server

To stop the Administration Server, log in to the WebLogic console using the URL listed in Chapter 15, "About Identity and Access Management Console URLs."

Then proceed as follows:

  1. Click the Control tab.

  2. Select AdminServer(admin).

  3. Click Shutdown and select Force Shutdown now.

  4. Click Yes when asked to confirm that you want to shut down the Administration Server.

15.1.3.4 Starting and Stopping WebLogic Managed Servers

Start and stop managed servers as follows.

15.1.3.4.1 Starting WebLogic Managed Servers

To start a managed server, log in to the WebLogic console using the URL listed in Chapter 15, "About Identity and Access Management Console URLs."

Then proceed as follows:

  1. Click the Control tab.

  2. Select Environment -> Servers from the Domain Structure menu.

  3. Select managed server for example wls_oim1

  4. Click the Start button.

  5. Click Yes when asked to confirm that you want to start the server(s).

15.1.3.4.2 Stopping WebLogic Managed Servers

To stop a Managed Server(s), log in to the WebLogic console using the URL listed in Chapter 15, "About Identity and Access Management Console URLs." Then proceed as follows:

  1. Select Environment -> Servers from the Domain Structure menu.

  2. Click the Control tab.

  3. Select Managed Server for example wls_oim1

  4. Click the Shutdown button and select Force Shutdown Now.

  5. Click Yes when asked to confirm that you want to shut down the server(s).

15.1.3.5 Starting and Stopping Node Manager

Start and stop Node Manager as follows.

15.1.3.5.1 Starting Node Manager

If the Node Manager being started is the one that controls the Administration Server, then prior to starting the Node Manager issue the command:

export JAVA_OPTIONS=-DDomainRegistrationEnabled=true

To start Node Manager, issue the commands:

cd SHARED_CONFIG_DIR/nodemanager/hostname
./startNodeManagerWrapper.sh
15.1.3.5.2 Stopping Node Manager

To stop Node Manager, kill the process started in the previous section.

15.2 About Identity and Access Management Console URLs

Table 15-1 lists the administration consoles used in this guide and their URLs.

Table 15-1 Console URLs

Domain Console URL User Name

IAMAccessDomain

WebLogic Administration Console

http://IADADMIN.mycompany.com/console

weblogic_idm
 

Enterprise Manager FMW Control

http://IADADMIN.mycompany.com/em

weblogic_idm
 

Access Management console

http://IADADMIN.mycompany.com/oamconsole

oamadmin
 

OAAM Server

https://SSO.mycompany.com/oaam_server

n/a
 

OAAM Administration Console

http://IADADMIN.mycompany.com/oaam_admin

oaamadmin

IAMGovernanceDomain

WebLogic Administration Console

http://IGDADMIN.mycompany.com/console

weblogic_idm
 

Enterprise Manager FMW Control

http://IGDADMIN.mycompany.com/em

weblogic_idm
 

Identity Manager System Administration Console

http://IGDADMIN.mycompany.com/sysadmin

xelsysadm
 

Identity Manager Self Service Console

https://SSO.mycompany.com/identity

xelsysadm
 

Authorization Policy Manager

http://IGDADMIN.mycompany.com/apm

oamadmin

15.3 Monitoring Enterprise Deployments

This section provides information about monitoring the Identity and Access Management enterprise deployment described in this manual.

This section contains the following topics:

15.3.1 Monitoring Oracle Unified Directory

You can check the status of Oracle Unified Directory by issuing the command:

OUD_ORACLE_INSTANCE/OUD/bin/status

This command accesses the locally running Oracle Unified Directory instance and reports the status of the directory, including whether or not replication and LDAP or LDAPS is enabled.

15.3.2 Monitoring WebLogic Managed Servers

You can use Oracle Enterprise Manager Fusion Middleware Control to monitor Managed Servers and other Fusion Middleware components, such as Access Manager, Oracle Identity Manager, Oracle Identity Federation, and SOA. For more information, see the administrator guides listed in the Preface under "Related Documents".

15.4 Auditing Identity and Access Management

Oracle Fusion Middleware Audit Framework is a new service in Oracle Fusion Middleware 11g, designed to provide a centralized audit framework for the middleware family of products. The framework provides audit service for platform components such as Oracle Platform Security Services (OPSS) and Oracle Web Services. It also provides a framework for JavaEE applications, starting with Oracle's own JavaEE components. JavaEE applications are able to create application-specific audit events. For non-JavaEE Oracle components in the middleware such as C or JavaSE components, the audit framework also provides an end-to-end structure similar to that for JavaEE applications.

Figure 15-1 is a high-level architectural diagram of the Oracle Fusion Middleware Audit Framework. For more information, see Oracle Fusion Middleware Application Security Guide.

Figure 15-1 Audit Event Flow

Surrounding text describes Figure 15-1 .

The Oracle Fusion Middleware Audit Framework consists of the following key components:

  • Audit APIs

    These are APIs provided by the audit framework for any audit-aware components integrating with the Oracle Fusion Middleware Audit Framework. During run-time, applications may call these APIs where appropriate to audit the necessary information about a particular event happening in the application code. The interface enables applications to specify event details such as username and other attributes needed to provide the context of the event being audited.

  • Audit Events and Configuration

    The Oracle Fusion Middleware Audit Framework provides a set of generic events for convenient mapping to application audit events. Some of these include common events such as authentication. The framework also enables applications to define application-specific events.

    These event definitions and configurations are implemented as part of the audit service in Oracle Platform Security Services. Configurations can be updated through Enterprise Manager (UI) and WLST (command-line tool).

  • The Audit Bus-stop

    Bus-stops are local files containing audit data before they are pushed to the audit repository. In the event where no database repository is configured, these bus-stop files can be used as a file-based audit repository. The bus-stop files are simple text files that can be queried easily to look up specific audit events. When a DB-based repository is in place, the bus-stop acts as an intermediary between the component and the audit repository. The local files are periodically uploaded to the audit repository based on a configurable time interval.

  • Audit Loader

    As the name implies, audit loader loads the files from the audit bus-stop into the audit repository. In the case of platform and JavaEE application audit, the audit loader is started as part of the JavaEE container start-up. In the case of system components, the audit loader is a periodically spawned process.

  • Audit Repository

    Audit Repository contains a pre-defined Oracle Fusion Middleware Audit Framework schema, created by Repository Creation Utility (RCU). Once configured, all the audit loaders are aware of the repository and upload data to it periodically. The audit data in the audit repository is expected to be cumulative and grow over time. Ideally, this should not be an operational database used by any other applications - rather, it should be a standalone RDBMS used for audit purposes only. In a highly available configuration, Oracle recommends that you use an Oracle Real Application Clusters (Oracle RAC) database as the audit data store.

  • Oracle Business Intelligence Publisher

    The data in the audit repository is exposed through pre-defined reports in Oracle Business Intelligence Publisher. The reports enable users to drill down the audit data based on various criteria. For example:

    • Username

    • Time Range

    • Application Type

    • Execution Context Identifier (ECID)

For more introductory information for the Oracle Fusion Middleware Audit Framework, see the "Introduction to Oracle Fusion Middleware Audit Framework" chapter in the Oracle Fusion Middleware Application Security Guide.

For information on how to configure the repository for Oracle Fusion Middleware Audit Framework, see the "Configuring and Managing Auditing" chapter in the Oracle Fusion Middleware Application Security Guide.

The EDG topology does not include Oracle Fusion Middleware Audit Framework configuration. The ability to generate audit data to the bus-stop files and the configuration of the audit loader are available once the products are installed. The main consideration is the audit database repository where the audit data is stored. Because of the volume and the historical nature of the audit data, it is strongly recommended that customers use a separate database from the operational store or stores being used for other middleware components.

15.5 Performing Backups and Recoveries

You can use the UNIX tar command for most backups. Typical usage is:

tar -czvpsPf BACKUP_LOCATION/backup_file.tar directories

You can use the UNIX tar command for recovery. Typical usage is:

tar -xzvpsPf BACKUP_LOCATION/backup_file.tar

For database backup and recovery, you can use the database utility RMAN. See the Oracle Database Backup and Recovery Reference for more information on using this command.

This section contains the following topics:

15.5.1 Peforming Baseline Backups

Perform baseline backups when building a system and when applying patches that update static artifacts, such as the Oracle binaries.

After performing a baseline backup, also perform a runtime backup.

Table 15-2 Static Artifacts to Back Up in the Identity and Access Management Enterprise Deployment

Type Host Location Tier

Oracle Home (database)

Oracle RAC database hosts:

IAMDBHOST1

IAMDBHOST2

User Defined

Database

Oracle Unified Directory Binaries

LDAPHOST1

LDAPHOST2

Middleware Home: DIR_MW_HOME

Directory Tier

Oracle Access Management Binaries

OAMHOST1

OAMHOST2

Middleware Home: IAD_MW_HOME

Application Tier

Oracle Identity Governance Binaries

OIMHOST1

OIMHOST2

Middleware Home: IGD_MW_HOME

Application Tier

Web Tier Binaries

WEBHOST1

WEBHOST2

Middleware Oracle home, WEB_ORACLE_HOME:

Web Tier

Install-Related Files

Each host

OraInventory:

ORACLE_BASE/oraInventory

/etc/oratab, /etc/oraInst.loc

~/bea/beahomelist (on hosts where WebLogic Server is installed)

Not applicable.

Note:

It is also recommended that you back up your load balancer configuration. Refer to your vendor documentation on how to do this.

For more information on backup and recovery of Oracle Fusion Middleware components, see Oracle Fusion Middleware Administrator's Guide.

15.5.2 Performing Runtime Backups

Perform runtime backups on an ongoing basis. These backups contain information on items that can change frequently, such as data in the database, domain configuration information, and identity information in LDAP directories.

Table 15-3 Run-Time Artifacts to Back Up in the Identity and Access Management Enterprise Deployments

Type Host Location Tier

IAMAccessDomain Home

OAMHOST1

OAMHOST2

Administration Server and Shared Files: IAD_ASERVER_HOME

Managed Servers: IAD_MSERVER_HOME

Application Tier

IAMGovernanceDomain Home

OIMHOST1

OIMHOST2

Administration Server and Shared Files: IGD_ASERVER_HOME

Managed Servers: IGD_MSERVER_HOME

Application Tier

Oracle HTTP Server

WEBHOST1

WEBHOST2

WEB_ORACLE_INSTANCE

Web Tier

Oracle RAC Databases

IAMDBHOST1

IAMDBHOST2

User defined

Directory Tier

Oracle Unified Directory

LDAPHOST1

LDAPHOST2

OUD_ORACLE_INSTANCE

Application Tier

15.5.3 Performing Backups During Installation and Configuration

It is an Oracle best practices recommendation to create a backup after successfully completing the installation and configuration of each tier, or at another logical point. Create a backup after verifying that the installation so far is successful. This is a quick backup for the express purpose of immediate restoration in case of problems in later steps. The backup destination is the local disk. You can discard this backup when the enterprise deployment setup is complete. After the enterprise deployment setup is complete, you can initiate the regular deployment-specific Backup and Recovery process.

For more details, see the Oracle Fusion Middleware Administrator's Guide.

For information on database backups, refer to the Oracle Database Backup and Recovery User's Guide.

This section contains the following topics:

15.5.3.1 Backing Up Middleware Home

Back up the Middleware homes whenever you create a new one or add components to it. The Middleware homes used in this guide are Oracle Identity Management and Oracle Identity and Access Management, as listed in Table 15-2.

15.5.3.2 Backing Up LDAP Directories

Whenever you perform an action which updates the data in LDAP, back up the directory contents.

This section contains the following topics:

15.5.3.2.1 Backing Up Oracle Unified Directory

To backup Oracle Unified Directory, perform the following steps:

  1. Shut down the Oracle Unified Directory Instances as described in Section 15.1, "Starting and Stopping Components."

  2. Back up OUD_ORACLE_INSTANCE directories on each host.

  3. Restart the Oracle Unified Directory instances as described in Section 15.1, "Starting and Stopping Components."

15.5.3.2.2 Backing Up Third-Party Directories

Refer to your operating system vendor's documentation for information about backing up directories.

15.5.3.3 Backing Up the Database

Whenever you create add a component to the configuration, back up the IAMDB database. Perform this backup after creating domains or adding components such as Oracle Access Management Access Manager or Oracle Identity Manager.

15.5.3.4 Backing Up the WebLogic Domain IAMGovernanceDomain

To back up the WebLogic domain, perform these steps:

  1. Shut down the WebLogic administration server and any managed servers running in the domain as described in Section 15.1, "Starting and Stopping Components."

  2. Back up the IGD_ASERVER_HOME directory from shared storage.

  3. Back up the IGD_MSERVER_HOME directory from each host.

  4. Restart the WebLogic Administration Server and managed servers.

15.5.3.5 Backing Up the WebLogic Domain IAMAccessDomain

To back up the WebLogic domain, perform these steps:

  1. Shut down the WebLogic administration server and any managed servers running in the domain as described in Section 15.1, "Starting and Stopping Components."

  2. Back up the IAD_ASERVER_HOME directory from shared storage.

  3. Back up the IAD_MSERVER_HOME directory from each host.

  4. Restart the WebLogic Administration Server and managed servers.

15.5.3.6 Backing Up the Web Tier

To back up the Web Tier, perform these steps:

15.5.3.6.1 Backing Up Oracle HTTP Server

Back up Oracle HTTP Server as follows:

  1. Shut down the Oracle HTTP Server as described in Section 15.1, "Starting and Stopping Components."

  2. Back up the WEB_ORACLE_INSTANCE directory on local storage.

  3. Start the Oracle HTTP Server as described in Section 15.1, "Starting and Stopping Components."

15.6 Patching Enterprise Deployments

It is recommended that you patch enterprise deployments by using the automated patching solution included with the Identity and Access Management Lifecycle Tools.

The process of applying patches can be summarized as follows:

  1. Create a patch top. A patch top directory contains patches, classified by each product to which patches apply.

  2. Run Patch Manager to generate a patch plan. Based on the deployment topology and patches provided, the Manager creates an optimal plan to apply those patches.

  3. Run the Patcher against all hosts which are affected by the plan. You might need to execute the Patcher on a given host multiple times if required by a given plan. As each Patcher invocation completes, it directs you where to run the Patcher next.

When the Patcher runs, it stops and starts server instances as necessary, and ensures that patches are applied in the correct order to satisfy dependencies.

Full details on how to use the IDM Patching Framework can be found in Oracle Fusion Middleware Patching Guide for Oracle Identity and Access Management. The Guide also contains instructions for patching the deployment manually if required, using the OPatch tool.

15.7 Preventing Timeouts for SQL

Most of the production deployment involves firewalls. Because database connections are made across firewalls, Oracle recommends that the firewall be configured so that the database connection is not timed out. For Oracle Real Application Clusters (Oracle RAC), the database connections are made on Oracle RAC VIPs and the database listener port. You must configure the firewall so it does not time out these connections. If such a configuration is not possible, set the SQLNET.EXPIRE_TIME=n parameter in the ORACLE_HOME/network/admin/sqlnet.ora file on the database server, where n is the time in minutes. Set this value to less than the known value of the timeout for the network device (that is, a firewall). For Oracle RAC, set this parameter in all of the Oracle home directories.

15.8 Manually Failing Over the WebLogic Administration Server

This section discusses how to fail over the Administration Server to a new host after the primary host fails. The example in this section shows how to fail the Access Management Administration Server from OAMHOST1 to OAMHOST2. If you are failing over the Oracle Identity Manager Administration server, substitute the appropriate values for that domain.

This section contains the following topics:

15.8.1 Failing Over the Administration Server to OAMHOST2

If a node fails, you can fail over the Administration Server to another node. This section describes how to fail over the Administration Server from OAMHOST1 to OAMHOST2.

Assumptions:

  • The Administration Server is configured to listen on IADADMINVHN.mycompany.com, and not on ANY address.

  • The Administration Server is failed over from OAMHOST1 to OAMHOST2, and the two nodes have these IP addresses:

    • OAMHOST1: 100.200.140.165

    • OAMHOST2: 100.200.140.205

    • IADADMINVHN: 100.200.140.206

      This is the Virtual IP address where the Administration Server is running, assigned to interface:index (for example, eth1:2), available in OAMHOST1 and OAMHOST2.

  • The domain directory where the Administration Server is running in OAMHOST1 is on a shared storage and is mounted also from OAMHOST2.

    Note:

    NM in OAMHOST2 does not control the domain at this point, since unpack/nmEnroll has not been run yet on OAMHOST2. But for the purpose of AdminServer failover and control of the AdminServer itself, Node Manager is fully functional

  • Oracle WebLogic Server and Oracle Fusion Middleware Components have been installed in OAMHOST2 as described in previous chapters. That is, the same path for IAD_ORACLE_HOME and IAD_MW_HOME that exists in OAMHOST1 is available in OAMHOST2.

The following procedure shows how to fail over the Administration Server to a different node, OAMHOST2.

  1. Stop the Administration Server on OAMHOST1 as described in Section 15.1, "Starting and Stopping Components."

  2. Migrate the IP address to the second node.

    1. Run the following command as root on OAMHOST1 (where x:y is the current interface used by IADADMINVHN.mycompany.com):

      /sbin/ifconfig x:y down

      For example:

      /sbin/ifconfig eth0:1 down

    2. Run the following command on OAMHOST2:

      /sbin/ifconfig interface:index IP_Address netmask netmask

      For example:

      /sbin/ifconfig eth0:1 100.200.140.206 netmask 255.255.255.0

    Note:

    Ensure that the netmask and interface to be used match the available network configuration in OAMHOST2.

  3. Update routing tables by using arping on OAMHOST2, for example:

    /sbin/arping -q -U -c 3 -I eth0 100.200.140.206

15.8.2 Starting the Administration Server on OAMHOST2

Perform the following steps to start Node Manager on OAMHOST2.

  1. On OAMHOST2, mount the Administration Server domain directory if it is not already mounted. For example:

    mount /u01/oracle

  2. Start Node Manager by using the following commands:

    cd WL_HOME/server/bin
./startNodeManager.sh

  3. Stop the Node Manager by killing the Node Manager process.

    Note:

    Starting and stopping Node Manager at this point is only necessary the first time you run Node Manager. Starting and stopping it creates a property file from a predefined template. The next step adds properties to that property file.

  4. Run the setNMProps.sh script to set the StartScriptEnabled property to true before starting Node Manager:

    cd ORACLE_COMMON_HOME/common/bin
./setNMProps.sh

    Note:

    You must use the StartScriptEnabled property to avoid class loading failures and other problems.

  5. Start the Node Manager as described in Section 15.1, "Starting and Stopping Components."

  6. Start the Administration Server on OAMHOST2.

    cd ORACLE_COMMON_HOME/common/bin
./wlst.sh

    Once in the WLST shell, execute the following commands:

    nmConnect('admin','Admin_Password', 'OAMHOST2','5556', 'IAMAccessDomain','/u1/oracle/config/domains/IAMAccessDomain')
nmStart('AdminServer')

  7. Test that you can access the Administration Server on OAMHOST2 as follows:

    1. Ensure that you can access the Oracle WebLogic Server Administration Console at:

      http://IADADMINVHN.mycompany.com/console.

    2. Check that you can access and verify the status of components in the Oracle Enterprise Manager at: http://IADADMINVHN.mycompany.com/em.

15.8.3 Validating Access to OAMHOST2 Through Oracle HTTP Server

Perform the same steps as in Section 11.1.1, "Verify Connectivity" This is to check that you can access the Administration Server when it is running on OAMHOST2.

15.8.4 Failing the Administration Server Back to OAMHOST1

This step checks that you can fail back the Administration Server, that is, stop it on OAMHOST2 and run it on OAMHOST1. To do this, migrate IADADMINVHN back to OAMHOST1 node as described in the following steps.

  1. Ensure that the Administration Server is not running on OAMHOST2. If it is, stop it from the WebLogic console, or by running the command stopWeblogic.sh from IAD_ASERVER_HOME/bin.

  2. On OAMHOST2, unmount the Administration server domain directory. For example:

    umount /u01/oracle

  3. On OAMHOST1, mount the Administration server domain directory. For example:

    mount /u01/oracle

  4. Disable the IADADMINVHN.mycompany.com virtual IP address on OAMHOST2 and run the following command as root on OAMHOST2:

    /sbin/ifconfig x:y down

    where x:y is the current interface used by IADADMINVHN.mycompany.com.

  5. Run the following command on OAMHOST1:

    /sbin/ifconfig interface:index 100.200.140.206 netmask 255.255.255.0

    Note:

    Ensure that the netmask and interface to be used match the available network configuration in OAMHOST1

  6. Update routing tables by using arping. Run the following command from OAMHOST1.

    /sbin/arping -q -U -c 3 -I interface 100.200.140.206

  7. If Node Manager is not already started on OAMHOST1, start it, as described in Section 15.1, "Starting and Stopping Components."

  8. Start the Administration Server again on OAMHOST1.

    cd ORACLE_COMMON_HOME/common/bin
./wlst.sh

    Once in the WLST shell, execute

    nmConnect(admin,'Admin_Pasword, OAMHOST1,'5556',
     'IAMAccessDomain','/u01/oracle/config/domains/IAMAccessDomain'
nmStart('AdminServer')

  9. Test that you can access the Oracle WebLogic Server Administration Console at:

    http://IADADMINVHN.mycompany.com:7001/console

    where 7001 is WLS_ADMIN_PORT in Section 7.1, "Assembling Information for Identity and Access Management Deployment."

  10. Check that you can access and verify the status of components in the Oracle Enterprise Manager at:

    http://IADADMIN.mycompany.com/em

15.9 Changing Startup Location

When the environment was deployed, start and stop scripts were generated to start and stop components in the topology. At the time of Deployment, the Access Domain Administration server was configured to start on OAMHOST1. If you want to permanently change this to start on OAMHOST2, perform the following steps.

Use the same steps, changing the name of the server and host, to change the Governance Domain Administration server to start on OIMHOST2 instead of OIMHOST1.

Edit the file serverInstancesInfo.txt, which is located in the directory: SHARED_CONFIG_DIR/scripts

Locate the line which looks like this:

OAMHOST1.mycompany.com AS AdminServer

Change OAMHOST1 to OAMHOST2 and save the file.

15.10 Troubleshooting

This section describes how to troubleshoot common issues that can arise with the Identity and Access Management enterprise deployment described in this manual.

This section contains the following topics:

15.10.1 Troubleshooting Identity and Access Management Deployment

This section describes some common problems related to Deployment. It contains the following topics:

15.10.1.1 Deployment Fails with Error: Incorrect Host or Domain Name Format for Attribute

Problem

Deployment fails with an error similar to this:

Incorrect host format for attibute : PRIMARY_OAM_SERVERS : server-123.mycompany.com

Due to a bug, one of the tools invoked during the deployment process cannot handle host names or domain names containing the hyphen (-) character.

Solution

Use host names and domain names that do NOT contain the hyphen (-) character.

15.10.1.2 Deployment Fails

Problem

Deployment fails.

Solution

Check the Deployment logs located in the directory:

LCM_HOME/provisioning/logs/hostname

where hostname is the host where the Deployment step failed.

15.10.2 Troubleshooting Start/Stop Scripts

This section describes some common problems related to Start/Stop scripts. It contains the following topics:

15.10.2.1 Preverify Inappropriately Fails with Insufficient Space

Problem

When preverify runs, it checks that sufficient space is available in the directory IDM_TOP. If you have created separate mount points for IDM_TOP/products and IDM_TOP/config, preverify does not add together the space allocated to the two mount points and fails the check inappropriately.

Solution

Disable the free space check by editing the file:

LCM_HOME/provisioning/idm-provisioning-build/idm-common-preverify-build.xml

Locate the entry:

    <target name="common-preverify-tasks">

Comment out the following entry so that after editing it looks like this:

            <!--antcall target="private-preverify-free-space"/-->

Save the file.

15.10.2.2 Start/Stop Scripts Fail to Start or Stop a Managed Server

Problem

Problem: Start/Stop scripts fail to start or stop a managed server.

The start/stop logs in the directory SHARED_CONFIG_DIR/scripts/logs contain an error similar to this:

weblogic.utils.AssertionError: ***** ASSERTION FAILED *****
        at weblogic.server.ServerLifeCycleRuntime.getStateRemote(ServerLifeCycleRuntime.java:734)
        at weblogic.server.ServerLifeCycleRuntime.getState(ServerLifeCycleRuntime.java:581)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Solution

  1. Shut down the failing managed server. You might have to kill the process.

  2. Back up the managed server's LDAP data, then remove it. For example:

    rm –rf LOCAL_CONFIG_DIR/domains/IAMAccessDomain/servers/server_name/data/ldap

    where server_name is the name of the failing managed server.

  3. Restart the managed server.

15.10.3 Troubleshooting Oracle Oracle Access Management Access Manager 11g

This section describes some common problems that can arise with Access Manager and the actions you can take to resolve the problem. It contains the following topics:

15.10.3.1 Access Manager Runs out of Memory

Problem

After Access Manager has been running for a while, you see the following error message in the output:

Attempting to allocate 1G bytes
There is insufficient native memory for the Java Runtime Environment to continue.

Possible reasons:

  • The system is out of physical RAM or swap space.

  • In 32 bit mode, the process size limit was reached.

Solutions

  • Reduce memory load on the system.

  • Increase physical memory or swap space.

  • Check if swap backing store is full.

  • Use 64 bit Java on a 64 bit OS.

  • Decrease Java heap size (-Xmx/-Xms).

  • Decrease number of Java threads.

  • Decrease Java thread stack sizes (-Xss).

  • Disable compressed references (-XXcompressedRefs=false).

  • Ensure that command line tool adrci can be executed from the command line.

    • at oracle.dfw.impl.incident.ADRHelper.invoke(ADRHelper.java:1309)

    • at oracle.dfw.impl.incident.ADRHelper.createIncident(ADRHelper.java:929

    • at oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl.createADRIncident(DiagnosticsDataExtractorImpl.java:1116)

  • On both OAMHOST1 and OAMHOST2, edit the file setSOADomainEnv.sh, which is located in IAD_MSERVER_HOME/bin and locate the line which begins:

    PORT_MEM_ARGS=

    Change this line so that it reads:

    PORT_MEM_ARGS="-Xms768m -Xmx2560m"

15.10.3.2 User Reaches the Maximum Allowed Number of Sessions

Problem

The Access Manager server displays an error message similar to this:

The user has already reached the maximum allowed number of sessions. Please close one of the existing sessions before trying to login again.

Solution

If users log in multiple times without logging out, they might overshoot the maximum number of configured sessions. You can modify the maximum number of configured sessions by using the Access Management Administration Console.

To modify the configuration by using the Access Management Administration Console, proceed as follows:

  1. Go to System Configuration -> Common Settings -> Session

  2. Increase the value in the Maximum Number of Sessions per User field to cover all concurrent login sessions expected for any user. The range of values for this field is from 1 to any number.

15.10.3.3 Policies Do Not Get Created When Oracle Access Management Access Manager is First Installed

Problem

The Administration Server takes a long time to start after configuring Access Manager.

Solution

Tune the Access Manager database. When the Administration server first starts after configuring Access Manager, it creates a number of default policies in the database. If the database is distant or in need of tuning, this can take a significant amount of time.

Resources
Authentication Policies
   Protected Higher Level Policy
   Protected Lower Level Policy
   Publicl Policy
Authorization Policies
   Authorization Policies

If you do not see these items, the initial population has failed. Check the Administration Server log file for details.

15.10.3.4 You Are Not Prompted for Credentials After Accessing a Protected Resource

Problem

When you access a protected resource, Access Manager should prompt you for your user name and password. For example, after creating a simple HTML page and adding it as a resource, you should see credential entry screen.

Solution

If you do not see the credential entry screen, perform the following steps:

  1. Verify that Host Aliases for IAMAccessDomain have been set. You should have aliases for IAMAccessDomain:80, IAMAccessDomain:Null, IADADMIN.mycompany.com:80, and SSO.mycompany.com:443, where Port 80 is HTTP_PORT and Port 443 is HTTP_SSL_PORT.

  2. Verify that WebGate is installed.

  3. Verify that ObAccessClient.xml was copied from IAD_ASERVER_HOME/output to the WebGate Lib directory and that OHS was restarted.

  4. When ObAccessClient.xml was first created, the file was not formatted. When the OHS is restarted, reexamine the file to ensure that it is now formatted. OHS gets a new version of the file from Access Manager when it first starts.

  5. Shut down the Access Manager servers and try to access the protected resource. You should see an error saying Access Manager servers are not available. If you do not see this error, re-install WebGate.

15.10.3.5 Cannot Log In to Access Management Console

Problem

You cannot log in to the Access Management Console. The Administration Server diagnostic log might contain an error message similar to this:

Caused by: oracle.security.idm.OperationFailureException:
oracle.security.am.common.jndi.ldap.PoolingException [Root exception is oracle.ucp.UniversalConnectionPoolException:
Invalid life cycle state.
 Check the status of the Universal Connection Pool]
         at
oracle.security.idm.providers.stdldap.UCPool.acquireConnection(UCPool.java:112)

Solution

Remove the /tmp/UCP* files and restart the Administration Server.

15.10.4 Troubleshooting Oracle Identity Manager

This section describes some common problems that can arise with Oracle Identity Manager and the actions you can take to resolve the problem. It contains the following topics:

15.10.4.1 java.io.FileNotFoundException When Running Oracle Identity Manager Configuration

Problem

When you run Oracle Identity Manager configuration, the error java.io.FileNotFoundException: soaconfigplan.xml (Permission denied) may appear and Oracle Identity Manager configuration might fail.

Solution

To workaround this issue:

  1. Delete the file /tmp/soaconfigplan.xml.

  2. Start the configuration again (OH/bin/config.sh).

15.10.4.2 ResourceConnectionValidationxception When Creating User in Oracle Identity Manager

Problem

If you are creating a user in Oracle Identity Manager (by logging into Oracle Identity Manager System Administration Console, clicking the Administration tab, clicking the Create User link, entering the required information in the fields, and clicking Save) in an active-active Oracle Identity Manager configuration, and the Oracle Identity Manager server that is handling the request fails, you may see a "ResourceConnectionValidationxception" in the Oracle Identity Manager log file, similar to:

[2010-06-14T15:14:48.738-07:00] [oim_server2] [ERROR] [] [XELLERATE.SERVER]
[tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default
(self-tuning)'] [userId: xelsysadm] [ecid:
004YGJGmYrtEkJV6u3M6UH00073A0005EI,0:1] [APP: oim#11.1.1.3.0] [dcid:
12eb0f9c6e8796f4:-785b18b3:12938857792:-7ffd-0000000000000037] [URI:
/admin/faces/pages/Admin.jspx] Class/Method:
PooledResourceConnection/heartbeat encounter some problems: Operation timed
out[[
com.oracle.oim.gcp.exceptions.ResourceConnectionValidationxception: Operation
timed out
        at
oracle.iam.ldapsync.impl.repository.LDAPConnection.heartbeat(LDAPConnection.ja
va:162)
        at
com.oracle.oim.gcp.ucp.PooledResourceConnection.heartbeat(PooledResourceConnec
tion.java:52)
         .
         .
         .

Solution

Despite this exception, the user is created correctly.

15.10.4.3 Oracle Identity Manager Reconciliation Jobs Fail

Problem

Oracle Identity Manager reconciliation jobs fail, or the following message is seen in the log files:

LDAP Error 53 : [LDAP: error code 53 - Full resync required. Reason: The provided cookie is older than the start of historical in the server for the replicated domain : dc=mycompany,dc=com]

This error is caused by the data in the Oracle Unified Directory change log cookie expiring because Oracle Unified Directory has not been written to for a certain amount of time.

Solution:

  1. Open a browser and go to the following location:

    http://igdadmin.mycompany.com/sysasdmin

  2. Log in a as xelsysadm using the COMMON_IDM_PASSWORD.

  3. Under System Management, click Scheduler.

  4. Under Search Scheduled Jobs, enter LDAP * (there is a space before *) and hit Enter.

  5. For each job in the search results, click on the job name on the left, then click Disable on the right.

    Do this for all jobs. If the job is already disabled do nothing.

  6. Run the following commands on LDAPHOST1:

    cd OUD_ORACLE_INSTANCE/OUD/bin
./ldapsearch -h ldaphost1 -p 1389 -D "cn=oudadmin" -b "" -s base "objectclass=*" lastExternalChangelogCookie

Password for user 'cn=oudadmin': <OudAdminPwd>
dn: lastExternalChangelogCookie: dc=mycompany,dc=com:00000140c682473c263600000862;

    Copy the output string that follows lastExternalChangelogCookie:. This value is required in the next step. For example,

    dc=mycompany,dc=com:00000140c682473c263600000862;

    The Hex portion must be 28 characters long. If this value has more than one Hex portion then separate the 28char portions with spaces. For example:

    dc=mycompany,dc=com:00000140c4ceb0c07a8d00000043 00000140c52bd0b9104200000042 00000140c52bd0ba17b9000002ac 00000140c3b290b076040000012c;

  7. Run each of the following LDAP reconciliation jobs once to reset the last change number.:

    • LDAP Role Delete Reconciliation

    • LDAP User Delete Reconciliation

    • LDAP Role Create and Update Reconciliation

    • LDAP User Create and Update Reconciliation

    • LDAP Role Hierarchy Reconciliation

    • LDAP Role Membership Reconciliation

    To run the jobs:

    1. Login to the OIM System Administration Console as the user xelsysadm.

    2. Under System Management, click Scheduler.

    3. Under Search Scheduled Jobs, enter LDAP * (there is a space before *) and hit Enter.

    4. Click on the job to be run.

    5. Set the parameter Last Change Number to the value obtained in step 6.

      For example:

      dc=mycompany,dc=com:00000140c4ceb0c07a8d00000043 00000140c52bd0b9104200000042 00000140c52bd0ba17b9000002ac 00000140c3b290b076040000012c;

    6. Click Run Now.

    7. Repeat for each of the jobs in the list at the beginning of this step.

  8. For each incremental recon job whose last changelog number has been reset, execute the job and check that the job now completes successfully.

  9. After the job runs successfully, re-enable periodic running of the jobs according to your requirements.

If the issue continues to occur, increase the cookie retention time to two months by running the following command on each OUD instance.

If, the error appears again after the incremental jobs have been re-enabled and run successfully ("Full resync required. Reason: The provided cookie is older..."), then increase the OUD cookie retention time. Although there is no hard and fast rule as to what this value should be, it should be long enough to avoid the issue, but small enough to avoid unnecessary resource consumption on OUD. One or two weeks should suffice; two week is given in the following example:

./dsconfig set-replication-server-prop --provider-name "Multimaster Synchronization" --set replication-purge-delay:8w -D cn=oudadmin --trustAll -p 4444 -h LDAPHOST1
 
Password for user 'cn=oudadmin':  <OudAdminPswd>
Enter choice [f]: f

15.10.5 Troubleshooting Oracle SOA Suite

This section describes some common problems that can arise with Oracle SOA Suite and the actions you can take to resolve the problem. It contains the following topics:

15.10.5.1 Transaction Timeout Error

Problem: The following transaction timeout error appears in the log:

Internal Exception: java.sql.SQLException: Unexpected exception while enlisting
 XAConnection java.sql.SQLException: XA error: XAResource.XAER_NOTA start()
failed on resource 'SOADataSource_soaedg_domain': XAER_NOTA : The XID
is not valid

Solution: Check your transaction timeout settings, and be sure that the JTA transaction time out is less than the DataSource XA Transaction Timeout, which is less than the distributed_lock_timeout (at the database).

With the out of the box configuration, the SOA data sources do not set XA timeout to any value. The Set XA Transaction Timeout configuration parameter is unchecked in the WebLogic Server Administration Console. In this case, the data sources use the domain level JTA timeout which is set to 30. Also, the default distributed_lock_timeout value for the database is 60. As a result, the SOA configuration works correctly for any system where transactions are expected to have lower life expectancy than such values. Adjust these values according to the transaction times your specific operations are expected to take.