31 Managing the Topology for an Enterprise Deployment

This chapter describes some operations that you can perform after you have set up the Identity and Access Management topology. These operations include monitoring, scaling, backing up your topology, and troubleshooting.

This chapter includes the following topics:

31.1 Starting and Stopping Enterprise Deployment Components

This section describes how to start, stop and restart the various components of the Oracle Enterprise Deployment.

This section contains the following topics:

31.1.1 Startup and Shutdown Order

When starting up your entire infrastructure, start the components in the following order, (ignoring those not in your topology):

  1. Database(s)

  2. Database Listener(s)

  3. Web hosts

  4. LDAP hosts

  5. OAM hosts

  6. OIM hosts

  7. Oracle Identity Manager Administration Server

  8. Oracle Identity Manager Managed Servers

  9. Oracle Access Management Administration Server

  10. Oracle Access Management Managed Servers

  11. Oracle Web Servers

  12. Oracle Mobile Security Access Servers

Note:

To shutdown the servers, follow the reverse order.

31.1.2 Stopping and Starting Exalogic vServers

This section describes how to stop and start Exalogic vServers.

This section contains the following topics

31.1.2.1 Stopping vServers

To stop a vServer, do the following:

Note:

Do not use the xm destroy command or Oracle VM Manager to stop a vServer. Use only Exalogic Control.

  1. Log in to the Exalogic Control as a Cloud User.

  2. From the navigation pane on the left, click vDC Management.

  3. Under vDCs, expand your cloud such as MyCloud.

  4. Expand Accounts.

  5. Expand the name of your account, such as Dept1.

    All the vServers in the account are displayed.

  6. Select the vServer you wish to stop.

    The dashboard of the vServer is displayed.

  7. From the actions pane on the right, click Stop vServer. Wait till the job succeeds in the jobs pane.

31.1.2.2 Starting vServers

To start a vServer, do the following:

Note:

Do not use the xm create command or Oracle VM Manager to start a vServer. Use only Exalogic Control.

  1. Log in to the Exalogic Control as a Cloud User.

  2. From the navigation pane on the left, click vDC Management.

  3. Expand your cloud, such as MyCloud.

  4. Expand Accounts.

  5. Expand the name of your account, such as Dept1.

    All the vServers in the account are displayed.

  6. Select the vServer you wish to start.

    The dashboard of the vServer is displayed.

  7. From the actions pane on the right, click Start vServer. Wait till the job succeeds in the jobs pane.

31.1.3 Starting and Stopping Directory Services

This section describes how to start and stop the directory services. This section includes the following topics:

31.1.3.1 Starting and Stopping Oracle Unified Directory

This section describes how to start and stop Oracle Unified Directory.

This section includes the following topics:

31.1.3.1.1 Starting Oracle Unified Directory

To start Oracle Unified Directory, run the following command:

LDAP_ORACLE_INSTANCE/OUD/bin/start-ds
31.1.3.1.2 Stopping Oracle Unified Directory

To stop Oracle Unified Directory, run the command:

LDAP_ORACLE_INSTANCE/OUD/bin/stop-ds

31.1.3.2 Starting and Stopping Oracle Internet Directory

This section describes how to start and stop Oracle Internet Directory.

This section includes the following topics:

31.1.3.2.1 Starting Oracle Internet Directory

To start Oracle Internet Directory, run the following command:

OID_ORACLE_INSTANCE/bin/opmnctl startall

You can verify that the system components have started, using the following command:

OID_ORACLE_INSTANCE/bin/opmnctl status -l
31.1.3.2.2 Stopping Oracle Internet Directory

To stop Oracle Internet Directory, run the following command:

OID_ORACLE_INSTANCE/bin/opmnctl stopall

31.1.3.3 Starting and Stopping Oracle Active Directory

Refer to the Oracle Active Directory documentation for instructions on starting and stopping Oracle Active Directory.

31.1.4 Starting and Stopping Node Manager

This section described how to start and stop the Node Manager.

This section includes the following topics:

31.1.4.1 Starting Node Manager

If the Node Manager being started is the one that controls the Administration Server, then prior to starting the Node Manager, run the command:

export JAVA_OPTIONS=-DDomainRegistrationEnabled=true

To start the Node Manager, run the following command:

cd SHARED_CONFIG_DIR/nodemanager/hostname
./startNodeManagerWrapper.sh

31.1.4.2 Stopping Node Manager

To stop the Node Manager, kill the process started in the previous section.

31.1.5 Starting and Stopping IAMAccessDomain Services

This section descries how to start and stop IAMAccessDomain services.

This section contains the following topics:

31.1.5.1 Starting and Stopping a WebLogic Administration Server

This section describes how to start and stop a WebLogic Administration Server.

This section includes the following topics:

Notes:

  • Admin_User and Admin_Password are only used to authenticate connections between Node Manager and clients. They are independent from the server administration ID and password and are stored in the file: IAD_ASERVER_HOME/config/nodemanager/nm_password.properties

  • If you are starting the IAMAccessDomain Administration server, ASERVER_HOME is IAD_ASERVER_HOME. If you are starting the IAMGovernanceDomain Administration server, ASERVER_HOME is IGD_ASERVER_HOME

31.1.5.1.1 Starting a WebLogic Administration Server

The recommended way to start the Administration server is to use WLST and connect to Node Manager:

cd ORACLE_COMMON_HOME/common/bin
./wlst.sh

Where ORACLE_COMMON_HOME is from the MW_HOME associated with the domain you are starting or stopping.

To start the Administration Server in the Access Domain, use the following command:

nmConnect('Admin_User','Admin_Password','IADADMINVHN','5556', 'IAMAccessDomain','IAD_ASERVER_HOME')
nmStart('AdminServer')

For example:

nmConnect('Admin_User','Admin_Password','IADADMINVHN','5556', 'IAMAccessDomain','/u01/oracle/config/domains/IAMAccessDomain')
nmStart('AdminServer')

Alternatively, you can start the Administration server by using the command:

ASERVER_HOME/bin/startWebLogic.sh

Note:

The Node Manager admin password is the COMMON_IAM_PASSWORD.
31.1.5.1.2 Stopping a WebLogic Administration Server

To stop the Administration Server, log in to the WebLogic console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs"

Then proceed as follows:

  1. Click the Control tab.

  2. Select AdminServer(admin).

  3. Click Shutdown and select Force Shutdown now.

  4. Click Yes when asked to confirm that you want to shut down the Administration Server.

31.1.5.2 Starting and Stopping Oracle Access Manager Weblogic Managed Servers

This section describes how to start and stop the Oracle Access Manager Managed Servers.

This section includes the following topics:

31.1.5.2.1 Starting Oracle Access Manager WebLogic Managed Servers

To start a Oracle Access Manager Managed Server(s), log in to the WebLogic console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs".

Then proceed as follows:

  1. Click the Control tab.

  2. Select Environment -> Servers from the Domain Structure menu.

  3. Select the Oracle Access Manager Managed Server. For example, wls_oam1.

  4. Click Start.

  5. Click Yes when asked to confirm that you want to start the server(s).

31.1.5.2.2 Stopping Oracle Access Manager WebLogic Managed Servers

To stop a Oracle Access Manager Managed Server(s), log in to the WebLogic console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs".

Then proceed as follows:

  1. Select Environment -> Servers from the Domain Structure menu.

  2. Click the Control tab.

  3. Select the Oracle Access Manager Managed Server. For example, wls_oam1.

  4. Click Shutdown and then click Force Shutdown Now.

  5. Click Yes when asked to confirm that you want to shut down the server(s).

31.1.5.3 Starting and Stopping Policy Manager Weblogic Managed Servers

This section describes how to start and stop Policy Manager Managed Servers.

This section includes the following topics:

31.1.5.3.1 Starting Policy Manager WebLogic Managed Servers

To start a Policy Manager Managed Server(s), log in to the WebLogic console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs".

Then proceed as follows:

  1. Click the Control tab.

  2. Select Environment -> Servers from the Domain Structure menu.

  3. Select the Policy Manager Managed Server. For example, wls_ama1.

  4. Click Start.

  5. Click Yes when asked to confirm that you want to start the server(s).

31.1.5.3.2 Stopping Policy Manager WebLogic Managed Servers

To stop a Policy Manager Managed Server(s), log in to the WebLogic console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs".

Then proceed as follows:

  1. Select Environment -> Servers from the Domain Structure menu.

  2. Click the Control tab.

  3. Select the Policy Manager Managed Server. For example, wls_ama1.

  4. Click Shutdown and then click Force Shutdown Now.

  5. Click Yes when asked to confirm that you want to shut down the server(s).

31.1.5.4 Starting and Stopping Mobile Security Manager Weblogic Managed Servers

This section describes how to start and stop Oracle Mobile Security Manager Managed Servers.

This section includes the following topics:

31.1.5.4.1 Starting Mobile Security Manager WebLogic Managed Servers

To start a Mobile Security Manager Managed Server(s), log in to the WebLogic console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs".

Then proceed as follows:

  1. Click the Control tab.

  2. Select Environment -> Servers from the Domain Structure menu.

  3. Select the Mobile Security Manager Managed Server. For example, wls_msm1.

  4. Click Start.

  5. Click Yes when asked to confirm that you want to start the server(s).

31.1.5.4.2 Stopping Mobile Security Manager WebLogic Managed Servers

To stop a Mobile Security Manager Managed Server(s), log in to the WebLogic console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs".

Then proceed as follows:

  1. Select Environment -> Servers from the Domain Structure menu.

  2. Click the Control tab.

  3. Select the Mobile Security Manager Managed Server. For example, wls_msm1.

  4. Click Shutdown and then click Force Shutdown Now.

  5. Click Yes when asked to confirm that you want to shut down the server(s).

31.1.6 Starting and Stopping IAMGovernanceDomain Services

This section descries how to start and stop IAMGovernanceDomain services.

This section contains the following topics:

31.1.6.1 Starting and Stopping a WebLogic Administration Server

This section describes how to start and stop a WebLogic Administration Server.

This section includes the following topics:

Notes:

  • Admin_User and Admin_Password are only used to authenticate connections between Node Manager and clients. They are independent from the server administration ID and password and are stored in the file: IAD_ASERVER_HOME/config/nodemanager/nm_password.properties

  • If you are starting the IAMAccessDomain Administration server, ASERVER_HOME is IAD_ASERVER_HOME. If you are starting the IAMGovernanceDomain Administration server, ASERVER_HOME is IGD_ASERVER_HOME

31.1.6.1.1 Starting a WebLogic Administration Server

The recommended way to start the Administration server is to use WLST and connect to Node Manager:

cd ORACLE_COMMON_HOME/common/bin
./wlst.sh

Where ORACLE_COMMON_HOME is from the MW_HOME associated with the domain you are starting or stopping.

To start the Administration Server in the IAMGovernanceDomain, use the following command:

nmConnect('Admin_User','Admin_Password','IADADMINVHN','5556', 'IAMGovernanceDomain','IAD_ASERVER_HOME')
nmStart('AdminServer')

For example:

nmConnect('Admin_User','Admin_Password','IADADMINVHN','5556', 'IAMGovernanceDomain','/u01/oracle/config/domains/IAMGovernanceDomain')
nmStart('AdminServer')

Alternatively, you can start the Administration server by using the command:

ASERVER_HOME/bin/startWebLogic.sh

Note:

The Node Manager admin password is the COMMON_IAM_PASSWORD.
31.1.6.1.2 Stopping a WebLogic Administration Server

To stop the Administration Server, log in to the WebLogic console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs"

Then proceed as follows:

  1. Click the Control tab.

  2. Select AdminServer(admin).

  3. Click Shutdown and select Force Shutdown now.

  4. Click Yes when asked to confirm that you want to shut down the Administration Server.

31.1.6.2 Starting and Stopping Oracle SOA Suite Weblogic Managed Servers

This section describes how to start and stop the Oracle SOA Suite Managed Servers.

This section includes the following topics:

31.1.6.2.1 Starting Oracle SOA Suite WebLogic Managed Servers

To start a Oracle SOA Suite Managed Server(s), log in to the WebLogic console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs".

Then proceed as follows:

  1. Click the Control tab.

  2. Select Environment -> Servers from the Domain Structure menu.

  3. Select the Oracle SOA Suite Managed Server. For example, wls_soa1.

  4. Click Start.

  5. Click Yes when asked to confirm that you want to start the server(s).

31.1.6.2.2 Stopping Oracle SOA Suite WebLogic Managed Servers

To stop a Oracle SOA Suite Managed Server(s), log in to the WebLogic console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs".

Then proceed as follows:

  1. Select Environment -> Servers from the Domain Structure menu.

  2. Click the Control tab.

  3. Select the Oracle SOA Suite Managed Server. For example, wls_soa1.

  4. Click Shutdown and then click Force Shutdown Now.

  5. Click Yes when asked to confirm that you want to shut down the server(s).

31.1.6.3 Starting and Stopping Oracle Identity Manager Weblogic Managed Servers

This section describes how to start and stop Oracle Identity Manager Managed Servers.

This section includes the following topics:

31.1.6.3.1 Starting Oracle Identity Manager WebLogic Managed Servers

To start a Oracle Identity Manager Managed Server(s), log in to the WebLogic console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs".

Then proceed as follows:

  1. Click the Control tab.

  2. Select Environment -> Servers from the Domain Structure menu.

  3. Select the Oracle Identity Manager Managed Server. For example, wls_oim1.

  4. Click Start.

  5. Click Yes when asked to confirm that you want to start the server(s).

31.1.6.3.2 Stopping Oracle Identity Manager WebLogic Managed Servers

To stop a Oracle Identity Manager Managed Server(s), log in to the WebLogic console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs".

Then proceed as follows:

  1. Select Environment -> Servers from the Domain Structure menu.

  2. Click the Control tab.

  3. Select the Oracle Identity Manager Managed Server. For example, wls_oim1.

  4. Click Shutdown and then click Force Shutdown Now.

  5. Click Yes when asked to confirm that you want to shut down the server(s).

31.1.6.4 Starting and Stopping Oracle BI Publisher Weblogic Managed Servers

This section describes how to start and stop Oracle BI Publisher Managed Servers.

This section includes the following topics:

31.1.6.4.1 Starting Oracle BI Publisher WebLogic Managed Servers

To start a Oracle BI Publisher Managed Server(s), log in to the WebLogic console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs".

Then proceed as follows:

  1. Click the Control tab.

  2. Select Environment -> Servers from the Domain Structure menu.

  3. Select the Oracle BI Publisher Managed Server. For example, wls_bi1.

  4. Click Start.

  5. Click Yes when asked to confirm that you want to start the server(s).

31.1.6.4.2 Stopping Oracle BI Publisher WebLogic Managed Servers

To stop Oracle BI Publisher Managed Server(s), log in to the WebLogic console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs".

Then proceed as follows:

  1. Select Environment -> Servers from the Domain Structure menu.

  2. Click the Control tab.

  3. Select the Oracle BI Publisher Managed Server. For example, wls_bi1.

  4. Click Shutdown and then click Force Shutdown Now.

  5. Click Yes when asked to confirm that you want to shut down the server(s).

31.1.7 Starting and Stopping Web Servers

This section describes how to start and stop web services like Oracle HTTP Server, Oracle Traffic Director, and Oracle Mobile Access Server.

This section includes the following topics:

31.1.7.1 Starting and Stopping Oracle HTTP Server

This section describes how to start and stop Oracle HTTP Server.

Prior to starting/stopping the Oracle HTTP server ensure that the environment variables WEB_ORACLE_HOME and ORACLE_INSTANCE are defined and that ORACLE_HOME/opmn/bin appears in the PATH. For example:

export ORACLE_HOME=WEB_ORACLE_HOME
export ORACLE_INSTANCE=WEB_ORACLE_INSTANCE
export PATH=$ORACLE_HOME/opmn/bin:$PATH

This section includes the following topics:

31.1.7.1.1 Starting Oracle HTTP Server

To start the Oracle HTTP Server, run the following command:

opmnctl startall
31.1.7.1.2 Stopping Oracle HTTP Server

To stop the Oracle HTTP Server, run the following command:

To stop the entire Web tier:

opmnctl stopall

To stop Oracle HTTP Server only:

opmnctl stoproc process-type=OHS

31.1.7.2 Starting the Oracle Traffic Director Instances

To start Oracle Traffic Director instances using the administration console, do the following

  1. Log in to the administration console using the URL listed in Section 31.2, "About Identity and Access Management Console URLs."

  2. Click Configurations at the upper left corner of the page.

    A list of the available configurations is displayed.

  3. Select the configuration for which you want to start the instance.

  4. In the navigation pane, select Instances.

  5. Click Start/Restart for the instance that you want to start.

This section also includes the following topics:

31.1.7.2.1 Starting and Stopping Oracle Traffic Director Administration Instances

OTD administration instances must be running to enable access to the OTD administration console and to enable the administration console to control remote OTD instances. To start the OTD administration console: perform the following steps.

Execute the command startserv located in the directory: WEB_ORACLE_INSTANCE/admin-server/bin

To stop the Administration Services, execute the command stopserv located in the directory: WEB_ORACLE_INSTANCE/admin-server/bin

Note:

If you are not running Oracle Traffic Director as root, manually stop the OTD failover groups first using the following command: 
OTD_ORACLE_HOME/bin/tadm stop-failover --instance-home=WEB_INSTANCE_HOME/ --config=login.example.com
31.1.7.2.2 Starting Oracle Traffic Director Instances

To start or restart all instances of the selected configuration, click Start/Restart Instances in the Common Tasks pane. To stop all instances of the configuration, click Stop Instances.

31.1.7.2.3 Starting Oracle Traffic Director Failover groups

If you started your OTD instances as the software owner rather than root, then start OTD failover groups using the following command when you are logged in as root:

WEB_ORACLE_HOME/bin/tadm start-failover --instance-home=WEB_INSTANCE_HOME/ --config=IAM

If you did not configure your Oracle Traffic Director to start as root, manually start the failover groups using the following command as root:

OTD_ORACLE_HOME/bin/tadm start-failover --instance-home=WEB_INSTANCE_HOME/ --config=login.example.com

31.1.7.3 Starting and Stopping Oracle Mobile Access Server

This section describes how to start and stop Oracle Mobile Access Server.

This section includes the following topics:

31.1.7.3.1 Starting Oracle Mobile Access Server

To start the Oracle Mobile Access Server, run the following command:

MSAS_INSTANCE_HOME/bin/startServer.sh
31.1.7.3.2 Stopping Oracle Mobile Access Server

To stop the Oracle Mobile Access Server, run the following command:

MSAS_INSTANCE_HOME/bin/stopServer.sh

31.2 About Identity and Access Management Console URLs

Table 31-1 lists the administration consoles used in this guide and their URLs.

Table 31-1 Console URLs

Domain Console URL Administrator User Name

IAMAccessDomain

WebLogic Administration Console

http://IADADMIN.example.com/console

weblogic_idm
 

Enterprise Manager FMW Control

http://IADADMIN.example.com/em

weblogic_idm
 

OAM console

http://IADADMIN.example.com/oamconsole

oamadmin
 

Access Management Policy Manager

http://IADADMIN.example.com/access

oamadmin

IAMGovernanceDomain

WebLogic Administration Console

http://IGDADMIN.example.com/console

weblogic_idm
 

Enterprise Manager FMW Control

http://IGDADMIN.example.com/em

weblogic_idm
 

Identity Manager System Administration Console

http://IGDADMIN.example.com/sysadmin

xelsysadm
 

Oracle Identity Self Service

https://prov.example.com/identity

xelsysadm

N/A

Exalogic Control (Enterprise Manager Operations Control)

https://exalogic:9943/emoc

  

N/A

Oracle Traffic Director Administration Console

https://OTDADMINVHN.example.com:8989

otdadmin

N/A

Oracle ZFS Storage Appliance Browser User Interface

https://exalogicsn01-priv:215

  

31.3 Monitoring Enterprise Deployments

This section provides information about monitoring the Identity and Access Management enterprise deployment described in this manual.

This section contains the following topics:

31.3.1 Monitoring Oracle Unified Directory

You can check the status of Oracle Unified Directory by issuing the command:

LDAP_ORACLE_INSTANCE/OUD/bin/status

This command prompts for the OUD Admin username and OUD_COMMON_PASSOWORD.

This command accesses the locally running Oracle Unified Directory instance and reports the status of the directory, including whether or not replication and LDAP or LDAPS is enabled.

31.3.2 Monitoring WebLogic Managed Servers

You can use Oracle Enterprise Manager Fusion Middleware Control to monitor Managed Servers and other Fusion Middleware components, such as Access Manager, Oracle Identity Manager, Oracle Identity Federation, and SOA. For more information, see the administrator guides listed in the Preface under "Related Documents".

31.4 Auditing Identity and Access Management

Oracle Fusion Middleware Audit Framework is a new service in Oracle Fusion Middleware 11g, designed to provide a centralized audit framework for the middleware family of products. The framework provides audit service for platform components such as Oracle Platform Security Services (OPSS) and Oracle Web Services. It also provides a framework for JavaEE applications, starting with Oracle's own JavaEE components. JavaEE applications are able to create application-specific audit events. For non-JavaEE Oracle components in the middleware such as C or JavaSE components, the audit framework also provides an end-to-end structure similar to that for JavaEE applications.

Figure 31-1 is a high-level architectural diagram of the Oracle Fusion Middleware Audit Framework. For more information, see Oracle Fusion Middleware Application Security Guide.

Figure 31-1 Audit Event Flow

Surrounding text describes Figure 31-1 .

The Oracle Fusion Middleware Audit Framework consists of the following key components:

  • Audit APIs

    These are APIs provided by the audit framework for any audit-aware components integrating with the Oracle Fusion Middleware Audit Framework. During run-time, applications may call these APIs where appropriate to audit the necessary information about a particular event happening in the application code. The interface enables applications to specify event details such as username and other attributes needed to provide the context of the event being audited.

  • Audit Events and Configuration

    The Oracle Fusion Middleware Audit Framework provides a set of generic events for convenient mapping to application audit events. Some of these include common events such as authentication. The framework also enables applications to define application-specific events.

    These event definitions and configurations are implemented as part of the audit service in Oracle Platform Security Services. Configurations can be updated through Enterprise Manager (UI) and WLST (command-line tool).

  • The Audit Bus-stop

    Bus-stops are local files containing audit data before they are pushed to the audit repository. In the event where no database repository is configured, these bus-stop files can be used as a file-based audit repository. The bus-stop files are simple text files that can be queried easily to look up specific audit events. When a DB-based repository is in place, the bus-stop acts as an intermediary between the component and the audit repository. The local files are periodically uploaded to the audit repository based on a configurable time interval.

  • Audit Loader

    As the name implies, audit loader loads the files from the audit bus-stop into the audit repository. In the case of platform and JavaEE application audit, the audit loader is started as part of the JavaEE container start-up. In the case of system components, the audit loader is a periodically spawned process.

  • Audit Repository

    Audit Repository contains a pre-defined Oracle Fusion Middleware Audit Framework schema, created by Repository Creation Utility (RCU). Once configured, all the audit loaders are aware of the repository and upload data to it periodically. The audit data in the audit repository is expected to be cumulative and grow over time. Ideally, this should not be an operational database used by any other applications - rather, it should be a standalone RDBMS used for audit purposes only. In a highly available configuration, Oracle recommends that you use an Oracle Real Application Clusters (Oracle RAC) database as the audit data store.

  • Oracle Business Intelligence Publisher

    The data in the audit repository is exposed through pre-defined reports in Oracle Business Intelligence Publisher. The reports enable users to drill down the audit data based on various criteria. For example:

    • Username

    • Time Range

    • Application Type

    • Execution Context Identifier (ECID)

For more introductory information for the Oracle Fusion Middleware Audit Framework, see the "Introduction to Oracle Fusion Middleware Audit Framework" chapter in the Oracle Fusion Middleware Application Security Guide.

For information on how to configure the repository for Oracle Fusion Middleware Audit Framework, see the "Configuring and Managing Auditing" chapter in the Oracle Fusion Middleware Application Security Guide.

The EDG topology does not include Oracle Fusion Middleware Audit Framework configuration. The ability to generate audit data to the bus-stop files and the configuration of the audit loader are available once the products are installed. The main consideration is the audit database repository where the audit data is stored. Because of the volume and the historical nature of the audit data, it is strongly recommended that customers use a separate database from the operational store or stores being used for other middleware components.

31.5 Performing Backups and Recoveries

You can use the UNIX tar command for most backups. Typical usage is:

tar -czvpsPf BACKUP_LOCATION/backup_file.tar directories

You can use the UNIX tar command for recovery. Typical usage is:

tar -xzvpsPf BACKUP_LOCATION/backup_file.tar

For database backup and recovery, you can use the database utility RMAN. See the Oracle Database Backup and Recovery Reference for more information on using this command.

This section contains the following topics:

31.5.1 Performing Baseline Backups

Perform baseline backups when building a system and when applying patches that update static artifacts, such as the Oracle binaries.

After performing a baseline backup, also perform a runtime backup.

Table 31-2 Static Artifacts to Back Up in the Identity and Access Management Enterprise Deployment

Type Host Location Tier

Oracle Home (database)

Oracle RAC database hosts:

IADDBHOST1

IADDBHOST2

User Defined

Database

Oracle Directory Binaries

LDAPHOST1

LDAPHOST2

Middleware Home: DIR_MW_HOME

Directory Tier

Oracle Access Management Binaries

OAMHOST1

OAMHOST2

Middleware Home: IAD_MW_HOME

Application Tier

Oracle Identity Governance Binaries

OIMHOST1

OIMHOST2

Middleware Home: IGD_MW_HOME

Application Tier

Web Tier Binaries

WEBHOST1

WEBHOST2

Middleware Oracle home, WEB_ORACLE_HOME:

Web Tier

Install-Related Files

Each host

OraInventory:

ORACLE_BASE/oraInventory

/etc/oratab, /etc/oraInst.loc

~/bea/beahomelist (on hosts where WebLogic Server is installed)

Not applicable.

Note:

It is also recommended that you back up your load balancer configuration. Refer to your vendor documentation on how to do this.

For more information on backup and recovery of Oracle Fusion Middleware components, refer to the following chapters in the Oracle Fusion Middleware Administrator's Guide:

31.5.2 Performing Runtime Backups

Perform runtime backups on an ongoing basis. These backups contain information on items that can change frequently, such as data in the database, domain configuration information, and identity information in LDAP directories.

Table 31-3 Run-Time Artifacts to Back Up in the Identity and Access Management Enterprise Deployments

Type Host Location Tier

IAMAccessDomain Home

OAMHOST1

OAMHOST2

Administration Server and Shared Files: IAD_ASERVER_HOME

Managed Servers: IAD_MSERVER_HOME

Application Tier

IAMGovernanceDomain Home

OIMHOST1

OIMHOST2

Administration Server and Shared Files: IGD_ASERVER_HOME

Managed Servers: IGD_MSERVER_HOME

Application Tier

Oracle HTTP Server

WEBHOST1

WEBHOST2

OHS_ORACLE_INSTANCE

Web Tier

Oracle Traffic Director

WEBHOST1

WEBHOST2

OTD_ORACLE_INSTANCE

Web Tier

Mobile Security Access Server

WEBHOST1

WEBHOST2

MSAS_ORACLE_INSTANCE

Web Tier

Oracle RAC Databases

IADDBHOST1

IADDBHOST2

User defined

Database

Oracle Unified Directory

LDAPHOST1

LDAPHOST2

OUD_ORACLE_INSTANCE

Directory Tier

Oracle Internet Directory

LDAPHOST1

LDAPHOST2

OID_ORACLE_INSTANCE

Directory Tier

31.5.3 Performing Backups During Installation and Configuration

It is an Oracle best practices recommendation to create a backup after successfully completing the installation and configuration of each tier, or at another logical point. Create a backup after verifying that the installation so far is successful. This is a quick backup for the express purpose of immediate restoration in case of problems in later steps. The backup destination is the local disk. You can discard this backup when the enterprise deployment setup is complete. After the enterprise deployment setup is complete, you can initiate the regular deployment-specific Backup and Recovery process.

For more details, see the Oracle Fusion Middleware Administrator's Guide.

For information on database backups, refer to the Oracle Database Backup and Recovery User's Guide.

This section contains the following topics:

31.5.3.1 Backing Up Middleware Home

Back up the Middleware homes whenever you create a new one or add components to it. The Middleware homes used in this guide are Oracle Identity Management and Oracle Identity and Access Management, as listed in Table 31-2.

31.5.3.2 Backing Up LDAP Directories

Whenever you perform an action which updates the data in LDAP, back up the directory contents.

This section contains the following topics:

31.5.3.2.1 Backing Up Oracle Unified Directory

To backup Oracle Unified Directory, perform the following steps:

  1. Shut down the Oracle Unified Directory Instances as described in Section 31.1, "Starting and Stopping Enterprise Deployment Components."

  2. Back up LDAP_ORACLE_INSTANCE directories on each host.

  3. Restart the Oracle Unified Directory instances as described in Section 31.1, "Starting and Stopping Enterprise Deployment Components."

31.5.3.2.2 Backing Up Third-Party Directories

Refer to your operating system vendor's documentation for information about backing up directories.

31.5.3.3 Backing Up the Database

Whenever you create add a component to the configuration, back up the IADDB database. Perform this backup after or adding components such as Oracle Access Management Access Manager or Oracle Identity Manager.

31.5.3.4 Backing Up the WebLogic Domain IAMGovernanceDomain

To back up the WebLogic domain, perform these steps:

  1. Shut down the WebLogic administration server and any managed servers running in the domain as described in Section 31.1, "Starting and Stopping Enterprise Deployment Components."

  2. Back up the IGD_ASERVER_HOME directory from shared storage.

  3. Back up the IGD_MSERVER_HOME directory from each host.

  4. Restart the WebLogic Administration Server and managed servers.

31.5.3.5 Backing Up the WebLogic Domain IAMAccessDomain

To back up the WebLogic domain, perform these steps:

  1. Shut down the WebLogic administration server and any managed servers running in the domain as described in Section 31.1, "Starting and Stopping Enterprise Deployment Components."

  2. Back up the IAD_ASERVER_HOME directory from shared storage.

  3. Back up the IAD_MSERVER_HOME directory from each host.

  4. Restart the WebLogic Administration Server and managed servers.

31.5.3.6 Backing Up the Web Tier

To back up the Web Tier, perform these steps:

31.5.3.6.1 Backing Up Oracle HTTP Server

Back up Oracle HTTP Server as follows:

  1. Shut down the Oracle HTTP Server as described in Section 31.1, "Starting and Stopping Enterprise Deployment Components."

  2. Back up the WEB_ORACLE_INSTANCE directory on local storage.

  3. Start the Oracle HTTP Server as described in Section 31.1, "Starting and Stopping Enterprise Deployment Components."

31.6 Patching Enterprise Deployments

It is recommended that you patch enterprise deployments by using the automated patching solution included with the Identity and Access Management Life Cycle Tools.

The process of applying patches can be summarized as follows:

  1. Create a patch top. A patch top directory contains patches, classified by each product to which patches apply.

  2. Run Patch Manager to generate a patch plan. Based on the deployment topology and patches provided, the Manager creates an optimal plan to apply those patches.

  3. Run the Patcher against all hosts which are affected by the plan. You might need to execute the Patcher on a given host multiple times if required by a given plan. As each Patcher invocation completes, it directs you where to run the Patcher next.

When the Patcher runs, it stops and starts server instances as necessary, and ensures that patches are applied in the correct order to satisfy dependencies.

Full details on how to use the IDM Patching Framework can be found in Oracle Fusion Middleware Patching Guide for Oracle Identity and Access Management. The Guide also contains instructions for patching the deployment manually if required, using the OPatch tool.

31.7 Preventing Timeouts for SQL

Most of the production deployment involves firewalls. Because database connections are made across firewalls, Oracle recommends that the firewall be configured so that the database connection is not timed out. For Oracle Real Application Clusters (Oracle RAC), the database connections are made on Oracle RAC VIPs and the database listener port. You must configure the firewall so it does not time out these connections. If such a configuration is not possible, set the SQLNET.EXPIRE_TIME=n parameter in the ORACLE_HOME/network/admin/sqlnet.ora file on the database server, where n is the time in minutes. Set this value to less than the known value of the timeout for the network device (that is, a firewall). For Oracle RAC, set this parameter in all of the Oracle home directories.

31.8 Manually Failing Over the WebLogic Administration Server

This section discusses how to fail over the Administration Server to a new host after the primary host fails. The example in this section shows how to fail the Access Management Administration Server from OAMHOST1 to OAMHOST2. If you are failing over the Oracle Identity Manager Administration server, substitute the appropriate values for that domain.

This section contains the following topics:

31.8.1 Failing Over the Administration Server to OAMHOST2

If a node fails, you can fail over the Administration Server to another node. This section describes how to fail over the Administration Server from OAMHOST1 to OAMHOST2.

Assumptions:

  • The Administration Server is configured to listen on IADADMINVHN.example.com, and not on ANY address.

  • The Administration Server is failed over from OAMHOST1 to OAMHOST2, and the two nodes have these IP addresses:

    • OAMHOST1: 100.200.140.165

    • OAMHOST2: 100.200.140.205

    • IADADMINVHN: 100.200.140.206

      This is the Virtual IP address where the Administration Server is running, assigned to interface:index (for example, eth1:2), available in OAMHOST1 and OAMHOST2.

  • The domain directory where the Administration Server is running in OAMHOST1 is on a shared storage and is mounted also from OAMHOST2.

    Note:

    NM in OAMHOST2 does not control the domain at this point, since unpack/nmEnroll has not been run yet on OAMHOST2. But for the purpose of AdminServer failover and control of the AdminServer itself, Node Manager is fully functional

  • Oracle WebLogic Server and Oracle Fusion Middleware Components have been installed in OAMHOST2 as described in previous chapters. That is, the same path for IAD_ORACLE_HOME and IAD_MW_HOME that exists in OAMHOST1 is available in OAMHOST2.

The following procedure shows how to fail over the Administration Server to a different node, OAMHOST2.

  1. Stop the Administration Server on OAMHOST1 as described in Section 31.1, "Starting and Stopping Enterprise Deployment Components."

  2. Migrate the IP address to the second node.

    1. Run the following command as root on OAMHOST1 (where x:y is the current interface used by IADADMINVHN.example.com):

      /sbin/ifconfig x:y down

      For example:

      /sbin/ifconfig eth0:1 down

    2. Run the following command on OAMHOST2:

      /sbin/ifconfig interface:index IP_Address netmask netmask

      For example:

      /sbin/ifconfig eth0:1 100.200.140.206 netmask 255.255.255.0

    Note:

    Ensure that the netmask and interface to be used match the available network configuration in OAMHOST2.

  3. Update routing tables by using arping on OAMHOST2, for example:

    /sbin/arping -q -U -c 3 -I eth0 100.200.140.206

31.8.2 Starting the Administration Server on OAMHOST2

Perform the following steps to start Node Manager on OAMHOST2.

  1. On OAMHOST2, mount the Administration Server domain directory if it is not already mounted. For example:

    mount /u01/oracle

  2. Start Node Manager by using the following commands:

    cd WL_HOME/server/bin
./startNodeManager.sh

  3. Stop the Node Manager by killing the Node Manager process.

    Note:

    Starting and stopping Node Manager at this point is only necessary the first time you run Node Manager. Starting and stopping it creates a property file from a predefined template. The next step adds properties to that property file.

  4. Run the setNMProps.sh script to set the StartScriptEnabled property to true before starting Node Manager:

    cd ORACLE_COMMON_HOME/common/bin
./setNMProps.sh

    Note:

    You must use the StartScriptEnabled property to avoid class loading failures and other problems.

  5. Start the Node Manager as described in Section 31.1, "Starting and Stopping Enterprise Deployment Components."

  6. Start the Administration Server on OAMHOST2.

    cd ORACLE_COMMON_HOME/common/bin
./wlst.sh

    Once in the WLST shell, execute the following commands:

    nmConnect('admin','Admin_Password', 'OAMHOST2','5556', 'IAMAccessDomain','/u1/oracle/config/domains/IAMAccessDomain')
nmStart('AdminServer')

  7. Test that you can access the Administration Server on OAMHOST2 as follows:

    1. Ensure that you can access the Oracle WebLogic Server Administration Console at:

      http://IADADMINVHN.example.com/console.

    2. Check that you can access and verify the status of components in the Oracle Enterprise Manager at: http://IADADMINVHN.example.com/em.

31.8.3 Validating Access to OAMHOST2 Through Oracle HTTP Server

Check if you can access the Administration Server when it is running on OAMHOST2.

31.8.4 Failing the Administration Server Back to OAMHOST1

This step checks that you can fail back the Administration Server, that is, stop it on OAMHOST2 and run it on OAMHOST1. To do this, migrate IADADMINVHN back to OAMHOST1 node as described in the following steps.

  1. Ensure that the Administration Server is not running on OAMHOST2. If it is, stop it from the WebLogic console, or by running the command stopWeblogic.sh from IAD_ASERVER_HOME/bin.

  2. On OAMHOST2, unmount the Administration server domain directory. For example:

    umount /u01/oracle

  3. On OAMHOST1, mount the Administration server domain directory. For example:

    mount /u01/oracle

  4. Disable the IADADMINVHN.example.com virtual IP address on OAMHOST2 and run the following command as root on OAMHOST2:

    /sbin/ifconfig x:y down

    where x:y is the current interface used by IADADMINVHN.example.com.

  5. Run the following command on OAMHOST1:

    /sbin/ifconfig interface:index 100.200.140.206 netmask 255.255.255.0

    Note:

    Ensure that the netmask and interface to be used match the available network configuration in OAMHOST1

  6. Update routing tables by using arping. Run the following command from OAMHOST1.

    /sbin/arping -q -U -c 3 -I interface 100.200.140.206

  7. If Node Manager is not already started on OAMHOST1, start it, as described in Section 31.1, "Starting and Stopping Enterprise Deployment Components."

  8. Start the Administration Server again on OAMHOST1.

    cd ORACLE_COMMON_HOME/common/bin
./wlst.sh

    Once in the WLST shell, execute

    nmConnect(admin,'Admin_Pasword, OAMHOST1,'5556',
     'IAMAccessDomain','/u01/oracle/config/domains/IAMAccessDomain'
nmStart('AdminServer')

  9. Test that you can access the Oracle WebLogic Server Administration Console at:

    http://IADADMINVHN.example.com:7001/console

    where 7001 is WLS_ADMIN_PORT in Section 8.1, "Summary of Virtual IP Addresses Required."

  10. Check that you can access and verify the status of components in the Oracle Enterprise Manager at:

    http://IADADMIN.example.com/em

31.9 Changing Startup Location

When the environment was deployed, start and stop scripts were generated to start and stop components in the topology. At the time of Deployment, the Access Domain Administration server was configured to start on OAMHOST1. If you want to permanently change this to start on OAMHOST2, perform the following steps.

Use the same steps, changing the name of the server and host, to change the Governance Domain Administration server to start on OIMHOST2 instead of OIMHOST1.

Edit the file serverInstancesInfo.txt, which is located in the directory: SHARED_CONFIG_DIR/scripts

Locate the line which looks like this:

OAMHOST1.example.com AS AdminServer

Change OAMHOST1 to OAMHOST2 and save the file.

31.10 Troubleshooting

This section describes how to troubleshoot common issues that can arise with the Identity and Access Management enterprise deployment described in this manual.

This section contains the following topics:

31.10.1 Troubleshooting Oracle Traffic Director

This section describes possible issues for Oracle Traffic Director (OTD). It contains the following topics:

31.10.1.1 OTD Failover Groups Show as Started, but IP Address Cannot be Pinged

Problem

OTD failover groups show as started, but IP address cannot be pinged.

Failover groups require a distinct Router ID on the system. If you reuse a Router ID, this behavior occurs. This can even occur if you remove and reinstall OTD.

Solution

To resolve this issue, recreate the failover group using a different Router ID

31.10.1.2 Error When Accessing SSL Terminated URL

Problem

When you access an SSL terminated URL, an error that says the browser cannot connect to the server, is displayed.

Solution

To resolve this issue, do the following:

  1. Ensure that the WebLogic plugin in enabled in the domain.

  2. Ensure that the SSL Passthrough is enabled in OTD.

  3. Ensure that the load balancer is adding WL-Proxy-SSL true and IS_SSL ssl to the HTTP request header. Different load balancers do this in different ways. On BigIP, you create an irule with the following content:

    # Notify the backend servers that this traffic was SSL offloaded by the F5.

##

when HTTP_REQUEST {

HTTP::header insert WL-Proxy-SSL true
HTTP::header insert IS_SSL ssl

}

31.10.1.3 Error When Creating Failover Groups

Problem

When creating failover groups, the following error is seen:

OTD-67322 The specified virtual IP 'x.x.x.x' cannot be bound to any of the network 
interfaces on the node 'hostname'. The IP addresses bound to the node are [......] 
check if the specified virtual IP is in the proper subnet. This error could also 
be caused if either the network interfaces on the node are not configured 
correctly or if the network prefix length is incorrect.

Solution

This is due to the IP address or CIDR being incompatible with the IP address or subnet already configured on the network card you wish to bind to. Choose a different IP address or CIDR.

31.10.2 Troubleshooting Identity and Access Management Deployment When Using IDMLCM

This section describes some common problems related to Deployment. It contains the following topics:

31.10.2.1 Deployment Fails

Problem

Deployment fails.

Solution

Check the Deployment logs located in the directory:

LCM_HOME/provisioning/logs/hostname

where hostname is the host where the Deployment step failed.

Rectify the error and re-deploy.

31.10.2.2 Deployment Fails with Error: Incorrect Host or Domain Name Format for Attribute

Problem

Deployment fails with an error similar to this:

Incorrect host format for attibute : PRIMARY_OAM_SERVERS : server-123.example.com

Due to a bug, one of the tools invoked during the deployment process cannot handle host names or domain names containing the hyphen (-) character.

Solution

Use host names and domain names that do NOT contain the hyphen (-) character.

31.10.2.3 Connection to Directory Failed Exception

Problem

You see the following error in the log configure log file:

oracle.idm.automation.exception.ExecutionFailedException: Connection to Directory failed: Host/Port details incorrect

Solution

  1. Check the property file mentioned in the log file output.

    A line similar to the following appears a bit farther in the log:

    See [/u01/oracle/products/products/access/iam/idmtools/bin/idmConfigTool.sh, -configOAM, input_file=/u01/lcm/tools/idmlcm/provisioning/idm-provisioning-build/config/config_oam.properties, log_file=/u01/lcm/provisioning/logs/slcn04cn10.example.com/idmautomation-configOAM.log, log_level=FINEST]{3}

    From this output you can see that the property file is called config_oam.properties. This file however, is moved from the location stated to the log directory. Examine this file and check that the entries IDSTORE_HOST/IDSTORE_PORT reference your load balancer/OTD directory entry (LBR_LDAP_HOST/LBR_LDAP_PORT).

  2. Validate that you can connect to the directory on the local host by telnetting to the LDAP_HOST and LDAP_PORT for example.

    telnet ldaphost1 1389

    If you see an entry similar to:

     Trying 10.245.169.148...
      Connected to slcn04cn10.example.com (10.245.169.148).
      Escape character is '^]'.

    Then you know that the directory was configured and is running.

    If it is not, the directory was not successfully configured, Check the standard directory log files for more information.

  3. Check that you can connect to the directory using the load balancer or OTD entry using LBR_LDAP_HOST and LBR_LDAP_PORT for example:

    telnet idstore.example.com 389

    If you don't see a connection, your load balancer or OTD instance is incorrectly configured. Recheck the configuration.

31.10.2.4 Deployment Fails on Install Phase with Permission Denied Error

Problem

During the Install phase, you may see an error similar to the following in the deployment log file:

[runIAMDeployment-install] [NOTIFICATION] [] 
[runIAMDeployment-install] [tid: 140] [ecid: 0000L3D9WwL72Fk5Gzl3if1ME9aO000013,0] 
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
oracle.idm.util.command.CommandException: Invalid shell command.[[
Message: cp: cannot create regular file
`/u01/lcm/lcmconfig/patch/patches/1446550807380/oam/21544485/etc/xml/GenericActions.xml: Permission denied
cp: cannot create regular file
`/u01/lcm/lcmconfig/patch/patches/1446550807380/oam/21544485/etc/xml/ShiphomeDirectoryStructure.xml: Permission denied

Solution

This occurs if the patch files have incorrect permissions. When you added patches to your repository, the patch files might not have write permissions which cause the patch manager process to fail.

To resolve the issue, change the permissions of the patch files in the repository to include write permission. For example:

chmod -R 755 REPOS_HOME/installers/iamsuite/patch

31.10.2.5 Deployment Fails While Configuring MSAS

Problem

The deployment fails in the Preconfig stage of the OAMHOST. The following error message is displayed in the LCM_HOME/provisioning/logs/oamhost1.example.com/configMSAS.log file:

 [wsm] [ERROR] [WSM-02381] 
 [oracle.wsm.resources.policymanager] [host: slc00drb] [nwaddr: 10.242.27.183] [tid: 1] [userId: 
user1] [ecid: 0000L4HZ^u717iK5IVG7yf1MIY1L000001,0] Unable to invoke method "post" of class 
com.sun.jersey.api.client.WebResource$Builder" with values "[Ljava.lang.Object;@497275cd"

Solution

This is caused by the IAD callback entry point being incorrectly configured. Ensure that iadinternal.example.com:port is correctly configured as described in Chapter 6, "Preparing the Load Balancer and Firewalls for an Enterprise Deployment".

31.10.2.6 Deployment Fails with Error: DiskSpaceCheck SEVERE Disk space check has failed

Problem

The IDMLCM tool health check verifies the disk space at IDM_TOP and fails, even though all the mounted storage shares under IDM_TOP have sufficient storage space, hence causing the deployment to fail.

Solution

The workaround for this issue is as follows:

  1. Look for the plugin definition with name "DiskSpaceCheck" in the IDMLCM_HOME/healthcheck/config/PreInstallChecks_mandatory.xml file, and comment out the plugin definition.

  2. Run the deployment again.

31.10.2.7 Preverify Inappropriately Fails with Insufficient Space

Problem

When preverify runs, it checks that sufficient space is available in the directory IDM_TOP. If you have created separate mount points for IDM_TOP/products and IDM_TOP/config, preverify does not add together the space allocated to the two mount points and fails the check inappropriately.

Solution

Disable the free space check by editing the file:

LCM_HOME/provisioning/idm-provisioning-build/idm-common-preverify-build.xml

Locate the entry:

    <target name="common-preverify-tasks">

Comment out the following entry so that after editing it looks like this:

            <!--antcall target="private-preverify-free-space"/-->

Save the file.

31.10.2.8 General Troubleshooting

Examine the log files in the directory LCM_HOME/provisioning/hostname. For example:

LCM_HOME/provisioning/hostname/runIAMDeployment-stage.log

This process identifies the cause of the failure.

If Pre-verify fails

If the pre-verify fails check this additional log file:

LCM_HOME/provisioning/hostname/healthchecker-preverify-error-check.log

31.10.3 Troubleshooting IDMLCM Start/Stop Scripts

This section describes some common problems related to Start/Stop scripts. It contains the following topics:

31.10.3.1 Start/Stop Scripts Fail to Start or Stop a Managed Server

Problem

Problem: Start/Stop scripts fail to start or stop a managed server.

The start/stop logs in the directory SHARED_CONFIG_DIR/scripts/logs contain an error similar to this:

weblogic.utils.AssertionError: ***** ASSERTION FAILED *****
        at weblogic.server.ServerLifeCycleRuntime.getStateRemote(ServerLifeCycleRuntime.java:734)
        at weblogic.server.ServerLifeCycleRuntime.getState(ServerLifeCycleRuntime.java:581)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Solution

  1. Shut down the failing managed server. You might have to kill the process.

  2. Back up the managed server's LDAP data, then remove it. For example:

    rm –rf LOCAL_CONFIG_DIR/domains/IAMAccessDomain/servers/server_name/data/ldap

    where server_name is the name of the failing managed server.

  3. Restart the managed server.

31.10.4 Troubleshooting Oracle Access Management Access Manager 11g

This section describes some common problems that can arise with Access Manager and the actions you can take to resolve the problem. It contains the following topics:

31.10.4.1 Access Manager Runs out of Memory

Problem

After Access Manager has been running for a while, you see the following error message in the output:

Attempting to allocate 1G bytes
There is insufficient native memory for the Java Runtime Environment to continue.

Possible reasons:

  • The system is out of physical RAM or swap space.

  • In 32 bit mode, the process size limit was reached.

Solutions

  • Reduce memory load on the system.

  • Increase physical memory or swap space.

  • Check if swap backing store is full.

  • Use 64 bit Java on a 64 bit OS.

  • Decrease Java heap size (-Xmx/-Xms).

  • Decrease number of Java threads.

  • Decrease Java thread stack sizes (-Xss).

  • Disable compressed references (-XXcompressedRefs=false).

  • Ensure that command line tool adrci can be executed from the command line.

    • at oracle.dfw.impl.incident.ADRHelper.invoke(ADRHelper.java:1309)

    • at oracle.dfw.impl.incident.ADRHelper.createIncident(ADRHelper.java:929

    • at oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl.createADRIncident(DiagnosticsDataExtractorImpl.java:1116)

  • On both OAMHOST1 and OAMHOST2, edit the file setSOADomainEnv.sh, which is located in IAD_MSERVER_HOME/bin and locate the line which begins:

    PORT_MEM_ARGS=

    Change this line so that it reads:

    PORT_MEM_ARGS="-Xms768m -Xmx2560m"

31.10.4.2 User Reaches the Maximum Allowed Number of Sessions

Problem

The Access Manager server displays an error message similar to this:

The user has already reached the maximum allowed number of sessions. Please close one of the existing sessions before trying to login again.

Solution

If users log in multiple times without logging out, they might overshoot the maximum number of configured sessions. You can modify the maximum number of configured sessions by using the Access Management Administration Console.

To modify the configuration by using the Access Management Administration Console, proceed as follows:

  1. Go to System Configuration -> Common Settings -> Session

  2. Increase the value in the Maximum Number of Sessions per User field to cover all concurrent login sessions expected for any user. The range of values for this field is from 1 to any number.

31.10.4.3 Policies Do Not Get Created When Oracle Access Management Access Manager is First Installed

Problem

The Administration Server takes a long time to start after configuring Access Manager.

Solution

Tune the Access Manager database. When the Administration server first starts after configuring Access Manager, it creates a number of default policies in the database. If the database is distant or in need of tuning, this can take a significant amount of time.

Resources
Authentication Policies
   Protected Higher Level Policy
   Protected Lower Level Policy
   Publicl Policy
Authorization Policies
   Authorization Policies

If you do not see these items, the initial population has failed. Check the Administration Server log file for details.

31.10.4.4 You Are Not Prompted for Credentials After Accessing a Protected Resource

Problem

When you access a protected resource, Access Manager should prompt you for your user name and password. For example, after creating a simple HTML page and adding it as a resource, you should see credential entry screen.

Solution

If you do not see the credential entry screen, perform the following steps:

  1. Verify that Host Aliases for IAMAccessDomain have been set. You should have aliases for IAMAccessDomain:80, IAMAccessDomain:Null, IADADMIN.example.com:80, and login.example.com:443, where Port 80 is HTTP_PORT and Port 443 is HTTP_SSL_PORT.

  2. Verify that WebGate is installed.

  3. Verify that ObAccessClient.xml was copied from IAD_ASERVER_HOME/output to the WebGate Lib directory and that OHS was restarted.

  4. When ObAccessClient.xml was first created, the file was not formatted. When the OHS is restarted, reexamine the file to ensure that it is now formatted. OHS gets a new version of the file from Access Manager when it first starts.

  5. Shut down the Access Manager servers and try to access the protected resource. You should see an error saying Access Manager servers are not available. If you do not see this error, re-install WebGate.

31.10.4.5 Cannot Log In to Access Management Console

Problem

You cannot log in to the Access Management Console. The Administration Server diagnostic log might contain an error message similar to this:

Caused by: oracle.security.idm.OperationFailureException:
oracle.security.am.common.jndi.ldap.PoolingException [Root exception is oracle.ucp.UniversalConnectionPoolException:
Invalid life cycle state.
 Check the status of the Universal Connection Pool]
         at
oracle.security.idm.providers.stdldap.UCPool.acquireConnection(UCPool.java:112)

Solution

Remove the /tmp/UCP* files and restart the Administration Server.

31.10.4.6 Oracle Coherence Cluster Startup Errors in WLS_AMA Server Logs

Problem

The WLS_AMA2 server has oam application deployment in failed state. The WLS_AMA2 server logs report request timeout exceptions while starting the cluster service, similar to following logs:

Oracle Coherence GE 3.7.1.13 <Warning> (thread=Cluster, member=n/a): Delaying 
formation of a new cluster; IpMonitor failed to verify the reachability of senior 
Member(Id=1, Timestamp=, Address=, MachineId=,
Location=site:,machine:iadadminvhn,process:8499, Role=WeblogicServer); if this 
persists it is likely the result of a local or remote firewall rule blocking
either ICMP pings, or connections to TCP port 7>

Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout 
during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=MasterMemberSet(
ThisMember=null
OldestMember=null
ActualMemberSet=MemberSet(Size=0
)
MemberId|ServiceVersion|ServiceJoined|MemberState
RecycleMillis=1200000
RecycleSet=MemberSet(Size=0
)
)
)
at
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:3)

at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)

at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)

Solution

This is a known issue. In some of the environments, the Access Policy Manager Server that is not running on the same host as the WebLogic Administration Server is unable to start the coherence cluster service, which results in the oam application deployment to be in failed state. To solve this issue, you must create a server instance for the effected Access Policy Manager Server by completing the following steps:

  1. Log in to the OAM console using the following URL:

    http://iadadmin.example.com/oamconsole

    Log in as the Access Manager administration user you created when you prepared the ID Store. For example, oamadmin.

  2. Click Configuration.

  3. Click Server Instances from the configuration launch pad.

  4. Click a new server instance for the Access Policy Manager WebLogic Managed Server, that is not running on the same machine as the IAMAccessDomain Admin Server. For example:

    • Name: WLS_AMA2

    • Port: 14150

    • Host: OAMHOST2 (For consolidated topology, the host will be IAMHOST2)

    Note:

    Provide the OAM Proxy details similar to the server instance for WLS_OAM.

  5. Click Apply.

31.10.4.7 Errors in log File when Starting OAM Servers

Problem

When you start the OAM Servers, errors similar to the following are seen in the log files which causes LCM heath check module to fail:

[wls_oam1] [TRACE:16] [] [oracle.oam.config] [tid: DistributedCacheWorker:4] [userId: <anonymous>] [ecid: 
0000LGmRJqxB9DE5N7P5ie1N5mOd000004,1:16514] [APP: oam_server#11.1.2.0.0] [SRC_CLASS: oracle.security.am.admin.config.util.MapUtil] [SRC_METHOD: 
getDefaultedStringValue] property not found at path:[Ljava.lang.String;@43537067 Defaulting to value:,
[2016-04-20T06:55:39.982+00:00] [wls_oam1] [TRACE:16] [] [oracle.oam.config] [tid: DistributedCacheWorker:4] [userId: <anonymous>] [ecid: 
0000LGmRJqxB9DE5N7P5ie1N5mOd000004,1:16514] [APP: oam_server#11.1.2.0.0] [SRC_CLASS: oracle.security.am.admin.config.util.MapUtil] [SRC_METHOD: getStringValue] THROW[[
oracle.security.am.admin.config.ConfigurationException: Cannot get java.lang.String value from configuration for key ResponseEscapeChar. Object null found.
at oracle.security.am.admin.config.util.MapUtil.handleFailedAttributeAccess(MapUtil.java:447)
at oracle.security.am.admin.config.util.MapUtil.getStringValue(MapUtil.java:130)
at oracle.security.am.admin.config.util.MapUtil.getDefaultedStringValue(MapUtil.java:147)
at oracle.security.am.engines.common.identity.provider.util.IdStoreConfig.initializeConfig(IdStoreConfig.java:76)
at oracle.security.am.engines.common.identity.provider.util.IdStoreConfig.<init>(IdStoreConfig.java:69)
at oracle.security.am.engines.common.identity.provider.util.IdStoreConfig.getConfig(IdStoreConfig.java:128)
at oracle.security.am.engines.common.identity.util.OAMUserAttribute.getStringValue(OAMUserAttribute.java:76)
at oracle.security.am.engines.common.identity.util.OAMUserAttribute.toString(OAMUserAttribute.java:114)
at java.lang.String.valueOf(String.java:2849)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at java.util.AbstractMap.toString(AbstractMap.java:523)
at java.lang.String.valueOf(String.java:2849)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at oracle.security.am.engines.common.identity.util.OAMIdentity.toString(OAMIdentity.java:678)
at java.lang.String.valueOf(String.java:2849)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at oracle.security.am.engines.sso.SSOSubject.toString(SSOSubject.java:238)
at java.lang.String.valueOf(String.java:2849)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at oracle.security.am.engines.sme.impl.SessionImpl.toString(SessionImpl.java:629)
at java.lang.String.valueOf(String.java:2849)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at oracle.security.am.engines.sme.mapimpl.db.DbOraSmeStore.loadSession(DbOraSmeStore.java:1705)
at oracle.security.am.engines.sme.mapimpl.db.DbOraSmeStore.loadSession(DbOraSmeStore.java:1691)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at oracle.security.am.foundation.mapimpl.coherence.store.DataConnectionUtility.invokeSqlOperationWithRetries(DataConnectionUtility.java:275)
at oracle.security.am.engines.sme.mapimpl.db.DbOraSmeStore.load(DbOraSmeStore.java:1284)
at com.tangosol.net.cache.ReadWriteBackingMap$CacheStoreWrapper.loadInternal(ReadWriteBackingMap.java:5676)
at com.tangosol.net.cache.ReadWriteBackingMap$StoreWrapper.load(ReadWriteBackingMap.java:4754)
at com.tangosol.net.cache.ReadWriteBackingMap.get(ReadWriteBackingMap.java:717)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$Storage.get(PartitionedCache.CDB:10)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onGetRequest(PartitionedCache.CDB:23)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$GetRequest.run(PartitionedCache.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:66)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
at java.lang.Thread.run(Thread.java:745)
]]

Solution

This occurs when OAM servers cannot communicate with each other using the coherence port. This is often caused by iptables. The workaround for this issue is as follows:

  1. Edit the file /etc/sysconfig/iptables on both OAMHOST1 and OAMHOST2 and add the following line:

    # Generated by iptables-save v1.4.7 on Tue Apr 19 10:02:45 2016
*filter
:INPUT ACCEPT [593:243587]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [614:423013]
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9095 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9097 -j ACCEPT
COMMIT

    In the above set of lines, 9095 and 9097 are the coherence ports being used.

  2. Save the file and restart the servers.

31.10.5 Troubleshooting Oracle Identity Manager

This section describes some common problems that can arise with Oracle Identity Manager and the actions you can take to resolve the problem. It contains the following topics:

31.10.5.1 java.io.FileNotFoundException When Running Oracle Identity Manager Configuration

Problem

When you run Oracle Identity Manager configuration, the error java.io.FileNotFoundException: soaconfigplan.xml (Permission denied) may appear and Oracle Identity Manager configuration might fail.

Solution

To workaround this issue:

  1. Delete the file /tmp/soaconfigplan.xml.

  2. Start the configuration again (OH/bin/config.sh).

31.10.5.2 ResourceConnectionValidationxception When Creating User in Oracle Identity Manager

Problem

If you are creating a user in Oracle Identity Manager (by logging into Oracle Identity Manager System Administration Console, clicking the Administration tab, clicking the Create User link, entering the required information in the fields, and clicking Save) in an active-active Oracle Identity Manager configuration, and the Oracle Identity Manager server that is handling the request fails, you may see a "ResourceConnectionValidationxception" in the Oracle Identity Manager log file, similar to:

[2010-06-14T15:14:48.738-07:00] [oim_server2] [ERROR] [] [XELLERATE.SERVER]
[tid: [ACTIVE].ExecuteThread: '0' for queue: 'weblogic.kernel.Default
(self-tuning)'] [userId: xelsysadm] [ecid:
004YGJGmYrtEkJV6u3M6UH00073A0005EI,0:1] [APP: oim#11.1.1.3.0] [dcid:
12eb0f9c6e8796f4:-785b18b3:12938857792:-7ffd-0000000000000037] [URI:
/admin/faces/pages/Admin.jspx] Class/Method:
PooledResourceConnection/heartbeat encounter some problems: Operation timed
out[[
com.oracle.oim.gcp.exceptions.ResourceConnectionValidationxception: Operation
timed out
        at
oracle.iam.ldapsync.impl.repository.LDAPConnection.heartbeat(LDAPConnection.ja
va:162)
        at
com.oracle.oim.gcp.ucp.PooledResourceConnection.heartbeat(PooledResourceConnec
tion.java:52)
         .
         .
         .

Solution

Despite this exception, the user is created correctly.

31.10.5.3 Oracle Identity Manager Reconciliation Jobs Fail

Problem

Oracle Identity Manager reconciliation jobs fail, or one of the following messages is seen in the log files:

  • Error-1

    LDAP Error 53 : [LDAP: error code 53 - Full resync required. Reason: The provided cookie is older than the start of historical in the server for the replicated domain : dc=example,dc=com]

  • Error-2

    LDAP: error code 53 - Invalid syntax of the provided cookie

This error is caused by the data in the Oracle Unified Directory change log cookie expiring because Oracle Unified Directory has not been written to for a certain amount of time.

Solution:

  1. Open a browser and go to the following location:

    http://igdadmin.example.com/sysadmin

  2. Log in a as xelsysadm using the COMMON_IDM_PASSWORD.

  3. Under System Management, click Scheduler.

  4. Under Search Scheduled Jobs, enter LDAP * (there is a space before *) and hit Enter.

  5. For each job in the search results, click on the job name on the left, then click Disable on the right.

    Do this for all jobs. If the job is already disabled do nothing.

  6. Run the following commands on LDAPHOST1:

    cd LDAP_ORACLE_INSTANCE/OUD/bin
./ldapsearch -h ldaphost1 -p 1389 -D "cn=oudadmin" -b "" -s base "objectclass=*" lastExternalChangelogCookie

Password for user 'cn=oudadmin': <OudAdminPwd>
dn: lastExternalChangelogCookie: dc=example,dc=com:00000140c682473c263600000862;

    Copy the output string that follows lastExternalChangelogCookie:. This value is required in the next step. For example,

    dc=example,dc=com:00000140c682473c263600000862;

    The Hex portion must be 28 characters long. If this value has more than one Hex portion then separate the 28char portions with spaces. For example:

    dc=example,dc=com:00000140c4ceb0c07a8d00000043 00000140c52bd0b9104200000042 00000140c52bd0ba17b9000002ac 00000140c3b290b076040000012c;

  7. Run each of the following LDAP reconciliation jobs once to reset the last change number.:

    • LDAP Role Delete Reconciliation

    • LDAP User Delete Reconciliation

    • LDAP Role Create and Update Reconciliation

    • LDAP User Create and Update Reconciliation

    • LDAP Role Hierarchy Reconciliation

    • LDAP Role Membership Reconciliation

    To run the jobs:

    1. Login to the OIM System Administration Console as the user xelsysadm.

    2. Under System Configuration, click Scheduler.

    3. Under Search Scheduled Jobs, enter LDAP * (there is a space before *) and hit Enter.

    4. Click on the job to be run.

    5. Set the parameter Last Change Number to the value obtained in step 6.

      For example:

      dc=example,dc=com:00000140c4ceb0c07a8d00000043 00000140c52bd0b9104200000042 00000140c52bd0ba17b9000002ac 00000140c3b290b076040000012c;

    6. Click Run Now.

    7. Repeat for each of the jobs in the list at the beginning of this step.

  8. For each incremental recon job whose last changelog number has been reset, execute the job and check that the job now completes successfully.

  9. After the job runs successfully, re-enable periodic running of the jobs according to your requirements.

If the error appears again after the incremental jobs have been re-enabled and run successfully ("Full resync required. Reason: The provided cookie is older..."), then increase the OUD cookie retention time. Although there is no hard and fast rule as to what this value should be, it should be long enough to avoid the issue, but small enough to avoid unnecessary resource consumption on OUD. One or two weeks should suffice. Run the following command on each OUD instance to increase the retention time to two weeks:

cd OUD_ORACLE_INSTANCE/bin

./dsconfig set-replication-server-prop --provider-name "Multimaster Synchronization" --set replication-purge-delay:2w -D cn=oudadmin --trustAll -p 4444 -h LDAPHOSTn

Password for user 'cn=oudadmin':  <OudAdminPswd>
Enter choice [f]: f

31.10.5.4 OIM Reconciliation Jobs Fail When Running Against Oracle Unified Directory

Problem: Reconciliation jobs fail when running against Oracle Unified Directory (OUD). The following error is seen in the OIM WebLogic Server logs:

LDAP: error code 53 - Invalid syntax of the provided cookie

Solution: Try out the workaround described in Section 31.10.5.3, "Oracle Identity Manager Reconciliation Jobs Fail". If that does not resolve the issue, try the following solution:

On each OIMHOST, update the IGD_MSERVER_HOME/config/fmwconfig/ovd/oim/adapters.os_xml file with the following parameter:

<param name="eclCookie" value="false"/>

Restart the OIM and SOA Managed Servers.

31.10.5.5 Cannot Open Reports from OIM Self Service Console

Problem: The reports cannot be opened from OIM Self Service Console.

Solution: When you enable the Identity Auditor feature in OIM, do the following configuration changes for the OIM-BI Publisher integration to work fine:

  1. Log in to the IAMGovernanceDomain Enterprise Management console.

  2. Open the system MBean browser and update the MBean "oracle.iam:Location=wls_oim1,name=Discovery,type=XMLConfig.DiscoveryConfig,XMLConfig=Config,Application=oim,ApplicationVersion=11.1.2.0.0" with Value as http://igdadmin.example.com/.

    Here, igdadmin.example.com is the Governance Domain admin Load balancer URL.

31.10.6 Troubleshooting Oracle SOA Suite

This section describes some common problems that can arise with Oracle SOA Suite and the actions you can take to resolve the problem. It contains the following topics:

31.10.6.1 Transaction Timeout Error

Problem: The following transaction timeout error appears in the log:

Internal Exception: java.sql.SQLException: Unexpected exception while enlisting
 XAConnection java.sql.SQLException: XA error: XAResource.XAER_NOTA start()
failed on resource 'SOADataSource_soaedg_domain': XAER_NOTA : The XID
is not valid

Solution: Check your transaction timeout settings, and be sure that the JTA transaction time out is less than the DataSource XA Transaction Timeout, which is less than the distributed_lock_timeout (at the database).

With the out of the box configuration, the SOA data sources do not set XA timeout to any value. The Set XA Transaction Timeout configuration parameter is unchecked in the WebLogic Server Administration Console. In this case, the data sources use the domain level JTA timeout which is set to 30. Also, the default distributed_lock_timeout value for the database is 60. As a result, the SOA configuration works correctly for any system where transactions are expected to have lower life expectancy than such values. Adjust these values according to the transaction times your specific operations are expected to take.

31.10.7 General Troubleshooting

This section describes the common issues and their workaround. This section includes the following topic:

31.10.7.1 Cannot Start Managed Server from WebLogic Console

When you start a Managed Server from the WebLogic Console, the following error is shown:

. For server WLS_BI1, the Node Manager associated with machine oimhost1 is not reachable.
. All of the servers selected are currently in a state which is incompatible with this operation or are not associated with a running Node Manager or you are not authorized to perform the action requested. No action will be performed.

Solution 1

Check if the Node Manager is started on the target host. If not, start it.

Solution 2

Verify that the domain is listed in the file nodemanager.domains, which is located in the directory SHARED_CONFIG_DIR/nodemanger/hostname. If not, do the following:

  1. Start the WebLogic Scripting Tool (WLST) by running the following command from the location MW_HOME/oracle_common/common/bin/:

    ./wlst.sh

  2. Connect to the domain you wish to add by running the following command:

    connect('weblogic_user','password','t3://ADMINVHN:AdminPort')

    In this command:

    weblogic_user is the WebLogic Administration user. For example, weblogic or weblogic_idmw.

    password is the password of the WebLogic Administration user.

    ADMINVHN is the Virtual host name of the Administration Server. For example, IGDADMINVHN or IADADMINVHN.

    adminPort is the port on which the Administration Server is running. For example, 7101.

    Sample Command:

    connect('weblogic_idm','mypasswd','t3://igdadminvhn.example.com:7001')

  3. Enrol the domain using the following command:

    nmEnroll(domainDir=absolute_path_to_the_domain,nm_Home=absolute_path_to_the_nodemanager_home)

    For example:

    nmEnroll(domainDir='/u02/private/oracle/config/domains/IAMGovernanceDomain/',nmHome='/u01/oracle/config/nodemanger/hostname)')

    Note:

    For Managed Servers, the domain home should always be specified as the local managed server directory.

31.10.7.2 Proxy Settings are Reset

Proxy settings are reset after changing via the MSM console. This happens because there is a different configuration file for the admin server and the managed servers.

Solution

  1. Locate the msm-config.xml file present under the $IAD_ASERVER_HOME/config/fmwconfig directory.

  2. Remove or rename the msm-config.xml file.