C Troubleshooting

The following topics provide troubleshooting tips:

• Introduction to Oracle Access Management Troubleshooting

• My Oracle Support for Additional Troubleshooting Information

• SQL Queries to List Sessions and Plugins from Database

• Oracle Access Management Console Inconsistent State

• AdminServer Won't Start if the Wrong Java Path Given with WebLogic Server Installation

• Agent Naming Not Unique

• Application URL Requirements

• Authentication Issues

• Authorization Issues

• Cannot Access Authentication LDAP or Database

• Cannot Find Configuration

• OAM unsupports Whole Server Migration

• Could Not Find Partial Trigger

• Denial of Service Attacks

• Disabling Windows Challenge/Response Authentication on IIS Web Servers

• Changing UserIdentityStore1 Type Can Lock Out Administrators

• IIS Web Server Issues

• jps Logger Class Instantiation Warning is Logged on Authentication

• Internationalization, Languages, and Translation

• Login Failure for a Protected Page

• OAM Metric Persistence Timer IllegalStateException: SafeCluster

• Partial Cluster Failure and Intermittent Login and Logout Failures

• RSA SecurID Issues and Logs

• Registration Issues

• Rowkey does not have any primary key attributes Error

• SELinux Issues

• Session Issues

• SSL versus Open Communication

• Start Up Issues

• Synchronizing OAM Server Clocks

• Time delay in configuration change

• Validation Errors

• Web Server Issues

• Windows Native Authentication

• WLST Commands for Multi-Data Centers

• Comparing Default Parameters and Values used in MDC Configuration for 11g and 12c

C.1 Introduction to Oracle Access Management Troubleshooting

Oracle Access Management is a business critical system; downtime comes with a potentially high cost to your business. The goal of system analysis is to quickly isolate and correct the cause of any problem. This requires a big picture view of your system and the tools to observe the live system and correlate components to the bigger picture.

To assist Administrators in performing a quick diagnosis, this section provides the following topics:

• System Analysis and Problem Scenarios

• LDAP Server or Identity Store Issues

• OAM Server or Host Issues

• Agent-Side Configuration and Load Issues

• Runtime Database (Audit or Session Data) Issues

• Change Propagation or Activation Issues

• Policy Store Database Issues

C.1.1 System Analysis and Problem Scenarios

System analysis includes understanding how the product works, what can go wrong, how likely the scenarios are, and the consequences or observable issues.

System problems can be divided into two basic categories:

• Cascading catastrophic failure

• Gradual breakdown in performance

Cascading catastrophic failure might be caused by:

• LDAP server is loaded and unresponsive

• Morning peak load starts

• Webgates send requests to the primary OAM Server

• Webgate requests time-out and Webgates retry to secondary OAM Server

Gradual breakdown in performance might occur over time when, for example:

• OAM is sized and rolled out for 10,000 users and 500 groups

• Over the course of a year, the number of users and groups increases significantly (to 50,000 users and 250 groups for example)

For information on the most commonly encountered issues, see the following topics:

• System Analysis and Problem Scenarios

• LDAP Server or Identity Store Issues

• OAM Server or Host Issues

• Agent-Side Configuration and Load Issues

• Runtime Database (Audit or Session Data) Issues

• Change Propagation or Activation Issues

• Policy Store Database Issues

C.1.2 LDAP Server or Identity Store Issues

This topic provides symptoms, probable cause, and steps to diagnose the following issues:

• Symptoms: Operational Slowness

• Symptoms: Total loss of service

Symptoms: Operational Slowness

• Poor user experience

• Agent time outs lead to retries

Cause

• Non-OAM load might be impacting OAM operations

• Capacity problems due to gradual increase in peak load

Symptoms: Total loss of service

Total loss of service

Cause

• Outage of all LDAP servers

• The load balancer is timing out old connections

Diagnosis

1. Shut down the LDAP server.

2. Restart your browser.

3. Try to access a protected site.

4. Review errors in the OAM Server log file, as described in Logging Component Event Messages (alternatively, in Monitoring Performance and Logs with Fusion Middleware Control).

5. Try to access Oracle Access Management Console.

6. Observe errors in WebLogic AdminServer log file.

7. Bring up the LDAP server again.

8. Retry access to a protected application.

9. Retry access to the Oracle Access Management Console.

10. Correct the issue based on the requirements in your environment.

C.1.3 OAM Server or Host Issues

This topic provides symptoms, probable cause, and steps to diagnose the following issues:

• Symptoms: Capacity Problems

• Symptoms: Interference with Other Services on the Host

Symptoms: Capacity Problems

• Poor user experience due to slow operations

• Agent time outs and retry can result in extra load

Cause

• CPU cycles

• Memory issues

Symptoms: Interference with Other Services on the Host

• Poor user experience due to slow operations

• Agent time outs and retry may result in extra load

Cause

• CPU cycle contention

• Memory contention

• File system full

Diagnosis: OAM Server

1. Shut down the OAM Server

2. Try to access a Webgate

3. Bring up the OAM Server

4. Use the Access Tester to test authentication and authorization as described in Validating Connectivity and Policies Using the Access Tester.

5. Use 'top' to figure out the CPU and Memory consumption of the OAM Server as you use the access tester

6. Get a thread dump of the OAM Server.

Diagnosis: AdminServer

1. Shut down the AdminServer.

2. Restart your browser and access a protected resource, which should work.

3. Use remote registration to register a new partner, as described in Registering and Managing OAM Agents (this should fail).

4. Startup OAM AdminServer.

C.1.4 Agent-Side Configuration and Load Issues

This topic provides symptoms, probable cause, and steps to diagnose time issues between agents and servers.

Symptoms

Difference in Clock time Between Agent and Server

• High CPU usage at both agent and server

• User experiences a system hang

Cause

• Agent thinks the token issued by the server is invalid

• Agent keeps going back to the server to re-issue the token

Diagnosis

1. Access protected resource.

2. Confirm: Client access hangs.

3. Confirm: High CPU usage on agent and server.

C.1.5 Runtime Database (Audit or Session Data) Issues

The audit and session functions are both write intensive operations. The policy database can be tuned for read intensive service.

Symptoms

• Audit and session operations are slow

• File system on the OAM Server is full with audit data that is not yet written to the database

• Loss of in-memory session data when one of the servers in the cluster fails

Cause

• Database is not tuned for write intensive operations

• Database is unavailable due to maintenance

• Space issues in the database

Diagnosis

1. Shut down the database used to store Audit and Session data.

2. Try to access a protected resource.

3. Review error and warning messages in the OAM Server log files, as described in Logging Component Event Messages (alternatively, in Monitoring Performance and Logs with Fusion Middleware Control).

C.1.6 Change Propagation or Activation Issues

This topic provides symptoms, probable cause, and steps to diagnose the following issues:

Symptoms

• Changes to policy do not take immediate effect

• Changes to system configuration do not take immediate effect

Cause

• Servers being too busy handling runtime requests (CPU contention)

Diagnosis

See "Policy Store Database Issues"

C.1.7 Policy Store Database Issues

This topic provides symptoms, probable cause, and steps to diagnose policy database issues.

Symptoms

No policy changes are allowed; no impact on runtime

Cause

• Database is unavailable (down for maintenance)

• Space issues in the database

Diagnosis

1. Shut down the database containing OAM policies.

2. Try to access a protected resource and observe the runtime access is not impacted.

3. Try to access the Oracle Access Management Console to edit policies, and then observe errors in the AdminServer log file.

C.2 My Oracle Support for Additional Troubleshooting Information

You can use My Oracle Support (formerly MetaLink) to help resolve Oracle Fusion Middleware problems. My Oracle Support contains several useful troubleshooting resources, such as:

• Knowledge base articles

• Community forums and discussions

• Patches and upgrades

• Certification information

Note:

You can also use My Oracle Support to log a service request.

You can access My Oracle Support at https://support.oracle.com.

C.3 SQL Queries to List Sessions and Plugins from Database

Sessions and plugins are stored in the Database; use SQL queries to list them.

List Sessions and Plugins	SQL queries
List Session from database	SELECT * FROM AM_SESSION
List plugins from database	SELECT * FROM OAM_FILE_ARTIFACTS WHERE path='/oam/plugins/'

C.4 Administrator Lockout

Problem

Administrator cannot successfully log in to the Oracle Access Management Console. The following message appears:

Manually Change Identity Store Settings at OPSS Level and configure the IDMDOmainAgent.

Cause

Access Manager secures the Oracle Access Management Console based on authentication information in the IAM Suite Application Domain: OAM Admin Console Policy. This policy relies on a single Authentication Scheme (OAMAdminConsoleScheme), which uses a Form challenge method and LDAP Authentication Module. The LDAP Authentication Module must be pointing to the User Identity Store designated as the System Store.

If, for example, your deployment is configured to use Oracle Internet Directory (with all Administrators, users, and groups defined therein) ensure that the LDAP Authentication Module points to this user identity store and that this is designated as the System Store.

Solution

1. Insert a user identity into both your designated system store and the Embedded LDAP store.

2. Log in to Oracle Access Management Console.

3. Configure the LDAP Authentication Module used by the designated System Store to point to the appropriate User Identity Store, as described in "Managing Native Authentication Modules".

C.5 Error During Federation Configuration After Upgrade from PS1 to PS2

IAM Suite is the OOTB Application Domain created when OAM 11.1.2 is installed. This Application Domain can be renamed after installation but when upgrading OAM to 11.1.2.2.0, it must be renamed back to IAM Suite. If it has been renamed, the upgrade operation will fail with this error seen in the WLS admin logs.

java.lang.NullPointerException
at
oracle.security.am.common.policy.tools.upgrade.r2ps2.bootstrap.FedR2PS2Bootstr
apHandler.createFedAuthnResource(FedR2PS2BootstrapHandler.java:505)
at
oracle.security.am.common.policy.tools.upgrade.r2ps2.bootstrap.FedR2PS2Bootstr
apHandler.doBootstrap(FedR2PS2BootstrapHandler.java:151)
at
oracle.security.am.common.policy.tools.upgrade.r2ps2.bootstrap.R2PS2BootstrapH
elper.doBootstrap(R2PS2BootstrapHelper.java:70)
at
oracle.security.am.common.policy.tools.PolicyComponentLifecycle.initialize(Pol
.
icyComponentLifecycle.java:99)

If the IAM Suite Application Domain has been renamed, it is required to rename it back to its original IAM Suite name prior to beginning the upgrade process. After the upgrade, the name can be changed back to a custom name.

C.6 Oracle Access Management Console Inconsistent State

Problem

Administrators performing updates concurrently will result in an inconsistent state within the system configuration of the Oracle Access Management Console.

Cause

Concurrent configuration updates are not supported.

Solution

Only one Administrator should be allowed to modify the system configuration at any given time.

C.7 AdminServer Won't Start if the Wrong Java Path Given with WebLogic Server Installation

WebLogic Server (wls1035_generic) installation is successful on Windows 64-bit with 32-bit Java (jdk1.6.0_24). When setup.exe is executed you must provide the path of the 64-bit java (jdk1.6.0_23) to successfully launch the install shield.

If you provide the 32-bit Java (jdk1.6.0_24) path, the install shield is not launched. However, if you execute config.cmd from \Middleware\Oracle_IDM1\common\bin, by default 32-bit Java (jdk1.6.0_24) path is used, but after successful installation Access Manager installation, you cannot start AdminServer.

On Windows host, the path to 32-bit JAVA_HOME (c:\Program files (x86)\java\jdkxxx) is not correctly handled by the startWeblogic.cmd. Replacing SUN_JAVA_HOME to use the path with the shorter name (c:\progra~2\java\jdkxxxx) works fine.

On Windows, the shorter names can be seen by executing "dir /X".

Alternatively, you can set windows cmd shell variable JAVA_HOME to path with shorter name and execute startWeblogic.cmd within that. For example:

>set JAVA_HOME=c:\progra~2\java\jdkXXX

>startweblogic.cmd

C.8 Agent Naming Not Unique

A unique identifying name for each Agent registration is preferred. However:

• If the Agent Name exists, no error occurs and the registration does not fail. Instead, Access Manager creates the policies if they are not already in place.

• If the host identifier exists, the unique Agent Base URL is added to the existing host identifier and registration proceeds.

C.9 Application URL Requirements

The number of characters allowed in a URL are based on browser version.

The main attribute that affects the size of a cookie is the length of the requested URL. Some of the system generated URLs for ADF applications are quite long and can cause the cookie to exceed the maximum size.

Another case is when using custom plug-ins. The data that a plug-in adds to the authentication context is persisted in the cookie and can cause the cookie size to grow.

Multiple wrong password attempts can also add more context data to the cookie. Combined with one of the above cases, the cookie size can rapidly grow.

Solutions

Ensure that your applications do not use URLs that exceed the length that Access Manager and the browser can handle.

The cookie cache mode can be changed to FORM mode from default COOKIE mode. FORM mode works with long URLs. The only difference in behavior is for programmatic authentication, which requires a proper form Submit to pass the OAM_REQ parameter set to the form. Custom credential collection pages need to handle the OAM_REQ parameter that is submitted with the form.

Also, to support long URLs, set the serverRequestCacheType parameter to FORM using the configRequestCacheType WLST command. For more information about its affect on ECC or DCC configurations, see  TempStateMode in Table 22-22

C.10 Authentication Issues

This section provides the following information:

• Anonymous Authentication Issues

• X.509Scheme and SSL Handshake Issues

• X.509 Protected Resource and Single Sign Off

• X509CredentialExtractor Certificate Validation Error

C.10.1 Anonymous Authentication Issues

Problem

Challenge Redirect URL can be NULL; however, Challenge Method cannot be NULL.

If you open the Anonymous authentication scheme to edit, and click Apply without adding a value for Challenge method, the following errors might appear:

Messages for this page are listed below. 

* Challenge Method You must make at least one selection. 
 
* Challenge Redirect You must enter a value. 

Solution

You must include both a challenge method and a challenge redirect whenever you edit an anonymous authentication scheme.

C.10.2 X.509Scheme and SSL Handshake Issues

The Access Manager X.509 Authentication Scheme relies on SSL to deliver the user's X.509 certificate to the OAM Server. The X.509 Authentication Scheme requires the X.509Plugin as the value of the Challenge Method (not the Authentication Module).

Problem

User has selected his certificate in the Browser but the Certificate is not available to the OAM Server.

Solution

The specific solution will depend on the reason for the SSL Handshake failure. For instance:

• For debugging SSL connections terminating on the Weblogic Server, please refer to http://docs.oracle.com/cd/E12840_01/wls/docs103/secmanage/ssl.html

• For debugging SSL connections terminating on the OHS server, see http://docs.oracle.com/cd/E12839_01/web.1111/e10144/under_mods.htm#i1007687.

Determine the reason for the SSL Handshake failure and the peer that is terminating the SSL Handshake. The solution will fall into the following categories:

• Configuration Issues

• Trust Issues

• Certificate Validation Issues

C.10.2.1 Configuration Issues

If you are encountering problems establishing a SSL connection with the default WebLogic server SSL implementation, switch to using the JSSE SSL implementation which is supported with WLS 10.3.3+.

The following list identifies other possible configuration issues.

• OHS plugin is incorrectly configured and not sending the user certificate to the WebLogic server.

• Cipher suites: As configured, are not compatible with the user certificate.

• Smart cards: The browser is not communicating with the smart card reader.

• PKCS#11 (or hardware cryptography): Ensure that the devices are in working order.

C.10.2.2 Trust Issues

The server name within the certificate does not match the host name. This check can be disabled through configuration.

The server does not contain a CA certificate on the user certificate path in its trust store.

C.10.2.3 Certificate Validation Issues

The following list identifies possible configuration issues.

• Certificate has expired.

• Certificate has been revoked.

• Certificate validation is not working because this is incorrectly configured or there are connectivity issues.

C.10.3 X.509 Protected Resource and Single Sign Off

Problem

Single Sign Off might not work after accessing the resource with X.509 authentication. When the user is logged out with the logout URL and tries to access the resource in the same browser, authentication might not occur. Instead, the user should be asked for authentication using the certificate pop up.

This can occur with any Agent type.

Solution

After executing the logout URL, click on Clear SSL State from the browser as follows, and the access the X.509-protected resource:

From the browser window, open the Tools menu, click Internet Options, choose Content, and then Clear SSL state.

C.10.4 X509CredentialExtractor Certificate Validation Error

Problem

Client certificate authentication works fine using the standard X509 Authentication Module after importing the root and sub CA certificates into the WebLogic Server and .oamkeystore keystores.

However, a certificate validation error can occur when using a Custom X509Plugin Authentication Module and root and sub CA certificates into the WebLogic Server and .oamkeystore keystores.

Solution

With the Custom X509Plugin Authentication Module the root and sub CA certificates must be added to the DOMAIN_HOME/config/fmwconfig/amtruststore because the X509CredentialExtractor plug-in loads certificates from this location.

C.11 Authorization Issues

This section provides the following topics:

• Authorization Condition Error

• LDAP Search Filter Test Results

• Authorization Header Response Names

C.11.1 Authorization Condition Error

An error is logged in the oam-server diagnostic log file whenever you create or edit an IPv4 range or temporal condition:

.... refreshPolicy specified but no response collector supplied 

Cause

This is a message that is erroneously being logged at the ERROR level.

Solution

The correct level of the message is INFO.

C.11.2 LDAP Search Filter Test Results

If too many results are returned, you are informed as follows:

Description of the illustration id_ldapfilter_results.gif

Solution

1. Click OK.

2. Click Test Filter to initiate a new test.

3. In the Edit Search Filter dialog, make your changes.

4. Check the Test Results.

C.11.3 Authorization Header Response Names

Some characters might not be usable within header response names or values, depending on whether the client receiving these responses is a Webgate, and if so which Web server is protected. Certain characters might be subject to automatic conversion to other characters in a server-specific way.

Oracle recommends that you refer to your Web server documentation for more details.

C.12 Cannot Access Authentication LDAP or Database

If the LDAP directory that is used for authentication is down or inaccessible (or the database that is configured as the policy store), it might be due to a heavy load or a timeout. You see a message when attempting to a protected resource that uses this LDAP or policy store.

Solution

1. Manually shut down the registered LDAP or database.

2. Restart the registered LDAP or database.

C.13 Cannot Find Configuration

This section lists the following errors and provides a solution or a workaround.

• Configuration Does Not Exist ...

C.13.1 Configuration Does Not Exist ...

If you attempt to create and apply configuration details for an OAM Server before configuring the OAM Server in the WebLogic Server domain, a message informs you of the following:

Configuration does not exist for path
/DeployedComponent/Server/oamServer/Instance/test

For more information, please see the server's error log for
an entry beginning with: Server Exception during PPR, #6.

To resolve this issue, you must configure the OAM Server in the WebLogic Server domain before you register the configuration with Access Manager.

C.14 OAM unsupports Whole Server Migration

Whole server migration is not supported for retaining virtual IP with OAM.

Solution

Whole server migration is not supported for OAM.

C.15 Could Not Find Partial Trigger

In the Administration Server output, you might see a "Could Not Find Partial Trigger" error (multiple times for each clicked policy configuration node or host identifier node) and also when you click any of other nodes in the navigation tree. This does not block functionality.

C.16 Denial of Service Attacks

A denial-of-service attack (DoS attack) or distributed denial-of-service attack (DDoS attack) is an attempt to make a computer resource unavailable to its intended users. One common method of attack involves saturating the target (victim) machine with external communication requests, such that it cannot respond to legitimate traffic, or responds so slowly as to be rendered effectively unavailable.

Denial of service attacks are classified into Authenticated and Unauthenticated Requests, and further classified as:

• NAP Requests

• HTTP Requests

Authenticated NAP Requests

For Authenticated NAP Requests, the OAM Server maintains a counter in the session and limits the number of retries. Despite this, after redirecting the user to an error page the user can repeat the cycle. This needlessly consumes server resources and can lead to OAM Server overloading.

Note:

To avoid OAM Server overloading with Authenticated NAP Requests, use relevant WebLogic overload configuration settings. These ensure that the server does not crash under load. However, this does not differentiate legitimate users from malicious users.

Authenticated HTTP Requests

You can handle a flood of HTTP Authenticated requests with a combination of WebLogic overload configuration and mod_security module settings.

Unauthenticated NAP Requests

Unauthenticated NAP Requests are handled by the WebLogic MDB pool throttling. This limits the number of NAP Requests that are forwarded to the OAM Server.

Again, this does not differentiate legitimate users from malicious ones.

Unauthenticated HTTP Requests

Configuring the mod_security module for the OHS server that front-ends the OAM Server enables rejection of malicious requests (unauthenticated HTTP Requests).

For more information, see:

• Protecting the OAM Server from Crashing Under Load

• Compensating for Network Latency

• Protecting OAM Servers from a Flood of HTTP Requests

C.16.1 Protecting the OAM Server from Crashing Under Load

If the number of requests to the OAM Server unexpectedly increases beyond what the server can handle, it could crash.

To limit the number of requests to the OAM Server:

1. In the WebLogic Console, use the Message Driven Bean pool to restrict the number of NAP requests to the OAM Server.

   MDBeans pull NAP requests from the Server queue and deliver NAP requests to the Server for processing. Limiting the number of MDBean instances helps control the number of requests that are processed at a given time.

2. In the WebLogic Console, configure the number of WebLogic worker threads that can be used (to restrict the number of requests to the OAM Server).

   MDBeans pull NAP requests from the server queue and deliver NAP requests to the Server for processing. Limiting the number of MDBean instances helps control the number of requests that are processed at a given time.

3. In the WebLogic Console, configure the number of WebLogic worker threads that can be used (to restrict the number of requests to the OAM Server).

   See the topic on Thread Management in the guide to Oracle Fusion Middleware Performance and Tuning for Oracle WebLogic Server.

4. In the WebLogic Console, specify a maximum incoming request size, complete message timeout, and set the number of file descriptors, to optimize performance as described in following topics in the Oracle Fusion Middleware Performance and Tuning for Oracle WebLogic Server:

   • Tuning Message Size

   • Tuning Complete Message Timeout

   • Tuning Number of File Descriptors

C.16.2 Compensating for Network Latency

Consider the scenario where Webgate sends an authentication request to the OAM Server. After successful credential collection and validation, the OAM Server creates the session and the relevant cookies (OAM_ID, ObSSOCookie). However, due to network latency, the response times out by the time the OAM Server sends it to the Webgate which triggers Webgate to re-send the authentication request to the Server. The OAM Server recognizes the session, then recreates the ObSSOCookie, and sends the response to the agent. If the network latency persists, the cycle continues in an infinite loop between the Server and the Webgate. The user is neither asked to login again nor presented with an error message.

See Also:

• Controlling Network Latency in Performance and Tuning Guide.

• Validating Connectivity and Policies Using the Access Tester

C.16.3 Protecting OAM Servers from a Flood of HTTP Requests

ModSecurity is a Web application firewall (WAF) that can be deployed as part of the existing Apache-based Web server infrastructure. This module can be plugged into the OHS Server that front-ends the OAM Server. In this way, Mod_security module protects the OAM Server from denial of service attacks.

A flexible rule engine is at the heart of ModSecurity. It implements the ModSecurity Rule Language, a specialized programming language designed to work with HTTP transaction data. A new configuration directive uses the httpd-guardian script to monitor for Denial of Service (DoS) attacks. By default httpd-guardian defends against clients that send more than 120 requests in a minute, or more than 360 requests in five minutes.

See Also:

http://www.modsecurity.org/documentation/modsecurity-apache/2.5.12/html-multipage/configuration-directives.html#N10689

To protect from a flood of HTTP Requests

1. Add the mod_security module to the OHS Server that front-ends the OAM Server.

2. In the OHS Server configuration, set the configuration directive to use the httpd-guardian script to monitor for Denial of Service (DoS) attacks.

   Syntax:

   SecGuardianLog |/path/to/httpd-guardian
   

   Example:

   SecGuardianLog |/usr/local/apache/bin/httpd-guardian
   

C.17 Diagnosing Initialization and Performance Issues

This section includes the following topics:

• Diagnosing an Initialization Issue

• Diagnosing a Performance Issue

• Diagnosing Out-of-Memory Issues With a Heap Dump

C.17.1 Diagnosing an Initialization Issue

Problem

OAM Server does not start up.

Solution

1. Locate and review the OAM Server log file on the computer hosting the OAM Server.

    DOMAIN_HOME/servers/SERVER-NAME/logs/SERVER-NAME-diagnostics.log
   

2. Enable logging for this computer, as described in Logging Component Event Messages:

    DOMAIN_HOME/config/fmwconfig/servers/SERVER-NAME/logging.xml
   

3. Restart the OAM Server, observe the behavior, check the log file again if needed.

C.17.2 Diagnosing a Performance Issue

Problem

Monitoring the OAM Server reveals a significant spike in latency during authentication.

Solution

1. Locate and review the OAM Server log file on the computer hosting the OAM Server.

    DOMAIN_HOME/servers/SERVER-NAME/logs/SERVER-NAME-diagnostics.log
   

2. Enable logging for this computer, as described in Logging Component Event Messages:

    DOMAIN_HOME/config/fmwconfig/servers/SERVER-NAME/logging.xml
   

3. Restart the OAM Server, observe the behavior, check the log file again if needed.

C.17.3 Diagnosing Out-of-Memory Issues With a Heap Dump

Problem

Debugging for all expression parsing and evaluation produced a significant performance drag within ~20 hours due to memory growth; running out of memory in ~50 hours.

Configuration: 2GB heap; 3 minute session timeout; jdbc connections tuned min=32 max=200; jdbc connection idle timeout disabled; jbo pool size min = 10 & max=150

Solution

To generate heap-dumps for comparison, you use the following command-line tools jmap for Sun jvm or jrcmd for jrockit jvm located under JAVA_HOME/bin.

For jrockit jvm

jrcmd pid <command> 
/jrockit_160_14_R27.6.5-32/bin/jrcmd 16775 heap_diagnostics
/jrockit_160_14_R27.6.5-32/bin/jrcmd 16775 print_threads
/jrockit_160_14_R27.6.5-32/bin/jrcmd 16775 jrarecording ....

For Sun jvm

jmap -histo <pid> 
jmap -dump:live,format=b,file=heap.bin <pid>

C.18 Disabling Windows Challenge/Response Authentication on IIS Web Servers

The IIS Web server on Windows supports Challenge/Response Authentication, which defaults to On when IIS is installed. This enables users to use their domain log-ins when requesting resources from IIS and can conflict with Access Manager's authentication.

For example, on the first request from an Internet Explorer (IE) browser to a resource on IIS protected by Access Manager with a basic authentication scheme, IE displays a login dialog box requesting a domain along with the user name and password login provided by Access Manager.

To disable Windows challenge/response authentication

1. Launch the Microsoft Management Console for IIS.

2. Select the Web Server Host under Internet Information Server in the left hand panel.

3. Right click and select Properties.

4. Scroll down and select Edit the Master Properties for WWW Service.

5. Select the Directory Security tab.

6. Select Edit Anonymous Access and Authentication Control.

7. Complete the appropriate step for your platform:

   Windows 2000: Clear the Integrate Windows Authentication box.

8. Click OK.

9. In the Windows IIS properties screen, click OK.

10. Close the Microsoft Management Console.

C.19 Changing UserIdentityStore1 Type Can Lock Out Administrators

An Identity Store that is designated as the System Store should not be edited to change the store type (from Embedded LDAP to OID, for instance) nor the connection URLs.

If you do need to change the Identity Store that is designated as the System Store should not be edited to change the store type, Oracle recommends that you create a new Identity Store and then edit that registration to mark it as your System Store.

C.20 IIS Web Server Issues

The following topics are provided to assist you:

• Form Authentication or Pass-Through Not Working

• IIS and General Web Component Guidelines

• Issues with IIS v6 Web Servers

• Page Cannot Be Displayed Error

• Removing and Reinstalling IIS DLLs

C.20.1 Form Authentication or Pass-Through Not Working

If form authentication or pass-through functionality is not working, the problem might be that either "UseWebGateExtForPassthrough" parameter is not set to true in the Webgate profile or that webgate.dll is not configured as Wild Card Application Mapping in IIS. In such cases, Webgate does not perform authentication or authorization for HTTP "POST" requests for the resources protected by form-based authentication.

Solution

Confirm that the UseWebGateExtForPassthrough parameter is configured in the Webgate profile with a value of true and that webgate.dll is configured as Wild Card Application Mapping.

C.20.2 IIS and General Web Component Guidelines

Following are some general guidelines to follow when installing Access Manager Webgates with IIS Web servers.

Account Privileges: The account that performs Access Manager installation must have administration privileges. The user account that is used to run OAM services must have the "Log on as a service" right, which can be set by selecting Administrative Tools, Local Policy, Local Policies, User Rights Assignments, Log on as a service.

IIS 6 Web Servers: You must run the WWW service in IIS 5.0 isolation mode. This is required by the ISAPI postgate filter. During Access Manager installation, this is usually set automatically. If it is not, you must set it manually for the Default Web site.

Webgate for IIS 7 Web Server: To use Form-based authentication without enabling pass through functionality (for example, "access/oblix/apps/webgate/bin/webgate.dll" is an action in the Form-based authentication scheme), ensure that the entry "<add segment="bin"/>" is not present in the applicationHost.config file. If the entry is present, you must remove it. Use the following steps to check this entry:

• Go to Windows\System32\inetsrv\config and open the file applicationHost.config.

• Search for the <hiddenSegments> module and remove the entry <add segment="bin"/> if it is present.

Webgate: When installing IIS Webgates, setting various permissions for the /access directory is required for IIS Webgates only when you are installing on a file system that supports NTFS. For example, suppose you install the ISAPI Webgate in Simple or Cert mode on a Windows 2000 computer running the FAT32 file system. The last installation panel provides instructions for manually setting various permissions that cannot be set on the FAT32 file system. In this case, these instructions may be ignored.

C.20.3 Issues with IIS v6 Web Servers

On IIS 6 Web servers only, you must run the WWW service in IIS 5.0 isolation mode, which is a requirement of the ISAPI postgate filter. This scenario will work if you have 32-bit Access Manager binaries running on a 32-bit Windows operating system. However, there is an issue if you attempt to run a 32-bit postgate.dll on a 64-bit Windows machine with IIS running in 32-bit mode.

Problem

When running IIS in IIS5.0 isolation mode, you see the following message:

"ISAPI Filter 'C:\webgate\access\oblix\apps\webgate\bin\webgate.dll' could not be loaded due to a configuration problem.

Cause

The current configuration only supports loading images built for an AMD 64-bit processor architecture. The data field contains the error number.

Solution

To learn more about this issue, including how to troubleshoot this kind of processor architecture mismatch error, see the following Web site:

http://go.microsoft.com/fwlink/?LinkId=29349

For more information, see Help and Support Center at:

http://go.microsoft.com/fwlink/events.asp

Problem

IIS5 never existed as 64-bit. However, IIS v6's IIS5 Compatibility Mode on 64-bit Windows computers only runs as 64-bit.

Cause

It is architecturally impossible run IIS5 Isolation Mode 32- bit on 64-bit Windows, as described in documentation available through the following URLs:

http://www.microsoft.com/communities/newsgroups/en-us/default.aspx?dg=microsoft.pu
blic.inetserver.iis&tid=5dd07102-8896-40cc-86cb-809060fa9426&cat=en_US_
02ceb021-bb43-476d-8f8f-6c00a363ccf5&lang=en&cr=US&p=1

http://blogs.msdn.com/david.wang/archive/2005/12/14/HOWTO-Diagnose-one-cause-of-W3
SVC-failing-to-start-with-Win32-Error-193-on-64bit-Windows.aspx

C.20.4 Page Cannot Be Displayed Error

A "The page cannot be displayed" error that appears after configuring Webgate for pass-through functionality, indicates a configuration issue.

Solution: Confirm that the UseWebGateExtForPassthrough parameter is configured in the Webgate profile with a value of true and that webgate.dll is configured as Wild Card Application Mapping.

C.20.5 Removing and Reinstalling IIS DLLs

When Access Manager is running with Microsoft's IIS Web server, you must manually uninstall and reinstall the following ISAPI filters when reinstalling Access Manager.

• tranfilter.dll

• oblixlock.dll (if you installed Webgate)

• webgate.dll (if you installed Webgate)

To remove and reinstall IIS DLLs

1. Uninstall Access Manager.

2. Manually uninstall the preceding DLLs.

3. Reinstall Access Manager.Active Directory.

4. Manually reinstall the DLLs.

Note:

These filters can change depending on the version of IIS you are using. If these filters do not exist or there are others present, contact Oracle to determine if the filters that are present need to be removed.

C.21 Import and File Upload Limits

The UPLOAD_MAX_MEMORY and UPLOAD_MAX_DISK_SPACE is set to "50mb". To upload more than 50mb, manually change these settings in web.xml.

To reset the memory and disk space parameters

1. Locate web.xml in WEB-INF/lib/ngam-ui.war.

2. Edit the file to change UPLOAD_MAX_MEMORY. For example:

   <context-param>
      <param-name>org.apache.myfaces.trinidad.UPLOAD_MAX_MEMORY</param-name> 
      <param-value>104857600</param-value>
   <context-param> 
   

3. Edit the file to change UPLOAD_MAX_DISK_SPACE. For example:

   <context-param>
      <param-name>org.apache.myfaces.trinidad.UPLOAD_MAX_DISK_SPACE</param-name> 
      <param-value>104857600</param-value>
   <context-param> 
   

4. Save the file.

5. Restart the OAM Server.

See Also:

"Providing File Upload Capability" in the Oracle Application Development Framework Developer's Guide.

C.22 jps Logger Class Instantiation Warning is Logged on Authentication

A jps logger class instantiation warning is might appear on the back end upon authentication. However, this is a harmless warning and no action is required.

C.23 Internationalization, Languages, and Translation

This section provides the following topics:

• Automatically Generated Descriptions Are Not Translated

• Console Looks Messy

• Authentication Fails: Users with Non-ASCII Characters

• Access Tester Does Not Work with Non-ASCII Agent Names

• Locales, Languages, and Oracle Access Management Console Login Page

C.23.1 Automatically Generated Descriptions Are Not Translated

The automatically generated Description for some components are not translated. This is expected and enables Administrators to change the Description to whatever they require. Following such a change, translation by Oracle is not possible.

C.23.2 Console Looks Messy

The Oracle Access Management Console displays policies and resources oddly when the input configuration file for remote registration is not in UTF-8 format or when the OAM Server is not started in UTF-8 locale (en_US.utf8, for instance).

Be sure to use UTF-8 encoding if creating a configuration file for the remote registration tool, oamreg, to generate authentication policies and protected resources. Also, be sure to start OAM Server in UTF-8 locale machines. Otherwise, the Oracle Access Management Console might display policies and resources oddly following successful inband registration.

C.23.3 Authentication Fails: Users with Non-ASCII Characters

Configure Access Manager to use Kerberos Authentication Scheme with WNA challenge method, and create a non-ASCII user in Microsoft Active Directory.

Problem

An exception occurs when trying to get user details to populate the subject with the user DN and GUID attributes. Authentication fails and an error is recorded in the OAM Server log when a non-ASCII user in Active Directory attempts to access an Access Manager-protected resource:

... Failure getting users by attribute : cn, value ....

Cause

The username in the attribute is passed without modification as a java string.

Solution

Non-ASCII users can access the resource protected by Kerberos WNA scheme now by applying this JVM system property (for the built-in WebLogic SPNEGO support):

-Dsun.security.krb5.msinterop.kstring=true

C.23.4 Access Tester Does Not Work with Non-ASCII Agent Names

Register a Webgate with Access Manager using a non-ASCII name. In the Access Tester, enter the valid IP Address, Port, and Agent ID (non-ASCII name), then click Connect.

Connection testing fails.

C.23.5 Locales, Languages, and Oracle Access Management Console Login Page

When the browser locale is not supported, the Oracle Access Management Console Login page shows as server locale. It should fall back to English. This is the expected behavior:

• If the client Locale is not supported, Oracle Access Management falls back to the server locale.

• If the server locale is not supported, Oracle Access Management falls back to English.

When users select an unsupported language and come to the Access Manager SSO page, it shows as server locale (German, for example). However, after logging in, all the pages are displayed as English.

To fall back to English

Disable the Access Manager SSO page and the original Access Manager login page also falls back to English.

C.24 Login Failure for a Protected Page

Problem

After installing OAM and protecting a page using a physical host and port, register the application using the OHS physical host and port. Login fails to prompt the user for credentials when accessing the protected page. The log file shows that the URL is re-directed to a Virtual Host despite the fact that all configuration and registration is setup correctly.

Solution

Remove any Virtual Host Directives from httpd.conf when protecting a page using the Oracle HTTP Server (OHS) physical host and port.

C.25 OAM Metric Persistence Timer IllegalStateException: SafeCluster

Problem

After using the WebLogic Configuration Wizard to create an OAM Server cluster on two computers, and starting AdminServer, all servers start up properly. After shut down, a third server is added using the WebLogic Server Administration Console to create a new managed server and add it to the cluster. The third server goes into Running mode when started, with some exceptions in the start up log.

... Exception in thread "OAM Metric Persistence Timer"

Solution

in addition to the actions in the WebLogic Administration Console, you must register the server using the Oracle Access Management Console to ensure that the server can identify itself.

Note:

When adding and registering a second server instance for the same computer, all port numbers must differ: OAM Proxy port; the "port" that must match the one in the WebLogic Server Console; and the Coherence port.

For server registration details, see "Managing Individual OAM Server Registrations".

C.26 Partial Cluster Failure and Intermittent Login and Logout Failures

Problem

In the event of a partial outage of Access Manager (on some, but not all instances of the cluster), end users might see intermittent login and logout failures.

Workarounds

1. Remove OHS from the deployment

2. Configure the OHS cluster such that each OHS instance is pinned to a WebLogic Server instance.

3. The WebLogic Server container with the malfunctioning Access Manager application must be removed from service (shutdown) and brought back up upon recovery.

C.27 RSA SecurID Issues and Logs

Each OAM SecurID Server must be registered as a separate agent with the RSA Authentication Manager. This provisions the OAM SecurID Server with its own node secret file. Every OAM SecurID Server must have its configuration file stored under $DOMAIN_HOME/config/fmwconfig/servers/$SERVER_NAME/oam.

If the RSA SecurID authentication plug-in returns an error, it is logged in the OAM Server log. Web Server logs can also provide clues as to what might be going wrong. Be sure the enable logging on your Web server.

If communication has been established between the Access Server and Authentication Manager, the sdadmin tool provides access to logs under the Report menu. Both Activity and Exception reports may give you helpful information.

Verify Authentication Manager Logging Configuration and Reports

1. Confirm that you have added the user and assigned a token using the Authentication Manager Administrator tool, sdadmin.

2. Verify that you have copied the sdconf.rec file to the OAM Server.

3. In the Authentication Manager console, Report menu, open Activity and Exception reports for helpful information.

Check SecurID Plug-In Parameters with Modified HTML Fields

If you have modified the HTML field names in the HTML forms, ensure that the RSA SecurID plug-in parameters are configured to match.

Remove the @ character From any Login Attribute Value

User login can fail if there is an at-sign (@) in the login attribute value. This is a known issue with SecurID.

C.28 Registration Issues

This section provides the following information.

• Problem: Remote Registration Tool Failure

• Problem: No ObAccessClient.xml File Generated

• Problem: Partner Registration Failure

• Problem: Remote Registration Failure in upgraded OAM12c environment

C.28.1 Problem: Remote Registration Tool Failure

Solution

Ensure that the agent name is unique (does not already exist) and that the AdminServer is running.

C.28.2 Problem: No ObAccessClient.xml File Generated

Solution

Protected and public resources must be described as relative URLs of the format '/index.html'. If the resource does not begin with a '/', no ObAccessClient.xml file will be generated.Verify the protected and public resource URLs and ensure all begin with a "/". For more information, see "Resource URL, Prefixes, and Patterns".

C.28.3 Problem: Partner Registration Failure

Partner registration can fail if you do not supply a unique agent name, which is also used to create an Application Domain. The agent name and Application Domain name must be the same and must be unique. Using the oamreg validate command can fail when the agent name does not match the Application Domain name.

Solution

Ensure that the agent name and Application Domain name are the same.

C.28.4 Problem: Remote Registration Failure in upgraded OAM12c environment

Remote registration fails if the user is not using Oracle Home of OAM12c and might encounter the following error:

SEVERE: Exception encountered: RemoteAgentRegistrationException. Specific exception:Cannot reach admin server at : http://slc12ors.us.oracle.com:7001. Please make sure the server url provided is valid and that the server is up before trying again.

Solution

Ensure that Oracle Home used in the remote registration is of OAM12c and not R2PS3.

C.29 Rowkey does not have any primary key attributes Error

While browsing across the Resources table in the Resource Type tab the following error message is logged:

@ <Error> 
<oracle.adfinternal.view.faces.model.binding.CurrencyRowKeySet> 
@ <BEA-000000> <ADFv: Rowkey does not have any primary key attributes. Rowkey:
oracle.jbo.Key[], table: model.ResTypeVOImpl@620289.> 

This is harmless and does not hinder any functionality.

C.30 SELinux Issues

Delivered with Oracle Enterprise Linux, SELinux modifications provide a variety of policies through the use of Linux Security Modules (LSM) within the Linux kernel.

SELinux requires performing additional steps after installing Access Manager Webgates and before starting the associated Web server.

Problem

The following errors could be reported in logs/console when starting a Web server on Linux distributions that have more strict SELinux policies in place (after installing an Webgate):

OAM Webgate

$Webgate_OH/webgate/ohs/lib/webgate.so: cannot restore segment prot after reloc: 
Permission denied. 

Cause

These errors are reported due to Secure Linux security context policies on files.

Solution

To avoid these errors and start the Web server, run following chcon commands to change the security context on files after installing each Access Manager Web component and before restarting the associated Web server. For more information on the chcon command, see your Linux documentation.

1. Run chcon -t texrel_shlib_t PATH_TO_LIBWEBPLUGINS.SO. For example:

   chcon -t texrel_shlib_t  /Webgate_install_dir/access/oblix/lib/webgate.so 
   ... and libxmlengine.so
   

2. Run chcon -t texrel_shlib_t PATH_TO_LIBWEBGATE.SO. For example:

   chcon -t texrel_shlib_t  /Webgate_install_dir/access/oblix/apps/webgate/
   bin/webgate.so 

C.31 Session Issues

This section provides the following details:

• Session Impersonation Not Enabled by Default

• Sessions with Oracle Access Manager 11.1.1 Integrated with Oracle Identity Federation 11.1.1

C.31.1 Session Impersonation Not Enabled by Default

Session impersonation is not enabled by default. You can update the value in oam-config.xml, then update the version of oam-config.xml to automatically propagate the ImpersonationConfig status to all managed servers without a restart.

To enable Session Impersonation

1. Back up DOMAIN_HOME/config/fmwconfig/oam-config.xml.

2. Set ImpersonationConfig to true:

   <Setting Name="ImpersonationConfig" Type="htf:map">
        <Setting Name="EnableImpersonation" Type="xsd:boolean">false</Setting> 
   </Setting>      
   

3. Configuration Version: Increment the Version xsd:integer as shown in the next to last line of this example (existing value (25, here) + 1):

   Example:

   <Setting Name="Version" Type="xsd:integer">
     <Setting xmlns="http://www.w3.org/2001/XMLSchema"
       Name="NGAMConfiguration" Type="htf:map:> 
     <Setting Name="ProductRelease" Type="xsd:string">11.1.1.3</Setting>
       <Setting Name="Version" Type="xsd:integer">25</Setting>
   </Setting>      
   

4. Save oam-config.xml.

C.31.2 Sessions with Oracle Access Manager 11.1.1 Integrated with Oracle Identity Federation 11.1.1

Expected Behavior: Oracle Identity Federation 11.1.1 session is not cleared

When Oracle Access Manager 11.1.1 is integrated with Oracle Identity Federation 11.1.1, and you clear the session using the console, only the Oracle Access Manager session is cleared. The Oracle Identity Federation session is not cleared.

C.32 SSL versus Open Communication

If both the SSL and Open ports of the Managed Server are enabled, then the Managed Server is set to the SSL port by default.

If you must use the non-ssl port, the credential collector URL the authentication scheme must be set to the absolute URL which points to 'http' as the protocol and non-ssl port.

C.33 Start Up Issues

This section lists the start up issues and its solution.

• AdminServer Startup (or Remote Registration Tool Failure) on AIX Platforms

C.33.1 AdminServer Startup (or Remote Registration Tool Failure) on AIX Platforms

Problem

AdminServer start up fails with following message:

"java.net.SocketException: 
No buffer space available".

Configuration for the number of AIX file descriptors set for the operating system is substantially high (ulimit file descriptor) resulting in a buffer overflow that causes remote registration failure with the following message:

The ulimit value is application dependent and applies exclusively to application program data and the application stack. The default number of open files setting (2000) is typically sufficient for most applications. If the value is too low, errors might occur when opening files or establishing connections. Because this value limits the number of file descriptors that a server process might open, a value that is too low prevents optimum performance. For the AIX operating system, the default setting is 2000.

Solution

Increasing the ulimit file descriptor limits might improve performance. Increasing some of the other limits might be needed depending on your application.

1. Log in as root.

2. Perform the following steps to change the open file limit to 10,000 files:

   1. Open the command window.

   2. Locate and edit /etc/security/limits file to add the following lines to the user account on which the AdminServer process runs:

      nofiles =  10000   
      nofiles_hard = 10000
      

   3. Save the file and restart AIX.

3. In a command window, decrease the TCP_TIMEWAIT interval with the following command to set the state to 15 seconds (which allows TCP to release closed connections faster and increases the number of available resources for open connections).

   /usr/sbin/no –o tcp_timewait =1
   

4. Tune the following parameters to 256k, as shown:

   no -a |grep space   
         tcp_recvspace = 262144   
         tcp_sendspace = 262144   
         udp_recvspace = 262144  
         udp_sendspace = 262144
   

5. Tune the following parameters as indicated here:

   no -o rfc1323=1  
   no -o sb_max=4194304

C.34 Synchronizing OAM Server Clocks

The state of a session is the source of truth for relying parties. Synchronization of system clocks of the various Servers is required.

The system clock of the relying party might be out of synchronization with the SME clock. If the relying party's clock is:

• Ahead of the session clock A relying party's request for authentication is made and the active sessionID is returned.

• Behind the session clock: Event notifications to the relying party help invalidate the session.

For example, if a Web server clock is ahead of the server clock, a request sent from the Webgate to the OAM Server will contain a time that, to the OAM Server, has not yet occurred. This can cause login events to fail. When running in Simple or Cert mode, time stamps might become out of sync, or the client certificate might appear to be invalid.

Note:

To avoid event notification issues, ensure that all OAM Server clocks are synchronized to Time Services such as NIST internet time service.

For successful operation:

• Ensure all computer clocks are synchronized. There is no tolerance level. If, for example, the Webgate clock is even slightly ahead of the OAM Server clock, a cookie generated by the Webgate will appear to be in the future and can cause problems in the OAM Server.

• Confirm that the clock on each computer running a Webgate is not running ahead of the OAM Servers with which it is associated. The OAM Server must be ahead of the Webgate clock by a maximum of 60 seconds.

C.35 Time delay in configuration change

If any configuration change is made, there may be a time delay for that change to be refreshed in the runtime. This time delay is because the servers will need few seconds to refresh the configuration after it was changed.

For Example, Suppose you create a new password after setting the “Disallowed Previous Passwords” option in the password policy, the previous password should not be allowed to access the protected resource. In this case, the previous password is valid if the user tries to use the password without waiting for 60 seconds.

Solution

When a configuration change is made, you should wait for a minimum of 60 seconds for the changes to reflect.

C.36 Validation Errors

The following sections provide information on the Validation Errors and solution to the problems.

• Resource not added to Authentication or Authorization Policy

• Validation Failure - "description" attribute is not valid

C.36.1 Resource not added to Authentication or Authorization Policy

Problem

While creating an Authentication or Authorization Policy, if you add a resource that is already used in another Authentication or Authorization Policy, a validation error appears when you click Apply. This is expected.

If you click OK in the error window and then attempt to add a valid resource that is not used within another Authentication or Authorization Policy, the resource is not added and the Authentication or Authorization Policy is not created.

Solution

1. Click Apply and close the Authentication or Authorization Policy page.

2. From the navigation tree, click the named policy again, click the Edit to open the page, and add the new resource.

C.36.2 Validation Failure - "description" attribute is not valid

Problem

A validation error appears if you enter an optional description longer than 200 characters.

Solution

Keep optional descriptions to 200 characters in length and less than 10 lines.

C.37 Web Server Issues

The following issues with Web servers may arise:

• Server Fails on an Apache Web Server

• Apache v2 on HP-UX

• Apache v2 Bundled with Red Hat Enterprise Linux 4

• Apache v2 Bundled with Security-Enhanced Linux

• Apache v2 on UNIX with the mpm_worker_module for Webgate

• Domino Web Server Issues

• Errors, Loss of Access, and Unpredictable Behavior

• Known Issues for ISA Web Server

• Oracle HTTP Server Fails to Start with LinuxThreads

• Oracle HTTP Server Webgate Fails to Initialize On Linux Red Hat 4

• Oracle HTTP Server Web Server Configuration File Issue

• Issues with IIS v6 Web Servers

• PCLOSE Error When Starting Sun Web Server

• Removing and Reinstalling IIS DLLs

C.37.1 Server Fails on an Apache Web Server

Symptom

You are running an Apache Web server, and an OAM Server fails, displaying the following message:

libthread panic: cannot create new lwp
(PID: 9035 LWP 2). stackrace:
ff3424cc
0

This symptom may be caused by the Apache Web server launching more instances of itself. This can happen when the server determines that more instances are needed to service the number of connections between one or more Webgates and the OAM Server.

The additional instances create even more connections, which exceed the number of connections by the OAM Server.

Solution

Reduce the number of MinSpareServers, MaxSpareServers, StartServers, and MaxClients parameters.

Go to the OAM Server's configuration directory and open the http.d configuration file.

Recommended parameter settings:

• MinSpareServers 1

• MaxSpareServers 5

• StartServers 3

• MaxClients 5

C.37.2 Apache v2 on HP-UX

When running Apache v2 on HP-UX, do not use nobody for User or Group, because shared memory may not work. Instead, use your login name as User Name with a your group as Group Name On HP-UX (on Solaris, "www" is equivalent to "nobody").

When running Apache v2 on HPUX 11.11, ensure that the AcceptMutex directive in the Apache httpd.conf file is set to "fcntl". If the directive is not present, add it to the httpd.conf file (AcceptMutex fcntl). For more information, see:

http://issues.apache.org/bugzilla/show_bug.cgi?id=22484

C.37.3 Apache v2 Bundled with Red Hat Enterprise Linux 4

Problem

After installing a Webgate on vendor-bundled Apache, the Web server may give the following error upon startup:

Error: Cannot load libgcc_s.so.1 library - Permission denied. 

Solution

Change the Security-Enhanced Linux (SELinux) policy rules for Access Manager Webgates.

C.37.4 Apache v2 Bundled with Security-Enhanced Linux

Errors might be reported in WebServer logs/console when starting a Web server on Linux distributions, which have stricter SELinux policies in place, after installing an Access Manager Web component. You can avoid these errors by running appropriate chcon commands for the installed Web component before restarting the Web server.

See Also:

"SELinux Issues"

C.37.5 Apache v2 on UNIX with the mpm_worker_module for Webgate

The following item is required only if you compile Apache v2 for Webgate on UNIX with the mpm_worker_module. In this case, you need to modify the thread.c file from the Apache source for the UNIX environment. Making this change ensures that the default pthread stacksize for Webgate produces optimal performance during multi-threaded server implementation. If this change is not made, the default pthread stack size would not be sufficient for Webgate and could result in a crash.

Apache 2.0 does not support the ThreadStackSize option. Therefore:

• With UNIX-based Apache v2.1 and later you must use the ThreadStackSize directive to set the size of the stack (for autodata) of threads that handle client connections and call modules to help process those connections.

• With UNIX-based Apache 2, it is best to use the compilable source while adding the mpm_worker_module and changing the thread.c file to avoid a stack overflow.

The following procedure shows how to modify the Apache v2.0 thread.c file to provide the default pthread stacksize needed by Webgate for optimal performance during multi-threaded server implementation. For details about the Apache v2.1+ ThreadStackSize directive, see http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadstacksize.

Note:

The following procedure should be performed only for the Apache 2.0 Webgate. Otherwise, the default pthread stack size is not sufficient for the Webgate and could result in a crash.

To modify the Apache v2.0 thread.c file for Webgate in a UNIX environment

1. Locate the thread.c file. For example:

   APACHE 2.0.52 source/srclib/apr/threadproc/unix/thread.c
   

2. Locate the function named apr_threadattr_create(apr_threadattr_t **new,apr_pool_t *pool) in the following code segment:

   **new,apr_pool_t *pool) in the following code segment:
   1-----> apr_status_t stat; 
   2 
   3-----> (*new) = (apr_threadattr_t *)apr_pcalloc(pool, sizeof(apr_threadattr_t)); 
   4-----> (*new)->attr = (pthread_attr_t *)apr_pcalloc(pool, sizeof(pthread_attr_t)); 
   5 
   6-----> if ((*new) == NULL || (*new)->attr == NULL) { 
   7----->              return APR_ENOMEM; 
   8-----> } 
   9 
   10----->(*new)->pool = pool; 
   11----->stat = pthread_attr_init((*new)->attr); 
   12 
   13-----> if (stat == 0) { 
   14----->            return APR_SUCCESS; 
   15-----> } 
   16----->#ifdef PTHREAD_SETS_ERRNO 
   17----->stat = errno; 
   18----->#endif 
   19 
   20----->return stat; 
   21 
   

3. Add the following code before line 13 shown earlier.

   int stacksize = 1 << 20; 
   pthread_attr_setstacksize(&(*new)->attr, stacksize);
   

4. Run configure, make, and make install to set up the Apache Web server with the mpm_worker_module.

C.37.6 Domino Web Server Issues

Failure Authentication Event: For Domino Web servers, the redirection of a URL through Access Manager may not work if the authentication type is set as Basic Over LDAP and the URL to be redirected is mentioned as one of the following:

• Either a relative path present on the same Web server
• Or the Full path URL on the same Web server containing a computer name defined in the host identifier string combinations.

To overcome a failure authentication event, you must set the redirected URL with a computer name that is not defined under the host identifier group. For example, the IP address of the computer.

This problem does not occur with a form-based authentication type.

Header Variables: It may not be possible to pass header variables other than REMOTE_USER to Webgates installed on Lotus Notes Domino Web servers when using Client Certificate authentication scheme.

For example, header variables cannot be set on the one request where Client Certificate authentication occurs. However, all other requests do allow header variables to be set.

C.37.7 Errors, Loss of Access, and Unpredictable Behavior

Symptom

If you installed Access Manager on UNIX under a different user ID than you used to create your Web server instance, Access Manager can become unstable. Users may experience behavior such as:

• Random bug report pages

• Failure to write to log file errors

• Loss of access to Web pages

Solution

Change file permissions using the chown command. Change the Access Manager directory to the same user ID that you used to create your Web server instance.

C.37.8 Known Issues for ISA Web Server

Webgate uses ISAPI extension for displaying user deny error message and for displaying the diagnostic page. However, ISA 2006 does not support extensions. Therefore:

• If the user is denied access by Webgate, the user gets Page Cannot be displayed error message instead of Access Manager denied access error message.

• The following diagnostic URL does not work for ISA: http(s)://hostname:port/access/oblix/apps/webgate/bin/webgate.dll?progid=1 for webgate.

C.37.9 Oracle HTTP Server Fails to Start with LinuxThreads

Problem

After installing a Webgate instance on an Oracle HTTP Server, the server does not start up.

Note:

When running Access Manager, LinuxThreads is used by default. This requires setting the environment variable LD_ASSUME_KERNEL to 2.4.19. If you are using NPTL with Access Manager, you do not set LD_ASSUME_KERNEL to 2.4.19.9

Cause

This occurs because Access Manager uses an older Linux threading model.

Solution

When using LinuxThreads mode, comment out the Perl module in the httpd.conf file, update the LD_ASSUME_KERNEL environment variable, and restart, as described in the following procedure.

To resolve the failure to start Oracle HTTP Server in LinuxThreads mode

1. Comment out the perl module in the httpd.conf file in the following location:

   Oracle HTTP Server 11g: $ORACLE_INSTANCE/config/OHS/ohs_name/httpd.conf

   Oracle HTTP Server v2: OH$/ohs/conf/httpd.conf

   Oracle HTTP Server v1.3: OH$/Apache/Apache/conf/httpd.conf

2. To update the LD_ASSUME_KERNEL value, open the following file in a text editor:

   OH$/opmn/conf/opm.xml
   

3. Find the following line:

   <process-type id="HTTP_Server" module-id="OHS">

4. Add the following information under the line you found in the previous step:

   <environment>
   <variable id="LD_ASSUME_KERNEL" value="2.4.19" />
   </environment>
   

5. Save this file.

6. Run the following commands to implement your changes:

   opmnctl stopall
   opmnctl startall

C.37.10 Oracle HTTP Server Webgate Fails to Initialize On Linux Red Hat 4

This situation might arise whether you are using Access Manager with LinuxThreads or NPTL.

Symptom

Webgate fails to initialize when installed on an Oracle HTTP Server running Red Hat Enterprise Server version 4.0 with a kernel version lower than 2.6.9-34.EL. Version 2.6.9-34.EL is supplied with the Red Hat version 4, update 3.

Solution

To prevent this problem, you must upgrade to Red Hat version 4, update 3 or higher.

C.37.11 Oracle HTTP Server Web Server Configuration File Issue

Problem

With Oracle Application Server 10.1.x, OC4J, when the httpd.conf file is modified automatically during Webgate installation, it can be corrupted.

Solution

Before installing Webgate, run the following command to prevent the httpd.conf file from being overwritten.

$ORACLE_HOME/dcm/bin/dcmctl updateConfig -ct ohs 

C.37.12 Issues with IIS v6 Web Servers

On IIS 6 Web servers only, you must run the WWW service in IIS 5.0 isolation mode, which is a requirement of the ISAPI postgate filter. This scenario will work if you have 32-bit Access Manager binaries running on a 32-bit Windows operating system. However, there is an issue if you attempt to run a 32-bit postgate.dll on a 64-bit Windows machine with IIS running in 32-bit mode.

Problem

When running IIS in IIS5.0 isolation mode, you see the following message:

"ISAPI Filter 'C:\webgate\access\oblix\apps\webgate\bin\webgate.dll' could not be loaded due to a configuration problem.

Cause

The current configuration only supports loading images built for an AMD 64-bit processor architecture. The data field contains the error number.

Solution

To learn more about this issue, including how to troubleshoot this kind of processor architecture mismatch error, see the following Web site:

http://go.microsoft.com/fwlink/?LinkId=29349

For more information, see Help and Support Center at:

http://go.microsoft.com/fwlink/events.asp

Problem

IIS5 never existed as 64-bit. However, IIS v6's IIS5 Compatibility Mode on 64-bit Windows computers only runs as 64-bit.

Cause

It is architecturally impossible run IIS5 Isolation Mode 32- bit on 64-bit Windows, as described in documentation available through the following URLs:

http://www.microsoft.com/communities/newsgroups/en-us/default.aspx?dg=microsoft.pu
blic.inetserver.iis&tid=5dd07102-8896-40cc-86cb-809060fa9426&cat=en_US_
02ceb021-bb43-476d-8f8f-6c00a363ccf5&lang=en&cr=US&p=1

http://blogs.msdn.com/david.wang/archive/2005/12/14/HOWTO-Diagnose-one-cause-of-W3
SVC-failing-to-start-with-Win32-Error-193-on-64bit-Windows.aspx

C.37.13 PCLOSE Error When Starting Sun Web Server

Symptom

When attempting to start the Sun Web server, you get an error like the following:

Unable to start, PCLOSE

Solution

Solution: A number of problems can cause this error:

• A syntax error in your obj.conf file

• Leading spaces in your obj.conf file

• Installing Access Manager as a different user ID than what you used to create your Web server instance

• A carriage return at the end of the obj.conf file

C.37.14 Removing and Reinstalling IIS DLLs

When Access Manager is running with Microsoft's IIS Web server, you must manually uninstall and reinstall the following ISAPI filters when reinstalling Access Manager.

• tranfilter.dll

• oblixlock.dll (if you installed Webgate)

• webgate.dll (if you installed Webgate)

To remove and reinstall IIS DLLs

1. Uninstall Access Manager.

2. Manually uninstall the preceding DLLs.

3. Reinstall Access Manager.Active Directory.

4. Manually reinstall the DLLs.

Note:

These filters can change depending on the version of IIS you are using. If these filters do not exist or there are others present, contact Oracle to determine if the filters that are present need to be removed.

C.38 Windows Native Authentication

Problem

After setting up Windows Native Authentication, and accessing the WNA-protected page, the browser might give an error indicating that the user name and/or password are incorrect.

Cause

The Identity Store used by Oracle Access Management might not point to Windows Active Directory. By default, the identity store is Embedded LDAP.

Solution

1. In the Oracle Access Management Console, review the identity store configuration: System Configuration, Data Sources, User Identity Store.

2. Confirm the LDAP store settings point to Active Directory.

C.39 WLST Commands for Multi-Data Centers

The following WebLogic Scripting Tool (WLST) commands are specific to Multi-Data Center deployment. More information is in the following sections.

• enableMultiDataCentreMode

• disableMultiDataCentreMode

• addPartnerForMultiDataCentre

• removePartnerForMultiDataCentre

• setMultiDataCenterType

• setMultiDataCenterWrite

• setMultiDataCentreClusterName

• validateMDCConfig

• exportAccessStore

• importAccessStore

See Also:

WebLogic Server WLST Online and Offline Command Reference

C.39.1 enableMultiDataCentreMode

Online command used to enable Multi-Data Center mode.

Description

This command enables Multi-Data Center mode. It takes a value equal to the full path to, and name of, the MDC.properties file.

Note:

Setting the SSO Token version to 5 is not supported from the administration console. To do this, modify the Access Manager Settings page and run the enableMultiDataCentreMode WLST command to set.

Syntax

enableMultiDataCentreMode(propfile="../MDC_properties/oamMDCProperty.properties")

Argument	Definition
propfile	Mandatory. Takes a value equal to the full path to, and name of, the oamMDCProperty.properties file. Table C-1 documents the properties that comprise the file. The example (following the table) is a sample oamMDCProperty.properties file.

Table C-1 oamMDC.properties Properties

Property	Definition
SessionMustBeAnchoredToDataCenterServicingUser	Takes a value of True (Invalidate) or False (No Invalidation).
SessionDataRetrievalOnDemand	Takes a value of True (Cross DC retrieval) or False (No). Data retrieval can be turned off without disabling MDC. If False, session data is not transferred but SSO is still performed as the user moves across DCs. NOTE: SessionDataRetrievalOnDemand must be set to False when deploying in Co-existence mode.
Reauthenticate	Takes a value of True (force reauthentication) or False (No forced reauthentication).
SessionDataRetrievalOnDemandMax_retry_attempts	Takes a value equal to a binary that represents the number of times to retry data retrieval when it fails. Default is 2.
SessionDataRetrievalOnDemandMax_conn_wait_time	Takes a value equal to a binary that represents the total amount of time in milliseconds to wait for a connection. Default is 1000.
SessionContinuationOnSyncFailure	Decides the session adoption action on fail over. When set to 'true', the session will continue on the DC servicing the current request even though the parent DC is down/not reachable. The session will be created in the DC servicing the current request from the mandatory minimal information available in the incoming token. When set to 'false', the user will be challenged on fail-over scenarios.
MDCGitoCookieDomain	Specifies the domain with which the OAM_GITO cookie should be set. In MDC deployments where a common cookie domain hierarchy cannot be derived, this setting should be commented or removed as described in Inactivity time outs scenario.

Sample oamMDCProperty.properties File

SessionMustBeAnchoredToDataCenterServicingUser=false
SessionDataRetrievalOnDemand=true
Reauthenticate=true
SessionDataRetrievalOnDemandMax_retry_attempts=3
SessionDataRetrievalOnDemandMax_conn_wait_time=80
SessionContinuationOnSyncFailure=true

#MDCGitoCookieDomain=.example.com <This setting should be provided only if there is a common cookie subdomain across the WGs and DCs>

Example

The following command enables this data center.

enableMultiDataCentreMode(propfile="../MDC_properties/oamMDCProperty.properties")

C.39.2 disableMultiDataCentreMode

Online command used to disable Multi-Data Center mode.

Description

This command disables Multi-Data Center mode.

Syntax

disableMultiDataCentreMode()

There are no arguments for this command.

Example

The following command disables Multi-Data Center mode.

disableMultiDataCentreMode()

C.39.3 addPartnerForMultiDataCentre

In an MDC deployment with n number of Data Centers, each Data Center has a registered partner to communicate with each of the other (n-1) Data Centers. This makes the total number of partner registrations (n) x (n-1). This online command is used to add a partner for inter Data Center OAP communication.

Note:

An MDC partner profile is exposed by each data center and used by other data centers to communicate with it. Registering an MDC partner is a two step process. Consider an MDC with three data centers. In DC1, expose an MDC partner profile by creating an OAM WebGate (DC1_MDC_Partner). Then, register DC1_MDC_Partner in DC2 and DC3 using addPartnerForMultiDataCentre. See addPartnerForMultiDataCentre for details.

Description

This command adds a partner to the Data Center. It takes a value equal to the full path to, and name of, the partnerInfo.properties file.

Syntax

addPartnerForMultiDataCentre(propfile="../MDC_properties/partnerInfo.properties")

Argument	Definition
propfile	Mandatory. Takes a value equal to the path to, and name of, the partnerInfo.properties file.
RESTEndpoint	Optional. Takes as a value the HTTP/HTTPS URL from which the Access Manager REST services can be accessed.

Table C-2 documents the properties that comprise partnerInfo.properties. See Multi-Data Center Security Modes for properties file samples.

Table C-2 partnerInfo.properties Properties

Property	Definition
remoteDataCentreClusterId	Cluster id of the remote Data Center with which the OAP communication needs to be established.
oamMdcAgentId	Partner ID of the registered partner profile in the remote Data Center. The "allow management operations" flag for this partner should be set in the remote Data Center.
PrimaryHostPort	Takes a fully-qualified-host-name:OAM-port for the primary Access Manager server corresponding to the remote DC identified by remoteDataCentreClusterId; for example: PrimaryHostPort=abc.example.com:5575
SecondaryHostPort	Takes a fully-qualified-host-name:OAM-port for the secondary Access Manager server corresponding to the remote DC identified by remoteDataCentreClusterId; for example: SecondaryHostPort=abc.example.com:5577 Consider an OAM MDC member Data Center with two managed servers at abc.example.com with ports as follows: oam_server1 (5575) and oam_server2 (5577). High availability/failover of the OAP SDK partner can be achieved by setting the PrimaryHostPort and SecondaryHostPort as below. PrimaryHostPort=abc.example.com:5575 SecondaryHostPort=abc.example.com:5577
AccessClientPasswd	The access client password of the MDC partner registered in the remote Data Center.
oamMdcSecurityMode	Defines the MDC security mode. Takes a value of OPEN/SIMPLE/CERT. (CERT Mode is preferred, SIMPLE is fine but OPEN is discouraged.) For SIMPLE and CERT modes, the following values should be supplied appropriately. For OPEN mode, these values are not applicable. See Multi-Data Center Security Modes.
agentVersion	Valid agent version 11g.
trustStorePath	Absolute path to the truststore file [SIMPE/CERT].
keyStorePath	Absolute path to the keyStore file [SIMPLE/CERT].
globalPassPhrase	Global passphrase set during the partner registration [SIMPLE/CERT].
keystorePassword	Key store password set during partner configuration [SIMPLE/CERT].

Example

The following command defines this data center as a Master.

addPartnerForMultiDataCentre(propfile="../MDC_properties/partnerInfo.properties")

C.39.4 removePartnerForMultiDataCentre

Online command used to remove a registered remote partner from the Data Center configuration.

Description

This command removes a registered remote partner from a configured Data Center. It takes a value equal to a valid remoteDataCentreClusterId.

Syntax

removePartnerForMultiDataCentre=("<cluster_ID>")

Argument	Definition
cluster_ID	Mandatory. Takes a string value equal to the cluster ID.

Example

The following command defines the partner to be removed.

removePartnerForMultiDataCentre("99bf9-adc2120609")

C.39.5 setMultiDataCenterType

Online command used to set the type of data center - either Master or Clone.

Description

In an MDC deployment one Data Center is designated as the Master and the others as a Clone. Essentially all MDC wide global configurations and policy updates should be applied to the Master and propagated to the Clones. This command sets the type of the data center. Values are Master or Clone.

Syntax

setMultiDataCenterType(DataCenterType="<Master|Clone>")

Argument	Definition
DataCenterType	Mandatory. Takes a string value of Master or Clone.

Example

The following command defines this data center as a Master.

setMultiDataCenterType(DataCenterType="Master")

C.39.6 setMultiDataCenterWrite

Online command used to set write protection for modifications to system and policy configurations on the Clone Data Center.

Description

A Clone Data Center can be write protected by setting WriteEnabledFlag to false. In this case, the Clone Data Center will not allow updates through the Oracle Access Management Console or WLST commands. Data synchronization will still continue to update as the command is used to write protect the Clone Data Center against accidental updates after the initial set up is complete.

Syntax

setMultiDataCenterWrite(WriteEnabledFlag="<true|false>")

Argument	Definition
WriteEnabledFlag	Mandatory. Takes a string value of true or false.

Example

The following example protects the Clone Data Center from accidental overwrites.

setMultiDataCenterWrite(WriteEnabledFlag = "false")

C.39.7 setMultiDataCentreClusterName

Online command to set the cluster name of the Data Center to the supplied string.

Description

This command sets the Multi-Data Center cluster name. Value is a string.

Syntax

setMultiDataCentreClusterName(clusterName="<string_value>")

Argument	Definition
clusterName	Mandatory. Takes a string equal to the cluster name.

Example

The following command enables this data center.

setMultiDataCentreClusterName(clusterName="MyCluster")

C.39.8 validateMDCConfig

Online command used to insure the Multi-Data Center configuration is correct.

Description

This command validates that the required entries in the Multi-Data Center configuration are present in oam-config.xml. For the MDC solution, a new Access Manager event named mdc_session_update is required to create or update MDC sessions during authorization. The Access Manager event model requires a set of configurations to be present in the oam-config.xml configuration file. The required configurations cannot be added statically so validateMDCConfig validates the required entries for mdc_session_update and seeds any configurations not already present.

Syntax

validateMDCConfig()

There are no arguments for this command.

Example

The following command validates the MDC configuration.

validateMDCConfig()

C.39.9 exportAccessStore

Online command to create a ZIP file of the Master Data Center UDM metadata.

Description

This command will create a ZIP file of the Master Data Center UDM metadata.

Syntax

exportAccessStore(toFile="<name and location of ZIP", namePath="/")

Example

exportAccessStore(toFile="/master/location/dc1metadata.zip", namePath="/")

C.39.10 importAccessStore

Online command to import a ZIP file of the Master Data Center UDM metadata to a Clone Data Center.

Description

This command will import a ZIP file of the Master Data Center UDM metadata to the Clone Data Center.

Syntax

importAccessStore(fromFile="<name and location of ZIP", namePath="/")

Example

importAccessStore(fromFile="/master/location/dc1metadata.zip", namePath="/")

C.40 Comparing Default Parameters and Values used in MDC Configuration for 11g and 12c

Configuring MDC in 12c environment is made simple with the new set of MDC Admin REST APIs. Many of the parameters and properties that were manually set in 11g are now defaulted in 12c.

The following table details the parameters in 12.2.1.3.0 which are defaulted. You can override the default setting, if required.

Parameter and Description	Mandatory / Optional	Default Value
mdcTopologyType (ACTIVE_ACTIVE or DISASTER_RECOVERY)	Mandatory	NA
masterMDCAgentID (MDC NAP Agent Name for Master)	Mandatory	NA
cloneMDCAgentID (MDC NAP Agent Name for Clone)	Mandatory	NA
accessClientPassword (password to be used for the MDC NAP agents in Master and Clone)	Mandatory	NA
cloneServerURL (URL of the clone admin server or the URL of the reverse proxy front-ending the clone admin server)	Mandatory	NA
agentKeyPassword (Agent Key Password used to register partners in CERT mode)	Mandatory	NA
certModeKeystorePassword (KeyStore Password used to protect clientTrustStore.jks and clientKeyStore.jks)	Mandatory	NA
masterServerURL (URL of the master admin server or URL of the reverse proxy front ending the master admin server)	Optional for Master configuration; Mandatory for Clone configuration	If not provided, during Master configuration it retrieves Admin server’s host and port details from MBean.
artifactPassword (password used for protecting the cloning artifacts.)	Mandatory	NA
cloneAdminUserNamePassword (UserName:Password of Clone DC Administrator; specify only when the Master and Clone Admin Users/Password are different)	Optional	UserName:Password of Master data center.
trustStorePath (path to clientTrustStore.jks file; specify only when the clientTrustStore.jks file is present in any folder other than DOMAIN_HOME/config/fmwconfig/oam-mdc-cert-artifacts)	Optional	CERT mode: %DOMAIN_HOME%/config/fmwconfig/oam-mdc-cert-artifacts SIMPLE mode: %DOMAIN_HOME%/output/webgate-ssl-SHA-256/
keyStorePath (path to clientKeyStore.jks file; specify only when the clientTrustStore.jks file is present in any folder other than DOMAIN_HOME/config/fmwconfig/oam-mdc-cert-artifacts)	Optional	CERT mode: %DOMAIN_HOME%/config/fmwconfig/oam-mdc-cert-artifacts SIMPLE mode: %DOMAIN_HOME%/output/webgate-ssl-SHA-256/
artifactsZipLocation (location where artifacts are stored; specify if artifacts are stored in a location other than /tmp)	Optional	/tmp
isMultiDataCenterEnabled (used to disable MDC)	Mandatory	NA
isBackwardCompatible (used to enable backward compatibility when master and clone data centers have different OAM versions.)	Mandatory	NA
clusterName (cluster name of a data center)	Mandatory	NA
masterArtifactsZipLocation (location where cloning artifacts are present in Master; specify only when artifactsZipLocation was used in input while configuring the Master data center)	Optional	/tmp