Skip Headers
Oracle® Fusion Middleware Administrator's Guide for Oracle Access Manager with Oracle Security Token Service
11g Release 1 (11.1.1)

Part Number E15478-05
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Index
Index
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
View PDF

I Troubleshooting

This chapter provides troubleshooting tips.

Introduction to OAM 11g Troubleshooting

OAM is a business critical system; downtime comes with a potentially high cost to your business. The goal of system analysis is to quickly isolate and correct the cause of any problem. This requires a big picture view of your system and the tools to observe the live system and correlate components to the bigger picture.

To assist administrators in performing a quick diagnosis, this section provides the following topics:

About System Analysis and Problem Scenarios

System analysis includes understanding how the product works, what can go wrong, how likely the scenarios are, and the consequences or observable issues.

System problems can be divided into two basic categories:

  • Cascading catastrophic failure

  • Gradual breakdown in performance

Cascading catastrophic failure might be caused by:

  • LDAP server is loaded and unresponsive

  • Morning peak load starts

  • Webgates send requests to the primary OAM Server

  • Webgate requests time-out and Webgates retry to secondary OAM Server

Gradual breakdown in performance might occur over time when, for example:

  • OAM is sized and rolled out for 10,000 users and 500 groups

  • Over the course of a year, the number of users and groups increases significantly (to 50,000 users and 250 groups for example)

For information on the most commonly encountered issues, see the following topics:

About LDAP Server or Identity Store Issues

This topic provides symptoms, probable cause, and steps to diagnose the following issues:

Symptoms: Operational Slowness

  • Poor user experience

  • Agent timeouts lead to retries

Cause

  • Non-OAM load might be impacting OAM operations

  • Capacity problems due to gradual increase in peak load

Symptoms: Total loss of service

Cause

  • Outage of all LDAP servers

  • The load balancer is timing out old connections

Diagnosis

  1. Shut down the LDAP server.

  2. Restart your browser.

  3. Try to access a protected site.

  4. Review errors in the OAM Server log file, as described in Chapter 22 (alternatively, in Chapter 26).

  5. Try to access Oracle Access Manager Console.

  6. Observe errors in WebLogic AdminServer log file.

  7. Bring up the LDAP server again.

  8. Retry access to a protected application.

  9. Retry access to the Oracle Access Manager Console.

  10. Correct the issue based on the requirements in your environment.

About OAM Server or Host Issues

This topic provides symptoms, probable cause, and steps to diagnose the following issues:

Symptoms: Capacity Problems

  • Poor user experience due to slow operations

  • Agent timeouts and retry can result in extra load

Cause

  • CPU cycles

  • Memory issues

Symptoms: Interference with Other Services on the Host

  • Poor user experience due to slow operations

  • Agent timeouts and retry may result in extra load

Cause

  • CPU cycle contention

  • Memory contention

  • File system full

Diagnosis: OAM Server

  1. Shut down the OAM Server

  2. Try to access a Webgate or mod_osso protected resource

  3. Bring up the OAM Server

  4. Use the Access Tester to test authentication and authorization as described in Chapter 14.

  5. Use 'top' to figure out the CPU and Memory consumption of the OAM Server as you use the access tester

  6. Get a thread dump of the OAM Server.

Diagnosis: OAM Admin Server

  1. Shut down the OAM AdminServer

  2. Restart your browser and access a protected resource, which should work.

  3. Use remote registration to register a new partner, as described in Chapter 10 (this should fail).

  4. Startup OAM AdminServer.

About Agent-Side Configuration and Load Issues

This topic provides symptoms, probable cause, and steps to diagnose time issues between agents and servers.

Symptoms: Difference in Clock time Between Agent and Server

  • High CPU usage at both agent and server

  • User experiences a system hang

Cause

  • Agent thinks the token issued by the server is invalid

  • Agent keeps going back to the server to re-issue the token

Diagnosis

  1. Access protected resource.

  2. Confirm: Client access hangs.

  3. Confirm: High CPU usage on agent and server.

About Runtime Database (Audit or Session Data) Issues

The audit and session functions are both write intensive operations. The policy database can be tuned for read intensive service.

Symptoms

  • Audit and session operations are slow

  • File system on the OAM Server is full with audit data that is not yet written to the database

  • Loss of in-memory session data when one of the servers in the cluster fails

Cause

  • Database is not tuned for write intensive operations

  • Database is unavailable due to maintenance

  • Space issues in the database

Diagnosis

  1. Shut down the database used to store Audit and Session data.

  2. Try to access a protected resource.

  3. Review error and warning messages in the OAM Server log files, as described in Chapter 22 (alternatively, in Chapter 26).

About Change Propagation or Activation Issues

This topic provides symptoms, probable cause, and steps to diagnose the following issues:

Symptoms

  • Changes to policy do not take immediate effect

  • Changes to system configuration do not take immediate effect

Cause

  • Servers being too busy handling runtime requests (CPU contention)

  • Coherence network slowness

Diagnosis: See "About Policy Store Database Issues"

About Policy Store Database Issues

This topic provides symptoms, probable cause, and steps to diagnose policy database issues.

Symptoms: No policy changes are allowed; no impact on runtime

Cause

  • Database is unavailable (down for maintenance)

  • Space issues in the database

Diagnosis

  1. Shut down the database containing OAM policies.

  2. Try to access a protected resource and observe the runtime access is not impacted.

  3. Try to access the Oracle Access Manager Console to edit policies, and then observe errors in the AdminServer log file.

Oracle Access Manager Console Inconsistent State

Problem

Administrators performing updates concurrently will result in an inconsistent state within the system configuration of the Oracle Access Manager Console.

Cause

Concurrent configuration updates are not supported.

Solution

Only one administrator should be allowed to modify the system configuration at any given time.

AdminServer Won't Start if the Wrong Java Path Given with WebLogic Server Installation

WebLogic Server (wls1035_generic) installation is successful on Windows 64-bit with 32-bit Java (jdk1.6.0_24). When setup.exe is executed you must provide the path of the 64-bit java (jdk1.6.0_23) to successfully launch the install shield.

If you provide the 32-bit Java (jdk1.6.0_24) path, the install shield is not launched. However, if you execute config.cmd from \Middleware\Oracle_IDM1\common\bin, by default 32-bit Java (jdk1.6.0_24) path is used, but after successful installation Oracle Access Manager installation, you cannot start AdminServer.

On Windows host, the path to 32-bit JAVA_HOME (c:\program files (x86)\java\jdkxxx) is not correctly handled by the startWeblogic.cmd. Replacing SUN_JAVA_HOME to use the path with the shorter name (c:\progra~2\java\jdkxxxx) works fine.

On Windows, the shorter names can be seen by executing "dir /X".

Alternatively, you can set windows cmd shell variable JAVA_HOME to path with shorter name and execute startWeblogic.cmd within that. For example:

>set JAVA_HOME=c:\progra~2\java\jdkXXX

>startweblogic.cmd

Agent Naming Not Unique

A unique identifying name for each Agent registration is preferred. However:

Application URL Requirements

The number of characters allowed in a URL are based on browser version.

Ensure that your applications do not use URLs that exceed the length that Oracle Access Manager and the browser can handle.

Authentication Issues

This section provides the following information:

Anonymous Authentication Issues

Problem

Challenge Redirect URL can be NULL; however, Challenge Method cannot be NULL.

If you open the Anonymous authentication scheme to edit, and click Apply without adding a value for Challenge method, the following errors might appear:

Messages for this page are listed below. 

* Challenge Method You must make at least one selection. 
 
* Challenge Redirect You must enter a value. 

Solution

You must include both a challenge method and a challenge redirect whenever you edit an anonymous authentication scheme.

X.509 Protected Resource and Single Sign Off

Problem

Single Sign Off might not work after accessing the resource with X.509 authentication. When the user is logged out with the logout URL and tries to access the resource in the same browser, authentication might not occur. Instead, the user should be asked for authentication using the certificate pop up.

This can occur with any Agent type.

Solution

After executing the logout URL, click on Clear SSL State from the browser as follows, and the access the X.509-protected resource:

From the browser window, open the Tools menu, click Internet Options, choose Content, and then Clear SSL state.

Authorization Issues

Problem: Constraint Error

An error is logged in the oam-server diagnostic log file whenever you create or edit an IPv4 range or temporal constraint:

.... refreshPolicy specified but no response collector supplied 

Cause

This is a message that is erroneously being logged at the ERROR level. The correct level of the message is INFO.

Cannot Access Authentication LDAP or Database

If the LDAP directory that is used for authentication is down or inaccessible (or the database that is configured as the policy store), it might be due to a heavy load or a timeout. You see a message when attempting to a protected resource that uses this LDAP or policy store.

Solution

  1. Manually shut down the registered LDAP or database.

  2. Restart the registered LDAP or database.

Cannot Find Configuration

Configuration Does Not Exist ...

If you attempt to create and apply configuration details for an OAM Server before configuring the OAM Server in the WebLogic Server domain, a message informs you of the following:

Configuration does not exist for path
/DeployedComponent/Server/oamServer/Instance/test

For more information, please see the server's error log for
an entry beginning with: Server Exception during PPR, #6.

To resolve this issue, you must configure the OAM Server in the WebLogic Server domain before you register the configuration with OAM 11g.

Could Not Find Partial Trigger

In the Administration Server output, you might see a "Could Not Find Partial Trigger" error (multiple times for each selected node when you click Policy Configuration tab or a host identifier node) and then you click any of other nodes in the navigation tree.

This does not block functionality.

Denial of Service Attacks

A denial-of-service attack (DoS attack) or distributed denial-of-service attack (DDoS attack) is an attempt to make a computer resource unavailable to its intended users. One common method of attack involves saturating the target (victim) machine with external communication requests, such that it cannot respond to legitimate traffic, or responds so slowly as to be rendered effectively unavailable.

Denial of service attacks are classified into Authenticated and Unauthenticated Requests, and further classified as:

Authenticated NAP Requests

For Authenticated NAP Requests, the OAM Server maintains a counter in the session and limits the number of retries. Despite this, after redirecting the user to an error page the user can repeat the cycle. This needlessly consumes server resources and can lead to OAM Server overloading.

Note:

To avoid OAM Server overloading with Authenticated NAP Requests, use relevant WebLogic overload configuration settings. These ensure that the server does not crash under load. However, this does not differentiate legitimate users from malicious users.

Authenticated HTTP Requests

You can handle a flood of HTTP Authenticated requests with a combination of WebLogic overload configuration and mod_security module settings.

Unauthenticated NAP Requests

Unauthenticated NAP Requests are handled by the WebLogic MDB pool throttling. This imits the number of NAP Requests that are forwarded to the OAM Server.

Again, this does not differentiate legitimate users from malicious ones.

Unauthenticated HTTP Requests

Configuring the mod_security module for the OHS server that front-ends the OAM Server enables rejection of malicious requests (unauthenticated HTTP Requests).

For more information, see:

Protecting the OAM Server from Crashing Under Load

If the number of requests to the OAM Server unexpectedly increases beyond what the server can handle, it could crash.

To limit the number of requests to the OAM Server:

  1. In the WebLogic Console, use the Message Driven Bean pool to restrict the number of NAP requests to the OAM Server.

    MDBeans pull NAP requests from the Server queue and deliver NAP requests to the Server for processing. Limiting the number of MDBean instances helps control the number of requests that are processed at a given time.

  2. In the WebLogic Console, configure the number of WebLogic worker threads that can be used (to restrict the number of requests to the OAM Server).

    MDBeans pull NAP requests from the server queue and deliver NAP requests to the Server for processing. Limiting the number of MDBean instances helps control the number of requests that are processed at a given time.

  3. In the WebLogic Console, configure the number of WebLogic worker threads that can be used (to restrict the number of requests to the OAM Server).

    See the topic on Thread Management in the guide to Oracle Fusion Middleware Performance and Tuning for Oracle WebLogic Server.

  4. In the WebLogic Console, specify a maximum incoming request size, complete message timeout, and set the number of file descriptors, to optimize performance as described in following topics in the Oracle Fusion Middleware Performance and Tuning for Oracle WebLogic Server:

    • Tuning Message Size

    • Tuning Complete Message Timeout

    • Tuning Number of File Descriptors

Compensating for Network Latency

Consider the scenario where Webgate sends an authentication request to the OAM Server. After successful credential collection and validation, the OAM Server creates the session and the relevant cookies (OAM_ID, ObSSOCookie). However, due to network latency, the response times out by the time the OAM Server sends it to the Webgate which triggers Webgate to re-send the authentication request to the Server. The OAM Server recognizes the session, then recreates the ObSSOCookie, and sends the response to the agent. If the network latency persists, the cycle continues in an infinite loop between the Server and the Webgate. The user is neither asked to login again nor presented with an error message.

Protecting OAM Servers from a Flood of HTTP Requests

ModSecurity is a Web application firewall (WAF) that can be deployed as part of the existing Apache-based Web server infrastructure. This module can be plugged into the OHS Server that front-ends the OAM Server. In this way, Mod_security module protects the OAM Server from denial of service attacks.

A flexible rule engine is at the heart of ModSecurity. It implements the ModSecurity Rule Language, a specialized programming language designed to work with HTTP transaction data. A new configuration directive uses the httpd-guardian script to monitor for Denial of Service (DoS) attacks. By default httpd-guardian defends against clients that send more than 120 requests in a minute, or more than 360 requests in five minutes.

See Also:

http://www.modsecurity.org/documentation/modsecurity-apache/2.5.12/html-multipage/configuration-directives.html#N10689

To protect from a flood of HTTP Requests

  1. Add the mod_security module to the OHS Server that front-ends the OAM Server.

  2. In the OHS Server configuration, set the configuration directive to use the httpd-guardian script to monitor for Denial of Service (DoS) attacks.

    Syntax:

    SecGuardianLog |/path/to/httpd-guardian
    

    Example:

    SecGuardianLog |/usr/local/apache/bin/httpd-guardian
    

Deployments with Freshly Installed OAM 10g Webgates

Use the OAM Server's diagnostic features to debug on the OAM Server side. This section includes the following topics:

Authentication Issues with OAM 10g Webgates

Use the following methods to troubleshoot authentication issues when you have freshly installed OAM 10g Webgates in your OAM 11g deployment.

  • Confirm that your request was protected using an http header trace like Internet Explorer HTTP Headers or Firefox Live HTTP Headers

  • Confirm that the request is sent to the OAM Server for authentication

    • GET /oam/server/obrareq.cgi?…..

    • Host: oam-server:port

Logout Issues with OAM 10g Webgates

Use the following methods to troubleshoot logout issues when you have freshly installed OAM 10g Webgates in your OAM 11g deployment.

  • Make liberal use of HTTP Header Trace

  • Confirm that the specific logout.html was copied to /access/oamsso folder in the 10g Webgate installation directory. If not present, you must create the logout.html as described in "Configuring Centralized Logout for 10g Webgate with OAM 11g Servers".

  • Change the OAM 10g Webgate's httpd.conf to remove the following lines:

    <LocationMatch "/oamsso/*">
    Satisfy any
    </LocationMatch>
    
  • From the Oracle Access Manager Console, confirm that the LogoutUrls parameter (/oamsso/logout.html) is configured for this Webgate

Diagnosing OAM 11g Initialization and Performance Issues

This section includes the following topics:

Diagnosing an Initialization Issue

Problem

OAM Server does not start up.

Solution

  1. Locate and review the OAM Server log file on the computer hosting the OAM Server.

    DOMAIN_HOME/servers/SERVER-NAME/logs/SERVER-NAME-diagnostics.log
    
  2. Enable logging for this computer, as described in Chapter 22, "Logging Component Event Messages":

    DOMAIN_HOME/config/fmwconfig/servers/SERVER-NAME/logging.xml
    
  3. Restart the OAM Server, observe the behavior, check the log file again if needed.

Diagnosing a Performance Issue

Problem

Monitoring the OAM Server reveals a significant spike in latency during authentication.

Solution

  1. Locate and review the OAM Server log file on the computer hosting the OAM Server.

    DOMAIN_HOME/servers/SERVER-NAME/logs/SERVER-NAME-diagnostics.log
    
  2. Enable logging for this computer, as described in Chapter 22, "Logging Component Event Messages":

    DOMAIN_HOME/config/fmwconfig/servers/SERVER-NAME/logging.xml
    
  3. Restart the OAM Server, observe the behavior, check the log file again if needed.

Diagnosing Out-of-Memory Issues With a Heap Dump

Problem

Debugging for all expression parsing and evaluation produced a significant performance drag within ~20 hours due to memory growth; running out of memory in ~50 hours.

Configuration: 2GB heap; 3 minute session timeout; jdbc connections tuned min=32 max=200; jdbc connection idle timeout disabled; jbo pool size min = 10 & max=150

Solution

To generate heap-dumps for comparison, you use the following command-line tools jmap for Sun jvm or jrcmd for jrockit jvm located under JAVA_HOME/bin.

For jrockit jvm

jrcmd pid <command> 
/jrockit_160_14_R27.6.5-32/bin/jrcmd 16775 heap_diagnostics
/jrockit_160_14_R27.6.5-32/bin/jrcmd 16775 print_threads
/jrockit_160_14_R27.6.5-32/bin/jrcmd 16775 jrarecording ....

For Sun jvm

jmap -histo <pid> 
jmap -dump:live,format=b,file=heap.bin <pid>

Disabling Windows Challenge/Response Authentication on IIS Web Servers

The IIS Web server on Windows supports Challenge/Response Authentication, which defaults to On when IIS is installed. This enables users to use their domain log-ins when requesting resources from IIS and can conflict with Oracle Access Manager's authentication.

For example, on the first request from an Internet Explorer (IE) browser to a resource on IIS protected by Oracle Access Manager with a basic authentication scheme, IE displays a login dialog box requesting a domain along with the user name and password login provided by Oracle Access Manager.

To disable Windows challenge/response authentication

  1. Launch the Microsoft Management Console for IIS.

  2. Select the Web Server Host under Internet Information Server in the left hand panel.

  3. Right click and select Properties.

  4. Scroll down and select Edit the Master Properties for WWW Service.

  5. Select the Directory Security tab.

  6. Select Edit Anonymous Access and Authentication Control.

  7. Complete the appropriate step for your platform:

    Windows 2000: Clear the Integrate Windows Authentication box.

  8. Click OK.

  9. In the Windows IIS properties screen, click OK.

  10. Close the Microsoft Management Console.

Changing UserIdentityStore1 Type Can Lock Out Administrators

An Identity Store that is designated as the System Store should not be edited to change the store type (from Embedded LDAP to OID, for instance) nor the connection URLs.

If you do need to change the Identity Store that is designated as the System Store should not be edited to change the store type, Oracle recommends that you create a new Identity Store and then edit that registration to mark it as your System Store.

IIS Web Server Issues

The following topics are provided to assist you:

Form Authentication or Pass-Through Not Working

If form authentication or pass-through functionality is not working, the problem might be that either "UseWebGateExtForPassthrough" parameter is not set to true in the Webgate profile or that webgate.dll is not configured as Wild Card Application Mapping in IIS. In such cases, Webgate does not perform authentication or authorization for HTTP "POST" requests for the resources protected by form-based authentication.

Solution: Confirm that the UseWebGateExtForPassthrough parameter is configured in the Webgate profile with a value of true and that webgate.dll is configured as Wild Card Application Mapping.

IIS and General Web Component Guidelines

Following are some general guidelines to follow when installing Oracle Access Manager Webgates with IIS Web servers.

Account Privileges: The account that performs Oracle Access Manager installation must have administration privileges. The user account that is used to run OAM services must have the "Log on as a service" right, which can be set by selecting Administrative Tools, Local Policy, Local Policies, User Rights Assignments, Log on as a service.

IIS 6 Web Servers: You must run the WWW service in IIS 5.0 isolation mode. This is required by the ISAPI postgate filter. During Oracle Access Manager installation, this is usually set automatically. If it is not, you must set it manually for the Default Web site.

Webgate for IIS 7 Web Server: To use Form-based authentication without enabling pass through functionality (for example, "access/oblix/apps/webgate/bin/webgate.dll" is an action in the Form-based authentication scheme), ensure that the entry "<add segment="bin"/>" is not present in the applicationHost.config file. If the entry is present, you must remove it. Use the following steps to check this entry:

  • Go to Windows\System32\inetsrv\config and open the file applicationHost.config.

  • Search for the <hiddenSegments> module and remove the entry <add segment="bin"/> if it is present.

Webgate: When installing IIS Webgates, setting various permissions for the /access directory is required for IIS Webgates only when you are installing on a file system that supports NTFS. For example, suppose you install the ISAPI Webgate in Simple or Cert mode on a Windows 2000 computer running the FAT32 file system. The last installation panel provides instructions for manually setting various permissions that cannot be set on the FAT32 file system. In this case, these instructions may be ignored.

Issues with IIS v6 Web Servers

On IIS 6 Web servers only, you must run the WWW service in IIS 5.0 isolation mode, which is a requirement of the ISAPI postgate filter. This scenario will work if you have 32-bit Oracle Access Manager binaries running on a 32-bit Windows operating system. However, there is an issue if you attempt to run a 32-bit postgate.dll on a 64-bit Windows machine with IIS running in 32-bit mode.

Problem

When running IIS in IIS5.0 isolation mode, you see the following message:

"ISAPI Filter 'C:\webgate\access\oblix\apps\webgate\bin\webgate.dll' could not be loaded due to a configuration problem.

Cause

The current configuration only supports loading images built for an AMD 64-bit processor architecture. The data field contains the error number.

Solution

To learn more about this issue, including how to troubleshoot this kind of processor architecture mismatch error, see the following Web site:

http://go.microsoft.com/fwlink/?LinkId=29349

For more information, see Help and Support Center at:

http://go.microsoft.com/fwlink/events.asp

Problem

IIS5 never existed as 64-bit. However, IIS v6's IIS5 Compatibility Mode on 64-bit Windows computers only runs as 64-bit.

Cause

It is architecturally impossible run IIS5 Isolation Mode 32- bit on 64-bit Windows, as described in documentation available through the following URLs:

http://www.microsoft.com/communities/newsgroups/en-us/default.aspx?dg=microsoft.pu
blic.inetserver.iis&tid=5dd07102-8896-40cc-86cb-809060fa9426&cat=en_US_
02ceb021-bb43-476d-8f8f-6c00a363ccf5&lang=en&cr=US&p=1
http://blogs.msdn.com/david.wang/archive/2005/12/14/HOWTO-Diagnose-one-cause-of-W3
SVC-failing-to-start-with-Win32-Error-193-on-64bit-Windows.aspx

Page Cannot Be Displayed Error

A "The page cannot be displayed" error that appears after configuring Webgate for pass-through functionality, indicates a configuration issue.

Solution: Confirm that the UseWebGateExtForPassthrough parameter is configured in the Webgate profile with a value of true and that webgate.dll is configured as Wild Card Application Mapping.

Removing and Reinstalling IIS DLLs

When Oracle Access Manager is running with Microsoft's IIS Web server, you must manually uninstall and reinstall the following ISAPI filters when reinstalling Oracle Access Manager.

  • tranfilter.dll

  • oblixlock.dll (if you installed Webgate)

  • webgate.dll (if you installed Webgate)

To remove and reinstall IIS DLLs

  1. Uninstall Oracle Access Manager.

  2. Manually uninstall the preceding DLLs.

  3. Reinstall Oracle Access Manager.Active Directory.

  4. Manually reinstall the DLLs.

Note:

These filters can change depending on the version of IIS you are using. If these filters do not exist or there are others present, contact Oracle to determine if the filters that are present need to be removed.

jps Logger Class Instantiation Warning is Logged on Authentication

A jps logger class instantiation warning is might appear on the back end upon authentication. However, this is a harmless warning and no action is required.

Languages and Translation

This section provides the following topics:

Automatically Generated Descriptions Are Not Translated

The automatically generated Description for each component of the Policy Configuration tab is not translated. This is expected and enables Administrators to change the Description to whatever they require. Following such a change, translation by Oracle is not possible.

Locales, Languages, and Oracle Access Manager Console Login Page

When the browser locale is not supported, the Oracle Access Manager Console Login page shows as server locale. It should fall back to English. This is the expected behavior:

  • If the client Locale is not supported, Oracle Access Manager falls back to the server locale.

  • If the server locale is not supported, Oracle Access Manager falls back to English.

When users select an unsupported language and come to the Oracle Access Manager SSO page, it shows as server locale (German, for example). However, after logging in, all the pages are displayed as English.

To fall back to English

Disable the Oracle Access Manager SSO page and the original Oracle Access Manager login page also falls back to English.

Console Looks Messy

The Oracle Access Manager Console displays policies and resources oddly when the input Fusion Applications configuration file for remote registration is not in UTF-8 format or when the OAM Server is not started in UTF-8 locale (en_US.utf8, for instance).

Be sure to use UTF-8 encoding if creating a Fusion Applications configuration file for the remote registration tool, oamreg, to generate authentication policies and protected resources. Also, be sure to start OAM Server in UTF-8 locale machines. Otherwise, the Oracle Access Manager Console might display policies and resources oddly following successful inband registration.

Login Failure for a Protected Page

Problem

After installing OAM and protecting a page using a physical host and port, register the partner application using the OHS physical host and port. Login fails to prompt the user for credentials when accessing the protected page. The log file shows that the URL is re-directed to a Virtual Host despite the fact that all configuration and registration is setup correctly.

Solution

Remove any Virtual Host Directives from httpd.conf when protecting a page using the Oracle HTTP Server (OHS) physical host and port.

OAM Metric Persistence Timer IllegalStateException: SafeCluster

Problem

After using the WebLogic Configuration Wizard to create an OAM Server cluster on two computers, and starting AdminServer, all servers start up properly. After shut down, a third server is added using the WebLogic Server Administration Console to create a new managed server and add it to the cluster. The third server goes into Running mode when started, with some exceptions in the start up log.

... Exception in thread "OAM Metric Persistence Timer"

Solution

in addition to the actions in the WebLogic Administration Console, you must register the server using the Oracle Access Manager Console to ensure that the server can identify itself.

Note:

When adding and registering a second server instance for the same computer, all port numbers must differ: OAM Proxy port; the "port" that must match the one in the WebLogic Server Console; and the Coherence port.

For server registration details, see "Managing Individual OAM Server Registrations".

Partial Cluster Failure and Intermittent Login and Logout Failures

Problem

In the event of a partial outage of Oracle Access Manager (on some, but not all instances of the cluster), end users might see intermittent login and logout failures.

Workarounds

  1. Remove OHS from the deployment

  2. Configure the OHS cluster such that each OHS instance is pinned to a WebLogic Server instance.

  3. The WebLogic Server container with the malfunctioning Oracle Access Manager application must be removed from service (shutdown) and brought back up upon recovery.

Registration Issues

Problem: Remote Registration Tool Failure

Solution

Ensure that the agent name is unique (does not already exist) and that the AdminServer is running.

Problem: No ObAccessClient.xml File Generated

Solution

Protected and public resources must be described as relative URLs of the format '/index.html'. If the resource does not begin with a '/', no ObAccessClient.xml file will be generated.Verify the protected and public resource URLs and ensure all begin with a "/". For more information, see "About the Resource URL".

Problem: Partner Registration Failure

Partner registration can fail if you do not supply a unique agent name, which is also used to create an application domain. The agent name and application domain name must be the same and must be unique. Using the oamreg validate command can fail when the agent name does not match the application domain name.

Solution

Ensure that the agent name and application domain name are the same.

Rowkey does not have any primary key attributes Error

While browsing across the Resources table in the Resource Type tab the following error message is logged:

@ <Error> 
<oracle.adfinternal.view.faces.model.binding.CurrencyRowKeySet> 
@ <BEA-000000> <ADFv: Rowkey does not have any primary key attributes. Rowkey:
oracle.jbo.Key[], table: model.ResTypeVOImpl@620289.> 

This is harmless and does not hinder any functionality.

SELinux Issues

Delivered with Oracle Enterprise Linux, SELinux modifications provide a variety of policies through the use of Linux Security Modules (LSM) within the Linux kernel.

SELinux requires performing additional steps after installing Oracle Access Manager Webgates and before starting the associated Web server.

Problem

The following errors could be reported in logs/console when starting a Web server on Linux distributions that have more strict SELinux policies in place (after installing an Oracle Access Manager Webgate):

OAM 11g Webgate

$Webgate_OH/webgate/ohs/lib/webgate.so: cannot restore segment prot after reloc: 
Permission denied. 

OAM 10g Webgate

$Webgate_install_dir/access/oblix/apps/webgate/bin/webgate.so: cannot restore segment prot after reloc: 
Permission denied. 

Cause

These errors are reported due to Secure Linux security context policies on files.

Solution

To avoid these errors and start the Web server, run following chcon commands to change the security context on files after installing each Oracle Access Manager Web component and before restarting the associated Web server. For more information on the chcon command, see your Linux documentation.

  1. Run chcon -t texrel_shlib_t PATH_TO_LIBWEBPLUGINS.SO. For example:

    chcon -t texrel_shlib_t  /Webgate_install_dir/access/oblix/lib/webgate.so 
    ... and libxmlengine.so
    
  2. Run chcon -t texrel_shlib_t PATH_TO_LIBWEBGATE.SO. For example:

    chcon -t texrel_shlib_t  /Webgate_install_dir/access/oblix/apps/webgate/
    bin/webgate.so 
    

Session Issues

This section provides the following details:

Session Impersonation Not Enabled by Default

Session impersonation is not enabled by default. You can update the value in oam-config.xml and update the version of oam-config.xml to automatically propagate the ImpersonationConfig status to all managed servers without a restart.

To enable Session Impersonation

  1. Back up DOMAIN_HOME/config/fmwconfig/oam-config.xml.

  2. Set ImpersonationConfig to true:

    <Setting Name="ImpersonationConfig" Type="htf:map">
         <Setting Name="EnableImpersonation" Type="xsd:boolean">false</Setting> 
    </Setting>      
    
  3. Configuration Version: Increment the Version xsd:integer as shown in the next to last line of this example (existing value (25, here) + 1):

    Example:

    <Setting Name="Version" Type="xsd:integer">
      <Setting xmlns="http://www.w3.org/2001/XMLSchema"
        Name="NGAMConfiguration" Type="htf:map:> 
      <Setting Name="ProductRelease" Type="xsd:string">11.1.1.3</Setting>
        <Setting Name="Version" Type="xsd:integer">25</Setting>
    </Setting>      
    
  4. Save oam-config.xml.

Sessions with Oracle Access Manager with Oracle Identity Federation

When Oracle Access Manager 11g is integrated with Oracle Identity Federation, and you use the Oracle Access Manager Session Management function to clear the session, only the Oracle Access Manager session is cleared (not the Oracle Identity Federation session).

SSL versus Open Communication

If both the SSL and Open ports of the Managed Server are enabled, then the Managed Server is set to the SSL port by default.

If you must use the non-ssl port, the credential collector URL the authentication scheme must be set to the absolute URL which points to 'http' as the protocol and non-ssl port.

Start Up Issues

Problem: AdminServer Startup (or Remote Registration Tool Failure) on AIX Platforms

AdminServer start up fails with following message:

"java.net.SocketException: 
No buffer space available".

Configration for the number of AIX file descriptors set for the operating sytem is substantially high (ulimit) resulting in a buffer overflow that causes remote registration failure with the following message:

The ulimit value is application dependent and applies exclusively to application program data and the application stack. The default number of open files setting (2000) is typically sufficient for most applications. If the value is too low, errors might occur when opening files or establishing connections. Because this value limits the number of file descriptors that a server process might open, a value that is too low prevents optimum performance. For the AIX operating system, the default setting is 2000.

Solution

Increasing the ulimit file descriptor limits might improve performance. Increasing some of the other limits might be needed depending on your application.

  1. Log in as root.

  2. Perform the following steps to change the open file limit to 10,000 files:

    1. Open the command window.

    2. Locate and edit /etc/security/limits file to add the following lines to the user account on which the AdminServer process runs:

      nofiles =  10000   
      nofiles_hard = 10000
      
    3. Save the file and restart AIX.

  3. In a command window, decrease the TCP_TIMEWAIT interval with the following command to set the state to 15 seconds (which allows TCP to release closed connections faster and increases the number of available resources for open connections).

    /usr/sbin/no –o tcp_timewait =1
    
  4. Tune the following parameters to 256k, as shown:

    no -a |grep space   
          tcp_recvspace = 262144   
          tcp_sendspace = 262144   
          udp_recvspace = 262144  
          udp_sendspace = 262144
    
  5. Tune the following parameters as indicated here:

    no -o rfc1323=1  
    no -o sb_max=4194304
    

Problem: Connection to OAM Server could not be established: Exception in connecting to server. Connection refused.

Cause:

This is normal and expected behavior for the Managed Server where the OAM Server runs because the IAMSuiteAgent agent is started before the OAM Server.

The IAMSuiteAgent is deployed on every WebLogic container. When the WebLogic container starts, the agent tries to connect to the OAM Server. If it fails to connect, this message is logged and the agent tries to establish the connection in subsequent requests. When the agent is successful, this message is no longer displayed.

Solution

If the connection to the OAM Server is not successful, the IAMSuiteAgent falls back and the WebLogic container handles protection (including login), if it is configured.

Synchronizing OAM Server Clocks

The state of a session is the source of truth for relying parties. Synchronization of system clocks of the various Servers is required.

The system clock of the relying party might be out of synchronization with the SME clock. If the relying party's clock is:

For example, if a Web server clock is ahead of the server clock, a request sent from the Webgate to the OAM Server will contain a time that, to the OAM Server, has not yet occurred. This can cause login events to fail. When running in Simple or Cert mode, time stamps might become out of sync, or the client certificate might appear to be invalid.

Note:

To avoid event notification issues, ensure that all OAM Server clocks are synchronized to Time Services such as NIST internet time service.

For successful operation:

Using Coherence

Oracle Access Manager 11g uses Oracle Coherence to replicate session states within a distributed installation. Coherence is used to communicate state changes between the Oracle Access Manager Console and OAM Servers.

Consider the following 2 distributed deployment topologies. Coherence relies on User Datagram Protocol (UDP) for cluster discovery and heartbeat. If a firewall exists between certain components of OAM 11g, then the corresponding UDP ports used by Coherence must be open. Otherwise, OAM 11g might not work correctly.

For example, the UDP ports used by Coherence must be opened as follows:

Oracle Access Manager 11g uses Oracle Coherence to provide a distributed cache with low-data access latencies and to transparently move data between distributed caches (and into the session store). Session data is redundant across these tiers. For example, when a session is created, it then exists within the local cache on the server that created it, the distributed cache, and (if enabled) within the session store database as well. For more information, see "Oracle Coherence and Session Management".

WARNING:

Oracle recommends that you do not modify Oracle Coherence settings unless requested to do so by an Oracle Support Representative.

Whether you are viewing Oracle Coherence settings for an individual server instance or Oracle Coherence details that are common to all OAM Servers, Oracle recommends that you do not modify Oracle Coherence settings unless requested to do so by an Oracle Support Representative.

Oracle Coherence logging appears in the WebLogic Server log only. There is no bridge from Oracle Coherence logging to Oracle Access Manager logging.

See Also:

Oracle Coherence documentation.

Validation Errors

Problem: Resource not added to Authentication or Authorization Policy

While creating an Authentication or Authorization Policy, if you add a resource that is already used in another Authentication or Authorization Policy, a validation error appears when you click Apply. This is expected.

If you click OK in the error window and then attempt to add a valid resource that is not used within another Authentication or Authorization Policy, the resource is not added and the Authentication or Authorization Policy is not created.

Solution

  1. Click Apply and close the Authentication or Authorization Policy page.

  2. From the navigation tree, click the named policy again, click the Edit to open the page, and add the new resource.

Problem: Validation Failure - "description" attribute is not valid

A validation error appears if you enter an optional description longer than 200 characters.

Solution

Keep optional descriptions to 200 characters in length and less than 10 lines.

Web Server Issues

The following issues with Web servers may arise:

Server Fails on an Apache Web Server

Symptom: You are running an Apache Web server, and an OAM Server fails, displaying the following message:

libthread panic: cannot create new lwp
(PID: 9035 LWP 2). stackrace:
ff3424cc
0

This symptom may be caused by the Apache Web server launching more instances of itself. This can happen when the server determines that more instances are needed to service the number of connections between one or more Webgates and the OAM Server.

The additional instances create even more connections, which exceed the number of connections by the OAM Server.

Solution: Reduce the number of MinSpareServers, MaxSpareServers, StartServers, and MaxClients parameters.

Go to the OAM Server's configuration directory and open the http.d configuration file.

Recommended parameter settings:

  • MinSpareServers 1

  • MaxSpareServers 5

  • StartServers 3

  • MaxClients 5

Apache v2 on HP-UX

When running Apache v2 on HP-UX, do not use nobody for User or Group, because shared memory may not work. Instead, use your login name as User Name with a your group as Group Name On HP-UX (on Solaris, "www" is equivalent to "nobody").

When running Apache v2 on HPUX 11.11, ensure that the AcceptMutex directive in the Apache httpd.conf file is set to "fcntl". If the directive is not present, add it to the httpd.conf file (AcceptMutex fcntl). For more information, see:

http://issues.apache.org/bugzilla/show_bug.cgi?id=22484

Apache v2 Bundled with Red Hat Enterprise Linux 4

After installing a Webgate on vendor-bundled Apache, the Web server may give the following error upon startup:

Error: Cannot load libgcc_s.so.1 library - Permission denied. 

Solution: Change the Security-Enhanced Linux (SELinux) policy rules for Oracle Access Manager Webgates as described in "Tuning Apache/IHS v2 for Oracle Access Manager Webgates".

Apache v2 Bundled with Security-Enhanced Linux

Errors might be reported in WebServer logs/console when starting a Web server on Linux distributions, which have stricter SELinux policies in place, after installing an Oracle Access Manager Web component. You can avoid these errors by running appropriate chcon commands for the installed Web component before restarting the Web server.

See Also:

"SELinux Issues"

Apache v2 on UNIX with the mpm_worker_module for Webgate

The following item is required only if you compile Apache v2 for Webgate on UNIX with the mpm_worker_module. In this case, you need to modify the thread.c file from the Apache source for the UNIX environment. Making this change ensures that the default pthread stacksize for Webgate produces optimal performance during multithreaded server implementation. If this change is not made, the default pthread stack size would not be sufficient for Webgate and could result in a crash.

Apache 2.0 does not support the ThreadStackSize option. Therefore:

  • With UNIX-based Apache v2.1 and later you must use the ThreadStackSize directive to set the size of the stack (for autodata) of threads that handle client connections and call modules to help process those connections.

  • With UNIX-based Apache 2, it is best to use the compilable source while adding the mpm_worker_module and changing the thread.c file to avoid a stack overflow.

The following procedure shows how to modify the Apache v2.0 thread.c file to provide the default pthread stacksize needed by Webgate for optimal performance during multi-threaded server implementation. For details about the Apache v2.1+ ThreadStackSize directive, see http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadstacksize.

Note:

The following procedure should be performed only for the Apache 2.0 Webgate. Otherwise, the default pthread stack size is not sufficient for the Webgate and could result in a crash.

To modify the Apache v2.0 thread.c file for Webgate in a UNIX environment

  1. Locate the thread.c file. For example:

    APACHE 2.0.52 source/srclib/apr/threadproc/unix/thread.c
    
  2. Locate the function named apr_threadattr_create(apr_threadattr_t **new,apr_pool_t *pool) in the following code segment:

    **new,apr_pool_t *pool) in the following code segment:
    1-----> apr_status_t stat; 
    2 
    3-----> (*new) = (apr_threadattr_t *)apr_pcalloc(pool, sizeof(apr_threadattr_t)); 
    4-----> (*new)->attr = (pthread_attr_t *)apr_pcalloc(pool, sizeof(pthread_attr_t)); 
    5 
    6-----> if ((*new) == NULL || (*new)->attr == NULL) { 
    7----->              return APR_ENOMEM; 
    8-----> } 
    9 
    10----->(*new)->pool = pool; 
    11----->stat = pthread_attr_init((*new)->attr); 
    12 
    13-----> if (stat == 0) { 
    14----->            return APR_SUCCESS; 
    15-----> } 
    16----->#ifdef PTHREAD_SETS_ERRNO 
    17----->stat = errno; 
    18----->#endif 
    19 
    20----->return stat; 
    21 
    
  3. Add the following code before line 13 shown earlier.

    int stacksize = 1 << 20; 
    pthread_attr_setstacksize(&(*new)->attr, stacksize);
    
  4. Run configure, make, and make install to set up the Apache Web server with the mpm_worker_module.

Domino Web Server Issues

Failure Authentication Event: For Domino Web servers, the redirection of a URL through Oracle Access Manager may not work if the authentication type is set as Basic Over LDAP and the URL to be redirected is mentioned as one of the following:

Either a relative path present on the same Web server
Or the Full path URL on the same Web server containing a computer name defined in the host identifier string combinations.

To overcome a failure authentication event, you must set the redirected URL with a computer name that is not defined under the host identifier group. For example, the IP address of the computer.

This problem does not occur with a form-based authentication type.

Header Variables: It may not be possible to pass header variables other than REMOTE_USER to Webgates installed on Lotus Notes Domino Web servers when using Client Certificate authentication scheme.

For example, header variables cannot be set on the one request where Client Certificate authentication occurs. However, all other requests do allow header variables to be set.

For more information, see Chapter 31, "Configuring Lotus Domino Web Servers for 10g Webgates".

Errors, Loss of Access, and Unpredictable Behavior

Symptom: If you installed Oracle Access Manager on UNIX under a different user ID than you used to create your Web server instance, Oracle Access Manager can become unstable. Users may experience behavior such as:

  • Random bug report pages

  • Failure to write to log file errors

  • Loss of access to Web pages

Solution: Change file permissions using the chown command. Change the Oracle Access Manager directory to the same user ID that you used to create your Web server instance.

Known Issues for ISA Web Server

Webgate uses ISAPI extension for displaying user deny error message and for displaying the diagnostic page. However, ISA 2006 does not support extensions. Therefore:

  • If the user is denied access by Webgate, the user gets Page Cannot be displayed error message instead of Oracle Access Manager denied access error message.

  • The following diagnostic URL does not work for ISA: http(s)://hostname:port/access/oblix/apps/webgate/bin/webgate.dll?progid=1 for webgate .

Oracle HTTP Server Fails to Start with LinuxThreads

After installing a Webgate instance on an Oracle HTTP Server, the server does not start up. This occurs because Oracle Access Manager uses an older Linux threading model.

Note:

When running Oracle Access Manager, LinuxThreads is used by default. This requires setting the environment variable LD_ASSUME_KERNEL to 2.4.19. If you are using NPTL with Oracle Access Manager, you do not set LD_ASSUME_KERNEL to 2.4.19.9

Solution: When using LinuxThreads mode, comment out the Perl module in the httpd.conf file, update the LD_ASSUME_KERNEL environment variable, and restart, as described in the following procedure.

To resolve the failure to start Oracle HTTP Server in LinuxThreads mode

  1. Comment out the Perl module in the httpd.conf file in the following location:

    Oracle HTTP Server 11g: ORACLE_INSTANCE/config/OHS/<ohs_name>/httpd.conf

    Oracle HTTP Server v2: OH$/ohs/conf/httpd.conf

    Oracle HTTP Server v1.3: OH$/Apache/Apache/conf/httpd.conf

  2. To update the LD_ASSUME_KERNEL value, open the following file in a text editor:

    OH$/opmn/conf/opm.xml
    
  3. Find the following line:

    <process-type id="HTTP_Server" module-id="OHS">
    
  4. 
    

    Add the following information under the line you found in the previous step:

    <environment>
    <variable id="LD_ASSUME_KERNEL" value="2.4.19" />
    </environment>
    
  5. Save this file.

  6. Run the following commands to implement your changes:

    opmnctl stopall
    opmnctl startall
    

Oracle HTTP Server Webgate Fails to Initialize On Linux Red Hat 4

This situation might arise whether you are using Oracle Access Manager with LinuxThreads or NPTL.

Symptom: Webgate fails to initialize when installed on an Oracle HTTP Server running Red Hat Enterprise Server version 4.0 with a kernel version lower than 2.6.9-34.EL. Version 2.6.9-34.EL is supplied with the Red Hat version 4, update 3.

Solution: To prevent this problem, you must upgrade to Red Hat version 4, update 3 or higher.

Oracle HTTP Server Web Server Configuration File Issue

Problem

With Oracle Application Server 10.1.x, OC4J, when the httpd.conf file is modified automatically during Webgate installation, it can be corrupted.

Solution

Before installing Webgate, run the following command to prevent the httpd.conf file from being overwritten.

$ORACLE_HOME/dcm/bin/dcmctl updateConfig -ct ohs 

Issues with IIS v6 Web Servers

On IIS 6 Web servers only, you must run the WWW service in IIS 5.0 isolation mode, which is a requirement of the ISAPI postgate filter. This scenario will work if you have 32-bit Oracle Access Manager binaries running on a 32-bit Windows operating system. However, there is an issue if you attempt to run a 32-bit postgate.dll on a 64-bit Windows machine with IIS running in 32-bit mode.

Problem

When running IIS in IIS5.0 isolation mode, you see the following message:

"ISAPI Filter 'C:\webgate\access\oblix\apps\webgate\bin\webgate.dll' could not be loaded due to a configuration problem.

Cause

The current configuration only supports loading images built for an AMD 64-bit processor architecture. The data field contains the error number.

Solution

To learn more about this issue, including how to troubleshoot this kind of processor architecture mismatch error, see the following Web site:

http://go.microsoft.com/fwlink/?LinkId=29349

For more information, see Help and Support Center at:

http://go.microsoft.com/fwlink/events.asp

Problem

IIS5 never existed as 64-bit. However, IIS v6's IIS5 Compatibility Mode on 64-bit Windows computers only runs as 64-bit.

Cause

It is architecturally impossible run IIS5 Isolation Mode 32- bit on 64-bit Windows, as described in documentation available through the following URLs:

http://www.microsoft.com/communities/newsgroups/en-us/default.aspx?dg=microsoft.pu
blic.inetserver.iis&tid=5dd07102-8896-40cc-86cb-809060fa9426&cat=en_US_
02ceb021-bb43-476d-8f8f-6c00a363ccf5&lang=en&cr=US&p=1
http://blogs.msdn.com/david.wang/archive/2005/12/14/HOWTO-Diagnose-one-cause-of-W3
SVC-failing-to-start-with-Win32-Error-193-on-64bit-Windows.aspx

PCLOSE Error When Starting Sun Web Server

Symptom: When attempting to start the Sun Web server, you get an error like the following:

Unable to start, PCLOSE

Solution: A number of problems can cause this error:

  • A syntax error in your obj.conf file

  • Leading spaces in your obj.conf file

  • Installing Oracle Access Manager as a different user ID than what you used to create your Web server instance

  • A carriage return at the end of the obj.conf file

Removing and Reinstalling IIS DLLs

When Oracle Access Manager is running with Microsoft's IIS Web server, you must manually uninstall and reinstall the following ISAPI filters when reinstalling Oracle Access Manager.

  • tranfilter.dll

  • oblixlock.dll (if you installed Webgate)

  • webgate.dll (if you installed Webgate)

To remove and reinstall IIS DLLs

  1. Uninstall Oracle Access Manager.

  2. Manually uninstall the preceding DLLs.

  3. Reinstall Oracle Access Manager.Active Directory.

  4. Manually reinstall the DLLs.

Note:

These filters can change depending on the version of IIS you are using. If these filters do not exist or there are others present, contact Oracle to determine if the filters that are present need to be removed.

Windows Native Authentication

Problem

After setting up Windows Native Authentication, and accessing the WNA-protected page, the browser might give an error indicating that the user name and/or password are incorrect.

Cause

The Identity Store used by Oracle Access Manager might not point to Windows Active Directory. By default, the identity store is Embedded LDAP.

Solution

  1. In the Oracle Access Manager Console, review the identity store configuration: System Configuration, Data Sources, User Identity Store.

  2. Confirm the LDAP store settings point to Active Directory.