This chapter provides troubleshooting tips.
AdminServer Won't Start if the Wrong Java Path Given with WebLogic Server Installation
Disabling Windows Challenge/Response Authentication on IIS Web Servers
Changing UserIdentityStore1 Type Can Lock Out Administrators
jps Logger Class Instantiation Warning is Logged on Authentication
OAM Metric Persistence Timer IllegalStateException: SafeCluster
Partial Cluster Failure and Intermittent Login and Logout Failures
OAM is a business critical system; downtime comes with a potentially high cost to your business. The goal of system analysis is to quickly isolate and correct the cause of any problem. This requires a big picture view of your system and the tools to observe the live system and correlate components to the bigger picture.
To assist administrators in performing a quick diagnosis, this section provides the following topics:
System analysis includes understanding how the product works, what can go wrong, how likely the scenarios are, and the consequences or observable issues.
System problems can be divided into two basic categories:
Cascading catastrophic failure
Gradual breakdown in performance
Cascading catastrophic failure might be caused by:
LDAP server is loaded and unresponsive
Morning peak load starts
Webgates send requests to the primary OAM Server
Webgate requests time-out and Webgates retry to secondary OAM Server
Gradual breakdown in performance might occur over time when, for example:
OAM is sized and rolled out for 10,000 users and 500 groups
Over the course of a year, the number of users and groups increases significantly (to 50,000 users and 250 groups for example)
For information on the most commonly encountered issues, see the following topics:
This topic provides symptoms, probable cause, and steps to diagnose the following issues:
Symptoms: Operational Slowness
Poor user experience
Agent timeouts lead to retries
Non-OAM load might be impacting OAM operations
Capacity problems due to gradual increase in peak load
Symptoms: Total loss of service
Outage of all LDAP servers
The load balancer is timing out old connections
Shut down the LDAP server.
Restart your browser.
Try to access a protected site.
Review errors in the OAM Server log file, as described in Chapter 22 (alternatively, in Chapter 26).
Try to access Oracle Access Manager Console.
Observe errors in WebLogic AdminServer log file.
Bring up the LDAP server again.
Retry access to a protected application.
Retry access to the Oracle Access Manager Console.
Correct the issue based on the requirements in your environment.
This topic provides symptoms, probable cause, and steps to diagnose the following issues:
Poor user experience due to slow operations
Agent timeouts and retry can result in extra load
CPU cycles
Memory issues
Symptoms: Interference with Other Services on the Host
Poor user experience due to slow operations
Agent timeouts and retry may result in extra load
CPU cycle contention
Memory contention
File system full
Shut down the OAM Server
Try to access a Webgate or mod_osso protected resource
Bring up the OAM Server
Use the Access Tester to test authentication and authorization as described in Chapter 14.
Use 'top' to figure out the CPU and Memory consumption of the OAM Server as you use the access tester
Get a thread dump of the OAM Server.
Shut down the OAM AdminServer
Restart your browser and access a protected resource, which should work.
Use remote registration to register a new partner, as described in Chapter 10 (this should fail).
Startup OAM AdminServer.
This topic provides symptoms, probable cause, and steps to diagnose time issues between agents and servers.
Symptoms: Difference in Clock time Between Agent and Server
High CPU usage at both agent and server
User experiences a system hang
Agent thinks the token issued by the server is invalid
Agent keeps going back to the server to re-issue the token
Access protected resource.
Confirm: Client access hangs.
Confirm: High CPU usage on agent and server.
The audit and session functions are both write intensive operations. The policy database can be tuned for read intensive service.
Audit and session operations are slow
File system on the OAM Server is full with audit data that is not yet written to the database
Loss of in-memory session data when one of the servers in the cluster fails
Database is not tuned for write intensive operations
Database is unavailable due to maintenance
Space issues in the database
Shut down the database used to store Audit and Session data.
Try to access a protected resource.
Review error and warning messages in the OAM Server log files, as described in Chapter 22 (alternatively, in Chapter 26).
This topic provides symptoms, probable cause, and steps to diagnose the following issues:
Changes to policy do not take immediate effect
Changes to system configuration do not take immediate effect
Servers being too busy handling runtime requests (CPU contention)
Coherence network slowness
Diagnosis: See "About Policy Store Database Issues"
This topic provides symptoms, probable cause, and steps to diagnose policy database issues.
Symptoms: No policy changes are allowed; no impact on runtime
Database is unavailable (down for maintenance)
Space issues in the database
Shut down the database containing OAM policies.
Try to access a protected resource and observe the runtime access is not impacted.
Try to access the Oracle Access Manager Console to edit policies, and then observe errors in the AdminServer log file.
Administrators performing updates concurrently will result in an inconsistent state within the system configuration of the Oracle Access Manager Console.
Concurrent configuration updates are not supported.
Only one administrator should be allowed to modify the system configuration at any given time.
WebLogic Server (wls1035_generic) installation is successful on Windows 64-bit with 32-bit Java (jdk1.6.0_24). When setup.exe is executed you must provide the path of the 64-bit java (jdk1.6.0_23) to successfully launch the install shield.
If you provide the 32-bit Java (jdk1.6.0_24) path, the install shield is not launched. However, if you execute config.cmd from \Middleware\Oracle_IDM1\common\bin, by default 32-bit Java (jdk1.6.0_24) path is used, but after successful installation Oracle Access Manager installation, you cannot start AdminServer.
On Windows host, the path to 32-bit JAVA_HOME (c:\program files (x86)\java\jdkxxx) is not correctly handled by the startWeblogic.cmd. Replacing SUN_JAVA_HOME to use the path with the shorter name (c:\progra~2\java\jdkxxxx) works fine.
On Windows, the shorter names can be seen by executing "dir /X".
Alternatively, you can set windows cmd shell variable JAVA_HOME to path with shorter name and execute startWeblogic.cmd within that. For example:
>set JAVA_HOME=c:\progra~2\java\jdkXXX
>startweblogic.cmd
A unique identifying name for each Agent registration is preferred. However:
If the Agent Name exists, no error occurs and the registration does not fail. Instead, Oracle Access Manager creates the policies if they are not already in place.
If the host identifier exists, the unique Agent Base URL is added to the existing host identifier and registration proceeds.
The number of characters allowed in a URL are based on browser version.
Ensure that your applications do not use URLs that exceed the length that Oracle Access Manager and the browser can handle.
This section provides the following information:
Challenge Redirect URL can be NULL; however, Challenge Method cannot be NULL.
If you open the Anonymous authentication scheme to edit, and click Apply without adding a value for Challenge method, the following errors might appear:
Messages for this page are listed below. * Challenge Method You must make at least one selection. * Challenge Redirect You must enter a value.
You must include both a challenge method and a challenge redirect whenever you edit an anonymous authentication scheme.
Single Sign Off might not work after accessing the resource with X.509 authentication. When the user is logged out with the logout URL and tries to access the resource in the same browser, authentication might not occur. Instead, the user should be asked for authentication using the certificate pop up.
This can occur with any Agent type.
After executing the logout URL, click on Clear SSL State from the browser as follows, and the access the X.509-protected resource:
From the browser window, open the Tools menu, click Internet Options, choose Content, and then Clear SSL state.
An error is logged in the oam-server diagnostic log file whenever you create or edit an IPv4 range or temporal constraint:
.... refreshPolicy specified but no response collector supplied
This is a message that is erroneously being logged at the ERROR level. The correct level of the message is INFO.
If the LDAP directory that is used for authentication is down or inaccessible (or the database that is configured as the policy store), it might be due to a heavy load or a timeout. You see a message when attempting to a protected resource that uses this LDAP or policy store.
Manually shut down the registered LDAP or database.
Restart the registered LDAP or database.
If you attempt to create and apply configuration details for an OAM Server before configuring the OAM Server in the WebLogic Server domain, a message informs you of the following:
Configuration does not exist for path /DeployedComponent/Server/oamServer/Instance/test For more information, please see the server's error log for an entry beginning with: Server Exception during PPR, #6.
To resolve this issue, you must configure the OAM Server in the WebLogic Server domain before you register the configuration with OAM 11g.
In the Administration Server output, you might see a "Could Not Find Partial Trigger" error (multiple times for each selected node when you click Policy Configuration tab or a host identifier node) and then you click any of other nodes in the navigation tree.
This does not block functionality.
A denial-of-service attack (DoS attack) or distributed denial-of-service attack (DDoS attack) is an attempt to make a computer resource unavailable to its intended users. One common method of attack involves saturating the target (victim) machine with external communication requests, such that it cannot respond to legitimate traffic, or responds so slowly as to be rendered effectively unavailable.
Denial of service attacks are classified into Authenticated and Unauthenticated Requests, and further classified as:
NAP Requests
HTTP Requests
For Authenticated NAP Requests, the OAM Server maintains a counter in the session and limits the number of retries. Despite this, after redirecting the user to an error page the user can repeat the cycle. This needlessly consumes server resources and can lead to OAM Server overloading.
Note:
To avoid OAM Server overloading with Authenticated NAP Requests, use relevant WebLogic overload configuration settings. These ensure that the server does not crash under load. However, this does not differentiate legitimate users from malicious users.You can handle a flood of HTTP Authenticated requests with a combination of WebLogic overload configuration and mod_security module settings.
Unauthenticated NAP Requests are handled by the WebLogic MDB pool throttling. This imits the number of NAP Requests that are forwarded to the OAM Server.
Again, this does not differentiate legitimate users from malicious ones.
Configuring the mod_security module for the OHS server that front-ends the OAM Server enables rejection of malicious requests (unauthenticated HTTP Requests).
For more information, see:
If the number of requests to the OAM Server unexpectedly increases beyond what the server can handle, it could crash.
To limit the number of requests to the OAM Server:
In the WebLogic Console, use the Message Driven Bean pool to restrict the number of NAP requests to the OAM Server.
MDBeans pull NAP requests from the Server queue and deliver NAP requests to the Server for processing. Limiting the number of MDBean instances helps control the number of requests that are processed at a given time.
In the WebLogic Console, configure the number of WebLogic worker threads that can be used (to restrict the number of requests to the OAM Server).
MDBeans pull NAP requests from the server queue and deliver NAP requests to the Server for processing. Limiting the number of MDBean instances helps control the number of requests that are processed at a given time.
In the WebLogic Console, configure the number of WebLogic worker threads that can be used (to restrict the number of requests to the OAM Server).
See the topic on Thread Management in the guide to Oracle Fusion Middleware Performance and Tuning for Oracle WebLogic Server.
In the WebLogic Console, specify a maximum incoming request size, complete message timeout, and set the number of file descriptors, to optimize performance as described in following topics in the Oracle Fusion Middleware Performance and Tuning for Oracle WebLogic Server:
Tuning Message Size
Tuning Complete Message Timeout
Tuning Number of File Descriptors
Consider the scenario where Webgate sends an authentication request to the OAM Server. After successful credential collection and validation, the OAM Server creates the session and the relevant cookies (OAM_ID, ObSSOCookie). However, due to network latency, the response times out by the time the OAM Server sends it to the Webgate which triggers Webgate to re-send the authentication request to the Server. The OAM Server recognizes the session, then recreates the ObSSOCookie, and sends the response to the agent. If the network latency persists, the cycle continues in an infinite loop between the Server and the Webgate. The user is neither asked to login again nor presented with an error message.
ModSecurity is a Web application firewall (WAF) that can be deployed as part of the existing Apache-based Web server infrastructure. This module can be plugged into the OHS Server that front-ends the OAM Server. In this way, Mod_security module protects the OAM Server from denial of service attacks.
A flexible rule engine is at the heart of ModSecurity. It implements the ModSecurity Rule Language, a specialized programming language designed to work with HTTP transaction data. A new configuration directive uses the httpd-guardian script to monitor for Denial of Service (DoS) attacks. By default httpd-guardian defends against clients that send more than 120 requests in a minute, or more than 360 requests in five minutes.
See Also:
http://www.modsecurity.org/documentation/modsecurity-apache/2.5.12/html-multipage/configuration-directives.html#N10689To protect from a flood of HTTP Requests
Add the mod_security module to the OHS Server that front-ends the OAM Server.
In the OHS Server configuration, set the configuration directive to use the httpd-guardian script to monitor for Denial of Service (DoS) attacks.
Syntax:
SecGuardianLog |/path/to/httpd-guardian
Example:
SecGuardianLog |/usr/local/apache/bin/httpd-guardian
Use the OAM Server's diagnostic features to debug on the OAM Server side. This section includes the following topics:
Use the following methods to troubleshoot authentication issues when you have freshly installed OAM 10g Webgates in your OAM 11g deployment.
Confirm that your request was protected using an http header trace like Internet Explorer HTTP Headers or Firefox Live HTTP Headers
Confirm that the request is sent to the OAM Server for authentication
GET /oam/server/obrareq.cgi?…..
Host: oam-server:port
Use the following methods to troubleshoot logout issues when you have freshly installed OAM 10g Webgates in your OAM 11g deployment.
Make liberal use of HTTP Header Trace
Confirm that the specific logout.html was copied to /access/oamsso folder in the 10g Webgate installation directory. If not present, you must create the logout.html as described in "Configuring Centralized Logout for 10g Webgate with OAM 11g Servers".
Change the OAM 10g Webgate's httpd.conf to remove the following lines:
<LocationMatch "/oamsso/*"> Satisfy any </LocationMatch>
From the Oracle Access Manager Console, confirm that the LogoutUrls parameter (/oamsso/logout.html) is configured for this Webgate
This section includes the following topics:
OAM Server does not start up.
Locate and review the OAM Server log file on the computer hosting the OAM Server.
DOMAIN_HOME/servers/SERVER-NAME/logs/SERVER-NAME-diagnostics.log
Enable logging for this computer, as described in Chapter 22, "Logging Component Event Messages":
DOMAIN_HOME/config/fmwconfig/servers/SERVER-NAME/logging.xml
Restart the OAM Server, observe the behavior, check the log file again if needed.
Monitoring the OAM Server reveals a significant spike in latency during authentication.
Locate and review the OAM Server log file on the computer hosting the OAM Server.
DOMAIN_HOME/servers/SERVER-NAME/logs/SERVER-NAME-diagnostics.log
Enable logging for this computer, as described in Chapter 22, "Logging Component Event Messages":
DOMAIN_HOME/config/fmwconfig/servers/SERVER-NAME/logging.xml
Restart the OAM Server, observe the behavior, check the log file again if needed.
Debugging for all expression parsing and evaluation produced a significant performance drag within ~20 hours due to memory growth; running out of memory in ~50 hours.
Configuration: 2GB heap; 3 minute session timeout; jdbc connections tuned min=32 max=200; jdbc connection idle timeout disabled; jbo pool size min = 10 & max=150
To generate heap-dumps for comparison, you use the following command-line tools jmap for Sun jvm or jrcmd for jrockit jvm located under JAVA_HOME/bin.
For jrockit jvm
jrcmd pid <command> /jrockit_160_14_R27.6.5-32/bin/jrcmd 16775 heap_diagnostics /jrockit_160_14_R27.6.5-32/bin/jrcmd 16775 print_threads /jrockit_160_14_R27.6.5-32/bin/jrcmd 16775 jrarecording ....
For Sun jvm
jmap -histo <pid> jmap -dump:live,format=b,file=heap.bin <pid>
The IIS Web server on Windows supports Challenge/Response Authentication, which defaults to On when IIS is installed. This enables users to use their domain log-ins when requesting resources from IIS and can conflict with Oracle Access Manager's authentication.
For example, on the first request from an Internet Explorer (IE) browser to a resource on IIS protected by Oracle Access Manager with a basic authentication scheme, IE displays a login dialog box requesting a domain along with the user name and password login provided by Oracle Access Manager.
To disable Windows challenge/response authentication
Launch the Microsoft Management Console for IIS.
Select the Web Server Host under Internet Information Server in the left hand panel.
Right click and select Properties.
Scroll down and select Edit the Master Properties for WWW Service.
Select the Directory Security tab.
Select Edit Anonymous Access and Authentication Control.
Complete the appropriate step for your platform:
Windows 2000: Clear the Integrate Windows Authentication box.
Click OK.
In the Windows IIS properties screen, click OK.
Close the Microsoft Management Console.
An Identity Store that is designated as the System Store should not be edited to change the store type (from Embedded LDAP to OID, for instance) nor the connection URLs.
If you do need to change the Identity Store that is designated as the System Store should not be edited to change the store type, Oracle recommends that you create a new Identity Store and then edit that registration to mark it as your System Store.
The following topics are provided to assist you:
If form authentication or pass-through functionality is not working, the problem might be that either "UseWebGateExtForPassthrough
" parameter is not set to true in the Webgate profile or that webgate.dll is not configured as Wild Card Application Mapping in IIS. In such cases, Webgate does not perform authentication or authorization for HTTP "POST" requests for the resources protected by form-based authentication.
Solution: Confirm that the UseWebGateExtForPassthrough
parameter is configured in the Webgate profile with a value of true
and that webgate.dll is configured as Wild Card Application Mapping.
Following are some general guidelines to follow when installing Oracle Access Manager Webgates with IIS Web servers.
Account Privileges: The account that performs Oracle Access Manager installation must have administration privileges. The user account that is used to run OAM services must have the "Log on as a service" right, which can be set by selecting Administrative Tools, Local Policy, Local Policies, User Rights Assignments, Log on as a service.
IIS 6 Web Servers: You must run the WWW service in IIS 5.0 isolation mode. This is required by the ISAPI postgate filter. During Oracle Access Manager installation, this is usually set automatically. If it is not, you must set it manually for the Default Web site.
Webgate for IIS 7 Web Server: To use Form-based authentication without enabling pass through functionality (for example, "access/oblix/apps/webgate/bin/webgate.dll" is an action in the Form-based authentication scheme), ensure that the entry "<add segment="bin"/>" is not present in the applicationHost.config file. If the entry is present, you must remove it. Use the following steps to check this entry:
Go to Windows\System32\inetsrv\config and open the file applicationHost.config.
Search for the <hiddenSegments
> module and remove the entry <add segment="bin"/>
if it is present.
Webgate: When installing IIS Webgates, setting various permissions for the /access directory is required for IIS Webgates only when you are installing on a file system that supports NTFS. For example, suppose you install the ISAPI Webgate in Simple or Cert mode on a Windows 2000 computer running the FAT32 file system. The last installation panel provides instructions for manually setting various permissions that cannot be set on the FAT32 file system. In this case, these instructions may be ignored.
On IIS 6 Web servers only, you must run the WWW service in IIS 5.0 isolation mode, which is a requirement of the ISAPI postgate filter. This scenario will work if you have 32-bit Oracle Access Manager binaries running on a 32-bit Windows operating system. However, there is an issue if you attempt to run a 32-bit postgate.dll on a 64-bit Windows machine with IIS running in 32-bit mode.
When running IIS in IIS5.0 isolation mode, you see the following message:
"ISAPI Filter 'C:\webgate\access\oblix\apps\webgate\bin\webgate.dll' could not be loaded due to a configuration problem.
The current configuration only supports loading images built for an AMD 64-bit processor architecture. The data field contains the error number.
To learn more about this issue, including how to troubleshoot this kind of processor architecture mismatch error, see the following Web site:
http://go.microsoft.com/fwlink/?LinkId=29349
For more information, see Help and Support Center at:
http://go.microsoft.com/fwlink/events.asp
IIS5 never existed as 64-bit. However, IIS v6's IIS5 Compatibility Mode on 64-bit Windows computers only runs as 64-bit.
It is architecturally impossible run IIS5 Isolation Mode 32- bit on 64-bit Windows, as described in documentation available through the following URLs:
http://www.microsoft.com/communities/newsgroups/en-us/default.aspx?dg=microsoft.pu blic.inetserver.iis&tid=5dd07102-8896-40cc-86cb-809060fa9426&cat=en_US_ 02ceb021-bb43-476d-8f8f-6c00a363ccf5&lang=en&cr=US&p=1
http://blogs.msdn.com/david.wang/archive/2005/12/14/HOWTO-Diagnose-one-cause-of-W3 SVC-failing-to-start-with-Win32-Error-193-on-64bit-Windows.aspx
A "The page cannot be displayed" error that appears after configuring Webgate for pass-through functionality, indicates a configuration issue.
Solution: Confirm that the UseWebGateExtForPassthrough
parameter is configured in the Webgate profile with a value of true
and that webgate.dll is configured as Wild Card Application Mapping.
When Oracle Access Manager is running with Microsoft's IIS Web server, you must manually uninstall and reinstall the following ISAPI filters when reinstalling Oracle Access Manager.
tranfilter.dll
oblixlock.dll
(if you installed Webgate)
webgate.dll
(if you installed Webgate)
To remove and reinstall IIS DLLs
Uninstall Oracle Access Manager.
Manually uninstall the preceding DLLs.
Reinstall Oracle Access Manager.Active Directory.
Manually reinstall the DLLs.
Note:
These filters can change depending on the version of IIS you are using. If these filters do not exist or there are others present, contact Oracle to determine if the filters that are present need to be removed.A jps logger class instantiation warning is might appear on the back end upon authentication. However, this is a harmless warning and no action is required.
This section provides the following topics:
The automatically generated Description for each component of the Policy Configuration tab is not translated. This is expected and enables Administrators to change the Description to whatever they require. Following such a change, translation by Oracle is not possible.
When the browser locale is not supported, the Oracle Access Manager Console Login page shows as server locale. It should fall back to English. This is the expected behavior:
If the client Locale is not supported, Oracle Access Manager falls back to the server locale.
If the server locale is not supported, Oracle Access Manager falls back to English.
When users select an unsupported language and come to the Oracle Access Manager SSO page, it shows as server locale (German, for example). However, after logging in, all the pages are displayed as English.
Disable the Oracle Access Manager SSO page and the original Oracle Access Manager login page also falls back to English.
The Oracle Access Manager Console displays policies and resources oddly when the input Fusion Applications configuration file for remote registration is not in UTF-8 format or when the OAM Server is not started in UTF-8 locale (en_US.utf8, for instance).
Be sure to use UTF-8 encoding if creating a Fusion Applications configuration file for the remote registration tool, oamreg, to generate authentication policies and protected resources. Also, be sure to start OAM Server in UTF-8 locale machines. Otherwise, the Oracle Access Manager Console might display policies and resources oddly following successful inband registration.
After installing OAM and protecting a page using a physical host and port, register the partner application using the OHS physical host and port. Login fails to prompt the user for credentials when accessing the protected page. The log file shows that the URL is re-directed to a Virtual Host despite the fact that all configuration and registration is setup correctly.
Remove any Virtual Host Directives from httpd.conf when protecting a page using the Oracle HTTP Server (OHS) physical host and port.
After using the WebLogic Configuration Wizard to create an OAM Server cluster on two computers, and starting AdminServer, all servers start up properly. After shut down, a third server is added using the WebLogic Server Administration Console to create a new managed server and add it to the cluster. The third server goes into Running mode when started, with some exceptions in the start up log.
... Exception in thread "OAM Metric Persistence Timer"
in addition to the actions in the WebLogic Administration Console, you must register the server using the Oracle Access Manager Console to ensure that the server can identify itself.
Note:
When adding and registering a second server instance for the same computer, all port numbers must differ: OAM Proxy port; the "port" that must match the one in the WebLogic Server Console; and the Coherence port.For server registration details, see "Managing Individual OAM Server Registrations".
In the event of a partial outage of Oracle Access Manager (on some, but not all instances of the cluster), end users might see intermittent login and logout failures.
Remove OHS from the deployment
Configure the OHS cluster such that each OHS instance is pinned to a WebLogic Server instance.
The WebLogic Server container with the malfunctioning Oracle Access Manager application must be removed from service (shutdown) and brought back up upon recovery.
Problem: Remote Registration Tool Failure
Ensure that the agent name is unique (does not already exist) and that the AdminServer is running.
Problem: No ObAccessClient.xml File Generated
Protected and public resources must be described as relative URLs of the format '/index.html'. If the resource does not begin with a '/', no ObAccessClient.xml file will be generated.Verify the protected and public resource URLs and ensure all begin with a "/". For more information, see "About the Resource URL".
Problem: Partner Registration Failure
Partner registration can fail if you do not supply a unique agent name, which is also used to create an application domain. The agent name and application domain name must be the same and must be unique. Using the oamreg validate command can fail when the agent name does not match the application domain name.
Ensure that the agent name and application domain name are the same.
While browsing across the Resources table in the Resource Type tab the following error message is logged:
@ <Error> <oracle.adfinternal.view.faces.model.binding.CurrencyRowKeySet> @ <BEA-000000> <ADFv: Rowkey does not have any primary key attributes. Rowkey: oracle.jbo.Key[], table: model.ResTypeVOImpl@620289.>
This is harmless and does not hinder any functionality.
Delivered with Oracle Enterprise Linux, SELinux modifications provide a variety of policies through the use of Linux Security Modules (LSM) within the Linux kernel.
SELinux requires performing additional steps after installing Oracle Access Manager Webgates and before starting the associated Web server.
The following errors could be reported in logs/console when starting a Web server on Linux distributions that have more strict SELinux policies in place (after installing an Oracle Access Manager Webgate):
OAM 11g Webgate
$Webgate_OH/webgate/ohs/lib/webgate.so: cannot restore segment prot after reloc: Permission denied.
OAM 10g Webgate
$Webgate_install_dir/access/oblix/apps/webgate/bin/webgate.so: cannot restore segment prot after reloc: Permission denied.
These errors are reported due to Secure Linux security context policies on files.
To avoid these errors and start the Web server, run following chcon
commands to change the security context on files after installing each Oracle Access Manager Web component and before restarting the associated Web server. For more information on the chcon
command, see your Linux documentation.
Run chcon -t texrel_shlib_t PATH_TO_LIBWEBPLUGINS.SO. For example:
chcon -t texrel_shlib_t /Webgate_install_dir/access/oblix/lib/webgate.so ... and libxmlengine.so
Run chcon -t texrel_shlib_t PATH_TO_LIBWEBGATE.SO. For example:
chcon -t texrel_shlib_t /Webgate_install_dir/access/oblix/apps/webgate/ bin/webgate.so
This section provides the following details:
Session impersonation is not enabled by default. You can update the value in oam-config.xml and update the version of oam-config.xml to automatically propagate the ImpersonationConfig status to all managed servers without a restart.
To enable Session Impersonation
Back up DOMAIN_HOME/config/fmwconfig/oam-config.xml.
Set ImpersonationConfig
to true
:
<Setting Name="ImpersonationConfig" Type="htf:map"> <Setting Name="EnableImpersonation" Type="xsd:boolean">false</Setting> </Setting>
Configuration Version: Increment the Version xsd:integer
as shown in the next to last line of this example (existing value (25, here) + 1):
Example:
<Setting Name="Version" Type="xsd:integer">
<Setting xmlns="http://www.w3.org/2001/XMLSchema"
Name="NGAMConfiguration" Type="htf:map:>
<Setting Name="ProductRelease" Type="xsd:string">11.1.1.3</Setting>
<Setting Name="Version" Type="xsd:integer">25</Setting>
</Setting>
Save oam-config.xml.
When Oracle Access Manager 11g is integrated with Oracle Identity Federation, and you use the Oracle Access Manager Session Management function to clear the session, only the Oracle Access Manager session is cleared (not the Oracle Identity Federation session).
If both the SSL and Open ports of the Managed Server are enabled, then the Managed Server is set to the SSL port by default.
If you must use the non-ssl port, the credential collector URL the authentication scheme must be set to the absolute URL which points to 'http' as the protocol and non-ssl port.
Problem: AdminServer Startup (or Remote Registration Tool Failure) on AIX Platforms
AdminServer start up fails with following message:
"java.net.SocketException: No buffer space available".
Configration for the number of AIX file descriptors set for the operating sytem is substantially high (ulimit) resulting in a buffer overflow that causes remote registration failure with the following message:
The ulimit value is application dependent and applies exclusively to application program data and the application stack. The default number of open files setting (2000) is typically sufficient for most applications. If the value is too low, errors might occur when opening files or establishing connections. Because this value limits the number of file descriptors that a server process might open, a value that is too low prevents optimum performance. For the AIX operating system, the default setting is 2000.
Increasing the ulimit file descriptor limits might improve performance. Increasing some of the other limits might be needed depending on your application.
Log in as root.
Perform the following steps to change the open file limit to 10,000 files:
Open the command window.
Locate and edit /etc/security/limits file to add the following lines to the user account on which the AdminServer process runs:
nofiles = 10000 nofiles_hard = 10000
Save the file and restart AIX.
In a command window, decrease the TCP_TIMEWAIT interval with the following command to set the state to 15 seconds (which allows TCP to release closed connections faster and increases the number of available resources for open connections).
/usr/sbin/no –o tcp_timewait =1
Tune the following parameters to 256k, as shown:
no -a |grep space tcp_recvspace = 262144 tcp_sendspace = 262144 udp_recvspace = 262144 udp_sendspace = 262144
Tune the following parameters as indicated here:
no -o rfc1323=1 no -o sb_max=4194304
Problem: Connection to OAM Server could not be established: Exception in connecting to server. Connection refused.
This is normal and expected behavior for the Managed Server where the OAM Server runs because the IAMSuiteAgent agent is started before the OAM Server.
The IAMSuiteAgent is deployed on every WebLogic container. When the WebLogic container starts, the agent tries to connect to the OAM Server. If it fails to connect, this message is logged and the agent tries to establish the connection in subsequent requests. When the agent is successful, this message is no longer displayed.
If the connection to the OAM Server is not successful, the IAMSuiteAgent falls back and the WebLogic container handles protection (including login), if it is configured.
The state of a session is the source of truth for relying parties. Synchronization of system clocks of the various Servers is required.
The system clock of the relying party might be out of synchronization with the SME clock. If the relying party's clock is:
Ahead of the session clock A relying party's request for authentication is made and the active sessionID is returned.
Behind the session clock: Event notifications to the relying party help invalidate the session.
For example, if a Web server clock is ahead of the server clock, a request sent from the Webgate to the OAM Server will contain a time that, to the OAM Server, has not yet occurred. This can cause login events to fail. When running in Simple or Cert mode, time stamps might become out of sync, or the client certificate might appear to be invalid.
Note:
To avoid event notification issues, ensure that all OAM Server clocks are synchronized to Time Services such as NIST internet time service.For successful operation:
Ensure all computer clocks are synchronized. There is no tolerance level. If, for example, the Webgate clock is even slightly ahead of the OAM Server clock, a cookie generated by the Webgate will appear to be in the future and can cause problems in the OAM Server.
Confirm that the clock on each computer running a Webgate is not running ahead of the OAM Servers with which it is associated. The OAM Server must be ahead of the Webgate clock by a maximum of 60 seconds.
Oracle Access Manager 11g uses Oracle Coherence to replicate session states within a distributed installation. Coherence is used to communicate state changes between the Oracle Access Manager Console and OAM Servers.
Consider the following 2 distributed deployment topologies. Coherence relies on User Datagram Protocol (UDP) for cluster discovery and heartbeat. If a firewall exists between certain components of OAM 11g, then the corresponding UDP ports used by Coherence must be open. Otherwise, OAM 11g might not work correctly.
For example, the UDP ports used by Coherence must be opened as follows:
The Oracle Access Manager Console is deployed within the intranet, and OAM Servers are deployed in the DMZ. In this case, the UDP ports used by Coherence must be opened on the firewall between the DMZ and the intranet.
The Oracle Access Manager Console and OAM Servers are deployed in different security zones of the DMZ, with firewalls between any two adjacent zones. In this case, the UDP ports used by Coherence must be opened on the firewall between the adjacent security zones, where one or more instances of Oracle Access Manager Console and OAM Servers run.
Oracle Access Manager 11g uses Oracle Coherence to provide a distributed cache with low-data access latencies and to transparently move data between distributed caches (and into the session store). Session data is redundant across these tiers. For example, when a session is created, it then exists within the local cache on the server that created it, the distributed cache, and (if enabled) within the session store database as well. For more information, see "Oracle Coherence and Session Management".
WARNING:
Oracle recommends that you do not modify Oracle Coherence settings unless requested to do so by an Oracle Support Representative.
Whether you are viewing Oracle Coherence settings for an individual server instance or Oracle Coherence details that are common to all OAM Servers, Oracle recommends that you do not modify Oracle Coherence settings unless requested to do so by an Oracle Support Representative.
Oracle Coherence logging appears in the WebLogic Server log only. There is no bridge from Oracle Coherence logging to Oracle Access Manager logging.
See Also:
Oracle Coherence documentation.Problem: Resource not added to Authentication or Authorization Policy
While creating an Authentication or Authorization Policy, if you add a resource that is already used in another Authentication or Authorization Policy, a validation error appears when you click Apply. This is expected.
If you click OK in the error window and then attempt to add a valid resource that is not used within another Authentication or Authorization Policy, the resource is not added and the Authentication or Authorization Policy is not created.
Click Apply and close the Authentication or Authorization Policy page.
From the navigation tree, click the named policy again, click the Edit to open the page, and add the new resource.
Problem: Validation Failure - "description" attribute is not valid
A validation error appears if you enter an optional description longer than 200 characters.
Keep optional descriptions to 200 characters in length and less than 10 lines.
The following issues with Web servers may arise:
Symptom: You are running an Apache Web server, and an OAM Server fails, displaying the following message:
libthread panic: cannot create new lwp (PID: 9035 LWP 2). stackrace: ff3424cc 0
This symptom may be caused by the Apache Web server launching more instances of itself. This can happen when the server determines that more instances are needed to service the number of connections between one or more Webgates and the OAM Server.
The additional instances create even more connections, which exceed the number of connections by the OAM Server.
Solution: Reduce the number of MinSpareServers
, MaxSpareServers
, StartServers
, and MaxClients
parameters.
Go to the OAM Server's configuration directory and open the http.d
configuration file.
Recommended parameter settings:
MinSpareServers
1
MaxSpareServers
5
StartServers
3
MaxClients
5
When running Apache v2 on HP-UX, do not use nobody
for User or Group, because shared memory may not work. Instead, use your login name as User Name with a your group as Group Name On HP-UX (on Solaris, "www" is equivalent to "nobody").
When running Apache v2 on HPUX 11.11, ensure that the AcceptMutex
directive in the Apache httpd.conf file is set to "fcntl
". If the directive is not present, add it to the httpd.conf file (AcceptMutex fcntl
). For more information, see:
After installing a Webgate on vendor-bundled Apache, the Web server may give the following error upon startup:
Error: Cannot load libgcc_s.so.1 library - Permission denied.
Solution: Change the Security-Enhanced Linux (SELinux) policy rules for Oracle Access Manager Webgates as described in "Tuning Apache/IHS v2 for Oracle Access Manager Webgates".
Errors might be reported in WebServer logs/console when starting a Web server on Linux distributions, which have stricter SELinux policies in place, after installing an Oracle Access Manager Web component. You can avoid these errors by running appropriate chcon
commands for the installed Web component before restarting the Web server.
See Also:
"SELinux Issues"The following item is required only if you compile Apache v2 for Webgate on UNIX with the mpm_worker_module. In this case, you need to modify the thread.c file from the Apache source for the UNIX environment. Making this change ensures that the default pthread stacksize for Webgate produces optimal performance during multithreaded server implementation. If this change is not made, the default pthread stack size would not be sufficient for Webgate and could result in a crash.
Apache 2.0 does not support the ThreadStackSize option. Therefore:
With UNIX-based Apache v2.1 and later you must use the ThreadStackSize directive to set the size of the stack (for autodata) of threads that handle client connections and call modules to help process those connections.
With UNIX-based Apache 2, it is best to use the compilable source while adding the mpm_worker_module and changing the thread.c file to avoid a stack overflow.
The following procedure shows how to modify the Apache v2.0 thread.c file to provide the default pthread stacksize needed by Webgate for optimal performance during multi-threaded server implementation. For details about the Apache v2.1+ ThreadStackSize directive, see http://httpd.apache.org/docs/2.2/mod/mpm_common.html#threadstacksize.
Note:
The following procedure should be performed only for the Apache 2.0 Webgate. Otherwise, the default pthread stack size is not sufficient for the Webgate and could result in a crash.To modify the Apache v2.0 thread.c file for Webgate in a UNIX environment
Locate the thread.c file. For example:
APACHE 2.0.52 source/srclib/apr/threadproc/unix/thread.c
Locate the function named apr_threadattr_create(apr_threadattr_t **new,apr_pool_t *pool) in the following code segment:
**new,apr_pool_t *pool) in the following code segment: 1-----> apr_status_t stat; 2 3-----> (*new) = (apr_threadattr_t *)apr_pcalloc(pool, sizeof(apr_threadattr_t)); 4-----> (*new)->attr = (pthread_attr_t *)apr_pcalloc(pool, sizeof(pthread_attr_t)); 5 6-----> if ((*new) == NULL || (*new)->attr == NULL) { 7-----> return APR_ENOMEM; 8-----> } 9 10----->(*new)->pool = pool; 11----->stat = pthread_attr_init((*new)->attr); 12 13-----> if (stat == 0) { 14-----> return APR_SUCCESS; 15-----> } 16----->#ifdef PTHREAD_SETS_ERRNO 17----->stat = errno; 18----->#endif 19 20----->return stat; 21
Add the following code before line 13 shown earlier.
int stacksize = 1 << 20; pthread_attr_setstacksize(&(*new)->attr, stacksize);
Run configure, make, and make install to set up the Apache Web server with the mpm_worker_module.
Failure Authentication Event: For Domino Web servers, the redirection of a URL through Oracle Access Manager may not work if the authentication type is set as Basic Over LDAP and the URL to be redirected is mentioned as one of the following:
To overcome a failure authentication event, you must set the redirected URL with a computer name that is not defined under the host identifier group. For example, the IP address of the computer.
This problem does not occur with a form-based authentication type.
Header Variables: It may not be possible to pass header variables other than REMOTE_USER to Webgates installed on Lotus Notes Domino Web servers when using Client Certificate authentication scheme.
For example, header variables cannot be set on the one request where Client Certificate authentication occurs. However, all other requests do allow header variables to be set.
For more information, see Chapter 31, "Configuring Lotus Domino Web Servers for 10g Webgates".
Symptom: If you installed Oracle Access Manager on UNIX under a different user ID than you used to create your Web server instance, Oracle Access Manager can become unstable. Users may experience behavior such as:
Solution: Change file permissions using the chown command. Change the Oracle Access Manager directory to the same user ID that you used to create your Web server instance.
Webgate uses ISAPI extension for displaying user deny error message and for displaying the diagnostic page. However, ISA 2006 does not support extensions. Therefore:
If the user is denied access by Webgate, the user gets Page Cannot be displayed error message instead of Oracle Access Manager denied access error message.
The following diagnostic URL does not work for ISA: http(s)://hostname:port/access/oblix/apps/webgate/bin/webgate.dll?progid=1 for webgate .
After installing a Webgate instance on an Oracle HTTP Server, the server does not start up. This occurs because Oracle Access Manager uses an older Linux threading model.
Note:
When running Oracle Access Manager, LinuxThreads is used by default. This requires setting the environment variable LD_ASSUME_KERNEL to 2.4.19. If you are using NPTL with Oracle Access Manager, you do not set LD_ASSUME_KERNEL to 2.4.19.9Solution: When using LinuxThreads mode, comment out the Perl module in the httpd.conf file, update the LD_ASSUME_KERNEL environment variable, and restart, as described in the following procedure.
To resolve the failure to start Oracle HTTP Server in LinuxThreads mode
Comment out the Perl module in the httpd.conf file in the following location:
Oracle HTTP Server 11g: ORACLE_INSTANCE/config/OHS
/<ohs_name>/
httpd.conf
Oracle HTTP Server v2: OH$
/ohs/conf/httpd.conf
Oracle HTTP Server v1.3: OH$
/Apache/Apache/conf/httpd.conf
To update the LD_ASSUME_KERNEL value, open the following file in a text editor:
OH$/opmn/conf/opm.xml
Find the following line:
<process-type id="HTTP_Server" module-id="OHS">
Add the following information under the line you found in the previous step:
<environment> <variable id="LD_ASSUME_KERNEL" value="2.4.19" /> </environment>
Save this file.
Run the following commands to implement your changes:
opmnctl stopall opmnctl startall
This situation might arise whether you are using Oracle Access Manager with LinuxThreads or NPTL.
Symptom: Webgate fails to initialize when installed on an Oracle HTTP Server running Red Hat Enterprise Server version 4.0 with a kernel version lower than 2.6.9-34.EL. Version 2.6.9-34.EL is supplied with the Red Hat version 4, update 3.
Solution: To prevent this problem, you must upgrade to Red Hat version 4, update 3 or higher.
With Oracle Application Server 10.1.x, OC4J, when the httpd.conf file is modified automatically during Webgate installation, it can be corrupted.
Before installing Webgate, run the following command to prevent the httpd.conf file from being overwritten.
$ORACLE_HOME/dcm/bin/dcmctl updateConfig -ct ohs
On IIS 6 Web servers only, you must run the WWW service in IIS 5.0 isolation mode, which is a requirement of the ISAPI postgate filter. This scenario will work if you have 32-bit Oracle Access Manager binaries running on a 32-bit Windows operating system. However, there is an issue if you attempt to run a 32-bit postgate.dll on a 64-bit Windows machine with IIS running in 32-bit mode.
When running IIS in IIS5.0 isolation mode, you see the following message:
"ISAPI Filter 'C:\webgate\access\oblix\apps\webgate\bin\webgate.dll' could not be loaded due to a configuration problem.
The current configuration only supports loading images built for an AMD 64-bit processor architecture. The data field contains the error number.
To learn more about this issue, including how to troubleshoot this kind of processor architecture mismatch error, see the following Web site:
http://go.microsoft.com/fwlink/?LinkId=29349
For more information, see Help and Support Center at:
http://go.microsoft.com/fwlink/events.asp
IIS5 never existed as 64-bit. However, IIS v6's IIS5 Compatibility Mode on 64-bit Windows computers only runs as 64-bit.
It is architecturally impossible run IIS5 Isolation Mode 32- bit on 64-bit Windows, as described in documentation available through the following URLs:
http://www.microsoft.com/communities/newsgroups/en-us/default.aspx?dg=microsoft.pu blic.inetserver.iis&tid=5dd07102-8896-40cc-86cb-809060fa9426&cat=en_US_ 02ceb021-bb43-476d-8f8f-6c00a363ccf5&lang=en&cr=US&p=1
http://blogs.msdn.com/david.wang/archive/2005/12/14/HOWTO-Diagnose-one-cause-of-W3 SVC-failing-to-start-with-Win32-Error-193-on-64bit-Windows.aspx
Symptom: When attempting to start the Sun Web server, you get an error like the following:
Unable to start, PCLOSE
Solution: A number of problems can cause this error:
A syntax error in your obj.conf
file
Leading spaces in your obj.conf
file
Installing Oracle Access Manager as a different user ID than what you used to create your Web server instance
A carriage return at the end of the obj.conf
file
When Oracle Access Manager is running with Microsoft's IIS Web server, you must manually uninstall and reinstall the following ISAPI filters when reinstalling Oracle Access Manager.
tranfilter.dll
oblixlock.dll
(if you installed Webgate)
webgate.dll
(if you installed Webgate)
To remove and reinstall IIS DLLs
Uninstall Oracle Access Manager.
Manually uninstall the preceding DLLs.
Reinstall Oracle Access Manager.Active Directory.
Manually reinstall the DLLs.
Note:
These filters can change depending on the version of IIS you are using. If these filters do not exist or there are others present, contact Oracle to determine if the filters that are present need to be removed.After setting up Windows Native Authentication, and accessing the WNA-protected page, the browser might give an error indicating that the user name and/or password are incorrect.
The Identity Store used by Oracle Access Manager might not point to Windows Active Directory. By default, the identity store is Embedded LDAP.
In the Oracle Access Manager Console, review the identity store configuration: System Configuration, Data Sources, User Identity Store.
Confirm the LDAP store settings point to Active Directory.