12 Monitoring Performance and Health

Monitoring performance refers to observing (viewing) performance metrics to make yourself aware of the state specific components. Monitoring health allows perimeter devices to check the health of an Access Manager server instance by hitting the heartbeat URL of the Managed Server. This chapter provides information on monitoring Oracle Access Management performance and Access Manager health. It contains the following sections.

Introduction to Performance Monitoring
Reviewing DMS Metric Tables
Monitoring Server Metrics
Monitoring SSO Agent Metrics
Introduction to OAM Proxy Metrics and Tuning
Reviewing OpenSSO Metrics in the DMS Console
Monitoring the Health of an Access Manager Server

12.1 Introduction to Performance Monitoring

Component performance metrics can be collected in memory during the completion of particular events. These metrics are kept only in memory so there are several mechanisms to extract and display them: EM, dmsSpy, and dmsDump, for example.

Note:

dmsSpy is a WebLogic Application Server (WAS) tool that displays raw DMS data specific to the WAS instance. Information is categorized by Noun Types (OAMS.OAM_ prefix for Oracle Access Management) and includes metrics pertaining to all DMS instrumented applications running in the WAS instance. To see the metrics on a WebLogic instance, go to http://hostname:port/dms/. For example:

http://samplehost:7001/dms/

Oracle Fusion Middleware Performance and Tuning Guide for details about instrumenting applications with Oracle Dynamic Monitoring Systems (DMS).

The metrics can be used to monitor the time spent in a particular area or track particular occurrences or state changes. Oracle Access Management uses the Oracle Dynamic Monitoring Systems (DMS) to measure application-specific performance information for OAM Servers and registered Agents. Administrators can monitor performance for Access Manager using the Monitoring command on the Actions menu under the System Configuration tab.

12.2 Reviewing DMS Metric Tables

Use this procedure to access the DMS console.

To access DMS console

In a browser window, go to the DMS Console using the following URL:
```
http:// <example_AdminServer:Port/dms/
```
Log in with your Oracle Access Management Administrator credentials.
In the DMS Metric Tables, click the desired metric from those listed to view the results on the right-side of the console.

12.3 Monitoring Server Metrics

This section provides the following topics:

Monitoring Server Instance Performance
Reviewing Server Metrics Using Oracle Access Management Console

12.3.1 Monitoring Server Instance Performance

Users with valid Oracle Access Management Administrator credentials can use the following procedure to display various performance metrics using the Oracle Access Management Console.

Prerequisites

The OAM Server must be running.

To monitor performance using Oracle Access Management Console

From the Oracle Access Management Console, click Server Instances and the desired server instance.
Server Instance:
1. From the Actions menu in the navigation tree, click Monitor Menu.
2. On the Monitor page, click the desired subtab to view results for the server instance:
  
  Server Processes Overview
  Session Operations
  Server Operations
  OAM Agents
3. Proceed to "Reviewing Server Metrics Using Oracle Access Management Console".
See also, "Introduction to OAM Proxy Metrics and Tuning".

12.3.2 Reviewing Server Metrics Using Oracle Access Management Console

This topic provides a look at the Server metrics available when you have a server instance selected in the navigation tree and you choose the Monitoring Menu command on the Actions menu under the System Configuration tab. Figure 12-1 shows the Server Processes page.

Figure 12-1 Server Processes Overview Page

Description of "Figure 12-1 Server Processes Overview Page"

Server Processes Overview provides the following OAM Server events, organized in individual columns on the tab.

Table 12-1 OAM Server Metrics: Server Processes Overview Tab

Server Metric Columns
Authorization Process
Authorization Requests
Authentication Process Failure
Authentication Process Success
Pre Authentication Process Failure
Pre Authentication Process Success

Figure 12-2 shows the Session Operations Monitoring tab after detaching the table to display all event metrics in individual columns.

Figure 12-2 OAM Server Metrics: Session Operations Monitoring Page

Description of "Figure 12-2 OAM Server Metrics: Session Operations Monitoring Page"

OAM Server Session Operations metrics include:

Table 12-2 OAM Server Metrics: Session Operations

Session Operations
Check Session Valid
Check Session Valid Failure
Check Session Valid Success
Create Session
Create Session Failure
Create Session Success
Destroy Session
Destroy Session Failure
Destroy Session Success
Delete Client Session
Delete Client Session Failure

Figure 12-3 shows the detached OAM Server Operations Monitoring page.

Figure 12-3 OAM Server Metrics: Server Operations Tab

Description of "Figure 12-3 OAM Server Metrics: Server Operations Tab"

OAM Server Operations metrics include those in Table 12-3.

Table 12-3 OAM Server Metrics: Server Operations Tab

OAM Server: Operations Metrics
Authentication Policy Response Failure
Authentication Policy Response Success
Authentication Scheme Response Failure
Authentication Scheme Response Success
Authentication Failure
Authentication Failure Responses
Authentication Policy Response
Authentication Requests
Authentication Scheme Response
Autorization Failure
Autorization Failure
Autorization Process Failure
Autorization Process Success

Figure 12-4 shows the OAM Server Metrics: OAM Agents tab with all available metrics showing.

Figure 12-4 OAM Server Metrics: OAM Agents Tab

Description of "Figure 12-4 OAM Server Metrics: OAM Agents Tab"

OAM Agent performance metrics include:

Agent Name
Agent Status
Version

12.4 Monitoring SSO Agent Metrics

This section describes how to review metrics for various components and how to determine whether tuning is needed. The following topics are included:

Monitoring Agent Metrics Using Oracle Access Management Console
Reviewing OAM Agent Metrics
Reviewing OSSO Agent Metrics

12.4.1 Monitoring Agent Metrics Using Oracle Access Management Console

Users with valid Oracle Access Management Administrator credentials can use the following procedure to display various SSO Agent performance metrics using the Oracle Access Management Console.

Prerequisites

The server and agent must be running.

To monitor SSO Agent performance using Oracle Access Management Console

From the Oracle Access Management Console, click SSO Agents.
Open the desired agent type node:
- OAM Agents
- OSSO Agents
- OpenSSO Agents: There is no way to monitor this Agent other than OpenSSO Proxy behavior with respect to Agent Requests. See "Reviewing OpenSSO Metrics Using the DMS Console".
Search for the desired agent to monitor, as usual.
In the Search Results table, highlight the desired agent SerialNumber and from the Actions menu select Monitor.
Proceed as needed.
- Reviewing OAM Agent Metrics
- Reviewing OSSO Agent Metrics

12.4.2 Reviewing OAM Agent Metrics

OAM Agent metrics are organized across the following tabs, as shown in Table 12-3:

Connectivity
Operations Overview
Operations Detail
Information

12.4.3 Reviewing OSSO Agent Metrics

When you have an OSSO Agent selected OSSO Agents Search Results table and choose Monitor from the table's Actions menu, the following metrics pages are available:

Figure 12-10, "OSSO Agent Monitoring Page with Operation Details"
Figure 12-11, "OSSO Agent Monitoring Process Overview Table"
Figure 12-12, "OSSO Agent Information Table"

Figure 12-10 OSSO Agent Monitoring Page with Operation Details

Description of "Figure 12-10 OSSO Agent Monitoring Page with Operation Details"

Figure 12-11 illustrates the detached OSSO 10g Agent Monitoring Process Overview table.

Figure 12-11 OSSO Agent Monitoring Process Overview Table

Description of "Figure 12-11 OSSO Agent Monitoring Process Overview Table "

Figure 12-12 illustrates the detached OSSO Agent Information table.

Figure 12-12 OSSO Agent Information Table

Description of "Figure 12-12 OSSO Agent Information Table "

12.5 Introduction to OAM Proxy Metrics and Tuning

This section provides the following topics:

About OAM Proxy Metrics
OAM Proxy Server Tuning Parameters

12.5.1 About OAM Proxy Metrics

Throughput refers to the number of requests processed per second. Latency refers to the time required to process a particular request. There is less than a 20% latency increase with the introduction of a proxy between Webgate and OAM Server.

Table 12-4 lists the various OAM Proxy metrics available.

Table 12-4 OAM Proxy Metrics

Metric	Description
handshakes.active	Number of active threads doing handshake
handshakes.avg	Average time spent performing initial handshake
handshakes.completed	Number of times an initial handshake has been executed
handshakes.maxTime	Maximum time spent performing initial handshake
handshakes.minTime	Minimum time spent performing initial handshake
handshakes.time	Total time spent performing initial handshake
failedHandshakes.count	Count of failed handshakes
peerCompatibilityFailures.count	Count of how many Peer Compatibility Check Failures have happened
openSecurityMode.count	Count of how many Open Security Mode handshakes have happened
simpleSecurityMode.count	Count of how many Simple Security mode handshakes have happened
SSLSecurityMode.count	Count of how many SSL Security Mode handshakes have happened
negotiateSecurityMode.active	Number of active threads doing security mode negotiation

12.5.2 OAM Proxy Server Tuning Parameters

Performance of the OAM Proxy can be tuned by changing its configuration through the Java EE container Administration Console.

Note:

Both the Java EE container Administrator and the Oracle Access Management Administrator can tune performance using the Java EE container Administration Console, which is outside the scope of this book.

Table 12-5 provides the tuning parameters for the OAM Proxy.

Table 12-5 OAM Proxy Tuning Parameters

Purpose	Parameter	Type	Value	Description
Denial of Service Attacks	ConnectionValidationInterval	Integer	120	The time interval in seconds for validating the connections periodically for denial of service attacks
	BacklogQueue	Integer	50	Maximum length of backlog queue
	MaxNAPHandShakeTime	Integer	100	The maximum time in milliseconds within which the client should complete the NAP handshake with client. If NAP handshake over a connection is not completed within this time, the connection will be marked as malicious

12.6 Reviewing OpenSSO Metrics in the DMS Console

This section provides the following topics:

OpenSSO Proxy Events and Metrics: Server
OpenSSO Proxy Metrics: Agent
Reviewing OpenSSO Metrics Using the DMS Console

12.6.1 OpenSSO Proxy Events and Metrics: Server

Throughput refers to the number of requests processed per second. Latency refers to the time required to process a particular request. The Events that can be monitored are described in Table 12-6.

Table 12-6 OpenSSO Proxy Server Events

Event	Description
Naming Service Request	This request is for naming lookups. One can monitor response time taken by the OpenSSO Proxy in servicing this request
Agent Authentication Process	Agent Authentication has been captured in two phases: AgentAuthentication_Login and AgentAuthentication_SubmitRequirements phase. The second phase refers to the phase after the credentials are submitted by the OpenSSO Agent for authentication The second phase refers to the phase after the credentials are submitted by the OpenSSO Agent for authentication.
Agent Session Validation	Agent Session Validation
User Authentication	This event is captured for Client SDK's only. One can monitor response time taken to authenticate client SDK's through this diagnostic event
User Session Validation	Time taken to validate User Session
User Authorization	Time taken for authorization as per the configured policy for the given resource

Table 12-7 lists the various OpenSSO Proxy metrics available for the named server.

Table 12-7 OpenSSO Proxy Metrics: Server

Metric	Description
AgentAuthentication_Login	Response time details for Authentication requests during login phase sent by the Agent to authenticate
AgentAuthentication_LoginFailures	Count of how many Agent Authentication requests during login phase have failed.
AgentAuthentication_SubmitRequirements	Response time details for Authentication requests during Submit Requirements phase send by the Agent to authenticate
AgentAuthentication_SubmitRequirementsFailures	Count of how many Agent Authentication requests during Submit Requirements phase have failed
NamingServiceRequest	Response time details for Naming Service Request operations
NamingServiceRequestFailures	Count of how many Naming Service Request operations have failed
UserAuthentication_SDK	Response time details for User Authentication requests
UserAuthentication_SDKFailures	Count of how many User authentication Requests have failed
UserAuthorization	Response time details for User Authorization operations
UserAuthorizationFailures	Count of how many user authorization operations have failed
ValidateAgentSession	Response time details for Agent Session Validation operation
ValidateAgentSessionFailures	Count of how many agent session validation operations have failed
ValidateUserSession	Response time details for User Session Validation operation
ValidateUserSessionFailures	Count of how many User session validation operations have failed.

12.6.2 OpenSSO Proxy Metrics: Agent

Table 2 lists the various OpenSSO Proxy metrics available for each OpenSSO Agent.

Table 12-8 OpenSSO Proxy Metrics: Agent

Metric	Description
AgentAuthentication_SubmitRequirements	Response time details for Authentication requests during Submit Requirements phase collected per Agent
AgentCacheMode	Specifies the cache mode for the client policy evaluator. Values can be: subtree or self
AgentFilterMode	Specifies how the agent filters requests to protected web applications. The global value functions as a default, and applies for protected applications that do not have their own filter settings
AgentHostName	The host name of OpenSSO Agent
AgentIPAddress	The IP Address of OpenSSO Agent
AgentMappingMode	Specifies the mechanism used to determine the user ID
AgentState	The state of OpenSSO Agent: enabled or disabled.
UserAttributeName	Specifies the data store attribute that contains the user ID
UserAuthorization	Response time details for User Authorization operations collected per Agent
UserIdentity	Specifies the session property name for the authenticated user's ID. Default is 'UserToken'
ValidateAgentSession	Response time details for Agent Session Validation operation collected per Agent
agentType	The type of OpenSSO agent: J2EE or Web Agent

12.6.3 Reviewing OpenSSO Metrics Using the DMS Console

User with valid Oracle Access Management Administrator credentials can use the procedure here to view OpenSSO Proxy metrics in the DMS console.

Prerequisites

The OAM Server must be running.

To access DMS console

In a brower window, go to the DMS Console using the following URL:
```
http:// <example_AdminServer:Port/dms/Spy
```
Log in with your Oracle Access Management Administrator credentials.
OpenSSO Agent Metrics: In the DMS Metric Tables, click OAMS.OAM_Server.OPENSSO_Agents.
OpenSSO Proxy Metrics: In the DMS Metric Tables, click OAMS.OAM_OpenSSOProxy and view the results on the right side of the console.

12.7 Monitoring the Health of an Access Manager Server

Access Manager Services are business critical and must always be available to control user access to an organization's protected web services and applications. Because hardware, network connectivity issues and other failures can happen, HeartBeat monitoring can be leveraged by Load Balancers to ensure user traffic is routed to healthy OAM Servers. For example, when there is a firewall installed between a User Agent or WebGate (10 or 11g) and the 10g or 11g Access Manager server, perimeter devices can check availability of the Access Manager server (its health) by hitting its HeartBeat URL. The following sections contain details.

Understanding WebGate and Access Manager Communications
Monitoring Access Manager Server Health

12.7.1 Understanding WebGate and Access Manager Communications

When deploying a network firewall between a WebGate and Access Manager server, the WebGate communicates using the OAP protocol by creating a TCP socket connection with Access Manager to establish a message channel. The WebGate uses the message channel to send different OAP messages necessary to serve the resource requests (isprotected, isauthorized, and the like). Now, consider a situation in which the WebGate/Oracle HTTP Server is idle. In this case, the WebGate has received no resource request and will not send any messages to Access Manager for authentication or authorization; there will also not be any read/write activity on the socket connection.

The firewall determines this connection is idle after 30-40 minutes of inactivity (depending on its configuration) and terminates the socket connection but does not inform/notify the WebGate or Access Manager server. In this case, when a request for a resource arrives at the WebGate and it sends a OAP message to the Access Manager server, it uses the existing connection and waits for a reply. Because the connection was dropped by the firewall, the WebGate does not receive any reply; so it waits for the TCP timeout. Following the TCP timeout, WebGate understands the message channel is of no use and starts the process to get a new message channel. TCP timeout is OS specific and may vary from several minutes to hours which makes the WebGate unable to process user requests.

Note:

The setKeepAlive WebGate parameter ensures that load balancers do not drop the OAP connection. See User-Defined WebGate Parameters for details.

12.7.2 Monitoring Access Manager Server Health

The OAM monitoring model allows Web Tier components (load balancers) to ping an OAM Managed Server's HeartBeat endpoint at a scheduled interval over HTTP(S). This allows Web Tier components to route incoming HTTP traffic away from unhealthy OAM Managed Server(s). Every OAM Managed Server exposes this HeartBeat URL:

Scheme://ManagedServerHost:ManagedServerPort/oam/server/HeartBeat

In this URL, the following is true:

scheme = https | http
ManagedServerHost = Host name of the Access Manager WLS Managed Server
ManagedServerPort = Port used by the Access Manager WLS Managed Server

The HeartBeat URL works as follows:

The Web Tier components will send an HTTP request to the HeartBeat endpoint of the Access Manager Managed Server.
The Access Manager Managed Server will then do the following:
- Verify Id Store Connectivity
- Verify Policy Store Connectivity
- Verify the Credential Collector URLs are reachable
- Sanity check the working of the Coherence Layer
- Check for NAP connectivity
If the above tests succeed, the Access Manager server is considered to be healthy and a HTTP 200 response is sent to the Load Balancer. Any other HTTP Status Code value signifies that the Access Manager Managed Server is not healthy.
When multiple Access Manager Managed Servers are present in the deployment, the Web Tier component will repeat this for each OAM Managed Server.

Note:

Neither the health status test results or check results can be communicated in the body of the HTTP Response.