14 Monitoring Performance

Service-oriented, distributed applications, which are characterized by modular development and dynamic binding, have a critical need for a single point of management from where one can monitor the behavior of the application as a whole, identify actual or potential problems, and take corrective action.

This chapter introduces the issues and tasks involved in monitoring the performance of distributed applications. It includes the following sections:

Monitoring Views and Dimensions
Using ECIDs to Track Requests
Setting up End-to-end Monitoring
User Roles and Privileges

To monitor the performance of distributed applications, you must be able to do the following.

Examine the user experience to assess the quality of service rendered and to understand use patterns.
Discover the components that make up the application, identify request flows of interest, and determine where performance issues or errors occur in the flow.
Find the root cause of poor performance and failure by looking at the infrastructure supporting the logical application, or by obtaining more detailed information.

Used together, the products described in this guide offer the functionality described above. You do not need to use all these to learn about your application's performance. For example, you could start by monitoring the end-user experience and then later, add transaction monitoring. The next section describes the different monitoring options that are available to you.

14.1 Monitoring Views and Dimensions

End-to-end performance monitoring requires multiple views and dimensions:

A complete view of the topology of the logical application, including routing schemes and database access
A complete view of the underlying infrastructure
Varying detail about the distributed application components used
For web-based applications, the ability to access html source for the web pages visited by users
Access to machine-level execution detail for application components running in a Java Virtual Machine
The ability to go from the logical to the physical view of the application

RUEI, BTM, JVMD, and Enterprise Manager provide the functionality required for end-to-end performance monitoring. As mentioned before, you do not need to install and configure all of these. You can use the piece that addresses your most immediate concerns and add more later.

Real User Experience Insight (RUEI)

Helps you identify problems with user interfaces, evaluate the quality of service offered, and understand and anticipate use patterns.
Business Transaction Management (BTM)

Discovers the components that make up your application and allows you to define transactions, which include operations that are of special interest. You can follow the work your application does as it crosses servers (tiers of execution) and also see the topology of your distributed application.
Java Virtual Machine Diagnostics (JVMD), Enterprise Manager

Provides a server-level view of the request flow and of the internal workings of the application execution environment for those services that execute in a Java Virtual Machine.
Request Instance Diagnostics (RID), Enterprise Manager

This JVMD view allows you to look at details of a single request, and query on things that touched a particular ECID.
Business Application Page, Enterprise Manager

Allows you to define Business Applications, in which context you van view and analyze RUEI and BTM information, and to access more detailed monitoring information.

Figure 14-1 illustrates how these components work together, both in scope and in depth, to give you end-to-end performance monitoring.

The End User Management pane illustrates RUEI monitoring. An end user completes a series of steps during a browsing session. RUEI monitors the actions of web users and can create reports, segmented in a variety of ways, that tell you who has requested a page, what pages were requested, which servers were affected, what the response time was, and what the throughput rate is for a given session or user flow.

By interacting with objects on web pages, users invoke request flows that are monitored by Business Transaction Management (BTM)., BTM allows us to discover back-end application components and to define and monitor the request flows (transactions) that are critical to our understanding of application performance. For a given time period, we can determine the number of started and completed transactions, the throughput, and the average and maximum response times.

Figure 14-1 Monitoring Application Performance

Finally, in a request flow, for any given sequence of operations supported by a particular server, we can invoke deep dive diagnostics (shown in Figure 14-1) to determine whether slow or faulty service is due to low-level issues. Here we can view detailed information for the period within which a given operation executes. We can look at stack frames for executing threads, thread state information, aggregate information about the frequency and cost of method execution, information about database locks, and we can also look at objects in the Java heap.

Another way to break down the end-to-end picture is to look at the layers of execution underlying a user click and understand how RUEI, BTM, and JVMD correspond to each layer. In the figure below we see that RUEI tells us about user clicks; that BTM tells us about instances of request flows and also about individual requests (or operations), and that JVMD tells us about the code executing in the Java Virtual Machine.

Figure 14-2 Components and Execution Layers

In addition to using RUEI, BTM, and JVMD to monitor end-to-end application performance, you can also use the Enterpriser Manager (EM) console to monitor Business Applications that include RUEI applications and BTM transactions. For more information see Chapter 18, "Monitoring Business Applications."

14.2 Using ECIDs to Track Requests

Because RUEI, BTM, and JVMD have a different focus and level of granularity, it helps to have some shared identifier to help us realize that we are looking at a shared process or element.

An Execution Context ID (ECID) is an identifier for tracking a request for components in the Oracle technology stack. An ECID is usually generated by the outer-most Oracle component handling the request and may be propagated to the Oracle components handling that request, potentially crossing server boundaries.

The creation and propagation of ECIDs enable the sharing of context and of diagnostic data between components. Although ECIDs are not universally used, where they are used, they provide good support for end-to-end diagnostic work.

Several technologies generate ECIDs for message traffic; these include RMI, JAX-RPC, JAX-WS, EJB, JMS, JDBC, Servlets, and SOA. (In some cases, ECIDs are supported only when communication occurs between WebLogic servers.)

Where ECIDs are used, they can help the user determine whether they are indeed looking at the same object across execution contexts. For example, you can correlate error messages from different target components if they share the same ECID.

The components used in end-to-end performance monitoring all support the use of ECIDs.

RUEI displays ECIDs assigned to page objects in the history shown for a particular page.
If BTM observes an incoming message to have an ECID, it assigns the ECID as an intrinsic property of the message. Users can then search for messages with a particular ECID and determine, when looking at table views of operations, which operations have the same ECID. (Request and response messages for the same operation can have different ECIDs.)
ECIDs are also used at the lowest level to further identify threads running in the Java Virtual Machine.
ECIDs can also be used in the correlation of log entries for Oracle Fusion Middleware components that use the Oracle Diagnostic Logging (ODL) framework.

To have ECIDs generated by default by an HTTP server or Web Logic server, follow the instructions given in My Oracle Support Knowledge Document 1527091.1.

14.3 Setting up End-to-end Monitoring

To obtain end-to-end monitoring, you must install, configure, and connect the products described in Section 14.1. You might not need to deploy all these pieces at once. You can start with the piece that gives you the functionality you need and add other pieces later.

This section describes the steps required to set up end-to-end application performance monitoring for each dimension of performance monitoring. The purpose of each step is explained, and references are given to the relevant documentation. This section includes the following:

Set up Enterprise Manager
Set up Java Virtual Machine Diagnostics
Set up Real User Experience Insight
Set up Business Transaction Management
Create the Business Application

Before looking at the set-up instructions, take a moment to look over the following illustrations, which provide a topological view of the pieces that you can deploy to enable end-to-end monitoring.

Figure 14-3 shows how the RUEI collector, EM agents, BTM observers, and JVMD agents are deployed in a monitored environment.

The RUEI collector must be deployed in front of the web server.
The EM agent must be deployed on the machine hosting the application servers and database servers used by the distributed application.
The BTM and JVMD agents must be deployed in the application servers where application components are deployed.

Of course, which of these you deploy, depends on the views you need. For example, if you are not interested in machine-level runtime information, you do not need to deploy the JVMD agent.

Figure 14-3 Agents and Observers in the Monitored Environment

Figure 14-4 shows how RUEI, OMS, JVMD, and BTM are connected to one another and to their corresponding data collection points.

It shows that each processing engine connects to its respective console, which allows the administrator to create and update monitored objects.
It shows how RUEI, JVMD, and BTM are connected to the Oracle Management Server, which allows the sharing of data that enables the creation and monitoring of Business Applications.

A minimal user environment is shown below the administrative layer.

Figure 14-4 Processing Engines in the Monitored Environment

14.3.1 Set up Enterprise Manager

You need Enterprise Manager to create and monitor Business Applications. Enterprise Manager is also required if you want to do deep-dive diagnostics by looking at machine-level performance data. For more info on this option, see Section 14.3.2, "Set up Java Virtual Machine Diagnostics."

To set up Enterprise Manager:

Install and configure Enterprise Manager. See Oracle Enterprise Manager Cloud Control Basic Installation Guide at the following URL for more information.

http://docs.oracle.com/cd/E24628_01/install.121/e22624/toc.htm
Install an Oracle management agent on the hosts where targets and application components monitored by RUEI or BTM are running. See "Installing Oracle Management Agent" in Oracle Enterprise Manager Cloud Control Basic Installation Guide at the following URL.

http://docs.oracle.com/cd/E24628_01/install.121/e22624/install_agent.htm
Launch Enterprise Manager and use the Enterprise Manager console to create and monitor Business Applications. See "Set up Business Transaction Management."

14.3.2 Set up Java Virtual Machine Diagnostics

To access machine-level performance data using the JVMD or RID views, you must install the JVMD manager and JVMD agents. JVMD is an integral part of Enterprise Manager, so the latter must be installed before you install JVMD.

To set up Java Virtual Machine Diagnostics:

Deploy the JVMD Manager (JVM Diagnostics Engine) in Enterprise Manager. You need this step to access JVM monitoring data. For information, see "Installing JVM Diagnostics" in Oracle Enterprise Manager Cloud Control Basic Installation Guide.

http://docs.oracle.com/cd/E24628_01/install.121/e22624/jvmd_installation.htm
Install a JVMD agent on all nodes where targets and services monitored by RUEI and BTM are running. You need this step to collect JVM data for a given server. For information, see "Installing JVM Diagnostics" in Oracle Enterprise Manager Cloud Control Basic Installation Guide.

14.3.3 Set up Real User Experience Insight

To obtain information about the user experience, you must install and configure RUEI, and then register it with Enterprise Manager.

To set up RUEI:

Install and configure RUEI. This step includes the following:
- Install collectors, processor, and reporter in the monitored environment.
- Install the reporter database.
- Configure the RUEI reporter.
Configuration teaches RUEI to identify users, to specify the collection of pages that make up an application, to specify the scope of monitoring, to configure mail notification, and to provide security options. It is also at this time that you set up a connection to the Oracle Enterprise Manager. For information, see Oracle Real User Experience Insight Installation Guide from the appropriate RUEI documentation library:

http://www.oracle.com/technetwork/documentation/realuserei-091455.html
Configure clickout functionality in RUEI. You need this step to be able to click out to external tools. In this case, you will want to enable clickout to BTM and JVMD. For information, see "Configuring Clickouts to External Tools" in Oracle Real User Experience Insight User's Guide.
Register RUEI with Enterprise Manager. Specify the port where the Reporter system can be accessed and provide access credentials. You need this step to establish communication between RUEI and EM. For information, see Chapter 18, "Monitoring Business Applications."

14.3.4 Set up Business Transaction Management

To discover your application components, to define transactions, and to define monitoring options for these, you must use Business Transaction Management. In the context of monitoring Business Applications, BTM is used to monitor cross-tier transactions; it is not needed for single-tier applications.

To set up Business Transaction Management:

Install and configure BTM. This step includes the following:
- Installation and configuration of central servers. (At this time, you can also configure the connection to the Oracle Enterprise Manager server.)
- Installation and configuration of monitors, which defines communication between monitors and observers.
- Installation of observers on every server hosting the services to be monitored.
For information, see Oracle Business Transaction Management Installation Guide at the following URL:

http://docs.oracle.com/cd/E24628_01/nav/assoproducts.htm
Enable access to JVMD. You need this step to access JVMD and RID views from BTM. For information, see "Enabling Access to the JVMD and RID Views" in Oracle Business Transaction Management Online Help.

http://docs.oracle.com/cd/E24628_01/nav/assoproducts.htm
Wait for traffic, discover services, and define transactions in BTM. Discovery in BTM is always dynamic. In a production environment, you must wait for traffic before you can discover application components. In a testing environment, you should run traffic to enable discovery. For more information, see Oracle Business Transaction Management Online Help.
Register BTM with Enterprise Manager. Specify the port where the Sphere can be accessed and provide access credentials. You need this step to establish communication between BTM and EM. For information, see Chapter 18, "Monitoring Business Applications."

14.3.5 Create the Business Application

For end-to-end monitoring, you want to create a Business Application that includes your RUEI applications, BTM transactions, and the system that supports these. You can build up your Business Application as you go along. You can start by including only the RUEI application, and then add any related transactions. You can even start by looking at the system that supports your distributed applications without including either a RUEI application or a BTM transaction.

To create a Business Application:

Create a system in Enterprise Manager that specifies the hosts and containers where monitored application components are running. These hosts and containers are the infrastructure of your distributed application. You need this step for Enterprise Manager to collect and return information about the health of the underlying infrastructure. For more information, see the online help for the Enterprise Manager Console.
Create a Business Application using the Enterprise Manager console. This step specifies the RUEI applications and BTM transactions to be included in a Business Application, and it specifies which system (Step 1) supports the Business Application. For information, see Chapter 18, "Monitoring Business Applications."
Monitor the Business Application. Use the Enterprise Manager Console to monitor the performance of your business application. For more information, see Chapter 18, "Monitoring Business Applications."
Edit the RUEI application if needed.

If you have defined a RUEI application and monitoring results in Enterprise Manager show that you need to change its definition to segment data differently or to re-set key performance indicators, you will need to use the RUEI console to change the application definition. For information, see Oracle Real User Experience Insight User's Guide at the following URL:

http://docs.oracle.com/cd/E48389_01/index.htm

Enterprise Manager is automatically updated with the new definitions.
Edit the BTM transaction if needed.

If you have defined a transaction and monitoring results suggest that you need to change the transaction definition to collect more or less data, to add properties or conditions, or to re-set key performance indicators, you will need to use the BTM console to change the transaction definition. For more information, see Oracle Business Management Online Help at the following URL.

http://docs.oracle.com/cd/E24628_01/nav/assoproducts.htm

EM is automatically updated with the new definitions.

14.4 User Roles and Privileges

User roles and privileges define accessibility to component functions. The following guidelines apply as you work with components to monitor application performance.

Overall, higher privileges are required to create entities in RUEI and BTM than to monitor them in EM.
Clicking through from one component to another exposes you to each console's native authentication system. Make sure that you have the privileges required for each component to perform your work.
With the exception of the admin and superAdmin rules, in EM roles are always associated with targets. What is visible to you in the EM console depends on your role with regard to a particular target.
You need the super admin role to register a BTM or RUEI system with EM. Once the RUEI or BTM system is registered with EM, you don't need the super admin role to create a business application.
You need Create Any Target privilege and View Target privilege on the RUEI or BTM system target to access the credentials used by EM to talk to RUEI and BTM.
You need Manage Business Application and Business Application Menu Item Application Performance Management resource privileges.
To view JVM Diagnostics data, you must have JVM Diagnostics User privileges.
To manage JVM Diagnostics operations such as creating and analyzing heap and thread snapshots, tracing threads, and so on, you must have JVM Diagnostics Administrator privileges.