13 Diagnosing the Oracle Fusion Applications Middle Tier Performance

This chapter describes how to diagnose Java applications in the middle tier using Oracle Enterprise Manager Cloud Control (Cloud Control):

This chapter includes the following topics:

Section 13.1, "Introduction to Java Diagnostics in the Middle Tier"
Section 13.2, "Diagnosing the Oracle Fusion Applications Middle Tier"
Section 13.3, "Working with Class Histograms"
Section 13.4, "Deploying/Upgrading Bulk Diagnostics Agents"
Section 13.5, "Upgrading/Redeploying ADP and JVMD Engines"
Section 13.6, "Working with Pool Threshold Violations"

13.1 Introduction to Java Diagnostics in the Middle Tier

Mission critical Java applications often suffer from availability and performance problems. Developers and IT administrators spend a lot of time diagnosing the root cause of these problems. Many times, the problems occurring in production environments either cannot be reproduced or may take too long to reproduce in other environments. This can cause severe impact on the business.

Oracle Enterprise Manager Cloud Control (Cloud Control) enables you to diagnose performance problems in Java applications in the production environment. By eliminating the need to reproduce problems, it reduces the time required to resolve these problems. This improves application availability and performance. Using Java Virtual Machine (JVM) diagnostics, you can identify the root cause of performance problems in the production environment without having to reproduce them in the test or development environment. It does not require complex instrumentation or restarting of the application to get in-depth application details. Application administrators will be able to identify Java problems or Database issues that are causing application downtime without any detailed application knowledge.

To configure your environment for JVM Diagnostics, see the "Using JVM Diagnostics" chapter in the Oracle Enterprise Manager Getting Started with Fusion Middleware Management.

13.2 Diagnosing the Oracle Fusion Applications Middle Tier

This section describes how to diagnose Java applications in the middle tier using Cloud Control:

Section 13.2.1, "Viewing JVM Pool Summary Information on a Domain"
Section 13.2.2, "Viewing JVM Summary Information on a Managed Server"
Section 13.2.3, "Viewing JVM Pool Summary Information on a JVM"
Section 13.2.4, "Finding the Top Java Methods"
Section 13.2.5, "Finding the Top SQL Queries"
Section 13.2.6, "Analyzing Live Stuck Threads"
Section 13.2.7, "Analyzing Stuck Threads from Specific Time Periods"
Section 13.2.8, "Drilling Down from JVM Diagnostics to SQL Instances"
Section 13.2.9, "Analyzing Potential Memory Leaks"

13.2.1 Viewing JVM Pool Summary Information on a Domain

To get started with monitoring Java Virtual Machine (JVM) Pool on a Domain using Cloud Control:

From the Targets menu, choose Targets > Middleware.

The Middleware target home page displays.
Search for the Oracle WebLogic Domains:
1. From the Search area, click Advanced Search.
2. From the Type list, select Oracle WebLogic Domain and deselect the other options.
3. Click Search.
Select a domain which has the JVM Diagnostics Agent deployed on it:
1. From the Target Navigation area, select Java Virtual Machine Pools and expand it.
  
  Note:
  
  All the WebLogic Domains that have a JVM Diagnostics Agent deployed on it appear under this folder.
From the WebLogic Domain, choose Diagnostics > Java Virtual Machine Pool Home.

The JVM Pool Home page displays.

Description of the illustration jvmpoolhome.gif

Take note of the following:

In the Summary section, Poll indicates if polling of that domain has been enabled. Polling Interval denotes the time interval (in milli seconds) at which the domain is polled.
In the Availability section, the JVMs in that pool are listed along with their availability status.
In the Realtime Thread States section, view the overall state of the threads including locks.
In the Incident section, the servers in the domain whose target status is Down are listed. The incidents will be automatically cleared when the underlying issues are resolved.

From the WebLogic Domain menu, choose Diagnostics > JVM Performance Diagnostics.

The JVM Pool Performance Diagnostics page displays. This page provides information for the pool of JVMs in the domain for the selected time period. You can also modify the time period using Edit Date and Time option at the top of the page.

The following figure shows the JVM Pool Performance Diagnostics page for the domain:

Description of the illustration jvmpoolperfdiag_domain.png

View the various charts:

Server State Charts For Selected Period section	This section displays the Active Threads, CPU Utilization, and Heap Utilization of IO, CPU, lock, and network resources during the selected time. The number of Java threads (daemon and non-daemon) that are currently running in the virtual machine for this Oracle WebLogic Domain: Active Threads by State: This chart displays the number of Java threads that are currently running in the domain or server. It is color-coded by thread state. JVM CPU Utilization (%): This chart shows the CPU utilization across the JVMs in the pool. JVM Heap Utilization (%): This chart shows the heap utilization across the JVMs in the pool.
Filter Options section	This section displays various options by which you can filter the JVMs in the pool.
Top Activities section	This section displays the top activities in the pool: Top Methods: This chart shows the most expensive Java methods in the selected time period. Top Requests: This chart shows the top page requests in the selected time period. Top DBWait Events: This chart shows the cross-tier correlation with the database. Top SQLs: This chart displays the list of SQL calls ordered by their cost (the number of samples). Top Databases: This chart shows the top impacted databases by the JVMs in this pool.
JVMs In Pool section	This section displays the Active Threads, CPU Utilization, and JVM Heap Utilization of IO, CPU, lock, and network resources during the selected time. Active Threads: This chart displays the number of active Java threads that are currently running in the domain. It is color-coded by thread state. CPU Utilization (%): This chart shows the CPU utilization across the JVMs in the pool. JVM Heap Utilization (%): This chart shows the heap utilization across the JVMs in the pool. Garbage Collections (Invocations/min): This chart shows the number of times the JVM garbage collector was invoked in the time period. It includes both major and minor garbage collections.

Note:

The Compared with feature enables you to compare the diagnostics across two specified periods of time.

13.2.2 Viewing JVM Summary Information on a Managed Server

To get started with monitoring Java Virtual Machine (JVM) on a Managed Server within a domain using Cloud Control:

From the Targets menu, choose Targets > Middleware.

The Middleware target home page displays.
Search for the Oracle WebLogic Domains:
1. From the Search area, click Advanced Search.
2. From the Type list, select Oracle WebLogic Domain and deselect the other options.
3. Click Search.
Select a domain which has the JVM Diagnostics Agent deployed on it:
1. From the Target Navigation area, select Java Virtual Machine Pools and expand it.
  
  Note:
  
  All the WebLogic Domains and Servers that have JVM Diagnostics Agents deployed on them appear under this folder.
Expand the WebLogic Domain and select a Managed Server.
From the WebLogic Server menu, choose Diagnostics > Java Virtual Machine Home.

The JVM Home page displays.

Description of the illustration jvmhome.gif

Take note of the following:

In the Summary section, the details of the JVM are displayed. The details contain the following information:
- JVM Pool
- Host machine
- Operating System (OS) running on the host machine
- JVM Vendor
- JVM Version
- Maximum JVM Heap Size
- Minimum JVM Heap Size
- JVMD Agent Optimization Level
- JVMD Agent Log Level
- JVMD Agent Build Number
- WebLogic Server.
In the Availability section, the availability status of the JVM is listed.
In the Realtime Thread States section, view the overall state of the threads including locks.
In the Incident section, the JVM corresponding to the server whose target status is Down is listed. This incident will be automatically cleared when the underlying issue is resolved.
In the Active Threads (Last 24 Hours) section, view a graph analyzing the active threads in the JVM in the last 24 hours.

From the WebLogic Server menu, choose Diagnostics > JVM Performance Diagnostics.

The JVM Performance Diagnostics page displays. This page provides information for a single JVM for the selected time period. You can also modify the time period using Edit Date and Time option at the top of the page.

The following figure shows the JVM Performance Diagnostics page for a single JVM:

Description of the illustration jvmperfdiag_wlserver.png

The Thread Transition Graph is displayed as ADF Bar Graph. By default, the data is shown for 30 minutes and the maximum time window is one hour.

View the various charts:

Server State Charts For Selected Period section	This section displays the Active Threads, CPU Utilization, and Heap Utilization of IO, CPU, lock, and network resources during the selected time. The number of Java threads (daemon and non-daemon) that are currently running in the virtual machine for this Oracle WebLogic Server: Active Threads by State: This chart displays the number of Java threads that are currently running in the server. It is color-coded by thread state. CPU Utilization (%): This chart shows the CPU utilization for the JVM. Heap Utilization (%): This chart shows the heap utilization for the JVM. Garbage Collections (Invocations/min): This chart shows the number of times the JVM garbage collector was invoked in the time period. It includes both major and minor garbage collections.
Filter Options section	This section displays various options by which you can filter the JVM.
General tab	This tab displays data from the JVM: Active Threads by State: This chart displays the number of Java threads that are currently running in server. It is color-coded by thread state. Top Requests: This chart shows the top page requests in the selected time period. Top Methods: This chart shows the top methods in the selected time period. Top SQLs: This chart shows the top SQLs in the selected time period. Top DBWait Events: This chart shows the top DB Wait Events in the selected time period. Top Databases: This chart shows the top impacted databses by the JVMs in this pool.
Threads tab	This tab displays data from the JVM. Threads State Transition: This chart shows how the threads have transitioned from one state to the other in the selected period. You can change the time interval and move it to a different time period by using the quick time selection control at the top of the page. You can hover over the colored bars to see the transition changes from one state to the other, for example from Runnable to Not Active or to Runnable. Click a bar graph in the State column to view a detailed analysis on the state of the thread. This feature allows you to analyze each sample (JVM snapshot at a specific time) in the monitored data. Metric By Active States: This chart shows how long each of the threads have been in the various states.

Note:

The Compared with feature enables you to compare the diagnostics across two specified periods of time.

13.2.3 Viewing JVM Pool Summary Information on a JVM

To get started with monitoring a Java Virtual Machine (JVM) in a Pool using Cloud Control:

From the Targets menu, choose Targets > Middleware.

The Middleware target home page displays.
Search for the Oracle WebLogic Domains:
1. From the Search area, click Advanced Search.
2. From the Type list, select Oracle WebLogic Domain and deselect the other options.
3. Click Search.
Select a domain which has the JVM Diagnostics Agent deployed on it:
1. From the Target Navigation area, select Java Virtual Machine Pools and expand it.
  
  Note:
  
  All the WebLogic Domains and Servers that have JVM Diagnostics Agents deployed on them appear under this folder.
  
  Note:
  
  The WebLogic Domain is suffixed with _jvmpool and the WebLogic Servers are suffixed with _jvm.
2. Select a WebLogic Domain or WebLogic Server under this folder.
When you select a WebLogic Domain, select Java Virtual Machine Pool Home from the Java Virtual Machine Pool menu.

The JVM Pool Summary page displays.

Description of the illustration jvmpoolsummary_domain.png

Take note of the following:

In the Summary section, Poll indicates if polling of that JVM Pool has been enabled. Polling Interval denotes the time interval (in milli seconds) at which the JVM Pool is polled.
In the Availability section, the availability status of all the JVMs is listed.
In the Realtime Thread States section, view the overall state of the threads including locks.
In the Incident section, the servers in the domain whose target status is Down are listed. The incidents will be automatically cleared when the underlying issues are resolved.

From the Java Virtual Machine Pool menu, choose JVM Performance Diagnostics.

The JVM Pool Performance Diagnostics page displays. This page provides information about all the JVMs in the pool for the selected time period. You can also modify the time period using Edit Date and Time option at the top of the page.

The following figure shows the JVM Pool Performance Diagnostics page for the domain:

Description of the illustration jvmpoolperfdiag_jvm_domain.png

View the various charts:

Server State Charts For Selected Period section	This section displays the Active Threads, CPU Utilization, and Heap Utilization of IO, CPU, lock, and network resources during the selected time. The number of Java threads (daemon and non-daemon) that are currently running in the virtual machine for this Oracle WebLogic Domain: Active Threads by State: This chart displays the number of Java threads that are currently running in the server. It is color-coded by thread state. CPU Utilization (%): This chart shows the CPU utilization across the JVM. Heap Utilization (%): This chart shows the heap utilization across the JVM. Garbage Collections (Invocations/min): This chart shows the number of times the JVM garbage collector was invoked in the time period. It includes both major and minor garbage collections.
Filter Options section	This section displays various options by which you can filter the JVMs in the pool.
Top Activities section	This section displays the top activities in the pool: Top Methods: This chart shows the most expensive Java methods in the selected time period. Top Requests: This chart shows the top page requests in the selected time period. Top DBWait Events: This chart shows the cross-tier correlation with the database. Top SQLs: This chart displays the list of SQL calls ordered by their cost (the number of samples). Top Databases: This chart shows the top impacted databases by the JVMs in this pool.
JVMs In Pool section	This section displays the Active Threads, CPU Utilization, and JVM Heap Utilization of IO, CPU, lock, and network resources during the selected time. Active Threads: This chart displays the number of active Java threads that are currently running in the domain. It is color-coded by thread state. CPU Utilization (%): This chart shows the CPU utilization across the JVMs in the pool. JVM Heap Utilization (%): This chart shows the heap utilization across the JVMs in the pool. Garbage Collections (Invocations/min): This chart shows the number of times the JVM garbage collector was invoked in the time period. It includes both major and minor garbage collections.

Note:

The Compared with feature enables you to compare the diagnostics across two specified periods of time.

When you select a WebLogic Server, select Java Virtual Machine Home from the Java Virtual Machine menu.

The JVM Summary page displays.

Description of the illustration jvmsummary_jvm_server.png

Take note of the following:

From the Java Virtual Machine menu, choose JVM Performance Diagnostics.

The JVM Performance Diagnostics page displays. This page provides information for a single JVM for the selected time period. You can also modify the time period using Edit Date and Time option at the top of the page.

The following figure shows the JVM Performance Diagnostics page for the JVM:

Description of the illustration jvmperfdiag_jvm_server.png

The Thread Transition Graph is displayed as ADF Bar Graph. By default, the data is shown for 30 minutes and the maximum time window is one hour.

View the various charts:

Server State Charts For Selected Period section	This section displays the Active Threads, CPU Utilization, and Heap Utilization of IO, CPU, lock, and network resources during the selected time. The number of Java threads (daemon and non-daemon) that are currently running in the virtual machine for this Oracle WebLogic Server: Active Threads by State: This chart displays the number of Java threads that are currently running in the server. It is color-coded by thread state. CPU Utilization (%): This chart shows the CPU utilization for the JVM. Heap Utilization (%): This chart shows the heap utilization for the JVM. Garbage Collections (Invocations/min): This chart shows the number of times the JVM garbage collector was invoked in the time period. It includes both major and minor garbage collections.
Filter Options section	This section displays various options by which you can filter the JVM.
General tab	This tab displays data from the JVM: Active Threads by State: This chart displays the number of Java threads that are currently running in server. It is color-coded by thread state. Top Requests: This chart shows the top page requests in the selected time period.
Threads tab	This tab displays data from the JVM: Threads State Transition: This chart shows how the threads have transitioned from one state to the other in the selected period. You can change the time interval and move it to a different time period by using the quick time selection control at the top of the page. You can hover over the colored bars to see the transition changes from one state to the other, for example from Runnable to Not Active or to Runnable. Click a bar graph in the State column to view a detailed analysis on the state of the thread. This feature allows you to analyze each sample (JVM snapshot at a specific time) in the monitored data. Metric By Active States: This chart shows how long each of the threads have been in the various states.

Note:

The Compared with feature enables you to compare the diagnostics across two specified periods of time.

13.2.4 Finding the Top Java Methods

If you have a slow-running application, locate the Java method causing the potential issue.

To find the top Java methods with Cloud Control:

From the Targets menu, choose Targets > Middleware.

The Middleware target home page displays.
Search for the Oracle WebLogic Server domains:
1. From the Search area, click Advanced Search.
2. From the Type list, select Oracle WebLogic Domain or Oracle WebLogic Server; deselect the other options.
3. Click Search.
Click a domain or one of its servers which has JVM Diagnostics agents deployed. For more information about installing JVM Diagnostics, see the Oracle Enterprise Manager Cloud Control Basic Installation Guide.

The WebLogic Server Domain Home page or WebLogic Server Home page displays.
From the WebLogic Domain menu or WebLogic Server menu, choose Diagnostics > JVM Performance Diagnostics.

The JVM Pool Performance Diagnostics page displays for an Oracle WebLogic Domain and the JVM Performance Diagnostics page displays for a Managed Server.
In the Top Methods section, review the list of methods ordered by their cost (the number of samples).
In the Top Methods section, click a method to add it as a filter.
In the Method Stack Tree dialog, click the plus sign next to the method to investigate further.
The Method Stack Tree dialog displays with a call stack that you can traverse.
In the Method Stack Tree dialog, select the method, and then click OK.

The Filter Options section auto-fills the information on the method and the charts update to reflect that method, for example:
- Heavy use of IO, CPU, lock, and network resources (Active Threads chart)
- Request causing it to be invoked (Top Requests chart)
- SQL or database state it causes (Top DBStates and Top SQLs charts)
After you are done with viewing the method, in Filter Options section, clear out the Method text box filter and click anywhere to remove the filter.

13.2.5 Finding the Top SQL Queries

If you suspect a slow SQL query is causing a network requests or IO issue, find the slowest SQL queries.

To find the top SQL calls using JVM diagnostics with Cloud Control:

From the Targets menu, choose Targets > Middleware.

The Middleware target home page displays.
Search for the Oracle WebLogic Server domains:
1. From the Search area, click Advanced Search.
2. From the Type list, select Oracle WebLogic Domain or Oracle WebLogic Server; deselect the other options.
3. Click Search.
Click a domain or one of its servers which has JVM Diagnostics agents deployed. For more information about installing JVM Diagnostics, see the Oracle Enterprise Manager Cloud Control Basic Installation Guide.

The WebLogic Server Domain Home page or WebLogic Server Home page displays.
From the WebLogic Domain menu or WebLogic Server menu, choose Diagnostics > JVM Performance Diagnostics.

The JVM Pool Performance Diagnostics page displays for an Oracle WebLogic Domain and the JVM Performance Diagnostics page displays for a Managed Server.
In the Top SQLs section, review the list of SQL calls ordered by their cost (the number of samples).
In the Top SQLs section, click a SQL call to view the charts for that call.

The Filter Options section auto-fills the information on the method and the charts update to reflect that method. Adding the statement as a filter enables you to see everything related to that SQL call, for example:
- Methods that invoke it (Top Methods chart)
- Request causing it to be invoked (Top Requests chart)
- Database state it causes (Top Databases chart)
After you are done viewing the method, in Filter Options section, clear out the SQL field and click anywhere to remove the filter.

13.2.6 Analyzing Live Stuck Threads

If application users report a spinning status indication after clicking in the application, investigate the live stuck threads. Follow the procedure given below to find and analyze the live stuck threads.

To find and analyze the live stuck threads:

From the Targets menu, choose Targets > Middleware.

The Middleware target home page displays.
Search for the Oracle WebLogic Server domains:
1. From the Search area, click Advanced Search.
2. From the Type list, select Oracle WebLogic Domain or Oracle WebLogic Server; deselect the other options.
3. Click Search.
Click a domain or one of its servers which has JVM Diagnostics agents deployed. For more information about installing JVM Diagnostics, see the Oracle Enterprise Manager Cloud Control Basic Installation Guide.

The WebLogic Server Domain Home page or WebLogic Server Home page displays.
From the WebLogic Domain menu, choose Diagnostics > Live Thread Analysis.

The Live Thread Analysis page displays.

Description of the illustration livethreadanalysis.png
In the JVMs section, select a thread to display details in the JVM Threads section.
In the JVM Threads section, look for a thread having the prefix [STUCK] and click it.
In the Thread Info and Thread Stack sections, look at the Current Call, File Name, Line, and State for the thread.

This information provides you with the key information on how to locate the code that is causing the problem:
- Current Call: This field displays the name of the method call where the code is stuck.
- File Name: This column identifies the file with the problem.
- Line: This column identifies the line number in the file where the problematic code is.
- State: This column displays the state of the thread (for example, CPU, IO, Network, DB Wait, Lock, and so on).
Look for the Lock Held in the Thread Info section.
If the stuck thread is in the DB Wait state, then click the link and go directly to the database session to see what that thread is doing in the database, or use the technique described in Section 13.2.8.

13.2.7 Analyzing Stuck Threads from Specific Time Periods

The procedure listed in this section allows you to find and analyze the stuck threads in the historical context.

To find the stuck threads in the historical context:

In Target Navigation area, expand Java Virtual Machine Pools > domain_name_jvmpool.
Select a Java Virtual Machine.
From the Java Virtual Machine menu, choose JVM Performance Diagnostics.
Description of the illustration jvmperfdiagnostics.png
In the Filter Options, enter [STUCK]% in the Thread Name field.
In the Threads sub-tab, drag the pointer to select the timeline.

Note:

Scroll left and right to select the appropriate date. After you select the date, select the required duration.

13.2.8 Drilling Down from JVM Diagnostics to SQL Instances

If you issue an SQL query and it does not return, then analyze the SQL statement.

To analyze SQL from Cloud Control:

From the Targets menu, choose Targets > Middleware.

The Middleware target home page displays.
Search for the Oracle WebLogic Server domains:
1. From the Search area, click Advanced Search.
2. From the Type list, select Oracle WebLogic Domain or Oracle WebLogic Server; deselect the other options.
3. Click Search.
Click a domain or one of its servers which has JVM Diagnostics agents deployed. For more information about installing JVM Diagnostics, see the Oracle Enterprise Manager Cloud Control Basic Installation Guide.

The WebLogic Server Domain Home page or WebLogic Server Home page displays.
From the WebLogic Domain menu or WebLogic Server menu, choose Diagnostics > JVM Performance Diagnostics.

The JVM Pool Performance Diagnostics page displays for an Oracle WebLogic Domain and the JVM Performance Diagnostics page displays for a Managed Server.
In the Top SQLs section, click a SQL call to view the charts for that call.
In Filter Options section, in the SQL field, copy the ID of the SQL call.
Click Databases in secondary navigator.
Select the relevant target database.
Click the Performance tab.
In the Additional Instance Monitoring Links section, click Search SQL.
Paste the ID of the SQL call into the relevant field with any other choices you may need and then click Search.
Analyze the SQL.

13.2.9 Analyzing Potential Memory Leaks

To find and analyze memory leaks, you can use Cloud Control to take and analyze snapshots of the heap.

Analyzing heap requires a large amount of free space in the Oracle Database tablespace being used. As a standard practice, ensure you have five times the size of heap dump file being loaded in the tablespace. Since you know the size of your dump file, make sure that there is adequate space to accommodate the dump file before it is loaded into the database.

To use heap snapshot to analyze memory leaks:

Task 1, "Review Metrics for a Potential Memory Leak"
Task 2, "Perform a Live Heap Analysis"
Task 3, "Create a Heap Snapshot"

Task 1 Review Metrics for a Potential Memory Leak

To create a snapshot of the heap for later loading and examination for leaks:

From the Targets menu, choose Targets > Middleware.

The Middleware target home page displays.
Search for the Oracle WebLogic Server domains:
1. From the Search area, click Advanced Search.
2. From the Type list, select Java Virtual Machine; deselect the other options.
3. Click Search.
Click a Java Virtual Machine with JVM Diagnostics agents deployed. For more information about installing JVM Diagnostics, see the Oracle Enterprise Manager Cloud Control Basic Installation Guide.

The Java Virtual Machine Home page displays.
From the Java Virtual Machine menu, choose Performance Summary.

The Performance Summary page displays.

Review the following metrics for any periods of time where the Warning Thresholds or Critical Thresholds were reached:

JVM GC Overhead	This metric shows the percentage of CPU the JVM is using for garbage collections in relation to total CPU usage including servicing application workload (the lower, the better). This metric and its trending can help determine when the garbage collector is making the CPU spin on garbage collection instead of on application workload.
JVM Heap Usage (%)	This metric shows the heap utilization for the JVM. This metric provides an indicator of heap size as they fluctuate between garbage collections.
JVM Heap Used After GC	This metric shows the percent of heap utilization used after garbage collection. This metric and its trend over time can provide a good indication that there is a leak. For example, if the chart trending up while the application load is stable, then it is possible there is a leak.

If any of the metrics exceed the Warning Thresholds or Critical Thresholds, it could indicate memory is a factor in the JVM performance and availability. It could mean there is a memory leak or that the JVM heap configuration is too small for the application load. If the heap configuration is correct, assume there is a leak and investigate the cause.

If any of the metrics exceed the Warning Thresholds or Critical Thresholds, proceed to Task 2.

Task 2 Perform a Live Heap Analysis

To create a snapshot of the heap for later loading and examination for leaks:

From the Java Virtual Machine menu, choose Live Heap Analysis.

The Live Heap Analysis page displays.
Review the top portion of the page to the analyze the heap and the number of objects added to the garbage collector; review the JVM Class Detail table to review the largest-size objects in the heap.

For more information about using this page, see the topic "Viewing the Real-Time Heap Data" in the Cloud Control online help.

Task 3 Create a Heap Snapshot

To create a snapshot of the heap for later loading and examination for leaks:

On the Live Heap Analysis page, click Create Heap Snapshot.

The Heap Snapshot page displays.
Provide the settings for your environment.

Notice the Heap Snapshot Type under Heap Snapshot Only option enables you to pick either Oracle's JVMD format or a HROFF for use with other tools.For more information about using this page, see the topic "Taking a Heap Snapshot" in the Cloud Control online help.
If you selected Heap Snapshot Only option, click Take Snapshot.

The heap snapshot is generated and the file name in which it is stored is displayed. You can upload the heap snapshot and analyze it using appropriate options from the Heap Snapshots menu.
From the Java Virtual Machine menu, choose Heap Snapshots.

The Available Heap Snapshots page displays.
Select the heap you created from the table, and then click Detail.

The Heaps > Roots page displays. The Roots tab displays the objects reachable by roots, which are objects that are directly reachable from the JVM itself.

Description of the illustration heapdumps.gif
Click a root name to drill down and view the objects that consume a lot of memory.

The Top 40 Objects page displays.
In the Heaps > Roots page, click Compare with to compare the current heap with another previously taken heap dump.

When comparing heaps, select the bigger one first. Otherwise you may see negative deltas.
In the Select a heap record dialog, select the second heap, and then click OK.
Compare both the heaps. Compare the number of objects (Objects) and the occupied memory size (Adjusted Memory) in each heap dump. This measure indicates the objects that are growing over the period of time when the snapshots were taken.
Drill down into the root which had the largest delta in order to find the biggest memory leak.

13.3 Working with Class Histograms

A class histogram is displayed in the form of a table when the optimization level of the jamagent is 0. The histogram displays the top 300 data rows sorted by the size. This feature allows you to monitor histograms for the analysis of the Virtual Machine.

This section contains the following sections:

Section 13.3.1, "Viewing Class Histogram"
Section 13.3.2, "Saving Class Histogram"
Section 13.3.3, "Scheduling Histogram Jobs"
Section 13.3.4, "Comparing Class Histograms"
Section 13.3.5, "Deleting Class Histograms".

13.3.1 Viewing Class Histogram

To get started with viewing the Class Histogram:

From the Targets menu, choose Targets > Middleware.

The Middleware target home page displays.
Select a Java Virtual Machine which is up and running.

The Java Virtual Machine Home page displays.
Select Live Heap Analysis from the Java Virtual Machine menu.

The Live Heap Analysis summary page displays. The JVM Class Details are displayed in the form of a table on this page.

13.3.2 Saving Class Histogram

To get started with saving the Class Histogram:

From the Targets menu, choose Targets > Middleware.

The Middleware target home page displays.
Select a Java Virtual Machine which is up and running.

The Java Virtual Machine home page displays.
Select Live Heap Analysis from the Java Virtual Machine menu.

The JVM Class Details are displayed in a tabular form.
Click Save.
Enter a name for the Snapshot in the corresponding field.
Provide a description for the snapshot.
A confirmation window appears.
Click View Saved Histograms to view a list of available Class Histograms.

13.3.3 Scheduling Histogram Jobs

Scheduling will allow you to insert JVM Class Histogram data into the repository by running the job at the defined time.

To get started with scheduling the running of histogram jobs:

Navigate to Live Heap Analysis summary page.
Click Schedule under the JVM Class Details.

The Schedule Settings window appears.
Enter a name and description for the job to be scheduled.
Select when to start the job.
- If you want to schedule the job with immediate effect, select Immediately.
- If you want to save the job to be run in future, select Later. Enter a date and time for the job.
Select the frequency at which you want to repeat the job from the Repeat drop-down list.
Select the option for the Grace Period. If you select the grace period, the job will remain active and run within the specified grace period.
Click OK to schedule the histogram job.
A confirmation window appears.

13.3.4 Comparing Class Histograms

The compare functionality allows you to compare any two class histogram snapshots listed in the table.

To get started with comparing the class histogram snapshots:

Navigate to Live Heap Analysis page.
Click View Saved Histograms under JVM Class Details.
Select any two class histograms (using the Ctrl key) under Available Class Histograms and click Compare.

The Compare Class Histograms page appears.

The comparison table displays the following details:
- Class name
- Instance size for both the snapshots
- Number of instances for both the snapshots.

13.3.5 Deleting Class Histograms

The remove functionality allows you to delete the saved class histograms from the repository.

To get started with deleting class histograms:

Navigate to the Live Heap Analysis summary page.
Click Views Saved Histograms under JVM Class Details.
Select the class histogram you want to delete from Available Class Histograms.
Click Remove.
Select OK in the confirmation window that appears.

13.4 Deploying/Upgrading Bulk Diagnostics Agents

Bulk deployment/upgradation of JVMD Agents overcomes the limitation of deploying/upgrading a JVMD agent on a single domain. This feature facilitates the management of deploying/upgrading JVMD agents on all discovered WebLogic domains using a single wizard.

To get started with deploying bulk diagnostic agents:

From the Setup menu, select Application Performance Management.

The Application Performance Management page appears. The number of JVMD Agents and ADP Agents deployed is displayed under Application Performance Management Agents.
Click Manage Diagnostic Agents to deploy diagnostic agents.

The Select Targets page appears.
Select Deploy under Operation.
Select the targets on which you want to deploy ADP and JVMD agents and click Next.

The Target Credentials page appears.
Select each target and specify the WebLogic Administration Server Host Credentials and Weblogic Domain Credentials.
Click Apply to save the changes made to the target credentials and click Next.
Select each domain and specify ADP Agent Configuration details. Click Next.

The JVMD Agents Configurations page appears.
Select each domain and specify the JVMD Agent Configuration details. Click Apply.
Click Next.
Specify the Host and WebLogic domain credentials for Enterprise Manager OMS.
Click Save to apply the changes made to the credentials and click Next.
Review all the target details, credentials, and engine details and click Deploy.
Click JVMD Agent Deployment Job on Diagnostics Agents Deployment Status to view the progress of the job.
A summary of the JVMD Agent Job progress appears.

To get started with upgradation of ADP and JVMD Agents:

Select Upgrade under Operation on Select Targets page to upgrade ADP and JVMD Agents to the latest version.

Note:

You can upgrade agents only for WebLogic Server targets.
Specify Target Credentials.
Specify Agent Configuration details.
Review the target details, credentials, and engine details. Click Upgrade.

You can view the progress of the agent deployment job.

13.5 Upgrading/Redeploying ADP and JVMD Engines

You can upgrade/redeploy the selected ADP/JVMD Engines on local or existing remote WebLogic Servers to the latest available version. When the upgrade is completed, the upgraded engine(s) gets redeployed.

To get started with upgrading the selected ADP/JVMD Engines:

From the Setup menu, select Application Performance Management.

The Application Performance Management page appears. The available JVMD Engines and ADP Engines are listed under Application Performance Management Engines.
Click the Upgrade icon next to JVMD Engines or ADP Engines to upgrade the corresponding engine(s) or click the Upgrade button.
Select the engine you want to upgrade.
Specify the Management Agent Host and WebLogic domain credentials for the selected engine(s). Click Apply and then Upgrade.
Click OK on the confirmation window to redeploy the selected engine.
Click the job link to view the upgrade job progress.

The summary of job execution is displayed.
When you navigate back to Application Performance Management page, you can see that the selected engines have been upgraded to the latest available version.

When there is no newer version available for upgrdae, you have the option to redeploy the same engine.

To redeploy the selected ADP/JVMD Engines:

From the Setup menu, select Application Performance Management.

The Application Performance Management page appears. The available JVMD Engines and ADP Engines are listed under Application Performance Management Engines.
Click the Redeploy button to redeploy the corresponding engine(s).
Select the engine you want to redeploy.
Specify the Management Agent Host and WebLogic domain credentials for the selected engine(s). Click Apply and then Redeploy. The selected engine is redeployed.
Click the job link to view the job progress.

Figure 13-1 Progress Page for JVMD Engine Redeployment

Figure 13-2 Progress Page for ADP Engine Redeployment

13.6 Working with Pool Threshold Violations

This feature allows you to monitor the Pool Threshold Violations and take corrective actions for the violations. The Threshold Violations and the corresponding Corrective Actions are displayed on the EM Event page. You have the option to setup the required corrective action for each of the threshold violations.

13.6.1 Viewing Threshold Violations

To get started with viewing the threshold violations:

From the Enterprise menu, select Monitoring > Incident Manager.
Select Events without incidents from the Views navigation pane.

The Incident Manager: Events without incidents page appears.
Select any message to view its details.
Select Edit JVM Pool threshold values under Actions within the General tab to setup corrective actions for the selected threshold violation.

The JVM Pool threshold value page appears. You can also navigate to this page through the Configure JVM Pool option in the context menu.

The pool threshold treetable contains information of threshold values and corresponding corrective actions for a Java Virtual Machine Pool target. Values of Threshold and Trigger Samples values can be changed inline on the table.
Click Add to add a corrective action for the corresponding threshold value.

The Add Corrective Action window appears. Corrective actions are of three types: No Action, TraceDump, Heap Dump.
Select the action type you want to add.

Provide the necessary details for the corresponding action type and click Ok.
Click Save to commit the changes made to the threshold violations.

Note:

If a corrective action was triggered by the threshold violation, the trace/heap dump files will be displayed under Event Details. A link to navigate to the trace/heap snapshot page will also be available.
After you have defined the corrective actions, click View Heap Snapshot under Diagnostics within the General tab of the Incident Manager: Events without incidents page.

This action takes you to the snapshot page of the heap dump.

13.6.2 Loading the Heaps

You can trigger a corrective action to perform the heap analysis. The heap created on the snapshot page is not loaded by default.

To get started with loading the heaps:

Navigate to the Java Virtual Machine Pool Home page.
Select Heap Snapshots from the Java Virtual Machine Pool menu.
Select the heap you want to load to the repository.
Click Load.

The Load Heap Snapshot page appears.
Configure the following details before you load the heap dump:
- JVM Diagnostics Agent Host credentials
- Host and path of the selected Heap loader
- Credentails for Heap Loader Host and JVM Diagnostics DB User.
Click Load Heap on the Load Heap Snapshot page.
Click Yes on the confirmation window to load the selected heap.

The selected heap is loaded to the repository.
Navigate to the Availble Heap Snapshots page.
Select the loaded heap from the Heap Snapshots table and click Detail to nalyze the heap dump.

The heap details are displayed.