22 Using Middleware Diagnostics Advisor

This section describes Middleware Diagnostics Advisor (MDA). MDA as a part of Enterprise Manager Cloud Control analyzes the entire stack and provides diagnostic findings by identifying the root cause of a problem.

The Middleware Diagnostics Advisor analyzes the entire stack and provides diagnostic findings by identifying the root cause of a problem. It correlates and analyzes the input and offers advice on how to resolve the problem. For example, it can help you identify a JDBC connection pool that is causing a performance bottleneck.

You can view the diagnostic findings for one or more servers in a WebLogic Domain if the Middleware Diagnostics Advisor has been enabled for the server(s).

This section covers the following:

Diagnosing Performance Issues with Oracle WebLogic Server

Oracle WebLogic Server (WLS) is an application server that provides high performance and scalability. WebLogic Server also simplifies deployment and management, and accelerates time to market with a modern, lightweight development platform.

In order to keep up the performance, and scalability of WLS, it is best to detect violations and provide insight to the cause of the violation, thus enabling faster remedial action. Performance related issues are detected based on the configuration and load of the server. The most common performance issues include slow response times, and application crashes. Using Middleware Diagnostics Advisor (MDA) adds value to the WebLogic Management Pack. To find out more about using MDA for WebLogic Servers, see Diagnosing Performance Issues Using Middleware Diagnostics Advisor.

Diagnosing Performance Issues Using Middleware Diagnostics Advisor

Middleware Diagnostics Advisor or MDA is a diagnostic module integrated within Enterprise Manager (EM) Cloud Control for diagnosing performance issues with middleware targets monitored by Enterprise Manager Cloud Control. Currently, MDA is supported for Oracle WebLogic Server 10g Release 3 (10.3) and higher. MDA monitors JDBC DataSources, and JMS Queues.

MDA enables you to easily identify the underlying states in the application server environment that are the causes for degradation in performance. These underlying states can manifest themselves as degradation in performance such as slow response for request, hung server, slow server, high memory utilization and high Disk I/O, and so on.

MDA analyses the performance of aspects like JMS message consumption time, etc. in a runtime environment. When the performance of the aspect degrades beyond a certain limit, MDA diagnoses the issue to find the underlying cause, and the problems detected by MDA are projected as Diagnostic Findings. However, individual one off issues, which do not affect the overall performance, are not isolated by MDA.

MDA diagnoses performances issues in the following areas and each of these areas is listed as a Finding Type in the Middleware Diagnostics Advisor Configuration page (see Figure 22-1):

JDBC Datasource Wait (JVMD mandatory)

Investigates excessive wait for JDBC Datasource connection.
Delivery of Message is Delayed (JVMD mandatory)

Investigates if the time taken to pick up a message for processing is higher than the specified time. If this is not checked it may lead to messages spending more time in the queue than expected.
Message Processing is slow (JVMD mandatory)

Investigates if the queue processing is slow because of which messages are being processed much slower than they are being received. If this is not checked it may cause the queue to grow and eventually lead to out-of-memory errors.

Functioning of Middleware Diagnostics Advisor

Middleware Diagnostics Advisor functions in the following way:

The MDA engine starts when the Oracle Management Service (OMS) is started.

The MDA engine is responsible for executing MDA analysis tasks.
Run the following command in the Enterprise Manager command line interface to enable MDA.

emcli update_mda_properties -props="MDA_AUTO_ENABLE:1"

Note:

MDA is not enabled in Enterprise Manager Cloud Control by default. This step is necessary to enable MDA.

If any targets have been added to MDA before performing this step, they can be explicitly enabled by executing the command:

emcli enable_mda_finding_types_for_targets -finding_types="finding_type_name"-targets="target_name:target_type"
The Enterprise Manager collected metric repository data and the JVMD agent collected repository data is accessed by MDA.
The MDA engine runs the analysis for each finding type as scheduled on all the applicable targets. For example, once an hour for JDBC Datasource Wait finding.

Note:

When a middleware target is in Black-out or Notification Black-out state, the data will not be processed for that target. Therefore, all scheduled analysis runs for such targets are skipped.
During the analysis, rules are applied to see if there are issues and the results are stored in the repository.
The findings are displayed on the Middleware Diagnostics Advisor page.

Prerequisites for Configuring Middleware Diagnostics Advisor

Before you begin using Middleware Diagnostics Advisor (MDA) for diagnosing performance issues, ensure the following prerequisites are met:

The WebLogic Server is discovered as a target in Enterprise Manager.
JVMD Manager is configured, and the JVMD Agent is deployed on the target server.
It is recommended to force the configuration collection for the target.

To force the configuration collection for the target, perform the following steps:
1. From the target menu, select Configuration and then select Last Collected.
2. Click Refresh.

Configuring Middleware Diagnostics Advisor

Middleware Diagnostic Advisor (MDA) has an auto-enable job that runs every hour to check for new targets. However, these jobs will not be enabled for MDA analysis by default unless you have run the command emcli update_mda_properties -props="MDA_AUTO_ENABLE:1". After the command has been executed successfully, and if the prerequisites (see Prerequisites for Configuring Middleware Diagnostics Advisor) are met, the target is automatically enabled for MDA analysis in the next run of the job. However, to immediately enable a newly discovered middleware target for MDA analysis follow the steps below:

On the Enterprise Manager home page, from the Targets menu, select Middleware.
Click the newly discovered target domain link from the Middleware targets list.
On the target domain home page, from the target menu select Diagnostics, and then select Middleware Diagnostics Advisor Configuration.
Click Health Check in the Registered Finding Types section.

A health check is performed, wherein in addition to performing a health check of MDA, any new middleware targets added in Enterprise Manager will also be discovered by MDA and enabled for all the applicable finding types.

Enabling Middleware Diagnostics Advisor for a Target

After the MDA_AUTO_ENABLE property has been set to "1" (enable) using the EM CLI verb emcli update_mda_properties -props="MDA_AUTO_ENABLE:1", any new WLS targets discovered in EM are enabled by default. However, if you do not want to enable every WLS server managed by the EM instance, you can set MDA_AUTO_ENABLE to "0" (disable), and individually enable specific targets either from UI (by following the steps below) or from EM CLI with the command emcli enable_mda_finding_types_for_targets -finding_types="finding_type_name"-targets="target_name:target_type".

To manually enable or disable MDA, follow these steps:

On the Enterprise Manager home page, from the Targets menu, select Middleware.
From the Middleware Features menu select Middleware Diagnostics Advisor Configuration.
On the Middleware Diagnostics Advisor Configuration page, from the Registered Finding Types section select any one of the Finding Type.
On the Targets section, select the target that you want to enable MDA for, and click Enable. See Figure 22-1.

Note:

Unless a target is discovered by MDA it will not be displayed in the targets list. To discover a target in MDA and enable it, see Configuring Middleware Diagnostics Advisor.

Figure 22-1 Middleware Diagnostics Advisor Configuration Page

Setting Up Middleware Diagnostics Advisor (MDA)

MDA can be set according to your preference. To set up MDA, follow these steps:

From the Setup menu, select Middleware Management, and then select Middleware Diagnostics Advisor.
On the Middleware Diagnostics Advisor Setup page, you can do the following:
- In the Analysis Job Configuration section, select Skip Analysis Runs for all MDA-Enabled Servers, if you want to skip all MDA analysis jobs.
- In the Purge Policy section you may choose to enable purging by selecting the Purge Data Older Than check box.
  
  To set your preferred frequency for purging data, enter the preferred number of days for which the data should be retained in the text box. There is an MDA job that runs every 24 hours which purges data from the repository.
  
  Note:
  
  This is a global setting and will be applied to all targets, and all users.
- In the Finding Threshold Configuration section, set the threshold or limit (in percentage) beyond which violations should result in a finding, by adjusting the Violations Percentage. The default value is 10%.
  
  Note:
  
  This setting is applicable only to JMS and JDBC findings.
- To set the wait time period (in minutes) beyond which any messages picked up will be considered as violations, adjust the JMS Wait Time. The default value is 1 minute.
  
  Note:
  
  This setting is applicable only to JMS wait time findings.
Click Apply.

Figure 22-2 shows the Middleware Diagnostics Advisor Setup page.

Figure 22-2 Middleware Diagnostics Advisor Setup Page

Note:

Additionally, you can also use EM CLI commands to configure MDA. For details, see Enterprise Manager Command Line Interface Guide.

Limiting the Scope of Middleware Diagnostics Advisor

You can limit the scope of Middleware Diagnostic Advisor (MDA) by disabling MDA on some or all of the targets. If disabled, no analysis runs will be scheduled for the disabled targets. The targets can also be enabled when required.

To disable a target for MDA analysis follow the steps below:

On the Enterprise Manager home page, from the Targets menu, select Middleware.
From the Middleware Features menu select Middleware Diagnostics Advisor Configuration.
On the Middleware Diagnostics Advisor Configuration page, from the Registered Finding Types section select any one of the Finding Type.
In the Targets section, select the target that you want to disable MDA for, and click Disable.

To enable a target again for MDA analysis, see Enabling Middleware Diagnostics Advisor for a Target.

For more information on limiting the scope of MDA, see the Oracle Enterprise Manager Command Line Interface guide.

Using Middleware Diagnostics Advisor to View and Diagnose Performance Issues

To use MDA to view and diagnose performance issues, follow these steps:

Navigate to the target home page.

Note:

The target can be an Oracle WebLogic Domain, Oracle WebLogic Cluster or an Oracle WebLogic Server. Go to an Oracle WebLogic Server home page to see the findings for only the server. Go to a domain or a cluster home page to see the findings for all the targets under them.
From the target specific menu, select Diagnostics, then select Middleware Diagnostics Advisor.
On the Middleware Diagnostics Advisor page, view the Timeline section for findings marked against the time at which the finding was recorded.

See Figure 22-3 for a sample of the Middleware Diagnostics Advisor page.

Note:

To view historic data adjust the settings in the View Data drop-down box.

Figure 22-3 Middleware Diagnostics Advisor Page
Click on the Quick View icon next to the finding link to have a quick look at the details of the finding.
To know more about the findings, if any, click on the finding link in the Timeline section.
The Middleware Diagnostics Advisor (MDA) Finding Details page provides more information on the finding, and the recommended workaround. The following details are provided:
- Finding
  
  The diagnostic finding for the Middleware domain.
```
For example: High number of messages reprocessed due to Transaction timeout.
```
  - Description - Target Type, Target, and Analysis Time
    
    The description and details related to the finding.
- Analysis
  - Rule
    
    The rule/s that are applied on the collected values to determine if there is a finding.
  - Execution Data
    
    The values collected during analysis.
- Recommendations
  - Action
    
    A solution or tip for the problem found.
  - Rationale
    
    The rationale applied for the suggested action.
- Charts
  
  The Charts section contains graphs pertaining to the finding.

Running an Unscheduled Middleware Diagnostics Advisor Analysis on a Target

The MDA engine runs analysis for each finding type as scheduled on all the applicable targets. You can also run an unscheduled on-demand MDA analysis on a target. Follow the steps below to do so:

On the Enterprise Manager home page, from the Targets menu, select Middleware.
Click the target domain link from the Middleware targets list.
On the target domain home page, from the target menu select Diagnostics, and then select Middleware Diagnostics Advisor Configuration.
On the Middleware Diagnostics Advisor Configuration page select the finding type from the Registered Finding Types section to list the applicable targets in the Targets section.
Select any one of the targets to view the last 10 analyses run on that target in the Analysis Runs section.

Note:

Ensure that the Show Analysis Runs with Findings Only check box is disabled to see all the ten analysis runs.
Click Run MDA Analysis to run an MDA analysis now.

Note:

MDA runs the analysis even if the target is in blackout or notification blackout state.

Troubleshooting Issues Related to Middleware Diagnostics Advisor

To troubleshoot issues related to MDA, follow the steps mentioned in the following table.

Table 22-1 Troubleshooting Tips for Middleware Diagnostics Advisor

Tip	How to
Ensure that MDA is enabled for the specified finding type on the target.	On the target domain home page, click the target menu. Select Diagnostics menu, and then select Middleware Diagnostics Advisor Configuration. Select the finding type. Select the target from the Targets section. Note: If the target is not visible in the Targets section, click Health Check in the Registered Finding Types section. Verify whether the status column for the target displays “Enabled" for the target. If the status is disabled, enable MDA for the specified finding type for the target. Refer the Enterprise Manager Command Line Interface Guide for more options.
Ensure that MDA auto-enable job is running properly	From the Enterprise menu, select Job and then Activity. In the Available Criteria section under Name, enter MDA_ADMIN_AUTO_ENABLE% in the text box and click the search icon. In the Available Criteria section under Target Type, select Targetless. In the Available Criteria section under Status, select Problems. View all the jobs displayed in the table. If there are no skipped or failed jobs, it indicates that the auto-enable job is running as expected.
Ensure that MDA analysis jobs are running properly.	From the Enterprise menu, select Job and then Activity. In the Available Criteria section under Name, enter MDA_ANALYSIS_RUN% in the text box and click the search icon. Note: There is one MDA_ANALYSIS_RUN job per Finding Type. By selecting MDA_ANALYSIS_RUN%, the jobs for all the Finding Types are displayed. In the Available Criteria section under Target Type, select Targetless. In the Available Criteria section under Status, select Problems. View all the jobs displayed in the table. If there are no skipped or failed jobs, it indicates that the analysis job is running as expected.
Ensure that the MDA engine is running normally.	Refer the Enterprise Manager Command Line Interface Guide for the command to check the status of MDA engine.