Diagnostic Tools

5 Diagnostic Tools

This chapter describes the diagnostic and monitoring tools you can use to help you develop and diagnose Fusion web applications problems.

This chapter includes the following sections:

Introduction to Monitoring and Diagnostic Tools

Oracle Fusion Middleware and Oracle ADF provide tools you can use to monitor and diagnose issues with your application.

The tools and procedures include:

Click History
Loggers
Diagnostics Framework
DMS metrics

Click History

Click History is a Fusion Middleware diagnostics feature that continuously reports the physical clicks on UI elements such as buttons and tabs, scrolling events, and graph draws. Use this feature to diagnose the cause of the incidents.

Click History captures ADF Faces events generated by user activities on ADF Faces components (for instance, clicking an ADF command button from a web browser). The data is continuously captured and stored in a circular buffer in memory. The buffer size is adjustable. When a diagnostic incident occurs, the data in the buffer is automatically written to disk as part of the diagnostic data. For more information about diagnostic incidents, see the Diagnosing Problems chapter of Administering Oracle Fusion Middleware.You do not need to configure Click History to capture the "click" events, it is automatically enabled by default. You can use Click History data to help diagnose problems. It is particularly useful to help determine the last user action that had caused the failure or problem.

The Click History record provides information about the UI component, the view, the region, the user, the Diagnostic Session Id (DSID), and other key application data. Click History captures its data using ADF EUM (End User Monitoring) services It has a circular buffer that is optimized specifically for EUM data. The data is packed in a single contiguous memory segment in the Java heap. It has minimal impact to Java Garbage Collection and its memory usage is strictly controlled. The default buffer size of 860 KB can hold approximately1,500 "click" events, which should be sufficient for incident analysis.

How to Obtain Click History

Click History is delivered as part of the default JRF Template. When you install Fusion Oracle Middleware or Oracle ADF, Click History will be installed as part of the Fusion Middleware framework.

How to Enable Click History for an Application

Click History is installed as part of ADF. However, the actual logging of Click History messages are performed by a separate thread independent of ADF, therefore, Click History imposes insignificant overhead for ADF.

Click History is enabled on an application by application basis. You enable an application for Click History by adding entries to its weblogic.xml and web.xml configuration files. For instructions, see Enabling the Application for RUEI and Click History in Developing Fusion Web Applications with Oracle Application Development Framework.

How to Enable and Disable Click History in WebLogic Server

After you have enabled Click History for an application, you can enable it for a specific WebLogic server instance by setting the oracle.clickhistory.EUM logger to the NOTIFICATION level.

You can a WLST setLogLevel command or the Enterprise Manager Cloud Control Support Workbench to set the logging level.

You do not need to restart the server for this to take effect.

To enable Click History using WLST, run the following command:

setLogLevel(logger='oracle.clickhistory.EUM', target='server_name', level='INFO', persist='1')

To disable Click History using WLST, run the following command:

setLogLevel(logger='oracle.clickhistory.EUM', target='server_name', level='OFF', persist='1')

For information on how to use Fusion Middleware Cloud Control Support Workbench to set the log level to enable Click History, see the Managing Log Files and Diagnostic Data section in Administering Oracle Fusion Middleware.

How to Adjust Click History Capacity

The Click History buffer size is defined in the logging.xml file. The default value is 860 KB or 880640 bytes. You can use the WLST configureLogHandler command to configure the buffer size (in bytes). You will need to restart the server for the change to take effect.

For example:

configureLogHandler(name="apps-clickhistory-handler", propertyName="bufferSize", propertyValue="100000000", target="your_server_name")

You can find the existing buffer size using the WLST listLogHandlers command.

For example:

listLogHandlers(name="apps-clickhistory-handler")

The command returns a list of information including bufferSize in bytes.

For example:

bufferSize=880640

How to Dump Click History Data Explicitly

When an incident occurs, Click History automatically dumps its data into a file. However, you can also access Click History data at any time using the WLST executeDump command.

To dump Click History data explicitly:

executeDump(name="odl.quicktrace", args={"handlerName" : "apps-clickhistory-handler"}, server="CRMCommonServer_1")

The Click History output file is a QuickTrace dump file. It has the same format as the ODL log file. This format can be parsed using the ODL LogQuery API as well as using the Enterprise Manager Cloud Control Support Workbench log query feature.

The Click History dump file format convention is:

odl_quicktrace<unique ID>_i<incident ID>.app-clickhistory-handler.dmp

The suffix is the QuickTrace handler name defined in the server's logging.xml.

For example:

odl_quicktrace502_i273.apps-clickhistory-handler.dmp

A sample dump file output is shown in the following example.

Sample Click History Dump
[2013-10-08T14:56:12.664-07:00] [AdminServer] [NOTIFICATION] [SOC] 
[oracle.clickhistory.EUM] [tid: 34] [userId: CASH_MANAGER] 
[ecid:a3da34f168f6c0de:-7bd21c26:14199e687e9:-8000-00000000000033d3,0]
[CH_RTY: oracle.adf.rich.Combobox ListOfValues]
[CH_CST: 1381269371002] [DSID: 0000K6PzLBrFS8G5Iz9Dif1IL7HS00000A]
[CH_CID: pt1:USma:0:MAnt2:1:pt1:Manua1:0:mrpanel1:mrbankAccountNameId]
[CH_VID: /ManualReconPage] [APP: PayablesApp]
[CH_CMP: oracle.adf.RichInputComboboxListOfValues]
[CH_CNM: Bank Account]
[CH_TYP: lovInternal
[CH_WID: 114iztpu6s_6]
[CH_RVD: /ManualReconLocalAreaFlow/ManualReconLocalAreaPF]
[CH_EID: a3da34f168f6c0de:-7bd21c26:14199e687e9:-8000-00000000000033d3]
[CH_FAM: org.apache.myfaces.trinidad.Input]
 
[2013-10-08T14:56:12.664-07:00] [AdminServer] [NOTIFICATION] [EOC]
[oracle.clickhistory.EUM] [tid: 34] [userId: CASH_MANAGER] 
[ecid: a3da34f168f6c0de:-7bd21c26:14199e687e9:-8000-000000000000339c,0]
[CH_CET: 1381269364245] 
[CH_CST: 1381269363617] 
[APP: PayablesApp] 
[DSID: 0000K6PzLBrFS8G5Iz9Dif1IL7HS00000A] 
[CH_TTT: 628] 
[CH_EID: a3da34f168f6c0de:-7bd21c26:14199e687e9:-8000-000000000000339c]

How to Interpret Click History Records

The Click History dump file is an ordinary ASCII file in the format of a log file. You can use any ASCII viewer to view its contents, but it would be best to use log file viewing tools to take advantage of the search and formatting functions they offer. The Click History messages written to the log file are logged by a separate thread that is independent of ADF.

The log file is located at:

${domain.home}/servers/${weblogic.Name}/logs/${weblogic.Name}-clickhistory.log

The Click History log data is automatically captured as part of a diagnostic incident. If you have an incident that includes Click History data, you can use the incident's readme.txt file to find the ECID (ExecutionContext ID), DSID (Diagnostics Session ID), user ID, and thread ID of the incident.You can also query Click History records based on these diagnostics flags.

The following information are logged:

Region/View Id
Component Id
Server time
Network time
Browser render time

By default, a pair of messages are logged for each click. The message Id for the first message is SOC (Start of Click). The second message EOC (End of Click) will not be logged until the subsequent click occurs. Timing details, such as TTT (Total Time). can be found only in the second message.

Table 5-1 lists the Click History attributes and their corresponding ADF UserActivityInfo fields.

Table 5-1 Click History Attribute Mapping

Click History Attributes	ADF UserActivityInfo fields
CH_CST	`userActivityInfo.getClientStartTime()`
CH_CET	`userActivityInfo.getClientEndTime()`
CH_VID	`userActivityInfo.getViewId()`
CH_WID	`userActivityInfo.getWindowId()`
CH_CID	`eventInfo.getComponentClientId()`
CH_CNM	`eventInfo.getComponentDisplayName()`
CH_FAM	`eventInfo.getComponentFamily()`
CH_CMP	`eventInfo.getComponentType()`
CH_RVD	`eventInfo.getRegionViewId()`
CH_RNM	`eventInfo.getRegionViewName()`
CH_RTY	`eventInfo.getRendererType()`
CH_TYP	`eventInfo.getType()`
CH_EID	`userActivityInfo.getContextId()`
CH_PEI	`userActivityInfo.getParentContextId()`
CH_TTT	Total Time = CH_CET - CH_CST
CH_RRT	`userActivityInfo.getResponseReceivedTime()`
CH_PRT	`userActivityInfo.getParentResponseReceivedTime()`
CH_RNT	Rendering Time = CH_CET - CH_RRT

Some of the viewing tools you can use to show the log file contents are:

Enterprise Manager Cloud Control Support Workbench

You can use Enterprise Manager Cloud Control Support Workbench to view and sort the log information.
WLST command

You can also use WLST command displayLog to view the record
Any text editor such as vi.

For example, search all Click History records with a matching DSID in order to return a list of the recent "click" events of a HTTP session.

For example:

[2013-01-21T20:50:16.403-08:00] [ProcurementServer_1] [NOTIFICATION] []
[oracle.clickhistory.EUM] [tid: 44] [userId: FUSION]
[ecid: 004ox_GMXwy6aMGpIws1yf00018d000gvb,0:2:8:2:2]
[CLICK_COMPTYPE: oracle.adf.RichCommandButton] [APP: ProcurementApp#V2.0]
[DSID: 0000JlW_fz5Fw0HpIsO5yf1GzEoI000Dh3]
[CLICK_VIEWID: /adfp-portlet-bridge-container/container-view]
[CLICK_REGIONVIEWID: [/BrowseCategoryFlow/BrowseCategory,
 /BrowseCategoryMainAreaFlow/BrowseCategoryMainArea,
 /FSGenericTaskFlow/DynamicRegionContainer]] 
[CLICK_STARTTIME: 1358830216656]
[CLICK_RENDERTYPE: oracle.adf.rich.Button]
[CLICK_TYPE: action]
[CLICK_COMPCLIENTID:_jpfcpncuivr__ns1588903364_j_id__
 ctru0:fragmentRegionStatic:0:dynamicRegion1:0:pt1:Brows1:0:
 ManageCategoryHierarchyPanel:ATT1:deleteWarningYES]
[CLICK_WINDOWID: k4coiwdvi_1]
[CLICK_REGIONVIEWNAME: [jpfcpncuivr_ns1588903364_j_id
 _ctru0:fragmentRegionStatic:0:dynamicRegion1:0:pt1:Brows1, _jpfcpncuivr
 _ns1588903364_j_id_ctru0:fragmentRegionStatic:0:dynamicRegion1, _jpfcpncuivr
 _ns1588903364_j_id__ctru0:fragmentRegionStatic]]
[CLICK_COMPDISPLAYNAME: Yes]

How to Get Click History DMS Metrics

You can determine Click History's buffer utilization and capacity by inspecting the DMS metrics. You can execute a WLST command to display the metrics. If multiple QuickTrace handlers are configured, look for the handler with this name: apps-clickhistory-handler.

For example:

wls:/base_domain/serverConfig>  displayMetricTables('QuickTraceInfo',
 servers=['your_server'])

QuickTraceInfo

Host:   xyz.com
Name:   apps-clickhistory-handler
Parent: /oracle/odl/quicktrace
Process:        your_server:7009
ServerName:     your_server
bufferElapsed.value:    134719    seconds
bufferRecordCount.value:        1579     records
bufferUsedPercentage.value:     100       percent
oldestTimestamp.value:  2012-06-26T18:38:48.261-0700    time

Viewing REST Metrics

Oracle ADF provides REST metrics that are used to analyze the system performance and monitor the status of the ADF application. Use DMS Spy Servlet to view the REST metrics.

ADF middle tier business components may be exposed as REST endpoints by building an ADF REST Application, see How to Create ADF REST Resources from View Object. When the service is deployed in a large scale clustered environment, the developers and system administrators need access to metrics that help analyze system performance or monitor system status.

Oracle Dynamic Monitoring Service calls are made from the REST servlet whenever it services a REST request, recording resource usage incurred during processing. For more information about the Oracle Dynamic Monitoring Service (DMS), see Using the Oracle Dynamic Monitoring Service.

The DMS Spy Servlet, which is optionally deployed into the container, allows you to view the state of the DMS metrics. The following is an example of an HTTP REST request from the DMS Spy Servlet: http://10.242.60.105:8888/RESTMar2016-RESTWebService-context- root/rest/v0/Emp/7654. For more information on how to send an HTTP REST request in JDeveloper or using the command line tool cURL, see Testing the ADF REST Resources Using Integrated WebLogic Server.

ADF REST web service returns a structured REST response. For more information on HTTP method support for ADF REST, see ADF REST Framework Reference.

At runtime, metrics are stored in memory and recorded to an ADF-REST deployment dashboard. For more information on how to get the ADF BC resource describe from which the specific HTTP REST request is made, see Retrieving the ADF REST Resource Describe.

Example 5-1 Sample Structured REST Response

{
  "Empno" : 7654,
  "Ename" : "MARTIN",
  "Job" : "SALESMAN",
  "Mgr" : 7098,
  "Hiredate" : "1981-09-28T00:00:00-07:00",
  "Sal" : 1250,
  "Com" : 1400,
  "Deptno" : 30,
  "links" : [ {
    "rel" : "self",
    "href" : "http://10.242.60.105:8888/RESTMar2016-RESTWebService-
context- root/rest/v0/Emp/7654",
    "name" : "Emp",
    "kind" : "item"
  }, {
    "rel" : "canonical",
    "href" : "http://10.242.60.105:8888/RESTMar2016-RESTWebService-
context- root/rest/v0/Emp/7654",
    "name" : "Emp",
    "kind" : "item"
  } ]
}

Types of REST Metrics

The REST web service has several types of metrics. The metrics help you to track performance information that developers and system administrators use to help analyze system performance or monitor system status.

Different HTTP REST requests result in different metrics displayed in ADF-REST dashboard.

Table 5-2 describes metrics that are associated with the ADF-REST application.

Table 5-2 DMS ADF-REST Metrics

Metric	Description
Name	Specifies the REST resource name.
Host	Specifies the host name servicing the request.
Process	Specifies the node name and port.
Execute	Specifies execution details.
HTTP Method	Specifies such HTTP methods as get, put, patch, etc.
InputByteCount	Specifies inbound payload size.
OutputByteCount	Specifies output payload size.
Request Type	Specifies HTTP request type.
Response Type	Specifies HTTP response type.
resultCode	Specifies HTTP result code.
Url	Specifies the URL of the REST request.
Username	Specifies the name of a user.

Figure 5-1 ADF_REST View of the DMS Spy Servlet

Description of "Figure 5-1 ADF_REST View of the DMS Spy Servlet"

Getting REST Metrics

ADF-REST metrics are shown in a table.

To view the contents of the ADF-REST metric table in the DMS Spy servlet, click the table name in the left column.

For more information on different types of REST metrics and their description see Types of REST Metrics.

ADF Logs

Oracle Fusion Middleware provides log files that record events such as error messages, warning messages, server information, and HTTP requests. Use the Fusion Middleware Control or the WLST commands to view, download, and configure settings for these log files.

Most Fusion Middleware log files are in the Oracle Diagnostic Logging (ODL) format. You can use Fusion Middleware Control or WLST commands to search, view, and download log files. You can also use these tools to perform various tasks related to logging, such as configuring settings, changing file locations, setting log levels, setting file format, and configuring tracing.

For more information about Fusion Middleware log files, see Managing Log Files and Diagnostic Data of the Administering Oracle Fusion Middleware.

JDeveloper also provides an ADF Logger and a log analyzer that you can use for monitoring and diagnostic purposes. JDeveloper generates the log file in the Oracle Diagnostic Logging (ODL) format that is used throughout Fusion Middleware. In JDeveloper, you use the Oracle Diagnostic Logging Configuration editor to set the log levels and other configuration parameters. Configuration information is stored in the logging.xml file.

After the log files have been created, you can use the Oracle Diagnostic Log Analyzer within JDeveloper to view the log entries. You can search and filter the log entries based on log levels, time frames, message content, and other parameters.

For more information about the ADF Loggers, see Using the ADF Logger in Developing Fusion Web Applications with Oracle Application Development Framework.

Diagnosing Problems using the Diagnostic Framework

Oracle Fusion Middleware provides a Diagnostic Framework to collect and manage information about a problem so that you can resolve it or send it to Oracle Support for resolution.

This Diagnostic Framework can help you to detect, diagnose, and resolve problems with your application. When a critical error occurs, the Diagnostic Framework immediately captures diagnostic data and associates the data and error with an incident number. Using this number, you can retrieve the data for analysis from the Automatic Diagnostic Repository (ADR).

Oracle ADF provides an ADFConfig dump which will execute when an INCIDENT_ERROR message is logged. You can also add code to invoke the dump in the application exception handlers. The following example shows a sample code you can add to your exception handler to invoke the ADFConfig dump.

IllegalArgumentException e = new IllegalArgumentException("test exception");
LoggerFactory.getFrameworkLogger().log(ODLLevel.INCIDENT_ERROR,
 "Test error message", e);

For more information about the Diagnostic Framework, see Diagnosing Problems of the Administering Oracle Fusion Middleware.

Finding ADF Runtime Version Information

Oracle ADF installation requires the ADF runtime JARs that must be available as a library in the target application server. Use the ADF runtime library version for diagnosing ADF application errors.

Knowing the ADF runtime JARs version numbers can be important information for debugging and diagnosing application problems. This information may be requested by Oracle Support.

You can access My Oracle Support at https://support.oracle.com.

You can find which versions of the ADF Runtime JARs are in effect using several methods:

Retrieving a CSV file that is generated when the application starts.
Using shell scripts in offline mode.
Using WLST commands.
Using Fusion Middleware Control.
Retrieving the dump files when an incident occurs.

How to Find Version Information by Starting the Application

When a deployed application is started, the runtime JARs version information is dumped into a CSV file located at:

<DomainName>/servers/<ServerName>/logs/<MyApp>-Versions.csv

Note the following:

oracle.adf.share.diagnostics.versions logger controls generation of runtime JARs versions at application startup. JARs versions are generated when the logger is set to level="FINEST".
On new WebLogic domains, oracle.adf.share.diagnostics.versions logger will be available in the logging.xml.
On existing domains, oracle.adf.share.diagnostics.versions logger will not be present in the logging.xml. Use Enterprise Manager Fusion Middleware Control or the WLST command to set the logger level.

How to Find Version Information by Using Shell Scripts in Offline Mode

In offline mode, you can use the printJarVersions.sh shell script to print the version information or to redirect the information to a file.

In the %MW_HOME%/oracle_common/common/bin directory, run the script:

bin$./printJarVersions.sh

You can redirect the output to a file:

bin$./printJarVersions.sh  > /home/trdsouza/export-JarVersions.csv

The version information have a format similar to the following (trimmed):

Jar Path,Oracle-Version,Oracle-Label,Oracle-Builder,Oracle-BuildTimestamp,
Specification-Version,Implementation-Version/ade/userYada/oracle/oracle_
common/modules/oracle.adf.model_12.2.1/adfm.jar,12.2.1.0.40.66.68,
JDEVADF_12.2.1.0.0_GENERIC_130608.2330.6668,user=userY
host=abc123456,2013-10-02 04:55:45 -0700,,

How to Find Version Information by Using the WLST Scripting Console

In offline mode, you can use the WLST exportJarVersions command to find the version information:

exportJarVersions('/home/userY/export-JarVersions.csv')

In online mode, you can use the WLST exportApplicationJarVersions command to find the version information:

exportApplicationJarVersions('MyApp','/home/userY/export-EarCrmCommon-Versions.csv')

If you want to find a specific JAR version, you can use the WLST exportApplicationSelectedJarVersions command:

exportApplicationSelectedJarVersions('MyApp','/home/userY/export-EarCrmCommon-Versions.csv', $ORACLE_HOME$/modules/oracle.adf.model_12.2.1/adfm.jar;$ORACLE_HOME$/modules/oracle.adf.view_12.2.1/adf-richclient-impl-12.jar')

How to Find Version Information Using Fusion Middleware Control

You can use Enterprise Manager Fusion Middleware Control to monitor Oracle products that are built using ADF. You can also use this tool to find the version information of runtime JAR files.

For instructions, see Finding Version Information of ADF Runtime JARs.

How to Find Version Information by Inspecting Diagnostic Dump Files

When an INCIDENT_ERROR occurs when the application is running in a WebLogic Domain, the Oracle Diagnostic Framework generates dump files in the WebLogic Domain directories. The location of these files are in:

<DomainName>/servers/<ServerName>/adr/diag/ofm/<DomainName>/<ServerName>/incident/

One of the generated dump files contains the application JAR version information. It is called adf_DiagnosticsJarsVersionDumpN.txt where N is a numerical value.

For example, <myDomainName>/servers/<myServerName>/adr/diag/ofm/<myDomainName>/<myServerName>/incident/adf_DiagnosticsJarsVersionDump45_i5.txt.

You can examine this file to view the version information. You can also upload this file to My Oracle Support for further analysis.

You can trigger a diagnostic dump from code by logging an INCIDENT_ERROR message.

Troubleshooting Oracle ADF for High Availability

You can deploy a Fusion Web application on high availability environments. After the deployment, you can monitor and diagnose the application deployment issues and the Oracle ADF replication and failover issues.

Troubleshooting Fusion web applications that are deployed to high availability environments require additional concerns. For information about deployment issues in high availability environments, see Troubleshooting High Availability Issues After Deployment in Developing Fusion Web Applications with Oracle Application Development Framework. For information about Oracle ADF replication issues in high availability environments, see Troubleshooting Oracle ADF Replication and Failover Issues in Developing Fusion Web Applications with Oracle Application Development Framework.