Skip Headers

Oracle® Application Server 10g Performance Guide
10g (9.0.4)
Part No. B10379-01
  Go To Documentation Library
Home
Go To Product List
Solution Area
Go To Table Of Contents
Contents
Go To Index
Index

Previous Next  

9 Instrumenting Applications With DMS

The Oracle Dynamic Monitoring Service (DMS) enables application developers, support analysts, system administrators, and others to measure application specific performance information. This chapter describes DMS and shows a sample application that demonstrates how to instrument Oracle Application Server Java applications using DMS.


Note:

Oracle Application Server provides a number of built-in metrics. Using DMS to instrument applications adds new metrics to the set of built-in metrics.

This chapter covers the following topics:

Introducing DMS Performance Metrics

The Dynamic Monitoring Service (DMS) API allows you to add performance instrumentation to Oracle Application Server applications. During runtime DMS collects performance information, called DMS metrics, that developers, system administrators, and support analysts use to help analyze system performance or monitor system status.

This section covers the following topics:


Note:

Oracle Application Server components, including OC4J, provide a number of predefined metrics. For a listing of the predefined metrics see Appendix A, " Performance Metrics".

Instrumenting Applications With DMS Metrics

DMS Instrumentation refers to the process of inserting DMS calls into application code. Using the DMS API is a simple and efficient way to enable your application to measure, collect, and save performance information.

To create DMS metrics developers add calls that notify DMS when events occur, when important intervals begin and end, or when pre-computed values change their state. At runtime, DMS stores metrics in memory and allows you to save or view the metrics.

Oracle Application Server includes built-in DMS metrics. By adding DMS calls to your applications, you expand the set of built-in metrics. When you instrument your applications with DMS calls, you use the same API that the built-in metrics use. In addition, to save and display your metrics, use the same monitoring tools that you use with built-in metrics.

Monitoring DMS Metrics

Monitoring DMS metrics refers to the process of retrieving performance metrics. When an application runs, DMS stores metrics in memory and allows you to show metrics on the console, save metrics to a file, or to view metrics using a web browser.

Oracle Application Server provides several runtime tools for viewing and saving DMS metrics, including dmstool, and the Spy and AggreSpy Servlets.

Example 9-1 shows a set of metrics output using dmstool.

Example 9-1 Set of Sample dmsDemo Metrics Using dmstool

/dmsDemo [type=n/a]
  /dmsDemo/BasicBinomial [type=MathSeries]
   computeSeries.active:        0       threads
   computeSeries.avg:         220.333   msecs
   computeSeries.completed:     3       ops
   computeSeries.maxActive:     1       threads
   computeSeries.maxTime:     435       msecs
   computeSeries.minTime:      35       msecs
   computeSeries.time:  661     msecs
   lastComputed.value:  100891344545564193334812497256
   loops.count: 147     ops

Understanding DMS Terminology (Nouns and Sensors)

This section introduces the terminology you need to understand to use DMS. Figure 9-1 illustrates the organization of a set of DMS metrics corresponding to the metrics in the demo application described in this chapter and the metrics shown in Example 9-1.

This section covers the following topics:

Figure 9-1 Organization of Sample Metrics From dmsDemo Application

dmsDemo sample metric organization, includes sensors
Description of the illustration asper020.gif

DMS Metrics

DMS Metrics track performance information that developers, system administrators, and support analysts use to help analyze system performance or monitor system status.

DMS Sensors

DMS Sensors measure performance data and allow DMS to define and collect a set of metrics. Certain metrics are always included with a Sensor and other metrics are optionally included with a Sensor.

DMS PhaseEvent Sensors

A DMS PhaseEvent Sensor measures the time spent in a specific section of code that has a beginning and an end. Use a PhaseEvent Sensor to track time in a method or in a block of code.

DMS can calculate optional metrics associated with a PhaseEvent, including: the average, maximum, and minimum time that is spent in the PhaseEvent.

Table 9-1 describes metrics available with a PhaseEvent.

Table 9-1 DMS PhaseEvent Sensor Metrics

Metric Description
sensor_name.time Specifies the total time spent in the phase sensor_name.

Default: time is a default PhaseEvent Sensor.

sensor_name.completed Specifies the number of times the phase sensor_name, has completed since the process was started.
sensor_name.minTime Specifies the minimum time spent in the phase sensor_name, for all the times the phase completed.
sensor_name.maxTime Specifies the maximum time spent in the phase sensor_name, over all the times the sensor_name phase completed.
sensor_name.avg Specifies the average time spent in the phase sensor_name, computed as the (time total)/(number of times the phase completed).
sensor_name.active Specifies the number of threads in the phase sensor_name, at the time the DMS statistics are gathered (the value may change over time).
sensor_name.maxActive Specifies the maximum number of concurrent threads in the phase sensor_name, since the process started.

DMS Event Sensors

A DMS Event Sensor is a Sensor that counts system events. Use a DMS Event Sensor to track system events that have a short duration, or where the duration of the event is not of interest but the occurrence of the event is of interest.

Table 9-2 describes the metric that is associated with an Event Sensor.

Table 9-2 DMS Event Sensor Metrics

Metric Description
sensor_name.count Specifies the number of times the event has occurred since the process started, where sensor_name is the name of the Event Sensor as specified in the DMS instrumentation API.

Default: count is the default metric for an Event Sensor. No other metrics are available for an Event Sensor.


DMS State Sensors

A DMS State Sensor is a Sensor to which you assign a precomputed value. The value can be of a variety of types, including: int, long, double, or Object. Use a State Sensor when you want to track system status information, or when you need a performance metric that is not associated with an event. For example, use State Sensors to represent queue lengths, pool sizes, buffer sizes, or host names.

Table 9-3 describes the State Sensor metrics. The minValue and maxValue optional metrics may only make sense when numeric values are associated with the State Sensor.

Table 9-3 DMS State Sensor Metrics

Metric Description
sensor_name.value Specifies the metric value for sensor_name, using the type assigned when sensor_name is created.

Default: value is the default State metric.

sensor_name.count Specifies the number of times sensor_name is updated.
sensor_name.minValue Specifies the minimum value for sensor_name since startup.
sensor_name.maxValue Specifies the maximum value this sensor_name since startup.

DMS Nouns

DMS Nouns (Nouns) organize performance data. Each Sensor, with its associated metrics is organized in a hierarchy according to Nouns. Nouns allow you to organize DMS metrics in a manner comparable to a directory structure in a file system. For example, Nouns can represent classes, methods, objects, queues, connections, applications, databases, or other objects that you want to measure.

A Noun type is a name that reflects the set of metrics being collected. For example, in the built-in metrics the Noun type oc4j_servlet represents the metrics collected for each servlet in each Web module within each J2EE application. And the Noun type JVM represents the set of metrics for each Java process (OC4J) currently running in the site.


Note:

In Appendix A, " Performance Metrics" the Noun type is called the metric table name.

The Noun naming scheme uses a '/' as the root of the hierarchy, with each Noun acting as a container under the root, or under its parent Noun.

DMS Object Relationships

This section describes the object relationships and attributes for DMS metrics, Sensors, and Nouns.

Table 9-4 describes the relationships between DMS objects. Figure 9-1 illustrates the relationships shown in Table 9-4 using a sample set of metrics.

Table 9-4 DMS Object Relationships and Attributes

Object Contains Attributes
Noun Sensors or other Nouns Name, Noun Type, Parent
Sensor Metrics Name, Description, Sensor Type

There are three Sensor Types: PhaseEvent, Event, and State.

Metric Value Name, Units designation

Isolating Expensive Intervals Using PhaseEvent Metrics

Carefully consider the requirements for new metrics when you add DMS instrumentation. It is important to add a sufficient number of metrics to validate that your code is behaving as desired.

As a guide, try to observe the following rules when you add DMS metrics:

  1. Add metrics only to provide an overview of the time the system spends in your block of code or module. You do not need to collect performance data for every method call, or for every distinct phase of your code or module.

  2. When your code calls external code that you do not control, and that you expect could take a significant amount of time, add a PhaseEvent Sensor to track the start and the completion of the external code.

Following these rules provides the following benefits:

  • Helps to limit the amount of information that DMS collects.

  • Allows those analyzing the system to prove that a module gives the expected runtime performance.

  • Ensures that people viewing DMS metrics can validate runtime performance without seeing an overwhelming amount of data.

  • Allows those analyzing system performance to separate and track your module from other system modules that are either expensive or failure prone.

Adding DMS Instrumentation To Java Applications

You can collect performance information in Java applications by adding DMS instrumentation to existing applications or by creating new applications that include DMS instrumentation.

The DMS samples shown in this chapter are supplied on the Oracle Technology Network Web site,

http://otn.oracle.com/tech/java/oc4j/demos/index.html

The DMS demo.zip file includes a ready to deploy .ear file and source code with build instructions. The demo includes two servlets, BasicBinomial.java and ImprovedBinomial.java.

The BasicBinomial servlet shows how to use the DMS API to add DMS Sensors.

The ImprovedBinomial servlet expands on the BasicBinomial and illustrates measuring the improved code, as compared with the BasicBinomial. ImprovedBinomial servlet also shows how to add more costly metrics that gather more detailed information. You can use ImprovedBinomial to compare the cost of enabling heavy instrumentation and the use of the DMS features that conditionally use instrumentation with the DMS SensorWeight feature.

Refer to the sample code for full details on the examples in this chapter.

To use DMS instrumentation, add DMS calls by performing the following steps:

Including DMS Imports

To use DMS you need to add DMS imports. The following example shows the imports that the sample application BasicBinomial.java requires.

import oracle.dms.instrument.DMSConsole;
import oracle.dms.instrument.Event;
import oracle.dms.instrument.Noun;
import oracle.dms.instrument.PhaseEvent;
import oracle.dms.instrument.State;
import oracle.dms.instrument.Sensor;  

Organizing Performance Data

Define DMS Nouns to organize Sensors and their associated metrics. DMS Nouns organize Sensors in a tree hierarchy in a manner comparable to a directory structure in a file system, starting with a root at the top of the tree.

Example 9-2 shows a section of code using Noun.create() from the BasicBinomial.java.

In Example 9-2, MathSeries specifies the Noun type. The Noun type is a name that reflects the set of metrics being collected. For example, MathSeries represents the metrics collected for the sample application containing a Binomial series computation. AggreSpy displays Sensors using the same Noun type together.

It is good practice to only use Noun types for Nouns that directly contain Sensors. When a Noun contains only Nouns, as in the Noun dmsDemo, and does not directly contain Sensors, AggreSpy displays the Noun type as a metric table, with no metrics. Example 9-2 shows the dmsDemo Noun that includes a Noun, BasicBinomial, but no Sensors. When the Noun type is not included for such a Noun, AggreSpy does not display a metric table associated with the Noun (the Noun’s children, with their associated Sensors are shown).


Note:

Start Noun type names with a capital letter to distinguish them from other DMS names.

Example 9-2 Using Noun.create To Organize Sensors

private Noun binRoot;    // root of the Binomial series DMS metrics
..
.
binRoot = Noun.create("/BasicBinomial", "MathSeries");

Defining and Using Metrics for Timing

To create metrics that time an interval, define and use a PhaseEvent Sensor as follows:

Defining PhaseEvent Sensors

Example 9-3 shows the DMS calls that declare and create the computeSeries PhaseEvent Sensor. This code defines a DMS metric named /Binomial/computeSeries.time.

PhaseEvent Sensors support a set of optional metrics, along with the default metric computeSeries.time (representing the time, as measured between the PhaseEvent start() and stop() calls). Associate optional metrics with PhaseEvent Sensors individually or as a complete set. Table 9-1 shows the available metrics for a PhaseEvent Sensor. The binComp.deriveMetric(Sensor.all) call in Example 9-3 enables all the supported optional metrics.


Note:

Using the method deriveMetric(Sensor.all) is recommended for adding optional metrics. Using this method with Sensor.all adds all metrics; this is good practice since the list of optional metrics could change in a future Oracle Application Server release.

Example 9-3 Defining PhaseEvent Sensors

private PhaseEvent binComp; // Time to compute Binomial series.
..
.   
binComp = PhaseEvent.create(binRoot, "computeSeries",                                  "Time to compute a Binomial series");binComp.deriveMetric(Sensor.all);

Using PhaseEvent Sensors

To use a PhaseEvent Sensor, an application calls the start() method to indicate the beginning of a phase, and subsequently calls the stop() method to indicate the completion of the phase.

Example 9-4 shows a code segment from BasicBinomial.java that uses the start() and stop() methods for the /Binomial/computeSeries.time metric. The long value named token that is returned from the PhaseEvent start() method must be passed to the corresponding PhaseEvent stop() method. This value is a timestamp representing the start time. Passing this value to the stop() method allows DMS to compute the PhaseEvent duration.


Note:

To assure that PhaseEvents are stopped, each PhaseEvent start() method, together with the code to be measured should be in a try block with the PhaseEvent stop() method in a corresponding finally block, as shown in Example 9-4.

Example 9-4 Using start() and stop() With PhaseEvent Sensors

long token = 0; // DMS
try {
    token = binComp.start(); // DMS
    BigInteger bins[] = bin(length);
    out.println("<H2>Binomial series for " +  length + "</H2>");
    for (int i = 0; i < length; i++)
        out.println("<br>" + bins[i]);
    }
    finally {
        binComp.stop(token); // DMS
        out.close();
    }

Defining and Using Metrics for Counting

To create metrics that count the occurrences of an event, define and use an Event Sensor as follows:

Defining Event Sensors

Example 9-5 shows the DMS calls that define an Event Sensor. This code allocates a counter and defines a DMS metric named /Binomial/loops.count.

Example 9-5 Defining Event Sensors

private Event binLoop;   // Loops needed for Binomial series.
.
.
.
   binLoop = Event.create(binRoot, "loops", "Iterations to compute series");

Using Event Sensors

DMS increments a counter when an application calls the occurred() method for an Event Sensor. Example 9-6 shows the occurred() call for an Event Sensor that increments the /Binomial/loops.count metric.

Example 9-6 Using occurred() With Event Sensors

binLoop.occurred();

Defining and Using Metrics for Recording Status Information

DMS captures status information with State Sensors. State sensors are general sensors, in that you use them to track Java Objects. DMS stores a reference to the Java Object and calls toString() on the object when the DMS value is sampled.

To create metrics that record status information define and use a State Sensor as follows:

Defining State Sensors

State Sensors support a set of optional DMS metrics, along with the default metric value. You associate optional metrics with State Sensors individually or as a complete set. Table 9-3 shows the available metrics for a State Sensor. Example 9-3 shows how to enable the optional metrics.

Example 9-7 shows the DMS calls that declare and create a State Sensor. This code defines a DMS metric named /Binomial/lastComputed.value. When you define a State Sensor, using the empty string in the create() method indicates that no units are associated with the State Sensor, as shown in Example 9-7.

Example 9-7 Defining State Sensors

private State binLast;   // Value of the last computed element in series.
.
.
.
binLast = State.create(binRoot, "lastComputed", "",
                       "Value of last computed series element", "0");

Using State Sensors

When an application calls a State Sensor’s update() method, DMS updates the value of the state sensor. Example 9-8 shows the update() call for a State Sensor that updates the /Binomial/lastComputed.value metric.

Example 9-8 Using update() With State Sensors

binLast.update(bins[k-1].toString());

Validating and Testing Applications Using DMS Metrics

You should test and verify the accuracy of the metrics that you add to Java applications.

This section covers the following topics:

Validating DMS Metrics

Use the dmstool and the other available DMS monitoring tools to verify and test new metrics.

Try to validate the following for new metrics:

  • Do expected metrics appear in the display? Test this by examining the code to make sure that all the metric names added using DMS instrumentation appear in your display or saved set of metrics.

  • Do unexpected metrics appear in the display? Verify that you have only added the metrics that you planned to add.

  • Are the metric values you see within reasonable ranges? Usually, upper and lower bounds for metrics can be established. You then test that the reported values for metrics do not exceed the expected bounds.

    For example, a "size of pool" metric should never report a negative value.

  • Are metric values accurate? This can be difficult to test; however, if an alternate means of measuring a particular metric is available then use it to verify metric values. For example, if you submit a known number of requests to a server and measure total time for the experiment, then you predict correct values for the relevant metrics and compare them with the actual monitored values. As another example, you can verify an Event Sensor count metric by examining records that you write to a log file or to the console.

    Check for timing inaccuracies that may apply for the metrics. Timing inaccuracies may be caused when low-resolution clocks time metrics for an interval of short duration. For example on Windows systems, the default Java clock advances only once every 15 milliseconds. DMS metrics reported for brief events on these systems must be analyzed with care.

  • When integrating DMS instrumentation with an existing package or when implementing a new feature, you should consider insulating a previously working system. For example, you could include an option to enable and disable new DMS metrics.

Testing DMS Metrics For Efficiency

The use of DMS metrics has some influence on application performance. When adding metrics, note the following:

  • The processing required for computing and storing metrics can slow down the execution of an application. DMS is fast and efficient, but it does have some required overhead cost. In addition, DMS cannot prevent developers from using the DMS API inefficiently. Therefore, before adding DMS instrumentation, establish reasonable expectations. After completing the implementation, measure the actual costs and compare them to your expectations. Be prepared to make changes to the implementation to reduce overhead costs until the measurements agree with expectations.

  • Make sure that new metrics are accurate. For most applications using DMS metrics, accuracy is more important than the performance cost of adding the DMS instrumentation. New DMS metrics should provide reliable and useful information.

  • DMS provides the DMSConsole.getSensorWeight() method to help you control the use of metrics. The central setting is an advisory measurement level that DMS does not enforce. To control which metrics to include, at runtime, the code must test the value for SensorWeight to determine whether to make DMS calls.

  • When integrating DMS instrumentation with an existing package or when implementing a new feature, you should consider insulating a previously working system. For example, you could include an option to enable and disable new DMS metrics.

  • Worrying about performance too soon often leads to costly design and implementation errors. According to Donald Knuth, "Premature optimization is the root of all evil".

  • You should run your performance tests with and without DMS enabled. If your tests show unacceptable results with DMS enabled, then you may want to re-design or re-implement metrics.

Understanding DMS Security Considerations

DMS metrics do not support user based access to DMS reports. When you define and use a DMS metric, the metric is available to any administrator that has access to DMS metrics. This means when you add DMS metrics, it is good practice to avoid placing customer sensitive information in the metrics.

When you add DMS instrumentation, the following users have access to the DMS metrics that you create:

Advanced DMS Topics

This section covers advanced DMS topics, including the following:

DMS Naming Conventions

Certain guidelines apply for defining DMS names. By following these guidelines, people viewing DMS metric reports can easily understand metrics across applications and across Oracle Application Server components.


Note:

View the naming conventions as guidelines; for each rule there may be an exception. In applying the rules, try to be as clear as possible, if there is a conflict, you may need to make an exception.

This section covers the following topics:

General DMS Naming

DMS metric names consist of a Sensor name, plus the "." character, plus the default metric or optional derivation. For example, the names: computeSeries.time, loops.count, and lastComputed.value are valid DMS metric names.

A Sensor base name is a simple string, not including the "." or the derivation. For example computeSeries, loops, and lastComputed are Sensor base names. A Sensor full name consists of the Sensor base name, preceded by the full name of its associated Noun, and a delimiter. For example, /Binomial/computeSeries, /Binomial/loops, and /Binomial/lastComputed.

A Noun base name is a simple string, not including a delimiter. For example Binomial is a noun base name. A Noun full name consists of the Noun base name, preceded by the full name of its parent, and a delimiter. For example /Binomial is a full Noun name.

General DMS Naming Rules and Character Sets

DMS names should be as compact as possible. Whenever possible, when you define Noun and Sensor names, avoid special characters such as white space, slashes, periods, parenthesis, commas, and control characters.

DMS replaces special characters with the "_" underscore character.


Note:

Oracle Application Server includes a number of built-in metrics. The Oracle Application Server built-in metrics do not always follow the DMS naming rules.

Noun and Noun Type Naming Rules

A Noun name should be a name which identifies a specific entity of interest.

Noun types should have names which clearly reflect the set of metrics being collected. For example, Servlet is the type for a Noun under which the metrics that are specific to a given servlet fall.

Noun type names should start with a capitol letter to distinguish them from other DMS names. All Nouns of a given type should contain the same set of sensors.

Sensor Naming Rules

  1. Sensor names should be descriptive, but not redundant. Sensor names should not contain any part of the Noun name hierarchy, or type, as this is redundant. Sensor names should avoid containing the specification of the units for the individual metrics. Where multiple words are required to describe a Sensor, the first word should start with a small letter, and the following words should start with capitol letters. For example computeSeries.

  2. Event Sensor and PhaseEvent Sensor names should have the form verbNoun where verb and Noun are interpreted as defined by English grammar. For example, activateInstance and runMethod. When a PhaseEvent monitors a function, method, or code block, it should be named to reflect the task performed as clearly as possible.

  3. The name of a State Sensor should be a Noun, possibly preceded by an adjective, which describes the semantics of the value which is tracked with this State. For example, lastComputed, totalMemory, port, availableThreads, activeInstances.

DMS Coding Recommendations

The following list includes coding recommendations for working with DMS.

  1. There is a global name space for DMS metrics. When you create a new Noun Sensor (PhaseEvent, Event, or State), its full name must not conflict with names in use by Oracle built-in metrics, or by other applications.

  2. Be sure all PhaseEvents are stopped. If the code block to be measured is not in a try block, then put it in a try block that includes PhaseEvent’s start(). Put the PhaseEvent’s stop() in a finally block.

  3. Use the DMS naming conventions.

  4. Avoid creating any DMS Sensor or Noun more than once. The DMS API allows this, and avoids creation of multiple objects, but DMS performs lookups for each subsequent call. Thus, you should define Sensors and Nouns during static initialization, or in the case of a Servlet, in the init() method.

  5. Assign a type for each Noun. If no type is assigned, the type is given the value "n/a" (not available). Nouns with the type specified as "n/a" are not shown in the Spy or AggreSpy display.

  6. Avoid creating PhaseEvents which do not measure a section of code that is expensive under some set of conditions.

  7. The DMS API calls are threadsafe; they provide sufficient synchronization to prevent races and access bugs.

Resetting and Destroying Sensors

The Sensor abstract class provides methods that you use to control PhaseEvent, Event, and State Sensors. The reset() method resets a Sensor’s metrics to initial values. The destroy() method removes a Sensor from DMS and releases references to its underlying resources.

Dumping DMS Metrics To Files

In a Java application, use the following methods to dump DMS metrics to a file.

The following code appends the current metrics to ./dms.log:

DMSConsole cons = new DMSConsole();
cons.dump();

The following code allows you to append or replace the contents of the specified file with the current metrics:

DMSConsole cons2 = new DMSConsole();
DMSConsole.dump("dmsmathseries.log", true, true);

The first argument specifies the file path name, the second argument specifies the output format, and the third argument specifies if the output is appended to the file or replaces the contents of the file.

Conditional Instrumentation Using DMS Sensor Weight

Use the DMS sensor weight feature to conditionally apply metrics. With sensor weight, you specify that applications execute expensive instrumentation only when the sensor weight is set to a particular value. Using this feature, enables you to include expensive metrics that you may only need for debugging.

Example 9-9 shows how to use DMSConsole.getSensorWeight() to test the value of the sensor weight, and optionally define and use a metric.

The sensor weight is set globally using the oracle.dms.sensors property on the command-line. Set this property using the OC4J startup options. Supported values for this property include: none, normal, heavy, and all.

Example 9-9 Using SensorWeight for Conditional Instrumentation

 /* DMS Method
   *
   * If the SensorWeight is high enough, return a phase with the
   * parameter in the name. Otherwise, return null.
   */
PhaseEvent heavyPhase(String param) {
PhaseEvent pe = null;
if (DMSConsole.getSensorWeight() > DMSConsole.NORMAL) {
   Noun base = Noun.create(binRoot, param, "MathSeries");
   pe = PhaseEvent.create(base, "computeSeries",
                                   "Time to compute a Binomial series");
   pe.deriveMetric(Sensor.all);
   }
return pe;