10 Configuring Policies

WLDF provides three main types of policies, which are differentiated by the sorts of data each can monitor:
  • Scheduled policies monitor diagnostic data that is generated by runtime MBeans according to a specific schedule. These policies can also be used to execute an action at a specific time or on a schedule.

  • Log policies monitor messages generated into the server or domain logs.

  • Instrumentation policies, also known as Event Data policies, monitor events generated by the WLDF Instrumentation component.

This chapter explains how to configure each policy type and includes the following sections:

For information about how to create a policy using the WebLogic Server Administration Console, see Create policies for a diagnostic system module in Oracle WebLogic Server Administration Console Online Help.

How Policies Are Configured

You can use any of the following tools to configure policies for diagnostic system modules:

  • WebLogic Server Administration Console

  • Fusion Middleware Control

  • WLST

  • REST

  • JMX application

This chapter refers primarily to using the WebLogic Server Administration Console or WLST.

The following table summarizes the attributes, elements, and options that you configure when creating a policy, and also identifies any requirements each configuration item has for specific policy types.

Item Description Policy Requirement

Rule Type

Attribute that determines the policy's type.

The default is Harvester.

Must be specified for log and instrumentation policies. Optional for scheduled policies.

Expression Language

Attribute that establishes the language used in the policy expression. The two supported languages are Java Expression Language (EL), and WLDF query language (deprecated).

Use EL in all policy types. The WLDF query language is supported, but deprecated.

Policy Expression

Expression that identifies a situation or condition that you want to trap for monitoring or diagnostic purposes. The expression can analyze log records, data events, or MBean metrics, depending on the rule type setting.

Optional for scheduled policies, but required for all others.

If a scheduled policy does not include an expression, the policy always executes the associated actions according to the Policy Schedule.

Actions

One or more operations that are executed when a policy expression is evaluated to true.

Optional.

Policy Schedule

A calendar-based schedule that determines when a scheduled policy is evaluated.

Required for all scheduled policies. Not available for log or instrumentation policies.

Alarm Options

Options that determine whether, or when, a policy can be evaluated again after it has been evaluated to true.

The default is None, which enables the policy to always be evaluated again.

Optional for all policy types.

Severity Option

Log message severity value that, when the policy is evaluated to true, is:

  1. Specified for the log message that is generated in the logging system.

  2. Passed to the actions that are configured with the policy.

The default is Notice.

Optional for all policy types.

Enablement Option

Flags that either enable or disable a policy from being evaluated.

The default is enabled.

Optional for all policy types.

Rule Type

When creating a policy, you must define its type in its rule type attribute. Policies with different rule types differ in two ways:

  • The syntax for specifying the conditions being monitored are unique to the rule type.

  • Log and instrumentation policies are triggered in real time, whereas scheduled policies are triggered by settings on the WLDFScheduleBean interface, described in Policy Schedule.

The way to define the rule type depends on the tool you use to create the policy:

  • If you are using the WebLogic Server Administration Console or Fusion Middleware Control, the rule type is determined by the policy type you are creating. For each of the policy types you can choose in either console, the following table identifies the corresponding rule type and WLDFWatchBean.RuleType attribute value that is defined for that policy:

    Policy Type . . . Rule Type WLDFWatchBean.RuleType Value

    Smart Rule

    Harvester

    Harvester
    

    Calendar Based

    Harvester

    Harvester
    

    Collected Metrics

    Harvester

    Harvester
    

    Server Log

    Log

    Log
    

    Domain Log

    Log

    DomainLog
    

    Event Data

    Instrumentation

    EventData
    

    For information about choosing a policy type using the WebLogic Server Administration Console, see Create policies for a diagnostic system module in Oracle WebLogic Server Administration Console Online Help.

  • If you are using WLST, REST, or JMX to configure a policy, you set the WLDFWatchBean.RuleType attribute as follows:

    Policy Type Rule Type Attribute

    Scheduled policy

    Harvester
    

    Log policy

    Log — for server log monitoring

    DomainLog — for domain log monitoring

    Instrumentation

    EventData — for instrumentation event monitoring

Expression Language

Policy expressions can be created using either of the following languages:

  • Java Expression Language (EL) (recommended)

  • WLDF query language (deprecated in WebLogic Server 12.2.1)

Note:

The policies described in this chapter are Java EL based. For information about configuring policies that use the WLDF query language, see WLDF Query Language-Based Policies.

If you have diagnostic system modules created with a previous release of WebLogic Server, WLDF supports expressions that use the WLDF query language. If you are creating new policies for either an existing or a new diagnostic system module, Oracle strongly recommends using Java EL as the policy expression language.

For more information about Java (EL), see the JSR-000341 Expression Language 3.0 specification at https://jcp.org/aboutJava/communityprocess/final/jsr341/index.html.

Policy Expression

A policy expression encapsulates all information necessary for specifying a rule that, when evaluated to true, causes the associated actions to be executed. When you use Java EL as the expression language, you can construct a policy expression that uses the following out-of-the-box resources to set the conditions that determine whether to fire an associated action:

  • Beans

    A bean is a Java object that can represent nearly anything in the domain, such as metrics from MBeans, or structured data surfaced by other beans.

  • Functions

    Functions are a set of operations that are provided either by EL itself, or by WLDF, that can be invoked on metric data.

  • Smart rules

    Smart Rules are special set of functions that encapsulate more complex logic and monitoring capabilities, and have specialized support in both the WebLogic Server Administration Console and Fusion Middleware Control. They can be used by themselves, or with other expression components as part of a larger, more complex expression.

Actions

Each policy can be associated with one or more actions that are executed whenever the policy evaluates to true. For information about configuring actions, see Configuring Actions.

Policy Schedule

All scheduled policies must be configured with a schedule. Scheduling allows policies to be evaluated according to a calendar schedule, at a specific time, after a duration of time, or at timed intervals.

You configure a policy schedule by setting attributes on the WLDFScheduleBean interface, which is a property of the WLDFWatchBean. You can set these attributes using the WebLogic Server Administration Console, WLST, REST, or a JMX application. When you are configuring new policies, the WebLogic Server Administration Console and Fusion Middleware Control provide convenient assistants and workflows for configuring common scheduling scenarios.

Note:

The WLDFScheduleBean is used for policy evaluation only when:

  • The configured policy rule type is "Harvester".

  • The configured expression language for the policy is "EL".

Note also that although scheduled policies that use the WLDFScheduleBean for scheduling are configured as Harvester types, the WLDF Harvester component is not used for scheduling.

Table 10-1 lists the attributes of the WLDFScheduleBean and their default values, which are the same as for the javax.ejb.ScheduleExpression interface. In addition, the syntax for specifying a value, range, list, or interval for a specific unit of time is also the same as that described for the ScheduleExpression interface. For more information, see http://docs.oracle.com/javaee/7/api/javax/ejb/ScheduleExpression.html.

Table 10-1 WLDFScheduleBean Attributes and Default Values

Attribute Description Default Allowable Values and Examples
second

One or more seconds within a minute

0

Allowable values: 0 to 59

Can be a value, range, list, or interval. To specify every n seconds of the minute, specify "*/n".

For example:

  • second = "30" — (value) run policy every 30 seconds within the minute

  • second = "10,20,30" — (list) run policy on seconds 10, 20 and 30 within the minute

  • second = "1-10" — (range) run policy on each of seconds 1 through 10 within the minute

  • second = "30/10" — (interval) run policy every 10 seconds within the minute, starting at second 30

  • second = "*/5" — (interval) run policy every 5 seconds within the minute

minute

One or more minutes within an hour

*/5

Allowable values: 0 to 59

Can be a value, range, list, or interval. To specify every n minutes of the hour, specify "*/n".

For example:

  • minute = "30" — (value) run policy every 30 minutes

    minute = "*/2" — (interval) run policy every two minutes of the hour

hour

One or more hours within a day

*

Allowable values: 0 to 23

Can be a value, range, list, or interval.

For example:

  • hour="16" — (value) run policy at 16:00.

  • hour = "*" — (range) run policy at every hour within a day.

dayOfWeek

One or more days within a week

*

Allowable values:

  • 0 to 7, where 0 and 7 represent Sunday. For example, dayOfWeek="3"

  • Sun, Mon, Tue, Wed, Thu, Fri, Sat. For example, dayOfWeek="Mon"

Can be a value, range, or list. For example:

  • dayOfWeek = "3" — run policy on Wednesday

  • dayOfWeek = "Mon-Fri" — run policy each day from Monday to Friday

  • dayOfWeek = "Mon, Wed, Fri" — run policy on Monday, Wednesday, and Friday

dayOfMonth

One or more days within a month

*

Allowable values:

  • 1 to 31

  • Last

  • -7 to -1

  • {1st, 2nd, 3rd, 4th, 5th, Last} {Sun, Mon, Tue, Wed, Thu, Fri, Sat}

Last represents the last day of the month.

-x (where x is in the range 7 to 1) means x days before the last day of the month.

1st, 2nd, and so on, specified with a day of the week identifies a single occurrence of that day within the month.

Can be a value, range, or list. For example:

  • dayOfMonth = "1" — run policy on first day of the month

  • dayOfMonth = "-3" — run policy on the third day before the end of the month

  • dayOfMonth = "2nd Mon" — run policy on the second Monday of the month

  • dayOfMonth = "1st Fri, 3rd Fri" — run policy on the first and third Friday of the month

  • dayOfMonth = "1 to 10" — run policy on each of the first 10 days of the month

month

One or more months within a year

*

Allowable values:

  • 1 to 12.

  • Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec

Can be a value, range, or list. For example:

  • month = "7" — run policy on the 7th month of the year

  • month = "Feb" — run policy in February

  • month = "1 - 3" — run policy on the first three months of the year

  • month = "Jan, Apr, Jul, Oct" — run policy in January, April, July, and October

year

A specific calendar year

*

Allowable values: a four-digit calendar year.

You can specify one value. For example:

  • year = "2015" — run policy in 2015

timezone

Time zone for the schedule

null

Defaults to the local VM time zone. You may use this attribute to specify a non-default time zone ID in whose context the schedule specification is to be evaluated.

Alarm Options

A policy that has been evaluated to true is referred to as having been triggered. For policies that are run repeatedly, an alarm determines when a policy can be evaluated again after it has been triggered. If a policy is configured with an alarm, a triggered policy is not evaluated again until the alarm is reset. For policies that are evaluated repeatedly, you can optionally define a minimum time that must transpire after a policy has been triggered before the policy can be evaluated again.

An alarm is important to configure for a policy that is run repeatedly to prevent the associated actions from being executed too frequently, such as generating a flood of emails or JMX notifications. For example, if you have a scheduled policy that executes a scale up action on a dynamic cluster, you should set an alarm that delays evaluating the policy again until the dynamic cluster is fully scaled up and is processing incoming requests. This delay is referred to as the alarm reset period. Without a proper alarm reset period, the scale up action could be executed again prematurely and counterproductively.

To configure an alarm for a policy, you specify the following:

  • The alarm type

  • The alarm reset period

The following table lists and describes each of the available alarm types:

Alarm Type Description
None

Allows the policy to be triggered whenever possible. This is the default.

AutomaticReset

Allows the policy to be triggered whenever possible, except that subsequent occurrences cannot occur any sooner than the interval specified in the alarm reset period.

ManualReset

Allows the policy to be triggered only once. After it is triggered, you must manually reset it to fire again. You can reset an alarm using a run-time MBean operation, either programmatically or with WLST. For example, you can use the resetWatchAlarm operation on the WLDFWatchNotificationRuntimeMBean.

Note the following alarm state behaviors:

  • When the alarm type is AutomaticReset, a policy enters the alarm state when triggered and stays in that state until the time interval specified by the alarm reset period has expired.

  • If a policy is configured with a ManualReset alarm, the policy enters the alarm state when triggered, and remains in that state until you manually reset it.

  • When a policy is in the alarm state, the policy is not evaluated again until the alarm is reset.

  • If a policy's alarm type is None, the configured action can be executed every time that the policy is triggered. The alarm state is never set in these cases.

Severity Option

Whenever a policy is triggered, a message is automatically generated in the logging system. The severity option is an optional value you can configure that:

  1. Gets assigned as the severity value of the log message generated in the logging system.

  2. Is also passed to the actions that are configured with the policy.

The severity option must be one that is defined for the WebLogic logging service in the weblogic.logging.Severities class. The accepted values are Info, Notice, Warning, Error, Critical, Alert, and Emergency. The default is Notice.

Enablement Option

Each policy can be individually enabled and disabled by using the Enabled attribute on that policy. The value you specify for this attribute is true or false. When disabled, a policy is not evaluated and its configured actions are not executed.

However, note that the WLDFWatchNotificationBean, which is the parent of all policy and action configurations in a diagnostic system module, also has an Enabled attribute. If the value of the WLDFWatchNotificationBean.Enabled attribute is false, all individual policies in the diagnostic system modules are disabled regardless of whether its policies are individually configured as enabled.

Configuring Scheduled Policies

Scheduled policies monitor diagnostic data that consists of data coming from MBeans within the WebLogic Server Runtime MBean Server, including the read-only configuration MBeans in the WebLogic Server Runtime MBean Server. These values, called metrics, originate from common WebLogic Server JMX data sources such as the following:

  • WebLogic Server Runtime MBean Server

  • Domain Runtime MBean Server

  • JVM platform MBean server

Scheduled policies are useful for monitoring run-time state information in the WebLogic Server environment. Examples of diagnostic data that is useful to monitor using scheduled policies are:

  • Changes over time in average JVM heap usage

    If the average amount of free heap reaches a particular threshold that is defined in the policy expression, the configured action is executed, such as sending an email to the system administrator.

  • Data from multiple services that are considered together, such as response-time metrics reported by a load balancer and message-backlog metrics from a message queue

    If the combination of data meets a particular set of criteria defined in the policy expression, the policy can fire a scaling action

The following sections explain how to configure, and show examples of, the three scheduled policy types:

See also Chaining Policies for information about how to create a policy expression that can reference the state of other policies defined within the same WLDF module as the beans. Policy chaining allows the state of one policy to be part of the expression of another.

Configuring Calendar Based Policies

The simplest type of scheduled policy is the calendar based policy. You can use a calendar based policy to fire any associated actions according to the policy's schedule.

Calendar-based policies are simply scheduled policies with no associated expression. This enables purely schedule-driven action execution; that is, the ability to unconditionally perform a set of actions on a specified schedule. If no expression is provided, when the scheduled time occurs, the policy treats the empty expression as a true result and executes the associated actions.

Note:

Calendar based policies are supported only for policies that: have the following configuration attributes:

  • The rule type specified as 'Harvester'

  • The expression language specified as 'EL'

The following example shows the configuration of a calendar based policy using WLST. This policy fires a scale up action at 3:00 a.m. on December 26.

calendarScaleUp=wn.lookupWatch('ChristmasReturnsScaleUpWatch')
if calendarScaleUp == None:
  print "Creating scale-up for the post-Christmas returns rush on Dec 26 at 3am"
  calendarScaleUp=wn.createWatch('ChristmasReturnsScaleUpWatch')
calendarScaleUp.setRuleType('Harvester')
calendarScaleUp.setExpressionLanguage('EL')
calendarScaleUp.getSchedule().setHour('3')
calendarScaleUp.getSchedule().setMinute('0')
calendarScaleUp.getSchedule().setSecond('0')
calendarScaleUp.getSchedule().setDayOfMonth('26')
calendarScaleUp.getSchedule().setMonth('Dec')
calendarScaleUp.setEnabled(false)
calendarScaleUp.addNotification(scaleUp)

Configuring Smart Rule Based Policies

Smart rules are prepackaged functions that greatly simplify the creation of policy expressions. The WebLogic Server Administration Console and Fusion Middleware Control, in particular, each contain a smart rule editor to greatly simplify the task of configuring a smart rule for the policy you are creating.

Smart rules perform a number of complex operations, but surface only a small number of configuration parameters that you set. Details about the specific low level metrics that are collected, how they are used, and so on, are hidden, thereby making them easy to use. Smart rules return only a Boolean value, which determines whether the policy is evaluated to true.

You use a smart rule as a predicate in policy expression by simply specifying the parameters required by that smart rule. For example, to evaluate whether a particular increase exists in the average thread pool queue length in the local server, you create a policy that specifies the ServerHighQueueLength smart rule as the policy expression and provide the following parameters:

  • The sampling period for collecting the value of the ThreadPoolRuntimeMBean.QueueLength attribute

  • Duration, or the most recent window of time, in which samples are retained

  • A threshold value that establishes the maximum acceptable number of threads in the queue

The smart rule takes responsibility for sampling the appropriate metrics over the specified time interval, computing averages, comparing threshold values, and determining whether the smart rule evaluates to true.

Note:

Smart rules are supported for use only in scheduled policies that are configured with Java EL as the expression language.

Types of Diagnostic Data that Smart Rules Evaluate

Smart rules can monitor trends in metrics in a server or cluster over time, including:

  • Average system throughput

  • Process CPU load

  • Pending user request count

  • Idle or stuck thread count

  • Incoming request queue size

  • System load average

  • JVM free heap size

  • Any metric value visible from JMX, such as custom MBean values

You can use smart rules as building blocks in policy expressions. In the simplest case, a single smart rule can be used by itself in a policy expression. You can also combine a smart rule with others, as well as with other EL constructs, to form more complex expressions.

For example, you can construct a policy that sends an email notification if all of the following conditions occur simultaneously in a server instance or cluster:

  • Low JVM free heap percentage

  • High number of stuck threads

  • High incoming requests queue size

For details about all the smart rules provided by WLDF, see Smart Rule Reference.

Smart Rule Example

The ClusterLowHeapFreePercent smart rule compares the average free heap across all Managed Servers in a cluster by monitoring the value of the JVMRuntimeMBean.HeapFreePercent attribute. A policy expression that uses this smart rule will be evaluated to true if a minimum percentage of Managed Servers in the cluster have an average free heap that is less than a particular threshold value.

The ClusterLowHeapFreePercent smart rule takes the following input parameters:

  • Cluster name

  • Sampling period — The frequency with which the value of the HeapFreePercent metric is collected

  • Retention window — A sliding window of time during which samples are retained. For example, the most recent five minutes.

  • percentFreeLimit — A value that represents the low free heap percentage threshold.

  • percentServersLimit — A percentage of Managed Servers in the cluster that must have an average free heap that is less than percentFreeLimit to cause the expression to evaluate to true.

The following is an example configuration of the ClusterLowHeapFreePercent smart rule:

wls:ClusterLowHeapFreePercent("myCluster","30 seconds","10 minutes",20,60)

From every Managed Server in myCluster, this smart rule collects the value of the HeapFreePercent every 30 seconds, retaining the most recent 10 minutes of data, and evaluates to true if at least 60 per cent of the Managed Servers in myCluster have an average free heap percentage that is less than 20 per cent.

This smart rule could be configured to fire an action when it evaluates to true, such as sending an email to the system administrator to report that a low free heap condition exists in the cluster. The system administrator can then take remedial action as necessary.

You can use smart rules in conjunction with scaling actions, described in Configuring Elastic Actions, to configure policy based scaling of a dynamic cluster. This capability enables automated elasticity in that cluster. For more information, including a demo that you can download and run, see Policy-Based Scaling in Configuring Elasticity in Dynamic Clusters for Oracle WebLogic Server.

Configuring Collected Metrics Based Policies

Collected metrics based policies, which are a type of scheduled policy, can use WLDF-provided beans and functions within their policy expressions. These beans are JavaBean objects that can obtain access to common WebLogic Server JMX data sources, such as the following:

  • WebLogic Server Runtime MBean Server

  • Domain Runtime MBean Server

  • JVM platform MBean server

The following sections explain how to configure collected metrics based policies using beans and functions:

Using Beans

Table 10-2 summarizes the beans provided by WebLogic Server. For complete reference information about each of these beans, see WLDF Beans Reference.

Table 10-2 Beans Provided by WebLogic Server

Name Prefix Scope Summary
runtime wls

Only available from partition scope diagnostic system module deployments and partitions

Provides access to MBeans in the local WebLogic Server Runtime MBean Server. These MBeans include both the read-only configuration MBean and RuntimeMBean instances.

domainRuntime wls

Administration Server

Provides access to MBeans on the Domain Runtime MBean Server (Administration Server only).

clusterRuntime wls

Administration Server

Provides a domain-wide access to cluster member data (Administraton Server only).

instrumentationEvent wls

Administration Server, Managed Server, and partitions

Provides access to instrumentation fields in instrumentation policy expressions.

log wls

Administration Server, Managed Server, and partitions

Provides access to log event fields in log policy expressions.

platform wls

Administration Server or Managed Server

Provides access to the JVM's platform MBean server.

Note that in the majority of cases, the platform bean is functionally equivalent to the runtime bean: WebLogic Server uses the JVM's platform MBean server to contain the WebLogic run-time MBeans by default.

partition wls

Partition scope WLDF diagnostic system module deployments.

Provides access to partition scope metrics.

This bean is available only to policies that are configured in a diagnostic system module that is deployed in the same partition to which this bean is scoped.

resource

n/a

Administration Server, Managed Server, and partitions

Provides access to beans and state information within a diagnostic system module deployment.

Access is restricted to policies that are configured within the same diagnostic system module.

Using Functions

In addition to the bundled functions and operators that come with Java EL by default, you can also use WLDF-provided functions within policy expressions for common operations with metric data and for retaining a set of metrics with history. For reference information about all the functions included in WLDF, see Functions Reference.

Chaining Policies

Within the same diagnostics system module, the expression in one policy can reference other policies as beans within that expression. In this way, complex policy expressions can be reused and "chained" together to allow the state of one policy to be part of the expression of another. This allows more complex, interrelated policies to be written, while keeping such policy configurations more readable and maintainable.

To allow access to policy states within an expression, you can use the resource bean within the global bean name space for each policy. The resource bean supports a Map attribute named watches, where each key in the map is the name of a policy defined within the same diagnostics system module.

Each value in the policy's map is a bean representing the named policy. These policy beans support a simple Boolean alarm attribute, which has the following semantics:

  • If the policy is configured with an alarm type other than None, the alarm attribute returns true if the policy is currently in the alarm state.

  • If no alarm type is configured on the policy, the alarm attribute yields the most recently evaluated result.

  • If the alarm attribute on a policy bean is accessed before the named policy has successfully completed an evaluation cycle, a NotEnoughDataException is thrown. This occurrence also has the effect of invalidating the expression during that evaluation cycle: the policy isn't disabled, but it is effectively a non-result when it occurs.

Configuring Log Policies

Use log policies to monitor the occurrence of specific messages or strings in the server or domain log. Policies of this type are triggered as a result of a log message containing the specified data being issued.

When creating a log policy, you can use the log bean in an EL expression to obtain access to log message fields. See logfor details about the log bean attributes.

Note:

Any log policies that search for the RUNNING state message ID should search for message ID BEA-000365, and not BEA-000360. The message ID BEA-000360 is issued immediately before the state change to RUNNING, and BEA-000365 is issued immediately afterward. Therefore, such log policies are able to find only message ID BEA-000365.

An example configuration for a server log policy is shown in Example 10-1.

Example 10-1 Sample Configuration for a Log Watch (in DIAG_MODULE.xml)

<wldf-resource xmlns="http://xmlns.oracle.com/weblogic/weblogic-diagnostics" 
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-diagnostics/1.0/weblogic-diagnostics.xsd"> 
  <name>mywldf1</name> 
  <watch-notification> 
    <enabled>true</enabled> 
    <log-watch-severity>Info</log-watch-severity> 
    <watch>
      <name>myLogWatch</name>
      <enabled>true</enabled>
      <rule-type>Log</rule-type>
      <rule-expression>log.messageId == 'BEA-000360'</rule-expression>
      <expression-language>EL</expression-language>
      <alarm-type>ManualReset</alarm-type>
    </watch>
    <smtp-notification> 
      <name>myMailNotif2</name> 
      <enabled>true</enabled> 
      <mail-session-jndi-name>myMailSession</mail-session-jndi-name> 
      <subject>This is a log alert</subject> 
      <recipient>username@emailservice.com</recipient> 
    </smtp-notification> 
  </watch-notification> 
</wldf-resource> 

In Example 10-1, note how the <rule-type> of Log causes messages or strings entered in the server log to be monitored. A <rule-type> of DomainLog monitors messages or strings in the domain log.

Configuring Instrumentation Policies

You use instrumentation policies to monitor the events from the WLDF Instrumentation component. Policies of this type are triggered as a result of the event being posted.

Example 10-2 shows an example configuration for an Instrumentation policy.

Example 10-2 Sample Configuration for an Instrumentation Policy (in DIAG_MODULE.xml)

  <watch-notification>
    <watch>
    <name>myInstWatch</name>
    <enabled>true</enabled>
    <rule-type>EventData</rule-type>
    <rule-expression>instrumentationEvent.payload &gt; 100000000 &amp;&amp; instrumentationEvent.monitor == 'Servlet_Around_Service'</rule-expression>
    <expression-language>EL</expression-language>
    <alarm-type>ManualReset</alarm-type>
    <notification>mySMTPNotification</notification>
  </watch>
  <smtp-notification>
    <name>mySMTPNotification</name>
    <enabled>true</enabled>
    <mail-session-jndi-name>myMailSession</mail-session-jndi-name>
    <subject xsi:nil="true"></subject>
    <body xsi:nil="true"></body>
    <recipient>username@emailservice.com</recipient>
  </smtp-notification>
</watch-notification>