10 Configuring Policies

The WebLogic Diagnostics Framework (WLDF) provides three main types of policies, which are differentiated by the sorts of data each can monitor. The policy types are:

• Scheduled policies, which monitor diagnostic data that is generated by runtime MBeans according to a specific schedule. These policies can also be used to execute an action at a specific time or on a schedule.

• Log policies, which monitor messages generated into the server or domain logs.

• Instrumentation policies, also known as Event Data policies, which monitor events generated by the WLDF Instrumentation component.

This chapter explains how to configure each policy type and includes the following sections:

For information about how to create a policy using the WebLogic Server Administration Console, see Create policies for a diagnostic system module in Oracle WebLogic Server Administration Console Online Help.

• How Policies Are Configured
  There are several components of a policy that you configure, such as the type, expression, corresponding actions to be executed when the policy is evaluated to true, and more.
• Configuring Scheduled Policies
  Scheduled policies monitor diagnostic data that consists of data coming from MBeans within the WebLogic Server Runtime MBean Server, including the read-only configuration MBeans in the WebLogic Server Runtime MBean Server.
• Configuring Log Policies
  Use log policies to monitor the occurrence of specific messages or strings in the server or domain log. Policies of this type are triggered as a result of a log message containing the specified data being issued.
• Configuring Instrumentation Policies
  You use instrumentation policies to monitor the events from the WLDF Instrumentation component. Policies of this type are evaluated as a result of an event being posted by the Instrumentation component, which occurs when code that matches a deployed Instrumentation monitor is exercised.
• Creating Complex Policy Expressions Using WLDF Java EL Extensions
  Oracle expects that the library of smart rules packaged with WLDF are sufficient for meeting the needs of creating scheduled policies that evaluate runtime performance data in a server or cluster. However, if you have a specific scheduled policy need that cannot be satisfied by a smart rule, WLDF also provides a set of extensions to Java EL. These extensions are intended for use in policies that evaluate very specific characteristics or trends in metrics collected from runtime MBean servers in your WebLogic domain.

How Policies Are Configured

There are several components of a policy that you configure, such as the type, expression, corresponding actions to be executed when the policy is evaluated to true, and more.

You can use any of the following tools to configure policies for diagnostic system modules:

• WebLogic Server Administration Console

• Fusion Middleware Control

• WLST

• REST

• JMX application

This chapter refers primarily to using the WebLogic Server Administration Console or WLST.

The following table summarizes the attributes, elements, and options that you configure when creating a policy, and also identifies any requirements each configuration item has for specific policy types.

Item	Description	Policy Requirement
Rule Type	Attribute that determines the policy's type. The default is Harvester.	Must be specified for log and instrumentation policies. Optional for scheduled policies.
Expression Language	Attribute that establishes the language used in the policy expression. The two supported languages are Java Expression Language (EL), and WLDF query language (deprecated).	Use EL in all policy types. The WLDF query language is supported, but deprecated.
Policy Expression	Expression that identifies a situation or condition that you want to trap for monitoring or diagnostic purposes. The expression can analyze log records, data events, or MBean metrics, depending on the rule type setting.	Optional for scheduled policies, but required for all others. If a scheduled policy does not include an expression, the policy always executes the associated actions according to the Policy Schedule.
Actions	One or more operations that are executed when a policy expression is evaluated to true.	Optional.
Policy Schedule	A calendar-based schedule that determines when a scheduled policy is evaluated.	Required for all scheduled policies. Not available for log or instrumentation policies.
Alarm Options	Options that determine whether, or when, a policy can be evaluated again after it has been evaluated to true. The default is None, which enables the policy to always be evaluated again.	Optional for all policy types.
Severity Option	Log message severity value that, when the policy is evaluated to true, is: Specified for the log message that is generated in the logging system. Passed to the actions that are configured with the policy. The default is Notice.	Optional for all policy types.
Enablement Option	Flags that either enable or disable a policy from being evaluated. The default is enabled.	Optional for all policy types.

• Rule Type
  
• Expression Language
  
• Policy Expression
  
• Actions
  
• Policy Schedule
  
• Alarm Options
  
• Severity Option
  
• Enablement Option
  

Rule Type

When creating a policy, you must define its type in its rule type attribute. Policies with different rule types differ in two ways:

• The syntax for specifying the conditions being monitored are unique to the rule type.

• Log and instrumentation policies are triggered in real time, whereas scheduled policies are triggered by settings on the WLDFScheduleBean interface, described in Policy Schedule.

The way to define the rule type depends on the tool you use to create the policy:

• If you are using the WebLogic Server Administration Console or Fusion Middleware Control, the rule type is determined by the policy type you are creating. For each of the policy types you can choose in either console, the following table identifies the corresponding rule type and WLDFWatchBean.RuleType attribute value that is defined for that policy:

  Policy Type . . .	Rule Type	WLDFWatchBean.RuleType Value
  Smart Rule	Harvester	Harvester
  Calendar Based	Harvester	Harvester
  Collected Metrics	Harvester	Harvester
  Server Log	Log	Log
  Domain Log	Log	DomainLog
  Event Data	Instrumentation	EventData

  For information about choosing a policy type using the WebLogic Server Administration Console, see Create policies for a diagnostic system module in Oracle WebLogic Server Administration Console Online Help.

• If you are using WLST, REST, or JMX to configure a policy, you set the WLDFWatchBean.RuleType attribute as follows:

  Policy Type	Rule Type Attribute
  Scheduled policy	Harvester
  Log policy	Log - for server log monitoring DomainLog - for domain log monitoring
  Instrumentation	EventData - for instrumentation event monitoring

Expression Language

Policy expressions can be created using either of the following languages:

• Java Expression Language (EL) (recommended)

• WLDF query language (deprecated in WebLogic Server 12.2.1)

See Java EL tutorials at https://docs.oracle.com/javaee/7/tutorial/jsf-el.htm#GJDDD. For more information about Java (EL), see the JSR-000341 Expression Language 3.0 specification at https://jcp.org/aboutJava/communityprocess/final/jsr341/index.html.

If you have diagnostic system modules created with a previous release of WebLogic Server, WLDF supports expressions that use the WLDF query language. If you are creating new policies for either an existing or a new diagnostic system module, Oracle strongly recommends using Java EL as the policy expression language.

Note:

The policies described in this chapter are Java EL based. For information about configuring policies that use the WLDF query language, see WLDF Query Language-Based Policies.

Policy Expression

A policy expression encapsulates all information necessary for specifying a rule that, when evaluated to true, causes the associated actions to be executed. When you use Java EL as the expression language, you can construct a policy expression that uses the following out-of-the-box resources to set the conditions that determine whether to fire an associated action:

• Beans

  A bean is a Java object that represents the data available for a policy expression to use, such as metrics from MBeans, log event information, or structured data surfaced by other beans. Beans are accessed in policy expressions using standard JavaBean conventions.

• Functions

  Functions are a set of operations that are provided either by EL itself, or by WLDF, that can be utilized from policy expressions to transform or evaluate data.

• Smart rules

  Smart Rules are special set of functions that encapsulate more complex logic and monitoring capabilities, and have specialized support in both the WebLogic Server Administration Console and Fusion Middleware Control. They can be used by themselves, or with other expression components as part of a larger, more complex expression.

Actions

Each policy can be associated with one or more actions that are executed whenever the policy evaluates to true. See Configuring Actions.

Policy Schedule

All scheduled policies must be configured with a schedule. Scheduling allows policies to be evaluated according to a calendar schedule, at a specific time, after a duration of time, or at timed intervals.

You configure a policy schedule by setting attributes on the WLDFScheduleBean interface, which is a property of the WLDFWatchBean. You can set these attributes using the WebLogic Server Administration Console, WLST, REST, or a JMX application. When you are configuring new policies, the WebLogic Server Administration Console and Fusion Middleware Control provide convenient assistants and workflows for configuring common scheduling scenarios.

Note:

The WLDFScheduleBean is used for policy evaluation only when:

• The configured policy rule type is "Harvester".

• The configured expression language for the policy is "EL".

Note also that although scheduled policies that use the WLDFScheduleBean for scheduling are configured as Harvester types, the WLDF Harvester component is not used for scheduling.

Table 10-1 lists the attributes of the WLDFScheduleBean and their default values, which are the same as for the javax.ejb.ScheduleExpression interface. In addition, the syntax for specifying a value, range, list, or interval for a specific unit of time is also the same as that described for the ScheduleExpression interface. See http://docs.oracle.com/javaee/7/api/javax/ejb/ScheduleExpression.html.

Table 10-1 WLDFScheduleBean Attributes and Default Values

Attribute	Description	Default	Allowable Values and Examples
second	One or more seconds within a minute	0	Allowable values: 0 to 59 Can be a value, range, list, or interval. To specify every n seconds of the minute, use "*/n". For example: second = "30" - (value) run policy every 30 seconds within the minute second = "10,20,30" - (list) run policy on seconds 10, 20, and 30 within the minute second = "1-10" - (range) run policy on each of seconds 1 through 10 within the minute second = "30/10" - (interval) run policy every 10 seconds within the minute, starting at second 30 second = "*/5" - (interval) run policy every 5 seconds within the minute
minute	One or more minutes within an hour	*/5	Allowable values: 0 to 59 Can be a value, range, list, or interval. To specify every n minutes of the hour, specify "*/n". For example: minute = "30" - (value) run policy every 30 minutes minute = "*/2" - (interval) run policy every two minutes of the hour
hour	One or more hours within a day	*	Allowable values: 0 to 23 Can be a value, range, list, or interval. For example: hour="16" - (value) run policy at 16:00. hour = "*" - (range) run policy at every hour within a day.
dayOfWeek	One or more days within a week	*	Allowable values: 0 to 7, where 0 and 7 represent Sunday. For example, dayOfWeek="3" Sun, Mon, Tue, Wed, Thu, Fri, Sat. For example, dayOfWeek="Mon" Can be a value, range, or list. For example: dayOfWeek = "3" - run policy on Wednesday dayOfWeek = "Mon-Fri" - run policy each day from Monday to Friday dayOfWeek = "Mon, Wed, Fri" - run policy on Monday, Wednesday, and Friday
dayOfMonth	One or more days within a month	*	Allowable values: 1 to 31 Last -7 to -1 {1st, 2nd, 3rd, 4th, 5th, Last} {Sun, Mon, Tue, Wed, Thu, Fri, Sat} Last represents the last day of the month. -x (where x is in the range 7 to 1) means x days before the last day of the month. 1st, 2nd, and so on, specified with a day of the week identifies a single occurrence of that day within the month. Can be a value, range, or list. For example: dayOfMonth = "1" - run policy on first day of the month dayOfMonth = "-3" - run policy on the third day before the end of the month dayOfMonth = "2nd Mon" - run policy on the second Monday of the month dayOfMonth = "1st Fri, 3rd Fri" - run policy on the first and third Friday of the month dayOfMonth = "1 to 10" - run policy on each of the first 10 days of the month
month	One or more months within a year	*	Allowable values: 1 to 12. Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec Can be a value, range, or list. For example: month = "7" - run policy on the 7th month of the year month = "Feb" - run policy in February month = "1 - 3" - run policy for the first three months of the year month = "Jan, Apr, Jul, Oct" - run policy in January, April, July, and October
year	A specific calendar year	*	Allowable values: a four-digit calendar year. You can specify one value. For example: year = "2015" - run policy in 2015
timezone	Time zone for the schedule	null	Defaults to the local VM time zone. You may use this attribute to specify a non-default time zone ID in whose context the schedule specification is to be evaluated.

Alarm Options

A policy that has been evaluated to true is referred to as having been triggered. For policies that are run repeatedly, an alarm determines when a policy can be evaluated again after it has been triggered. If a policy is configured with an alarm, a triggered policy is not evaluated again until the alarm is reset. For policies that are evaluated repeatedly, you can optionally define a minimum time that must transpire after a policy has been triggered before the policy can be evaluated again.

An alarm is important to configure for a policy that is run repeatedly to prevent the associated actions from being executed too frequently, such as generating a flood of emails or JMX notifications. For example, if you have a scheduled policy that executes a scale up action on a dynamic cluster, you should set an alarm that delays evaluating the policy again until the dynamic cluster is fully scaled up and is processing incoming requests. This delay is referred to as the alarm reset period. Without a proper alarm reset period, the scale up action could be executed again prematurely and counterproductively.

To configure an alarm for a policy, you specify the following:

• The alarm type

• The alarm reset period

The following table lists and describes each of the available alarm types:

Alarm Type	Description
None	Allows the policy to be triggered whenever possible. This is the default.
AutomaticReset	Allows the policy to be triggered whenever possible, except that subsequent occurrences cannot occur any sooner than the interval specified in the alarm reset period.
ManualReset	Allows the policy to be triggered only once. After it is triggered, you must manually reset it to fire again. You can reset an alarm using a run-time MBean operation, either programmatically or with WLST. For example, you can use the resetWatchAlarm operation on the WLDFWatchNotificationRuntimeMBean.

Note the following alarm state behaviors:

• When the alarm type is AutomaticReset, a policy enters the alarm state when triggered and stays in that state until the time interval specified by the alarm reset period has expired.

• If a policy is configured with a ManualReset alarm, the policy enters the alarm state when triggered, and remains in that state until you manually reset it.

• When a policy is in the alarm state, the policy is not evaluated again until the alarm is reset.

• If a policy's alarm type is None, the configured action can be executed every time that the policy is triggered. The alarm state is never set in these cases.

Severity Option

Whenever a policy is triggered, a message is automatically generated in the logging system. The severity option is an optional value you can configure that:

1. Gets assigned as the severity value of the log message generated in the logging system.

2. Is also passed to the actions that are configured with the policy.

The severity option must be one that is defined for the WebLogic logging service in the weblogic.logging.Severities class. The accepted values are Info, Notice, Warning, Error, Critical, Alert, and Emergency. The default is Notice.

Enablement Option

Each policy can be individually enabled and disabled by using the Enabled attribute on that policy. The value you specify for this attribute is true or false. When disabled, a policy is not evaluated and its configured actions are not executed.

However, note that the WLDFWatchNotificationBean, which is the parent of all policy and action configurations in a diagnostic system module, also has an Enabled attribute. If the value of the WLDFWatchNotificationBean.Enabled attribute is false, all individual policies in the diagnostic system modules are disabled regardless of whether its policies are individually configured as enabled.

Configuring Scheduled Policies

Scheduled policies monitor diagnostic data that consists of data coming from MBeans within the WebLogic Server Runtime MBean Server, including the read-only configuration MBeans in the WebLogic Server Runtime MBean Server.These values, called metrics, originate from common WebLogic Server JMX data sources such as the following:

• WebLogic Server Runtime MBean Server

• Domain Runtime MBean Server

• JVM platform MBean server

Scheduled policies are useful for monitoring run-time state information in the WebLogic Server environment. Examples of diagnostic data that is useful to monitor using scheduled policies are:

• Changes over time in average JVM heap usage

  If the average amount of free heap reaches a particular threshold that is defined in the policy expression, the configured action is executed, such as sending an email to the system administrator.

• Data from multiple services that are considered together, such as response-time metrics reported by a load balancer and message-backlog metrics from a message queue

  If the combination of data meets a particular set of criteria defined in the policy expression, the policy can fire a scaling action

See also Chaining Policies for information about how to create a policy expression that can reference the state of other policies defined within the same WLDF module as the beans. Policy chaining allows the state of one policy to be part of the expression of another.

The following sections explain how to configure, and show examples of, the three scheduled policy types:

• Configuring Calendar Based Policies
  
• Configuring Smart Rule Based Policies
  
• Chaining Policies
  

Configuring Calendar Based Policies

The simplest type of scheduled policy is the calendar based policy. You can use a calendar based policy to fire any associated actions according to the policy's schedule.

Calendar-based policies are simply scheduled policies with no associated expression. This enables purely schedule-driven action execution; that is, the ability to unconditionally perform a set of actions on a specified schedule. If no expression is provided, when the scheduled time occurs, the policy treats the empty expression as a true result and executes the associated actions.

Note:

Calendar based policies are supported only for policies that: have the following configuration attributes:

• The rule type specified as 'Harvester'

• The expression language specified as 'EL'

The following example shows the configuration of a calendar based policy using WLST. This policy fires a scale up action at 3:00 a.m. on December 26.

calendarScaleUp=wn.lookupWatch('ChristmasReturnsScaleUpWatch')
if calendarScaleUp == None:
  print "Creating scale-up for the post-Christmas returns rush on Dec 26 at 3am"
  calendarScaleUp=wn.createWatch('ChristmasReturnsScaleUpWatch')
calendarScaleUp.setRuleType('Harvester')
calendarScaleUp.setExpressionLanguage('EL')
calendarScaleUp.getSchedule().setHour('3')
calendarScaleUp.getSchedule().setMinute('0')
calendarScaleUp.getSchedule().setSecond('0')
calendarScaleUp.getSchedule().setDayOfMonth('26')
calendarScaleUp.getSchedule().setMonth('Dec')
calendarScaleUp.setEnabled(false)
calendarScaleUp.addNotification(scaleUp)

Configuring Smart Rule Based Policies

Smart rules are prepackaged functions that greatly simplify the creation of policy expressions. The WebLogic Server Administration Console and Fusion Middleware Control, in particular, each contain a smart rule editor to greatly simplify the task of configuring a smart rule for the policy you are creating.

Smart rules perform a number of complex operations, but surface only a small number of configuration parameters that you set. Details about the specific low level metrics that are collected, how they are used, and so on, are hidden, thereby making them easy to use. Smart rules return only a Boolean value, which determines whether the policy is evaluated to true.

You use a smart rule as a predicate in policy expression by simply specifying the parameters required by that smart rule. For example, to evaluate whether a particular increase exists in the average thread pool queue length in the local server, you create a policy that specifies the ServerHighQueueLength smart rule as the policy expression and provide the following parameters:

• The sampling period for collecting the value of the ThreadPoolRuntimeMBean.QueueLength attribute

• Duration, or the most recent window of time, in which samples are retained

• A threshold value that establishes the maximum acceptable number of threads in the queue

The smart rule takes responsibility for sampling the appropriate metrics over the specified time interval, computing averages, comparing threshold values, and determining whether the smart rule evaluates to true.

Note:

Smart rules are supported for use only in scheduled policies that are configured with Java EL as the expression language.

• Types of Diagnostic Data that Smart Rules Evaluate
  
• Smart Rule Example
  

Types of Diagnostic Data that Smart Rules Evaluate

Smart rules can monitor trends in metrics in a server or cluster over time, including:

• Average system throughput

• Process CPU load

• Pending user request count

• Idle or stuck thread count

• Incoming request queue size

• System load average

• JVM free heap size

• Any metric value visible from JMX, such as custom MBean values

You can use smart rules as building blocks in policy expressions. In the simplest case, a single smart rule can be used by itself in a policy expression. You can also combine a smart rule with others, as well as with other EL constructs, to form more complex expressions.

For example, you can construct a policy that sends an email notification if all of the following conditions occur simultaneously in a server instance or cluster:

• Low JVM free heap percentage

• High number of stuck threads

• High incoming requests queue size

For details about all the smart rules provided by WLDF, see Smart Rule Reference.

Smart Rule Example

The ClusterLowHeapFreePercent smart rule compares the average free heap across all Managed Servers in a cluster by monitoring the value of the JVMRuntimeMBean.HeapFreePercent attribute. A policy expression that uses this smart rule will be evaluated to true if a minimum percentage of Managed Servers in the cluster have an average free heap that is less than a particular threshold value.

The ClusterLowHeapFreePercent smart rule takes the following input parameters:

• Cluster name

• Sampling period — The frequency with which the value of the HeapFreePercent metric is collected

• Retention window — A sliding window of time during which samples are retained. For example, the most recent five minutes.

• percentFreeLimit — A value that represents the low free heap percentage threshold.

• percentServersLimit — A percentage of Managed Servers in the cluster that must have an average free heap that is less than percentFreeLimit to cause the expression to evaluate to true.

The following is an example configuration of the ClusterLowHeapFreePercent smart rule:

wls:ClusterLowHeapFreePercent("myCluster","30 seconds","10 minutes",20,60)

For every Managed Server in myCluster, this smart rule collects the value of the HeapFreePercent every 30 seconds, retaining the most recent 10 minutes of data, and evaluates to true if at least 60 per cent of the Managed Servers in myCluster have an average free heap percentage that is less than 20 per cent.

This smart rule could be configured to fire an action when it evaluates to true, such as sending an email to the system administrator to report that a low free heap condition exists in the cluster. The system administrator can then take remedial action as necessary.

You can use smart rules in conjunction with scaling actions, described in Configuring Elastic Actions, to configure policy based scaling of a dynamic cluster. This capability enables automated elasticity in that cluster. For more information, including a demo that you can download and run, see Policy-Based Scaling in Configuring Elasticity in Dynamic Clusters for Oracle WebLogic Server.

Chaining Policies

Within the same diagnostics system module, the expression in one policy can reference other policies as beans within that expression. In this way, complex policy expressions can be reused and "chained" together to allow the state of one policy to be part of the expression of another. This allows more complex, interrelated policies to be written, while keeping such policy configurations more readable and maintainable.

To allow access to policy states within an expression, you can use the resource bean within the global bean name space for each policy. The resource bean supports a Map attribute named watches, where each key in the map is the name of a policy defined within the same diagnostics system module.

Each value in the policy's map is a bean representing the named policy. These policy beans support a simple Boolean alarm attribute, which has the following semantics:

• If the policy is configured with an alarm type other than None, the alarm attribute returns true if the policy is currently in the alarm state.

• If no alarm type is configured on the policy, the alarm attribute yields the most recently evaluated result.

• If the alarm attribute on a policy bean is accessed before the named policy has successfully completed an evaluation cycle, a NotEnoughDataException is thrown. This occurrence also has the effect of invalidating the expression during that evaluation cycle: the policy isn't disabled, but it is effectively a non-result when it occurs.

Configuring Log Policies

Use log policies to monitor the occurrence of specific messages or strings in the server or domain log. Policies of this type are triggered as a result of a log message containing the specified data being issued.

When creating a log policy, you can use the log bean in a policy expression to obtain access to data to log message fields. See log for details about the available log bean attributes.

The following example looks for a log message indicating that the server is entering the RUNNING state:

  w=cmo.createWatch("ServerLogRunningState")
  w.setExpressionLanguage('EL')
  w.setRuleType('Log')
  w.setRuleExpression("log.messageId == 'BEA-000365' and log.logMessage.contains('RUNNING')")

You can also use java methods and field accessors to access the data in log, since the log bean is a simple JavaBean object. An equivalent policy expression of the above example is:

  w=cmo.createWatch("ServerLogRunningState2")
  w.setExpressionLanguage('EL')
  w.setRuleType('Log')
  w.setRuleExpression("log.getMessageId().contains('000365') && log.getLogMessage().contains('RUNNING')")

Note:

Any log policies that search for the RUNNING state message ID should search for message ID BEA-000365, and not BEA-000360. The message ID BEA-000360 is issued immediately before the state change to RUNNING, and BEA-000365 is issued immediately afterward. WLDF does not start rule evaluation until the server is in the RUNNING state. Therefore, such log policies are able to find only message ID BEA-000365.

Configuring Instrumentation Policies

You use instrumentation policies to monitor the events from the WLDF Instrumentation component. Policies of this type are evaluated as a result of an event being posted by the Instrumentation component, which occurs when code that matches a deployed Instrumentation monitor is exercised.

Instrumentation policy expressions utilize a single bean named instrumentationEvent. This bean provides access to the data that is captured in an Instrumentation event. As with Log, DomainLog, and Collected Metrics policies, you can access data in the Instrumentation event using JavaBean conventions in the policy expression. See the set of fields that are accessible on the instrumentationEvent bean.

The following example shows how to access data in an Instrumentation policy using the instrumentationEvent bean:

instrumentationEvent.payload > 100000000 && instrumentationEvent.monitor == 'Servlet_Around_Service'

This policy triggers when the monitor event is of type “Servlet_Around_Service” and the payload value (in this case, the execution time of the servlet recorded by the Servlet_Around_Service monitor) is greater than 100000000 nanoseconds (100 milliseconds). You can also use java methods and field accessors to access data in instrumentationEvent, since the instrumentationEvent bean is a simple JavaBean object . An equivalent policy expression of the example above can be given as:

instrumentationEvent.getPayload() > 100000000 && instrumentationEvent.getMonitor().equals(‘Servlet_Around_Service')

Example 10-1 shows an example configuration for an Instrumentation policy.

Example 10-1 Sample Configuration for an Instrumentation Policy (in DIAG_MODULE.xml)

  <watch-notification>
    <watch>
    <name>myInstWatch</name>
    <enabled>true</enabled>
    <rule-type>EventData</rule-type>
    <rule-expression>instrumentationEvent.payload &gt; 100000000 &amp;&amp; instrumentationEvent.monitor == 'Servlet_Around_Service'</rule-expression>
    <expression-language>EL</expression-language>
    <alarm-type>ManualReset</alarm-type>
    <notification>mySMTPNotification</notification>
  </watch>
  <smtp-notification>
    <name>mySMTPNotification</name>
    <enabled>true</enabled>
    <mail-session-jndi-name>myMailSession</mail-session-jndi-name>
    <subject xsi:nil="true"></subject>
    <body xsi:nil="true"></body>
    <recipient>[e-mail]</recipient>
  </smtp-notification>
</watch-notification>

Creating Complex Policy Expressions Using WLDF Java EL Extensions

Oracle expects that the library of smart rules packaged with WLDF are sufficient for meeting the needs of creating scheduled policies that evaluate runtime performance data in a server or cluster. However, if you have a specific scheduled policy need that cannot be satisfied by a smart rule, WLDF also provides a set of extensions to Java EL. These extensions are intended for use in policies that evaluate very specific characteristics or trends in metrics collected from runtime MBean servers in your WebLogic domain.

The contents of this section are targeted to developers who are knowledgeable of complex programming techniques. Experience with Java EL is highly recommended.

Using WLDF Beans and Functions

WLDF leverages Java EL as the language for writing policy expressions. Java EL is a standard, extensible, and robust scripting language. WLDF has adopted and extended Java EL to provide access to WebLogic diagnostic data and events for writing policy expressions. WLDF provides a set of functions and JavaBean objects for writing policy expressions that use the following diagnostic data and events:

• WebLogic Runtime MBean data

• WebLogic Logging events

• WebLogic Instrumentation events

You can utilize all the features available within Java EL in conjunction with these WLDF extensions to write policy expressions. Collected metrics based policies, which are a type of scheduled policy, can use WLDF-provided beans and functions within their policy expressions. These beans are JavaBean objects that can obtain access to common WebLogic Server JMX data sources, such as the following:

• WebLogic Server Runtime MBean Server

• Domain Runtime MBean Server

• JVM platform MBean server

The following sections explain how to configure collected metrics based policies using beans and functions:

• Writing Collected Metrics Policy Expressions Using Beans
  
• Writing Collected Metrics Policy Expressions Using Functions
  

Writing Collected Metrics Policy Expressions Using Beans

Table 10-2 summarizes the beans provided by WebLogic Server. For complete reference information about each of these beans, see WLDF Beans Reference.

Table 10-2 Beans Provided by WebLogic Server

Name	Prefix	Scope	Summary
runtime	wls	Only available from partition scope diagnostic system module deployments and partitions	Provides access to MBeans in the local WebLogic Server Runtime MBean Server. These MBeans include both the read-only configuration MBean and RuntimeMBean instances.
domainRuntime	wls	Administration Server	Provides access to MBeans on the Domain Runtime MBean Server (Administration Server only).
clusterRuntime	wls	Administration Server	Provides domain-wide access to cluster member data (Administraton Server only).
platform	wls	Administration Server or Managed Server	Provides access to the JVM's platform MBean server. Note that in the majority of cases, the platform bean is functionally equivalent to the runtime bean: WebLogic Server uses the JVM's platform MBean server to contain the WebLogic run-time MBeans by default.
partition	wls	Partition scope WLDF diagnostic system module deployments.	Provides access to partition scope metrics. This bean is available only to policies that are configured in a diagnostic system module that is deployed in the same partition to which this bean is scoped. Note: WebLogic Server Multitenant domain partitions, resource groups, resource group templates, and virtual targets are deprecated in WebLogic Server 12.2.1.4.0 and will be removed in the next release.
resource	n/a	Administration Server, Managed Server, and partitions	Provides access to beans and state information within a diagnostic system module deployment. Access is restricted to policies that are configured within the same diagnostic system module.

• Accessing MBean Data in Collected Metrics
  
• Working with Complex MBean Attributes
  
• Performing Bulk Queries on Collected Metrics from MBeans
  

Accessing MBean Data in Collected Metrics

The beans described in Table 10-2 provide access to WebLogic Server Runtime MBean metrics. In policy expressions that use Java EL, metric data from each of these runtime MBeans is accessed using a WLDF-provided bean using the following syntax:

wls.bean-name.attribute-or-operation.attribute-or-operation…

All EL-based policy expressions that use the WLDF beans must begin with the namespace prefix wls . The prefix wls is similar to a namespace that contains all the WLDF beans that can be used in the policy expressions. Beans and their attributes and methods are accessed using standard JavaBean conventions. The following example shows a simple policy expression that returns true when the value of HeapFreePercent attribute of JVMRuntimeMBean is less than 20:

wls.runtime.serverRuntime.JVMRuntime.heapFreePercent < 20

The preceding policy expression example accesses the value of HeapFreePercent in the following sequence:

1. The runtime bean is accessed from the wls bean namespace.

   The runtime bean provides an entry point into the metrics collected by the local runtime MBean and also into the read-only configuration MBean data in the WebLogic Server Runtime MBean Server.

2. The serverRuntime attribute is accessed from the runtime bean.

   The serverRuntime attribute of the runtime bean corresponds directly to the ServerRuntimeMBean instance in the local running server instance wherever the expression is being evaluated.

3. The JVMRuntime attribute, which corresponds to the JVMRuntimeMBean instance under the local ServerRuntimeMBean, is accessed from the serverRuntime bean.

4. The heapFreePercent attribute is accessed from the returned JVMRuntime instance.

From the runtime bean, runtime metrics and monitoring data are available through the serverRuntime attribute, and the domain attribute provides access to the current configuration data in the local read-only DomainMBean tree. This access allows policies to examine the current in-memory configuration within a policy expression.

MBeans that are accessed as bean attributes from the WLDF-provided expression beans have read-only access to most of the attributes and some operations available to the expression as defined in the MBean Reference for Oracle WebLogic Server, with some exceptions for security purposes.

Note:

There are slight differences in syntax between JMX and JavaBean conventions when accessing attributes. For example, JavaBean conventions for accessing the JMX attribute HeapFreePercent require using “camel-case” syntax. When using JMX, the attribute is accessed by the name HeapFreePercent. However, in EL expressions, the same attribute is accessed as heapFreePercent.

Working with Complex MBean Attributes

Some MBean attributes return complex objects; for example, the HealthState attribute of the ServerRuntimeMBean. Such attributes can be accessed using JavaBean conventions. In the following example, the policy expression returns true if the health state of the server is a non-zero value:

wls.runtime.serverRuntime.healthState.state != 0

Working with Array Attributes

Many WLDF bean attributes return arrays of child MBeans. To work with collections, such as arrays, Java EL provides the stream operator to convert arrays and lists into stream objects that can be fed into other Java EL and WLDF functions and operators. In the following example, the policy expression examines the state of all JDBCDataSourceRuntimeMBean instances in the local server instance, and returns true if any of them are in the Overloaded state:

wls.runtime.serverRuntime.JDBCServiceRuntime.JDBCDataSourceRuntimeMBeans.stream().anyMatch(ds -> ds.state == “Overloaded”)

The policy expression executes in the following sequence:

1. The JDBCServiceRuntimeMBean child is accessed from the ServerRuntimeMBean.

2. The array attribute JDBCDataSourceRuntimeMBeans is accessed from the JDBCServiceRuntimeMBean.

3. The Java EL stream operator is utilized to convert the array to a stream so that it can be used with WLDF and standard Java EL collection operations.

4. The anyMatch collection operation is used to look for the Overloaded state on any of the returned JDBCDataSourceRuntimeMBean instances.

5. If the anyMatch operation matches the Overloaded state, returns true.

Performing Bulk Queries on Collected Metrics from MBeans

The MBeans defined in Table 10-2 are used in collected metrics policy expressions. All of these beans support a query method that allows to perform a query for a set of MBean attribute values against a homogeneous set of MBeans.

The method takes the following syntax:

query(target-list, object-name-pattern, attribute-expression)

The query method returns an iterable list of values that is obtained using the attribute-expression on each matching MBean instance.

Table 10-3 Method Parameters

Parameter	Description
target-list	This argument is applicable only for domainRuntime bean which is available only for policies executing on the Administration Server. The bean supports an overloaded variant that takes an array of targets. It is a list of servers or clusters in the domain. The argument allows the policy expression to examine MBean values across the domain in the same expression.
object-name-pattern	This argument takes any valid JMX ObjectName pattern that is specified as a string value enclosed by single quote (') characters. For example: 'com.bea:Type=ServletRuntime,*'
attribute-expression	This argument is a quoted EL subexpression that is used to access an attribute from each of the MBeans matching the object-name-pattern argument. The attribute-expression argument can be either of the following types: A simple attribute available on the MBean. An attribute of a complex type that uses a JavaBean-style expression to access the values within that complex structure. Note: It is expected that attribute-expression ultimately resolves to a single scalar value, and not a complex structure.

The values returned by the query method can be used as a part of the larger policy expression that examines those values.

Note:

The intended use of the query method is to operate against a homogeneous set of MBean instances, but there is no enforcement mechanism to ensure that the specified MBeans must all be of the same type. Therefore, if you do specify an object-name-pattern that encompasses MBeans of different types, errors can result when the policy expression is evaluated.

Example 10-2 Examples of Using the query Method

Table 10-4 lists some examples of using the query method in policy expressions.

Note:

The examples show how to use the query method and are not complete policy expressions.

Table 10-4 query Method Examples

Example	Description
wls.runtime.query('com.bea:Type=ServletRuntime,*', 'ExecutionTimeAverage')	The query method is used for all the instances of ServletRuntimeMBean in the local server and returns the value of ExecutionTimeAverage for each instance in the returned iterable stream.
wls.domainRuntime.query(['cluster1'], 'com.bea:Type=ThreadPoolRuntime,*', 'PendingUserRequestCount')	The domainRuntime bean is used to query all values of PendingUserRequestCount across all instances of ThreadPoolRuntimeMBean in the cluster cluster1. Any values found are returned in the Iterable set returned by the method call.

The use of query method in policy expression and the result set are represented in the following illustration:

Figure 10-1 Result Set of query in Policy Expression

Description of "Figure 10-1 Result Set of query in Policy Expression"

The following is a complete example of a policy expression that uses the query method to determine whether the StuckThreadCount attribute on any WorkManagerRuntimeMBean in the local WebLogic Server instance is greater than zero:

wls.runtime.query('com.bea:Type=WorkManagerRuntime,*', 'StuckThreadCount').stream().anyMatch( x -> x > 0 )

The values of StuckThreadCount for all instances of WorkManagerRuntimeMBean are queried, and each value is examined to see if it is greater than zero, which indicates a stuck thread in the server. The stream collection operation is part of the Java EL standard, and is used for converting an iterable set into a stream that can be used with Java EL collection operations, such as anyMatch in the example.

Writing Collected Metrics Policy Expressions Using Functions

In addition to the bundled functions and collection operations that come with Java EL by default, there are also a set of WLDF-provided functions for use within policy expressions for common operations with metric data and for retaining a set of metrics with history.

The set of WLDF-provided functions includes:

• wls:tableChanges

• wls:tableAverages

• wls:extract

• wls:average

• wls:changes

• wls:aliveServersCount

For complete details about each EL function provided by WLDF, see Functions Reference.

Functions are invoked using the prefix wls:

wls:<function-call>

For example, wls:aliveServersCount('cluster1') invokes the aliveServersCount() function provided by WLDF for the cluster cluster1.

Collection Operations

WLDF also provides a set of collection operations that can be invoked similar to the collection operations provided by Java EL. The set of WLDF-provided collection operations includes:

• tableAverages

• percenMatch

• collection

• flatten

• Examining Trends in Metric Values over Time
  
• Extracting and Examining Collected Metrics in Policy Expressions
  
• Lifecycle of Data Collection
  

Examining Trends in Metric Values over Time

You can look for trends in metric data over time instead of assessing the instantaneous values. Use the wls:extract function to extract a table of time series from a specified set of input sources, based on a specified sampling rate schedule and time window. 

The extract function has the following syntax:

wls:extract(sources, sampling rate, retention window)

The method returns an iterable set that consists of a two dimensional set of results. The metric input to the function comes from multiple MBean instances during the course of a specific interval of time defined by the retention window parameter. The resulting data is similar to a table where each row is a set of values from a particular MBean instance over the time window.

Parameters

Table 10-5 Parameters Description for extract() Function

Parameters	Description
sources	Set of metric sources, which can be identified as a query method or as a quoted Java EL expression.
sampling rate	String that identifies the frequency with which data is collected. You can specify this string as hours, minutes, or seconds. The syntax is flexible, allowing you to specify 30 seconds, for example, as “30s”, “30sec”, or “30 seconds”. Note: The frequency only applies to the rate of collection of the metric, and is independent of the overall policy evaluation schedule.
retention window	String that identifies the retention window over which to observe values from the sources input with syntax identical to the sampling rate parameter. It implements the sliding window algorithm in which the oldest data in the set is aged out when the array is full. See retention window.

Example 10-3 Examples of Using the extract Function

Table 10-6 lists example usages of the extract function.

Note:

The examples show how to invoke the extract function and are not complete policy expressions.

Table 10-6 extract Function Examples

Example	Description
wls:extract("wls.runtime.serverRuntime.threadPoolRuntime.pendingUserRequestCount", "30s", "2m")	The extract function is invoked with an EL expression as the first argument which observes and collects the values of the PendingUserRequestCount attribute on the ThreadPoolRuntimeMBean at 30-second intervals and retains them over a period of 2 minutes. In this example, ThreadPoolRuntimeMBean is a singleton, and only the local WebLogic Server instance is monitored. Therefore, only a single row of values is returned in the table of values.
wls:extract(wls.runtime.query("com.bea:Type=ThreadPoolRuntime,*", "PendingUserRequestCount"), "30s", "2m")	The extract function is used with the result of a query method invocation as input.
wls:extract(wls.domainRuntime.query(['cluster1'], 'com.bea:Type=ThreadPoolRuntime,*', 'PendingUserRequestCount'), '30s', '2m')	The extract function is used with the query method of the domainRuntime bean to collect the value of PendingUserRequestCount attribute on all ThreadPoolRuntimeMBean instances on every server in cluster1. The result set for this call consists of a row of values for each ThreadPoolRuntimeMBean instance in each active server instance in cluster1 .

Extracting and Examining Collected Metrics in Policy Expressions

The extract function returns a table of scalar values. You can use any collection operation to examine or manipulate the result set. WLDF provides more collection operations that are intended for use with the data returned from extract function, such as tableAverages, percentMatch, collection, and flatten.

Operations	Description
tableAverages	Computes the average value for each row in the table.
percentMatch	Examines all the computed averages from tableAverages.
collection	Returns the two dimensional set of values in tabular form, which can be then converted to Java EL collection stream using the stream operator and can be directly manipulated in other Java EL collection operators.
flatten	Converts the two dimensional set of values returned by extract function into a linear collection of values.

The result of extract can then be fed into other functions or operations as part of an overall policy expression.

In the following example of a policy expression, the extract function collects the value for the PendingUserRequestCount attribute across the servers in cluster1. The result is combined with the tableAverages and percentMatch collection operations to produce a boolean value.

wls:extract(wls.domainRuntime.query({'cluster1'}, 'com.bea:Type=ThreadPoolRuntime,*', 'PendingUserRequestCount'), '30s', '2m').tableAverages().stream().percentMatch(pendingCount -> pendingCount > 100) > 0.75

This policy expression returns true when the average value of the attribute PendingUserRequestCount over the 2-minutes window is greater than 100 on 75% of the servers in cluster1. The policy expression executes in the following sequence:

1. The extract function creates a table of values for the attribute PendingUserRequestCount, where each row is one set of values from a server in cluster1 over a 2-minutes window.

2. The tableAverages operation computes the average value over the 2-minutes window for each row in the table returned by the extract function.

3. stream is a standard Java EL collection operation used to convert the vector result of tableAverages to a Java EL stream.

4. The percentMatch operation examines all the computed averages from tableAverages, and computes the percentage of values in that set that are greater than 100.

5. The result of percentMatch is a value between 0 and 1 and is compared with 0.75, the desired threshold.

Lifecycle of Data Collection

The extract function extracts data from a specified input source over a defined period of time. When the extract function is first encountered in an expression by the WLDF policy engine, it starts the collection of the desired metrics indicated in the policy expression. Samples are collected by the policy engine until the policy using the extract function is disabled or undeployed.

Policy expressions that use the extract function is not evaluated until enough data has been collected for the desired metrics to satisfy the sliding window interval specified in the invocation. If the function invocation specifies that a 5-minutes window of data is required, then 5 minutes of data collection must take place from the moment the policy is deployed before the expression can be successfully evaluated.

In the following example, the expression does not evaluate until 2 minutes of data for the PendingUserRequestCount attribute is collected.

wls:extract(wls.runtime.query("com.bea:Type=ThreadPoolRuntime,*", "PendingUserRequestCount"), "30s", "2m")