Set Up Alert Rules

7 Set Up Alert Rules

Alert rules allow you to set up metric thresholds and define how to be notified if an entity is down or metrics thresholds are exceeded.

You must have Oracle Management Cloud Administrator privileges to set up Alert Rules.

Topics

Typical Workflow for Setting Up Alert Rules

Set up Alert Rules for your monitored infrastructure to trigger alerts on impending issues.

Table 7-1 Typical Workflow for Setting Up Alert Rules

Task	Description	More Information
Select the entities for which you want to set up alerts.	Choose individual entities or entity types to set up alerts on.	Set Up Alert Thresholds and Notifications
Specify the alert conditions and notification options.	Select threshold conditions that apply to your entities in order to monitor performance. Select your notification options for performance and availability alerts.	Set Up Alert Thresholds and Notifications

For a brief video demonstrating how to create an Alert Rule, see Set Up an Alert Rule.

Set Up Alert Thresholds and Notifications

Oracle Management Cloud (OMC) collects performance and availability metrics for all entities set up for monitoring. The alerts sub-system informs you of availability or performance problems. Alert Rules enable you to define how alerts are triggered (if they are not automatic) and how you get notified.

Setting up alerts thresholds or notifications requires an understanding of how entities are defined in Oracle Management Cloud. Each entity is defined using a multi-level hierarchical model as pictured below.

This image shows the entities definition model

Each entity or entity type is defined by a set of characteristics, it has a parent and may have other children. For example, a Generic Host is an operating system (OS) independent Target and it has children entities that are specific OS Hosts, like Linux, Windows and so on. . The metrics collection functionality takes advantage of this model so each monitored entity has entity-specific metrics as well as metrics inherited from each level it descended from. For example, OMC collects metrics at level 3 that are common to all Generic Hosts, independent of the vendor. A Linux Host, since its parent is a Generic Host, inherits all the metrics collected for Generic Hosts and its ancestors, as well as Linux specific ones, if any.

Availability problems (for example, an entity is down, or agents or hosts are unavailable) are automatically detected by Oracle Infrastructure Monitoring and availability alerts are automatically generated. Performance metrics deviations, however, can be monitored by setting up Alert Rules. In addition, Alert Rules allow you to specify how to be notified when alerts are triggered.

Create an Alert Rule

To set up Alert Rules for your enterprise, you must be logged in to the services as an Oracle Management Cloud Administrator user. Navigate to the main service menu, go to the Alert Rules page and select Create Alert Rule.

Alert Rules

Choose a sample rule if any of the sample rules provided seem appropriate for your environment. Sample rules help you populate the rule definitions with values typical for a particular entity type.
Enter a name and optionally a description for your new alert rule.
Alert rules can apply to one or more entity types (based on the entities definition model described above), groups, or they can apply to an individual entity.
You can choose one of these options:
- Entity Types if the rule will apply to a set of entities of the same type. This includes specific entity types (for example, MySQL database or Tomcat, which are entities with no children in the definition model above) or generic entity types. Here are some examples of generic entity types:
  
  —A Relational Database, an entity type with several children in the definition model above; choose this to indicate that this rule applies to all entities that have the Relational Database as a parent.
  
  —A Hosted Target; choose this to indicate that this rule applies to all entities that run on a host, such as databases and application servers.
  
  —A Target; choose this if the alert rule will apply to all Hosted Targets and Generic Hosts entities and their children.
  
  —Any other custom defined entity type.
- Individual Entities, if the rule will apply to specific entities defined in your monitoring environment.
  
  Your selection will determine the options that follow in selecting alert conditions.
Best Practice Tip: As mentioned above, alert rules can also apply to groups or composite entities. You can have an alert rule automatically apply to specific entities, by specifying a group, such as a dynamic group, as the entity for the alert rule. As more entities are monitored, you can add them to the group and the alert rule will automatically apply. You can further streamline this by creating dynamic groups based on tags. When you add an entity and also add the appropriate tag, the entity automatically joins the dynamic group and thus be part of the alert rule.

Specify the alert conditions. Availability monitoring is based on the availability of an entity. For performance monitoring, alert conditions are based on set thresholds of various performance metrics. Select one of the following then click Save to save these settings:

Fixed Threshold condition to generate an alert when a metric exceeds a set metric threshold value. From the metric drop-down, choose the metric of interest and the operator. Use the graphical display (chart) of historical values of that particular metric to guide you in entering the Warning and Critical values. Hover over the graph to see the metric values at various points in time. If metrics are is associated with key values, you can set individual warning and critical thresholds values for each key value. For example, the File System Space Available % host metric has mount points as key values. You can set warning and critical thresholds for each mount point defined on that host.

The alert will be triggered immediately, unless you specify the number of minutes that the value should remain outside the threshold before an alert is generated.

You can also use a statistical value of the metric (Average, Sum, Minimum, Maximum, Count) as the value to be used when evaluating against specified metric thresholds. For example, if CPU utilization % has a warning threshold of 70% and a critical threshold of 90%, you can specify that when the Average CPU utilization value remains over 70% for more than 30 minutes, generate an alert.

You can select a standard alert message or, select to customize the message when the alert first triggers as well as the message sent when the alert clears. Use the provided tokens, encapsulated as %{token_name}%, to construct a meaningful custom alert message. This table lists the tokens you can use:

Table 7-2 Valid Tokens

Token Name	Description
sys.entityDisplayName	Name of the entity
sys.entityId	Entity ID
sys.value	Value of the metric that triggered the alert
sys.criticalThreshold	Critical threshold used in evaluating the alert condition
sys.warningThreshold	Warning threshold used in evaluating the alert condition
sys.operator	Operator used in evaluating the alert condition
sys.conditionName	Alert condition name
sys.ruleName	Alert Rule name
sys.slidingWindowSize	Number of minutes that the metric crossed the threshold before the alert was triggered
sys.defaultMessage	Show the default message normally generated for the alert condition. Additional text can be added before and after this token. The additional text can include other alert message tokens. Example `"messageTemplates": { "warning": "Warning Default message : % {sys.defaultMessage} %. Please contact the administrator to report the issue for entity %{sys.entityDisplayName}%. "}` Generated Message Warning Default message : TestMetric1_num for TestMsgTempEntity is 11; it is greater than expected value of 10 . Please contact the administrator to report the issue for entity TestMsgTempEntity.

For a complete list and descriptions of all metrics collected for each entity type, see List of Supported Entities in Metric Reference for Oracle Infrastructure Monitoring.

Alert Condition Notes

When creating an alert condition as part of an Alert Rule, you can specify notes that contain runbook instructions (or a runbook URL) that should be followed when responding to an alert.

Other Types of Alert Conditions
In addition to alerting for specific metric thresholds, you can also have Oracle Infrastructure Monitoring send alert notifications based on the following conditions:

Alert Anomalies: Alerts can be triggered when metric threshold values occur outside the bounds of expected normal values. Oracle Infrastructure Monitoring determines what is normal behavior for a monitored entity by creating metric baselines. A baseline is an expected range of values for a metric based on its past performance. Baselines are automatically tracked and adjusted for seasonality, or the ability to adjust baseline values according to the time of day, or day of week once there is sufficient historical metric data.

When monitoring for anomalies, you may not want to raise an alert every time an anomalous condition occurs. Instead, you may only want to raise an alert if a metric anomaly occurs for a specific duration. For example, you don’t care if there is a brief spike in a host’s CPU usage, but you do want to be alerted if CPU usage remains abnormally high for 5 minutes or more. When defining an Alert Condition, you can specify duration-based anomalies.

Early Warning: Early warning conditions will trigger an alert when, based on historical metric data, Oracle Infrastructure Monitoring predicts the provided alert threshold will be crossed in the future.
Availability alerts are automatically raised by Oracle Management Cloud when entities are detected to be Down or Not Heard From in the case of hosts, agents, gateways and data collectors. Down alerts or Not Heard From alerts on host and agent entities are alerts of Fatal severity. To get notifications for alerts of Fatal severity, choose the Availability alert condition.

Specify the recipients for the alert notifications by choosing the appropriate notification channel. See Set Up Notification Channels for more information.
Customizing When Notifications are Sent: By default, notifications are sent when alerts are first raised, when they worsen in severity, or when they are closed. If the default notification settings don't fit your needs, you can change them by setting various options under Advanced Options. You can change:
- The alert conditions under which a notification should be sent (use default or on every severity change).
  
  Note:
  This option only applies to Email, Mobile, and Slack notification channels.
- The alert conditions under which a notification should be suppressed (based on selected severities).
  
  Note:
  This option only applies to Email, Mobile, and Slack notification channels.
- Whether or not a notification should be sent repeatedly. Repeat notifications allow administrators to be notified at specified intervals until an alert is either cleared or acknowledged or the number of Maximum number of repeat notifications has been reached.
  
  Note:
  This option only applies to Email notification channels.
Click Save to save the alert rule.

Once the alert rule has been created, it appears in the table on the Alert Rules page where it can be enabled, disabled, or deleted. Disabled alert rules do not create alerts or send notifications. Alert rules can also be enabled/disabled while editing/creating the alert rule.

Set Up Notification Channels

When an Oracle Management Cloud alert is raised, worsens in severity, or clears, you may want to be actively notified through email, by push notifications (mobile devices), or have a third-party application take action. These types of notifications are defined using notification channels.

Types of Notification Channels

Classes of notification destinations are called notification channels, Notification channels allow you to set up and reuse functional groups of notification recipients, such as regional administrators, IT managers, or other Web servers without having to specify large numbers of individual destinations repeatedly. Once you set up a notification channel, you can reuse the channels across different alert rules.

Available of notification channels:

Email: Allows you to manage/logically organize the email recipients of alert notifications. For example, you can create an email channel for the On-Call Team containing the email addresses of the IT operators that staff the on-call team, or a DBA Team email channel containing all email addresses of database administrators in your department.

Create an Email Notification Channel
Mobile: Allows you to send push notifications to one or more mobile phones associated with Oracle Management Cloud users.

Create a Mobile Notification Channel
Integration (WebHook): Allows you to send HTTP POST (WebHook) messages to a destination URL. A WebHook notification message contains a JSON payload that specifies pertinent information to a destination Web application.

Create a WebHook Notification Channel (Integration)
PagerDuty: Allows you to integrate Oracle Management Cloud's monitoring capability with PagerDuty's incident response services.

Create a PagerDuty Notification Channel
ServiceNow: Allows you to integrate Oracle Management Cloud’s monitoring capability with ServiceNow’s incident management services.

Create a ServiceNow Notification Channel
Slack: Allows you to send Oracle Management Cloud notifications to a specified Slack channel.

Create a Slack Notification Channel

Create an Email Notification Channel

Notification channels can be defined while creating/editing an alert rule from the Alert Rules UI or via the Notification Channel UI.

To create an email notification channel from the Notification Channel UI:

From the Management Cloud side menu, select Administration, and then Notification Channels.
On the Notification Channels page, click, Create Notification Channel. The four notification channel options are displayed.
Click on the desired notification channel type and specify the requisite information.

The most common method of notification is via email. Using an email notification channel lets you specify the channel and rather than having to specify individual emails every time. For an email notification channel, you specify:
- Channel Name: An intuitive name for this notification channel.
- Email Addresses: A comma-separated list of recipient email addresses .
- Email Language: The language that will be used for the date on which the alert was raised
- Email Timezone: The timezone that will be used for the time on which the alert was raised

Create a Mobile Notification Channel

In order to receive push notification on your mobile device, the Oracle Management Cloud Mobile application must be installed on your device and signed into. As shown below, the Oracle Management Cloud Mobile application offers anywhere access to information about the entire IT infrastructure managed by Oracle Management Cloud.

Graphic shows home and alert pages of the Oracle Management Cloud mobile app.

From the Management Cloud side menu, select Administration, and then Notification Channels.
On the Notification Channels page, click, Create Notification Channel. The four notification channel options are displayed.
Click on the desired notification channel type and specify the requisite information.

For a Mobile notification channel, you specify:
- Channel Name: An intuitive name for this notification channel.
- OMC User Names: A comma-separated list of recipient Oracle Management Cloud users.

Create a WebHook Notification Channel (Integration)

In addition to notifying people, Oracle Management Cloud can also send relevant information to third-party Web applications (such as Slack or Hipchat) when an alert is raised, thus allowing you extend Oracle Management Cloud functionality by having third-party applications carry out actions in response to an Oracle Management Cloud alert notification. This type of system integration is achieved using WebHooks (HTTP POST message containing a JSON payload that is sent to a destination URL).

From the Management Cloud side menu, select Administration, and then Notification Channels.
On the Notification Channels page, click, Create Notification Channel.
Click on the desired notification channel type and specify the requisite information. To define a WebHook notification channel, you need to specify:.
- URL: Destination URL where the JSON payload is sent. If you want to specify a port number, please note that only port numbers 80 and 443 are supported.
- Authentication Type: Type of authentication used when accessing the destination URL. You can specify None or Basic (Username and Password). When specifying Basic credentials, you choose whether you want to use Existing Credentials, or create New Credentials. For every new credential, you specify a Credential Name that will be used for future identificaiton. If you select the Existing Credentials option instead of New Credentials, the Credential Name will appear as a selectable item in the drop-down lilst.
- Setting HTTP Header and Payload Values
  
  A field value can be a fixed string or token (i.e., payload variable) or combination of both.
  
  When the value is a combination of fixed strings and tokens, the following rules must be followed.
  - Value should start with ${
  - Value should end with }
  - Each fixed string should be enclosed inside a pair of single quotes.
  - Token names should be entered without enclosing them inside ${ and }
  - To enter consecutive multiple tokens, place a pair of single quotes between each pair of tokens for separation, e.g., alert.id''alert.message.
  Examples:
```
${'Alert ID: 'alert.id}
${'Alert ID: 'alert.id' Alert Message: 'alert.message}
${'Two tokens with a space between them: 'alert.id' 'alert.message}
${'Two tokens without any space between them: 'alert.id''alert.message}
```
- HTTP Headers: Headers to be included with the message. For each header, value can be a fixed string, payload variable, combination of both. When the value is a combination of fixed strings and payload variables, follow the conventions listed in the previous bullet (Setting HTTP Header and Payload Values).
- Payload: The JSON payload specifying the requisite information sent to the destination URL. Payload consists of the following JSON code:
```
{
    "alertId": "${alert.id}",
    "ruleName": "${rule.ruleName}",
    "conditionName": "${rule.conditionName}",
    "updateType": "${updateType}",
    "message": "${alert.message}",
    "severity": "${alert.severity}",
    "time": "${alert.time}",
    "eventName": "${alert.eventName}",
    "alertDetailUrl": "${alert.detailUiUrl}",
    "entityId": "${entity.id}",
    "entityName": "${entity.name}",
    "entityType": "${entity.type}",
    "entityDisplayName": "${entity.displayName}",
    "entityHostName": "${entity.hostName}"
}
```

The JSON payload provides 14 tokens that are used to pass information to the destination URL. You can modify the JSON payload to send specific information required by the receiving Web application.

Token Name	Description
${alert.id}	ID of the alert. Example: 1234
${updateType}	Type of update. Example: created, updated, closed
${rule.ruleName}	Name of the alert rule that triggered the alert.
${rule.conditionName}	Name of the condition triggering the alert.
${alert.eventName}	Name of the event.
${alert.message}	Text of the alert message.
${alert.severity}	Severity of the alert. Example: warning, critical
${alert.time}	Date/time when the update occurred Example: 2017-07-05T20:22:04.134Z
${alert.detailUiUrl}	URL where explicit details about the alert can be found.
${entity.id}	ID of the entity.
${entity.name}	Name of the entity impacted (internal name).
${entity.displayName}	Display name of entity impacted.
${entity.type}	Type of the entity. Example: APM Server Request, host(linux)
${entity.hostName}	Host name of the host where the entity resides.
${entity.shortHostName}	Shortened host name. This name does not include the domain name.

Create a PagerDuty Notification Channel

When an Oracle Management Cloud alert is raised, you can have that alert sent to PagerDuty for incident management. Oracle Management Cloud updates the same ticket when the alert is updated by PagerDuty, and closes the ticket when PagerDuty clears the alert. Integrating Oracle Management Cloud with PagerDuty is a two-phase process: Configure PagerDuty to receive Oracle Management Cloud alerts and then create a PagerDuty notification channel.

Obtaining the PagerDuty Authorization and Integration Keys

You need to obtain both an Authorization Key and an Integration Key from PagerDuty before you can define the PagerDuty notification channel.

Go to the PagerDuty authentication page and log in.
From the PagerDuty Configuration menu, select API Access. The API Access Keys page displays.
Click Create New API Key. The Create API Key dialog displays.
Enter a useful description and ensure the V2 Current option is selected. Click Create Key. The New API Key dialog displays.
Copy and save the API key. You will need this key later when defining the PagerDuty notification channel (Authorization Key). Close the dialog.

Next, you need to create a service key (Integration Key).
From the PagerDuty Configuration menu, select Services.
Click Add New Service. The Add a Service page displays.
Enter a name for your service and a description.
Under Integration Settings, select Oracle Management Cloud from the Integration Type drop-down menu.

Note:
To narrow the list of menu options, you can enter OMC in the search field.
Make modifications to the Incident Settings, if necessary.
Click Add Service. The Service Details page for your newly defined service displays.
Copy and save the Integration Key. You will need this key later when defining the PagerDuty notification channel (Integration Key).

Now that you have both the PagerDuty Authorization (API key) and Integration keys, you define the actual PagerDuty notification channel.
From Oracle Management Cloud, navigate to the Notification Channels page and choose the PagerDuty notification channel type. The Create PagerDuty Channel dialog displays.

Enter the requisite information and click Create.

For a PagerDuty notification channel, you specify:
- Channel Name: An intuitive name for this notification channel.
- Existing or New Credentials: Choose the New Credentials option.
- Authorization Key: The PagerDuty API Key created earlier.
- Integration Key: The PagerDuty Integration Key created earlier.

Create a ServiceNow Notification Channel

You can send an Oracle Management Cloud alert to ServiceNow for incident management.

From the Management Cloud side menu, select Administration, and then Notification Channels.
On the Notification Channels page, click, Create Notification Channel.
Click on the desired notification channel type and specify the requisite information,.

Enter the ServiceNow username, password, and instance name as shown in the Create ServiceNow Channel dialog.
Specify the payload to be sent to ServiceNow (Default Payload or Custom Payload). See below for more information about ServiceNow payloads.

Note:

When specifying the ServiceNow Instance, do not specify the full domain. For example, if the full domain is myinstance.service-now.com, then you only need to enter myinstance.

ServiceNow Payloads

Oracle Management Cloud sends notifications to ServiceNow channels when an alert is created or updated or closed. Payload for the following three types of notifications differs from one another.

Default Payload Sent to ServiceNow

Oracle Management Cloud sends the following information by default The following table lists the default field values.

Field Name	Default Value	When it is sent:
impact	2	The alert is created.
urgency	2	The alert is created.
short_description	`${alert.message}`	All updates
description	`${alert.message}`	All updates
comments	${'[code]<div style=\"padding:0 1em\"><h3><b>Incident Details</b></h3><dl style=\"padding-left:1em\"><dt>Alert ID:</dt><dd><a href=\"'alert.detailUiUrl'\" target=\"_blank\">'alert.id'</a></dd><dt>Event Name:</dt><dd>'alert.eventName'</dd><dt>Event Message:</dt><dd>'alert.message'</dd><dt>Entity Name:</dt><dd>'entity.name'</dd><dt>Entity ID:</dt><dd>'entity.id'</dd><dt>OMC Severity:</dt><dd>'alert.severity'</dd><dt>Rule:</dt><dd>'rule.ruleName'</dd><dt>Note:</dt><dd>'rule.note.text'</dd><dt>Raised On:</dt><dd>'alert.time'</dd></dl><h4>Created by Oracle Management Cloud ServiceNow Connector</h4></div>[/code]'}	All updates
state	6	The alert is cleared.
close_code	Solved (Work Around)	The alert is cleared.
close_notes	Management Cloud Resolution	The alert is cleared.

Customized Field Values

A field value can be a fixed string, a token, or combination of both.

When the value is a combination of fixed strings and tokens, the following rules must be followed.

Value should start with ${
Value should end with }
Each fixed string should be enclosed inside a pair of single quotes.
Token names should be entered without enclosing them inside ${ and }
To enter consecutive multiple tokens, place a pair of single quotes between each pair of tokens for separation, e.g.,alert.id''alert.message.

Examples:

${'Alert ID: 'alert.id}
${'Alert ID: 'alert.id' Alert Message: 'alert.message}
${'Two tokens with a space between them: 'alert.id' 'alert.message}
${'Two tokens without any space between them: 'alert.id''alert.message}

Create a Slack Notification Channel

You can send a Oracle Management Cloud notifications to an existing Slack channel by first creating a WebHook in Slack and then creating an Oracle Management Cloud notification channel.

Create a WebHook in Slack

This channel will receive all alerts from Oracle Management Cloud.

Create the incoming WebHook configuration.

https://<youraccount>.slack.com/apps/A0F7XDUAZ-incoming-webhooks

Click Add Configuration.
Select channel under Post to Channel.
Click Add Incoming Webhooks Integration.
From confirmation page, capture the WebHook URL. This URL will be required when you create the notification channel in Oracle Management Cloud.

You will need to remember the channel name as this will be used when you create the Slack notification channel in Oracle Management Cloud.

Create a Slack Notification Channel

From the Management Cloud side menu, select Administration, and then Notification Channels.
On the Notification Channels page, click, Create Notification Channel.
Click on the Slack notification channel type and specify the requisite information,.

Enter the Channel Name (name of the Slack channel to be displayed in the Oracle Management Cloud console), URL (the Slack URL), and Team Channel (the Slack team channel you are sending the notification to).
Click Create. A test message will be sent to Slack to confirm that the integration is working.