3 Notifications

The notification system allows you to notify Enterprise Manager administrators of alerts, policy violations, and the status changes of job executions. In addition to notifying administrators, the notification system can perform actions such as executing operating system commands (including scripts) and PL/SQL procedures when an alert is triggered. This capability allows you to implement automatically specific IT practices under particular alert conditions. For example, if an alert is generated when monitoring the operational (up/down) status of a database, you may want the notification system to automatically open an in-house trouble-ticket using an OS script so that the appropriate IT staff can respond in a timely manner.

By using Simple Network Management Protocol (SNMP) traps, the Enterprise Manager notification system also allows you to send traps to SNMP-enabled third-party applications such as HP OpenView. Some administrators may want to send third-party applications a notification when a certain metric has exceeded a threshold.

This chapter covers the following:

Setting Up Notifications

All Enterprise Manager administrators can set up e-mail notifications for themselves. Super Administrators also have the ability to set up notifications for other Enterprise Manager administrators.

Setting Up a Mail Server for Notifications

Before Enterprise Manager can send e-mail notifications, you must first specify the Outgoing Mail (SMTP) servers to be used by the notification system. Once set, you can then define e-mail notifications for yourself or, if you have Super Administrator privileges, other Enterprise Manager administrators.

You specify the Outgoing Mail (SMTP) server on the Notification Methods page (Figure 3-1). Display the Notification Methods page by clicking Setup on any page in the Grid Control console and clicking Notification Methods in the vertical navigation bar.

Note:

You must have Super Administrator privileges in order to set up SMTP servers.

Specify one or more outgoing mail server names, the mail server authentication credentials (User Name, Password, and Confirm Password), if required, the name you want to appear as the sender of the notification messages, and the e-mail address you want to use to send your e-mail notifications. This address, called the Sender's Mail Address, must be a valid address on each mail server that you specify. A message will be sent to this e-mail address if any problem is encountered during the sending of an e-mail notification. Example 3-1 shows sample notification method entries.

Example 3-1 Mail Server Settings

  • Outgoing Mail (SMTP) Server - smtp01.mycorp.com:587, smtp02.mycorp.com

  • User Name - myadmin

  • Password - ******

  • Confirm Password - ******

  • Identify Sender As - Enterprise Manager

  • Sender's E-mail Address - mgmt_rep@mycorp.com

  • Use Secure Connection - No: E-mail is not encrypted. SSL: E-mail is encrypted using the Secure Sockets Layer protocol. TLS, if available: E-mail is encrypted using the Transport Layer Security protocol if the mail server supports TLS. If the server does not support TLS, the e-mail is automatically sent as plain text.

Figure 3-1 Defining a Mail Server

Definition entry fields on Notification Methods page.
Description of "Figure 3-1 Defining a Mail Server"

Note:

The e-mail address you specify on this page is not the e-mail address to which the notification is sent. You will have to specify the e-mail address (where notifications will be sent) from the Preferences General page.

After configuring the e-mail server, click Test Mail Servers to verify your e-mail setup. You should verify that an e-mail message was received by the e-mail account specified in the Sender's E-mail Address field.

Defining multiple mail servers will improve the reliability of e-mail notification delivery and spread the load across multiple systems. The Management Service makes use of each mail server to send e-mails and the behavior is controlled by the following parameters found in the $ORACLE_HOME/sysman/config/emoms.properties file.

Example 3-2 Management Service Parameters

# The maximum number of emails that can be sent in a single connection to an

# email server

# em.notification.emails_per_connection=20

#

# The maximum number of emails that can be sent in a minute

# em.notification.emails_per_minute=250

Based on the defaults in Example 3-2, the first mail server is used to send 20 e-mails before the Management Service switches to the next mail server which is used to send another 20 e-mails before switching to the next mail server. This prevents one mail server from becoming overloaded and should improve overall reliability and throughput.

Setting Up Repeat Notifications

Repeat notifications allow administrators to be notified repeatedly until an alert is either acknowledged or the number of Maximum Repeat Notifications has been reached. Enterprise Manager supports repeat notification for all notification methods (e-mail, OS command, PL/SQL procedure, and SNMP trap). To enable this feature for a notification method, select the Send Repeat Notifications option. In addition to setting the maximum number of repeat notifications, you can also set the time interval at which the notifications are sent.

Important:

If the Grid Control Repository database version is 9.2, the aq_tm_processes init.ora parameter must be set to at least 1 to enable repeat notification functionality.

Repeat Notifications for Rules

Setting repeat notifications globally at the notification method level may not be provide sufficient flexibility. For example, you may want to have different repeat notification settings based on the metric type and/or alert severity. Enterprise Manager accomplishes this by allowing you to set repeat notifications for individual notification rules. Repeat notifications set at the rule level take precedence over those defined at the notification method level.

Important:

Repeat notifications for rules will only be sent if the Send Repeat Notifications option is enabled in the Notification Methods page.

For PL/SQL, OS command, and SNMP trap notification methods, you must enable each method to support repeat notifications. You can select Supports Repeat Notifications option when adding a new notification method or by editing an existing method.

Figure 3-2 Enabling Repeat Notification for an OS Command Notification Method

Description of Figure 3-2 follows
Description of "Figure 3-2 Enabling Repeat Notification for an OS Command Notification Method"

Setting Up E-mail for Yourself

If you want to receive notifications by e-mail, you will need to specify your e-mail address(s) in the General page under the Preferences link in the Grid Control console. In addition to defining notification e-mail addresses, you associate the notification message format (long or short) to be used for your e-mail address.

Setting up e-mail involves three steps:

Step 1: Define e-mail addresses.

Step 2: Set up a Notification Schedule.

Step 3: Subscribe to receive e-mail for notification rules.

Defining E-mail Addresses

An e-mail address can have up to 128 characters. There is no upper limit with the number of e-mail addresses.

To add an e-mail address:

  1. From the Grid Control console, click Preferences. By default the General page is selected.

  2. Click Add Another Row to create a new e-mail entry field in the E-mail Addresses table.

  3. Specify the e-mail associated with your Enterprise Manager account. All e-mail notifications you receive from Enterprise Manager will be sent to the e-mail addresses you specify.

    For example, user1@oracle.com

    Select the message format for your e-mail address. The Long Format sends a HTML formatted e-mail that contains detailed information. Example 3-3 shows a typical notification that uses the long format.

    The Short Format (Example 3-4) sends a concise, text e-mail that is limited to a configurable number of characters, thereby allowing the e-mail be received as an SMS message or page. The content of the message can be sent entirely in the subject, entirely in the body or split across the subject and body. For example, in the last case, the subject could contain the severity type (for example, Critical) and the target name. The body could contain the time the severity occurred and the severity message. Since the message length is limited, some of this information may be truncated. If truncation has occurred there will be an ellipsis end of the message.

  4. Click Apply to save your e-mail address.

Example 3-3 Long E-mail Notification for Alerts

Name=myhost.com

Type=Host

Host=myhost.com

Metric=Filesystem Space Available (%)

Mount Point =/usr

Timestamp=06-OCT-2006 16:27:05 US/Pacific

Severity=Warning

Message=Filesystem / has only 76.07% available space

Rule Name=Host Availability and Critical States

Rule Owner=SYSMAN

Example 3-4 Short E-mail Notification for Alerts

Subject is :

EM:Unreachable Start:myhost

Body is :

Nov 16, 2006 2:02:19 PM EST:Agent is Unreachable (REASON = Connection refused) 

but the host is UP

More about Short E-mail Format

Enterprise Manager does not directly support message services such as paging or SMS, but instead relies on external gateways to, for example, perform the conversion from e-mail to page.

Entries in the emoms.properties file define the size and format of the short e-mail.

You must establish the maximum size your device can support and whether the message is sent in subject, body or both.

emoms.properties Entries for a Short E-mail Format

# The maximum size of a short format email

# em.notification.short_format_length=155

# The format of the short email. It can be set to subject, body or both.

#

# When set to subject the entire message is sent in the subject i.e.

#  EM:<severity>:<target>:<message>:<timestamp>

# When set to body the entire message is sent in the body i.e.

#  EM:<severity>:<target>:<message>:<timestamp>

# When set to both the message is split i.e. the subject contains

#  EM:<severity>:<target>

# and the body contains

#  <message>:<timestamp>

# In all cases the message is truncated to the length specified in the

# em.notification.short_format_length parameter

# em.notification.short_format=both

Setting Up a Notification Schedule

Once you have defined your e-mail notification addresses, you will need to define a notification schedule. For example, if your e-mail addresses are user1@oracle.com, user2@oracle.com, user3@oracle.com, you can choose to use one or more of these e-mail addresses for each time period in your notification schedule.

Note:

When you enter e-mail addresses for the first time, a 24x7 weekly notification schedule is set automatically. You can then review and modify the schedule to suit your monitoring needs.

A notification schedule is a repeating schedule used to specify your on-call schedule—the days and time periods and e-mail addresses that should be used by Enterprise Manager to send notifications to you. Each administrator has exactly one notification schedule. When a notification needs to be sent to an administrator, Enterprise Manager consults that administrator's notification schedule to determine the e-mail address to be used. Depending on whether you are Super Administrator or a regular Enterprise Manager administrator, the process of defining a notification schedule differs slightly.

If you are a regular Enterprise Manager administrator and are defining your own notification schedule:

  1. From the Enterprise Manager Grid Control, click Preferences at the top of the page. By default the General page is selected.

  2. Click Notification Schedule in the vertical navigation bar. Your Notification Schedule page appears.

  3. Follow the directions on the Notification Schedule page to specify when you want to receive e-mails.

Subscribe to Receive E-mail for Notification Rules

A notification rule is a user-defined rule that defines the criteria by which notifications should be sent for alerts, policy violations, corrective action execution status, and job execution status. Specifically, in each rule, you can specify the criteria you are interested in and the notification methods (such as e-mail) that should be used for sending these notifications. For example, you can set up a rule that when any database goes down or any database backup job fails, e-mail should be sent and the "log trouble ticket" notification method should be called. Or you can define another rule such that when the CPU or Memory Utilization of any host reach critical severities, SNMP traps should be sent to another management console. During notification rule creation, you specify criteria such as the targets you are interested in, their monitored metrics, associated alert severity conditions (clear, warning, critical), policy violations, corrective action execution status, or job execution status, and the associated notification method.

To subscribe to a notification rule you create, while creating the rule, go to the Actions page and check the Send Me E-mail option.

Out-of-Box Notification Rules

Enterprise Manager Grid control comes with out-of-box notification rules that cover the most common alert conditions. When you install the Oracle Management Service, you are given the option to receive e-mail notifications for critical alerts. If you choose this option, and if an e-mail address for the SYSMAN user was specified, then some default notification rules are created that cover the availability and critical states for common target types and would also be configured to send e-mail notifications to the SYSMAN e-mail address for the conditions defined in the notification rules.

You can access the out-of-box notification rules by clicking on Preferences on any page in the Enterprise Manager console and clicking Public Rules in the vertical navigation bar. If the conditions defined in the out-of-box notification rules meet your requirements, you can simply subscribe to receive e-mail notifications for the conditions defined in the rule by clicking on Subscribe column in the row of the Public Rules table that corresponds to the notification rule that you are interested in. Click Apply to save your changes.

Table 3-1 lists all default notification rules. These are all owned by the SYSMAN user and are public rules.

Table 3-1 Default Notification Rules

Name Description Applies to Targets of the Type Send Notification on the Following Availability States Send Notification if Any of the Metrics is at CRITICAL Alert Severity

Agent Upload Problems

System-generated notification rule for monitoring Agents that may have problems uploading data to the Management Service.

Oracle Management Service and Repository

N/A

Count of targets not uploading data

Agents Unreachable

System-generated notification rule for monitoring Agents that lose contact with the Management Service due to network problems, host problems or Agents going down.

Agents

Agent UnreachableAgent Unreachable Resolved

N/A

Application Server Availability and Critical States

System-generated notification rule for monitoring Application Servers' availability, and critical metric statuses.

Application Servers

Down

CPU Usage (%)

Database Availability and Critical States

System-generated notification rule for monitoring Databases' availability, and critical metric statuses.

Databases (single instance only)

Down

Process Limit Usage (%)

Session Limit Usage (%)

Blocking Session Count All Objects

Archiver Hung Alert Log Error Status

Data Block Corruption Alert Log Error Status

Generic Alert Log Error Status

Media Failure Alert Log Error Status

Session Terminated Alert Log Error Status

Archive Area Used (%) All Objects

Segments Not Able to Extend Count All Objects

Segments Approaching Maximum Extents Count All Objects

Tablespace Space Used (%) All Objects

Wait Time (%)

HTTP Server Availability and Critical States

System-generated notification rule for monitoring HTTP Server's availability, and critical metric statuses.

Oracle HTTP Server

Down

CPU Usage (%)

Percentage of Busy Processes

Active HTTP Connections

Request Processing Time (seconds)

Host Availability and Critical States

System-generated notification rule for monitoring Hosts' availability, and critical metric statuses.

Hosts

Agent Unreachable

Agent Unreachable Resolved

Average Disk I/O Service Time (ms)

Disk Device Busy (%)

Filesystem Space Available (%)

CPU in I/O Wait (%)

Run Queue Length (5 minute average)

CPU Utilization (%)

Memory Utilization (%)

Memory Page Scan Rate (per second)

Swap Utilization (%)

Network Interface Combined Utilization (%)

Listener Availability

System-generated notification rule for monitoring database Listeners' availability, and critical metric statuses.

Listeners

Down

N/A

Misconfigured Agents

System-generated notification rule for misconfigured agents.

Agent

Agent Unreachable

Agent Unreachable Resolved

Consecutive severity upload failure count Consecutive heartbeat failure count

MS Agent time skew (mins)Consecutive metadata upload failure count

OC4J Availability and Critical States

System-generated notification rule for monitoring OC4J instance's availability, and critical metric statuses.

OC4J

Down

CPU Usage (%)

OC4J Instance - Request Processing Time (seconds)

OC4J Instance - Active Sessions

OMS Service Initialization Errors

System-generated notification rule for monitoring OMS service initialization errors.

OMS and Repository

N/A

Service Status

PAF Status Notification

System-generated notification rule for Provisioning Advisor Framework: Notifies the instance creator of any status updates.

N/A

Up

Down

Corrective Actions on Target Down

Agent Unreachable

Agent Unreachable Resolved

Metric Error Detected

Metric Error ResolvedBlackout Started

Blackout Ended

N/A

Repository Operations Availability

System-generated notification rule for monitoring the availability of the DBMS jobs that are part of the Management Repository.

OMS and Repository

Critical

DBMS Job UpDown

Violation Notification for Database Security Policies

System-generated notification rule for monitoring the secureness of the database configuration.

Databases

Critical

N/A

Web Cache Availability and Critical States

System-generated notification rule for monitoring Web Cache's instance's availability, and critical metric statuses.

Oracle Web Cache

Down

Hits (% of requests)

Web Cache CPU Usage (%)


Creating Your Own Notification Rules

If you find that the default notification rules do not meet your needs, you can define your own custom rules. The following procedure documents the process of notification rule creation for non-Super Administrators.

To create your own notification rule:

  1. From the Enterprise Manager Grid Control, click Preferences.

  2. Click My Rules in the vertical navigation bar.

    If you are not logged in as an administrator with Super Administrator privileges, you will see a link for My Rules instead of Rules as in the case of an administrator with Super Administrator privileges.

  3. Click Create.

    Enterprise Manager displays the Create Notification Rule pages. Enter the requisite information on each page to create your notification rule.

    When you specify the notification rule properties, check Make Public in the General page if you want other non-privileged users to be able to view and share that rule. For example, it allows other administrators to later specify that they should receive e-mail for this rule.

    When you specify the notification rule, you will only be able to choose from e-mail and SNMP traps. Specifying custom commands and PL/SQL procedures is an option that is only available to Super Administrators. To receive e-mail notifications for conditions defined in the rule, go to the Actions page and check the Send Me E-Mail option.

Specifying Additional Alert Duration Criteria

You can set additional alert duration criteria for a notification rule and have the rule apply only to alerts that have been open for at least a certain amount of time and have not been acknowledged. These criteria apply only to Target Down, Agent Unreachable, Metric alerts, Policy Violations, Blackout Started and Metric Error Start alerts.

Typical scenarios where you would use additional alert criteria are:

  • For log alerts that have been open for at least 7 days, clear the alerts

  • For alerts that have been open for at least 48 hours and have not been acknowledged, send e-mail to the DBA Manager

To specify additional alert duration criteria:

  1. Create or edit a notification rule. Additional alert criteria can be added from the Availability, Metrics, or Policy tab.

  2. From one of the aforementioned tabs, go to the Additional Alert Criteria section and click Add. The Additional Alert Criteria page appears.

  3. Specify the alert duration criteria and click Continue.

Setting Up E-mail for Other Administrators

If you have Super Administrator privileges, you can set up e-mail notifications for other Enterprise Manager administrators. To set up e-mail notifications for other Enterprise Manager administrators, you need to:

Step 1: Ensure Each Administrator Account has an Associated E-mail Address

Each administrator to which you want to send e-mail notifications must have a valid e-mail address.

  1. Click Setup.

  2. Click Administrators from the vertical navigation bar.

  3. For each administrator, define an e-mail address. This sets up a 24x7 notification schedule for this user that uses all the e-mail addresses specified.

Enterprise Manager also allows you to specify an administrator address when editing an administrator's notification schedule.

Step 2: Define Administrators' Notification Schedules

Once you have defined e-mail notification addresses for each administrator, you will need to define their respective notification schedules. Although a default 24x7 notification schedule is created when you specify an e-mail address for the first time, you should review and edit the notification schedule as needed.

  1. Click Setup.

  2. From the vertical navigation bar, click Schedules (under Notification). The Notification Schedule page appears.

  3. Specify the administrator who's notification schedule you wish to edit and click Change.

  4. Click Edit Schedule Definition. The Edit Schedule Definition: Time Period page appears. If necessary, modify the rotation schedule.

  5. Click Continue. The Edit Schedule Definition: E-mail Addresses page appears.

  6. Follow the directions on the Edit Schedule Definition: E-mail Addresses page to modify the notification schedule.

  7. Click Finish when you are done.

  8. Repeat steps three through seven for each administrator.

Step 3: Assign Notification Rules to Administrators

With the notification schedules set, you now need to assign the appropriate notification rules for each designated administrator.

  1. Click Setup.

  2. From the vertical navigation bar, click Administrators.

  3. Select the desire administrator.

  4. Click Subscribe to Rules. The Subscribe <administrator> to Public Notification Rules page appears.

  5. Select the desired notification rules and click Subscribe.

  6. Click OK when you are finished.

  7. Repeat steps three through six for each administrator.

E-mail Customization

Enterprise Manager allows Super Administrators to customize global e-mail notifications for the four alert types (Metric Alert, Target Availability, Policy Violation, and Job Status Change). Using predefined building blocks (called attributes and labels) contained within a simple script, Super Administrators can customize alert e-mails by selecting from a wide variety of information content.

To customize an e-mail:

  1. Access the E-mail Customization page. Setup-->E-mail Customization

  2. Choose the Alert Type and Format.

  3. Click Edit. The Edit E-mail Template page is displayed.

From the Edit E-mail Template page, you can modify the content of the e-mail template Enterprise Manager uses to generate e-mail notifications. Extensive information on script formatting, syntax, and options is available from the Edit E-mail Template page via imbedded assistance and online help.

Figure 3-3 E-mail Customization

Description of Figure 3-3 follows
Description of "Figure 3-3 E-mail Customization"

E-mail Customization Reference

The following reference summarizes the semantics and component syntax of the pseudo-language used to define e-mails. The pseudo-language provides you with a simple, yet flexible way to customize e-mail notifications. The following is a summary of pseudo-language conventions/limitations:

  • You can add comments (or any free-form text) using separate lines beginning with "--" or at end of lines.

  • You can use attributes.

  • You can use IF & ELSE & ENDIF control structures. You can also use multiple conditions using "AND" or "OR". Nested IF statements are not supported.

  • You can insert spaces for formatting purposes. Spaces at the beginning of a line will be ignored in the actual e-mail. To insert spaces at the beginning of a line, use the [SP] attribute.

  • Use "/" to escape and "[" or "]" if you want to add attribute names, operators, or IF clauses to the actual e-mail.

  • HTML is not supported.

Reserved Words and Operators

The following table lists all reserved words and operators used when modifying e-mail scripts.

Table 3-2 Reserved Words and Operators

Reserved Word/Operator Description

IF, ELSIF, ENDIF, ELSE

Used in IF-ELSE constructs.

AND, OR

Boolean operators – used in IF-ELSE constructs only.

NULL

To check NULL value for attributes - used in IF-ELSE constructs only.

|

Pipe operator – used to show the first non-NULL value in a list of attributes.

For example:

METRIC_NAME|POLICY_NAME

EQ, NEQ

Equal and Not-Equal operators – applicable to NULL, STRING and NUMERIC values.

/

Escape character – used to escape reserved words and operators. Escape characters signify that what follows the escape character takes an alternative interpretation.

[ , ]

Delimiters used to demarcate attribute names and IF clauses.


Syntax Elements

Literal Text

You can specify any text as part of the e-mail content. The text will be displayed in the e-mail and will not be translated if the Oracle Management Services (OMS) language setting is changed. For example, 'my Oracle Home' appears as 'my Oracle Home' in the generated e-mail.

Predefined Attributes

Predefined attributes/labels will be substituted with actual values in a specific context. To specify a predefined attribute/label, use the following syntax:

[PREDEFINED_ATTR]

Attribute names can be in either UPPER or LOWER case. The parsing process is case-insensitive.

A pair of square brackets is used to distinguish predefined attributes from literal text. For example, for a job e-mail notification, the actual job name will be substituted for [JOB_NAME]. For a metric e-mail notification, the actual metric column name will be substituted for [METIRC_COLUMN].

You can use the escape character “/” to specify words and not have them interpreted as predefined labels/attributes. For example, "/[NEW/]” will not be considered as the predefined attribute [NEW] when parsed.

Operators

EQ, NEQ – for text and numeric values

NULL- for text and numeric values

GT, LT, GE, LE – for numeric values

Control Structures

The following table lists acceptable script control structures.

Table 3-3 Control Structures

Control Structure Description

Pipe "|"

Two or more attributes can be separated by '|' character. For example,

[METRIC_NAME|POLICY_NAME]

In this example, only the applicable attribute within the current alert context will be used (replaced by the actual value) in the e-mail. If more than one attributes are applicable, only the left-most attribute is used.

IF

Allows you to make a block of text conditional. Only one level of IF and ELSIF is supported. Nested IF constructs are not supported.

All attributes can be used in IF or ELSIF evaluation using EQ/NEQ operators on NULL values. Other operators are allowed for “SEVERITY” and “REPEAT_COUNT” only.

Inside the IF block, the values need to be contained within quotation marks “ ”. Enterprise Manager will extract the attribute name and its value based on the position of “EQ” and other key words such as “and”, “or”. For example,

[IF REPEAT_COUNT EQ “1” AND SEVERITY EQ “CRITICAL” THEN]

The statement above will be true when the attributes of the alert match the following condition:

  • Attribute Name: REPEAT_COUNT

  • Attribute Value: 1

  • Attribute Name: SEVERITY

  • Attribute Value: CRITICAL

Example IF Block:

[IF JOB_NAME NEQ NULL]

       [JOB_NAME_LABEL]=[JOB_NAME] 

       [JOB_OWNER_LABEL]=[JOB_OWNER] 

[ENDIF] 

 

[IF SEVERITY EQ CRITICAL ]

       [MTRIC_NAME_LABEL]=[METRIC_NAME] 

       [METRIC_VALUE_LABEL]=[METRIC_VALUE] 

       [TARGET_NAME_LABEL]=[TARGET_NAME] 

       [KEY_VALUES] 

[ENDIF] 

Example IF and ELSEIF Block:

[IF SEVERITY EQ CRITICAL]           statement1[ELSIF SEVERITY EQ WARNING]           statement2[ELSIF SEVERITY EQ CLEAR]           statement3[ELSE]           statement4[ENDIF]


Comments

You can add comments to your script by prefacing a single line of text with two hyphens "--". For example,

 -- Code added on 8/3/2009     [IF REPEAT_COUNT NEQ NULL]     . . . 

Comments may also be placed at the end of a line of text.

[IF SEVERITY_SHORT EQ W] -- for Warning alert

HTML Tags in Customization Content

Use of HTML tags is not supported.

When Enterprise Manager parses the e-mail script, it will convert the “<” and “>” characters of HTML tags into encoded format (&lt; and &gt;). This ensures that the HTML tag is not treated as HTML by the destination system.

Examples

E-mail customization template scripts support three main operators.

  • Comparison operators: EQ/NEQ/GT/LT/GE/LELogic operators: AND/ORPipeline operator: |

Extending Notification Beyond E-mail

Notification Methods are the mechanisms by which alerts are sent. Enterprise Manager Super Administrators can set up e-mail notifications by configuring the 'e-mail' notification method. Most likely this would already have been setup as part of the Oracle Management Service installation.Enterprise Manager Super Administrators can also define other custom notification methods. For example, alerts may need to be forwarded to a 3rd party trouble-ticketing system. Assuming APIs to the third-party trouble-ticketing system are available, a custom notification method can be created to call a custom OS script that has the appropriate APIs. The custom notification method can be named in a user-friendly fashion, for example, "Log trouble ticket". Once that is defined, any time an administrator needs to send alerts to the trouble-ticketing system, he merely needs to invoke the now globally available notification method called "Log trouble ticket". Custom notification methods can be defined based on any custom OS script, any custom PL/SQL procedure, or by sending SNMP traps. A fourth type of notification method (Java Callback) exists to support Oracle internal functionality and cannot be created or edited by Enterprise Manager administrators.

Only Super Administrators can define OS Command, PL/SQL, and SNMP Trap notification methods. However, any Enterprise Manager administrator can add these notification methods (once defined by the Super Administrator) to their notification rules.

Through the Notification Methods page, you can:

  • Set up the outgoing mail servers if you plan to send e-mail notifications through notification rules

  • Create other custom notification methods using OS and PL/SQL scripts and SNMP traps.

  • Set global repeat notifications.

Custom Notification Methods Using Scripts and SNMP Traps

You can create other custom notification methods based on OS scripts, PL/SQL procedures, or SNMP traps. Any administrator can then use these methods in Notification Rules.

Adding a Notification Method based on an OS Command or Script

Complete the following four steps to define a notification method based on an OS command or script.

Note:

Notification methods based on OS commands must be configured by an administrator with Super Administrator privileges.

Step 1: Define your OS command or script.

You can specify an OS command or script that will be called by the notification system. You can use target and alert or policy violation context information, corrective action execution status and job execution status within the body of the script. Passing metric severity attributes (severity level, type, notification rule, rule owner, or rule owner, and so on) or policy violation information to OS commands/scripts allows you to customize automated responses to alerts or policy violations. For example, if an OS script opens a trouble ticket for an in-house support trouble ticket system, you will want to pass severity levels (critical, warning, and so on) to the script to open a trouble ticket with the appropriate details and escalate the problem. For more information on passing specific types of information to OS Command or Scripts, see

Step 2: Deploy the script on each Management Service host.

You must deploy the OS Command or Script on each Management Service host machine that connects to the Management Repository. The OS Command is run as the user who started the Management Service.

The OS Command or Script should be deployed on the same location on each Management Service host machine. The OS Command should be an absolute path, for example, /u1/bin/logSeverity.sh. The command is run by the user who started the Management Service. If an error is encountered during the running of the OS Command, the Notification System can be instructed to retry the sending of the notification to the OS Command by returning an exit code of 100. The procedure is initially retried after one minute, then two minutes, then three minutes and so on, until the notification is a day old, at which point it will be purged.

Example 3-5 shows the parameter in emoms.properties that controls how long the OS Command can execute without being killed by the Management Service. This is to prevent OS Commands from running for an inordinate length of time and blocking the delivery of other notifications. By default the command is allowed to run for 30 seconds before it is killed.

Example 3-5 Parameter in emoms.properties File

# The amount of time in seconds after which an OS Command started by the

# Notification System will be killed if it has not exited

# em.notification.os_cmd_timeout=30

Step 3: Register your OS Command or Script as a new Notification Method.

Add this OS command as a notification method that can be called in Notification Rules. Log in as a Super Administrator, click Setup and then Notification Methods from the vertical navigation bar. From this page, you can define a new notification based on the 'OS Command' type. See "Adding a Notification Method based on an OS Command or Script" .

The following information is required for each OS command notification method:

  • Name

  • Description

    Both Name and Description should be clear and intuitive so that the function of the method is clear to other administrators.

  • OS Command

You must enter the full path of the OS command or script in the OS command field (for example, /u1/bin/myscript.sh). For environments with multiple Management Services, the path must be exactly the same on each machine that has a Management Service. Command line parameters can be included after the full path (for example, /u1/bin/myscript.sh arg1 arg2).

Example 3-6 shows information required for the notification method.

Example 3-6 OS Command Notification Method

Name Trouble Ticketing

Description Notification method to log trouble ticket for a severity occurrence

OS Command /private/mozart/bin/logTicket.sh

Note:

There can be more than one OS Command configured per system.

Step 4: Assign the notification method to a rule.

You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a single rule, see "Assigning Rules to Methods" or "Assigning Methods to Rules".

Passing Alert and Policy Violation Information to an OS Command or Script

The notification system passes severity information to an OS script or executable using system environment variables.

Conventions used to access environmental variables vary depending on the operating system:

  • UNIX: $ENV_VARIABLE

  • Windows:%ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.

Table 3-4 Environment Variables

Environment Variable Description

TARGET_NAME

Name of the target on which the severity occurred.

TARGET_TYPE

Type of target on which the severity occurred. Targets are defined as any monitorable entity, such as Host, Database, Listener, or Oracle HTTP Server. You can view the type of a monitored target on the All Targets page.

HOST

Name of the machine on which the target resides.

METRIC

Metric generating the severity. This variable is not set for policy violations.

METRIC_VALUE

The value of the metric when the threshold was exceeded. Not set for policy violations

CYCLE_GUID

A unique identifier for a metric alert cycle, which starts from the time the metric alert is initially generated until the time it is clear.

POLICY_RULE

The name of the policy when the threshold was exceeded. Not set for metric severities

KEY_VALUE

For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity.

KEY_VALUE_NAME

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

VIOLATION_CONTEXT

A comma-separated list of name-value pairs that shows the alert context for a policy violation.

TIMESTAMP

Time when the severity occurred.

SEVERITY

Type of severity. For example, severity for a target's (availability) status metric are:

  • UP

  • DOWN

  • UNREACHABLE CLEAR

  • UNREACHABLE START

  • BLACKOUT END

  • BLACKOUT START

Other metrics can have any of the following severities:

  • WARNING

  • CRITICAL

  • CLEAR

  • METRIC ERROR CLEAR

  • METRIC ERROR START

MESSAGE

Message for the alert that provides details about what triggered the condition.

RULE_NAME

Name of the notification rule to which the OS Command notification method was assigned.

RULE_OWNER

Name of the Enterprise Manager administrator who owns the notification rule.


Your script may reference some or all of these variables.

The sample OS script shown in Example 3-7 appends environment variable entries to a log file. In this example, the script logs a severity occurrence to a file server. If the file server is unreachable then an exit code of 100 is returned to force the Oracle Management Service Notification System to retry the notification

Example 3-7 Sample OS Command Script

#!/bin/ksh


LOG_FILE=/net/myhost/logs/severity.log

if test -f $LOG_FILE

then

echo $TARGET_NAME $MESSAGE $TIMESTAMP >> $LOG_FILE

else

   exit 100

fi

Example 3-8 shows an OS script that logs alert information to the file 'alertmsg.txt'. The file is saved to the /u1/results directory.

Example 3-8 Alert Logging Script

#!/usr/bin/sh

echo "Alert logged:" > /u1/results/alertmsg.txt

echo "\n" >> /u1/results/alertmsg.txt

echo "target name is " $TARGET_NAME >> /u1/results/alertmsg.txt

echo "target type is " $TARGET_TYPE >> /u1/results/alertmsg.txt

echo "target is on host " $HOST >> /u1/results/alertmsg.txt

echo "metric in alert is " $METRIC >> /u1/results/alertmsg.txt

echo "metric index is " $KEY_VALUE >> /u1/results/alertmsg.txt

echo "timestamp is " $TIMESTAMP >> /u1/results/alertmsg.txt

echo "severity is " $SEVERITY >> /u1/results/alertmsg.txt

echo "message is " $MESSAGE >> /u1/results/alertmsg.txt

echo "notification rule is " $RULE_NAME >> /u1/results/alertmsg.txt

echo "rule owner is " $RULE_OWNER >> /u1/results/alertmsg.txt

exit 0

Example 3-9 shows a script that sends an alert to an HP OpenView console from Enterprise Manager Grid Control. When a metric alert is triggered, the Enterprise Manager Grid Control displays the alert. The HP OpenView script is then called, invoking opcmsg and forwarding the information to the HP OpenView management server.

Example 3-9 HP OpenView Script

/opt/OV/bin/OpC/opcmsg severity="$SEVERITY" app=OEM msg_grp=Oracle msg_text="$MESSAGE" object="$TARGET"

Adding a Notification Method Based on a PL/SQL Procedure

Complete the following four steps to define a notification method based on a PL/SQL procedure.

Step 1: Define the PL/SQL procedure.

The procedure must have one of the following signatures depending on the type of notification that will be received.

For alerts and policy violations:

PROCEDURE p(severity IN MGMT_NOTIFY_SEVERITY)

For job execution status changes:

PROCEDURE p(job_status_change IN MGMT_NOTIFY_JOB)

For corrective action status changes:

PROCEDURE p(ca_status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION)

Note:

The notification method based on a PL/SQL procedure must be configured by an administrator with Super Administrator privileges before a user can select it while creating/editing a notification rule.

For more information on passing specific types of information to scripts or PL/SQL procedures, see the following sections:

"Passing Alert and Policy Violation Information to a PL/SQL Procedure"

"Passing Corrective Action Status Change Information"

"Passing Job Execution Status Information"

Step 2: Create the PL/SQL procedure on the Management Repository.

Create the PL/SQL procedure on the repository database using one of the following procedure specifications:

PROCEDURE p(severity IN MGMT_NOTIFY_SEVERITY)

PROCEDURE p(job_status_change IN MGMT_NOTIFY_JOB)

PROCEDURE p(ca_status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION)

The PL/SQL procedure must be created on the repository database using the database account of the repository owner (such as SYSMAN)

If an error is encountered during the running of the procedure, the Notification System can be instructed to retry the sending of the notification to the procedure by raising a user defined exception that uses the error code -20000. See Example 3-11, "PL/SQL Procedure Using a Severity Code". The procedure initially retried after one minute, then two minutes, then three minutes and so on, until the notification is a day old, at which point it will be purged.

Step 3: Register your PL/SQL procedure as a new notification method.

Log in as a Super Administrator, click Setup and then Notification Methods from the vertical navigation bar. From this page, you can define a new notification based on 'PL/SQL Procedure'. See "Adding a Notification Method Based on a PL/SQL Procedure".

Make sure to use a fully qualified name that includes the schema owner, package name and procedure name. The procedure will be executed by the repository owner and so the repository owner must have execute permission on the procedure.

Create a notification method based on your PL/SQL procedure. The following information is required when defining the method:

  • Name

  • Description

  • PL/SQL Procedure

You must enter a fully qualified procedure name (for example, OWNER.PKGNAME.PROCNAME) and ensure that the owner of the Management Repository has execute privilege on the procedure.

An example of the required information is shown in Example 3-10.

Example 3-10 PL/SQL Procedure Required Information

Name Open trouble ticket

Description Notification method to open a trouble ticket in the event

PLSQL Procedure ticket_sys.ticket_ops.open_ticket

Step 4: Assign the notification method to a rule.

You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a single rule, see "Assigning Rules to Methods" or "Assigning Methods to Rules".

There can be more than one PL/SQL-based method configured for your Enterprise Manager environment.

Information about the severity types that relate to a target's availability, and how metric severity and policy violation information is passed to the PLSQL procedure is covered in the next section.

Passing Alert and Policy Violation Information to a PL/SQL Procedure

Passing metric severity attributes (severity level, type, notification rule, rule owner, or rule owner, and so on) or policy violation information to PL/SQL procedures allows you to customize automated responses to alerts or policy violations.

The notification system passes information about metric severities or policy violations to a PL/SQL procedure using the MGMT_NOTIFY_SEVERITY object. An instance of this object is created for every alert or policy violation. When an alert or policy violation occurs, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_SEVERITY object that has been passed to it.

The following table lists all metric severity attributes that can be passed:

Table 3-5 Metric Severity Attributes

Attribute Datatype Additional Information

TARGET_NAME

VARCHAR2(256)

Name of the target on which the severity occurred.

TARGET_TYPE

VARCHAR2(64)

Type of target on which the severity occurred. Targets are defined as any monitorable service.

TIMEZONE

VARCHAR2(64)

The target's regional timezone

HOST_NAME

VARCHAR2(128)

Name of the machine on which the target resides.

METRIC_NAME

VARCHAR2(64)

Metric or policy generating the severity.

METRIC_DESCRIPTION

VARCHAR2(128)

Meaningful description of the metric that can be understood by other administrators.

METRIC_COLUMN

VARCHAR2(64)

For table metrics, the metric column contains the name of the column in the table that is being defined. If the metric that is being defined is not a table metric, the value in this column is a single space. This attribute is not used for policy violations.

METRIC_VALUE

VARCHAR2(1024)

The value of the metric.

KEY_VALUE

VARCHAR2(1290)

For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity.

KEY_VALUE_NAME

VARCHAR2(512)

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

KEY_VALUE_GUID

VARCHAR2(256)

GUID associated with a composite key value name.

CTXT_LIST

MGMT_NOTIFY_COLUMNS

Details of the alert context.

COLLECTION_TIMESTAMP

DATE

The time when the target status change was last detected and logged in the management repository.

SEVERITY_CODE

NUMBER

Numeric code identifying the severity level. See Severity Code table below.

MESSAGE

VARCHAR2(4000)

An optional message that is generated when the alert is created that provides additional information about the alert condition.

SEVERITY_GUID

RAW(16)

Severity global unique identifier.

METRIC_GUID

RAW(16)

Metric global unique identifier.

TARGET_GUID

RAW(16)

Target global unique identifier.

RULE_OWNER

VARCHAR2(64)

Name of the Enterprise Manager administrator who owns the rule.

RULE_NAME

VARCHAR2(132)

Name of the notification rule that resulted in the severity.


When a severity occurs for the target, the notification system creates an instance of the MGMT_NOTIFY_SEVERITY object and populates it with values from the severity. The severity codes in Table 3-6 have been defined as constants in the MGMT_GLOBAL package and can be used to determine the type of severity in the severity_code field of the MGMT_NOTIFY_SEVERITY object.

Table 3-6 Severity Codes

Name Datatype Value

G_SEVERITY_COMMENT

NUMBER(2)

10

G_SEVERITY_CLEAR

NUMBER(2)

15

G_SEVERITY_WARNING

NUMBER(2)

20

G_SEVERITY_CRITICAL

NUMBER(2)

25

G_SEVERITY_UNREACHABLE_CLEAR

NUMBER(3)

115

G_SEVERITY_UNREACHABLE_START

NUMBER(3)

125

G_SEVERITY_BLACKOUT_END

NUMBER(3)

215

G_SEVERITY_BLACKOUT_START

NUMBER(3)

225

G_SEVERITY_ERROR_END

NUMBER(3)

315

G_SEVERITY_ERROR_START

NUMBER(3)

325

G_SEVERITY_NO_BEACONS

NUMBER(3)

425

G_SEVERITY_UNKNOWN

NUMBER(3)

515


Example 3-11 PL/SQL Procedure Using a Severity Code

CREATE TABLE alert_log (target_name VARCHAR2(64),

alert_msg VARCHAR2(4000),

occured DATE);


PROCEDURE LOG_CRITICAL_ALERTS(severity IN MGMT_NOTIFY_SEVERITY)

IS

BEGIN

-- Log all critical severities

   IF severity.severity_code = MGMT_GLOBAL.G_SEVERITY_CRITICAL

   THEN

         BEGIN

                 INSERT INTO alert_log (target_name, alert_msg, occured)

                 VALUES (severity.target_name, severity.message,

                                 severity.collection_timestamp);

         EXCEPTION

         WHEN OTHERS

         THEN

         -- If there are any problems then get the notification retried

                 RAISE_APPLICATION_ERROR(-20000, 'Please retry');

                 END;

                 COMMIT;

   END IF;

END LOG_CRITICAL_ALERTS;

Adding a Notification Method Based on an SNMP Trap

Enterprise Manager supports integration with third-party management tools through the SNMP. For example, you can use SNMP to notify a third-party application that a selected metric has exceeded its threshold.

The trap is an SNMP Version 1 trap and is described by the MIB definition shown at the end of this chapter. See "Management Information Base (MIB)".

For more comprehensive configuration information, see the documentation specific to your platform; SNMP configuration differs from platform to platform.

Note:

Notification methods based on SNMP traps must be configured by an administrator with Super Administrator privileges before any user can then choose to select one or more of these SNMP trap methods while creating/editing a notification rule.

Step 1: Define a new notification method based on an SNMP trap.

Log in to Enterprise Manager as a Super Administrator. Click Setup and then click Notification Method from the vertical navigation bar to access the Notification Methods page. From this page you can add a new method based on an SNMP trap.

You must provide the name of the host (machine) on which the SNMP master agent is running and other details as shown in the following example. In Example 3-12, the SNMP host will receive your SNMP traps.

Example 3-12 SNMP Trap Required Information

Name HP OpenView Console

Description Notification method to send trap to HP OpenView console

SNMP Trap Host Name machine1.us.oracle.com

SNMP Host Port 162

SNMP Community public

This SNMP host will receive your SNMP traps.

Note:

A Test Trap button exists for you to test your setup.

Metric severity information will be passed as a series of variables in the SNMP trap.

An example SNMP Trap is shown in Example 3-13. Each piece of information is sent as a variable embedded in the SNMP Trap.

Example 3-13 SNMP Trap

Tue Oct 28 05:00:02 2006


Command: 4

   Enterprise: 1.3.6.1.4.1.111.15.2

   Agent: 138.1.6.200

   Generic Trap: 6

   Specific Trap: 1

   Time Stamp: 8464:39.99

   Count: 11


Name: 1.3.6.1.4.1.111.15.1.1.1.2.1

   Kind: OctetString

   Value: "mydatabase"


Name: 1.3.6.1.4.1.111.15.1.1.1.3.1

   Kind: OctetString

   Value: "Database"


Name: 1.3.6.1.4.1.111.15.1.1.1.4.1

  Kind: OctetString

  Value: "myhost.com"


Name: 1.3.6.1.4.1.111.15.1.1.1.5.1

   Kind: OctetString

   Value: "Owner's Invalid Object Count"


Name: 1.3.6.1.4.1.111.15.1.1.1.6.1

   Kind: OctetString

   Value: "Invalid Object Owner"


Name: 1.3.6.1.4.1.111.15.1.1.1.7.1

   Kind: OctetString

   Value: "SYS"


Name: 1.3.6.1.4.1.111.15.1.1.1.8.1

   Kind: OctetString

   Value: "28-OCT-2006 04:59:10 (US/Eastern GMT)"


Name: 1.3.6.1.4.1.111.15.1.1.1.9.1

   Kind: OctetString

   Value: "Warning"


Name: 1.3.6.1.4.1.111.15.1.1.1.10.1

   Kind: OctetString

   Value: "12 object(s) are invalid in the SYS schema."


Name: 1.3.6.1.4.1.111.15.1.1.1.11.1

   Kind: OctetString

   Value: "Database Metrics"


Name: 1.3.6.1.4.1.111.15.1.1.1.12.1

   Kind: OctetString

   Value: "SYSMAN"

Step 2: Assign the notification method to a rule.

You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a rule, see "Assigning Methods to Rules" or "Assigning Rules to Methods".

Passing Corrective Action Status Change Information

Passing corrective action status change attributes (such as new status, job name, job type, notification rule, or rule owner) to PL/SQL procedures or OS commands/scripts allows you to customize automated responses to status changes. For example, you many want to call an OS script to open a trouble ticket for an in-house support trouble ticket system if a critical corrective action fails to run. In this case, you will want to pass status (for example, Problems or Aborted) to the script to open a trouble ticket and escalate the problem.

Passing Corrective Action Execution Status to an OS Command or Script

The notification system passes information to an OS script or executable via system environment variables. Conventions used to access environmental variables vary depending on the operating system:

  • UNIX: $ENV_VARIABLE

  • MS Windows: %ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.

Table 3-7 Environment Variables

Environment Variable Description

JOB_NAME

The name of the corrective action.

JOB_OWNER

The owner of the corrective action.

JOB_TYPE

The type of corrective action.

JOB_STATUS

The corrective action status.

TIMESTAMP

Time when the severity occurred.

NUM_TARGETS

The number of targets.

TARGET_NAMEn

The name of the nth target on which the corrective action ran. Example: TARGET_NAME1, TARGET_NAME2.

METRIC

The name of the metric in the alert that caused the corrective action to run. Not set for policy violations.

POLICY_RULE

The name of the policy rule in the alert that caused the corrective action to run. Not set for metric severities.

METRIC_VALUE

The value of the metric column in the alert that caused the corrective action to run.

VIOLATION_CONTEXT

A comma-separated list of name-value pairs that show the policy violation context.

KEY_VALUE_NAME

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

KEY_VALUE

For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity.

SEVERITY

Type of alert severity. For example, severity types that relate to a target's availability are:

  • UP

  • DOWN

  • UNREACHABLE CLEAR

  • UNREACHABLE START

  • BLACKOUT END

  • BLACKOUT START

Other metrics can have any of the following severities:

  • WARNING

  • CRITICAL

  • CLEAR

  • METRIC ERROR CLEAR

  • METRIC ERROR START

RULE_NAME

Name of the notification rule that resulted in the execution of the corrective action.

RULE_OWNER

Name of the Enterprise Manager administrator who owns the notification rule.


Passing Corrective Action Execution Status to a PLSQL Procedure

The notification system passes corrective action status change information to a PL/SQL procedure via the MGMT_NOTIFY_CORRECTIVE_ACTION object. An instance of this object is created for every status change. When a corrective action executes, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_CORRECTIVE_ACTION object that has been passed to it.

Table 3-8 lists all corrective action status change attributes that can be passed:

Table 3-8 Corrective Action Status Attributes

Attribute Datatype Additional Information

JOB_NAME

VARCHAR2(128)

The corrective action name.

JOB_OWNER

VARCHAR(256)

The owner of the corrective action.

JOB_TYPE

VARCHAR2(32)

The type of the corrective action.

JOB_STATUS

NUMBER

The new status of the corrective action. See Table 3-9, "Corrective Action Status Codes" for a list of possible status conditions.

STATE_CHANGE_GUID

RAW(16)

The GUID of the state change record.

JOB_GUID

RAW(16)

The unique id of the corrective action.

EXECUTION_ID

RAW(16)

The unique id of the corrective action execution.

TARGETS

SMP_EMD_NVPAIR_ARRAY

An array of the target name/target type pairs that the corrective action runs on.

METRIC_NAME

VARCHAR2(256)

The name of the metric/policy rule in the alert that caused the corrective action to run.

METRIC_COLUMN

VARCHAR2(64)

The name of the metric in the alert that caused the corrective action to run. This is not used for policy violations.

METRIC_VALUE

VARCHAR2(1024)

The value of the metric in the alert that caused the corrective action to run.

SEVERITY_CODE

NUMBER

The severity code of the alert that caused the corrective action to run. See Table 3-6, "Severity Codes".

KEY_VALUE_NAME

VARCHAR2(512)

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

KEY_VALUE

VARCHAR2(1290)

For table metrics, this column contains the value of the key column for the row in the table whose thresholds are being defined. If the thresholds are not for a table metric, or the thresholds apply for all rows in the metric column, then the value in this column will contain a single space.

KEY_VALUE_GUID

RAW(16)

GUID associated with a composite key value name.

CTXT_LIST

MGMT_NOTIFY_COLUMNS

Details of the corrective action status change context.

RULE_OWNER

VARCHAR2(64)

The owner of the notification rule that caused the PL/SQL notification to be sent.

RULE_NAME

VARCHAR2(132)

The name of the notification rule that caused the PL/SQL notification method to be invoked.

OCCURRED_DATE

DATE

The time and date when the status change happened.


The following status codes are possible values for the job_status field of the MGMT_NOTIFY_CORRECTIVE_ACTION object.

Table 3-9 Corrective Action Status Codes

Name Datatype Value

SCHEDULED_STATUS

NUMBER(2)

1

EXECUTING_STATUS

NUMBER(2)

2

ABORTED_STATUS

NUMBER(2)

3

FAILED_STATUS

NUMBER(2)

4

COMPLETED_STATUS

NUMBER(2)

5

SUSPENDED_STATUS

NUMBER(2)

6

AGENTDOWN_STATUS

NUMBER(2)

7

STOPPED_STATUS

NUMBER(2)

8

SUSPENDED_LOCK_STATUS

NUMBER(2)

9

SUSPENDED_EVENT_STATUS

NUMBER(2)

10

SUSPENDED_BLACKOUT_STATUS

NUMBER(2)

11

STOP_PENDING_STATUS

NUMBER(2)

12

SUSPEND_PENDING_STATUS

NUMBER(2)

13

INACTIVE_STATUS

NUMBER(2)

14

QUEUED_STATUS

NUMBER(2)

15

FAILED_RETRIED_STATUS

NUMBER(2)

16

WAITING_STATUS

NUMBER(2)

17

SKIPPED_STATUS

NUMBER(2)

18

REASSIGNED_STATUS

NUMBER(2)

20


Example 3-14 PL/SQL Procedure Using a Status Code

CREATE TABLE ca_log (jobid RAW(16),                     occured DATE);


CREATE OR REPLACE PROCEDURE LOG_PROBLEM_CAS(status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION)

IS

BEGIN

 -- Log all failed corrective actions

   IF status_change.job_status = MGMT_JOBS.FAILED_STATUS

   THEN

         BEGIN

                 INSERT INTO ca_log (jobid, occured)

                 VALUES (status_change.job_guid, SYSDATE);

         EXCEPTION

         WHEN OTHERS

         THEN

         -- If there are any problems then get the notification retried

                 RAISE_APPLICATION_ERROR(-20000, 'Please retry');

                 END;

                 COMMIT;

   END IF;

END LOG_PROBLEM_CAS;

Passing Job Execution Status Information

Passing job status change attributes (such as new status, job name, job type, notification rule, or rule owner) to PL/SQL procedures or OS commands/scripts allows you to customize automated responses to status changes. For example, you many want to call an OS script to open a trouble ticket for an in-house support trouble ticket system if a critical job fails to run. In this case you will want to pass status (for example, Problems or Aborted) to the script to open a trouble ticket and escalate the problem.

Passing Job Execution Status to a PLSQL Procedure

The notification system passes job status change information to a PL/SQL procedure via the MGMT_NOTIFY_JOB object. An instance of this object is created for every status change. When a job changes status, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_JOB object that has been passed to it.

Table 3-10 lists all corrective action status change attributes that can be passed:

Table 3-10 Job Status Attributes

Attribute Datatype Additional Information

job_name

VARCHAR2(128)

The job name.

job_owner

VARCHAR2(256)

The owner of the job.

job_type

VARCHAR2(32)

The type of the job.

job_status

NUMBER

The new status of the job.

state_change_guid

RAW(16)

The GUID of the state change record.

job_guid

RAW(16)

The unique id of the job.

execution_id

RAW(16)

The unique id of the execution.

targets

SMP_EMD_NVPAIR_ARRAY

An array of the target name/target type pairs that the job runs on.

rule_owner

VARCHAR2(64)

The name of the notification rule that cause the notification to be sent.

rule_name

VARCHAR2(132)

The owner of the notification rule that cause the notification to be sent.

occurred_date

DATE

The time and date when the status change happened.


When a job status change occurs for the job, the notification system creates an instance of the MGMT_NOTIFY_JOB object and populates it with values from the status change. The following status codes have been defined as constants in the MGMT_JOBS package and can be used to determine the type of status in the job_status field of the MGMT_NOTIFY_JOB object.

Table 3-11 Job Status Codes

Name Datatype Value

SCHEDULED_STATUS

NUMBER(2)

1

EXECUTING_STATUS

NUMBER(2)

2

ABORTED_STATUS

NUMBER(2)

3

FAILED_STATUS

NUMBER(2)

4

COMPLETED_STATUS

NUMBER(2)

5

SUSPENDED_STATUS

NUMBER(2)

6

AGENTDOWN_STATUS

NUMBER(2)

7

STOPPED_STATUS

NUMBER(2)

8

SUSPENDED_LOCK_STATUS

NUMBER(2)

9

SUSPENDED_EVENT_STATUS

NUMBER(2)

10

SUSPENDED_BLACKOUT_STATUS

NUMBER(2)

11

STOP_PENDING_STATUS

NUMBER(2)

12

SUSPEND_PENDING_STATUS

NUMBER(2)

13

INACTIVE_STATUS

NUMBER(2)

14

QUEUED_STATUS

NUMBER(2)

15

FAILED_RETRIED_STATUS

NUMBER(2)

16

WAITING_STATUS

NUMBER(2)

17

SKIPPED_STATUS

NUMBER(2)

18

REASSIGNED_STATUS

NUMBER(2)

20


Example 3-15 PL/SQL Procedure Using a Status Code (Job)

CREATE TABLE job_log (jobid RAW(16),                    occured DATE);


CREATE OR REPLACE PROCEDURE LOG_PROBLEM_JOBS(status_change IN MGMT_NOTIFY_JOB)

IS

BEGIN

 -- Log all failed jobs

   IF status_change.job_status = MGMT_JOBS.FAILED_STATUS

   THEN

         BEGIN

                 INSERT INTO job_log (jobid, occured)

                 VALUES (status_change.job_guid, SYSDATE);

         EXCEPTION

         WHEN OTHERS

         THEN

         -- If there are any problems then get the notification retried

                 RAISE_APPLICATION_ERROR(-20000, 'Please retry');

                 END;

                 COMMIT;

   END IF;

END LOG_PROBLEM_JOBS;

Passing Job Execution Status to an OS Command or Script

The notification system passes job execution status information to an OS script or executable via system environment variables. Conventions used to access environmental variables vary depending on the operating system:

  • UNIX: $ENV_VARIABLE

  • MS Windows: %ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.

Table 3-12 Environment Variables

Environment Variable Description

JOB_NAME

The name of the job.

JOB_OWNER

The owner of the job.

JOB_TYPE

The type of job.

JOB_STATUS

The job status.

TIMESTAMP

Time when the severity occurred.

NUM_TARGETS

The number of targets.

TARGET_NAMEn

The name of the nth target. For example, TARGET_NAME1, TARGET_NAME2.

TARGET_TYPEn

The type of the nth target. For example TARGET_TYPE1, TARGET_TYPE2.

RULE_NAME

Name of the notification rule that resulted in the severity.

RULE_OWNER

Name of the Enterprise Manager administrator who owns the notification rule.


Passing User-Defined Target Properties to Notification Methods

Enterprise Manager allows you to define target properties (accessed via Related Links on the target home page) that can be used to store environmental or usage context information specific to that target. Target property values are passed to custom notification methods where they can be processed using conditional logic or simply passed as additional alert information to third-party devices, such as ticketing systems. By default, Enterprise Manager passes all defined target properties to notification methods.

Note:

Target properties are not passed to notification methods when short e-mail format is used.

Figure 3-4 Host Target Properties

Description of Figure 3-4 follows
Description of "Figure 3-4 Host Target Properties"

Assigning Methods to Rules

For each notification rule, you can assign one or more notification methods to be called when any of the criteria in the notification rule is met.

  1. From the Enterprise Manager Grid Control, click Preferences at the top of the page.

  2. Click Notification Rules in the vertical navigation bar.

    The Enterprise Manager Grid Control displays the Notification Rules page. Any notification rules already created are listed in the Notification Rules table.

  3. Click Assign Methods to Multiple Rules.

  4. Perform your assignments.

Figure 3-5 Assigning Methods to Rules

This graphic shows the Assign Methods to Multiple rule page
Description of "Figure 3-5 Assigning Methods to Rules"

Assigning Rules to Methods

For each notification method, you can associate one or more notification rules that will use that method to send notifications.

  1. From the Enterprise Manager Grid Control, click Preferences at the top of the page.

  2. Click Notification Rules in the vertical navigation bar.

    The Enterprise Manager Grid Control displays the Notification Rules page. Any notification rules already created are listed in the Notification Rules table.

  3. Click Assign Methods to Multiple Rules.

  4. From the View menu, select By Method.

  5. Perform your assignments.

Figure 3-6 Assign Rules to Methods

graphic shows the notification method page

Notification Coverage

To reduce the likelihood of an alert triggering and no administrator being notified because there was no notification rule covering that condition, you can use the Information Publisher (Enterprise Manager Report system) to view, for each target, the metrics monitored for that target and associated notification rules. Information Publisher provides an out-of-box report specifically designed for this purpose. You can run this report from the Report Definitions page (Reports tab) under Monitoring-->Alerts and Policy Violations--> Notification Rule Coverage for Metric Alerts and Availabilities (Target)

Management Information Base (MIB)

Enterprise Manager Grid Control can send SNMP Traps to third-party, SNMP-enabled applications. Details of the trap contents can be obtained from the management information base (MIB) variables. The following sections discuss Enterprise Manager MIB variables in detail.

About MIBs

A MIB is a text file, written in ASN.1 notation, which describes the variables containing the information that SNMP can access. The variables described in a MIB, which are also called MIB objects, are the items that can be monitored using SNMP. There is one MIB for each element being monitored. Each monolithic or subagent consults its respective MIB in order to learn the variables it can retrieve and their characteristics. The encapsulation of this information in the MIB is what enables master agents to register new subagents dynamically — everything the master agent needs to know about the subagent is contained in its MIB. The management framework and management applications also consult these MIBs for the same purpose. MIBs can be either standard (also called public) or proprietary (also called private or vendor).

The actual values of the variables are not part of the MIB, but are retrieved through a platform-dependent process called "instrumentation". The concept of the MIB is very important because all SNMP communications refer to one or more MIB objects. What is transmitted to the framework is, essentially, MIB variables and their current values.

Reading the MIB Variable Descriptions

This section covers the format used to describe MIB variables. Note that the STATUS element of SNMP MIB definition, Version 2, is not included in these MIB variable descriptions. Since Oracle has implemented all MIB variables as CURRENT, this value does not vary.

Variable Name

Syntax

Maps to the SYNTAX element of SNMP MIB definition, Version 2.

Max-Access

Maps to the MAX-ACCESS element of SNMP MIB definition, Version 2.

Status

Maps to the STATUS element of SNMP MIB definition, Version 2.

Explanation

Describes the function, use and precise derivation of the variable. (For example, a variable might be derived from a particular configuration file parameter or performance table field.) When appropriate, incorporates the DESCRIPTION part of the MIB definition, Version 2.

Typical Range

Describes the typical, rather than theoretical, range of the variable. For example, while integer values for many MIB variables can theoretically range up to 4294967295, a typical range in an actual installation will vary to a lesser extent. On the other hand, some variable values for a large database can actually exceed this "theoretical" limit (a "wraparound"). Specifying that a variable value typically ranges from 0 to 1,000 or 1,000 to 3 billion will help the third-party developer to develop the most useful graphical display for the variable.

Significance

Describes the significance of the variable when monitoring a typical installation. Alternative ratings are Very Important, Important, Less Important, or Not Normally Used. Clearly, the DBA will want to monitor some variables more closely than others. However, which variables fall into this category can vary from installation to installation, depending on the application, the size of the database, and on the DBA's objectives. Nevertheless, assessing a variable's significance relative to the other variables in the MIB can help third-party developers focus their efforts on those variables of most interest to the most DBAs.

Related Variables

Lists other variables in this MIB, or other MIBs implemented by Oracle, that relate in some way to this variable. For example, the value of this variable might derive from that of another MIB variable. Or perhaps the value of this variable varies inversely to that of another variable. Knowing this information, third-party developers can develop useful graphic displays of related MIB variables.

Suggested Presentation

Suggests how this variable can be presented most usefully to the DBA using the management application: as a simple value, as a gauge, or as an alarm, for example.

MIB Definition

Example 3-16 shows a typical MIB definition used by Enterprise Manager.

Example 3-16 MIB Definition

ORACLE-ENTERPRISE-MANAGER-4-MIB DEFINITIONS ::= BEGIN

IMPORTS

    TRAP-TYPE

        FROM RFC-1215

    DisplayString

        FROM RFC1213-MIB

    OBJECT-TYPE

        FROM RFC-1212

    enterprises

        FROM RFC1155-SMI;

oracle OBJECT IDENTIFIER ::= { enterprises  111 }

oraEM4 OBJECT IDENTIFIER ::= { oracle  15 }

oraEM4Objects OBJECT IDENTIFIER ::= { oraEM4  1 }

oraEM4AlertTable OBJECT-TYPE

    SYNTAX  SEQUENCE OF OraEM4AlertEntry

    ACCESS  not-accessible

    STATUS  mandatory

    DESCRIPTION

     "Information on alerts generated by Oracle Enterprise Manager. This table is

 not queryable; it exists only to document the variables included in the

 oraEM4Alert trap.  Each trap contains a single instance of each variable in the

 table."

    ::= { oraEM4Objects  1 }

oraEM4AlertEntry OBJECT-TYPE

    SYNTAX  OraEM4AlertEntry

    ACCESS  not-accessible

    STATUS  mandatory

    DESCRIPTION

     "Information about a particular Oracle Enterprise Manager alert."

    INDEX   { oraEM4AlertIndex }

    ::= { oraEM4AlertTable  1 }

OraEM4AlertEntry ::=

    SEQUENCE {

        oraEM4AlertIndex

            INTEGER,

        oraEM4AlertTargetName

       DisplayString,

        oraEM4AlertTargetType

       DisplayString,

        oraEM4AlertHostName

       DisplayString,

        oraEM4AlertMetricName

       DisplayString,

        oraEM4AlertKeyName

       DisplayString,

        oraEM4AlertKeyValue

       DisplayString,

        oraEM4AlertTimeStamp

       DisplayString,

        oraEM4AlertSeverity

       DisplayString,

        oraEM4AlertMessage

       DisplayString,

        oraEM4AlertRuleName

       DisplayString

        oraEM4AlertRuleOwner

       DisplayString

    oraEM4AlertMetricValue

           DisplayString,

        oraEM4AlertContext

           DisplayString

    }

oraEM4AlertIndex OBJECT-TYPE

    SYNTAX  INTEGER

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "Index of a particular alert, unique only at the moment an alert is generated."

    ::= { oraEM4AlertEntry  1 }

oraEM4AlertTargetName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the target to which this alert applies."

    ::= { oraEM4AlertEntry  2 }

oraEM4AlertTargetType OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The type of the target to which this alert applies."

    ::= { oraEM4AlertEntry  3 }

oraEM4AlertHostName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the host on which this alert originated."

    ::= { oraEM4AlertEntry  4 }

oraEM4AlertMetricName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the metric or policy which generated this alert."

    ::= { oraEM4AlertEntry  5 }

oraEM4AlertKeyName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the key-column, if present, for the metric which generated this alert."

    ::= { oraEM4AlertEntry  6 }

oraEM4AlertKeyValue OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The value of the key-column, if present, for the metric which generated this alert."

    ::= { oraEM4AlertEntry  7 }

oraEM4AlertTimeStamp OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The time at which this alert was generated."

    ::= { oraEM4AlertEntry  8 }

oraEM4AlertSeverity OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The severity of the alert e.g. Critical."

    ::= { oraEM4AlertEntry  9 }

oraEM4AlertMessage OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The message associated with the alert."

    ::= { oraEM4AlertEntry  10 }

oraEM4AlertRuleName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the notification rule that caused this notification."

    ::= { oraEM4AlertEntry  11 }

oraEM4AlertRuleOwner OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The owner of the notification rule that caused this notification."

    ::= { oraEM4AlertEntry  12 }

oraEM4AlertMetricValue OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The value of the metric which caused this alert to be generated."

    ::= { oraEM4AlertEntry  13 }

oraEM4AlertContext OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "A comma separated list of metric column names and values associated with the metric that caused this alert to be generated."

    ::= { oraEM4AlertEntry  14 }

oraEM4Traps OBJECT IDENTIFIER ::= { oraEM4  2 }

oraEM4Alert TRAP-TYPE

    ENTERPRISE  oraEM4Traps

    VARIABLES   { oraEM4AlertTargetName, oraEM4AlertTargetType,

                  oraEM4AlertHostName, oraEM4AlertMetricName,

                  oraEM4AlertKeyName, oraEM4AlertKeyValue, oraEM4AlertTimeStamp,

                  oraEM4AlertSeverity, oraEM4AlertMessage,

                  oraEM4AlertRuleName, oraEM4AlertRuleOwner,

                  oraEM4AlertMetricValue, oraEM4AlertContext }

    DESCRIPTION

     "The variables included in the oraEM4Alert trap."

    ::= 1

oraEM4JobAlertTable OBJECT-TYPE

    SYNTAX  SEQUENCE OF OraEM4JobAlertEntry

    ACCESS  not-accessible

    STATUS  mandatory

    DESCRIPTION

     "Information on alerts generated by Oracle Enterprise Manager. This table is

 not queryable; it exists only to document the variables included in the

 oraEM4JobAlert trap.  Each trap contains a single instance of each variable in

 the table."

    ::= { oraEM4Objects  2 }

oraEM4JobAlertEntry OBJECT-TYPE

    SYNTAX  OraEM4AlertEntry

    ACCESS  not-accessible

    STATUS  mandatory

    DESCRIPTION

     "Information about a particular Oracle Enterprise Manager alert."

    INDEX   { oraEM4JobAlertIndex }

    ::= { oraEM4JobAlertTable  1 }

OraEM4JobAlertEntry ::=

    SEQUENCE {

        oraEM4JobAlertIndex

            INTEGER,

        oraEM4JobAlertJobName

       DisplayString,

        oraEM4JobAlertJobOwner

       DisplayString,

        oraEM4JobAlertJobType

       DisplayString,

        oraEM4JobAlertJobStatus

       DisplayString,

        oraEM4JobAlertTargets

       DisplayString,

        oraEM4JobAlertTimeStamp

       DisplayString,

        oraEM4JobAlertRuleName

       DisplayString

        oraEM4JobAlertRuleOwner

       DisplayString,

    oraEM4JobAlertMetricName

           DisplayString,

    oraEM4JobAlertMetricValue

           DisplayString,

        oraEM4JobAlertContext

           DisplayString,

        oraEM4JobAlertKeyName

       DisplayString,

        oraEM4JobAlertKeyValue

       DisplayString,

        oraEM4JobAlertSeverity

       DisplayString

    }

oraEM4JobAlertIndex OBJECT-TYPE

    SYNTAX  INTEGER

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "Index of a particular alert, unique only at the moment an alert is

 generated."

    ::= { oraEM4JobAlertEntry  1 }

oraEM4JobAlertJobName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the job to which this alert applies."

    ::= { oraEM4JobAlertEntry  2 }

oraEM4JobAlertJobOwner OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The owner of the job to which this alert applies."

    ::= { oraEM4JobAlertEntry  3 }

oraEM4JobAlertJobType OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The type of the job to which this alert applies."

    ::= { oraEM4JobAlertEntry  4 }

oraEM4JobAlertJobStatus OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The status of the job to which this alert applies."

    ::= { oraEM4JobAlertEntry  5 }

oraEM4JobAlertTargets OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "A comma separated list of target to which this alert applies."

    ::= { oraEM4JobAlertEntry  6 }

oraEM4JobAlertTimeStamp OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The time at which this job status changed causing this alert."

    ::= { oraEM4JobAlertEntry  7 }

oraEM4JobAlertRuleName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the notification rule that caused this notification."

    ::= { oraEM4JobAlertEntry  8 }

oraEM4JobAlertRuleOwner OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The owner of the notification rule that caused this notification."

    ::= { oraEM4JobAlertEntry  9 }

oraEM4JobAlertMetricName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the metric or policy which caused the Corrective Action to run

 that caused this alert."

    ::= { oraEM4JobAlertEntry  10 }

oraEM4JobAlertMetricValue OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The value of the metric which caused the Corrective Action to run that

 caused this alert."

    ::= { oraEM4JobAlertEntry  11 }

oraEM4JobAlertContext OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "A comma separated list of metric column names and values associated with the

 metric which caused the Corrective Action to run that caused this alert."

    ::= { oraEM4JobAlertEntry  12 }

oraEM4JobAlertKeyName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the key-column, if present, for the metric which caused the

 Corrective Action to run that generated this alert."

    ::= { oraEM4JobAlertEntry  13 }

oraEM4JobAlertKeyValue OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The value of the key-column, if present, for the metric which caused the

 Corrective Action to run that generated this alert."

    ::= { oraEM4JobAlertEntry  14 }

oraEM4JobAlertSeverity OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The severity of the metric which caused the Corrective Action to run that

 generated this alert e.g. Critical."

    ::= { oraEM4JobAlertEntry  15 }

oraEM4JobAlert TRAP-TYPE

    ENTERPRISE  oraEM4Traps

    VARIABLES   { oraEM4JobAlertJobName, oraEM4JobAlertJobOwner,

                  oraEM4JobAlertJobType, oraEM4JobAlertJobStatus,

                  oraEM4JobAlertTargets, oraEM4JobAlertTimeStamp,

                  oraEM4JobAlertRuleName, oraEM4JobAlertRuleOwner,

                  oraEM4JobAlertMetricName, oraEM4JobAlertMetricValue,

                  oraEM4JobAlertContext, oraEM4JobAlertKeyName,

                  oraEM4JobAlertKeyValue, oraEM4JobAlertSeverity }

    DESCRIPTION

     "The variables included in the oraEM4JobAlert trap."

    ::= 2

END

Troubleshooting Notifications

To function properly, the notification system relies on various components of Enterprise Manager and your IT infrastructure. For this reason, there can be many causes of notification failure. The following guidelines and suggestions can help you isolate potential problems with the notification system.

General Setup

The first step in diagnosing notification issues is to ensure that you have properly configured and defined your notification environment.

OS Command, PL/SQL and SNMP Trap Notifications

Make sure all OS Command, PLSQL and SNMP Trap Notification Methods are valid by clicking the Test button. This will send a test notification and show any problems the OMS has in contacting the method. Make sure that your method was called, for example, if the OS Command notification is supposed to write information to a log file, check that it has written information to its log file.

E-mail Notifications

  • Make sure an e-mail gateway is set up under the Notification Methods page of Setup. The Sender's e-mail address should be valid. Clicking the Test button will send an e-mail to the Sender's e-mail address. Make sure this e-mail is received. Note that the Test button ignores any Notification Schedule.

  • Make sure an e-mail address is setup under General page of Preferences. Clicking the Test button will send an e-mail to specified address and you should make sure this e-mail is received. Note that the Test button ignores any Notification Schedule.

  • Make sure an e-mail schedule is defined under the Schedule page of Preferences. No e-mails will be sent unless a Notification Schedule has been defined.

  • Make sure a Notification Rule is defined to match the target, metric, severity and availability states you are interested and make sure e-mail and notification methods are assigned to the rule. A summary of the notification rule can be checked by going to the Rules page under Setup and clicking the rule name.

Notification System Errors

For any alerts involving problems with notifications, check the following for notification errors.

  • Any serious errors in the Notification System are logged as system errors in the MGMT_SYSTEM_ERROR_LOG table. These errors can be seen in the Errors page under Management Services and Repository under Setup.

  • Check for any delivery errors. From the Alerts section of a target home page, click on the alert message to access the metric details page. In the Alert History section, click on the Details icon for more information about the alert. The details will give the reason why the notification was not delivered. Delivery errors are stored in MGMT_NOTIFICATION_LOG with the DELIVERED column set to 'N'.

  • Severities will not be displayed in the Grid Control console if no metric values have been loaded for the metric associated with the severity.

Notification System Trace Messages

The Notification System can produce trace messages in sysman/log/emoms.trc file.

Tracing is configured by setting the log4j.em.notification property flag using the emctl set property command. You can set the trace level to INFO, WARN, DEBUG. For example,

./emctl set property -sysman_pwd your_sysman_password -name log4j.em.notification -value DEBUG

Trace messages contain the string "em.notification". If you are working in a UNIX environment, you can search for messages in the emoms.trc file using the grep command. For example,

grep em.notification emoms.trc

What to look for in the trace file.

The following entries in the emoms.trc file are relevant to notifications.

Normal Startup Messages

When the OMS starts, you should see these types of messages.

2006-11-08 03:18:45,385 [Orion Launcher] INFO  em.notification init.1279 - Short format maximum length is 155


2006-11-08 03:18:45,386 [Orion Launcher] INFO  em.notification init.1297 - Short format is set to both subject and body


2006-11-08 03:18:45,387 [NotificationMgrThread] INFO  em.notification run.1010 - Waiting for connection to EM Repository...


2006-11-08 03:18:46,006 [NotificationMgrThread] INFO  em.notification run.1041 - Registering for Administrative Queue Name...


2006-11-08 03:18:46,250 [NotificationMgrThread] INFO  em.notification run.1078 - Administrative Queue is ADM21


2006-11-08 03:18:46,250 [NotificationMgrThread] INFO  em.notification run.1089 - Creating thread pool: min = 6 max = 24


2006-11-08 03:18:48,206 [NotificationMgrThread] INFO  em.notification handleAdminNotification.655 - Handling notifications for EMAIL1

Notification Delivery Messages

2006-11-08 03:18:45,387 [NotificationMgrThread] INFO  em.notification run.682 - Notification ready on EMAIL1


2006-11-08 03:18:46,006 [DeliveryThread-EMAIL1] INFO  em.notification run.114 - Deliver to SYSMAN/admin@oracle.com


2006-11-08 03:18:47,006 [DeliveryThread-EMAIL1] INFO  em.notification run.227 - Notification handled for SYSMAN/admin@oracle.com

Notification System Error Messages

2006-11-08 07:26:30,242 [NotificationMgrThread] ERROR  em.notification getConnection.237 - Failed to get a connection Io exception: The Network Adapter could not establish the connection

E-mail Errors

The SMTP gateway is not set up correctly:

Failed to send e-mail to my.admin@oracle.com: For e-mail notifications to be sent, your Super Administrator must configure an Outgoing Mail (SMTP) Server within Enterprise Manager. (SYSMAN, myrule)

Invalid host name:

Failed to connect to gateway: badhost.us.oracle.com: Sending failed;

nested exception is:

javax.mail.MessagingException: Unknown SMTP      host: badhost.us.oracle.com;

Invalid e-mail address:

Failed to connect to gateway: rgmemeasmtp.oraclecorp.com: Sending failed;

nested exception is:

javax.mail.MessagingException: 550 5.7.1         <smpemailtest_ie@oracle.com>... Access denied

Always use the Test button to make sure the e-mail gateway configuration is valid. Check that an e-mail is received at the sender's e-mail address

OS Command Errors

When attempting to execute an OS command or script, the following errors may occur. Use the Test button to make sure OS Command configuration is valid. If there are any errors, they will appear in the console.

Invalid path or no read permissions on file:

Could not find /bin/myscript (stacb10.us.oracle.com_Management_Service) (SYSMAN, myrule )

No execute permission on executable:

Error calling /bin/myscript: java.io.IOException: /bin/myscript: cannot execute (stacb10.us.oracle.com_Management_Service) (SYSMAN, myrule ) 

Timeout because OS Command ran too long:

Timeout occurred running /bin/myscript (stacb10.us.oracle.com_Management_Service) (SYSMAN, myrule )

Any errors such as out of memory or too many processes running on OMS machine will be logged as appropriate.

Always use the Test button to make sure OS Command configuration is valid.

SNMP Trap Errors

Use the Test button to make sure SNMP Trap configuration is valid.

Other possible SNMP trap problems include: invalid host name, port, or community for a machine running an SNMP Console.

PL/SQL Errors

When attempting to execute an PL/SQL procedure, the following errors may occur. Use the Test button to make sure the procedure is valid. If there are any errors, they will appear in the console.

Procedure name is invalid or is not fully qualified. Example: SCOTT.PKG.PROC

Error calling PL/SQL procedure plsql_proc: ORA-06576: not a valid function or procedure name (SYSMAN, myrule) 

Procedure is not the correct signature. Example: PROCEDURE p(s IN MGMT_NOTIFY_SEVERITY)

Error calling PL/SQL procedure plsql_proc: ORA-06553: PLS-306: wrong number or types of arguments in call to 'PLSQL_PROC' (SYSMAN, myrule)

Procedure has bug and is raising an exception.

Error calling PL/SQL procedure plsql_proc: ORA-06531: Reference to uninitialized collection (SYSMAN, myrule)

Care should be taken to avoid leaking cursors in your PL/SQL. Any exception due to this condition will result in delivery failure with the message being displayed in the Details section of the alert in the Grid Control console.

Always use the Test button to make sure the PL/SQL configuration is valid.