Skip Headers
Oracle® Enterprise Manager Cloud Control Administrator's Guide
12c Release 4 (12.1.0.4)

E24473-30
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Index
Index
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
PDF · Mobi · ePub

4 Using Notifications

The notification system allows you to notify Enterprise Manager administrators when specific incidents, events, or problems arise.

Note:

This chapter assumes that you are familiar with incident management. For information about monitoring and managing your IT infrastructure via incident management, see Chapter 3, "Using Incident Management".

As an integral part of the management framework, notifications can also perform actions such as executing operating system commands (including scripts) and PL/SQL procedures when specific incidents, events, or problems occur. This capability allows you to automate IT practices. For example, if an incident (such as monitoring of the operational (up/down) status of a database) arises, you may want the notification system to automatically open an in-house trouble-ticket using an OS script so that the appropriate IT staff can respond in a timely manner.

By using Simple Network Management Protocol (SNMP) traps, the Enterprise Manager notification system also allows you to send traps to SNMP-enabled third-party applications such as HP OpenView for events published in Enterprise Manager. Some administrators may want to send third-party applications a notification when a certain metric has exceeded a threshold.

This chapter covers the following:

4.1 Setting Up Notifications

All Enterprise Manager administrators can set up e-mail notifications for themselves. Super Administrators also have the ability to set up notifications for other Enterprise Manager administrators.

4.1.1 Setting Up a Mail Server for Notifications

Before Enterprise Manager can send e-mail notifications, you must first specify the Outgoing Mail (SMTP) servers to be used by the notification system. Once set, you can then define e-mail notifications for yourself or, if you have Super Administrator privileges, you can also define notifications for other Enterprise Manager administrators.

You specify the Outgoing Mail (SMTP) server on the Notification Methods page (Figure 4-1). To display the Notification Methods page, from the Setup menu, select Notifications, then select Notification Methods.

Note:

You must have Super Administrator privileges in order to set up SMTP servers.

Specify one or more outgoing mail server names, the mail server authentication credentials (User Name, Password, and Confirm Password), if required, the name you want to appear as the sender of the notification messages, and the e-mail address you want to use to send your e-mail notifications. This address, called the Sender's Mail Address, must be a valid address on each mail server that you specify. A message will be sent to this e-mail address if any problem is encountered during the sending of an e-mail notification. Example 4-1 shows sample notification method entries.

Example 4-1 Mail Server Settings

  • Outgoing Mail (SMTP) Server - smtp01.example.com:587, smtp02.example.com

  • User Name - myadmin

  • Password - ******

  • Confirm Password - ******

  • Identify Sender As - Enterprise Manager

  • Sender's E-mail Address - mgmt_rep@example.com

  • Use Secure Connection - No: E-mail is not encrypted. SSL: E-mail is encrypted using the Secure Sockets Layer protocol. TLS, if available: E-mail is encrypted using the Transport Layer Security protocol if the mail server supports TLS. If the server does not support TLS, the e-mail is automatically sent as plain text.

Figure 4-1 Defining a Mail Server

Description of Figure 4-1 follows
Description of "Figure 4-1 Defining a Mail Server"

Note:

The e-mail address you specify on this page is not the e-mail address to which the notification is sent. You will have to specify the e-mail address (where notifications will be sent) from the Password and E-mail page. From the Setup menu, choose MyPreferences and then Enterprise Manager Password & E-mail.

As standard practice, each user should have their own e-mail address.

After configuring the e-mail server, click Test Mail Servers to verify your e-mail setup. You should verify that an e-mail message was received by the e-mail account specified in the Sender's E-mail Address field.

Defining multiple mail servers will improve the reliability of e-mail notification delivery. E-mail notifications will be delivered if at least one e-mail server is up. The notification load is balanced across multiple e-mail servers by the OMS, which switches through them (servers are allocated according to availability) after 20 e-mails have been sent. Switching is controlled by the oracle.sysman.core.notification.emails_per_connection emoms property.

Setting the Cloud Control Console URL when Using an SLB

If you have a multi-OMS environment with a Server Load Balancer (SLB) configured for the OMS instances, you should update the console URL to ensure that any emails from Enterprise Manager direct you to the Enterprise Manager console through the SLB URL and not the specific OMS URL from which the email may have originated.

To change the console URL:

  1. From the Setup menu, select Manage Cloud Control, and then Health Overview. The Management Services and Repository page displays.

  2. On the Management Services and Repository page, in the Overview section, click Add/Edit against the Console URL label.

    edit console url

    The Console URL page displays. console URL change page

  3. Modify the Console URL to the SLB URL.

    Examples:

    http://www.example.com

    https://www.example.com:4443.

    Note that path, typically /em, should not be specified.

  4. Click OK.

4.1.2 Setting Up E-mail for Yourself

If you want to receive notifications by e-mail, you will need to specify your e-mail address(s) in the Password & E-mail page (from the Setup menu, select MyPreferences, then select Enterprise Manager Password & E-mail). In addition to defining notification e-mail addresses, you associate the notification message format (long, short, pager) to be used for your e-mail address.

Setting up e-mail involves three steps:

Step 1: Define an e-mail addresses.

Step 2: Set up a Notification Schedule.

Step 3: Subscribe to incident rules in order to receive e-mails.

4.1.2.1 Defining E-mail Addresses

An e-mail address can have up to 128 characters. There is no upper limit with the number of e-mail addresses.

To add an e-mail address:

  1. From username drop-down menu, select Enterprise Manager Password & E-mail.

  2. Click Add Another Row to create a new e-mail entry field in the E-mail Addresses table.

  3. Specify the e-mail associated with your Enterprise Manager account. All e-mail notifications you receive from Enterprise Manager will be sent to the e-mail addresses you specify.

    For example, user1@oracle.com

    Select the E-mail Type (message format) for your e-mail address. E-mail (Long) sends a HTML formatted e-mail that contains detailed information. Example 4-2 shows a typical notification that uses the long format.

    E-mail (Short) and Pager(Short) (Example 4-3) send a concise, text e-mail that is limited to a configurable number of characters, thereby allowing the e-mail be received as an SMS message or page. The content of the message can be sent entirely in the subject, entirely in the body or split across the subject and body.

    For example, in the last case, the subject could contain the severity type (for example, Critical) and the target name. The body could contain the time the severity occurred and the severity message. Since the message length is limited, some of this information may be truncated. If truncation has occurred there will be an ellipsis end of the message. Pager(Short) addresses are used for supporting the paging feature in incident rules. Note that the incident rules allow the rule author to designate some users to receive a page for critical issues.

  4. Click Apply to save your e-mail address.

Example 4-2 Long E-mail Notification for Metric Alerts

Target type=Host Target name=machine6140830.example.com Message=Filesystem / has 54.39% available space, fallen below warning (60) or critical (30) threshold. Severity=Warning Event reported time=Apr 28, 2011 2:33:55 PM PDT Event Type=Metric Alert Event name=Filesystems:Filesystem Space Available (%) Metric Group=FilesystemsMetric=Filesystem Space Available (%)Metric value=54.39Key Value=/Key Column 1=Mount PointRule Name=NotifRuleSet1,Event rule1 Rule Owner=SYSMAN

Example 4-3 Short E-mail Notification for Alerts

Subject is :
EM:Unreachable Start:myhost
Body is :
Nov 16, 2006 2:02:19 PM EST:Agent is Unreachable (REASON = Connection refused) 
but the host is UP

More about E-mail(Short) and Pager(Short) Formats

Enterprise Manager does not directly support message services such as paging or SMS, but instead relies on external gateways to, for example, perform the conversion from e-mail to page. Beginning with Enterprise Manager 12c, the notification system allows you to tag e-mail addresses explicitly as 'page' or 'e-mail'. Explicit system differentiation between these two notification methods allows you to take advantage of the multiple action capability of incident rules. For example, the e-mail versus page distinction is required in order to send you an e-mail if an event severity is 'warning' or page you if the severity is 'critical'. To support this capability, a Pager format has been made available that sends an abbreviated version of the short format e-mail.

Note:

To receive a Page, an administrator should be added to the Page Notification option in the Incident Rule.

4.1.2.2 Setting Up a Notification Schedule

Once you have defined your e-mail notification addresses, you will need to define a notification schedule. For example, if your e-mail addresses are user1@oracle.com, user2@oracle.com, user3@oracle.com, you can choose to use one or more of these e-mail addresses for each time period in your notification schedule. Only e-mail addresses that have been specified with your user preferences (Enterprise Manager Password and Email page) can be used in the notification schedule.

Note:

When you enter e-mail addresses for the first time, a 24x7 weekly notification schedule is set automatically. You can then review and modify the schedule to suit your monitoring needs.

A notification schedule is a repeating schedule used to specify your on-call schedule—the days and time periods and e-mail addresses that should be used by Enterprise Manager to send notifications to you. Each administrator has exactly one notification schedule. When a notification needs to be sent to an administrator, Enterprise Manager consults that administrator's notification schedule to determine the e-mail address to be used. Depending on whether you are Super Administrator or a regular Enterprise Manager administrator, the process of defining a notification schedule differs slightly.

Figure 4-2 Notification Schedule Page

Graphic shows notification schedule page.

If you are a regular Enterprise Manager administrator and are defining your own notification schedule:

  1. From Setup menu, select Notifications, then select My Notification Schedule.

  2. Follow the directions on the Notification Schedule page to specify when you want to receive e-mails.

4.1.2.3 Subscribe to Receive E-mail for Incident Rules

An incident rule is a user-defined rule that specifies the criteria by which notifications should be sent for specific events that make up the incident. An incident rule set, as the name implies, consists of one or more rules associated with the same incident.

When creating an incident rule, you specify criteria such as the targets you are interested in, the types of events to which you want the rule to apply. Specifically, for a given rule, you can specify the criteria you are interested in and the notification methods (such as e-mail) that should be used for sending these notifications. For example, you can set up a rule that when any database goes down or any database backup job fails, e-mail should be sent and the "log trouble ticket" notification method should be called. Or you can define another rule such that when the CPU or Memory Utilization of any host reach critical severities, SNMP traps should be sent to another management console. Notification flexibility is further enhanced by the fact that with a single rule, you can perform multiple actions based on specific conditions. Example: When monitoring a condition such as machine memory utilization, for an incident severity of 'warning' (memory utilization at 80%), send the administrator an e-mail, if the severity is 'critical' (memory utilization at 99%), page the administrator immediately.

You can subscribe to a rule you have already created.

  1. From the Setup menu, select Incidents, then select Incident Rules.

  2. On the Incident Rules - All Enterprise Rules page, click on the rule set containing incident escalation rule in question and click Edit... Rules are created in the context of a rule set.

    Note: In the case where there is no existing rule set, create a rule set by clicking Create Rule Set... You then create the rule as part of creating the rule set.

  3. In the Rules section of the Edit Rule Set page, highlight the escalation rule and click Edit....

  4. Navigate to the Add Actions page.

  5. Select the action that escalates the incident and click Edit...

  6. In the Notifications section, add the DBA to the E-mail cc list.

  7. Click Continue and then navigate back to the Edit Rule Set page and click Save.

Out-of-Box Incident Rules

Enterprise Manager comes with two incident rule sets that cover the most common monitoring conditions, they are:

  • Incident Management Ruleset for All Targets

  • Event Management Ruleset for Self Update

If the conditions defined in the out-of-box incident rules meet your requirements, you can simply subscribe to receive e-mail notifications for the conditions defined in the rule using the subscribe procedure shown in the previous section.

The out-of-box incident rule set for all targets does not generate emails for warning alerts by default.

Creating Your Own Incident Rules

You can define your own custom rules. The following procedure documents the process of incident rule creation for non-Super Administrators.

To create your own incident rule:

  1. From the Setup menu, select Incidents, then select Incident Rules.

    The Incident Rules page displays. From this page you can create a new rule set, to which you can add new rules. Alternatively, if you have the requisite permissions, you can add new rules to existing

  2. Click Create Rule Set...

    The create rule set page displays.

  3. Specify the Name, Description, and the Targets to which the rules set should apply.

  4. Click the Rules tab, then click Create.

  5. Choose the incoming incident, event or problem to which you want the rule to apply. See "Setting Up Rule Sets" for more information.

  6. Click Continue.

    Enterprise Manager displays the Create Incident Rule pages. Enter the requisite information on each page to create your incident rule.

  7. Follow the wizard instructions to create your rule.

    Once you have completed defining your rule, the wizard returns you to the create rule set page.

  8. Click Save to save the incident rule set.

4.1.3 Setting Up E-mail for Other Administrators

If you have Super Administrator privileges, you can set up e-mail notifications for other Enterprise Manager administrators. To set up e-mail notifications for other Enterprise Manager administrators, you need to:

Step 1: Ensure Each Administrator Account has an Associated E-mail Address

Each administrator to which you want to send e-mail notifications must have a valid e-mail address.

  1. From the Setup menu, select Security and then Administrators.

  2. For each administrator, define an e-mail address. This sets up a 24x7 notification schedule for this user that uses all the e-mail addresses specified. By default, this adds the Email ID with type set to Email Long. It is not possible to specify the Email Type option here.

Enterprise Manager also allows you to specify an administrator address when editing an administrator's notification schedule.

Step 2: Define Administrators' Notification Schedules

Once you have defined e-mail notification addresses for each administrator, you will need to define their respective notification schedules. Although a default 24x7 notification schedule is created when you specify an e-mail address for the first time, you should review and edit the notification schedule as needed.

  1. From the Setup menu, select Notifications, then select Notification Schedule.

    From the vertical navigation bar, click Schedules (under Notification). The Notification Schedule page appears.

  2. Specify the administrator who's notification schedule you wish to edit and click Change.

  3. Click Edit Schedule Definition. The Edit Schedule Definition: Time Period page appears. If necessary, modify the rotation schedule.

  4. Click Continue. The Edit Schedule Definition: E-mail Addresses page appears.

  5. Follow the directions on the Edit Schedule Definition: E-mail Addresses page to modify the notification schedule.

  6. Click Finish when you are done.

  7. Repeat steps three through seven for each administrator.

Step 3: Assign Incident Rules to Administrators

With the notification schedules set, you now need to assign the appropriate incident rules for each designated administrator.

  1. From the Setup menu, select Incidents, then select Incident Rules.

  2. Select the desired Ruleset and click Edit.

  3. Click on the Rules tab, select the desired rule and click Edit.

  4. Click Add Actions, select desire action and click Edit.

  5. Enter the Administrator name on either E-mail To or E-mail Cc field in the Basic Notification region.

  6. Click Continue, click Next, click Next, click Continue, and finally click Save.

4.1.4 E-mail Customization

Enterprise Manager allows Super Administrators to customize global e-mail notifications for the following types: All events, incidents, problems, and specific event types installed. You can alter the default behavior for all events by customizing Default Event Email Template. In addition, you can further customize the behavior for a specific event type by customizing the template for the event type. For instance, you can customize the Metric Alert Events template for the metric alert event type. Using predefined building blocks (called attributes and labels) contained within a simple script, Super Administrators can customize alert e-mails by selecting from a wide variety of information content.

To customize an e-mail:

  1. From the Setup menu, select Notifications, then select Customize Email Formats.

  2. Choose the Type and Format.

  3. Click Customize. The Customize Email Template page displays.

From the Customize E-mail Template page, you can modify the content of the e-mail template Enterprise Manager uses to generate e-mail notifications. Extensive information on script formatting, syntax, and options is available from the Edit E-mail Template page via imbedded assistance and online help.

Figure 4-3 E-mail Customization

Description of Figure 4-3 follows
Description of "Figure 4-3 E-mail Customization"

4.1.4.1 E-mail Customization Reference

The following reference summarizes the semantics and component syntax of the pseudo-language used to define e-mails. The pseudo-language provides you with a simple, yet flexible way to customize e-mail notifications. The following is a summary of pseudo-language conventions/limitations:

  • You can add comments (or any free-form text) using separate lines beginning with "--" or at end of lines.

  • You can use attributes.

  • You can use IF & ELSE & ENDIF control structures. You can also use multiple conditions using "AND" or "OR". Nested IF statements are not supported.

  • You can insert spaces for formatting purposes. Spaces at the beginning of a line will be ignored in the actual e-mail. To insert spaces at the beginning of a line, use the [SP] attribute.

  • Use "/" to escape and "[" or "]" if you want to add attribute names, operators, or IF clauses to the actual e-mail.

  • HTML is not supported.

Reserved Words and Operators

The following table lists all reserved words and operators used when modifying e-mail scripts.

Table 4-1 Reserved Words and Operators

Reserved Word/Operator Description

IF, ELSIF, ENDIF, ELSE

Used in IF-ELSE constructs.

AND, OR

Boolean operators – used in IF-ELSE constructs only.

NULL

To check NULL value for attributes - used in IF-ELSE constructs only.

|

Pipe operator – used to show the first non-NULL value in a list of attributes.

For example:

METRIC_NAME|SEVERITY

EQ, NEQ

Equal and Not-Equal operators – applicable to NULL, STRING and NUMERIC values.

/

Escape character – used to escape reserved words and operators. Escape characters signify that what follows the escape character takes an alternative interpretation.

[ , ]

Delimiters used to demarcate attribute names and IF clauses.


Syntax Elements

Literal Text

You can specify any text as part of the e-mail content. The text will be displayed in the e-mail and will not be translated if the Oracle Management Services (OMS) language setting is changed. For example, ’my Oracle Home' appears as ’my Oracle Home' in the generated e-mail.

Predefined Attributes

Predefined attributes/labels will be substituted with actual values in a specific context. To specify a predefined attribute/label, use the following syntax:

[PREDEFINED_ATTR]

Attribute names can be in either UPPER or LOWER case. The parsing process is case-insensitive.

A pair of square brackets is used to distinguish predefined attributes from literal text. For example, for a job e-mail notification, the actual job name will be substituted for [EXECUTION_STATUS]. For a metric alert notification, the actual metric column name will be substituted for [METIRC_COLUMN].

You can use the escape character ”/” to specify words and not have them interpreted as predefined labels/attributes. For example, "/[NEW/]” will not be considered as the predefined attribute [NEW] when parsed.

Operators

EQ, NEQ – for text and numeric values

NULL- for text and numeric values

GT, LT, GE, LE – for numeric values

Control Structures

The following table lists acceptable script control structures.

Table 4-2 Control Structures

Control Structure Description

Pipe "|"

Two or more attributes can be separated by ’|' character. For example,

[METRIC_NAME|SEVERITY]

In this example, only the applicable attribute within the current alert context will be used (replaced by the actual value) in the e-mail. If more than one attribute is applicable, only the left-most attribute is used.

IF

Allows you to make a block of text conditional. Only one level of IF and ELSIF is supported. Nested IF constructs are not supported.

All attributes can be used in IF or ELSIF evaluation using EQ/NEQ operators on NULL values. Other operators are allowed for ”SEVERITY” and ”REPEAT_COUNT” only.

Inside the IF block, the values need to be contained within quotation marks ” ”. Enterprise Manager will extract the attribute name and its value based on the position of ”EQ” and other key words such as ”and”, ”or”. For example,

[IF REPEAT_COUNT EQ ”1” AND SEVERITY EQ ”CRITICAL” THEN]

The statement above will be true when the attributes of the alert match the following condition:

  • Attribute Name: REPEAT_COUNT

  • Attribute Value: 1

  • Attribute Name: SEVERITY

  • Attribute Value: CRITICAL

Example IF Block:

[IF EXECUTION_STATUS NEQ NULL]
       [JOB_NAME_LABEL]=[EXECUTION_STATUS] 
       [JOB_OWNER_LABEL]=[JOB_OWNER] 
[ENDIF] 
 
[IF SEVERITY_CODE EQ CRITICAL ]
       [MTRIC_NAME_LABEL]=[METRIC_GROUP] 
       [METRIC_VALUE_LABEL]=[METRIC_VALUE] 
       [TARGET_NAME_LABEL]=[TARGET_NAME] 
       [KEY_VALUES] 
[ENDIF] 

Example IF and ELSEIF Block:

[IF SEVERITY_CODE EQ CRITICAL]           statement1[ELSIF SEVERITY_CODE EQ WARNING]           statement2[ELSIF SEVERITY_CODE EQ CLEAR]           statement3[ELSE]           statement4[ENDIF]

Comments

You can add comments to your script by prefacing a single line of text with two hyphens "--". For example,

 -- Code added on 8/3/2009     [IF REPEAT_COUNT NEQ NULL]     . . . 

Comments may also be placed at the end of a line of text.

[IF SEVERITY_SHORT EQ W] -- for Warning alert

HTML Tags in Customization Content

Use of HTML tags is not supported.

When Enterprise Manager parses the e-mail script, it will convert the ”<” and ”>” characters of HTML tags into encoded format (&lt; and &gt;). This ensures that the HTML tag is not treated as HTML by the destination system.

Examples

E-mail customization template scripts support three main operators.

  • Comparison operators: EQ/NEQ/GT/LT/GE/LELogic operators: AND/ORPipeline operator: |

4.1.5 Setting Up Repeat Notifications

Repeat notifications allow administrators to be notified repeatedly until an incident is either acknowledged or the number of Maximum Repeat Notifications has been reached. Enterprise Manager supports repeat notification for all notification methods (e-mail, OS command, PL/SQL procedure, and SNMP trap).

Configuring Repeat Notifications Globally

To enable repeat notifications for a notification method (globally), select the Send Repeat Notifications option on the Notification Methods page . In addition to setting the maximum number of repeat notifications, you can also set the time interval at which the notifications are sent.

Important:

For Oracle database versions 10 and higher, it is recommend that no modification be made to aq_tm_processes init.ora parameter. If, however, this parameter must be modified, its value should be at least one for repeat notification functionality. If the Enterprise Manager Repository database version is 9.2, the aq_tm_processes init.ora parameter must be set to at least one to enable repeat notification functionality.

Configuring Repeat Notifications Via Incident Rules

Setting repeat notifications globally at the notification method level may not provide sufficient flexibility. For example, you may want to have different repeat notification settings based on event type. Enterprise Manager accomplishes this by allowing you to set repeat notifications for individual incident rule sets or individual rules within a rule set. Repeat notifications set at the rule level take precedence over those defined at the notification method level.

Important:

Repeat notifications will only be sent if the Send Repeat Notifications option is enabled in the Notification Methods page.

Non-E-mail Repeat Notifications

For non-e-mail repeat notifications (PL/SQL, OS command, and SNMP trap notification methods), you must enable each method to support repeat notifications. You can select Supports Repeat Notifications option when adding a new notification method or by editing an existing method.

Figure 4-4 Enabling Repeat Notification for an OS Command Notification Method

Description of Figure 4-4 follows
Description of "Figure 4-4 Enabling Repeat Notification for an OS Command Notification Method"

4.2 Extending Notification Beyond E-mail

Notification Methods are the mechanisms by which notifications are sent. Enterprise Manager Super Administrators can set up e-mail notifications by configuring the 'e-mail' notification method. Most likely this would already have been set up as part of the Oracle Management Service installation.

Enterprise Manager Super Administrators can also define other custom notification methods. For example, event notifications may need to be forwarded to a 3rd party trouble-ticketing system. Assuming APIs to the third-party trouble-ticketing system are available, a custom notification method can be created to call a custom OS script that has the appropriate APIs. The custom notification method can be named in a user-friendly fashion, for example, "Log trouble ticket". Once the custom method is defined, whenever an administrator needs to send alerts to the trouble-ticketing system, he simply needs to invoke the now globally available notification method called "Log trouble ticket".

Custom notification methods can be defined based on any custom OS script, any custom PL/SQL procedure, or by sending SNMP traps. A fourth type of notification method (Java Callback) exists to support Oracle internal functionality and cannot be created or edited by Enterprise Manager administrators.

Only Super Administrators can define OS Command, PL/SQL, and SNMP Trap notification methods. However, any Enterprise Manager administrator can add these notification methods (once defined by the Super Administrator) as actions to their incident rules.

Through the Notification Methods page, you can:

  • Set up the outgoing mail servers if you plan to send e-mail notifications through incident rules

  • Create other custom notification methods using OS and PL/SQL scripts and SNMP traps.

  • Set global repeat notifications.

4.3 Sending Notifications Using OS Commands and Scripts

Notification system can invoke a custom script when an incident rule matches the OS Command advanced notification action. A custom script receives notifications for matching events, incidents and problem through environment variables.

The length of any environment variable's value is limited to 512 characters by default. Configure emoms property named oracle.sysman.core.notification.oscmd.max_env_var_length for changing the default limit.

Important:

Notification methods based on OS commands must be configured by an administrator with Super Administrator privileges.

Running OS Command Scripts:

Running an OS command such as "sudo" for receiving notifications will fail because the command does not have read permission of the OMS account. The OMS account must have read permission over the OS command in order to send notifications.

To overcome the permissions problem, embed the command in a wrapper script that is readable by the OMS administrator account. Once the command is contained within the wrapper script, you then specify this script in place of the OS command.

Registering a Custom Script

In order to use a custom script, you must first register the script with the notification system. This is performed in four steps:

  1. Define your OS command or script.

  2. Deploy the script on each Management Service host.

  3. Register your OS Command or Script as a new Notification Method.

  4. Assign the notification method to an incident rule.

Step 1: Define your OS command or script.

You can specify an OS command or script that will be called by the notification system when an incident rule matches the OS Command advanced notification action. You can use incident, event, or problem context information, corrective action execution status and job execution status within the body of the script. Passing this contextual information to OS commands/scripts allows you to customize automated responses specific event conditions. For example, if an OS script opens a trouble ticket for an in-house support trouble ticket system, you will want to pass severity levels (critical, warning, and so on) to the script to open a trouble ticket with the appropriate details and escalate the problem. For more information on passing specific types of information to OS Command or Scripts, see:

Step 2: Deploy the script on each Management Service host.

You must deploy the OS Command or Script on each Management Service host machine that connects to the Management Repository. The OS Command is run as the user who started the Management Service. The OS Command or Script should be deployed on the same location on each Management Service host machine.

Important:

Both scripts and OS Commands should be specified using absolute paths. For example, /u1/bin/logSeverity.sh.

The command is run by the user who started the Management Service. If an error is encountered during the running of the OS Command, the Notification System can be instructed to retry the sending of the notification to the OS Command by returning an exit code of 100. The procedure is initially retried after one minute, then two minutes, then three minutes, eventually progressing to 30 minutes. From here, the procedure is retried every 30 minutes until the notification is a 24 hours old. The notification will be then be purged.

Example 4-4 shows the parameter in emoms.properties that controls how long the OS Command can execute without being killed by the Management Service. This prevents OS Commands from running for an inordinate length of time and blocks the delivery of other notifications. By default the command is allowed to run for 30 seconds before it is killed. The oracle.sysman.core.notification.os_cmd_timeout emoms property can be configured to change the default timeout value.

Example 4-4 Changing the oracle.sysman.core.notification.os_cmd_timeout emoms Property

emctl set property -name oracle.sysman.core.notification.os_cmd_timeout value 30
 

Step 3: Register your OS Command or Script as a new Notification Method.

Add this OS command as a notification method that can be called in incident rules. Log in as a Super Administrator. From the Setup menu, select Notifications, then select Notification Methods. From this page, you can define a new notification based on the 'OS Command' type. See "Sending Notifications Using OS Commands and Scripts".

The following information is required for each OS command notification method:

  • Name

  • Description

    Both Name and Description should be clear and intuitive so that the function of the method is clear to other administrators.

  • OS Command

You must enter the full path of the OS command or script in the OS command field (for example, /u1/bin/myscript.sh). For environments with multiple Management Services, the path must be exactly the same on each machine that has a Management Service. Command line parameters can be included after the full path (for example, /u1/bin/myscript.sh arg1 arg2).

Example 4-5 shows information required for the notification method.

Example 4-5 OS Command Notification Method

Name Trouble Ticketing
Description Notification method to log trouble ticket for a severity occurrence
OS Command /private/mozart/bin/logTicket.sh

Note:

There can be more than one OS Command configured per system.

Figure 4-5 Notification Method: Add OS Command

Add OS command

Step 4: Assign the notification method to an incident rule.

You can edit an existing rule (or create a new instance rule), then go to the Methods page. From the Setup menu, choose Incidents and then Incident Rules. The Incident Rules page provides access to all available rule sets.

For detailed reference information on passing event, incident, and problem information to an OS Command or script, see "Passing Event, Incident, Problem Information to an OS Command or Script".

4.3.1 Script Examples

The sample OS script shown in Example 4-6 appends environment variable entries to a log file. In this example, the script logs a severity occurrence to a file server. If the file server is unreachable then an exit code of 100 is returned to force the Oracle Management Service Notification System to retry the notification

Example 4-6 Sample OS Command Script

#!/bin/ksh

LOG_FILE=/net/myhost/logs/event.log
if test -f $LOG_FILE
then
echo $TARGET_NAME $MESSAGE $EVENT_REPORTED_TIME >> $LOG_FILE
else
   exit 100
fi

Example 4-7 shows an OS script that logs alert information for both incidents and events to the file 'oscmdNotify.log'. The file is saved to the /net/myhost/logs directory.

Example 4-7 Alert Logging Scripts

#!/bin/sh#
  LOG_FILE=/net/myhost/logs/oscmdNotify.log

  echo '-------------' >> $LOG_FILE

  echo 'issue_type=' $ISSUE_TYPE >> $LOG_FILE
  echo 'notif_type=' $NOTIF_TYPE >> $LOG_FILE
  echo 'message=' $MESSAGE >> $LOG_FILE
  echo 'message_url'  = $MESSAGE_URL >>$LOG_FILE
  echo 'severity=' $SEVERITY >> $LOG_FILE
  echo 'severity_code'  = $SEVERITY_CODE >>$LOG_FILE
  echo 'ruleset_name=' $RULESET_NAME >> $LOG_FILE
  echo 'rule_name=' $RULE_NAME >> $LOG_FILE
  echo 'rule_owner=' $RULE_OWNER >> $LOG_FILE
  echo 'repeat_count=' $REPEAT_COUNT >> $LOG_FILE
  echo 'categories_count'  = $CATEGORIES_COUNT >>$LOG_FILE
  echo 'category_1'  = $CATEGORY_1 >>$LOG_FILE
  echo 'category_2'  = $CATEGORY_2 >>$LOG_FILE
  echo 'category_code_1'  = $CATEGORY_CODE_1 >>$LOG_FILE
  echo 'category_code_2'  = $CATEGORY_CODE_2 >>$LOG_FILE
  echo 'category_codes_count'  = $CATEGORY_CODES_COUNT >>$LOG_FILE

# event
if [ $ISSUE_TYPE -eq 1 ]
then
  echo 'host_name=' $HOST_NAME >> $LOG_FILE
  echo 'event_type=' $EVENT_TYPE >> $LOG_FILE
  echo 'event_name=' $EVENT_NAME >> $LOG_FILE
  echo 'event_occurrence_time=' $EVENT_OCCURRENCE_TIME >> $LOG_FILE
  echo 'event_reported_time=' $EVENT_REPORTED_TIME >> $LOG_FILE
  echo 'sequence_id=' $SEQUENCE_ID >> $LOG_FILE
  echo 'event_type_attrs=' $EVENT_TYPE_ATTRS >> $LOG_FILE
  echo 'source_obj_name=' $SOURCE_OBJ_NAME >> $LOG_FILE
  echo 'source_obj_type=' $SOURCE_OBJ_TYPE >> $LOG_FILE
  echo 'source_obj_owner=' $SOURCE_OBJ_OWNER >> $LOG_FILE
  echo 'target_name'  = $TARGET_NAME >>$LOG_FILE
  echo 'target_url'  = $TARGET_URL >>$LOG_FILE
  echo 'target_owner=' $TARGET_OWNER >> $LOG_FILE
  echo 'target_type=' $TARGET_TYPE >> $LOG_FILE
  echo 'target_version=' $TARGET_VERSION >> $LOG_FILE
  echo 'lifecycle_status=' $TARGET_LIFECYCLE_STATUS >> $LOG_FILE
  echo 'assoc_incident_escalation_level'  = $ASSOC_INCIDENT_ESCALATION_LEVEL >>$LOG_FILE
  echo 'assoc_incident_id'  = $ASSOC_INCIDENT_ID >>$LOG_FILE
  echo 'assoc_incident_owner'  = $ASSOC_INCIDENT_OWNER >>$LOG_FILE
  echo 'assoc_incident_acknowledged_by_owner'  = $ASSOC_INCIDENT_ACKNOWLEDGED_BY_OWNER >>$LOG_FILE
  echo 'assoc_incident_acknowledged_details'  = $ASSOC_INCIDENT_ACKNOWLEDGED_DETAILS >>$LOG_FILE
  echo 'assoc_incident_priority'  = $ASSOC_INCIDENT_PRIORITY >>$LOG_FILE
  echo 'assoc_incident_status'  = $ASSOC_INCIDENT_STATUS >>$LOG_FILE
  echo 'ca_job_status'  = $CA_JOB_STATUS >>$LOG_FILE
  echo 'event_context_attrs'  = $EVENT_CONTEXT_ATTRS >>$LOG_FILE
  echo 'last_updated_time'  = $LAST_UPDATED_TIME >>$LOG_FILE
  echo 'sequence_id'  = $SEQUENCE_ID >>$LOG_FILE
  echo 'test_date_attr_noref'  = $TEST_DATE_ATTR_NOREF >>$LOG_FILE
  echo 'test_raw_attr_noref'  = $TEST_RAW_ATTR_NOREF >>$LOG_FILE
  echo 'test_str_attr1'  = $TEST_STR_ATTR1 >>$LOG_FILE
  echo 'test_str_attr2'  = $TEST_STR_ATTR2 >>$LOG_FILE
  echo 'test_str_attr3'  = $TEST_STR_ATTR3 >>$LOG_FILE
  echo 'test_str_attr4'  = $TEST_STR_ATTR4 >>$LOG_FILE
  echo 'test_str_attr5'  = $TEST_STR_ATTR5 >>$LOG_FILE
  echo 'test_str_attr_ref'  = $TEST_STR_ATTR_REF >>$LOG_FILE
  echo 'total_occurrence_count'  = $TOTAL_OCCURRENCE_COUNT >>$LOG_FILE
fi

# incident
if [ $ISSUE_TYPE -eq 2 ]
then
  echo 'action_msg=' $ACTION_MSG >> $LOG_FILE
  echo 'incident_id=' $INCIDENT_ID >> $LOG_FILE
  echo 'incident_creation_time=' $INCIDENT_CREATION_TIME >> $LOG_FILE
  echo 'incident_owner=' $INCIDENT_OWNER >> $LOG_FILE
echo 'incident_acknowledged_by_owner'  = $INCIDENT_ACKNOWLEDGED_BY_OWNER >>$LOG_FILE
  echo 'incident_status'  = $INCIDENT_STATUS >>$LOG_FILE
  echo 'last_modified_by=' $LAST_MODIFIED_BY >> $LOG_FILE
  echo 'last_updated_time=' $LAST_UPDATED_TIME >> $LOG_FILE
  echo 'assoc_event_count=' $ASSOC_EVENT_COUNT >> $LOG_FILE
  echo 'adr_incident_id=' $ADR_INCIDENT_ID >> $LOG_FILE
  echo 'occurrence_count=' $OCCURRENCE_COUNT >> $LOG_FILE
  echo 'escalated=' $ESCALATED >> $LOG_FILE
  echo 'escalated_level=' $ESCALATED_LEVEL >> $LOG_FILE
  echo 'priority=' $PRIORITY >> $LOG_FILE
  echo 'priority_code'  = $PRIORITY_CODE >>$LOG_FILE
  echo 'ticket_id=' $TICKET_ID >> $LOG_FILE
  echo 'ticket_status=' $TICKET_STATUS >> $LOG_FILE
  echo 'ticket_url=' $TICKET_ID_URL >> $LOG_FILE
  echo 'total_duplicate_count=' $TOTAL_DUPLICATE_COUNT >> $LOG_FILE
  echo 'source_count=' $EVENT_SOURCE_COUNT >> $LOG_FILE
  echo 'event_source_1_host_name'  = $EVENT_SOURCE_1_HOST_NAME >>$LOG_FILE
  echo 'event_source_1_target_guid'  = $EVENT_SOURCE_1_TARGET_GUID >>$LOG_FILE
  echo 'event_source_1_target_name'  = $EVENT_SOURCE_1_TARGET_NAME >>$LOG_FILE
  echo 'event_source_1_target_owner'  = $EVENT_SOURCE_1_TARGET_OWNER >>$LOG_FILE
  echo 'event_source_1_target_type'  = $EVENT_SOURCE_1_TARGET_TYPE >>$LOG_FILE
  echo 'event_source_1_target_url'  = $EVENT_SOURCE_1_TARGET_URL >>$LOG_FILE
  echo 'event_source_1_target_lifecycle_status'  = $EVENT_SOURCE_1_TARGET_LIFECYCLE_STATUS >>$LOG_FILE
  echo 'event_source_1_target_version'  = $EVENT_SOURCE_1_TARGET_VERSION >>$LOG_FILE
fi
exit 0

Example 4-8 shows a script that sends an alert to an HP OpenView console from Enterprise Manager Cloud Control. When a metric alert is triggered, the Enterprise Manager Cloud Control displays the alert. The HP OpenView script is then called, invoking opcmsg and forwarding the information to the HP OpenView management server.

Example 4-8 HP OpenView Script

/opt/OV/bin/OpC/opcmsg severity="$SEVERITY" app=OEM msg_grp=Oracle msg_text="$MESSAGE" object="$TARGET_NAME"

4.3.2 Migrating pre-12c OS Command Scripts

This section describes how to map pre-12c OS Command notification shell environment variables to 12c OS Command shell environment variables.

Note:

Policy Violations are no longer supported beginning with the Enterprise Manager 12c release.

4.3.2.1 Migrating Metric Alert Event Types

Following table is the mapping for the OS Command shell environment variables when the event_type is metric_alert.

Table 4-3 Pre-12c/12c metric_alert Environment Variable Mapping

Pre-12c Environment Variable Corresponding 12c Environment Variables

ACKNOWLEDGED

ASSOC_INCIDENT_ACKNOWLEDGED_BY_OWNER

ACKNOWLEDGED_BY

ASSOC_INCIDENT_OWNER

CYCLE_GUID

CYCLE_GUID

HOST

HOST_NAME

KEY_VALUE

Note: See detail description below.

KEY_VALUE_NAME

Note: See detail description below

MESSAGE

MESSAGE

METRIC

METRIC_COLUMN

NOTIF_TYPE

NOTIF_TYPE; use the map in section 2.3.5

REPEAT_COUNT

REPEAT_COUNT

RULE_NAME

RULESET_NAME

RULE_OWNER

RULE_OWNER

SEVERITY

SEVERITY

TARGET_NAME

TARGET_NAME

TARGET_TYPE

TARGET_TYPE

TIMESTAMP

EVENT_REPORTED_TIME

METRIC_VALUE

VALUE

VIOLATION_CONTEXT

EVENT_CONTEXT_ATTRS

VIOLATION_GUID

SEVERITY_GUID

POLICY_RULE

No mapping, Obsolete as of Enterprise Manager 12c release.


To obtain KEY_VALUE_NAME and KEY_VALUE, perform the following steps.

  • If $NUM_KEYS variable is null, then $KEY_VALUE_NAME and $KEY_VALUE are null.

  • If $NUM_KEYS equals 1

    KEY_VALUE_NAME=$KEY_COLUMN_1

    KEY_COLUMN_1_VALUE

  • If $NUM_KEYS is greater than 1

    KEY_VALUE_NAME="$KEY_COLUMN_1;$KEY_COLUMN_2;..;KEY_COLUMN_x"

    KEY_VALUE="$KEY_COLUMN_1_VALUE;$KEY_COLUMN_2_VALUE;..;KEY_COLUMN_x_VALUE "

    Where x is the value of $NUM_KEYS and ";" is the separator.

4.3.2.2 Migrating Target Availability Event Types

Following table is the mapping for the OS Command shell environment variables when the event_type is 'target_availability'.

Table 4-4 pre-12c/12c target_availability Environment Variable Mappings

Pre-12c Environment Variable Corresponding 12c Environment Variables

TARGET_NAME

TARGET_NAME

TARGET_TYPE

TARGET_TYPE

METRIC

Status

CYCLE_GUID

CYCLE_GUID

VIOLATION_CONTEXT

EVENT_CONTEXT_ATTRS

SEVERITY

TARGET_STATUS

HOST

HOST_NAME

MESSAGE

MESSAGE

NOTIF_TYPE

NOTIF_TYPE; use the map in section 2.3.5

TIMESTAMP

EVENT_REPORTED_TIME

RULE_NAME

RULESET_NAME

RULE_OWNER

RULE_OWNER

REPEAT_COUNT

REPEAT_COUNT

KEY_VALUE

""

KEY_VALUE_NAME

""


4.3.2.3 Migrating Job Status Change Event Types

Following table is the mapping for the OS Command shell environment variables when the event_type is 'job_status_change'.

Table 4-5 pre-12c/12c job_status_change Environment Variable Mappings

Pre-12c Environment Variable Corresponding 12c Environment Variables

JOB_NAME

SOURCE_OBJ_NAME

JOB_OWNER

SOURCE_OBJ_OWNER

JOB_TYPE

SOURCE_OBJ_SUB_TYPE

JOB_STATUS

EXECUTION_STATUS

NUM_TARGETS

1 if $ TARGET_NAME is not null, 0 otherwise

TARGET_NAME1

TARGET_NAME

TARGET_TYPE1

TARGET_TYPE

TIMESTAMP

EVENT_REPORTED_TIME

RULE_NAME

RULESET_NAME

RULE_OWNER

RULE_OWNER


4.3.2.4 Migrating Corrective Action-Related OS Scripts

Refer to section "Migrating Metric Alert Event Types" for mapping the following environment variables while receiving notifications for corrective actions.

KEY_VALUE

KEY_VALUE_NAME

METRIC

METRIC_VALUE

RULE_NAME

RULE_OWNER

SEVERITY

TIMESTAMP

VIOLATION_CONTEXT

Use the map below for mapping other environment variables.

Table 4-6 pre-12c/12c Corrective Action Environment Variable Mappings

Pre-12c Environment Variable Corresponding 12c Environment Variables

NUM_TARGETS

1

TARGET_NAME1

TARGET_NAME

TARGET_TYPE1

TARGET_TYPE

JOB_NAME

CA_JOB_NAME

JOB_OWNER

CA_JOB_OWNER

JOB_STATUS

CA_JOB_STATUS

JOB_TYPE

CA_JOB_TYPE


4.3.2.5 Notification Type Mapping

Table 4-7 pre-12c/12c notif_type Mappings

notif_type (12c) notif_type (Pre-12c)

NOTIF_NORMAL

1

NOTIF_REPEAT

4

NOTIF_DURATION

9

NOTIF_RETRY

2


4.4 Sending Notifications Using PL/SQL Procedures

A user-defined PL/SQL procedure can receive notifications for matching events, incidents and problems.

Note:

When upgrading from pre-12c to 12c versions of Enterprise Manager, existing pre-12c PL/SQL advanced notification methods will continue to work without modification. You should, however, update the procedures to use new signatures.

New PL/SQL advanced notification methods created with Enterprise Manager 12c must use the new signatures documented in the following sections.

Complete the following four steps to define a notification method based on a PL/SQL procedure.

4.4.1 Defining a PL/SQL-based Notification Method

Creating a PL/SQL-based notification method consists of four steps:

  1. Define the PL/SQL procedure.

  2. Create the PL/SQL procedure on the Management Repository.

  3. Register your PL/SQL procedure as a new notification method.

  4. Assign the notification method to an incident rule.

Step 1: Define the PL/SQL Procedure

The procedure must have one of the following signatures depending on the type of notification that will be received.

For Events:

PROCEDURE event_proc(event_msg IN gc$notif_event_msg)

For Incidents:

PROCEDURE incident_proc(incident_msg IN gc$notif_incident_msg)

For Problems:

PROCEDURE problem_proc(problem_msg IN gc$notif_problem_msg)

Note:

The notification method based on a PL/SQL procedure must be configured by an administrator with Super Administrator privileges before a user can select it while creating/editing a incident rule.

For more information on passing specific types of information to scripts or PL/SQL procedures, see the following sections:

"Passing Information to a PL/SQL Procedure"

"Passing Corrective Action Status Change Information"

"Passing Job Execution Status Information"

Step 2: Create the PL/SQL procedure on the Management Repository.

Create the PL/SQL procedure on the repository database using one of the following procedure specifications:

PROCEDURE event_proc(event_msg IN gc$notif_event_msg)

PROCEDURE incident_proc(incident_msg IN gc$notif_incident_msg)

PROCEDURE problem_proc(problem_msg IN gc$notif_problem_msg)

The PL/SQL procedure must be created on the repository database using the database account of the repository owner (such as SYSMAN)

If an error is encountered during the running of the procedure, the Notification System can be instructed to retry the sending of the notification to the procedure by raising a user-defined exception that uses the error code -20000. The procedure initially retried after one minute, then two minutes, then three minutes and so on, until the notification is a day old, at which point it will be purged.

Step 3: Register your PL/SQL procedure as a new notification method.

Log in as a Super Administrator. From the Setup menu, choose Notifications and then Notification Methods to access the Notification Methods page. From this page, you can define a new notification based on 'PL/SQL Procedure'. See Section 4.4, "Sending Notifications Using PL/SQL Procedures".

Make sure to use a fully qualified name that includes the schema owner, package name and procedure name. The procedure will be executed by the repository owner and so the repository owner must have execute permission on the procedure.

Create a notification method based on your PL/SQL procedure. The following information is required when defining the method:

  • Name

  • Description

  • PL/SQL Procedure

You must enter a fully qualified procedure name (for example, OWNER.PKGNAME.PROCNAME) and ensure that the owner of the Management Repository has execute privilege on the procedure.

An example of the required information is shown in Example 4-9.

Example 4-9 PL/SQL Procedure Required Information

Name Open trouble ticket
Description Notification method to open a trouble ticket in the event
PLSQL Procedure ticket_sys.ticket_ops.open_ticket

Figure 4-6 illustrates how to add a PL/SQL-based notification method from the Enterprise Manager UI.

Figure 4-6 Adding a PL/SQL Procedure

adding a pl/sql procedure

Step 4: Assign the notification method to an incident rule.

You can edit an existing rule (or create a new incident rule). From the Setup menu, select Incidents and then select Incident Rules. The Incident Rules page displays. From here, you can add an action to a rule specifying the new PL/SQL procedure found under Advanced Notification Method.

There can be more than one PL/SQL-based method configured for your Enterprise Manager environment.

See "Passing Information to a PL/SQL Procedure" for more information about how incident, event, and problem information is passed to the PLSQL procedure.

Example 4-10 PL/SQL Script

-- Assume log_table is created by following DDL
-- CREATE TABLE log_table (message VARCHAR2(4000)) ;
-- Define PL/SQL notification method for Events
CREATE OR REPLACE PROCEDURE log_table_notif_proc(s IN GC$NOTIF_EVENT_MSG)
IS
  l_categories gc$category_string_array;
  l_category_codes gc$category_string_array;
  l_attrs gc$notif_event_attr_array;
  l_ca_obj gc$notif_corrective_action_job;
BEGIN
  INSERT INTO log_table VALUES ('notification_type: ' || s.msg_info.notification_type);
  INSERT INTO log_table VALUES ('repeat_count: ' || s.msg_info.repeat_count);
  INSERT INTO log_table VALUES ('ruleset_name: ' || s.msg_info.ruleset_name);
  INSERT INTO log_table VALUES ('rule_name: ' || s.msg_info.rule_name);
  INSERT INTO log_table VALUES ('rule_owner: ' || s.msg_info.rule_owner);
  INSERT INTO log_table VALUES ('message: ' || s.msg_info.message);
  INSERT INTO log_table VALUES ('message_url: ' || s.msg_info.message_url);
  INSERT INTO log_table VALUES ('event_instance_guid: ' || s.event_payload.event_instance_guid);
  INSERT INTO log_table VALUES ('event_type: ' || s.event_payload.event_type);
  INSERT INTO log_table VALUES ('event_name: ' || s.event_payload.event_name);
  INSERT INTO log_table VALUES ('event_msg: ' || s.event_payload.event_msg);
  INSERT INTO log_table VALUES ('source_obj_type: ' || s.event_payload.source.source_type);
  INSERT INTO log_table VALUES ('source_obj_name: ' || s.event_payload.source.source_name);
  INSERT INTO log_table VALUES ('source_obj_url: ' || s.event_payload.source.source_url);
  INSERT INTO log_table VALUES ('target_name: ' || s.event_payload.target.target_name);
  INSERT INTO log_table VALUES ('target_url: ' || s.event_payload.target.target_url);
  INSERT INTO log_table VALUES ('severity: ' || s.event_payload.severity);  INSERT INTO log_table VALUES ('severity_code: ' || s.event_payload.severity_code);
  INSERT INTO log_table VALUES ('event_reported_date: ' || to_char(s.event_payload.reported_date, 'D MON DD HH24:MI:SS'));

  l_categories := s.event_payload.categories;
  IF l_categories IS NOT NULL
  THEN
    FOR c IN 1..l_categories.COUNT
    LOOP
      INSERT INTO log_table VALUES ('category ' || c || ' - ' || l_categories(c));
    END LOOP;
  END IF;

  l_category_codes := s.event_payload.category_codes;
  IF l_categories IS NOT NULL
  THEN
    FOR c IN 1..l_category_codes.COUNT
    LOOP
      INSERT INTO log_table VALUES ('category_code ' || c || ' - ' || l_category_codes(c));
    END LOOP;
  END IF;

  l_attrs := s.event_payload.event_attrs;
  IF l_attrs IS NOT NULL
  THEN
    FOR c IN 1..l_attrs.COUNT
    LOOP
        INSERT INTO log_table VALUES ('EV.ATTR name=' || l_attrs(c).name || ' value=' || l_attrs(c).value || ' nls_value=' || l_attrs(c).nls_value);    END LOOP;
  END IF;

COMMIT ;
END ;
/

Example 4-11 PL/SQL Script to Log Events to a Table

CREATE TABLE event_log (
  notification_type      VARCHAR2(32),
  repeat_count           NUMBER,
  ruleset_name           VARCHAR2(256),
  rule_owner             VARCHAR2(256),
  rule_name              VARCHAR2(256),
  message                VARCHAR2(4000),
  message_url            VARCHAR2(4000),
  event_instance_guid    RAW(16),
  event_type             VARCHAR2(20),
  event_name             VARCHAR2(512),
  event_msg              VARCHAR2(4000),
  categories             VARCHAR2(4000),
  source_obj_type        VARCHAR2(120),
  source_obj_name        VARCHAR2(256),
  source_obj_url         VARCHAR2(4000),
  severity               VARCHAR2(128),
  severity_code          VARCHAR2(32),
  target_name            VARCHAR2(256),
  target_type            VARCHAR2(128),
  target_url             VARCHAR2(4000),
  host_name              VARCHAR2(256),
  timezone               VARCHAR2(64),
  occured                DATE,
  ca_guid                RAW(16),
  ca_name                VARCHAR2(128),
  ca_owner               VARCHAR2(256),
  ca_type                VARCHAR2(256),
  ca_status              VARCHAR2(64),
  ca_status_code         NUMBER,
  ca_job_step_output     VARCHAR2(4000),
  ca_execution_guid      RAW(16),
  ca_stage_change_guid   RAW(16)
)
;

CREATE OR REPLACE PROCEDURE log_event(s IN GC$NOTIF_EVENT_MSG)
IS
   l_categories gc$category_string_array;
   l_ca_obj gc$notif_corrective_action_job;
   l_categories_new VARCHAR2(1000);
BEGIN
    -- save event categories
   l_categories := s.event_payload.categories;
        IF l_categories IS NOT NULL
   THEN
     FOR c IN 1..l_categories.COUNT
     LOOP
       l_categories_new := (l_categories_new|| c || ' - ' || l_categories(c)||',');
     END LOOP;
   END IF;

   -- save event message
   IF s.msg_info.notification_type = 'NOTIF_CA' AND s.event_payload.corrective_action IS NOT NULL
   THEN
     l_ca_obj := s.event_payload.corrective_action;
      INSERT INTO event_log (notification_type, repeat_count, ruleset_name, rule_name, rule_owner, message, message_url, event_instance_guid, event_type, event_name, event_msg, categories, source_obj_type, source_obj_name, source_obj_url, severity, severity_code, target_name, target_type, target_url, host_name, timezone, occured, ca_guid, ca_name, ca_owner, ca_type, ca_status, ca_status_code, ca_job_step_output, ca_execution_guid, ca_stage_change_guid)
      VALUES (s.msg_info.notification_type, s.msg_info.repeat_count, s.msg_info.ruleset_name, s.msg_info.rule_name,s.msg_info.rule_owner, s.msg_info.message, s.msg_info.message_url, s.event_payload.event_instance_guid, s.event_payload.event_type, s.event_payload.event_name, s.event_payload.event_msg, l_categories_new, s.event_payload.source.source_type, s.event_payload.source.source_name, s.event_payload.source.source_url, s.event_payload.severity, s.event_payload.severity_code, s.event_payload.target.target_name, s.event_payload.target.target_type, s.event_payload.target.target_url, s.event_payload.target.host_name, s.event_payload.target.target_timezone, s.event_payload.occurrence_date, l_ca_obj.JOB_GUID, l_ca_obj.JOB_NAME, l_ca_obj.JOB_OWNER, l_ca_obj.JOB_TYPE, l_ca_obj.JOB_STATUS, l_ca_obj.JOB_STATUS_CODE, l_ca_obj.JOB_STEP_OUTPUT, l_ca_obj.JOB_EXECUTION_GUID, l_ca_obj.JOB_STATE_CHANGE_GUID);   ELSE
      INSERT INTO event_log (notification_type, repeat_count, ruleset_name, rule_name, rule_owner, message, message_url, event_instance_guid, event_type, event_name, event_msg, categories, source_obj_type, source_obj_name, source_obj_url, severity, severity_code, target_name, target_type, target_url, host_name, timezone, occured, ca_guid, ca_name, ca_owner, ca_type, ca_status, ca_status_code, ca_job_step_output, ca_execution_guid, ca_stage_change_guid)
      VALUES (s.msg_info.notification_type, s.msg_info.repeat_count, s.msg_info.ruleset_name, s.msg_info.rule_name, s.msg_info.rule_owner, s.msg_info.message, s.msg_info.message_url, s.event_payload.event_instance_guid, s.event_payload.event_type, s.event_payload.event_name, s.event_payload.event_msg, l_categories_new, s.event_payload.source.source_type, s.event_payload.source.source_name, s.event_payload.source.source_url, s.event_payload.severity, s.event_payload.severity_code, s.event_payload.target.target_name, s.event_payload.target.target_type, s.event_payload.target.target_url, s.event_payload.target.host_name, s.event_payload.target.target_timezone, s.event_payload.occurrence_date, null,null,null,null,null,null,null,null,null);
   END IF;
   COMMIT;
END log_event;
/

Example 4-12 PL/SQL Script to Log Incidents to a Table

CREATE TABLE incident_log ( 
  notification_type      VARCHAR2(32), 
  repeat_count           NUMBER, 
  ruleset_name           VARCHAR2(256), 
  rule_owner             VARCHAR2(256), 
  rule_name              VARCHAR2(256), 
  message                VARCHAR2(4000), 
  message_url            VARCHAR2(4000), 
  incident_id            VARCHAR2(128), 
  ticket_url             VARCHAR2(4000), 
  assoc_event_cnt        NUMBER, 
  severity               VARCHAR2(128), 
  severity_code          VARCHAR2(32), 
  priority               VARCHAR2(128), 
  priority_code          VARCHAR2(32), 
  status                 VARCHAR2(32), 
  categories             VARCHAR2(1000), 
  target_name            VARCHAR2(256), 
  target_type            VARCHAR2(128), 
  host_name              VARCHAR2(256), 
  timezone               VARCHAR2(64), 
  occured                DATE 
) 
; 
   CREATE OR REPLACE PROCEDURE log_incident(s IN GC$NOTIF_INCIDENT_MSG) 
IS 
   l_src_info_array GC$NOTIF_SOURCE_INFO_ARRAY; 
   l_src_info  GC$NOTIF_SOURCE_INFO; 
   l_categories gc$category_string_array; 
   l_target_obj GC$NOTIF_TARGET; 
   l_target_name VARCHAR2(256); 
   l_target_type VARCHAR2(256); 
   l_target_timezone VARCHAR2(256); 
   l_hostname VARCHAR2(256); 
   l_categories_new VARCHAR2(1000); 
BEGIN 
     -- Save Incident categories 
  IF l_categories IS NOT NULL 
   THEN 
     FOR c IN 1..l_categories.COUNT 
     LOOP 
       l_categories_new := (l_categories_new|| c || ' - ' || l_categories(c)||','); 
     END LOOP; 
   END IF; 

   -- GET target info 
   l_src_info_array := s.incident_payload.incident_attrs.source_info_arr; 
   IF l_src_info_array IS NOT NULL 
   THEN 
     FOR I IN 1..l_src_info_array.COUNT 
     LOOP 
       IF l_src_info_array(I).TARGET IS NOT NULL 
       THEN 
         l_target_name := l_src_info_array(I).TARGET.TARGET_NAME; 
         l_target_type := l_src_info_array(I).TARGET.TARGET_TYPE; 
         l_target_timezone := l_src_info_array(I).TARGET.TARGET_TIMEZONE; 
         l_hostname := l_src_info_array(I).TARGET.HOST_NAME; 
      END IF; 
     END LOOP; 
   END IF; 

   -- save Incident notification message    INSERT INTO incident_log(notification_type, repeat_count, ruleset_name, rule_owner, rule_name, message, message_url, incident_id, ticket_url, assoc_event_cnt, severity, severity_code, priority, priority_code, status, categories, target_name, target_type, host_name, timezone, occured) 
   VALUES (s.msg_info.notification_type, s.msg_info.repeat_count, s.msg_info.ruleset_name, s.msg_info.rule_owner, s.msg_info.rule_name, s.msg_info.message, s.msg_info.message_url, s.incident_payload.incident_attrs.id, s.incident_payload.ticket_url, s.incident_payload.assoc_event_count, s.incident_payload.incident_attrs.severity, s.incident_payload.incident_attrs.severity_code, s.incident_payload.incident_attrs.priority, s.incident_payload.incident_attrs.priority_code, s.incident_payload.incident_attrs.STATUS, l_categories_new, l_target_name, l_target_type, l_hostname,l_target_timezone, s.incident_payload.incident_attrs.creation_date); 
   COMMIT; 
END log_incident; 
/

Example 4-13 PL/SQL Script to Log Problems to a Table

CREATE TABLE problem_log ( 
  notification_type      VARCHAR2(32), 
  repeat_count           NUMBER, 
  ruleset_name           VARCHAR2(256), 
  rule_owner             VARCHAR2(256), 
  rule_name              VARCHAR2(256), 
  message                VARCHAR2(4000), 
  message_url            VARCHAR2(4000), 
  problem_key            VARCHAR2(850), 
  assoc_incident_cnt     NUMBER, 
  problem_id             NUMBER, 
  owner                  VARCHAR2(256), 
  severity               VARCHAR2(128), 
  severity_code          VARCHAR2(32), 
  priority               VARCHAR2(128), 
  priority_code          VARCHAR2(32), 
  status                 VARCHAR2(32), 
  categories             VARCHAR2(1000), 
  target_name            VARCHAR2(256), 
  target_type            VARCHAR2(128), 
  host_name              VARCHAR2(256),   timezone               VARCHAR2(64), 
  occured                DATE 
) 
; 
     CREATE OR REPLACE PROCEDURE log_problem(s IN GC$NOTIF_PROBLEM_MSG) 
IS 
   l_src_info_array GC$NOTIF_SOURCE_INFO_ARRAY; 
   l_src_info  GC$NOTIF_SOURCE_INFO; 
   l_categories gc$category_string_array; 
   l_target_obj GC$NOTIF_TARGET; 
   l_target_name VARCHAR2(256); 
   l_target_type VARCHAR2(256); 
   l_target_timezone VARCHAR2(256); 
   l_hostname VARCHAR2(256); 
   l_categories_new VARCHAR2(1000); 
BEGIN 
     -- Save Problem categories 
  l_categories := s.problem_payload.problem_attrs.categories; 
  IF l_categories IS NOT NULL 
   THEN 
     FOR c IN 1..l_categories.COUNT 
     LOOP 
       l_categories_new := (l_categories_new|| c || ' - ' || l_categories(c)||','); 
     END LOOP; 
   END IF; 

   -- GET target info 
   l_src_info_array := s.problem_payload.problem_attrs.source_info_arr; 
   IF l_src_info_array IS NOT NULL 
   THEN 
     FOR I IN 1..l_src_info_array.COUNT 
     LOOP 
       IF l_src_info_array(I).TARGET IS NOT NULL 
       THEN 
         l_target_name := l_src_info_array(I).TARGET.TARGET_NAME; 
         l_target_type := l_src_info_array(I).TARGET.TARGET_TYPE; 
         l_target_timezone := l_src_info_array(I).TARGET.TARGET_TIMEZONE; 
         l_hostname := l_src_info_array(I).TARGET.HOST_NAME; 
      END IF; 
     END LOOP; 
   END IF; 

  -- save Problem notification message 
   INSERT INTO problem_log(notification_type, repeat_count, ruleset_name, rule_owner, rule_name, message, message_url, problem_key, assoc_incident_cnt, problem_id, owner,  severity, severity_code, priority, priority_code, status, categories, target_name, target_type, host_name, timezone, occured) 
   VALUES (s.msg_info.notification_type, s.msg_info.repeat_count, s.msg_info.ruleset_name, s.msg_info.rule_owner, s.msg_info.rule_name, s.msg_info.message, s.msg_info.message_url, s.problem_payload.problem_key,            s.problem_payload.ASSOC_INCIDENT_COUNT, s.problem_payload.problem_attrs.id,            s.problem_payload.problem_attrs.owner, s.problem_payload.problem_attrs.severity,            s.problem_payload.problem_attrs.severity_code, s.problem_payload.problem_attrs.PRIORITY, s.problem_payload.problem_attrs.PRIORITY_CODE, s.problem_payload.problem_attrs.status, l_categories_new, l_target_name, l_target_type, l_hostname,l_target_timezone, s.problem_payload.problem_attrs.CREATION_DATE); 
   COMMIT; 
END log_problem; 
/

4.4.2 Migrating Pre-12c PL/SQL Advanced Notification Methods

Pre-12c notifications map to event notifications in Enterprise Manager 12c. The event types metric_alert, target_availability and job_status_alert correspond to the pre-12c notification functionality.

Note:

Policy Violations are no longer available beginning with Enterprise Manager 12c.

This section describes the mapping between Enterprise Manager 12c PL/SQL notification payload to the pre-12c PL/SQL notification payload. You can use this information for updating the existing pre-12c PL/SQL user callback procedures to use the 12c PL/SQL notification payload.Please note that Policy Violations are no longer supported in the 12c release.

4.4.2.1 Mapping for MGMT_NOTIFY_SEVERITY

When event type is metric_alert

Use the following map when gc$notif_event_payload .event_type='metric_alert'.

Table 4-8 Metric Alert Mapping

MGMT_NOTIFY_SEVERITY 12c Notification Payload

TARGET_NAME

gc$notif_target.target_name

TARGET_TYPE

gc$notif_target.target_type

TIMEZONE

gc$notif_target.target_timezone

HOST_NAME

gc$notif_target.host_name

MERTIC_NAME

gc$notif_event_attr.value where its name=' metric_group' in gc$notif_event_attr_array.

METRIC_DESCRIPTION

gc$notif_event_attr.value where its name=' metric_description' in gc$notif_event_attr_array.

METRIC_COLUMN

gc$notif_event_attr.value where its name=' metric_column' in gc$notif_event_attr_array.

METRIC_VALUE

gc$notif_event_attr.value where its name=' value' in gc$notif_event_attr_array.

KEY_VALUE

It is applied for multiple keys based metric when value of gc$notif_event_attr.name='num_keys' is not null and is greater than 0 in gc$notif_event_attr_array.

See detail descriptions below.

KEY_VALUE_NAME

It is applied for multiple keys based metric when value of gc$notif_event_attr.name='num_keys' is not null and is greater than 0 in gc$notif_event_attr_array.

See detail descriptions below.

KEY_VALUE_GUID

gc$notif_event_attr.value where its name='key_ value' in gc$notif_event_attr_array.

CTXT_LIST

gc$notif_event_context_array

COLLECTION_TIMESTAMP

gc$notif_event_payload. reported_date

SEVERITY_CODE

Derive from gc$notif_event_payload.severity_code, see Table 4-9, "Severity Code Mapping".

MESSAGE

gc$notif_msg_info.message

SEVERITY_GUID

gc$notif_event_attr.value where its name=' severity_guid' in gc$notif_event_attr_array.

METRIC_GUID

gc$notif_event_attr.value where its name=' metric_guid' in gc$notif_event_attr_array.

TARGET_GUID

gc$notif_target.target_guid

RULE_OWNER

gc$notif_msg_info.rule_owner

RULE_NAME

gc$notif_msg_info.ruleset_name


The following example illustrates how to obtain similar pre-12c KEY_VALUE and KEY_VALUE_NAME from an Enterprise Manager 12c notification payload.

Example 4-14 Extracting KEY_VALUE and KEY_VALUE_NAME

-- Get the pre-12c KEY_VALUE and KEY_VALUE_NAME from an Enterprise Manager 12c
-- notification payload
-- parameters
--   IN Parameters:
--      event_msg : The event notification payload
--   OUT Parameters
--      key_value_name_out : the KEY_VALUE_NAME backward compitable to pre-12c 
--                           notification payload
--      key_value_out      : the KEY_VALUE backward compitable to pre-12c 
--                           notification payload
--
CREATE OR REPLACE PROCEDURE get_pre_12c_key_value(
          event_msg IN GC$NOTIF_EVENT_MSG,
          key_value_name_out OUT VARCHAR2,
          key_value_out OUT VARCHAR2)
IS

  l_key_columns MGMT_SHORT_STRING_ARRAY := MGMT_SHORT_STRING_ARRAY();
  l_key_column_values MGMT_MEDIUM_STRING_ARRAY := MGMT_MEDIUM_STRING_ARRAY();
  l_key_value VARCHAR2(1790) := NULL;
  l_num_keys NUMBER := 0;
  l_attrs gc$notif_event_attr_array;
  l_key_value_name VARCHAR2(512);
BEGIN
  l_attrs := event_msg.event_payload.event_attrs;
  key_value_name_out := NULL;
  key_value_out := NULL;

  IF l_attrs IS NOT NULL AND
     l_attrs.COUNT > 0
  THEN
    l_key_columns.extend(7);
    l_key_column_values.extend(7);
    FOR c IN 1..l_attrs.COUNT
    LOOP
      CASE l_attrs(c).name
        WHEN 'num_keys' THEN
          BEGIN
            l_num_keys := to_number(l_attrs(c).value);
          EXCEPTION
          WHEN OTHERS THEN
            -- should never happen, but guard against it l_num_keys := 0;
          END;
        WHEN 'key_value' THEN
          l_key_value := substr(l_attrs(c).nls_value,1,1290);
        WHEN 'key_column_1' THEN
          l_key_columns(1) := substr(l_attrs(c).nls_value,1,64);
        WHEN 'key_column_2' THEN
          l_key_columns(2) := substr(l_attrs(c).nls_value,1,64);
        WHEN 'key_column_3' THEN
          l_key_columns(3) := substr(l_attrs(c).nls_value,1,64);
        WHEN 'key_column_4' THEN
          l_key_columns(4) := substr(l_attrs(c).nls_value,1,64);
        WHEN 'key_column_5' THEN
          l_key_columns(5) := substr(l_attrs(c).nls_value,1,64);
        WHEN 'key_column_6' THEN
          l_key_columns(6) := substr(l_attrs(c).nls_value,1,64);
        WHEN 'key_column_7' THEN
          l_key_columns(7) := substr(l_attrs(c).nls_value,1,64);
        WHEN 'key_column_1_value' THEN
          l_key_column_values(1) := substr(l_attrs(c).nls_value,1,256);
        WHEN 'key_column_2_value' THEN
          l_key_column_values(2) := substr(l_attrs(c).nls_value,1,256);
        WHEN 'key_column_3_value' THEN
          l_key_column_values(3) := substr(l_attrs(c).nls_value,1,256);
        WHEN 'key_column_4_value' THEN
          l_key_column_values(4) := substr(l_attrs(c).nls_value,1,256);
        WHEN 'key_column_5_value' THEN
          l_key_column_values(5) := substr(l_attrs(c).nls_value,1,256);
        WHEN 'key_column_6_value' THEN
          l_key_column_values(6) := substr(l_attrs(c).nls_value,1,256);
        WHEN 'key_column_7_value' THEN
          l_key_column_values(7) := substr(l_attrs(c).nls_value,1,256);
        ELSE
          NULL;
      END CASE;
    END LOOP;

    -- get key_value and key_value_name when l_num_keys > 0 

    IF l_num_keys > 0
    THEN
      -- get key value name
      IF l_key_columns IS NULL OR l_key_columns.COUNT = 0
      THEN
        key_value_name_out := NULL;
      ELSE
        l_key_value_name := NULL;
        FOR i in l_key_columns.FIRST..l_num_keys
        LOOP
          IF i > 1
          THEN
            l_key_value_name := l_key_value_name || ';';
          END IF;
          l_key_value_name := l_key_value_name || l_key_columns(i);
        END LOOP;
        key_value_name_out := l_key_value_name;
      END IF;
            -- get key_value
      IF l_num_keys = 1
      THEN
        key_value_out := l_key_value;
      ELSE
        l_key_value := NULL;
        IF l_key_column_values IS NULL OR l_key_column_values.COUNT = 0
        THEN
          key_value_out := NULL;
        ELSE 
          FOR i in l_key_column_values.FIRST..l_num_keys
          LOOP
            IF i > 1
            THEN
              l_key_value := l_key_value || ';';
            END IF;
            l_key_value := l_key_value || l_key_column_values(i);
                        END LOOP;
          --  max length for key value in pre-12c = 1290
          key_value_out := substr(l_key_value,1,1290);
        END IF;
      END IF;
    END IF; -- l_num_keys > 0
  END IF;  -- l_attrs IS NOT NULL
END get_pre_12c_key_value;
/

When the event type is metric_alert:

Use the following severity code mapping from 12c to pre-12c when the event type is metric_alert.

Table 4-9 Severity Code Mapping

12c Severity Code Pre-12c Severity Code

GC_EVENT_RECEIVER.FATAL 32

MGMT_GLOBAL.G_SEVERITY_CRITICAL 25

GC_EVENT_RECEIVER.CRITICAL 16

MGMT_GLOBAL.G_SEVERITY_CRITICAL 25

GC_EVENT_RECEIVER.WARNING 8

MGMT_GLOBAL.G_SEVERITY_WARNING 20

GC_EVENT_RECEIVER.CLEAR 0

MGMT_GLOBAL.G_SEVERITY_CLEAR 15


When event type is target_availability:

Use the following map when gc$notif_event_payload .event_type='target_availability'.

Table 4-10 Target Availability Mapping

MGMT_NOTIFY_SEVERITY 12c Notification Payload

TARGET_NAME

gc$notif_target.target_name

TARGET_TYPE

gc$notif_target.target_type

TIMEZONE

gc$notif_target.target_timezone

HOST_NAME

gc$notif_target.host_name

MERTIC_NAME

Use fixed value "Response".

METRIC_DESCRIPTION

NULL

METRIC_COLUMN

Use fixed value "Status".

METRIC_VALUE

gc$notif_event_attr.value where its name='target_status' in gc$notif_event_attr_array.

KEY_VALUE

NULL

KEY_VALUE_NAME

NULL

KEY_VALUE_GUID

NULL

CTXT_LIST

gc$notif_event_context_array

COLLECTION_TIMESTAMP

gc$notif_event_payload. reported_date

SEVERITY_CODE

gc$notif_event_attr.value where its name=' avail_severity' in gc$notif_event_attr_array.

MESSAGE

gc$notif_msg_info.message

SEVERITY_GUID

gc$notif_event_attr.value where its name=' severity_guid' in gc$notif_event_attr_array.

METRIC_GUID

gc$notif_event_attr.value where its name=' metric_guid_id' in gc$notif_event_attr_array.

TARGET_GUID

gc$notif_target.target_guid

RULE_OWNER

gc$notif_msg_info.rule_owner

RULE_NAME

gc$notif_msg_info.ruleset_name


4.4.2.2 Mapping for MGMT_NOTIFY_JOB

Use the following map when gc$notif_event_payload .event_type=job_status_change'.

Table 4-11 Job Status Change Mapping

MGMT_NOTIFY_JOB 12c Notification Payload

JOB_NAME

gc$notif_source.source_name

JOB_OWNER

gc$notif_source.source_owner

JOB_TYPE

gc$notif_source.source_sub_type

JOB_STATUS

gc$notif_event_attr.value where its name=' execution_status_code' in gc$notif_event_attr_array.

STATE_CHANGE_GUID

gc$notif_event_attr.value where its name=' state_change_guid' in gc$notif_event_attr_array.

JOB_GUID

gc$notif_source.source_guid

EXECUTION_ID

gc$notif_event_attr.value where its name=' execution_id' in gc$notif_event_attr_array.

TARGETS

gc$notif_target.target_name, gc$notif_target.target_type

RULE_OWNER

gc$notif_msg_info.rule_owner

RULE_NAME

gc$notif_msg_info.ruleset_name

OCCURRED_DATE

gc$notif_event_payload. reported_date


4.4.2.3 Mapping for MGMT_NOTIFY_CORRECTIVE_ACTION

Note that corrective action related payload is populated when gc$notif_msg_info.notification_type is set to NOTIF_CA.

For mapping the following attributes, use the mapping information provided for MGMT_NOTIFY_SEVERITY object Table 4-8, "Metric Alert Mapping"

MERTIC_NAME

METRIC_COLUMN

METRIC_VALUE

KEY_VALUE

KEY_VALUE_NAME

KEY_VALUE_GUID

CTXT_LIST

RULE_OWNER

RULE_NAME

OCCURRED_DATE

For mapping the job related attributes in MGMT_NOTIFY_CORRECTIVE_ACTION object, use the following map.

Table 4-12 Corrective Action Mapping

MGMT_NOTIFY_CORRECTIVE_ACTION 12c Notification Payload

JOB_NAME

gc$ notif_corrective_action_job.job_name

JOB_OWNER

gc$ notif_corrective_action_job.job_owner

JOB_TYPE

gc$ notif_corrective_action_job.job_type

JOB_STATUS

gc$ notif_corrective_action_job.status_code

STATE_CHANGE_GUID

gc$ notif_corrective_action_job. job_state_change_guid

JOB_GUID

gc$ notif_corrective_action_job. job _guid

EXECUTION_ID

gc$ notif_corrective_action_job. job_execution_guid

OCCURRED_DATE

gc$ notif_corrective_action_job.occurred_date

TARGETS

There can be at most one target. Use the values from gc$notif_target.target_name, gc$notif_target.target_type for the associated target.


4.5 Sending SNMP Traps to Third Party Systems

Enterprise Manager supports integration with third-party management tools through the Simple Network Management Protocol (SNMP). For example, you can use SNMP to notify a third-party application that a selected metric has exceeded its threshold.

Important:

In order for a third-party system to interpret traps sent by the OMS, the omstrap.v1 file must first be loaded into the third-party SNMP console. For more information about this file and its location, see "MIB Definition".

The Enterprise Manager 12c version of the MIB file incorporates the 10g and 11g MIB content, thus ensuring backward compatibility with earlier Enterprise Manager releases.

Enterprise Manager supports both SNMP Version 1 and Version 3 traps. The traps are described by the MIB definition shown in Appendix B, "Enterprise Manager MIB Definition." See "Management Information Base (MIB)" for an explanation of how the MIB works. If you are using Enterprise Manager 12c, see Appendix A, "Interpreting Variables of the Enterprise Manager MIB" and Appendix B, "Enterprise Manager MIB Definition." If you are upgrading from a pre-12c version of Enterprise Manager, see Appendix C, "SNMP Trap Mappings" for specific version mappings.

For Enterprise Manager 12c, SNMP traps are delivered for event notifications only. SNMP trap notifications are not supported for incidents or problems.

Note:

Notification methods based on SNMP traps must be configured by an administrator with Super Administrator privileges before any user can then choose to select one or more of these SNMP trap methods while creating/editing a incident rule.

4.5.1 SNMP Version 1 Versus SNMP Version 3

SNMP Version 3 shares the same basic architecture of Version 1, but adds numerous enhancements to SNMP administration and security. The primary enhancement relevant to Enterprise Manager involves additional security levels that provide both authentication and privacy as well as authorization and access control.

User-based Security Model (USM)

USM defines the security-related procedures followed by an SNMP engine when processing SNMP messages. Enterprise Manager SNMP V3 support takes advantage of this added SNMP message-level security enhancement to provide a secure messaging environment.

USM protects against two primary security threats:

  • Modification of information: The modification threat is the danger that some unauthorized entity may alter in-transit SNMP messages generated on behalf of an authorized principal in such a way as to effect unauthorized management operations, including falsifying the value of an object.

  • Masquerade: The masquerade threat is the danger that management operations not authorized for some user may be attempted by assuming the identity of another user that has the appropriate authorizations.

For both SNMP versions, the basic methodology for setting up Enterprise Manager advanced notifications using SNMP traps remains the same:

  1. Define the notification method based on an SNMP trap.

  2. Assign the notification method to an incident rule.

4.5.2 Working with SNMP V3 Trap Notification Methods

The procedure for defining an SNMP V3 trap notification method differs slightly from that of V1. Beginning with Enterprise Manager Release 12.1.0.4, a separate interface consolidates key information and configuration functionality pertaining to SNMP V3 trap notification methods. The SNMP V3 Trap interface helps guide you through the process of creating SNMP notification methods, enabling the OMS to send SNMP traps, and defining user security settings for SNMP trap notifications.

4.5.2.1 Configuring the OMS to Send SNMP Trap Notifications

Before creating an SNMP Trap notification method, you must enable at least one OMS is your environment to handle SNMP Trap notifications. For SNMP V3, the OMS serves as an SNMP Agent which sends traps to the SNMP Manager that is monitoring all SNMP Agents deployed in the network.

  1. From the Setup menu, select Notifications and then SNMP V3 Traps. The Getting Started page displays. This page documents the high-level workflow for configuring Enterprise Manager to send traps to third-party SNMP Managers.

    displays the SNMP V3 getting started page
  2. Click the Configuration tab. The Configuration page displays.

    displays the configuration page
  3. In the OMS Configuration region, select the OMS you wish to enable.

  4. Check the following for each OMS and make changes, if necessary:

    • OMS requires a port for SNMPv3 traps. Check if the default port can be used by OMS.

    • OMS requires a unique Engine ID for sending traps. By default, it is being generated from the host name and port.

  5. Click Enable.

4.5.2.2 Creating/Editing an SNMP V3 Trap Notification Method

Once an OMS has been enabled to send SNMP traps notifications, the next step is to create a notification method than can be used by an incident rule.

  1. From the Setup menu, select Notifications and then SNMP V3 Traps. The Getting Started page displays.

    Note:

    If want to edit an existing Notification Method, select the desired method from the Notification Methods region and click Edit.
  2. Click the Configuration tab. The Configuration page displays.

    displays the configuration page
  3. From the Notification Methods region, click Create. The SNMPv3 Traps: Create Notification Method page displays. shows the create notification method page

  4. Enter the requisite Notification Method definition parameters. Note: You can enable Repeat Notifications at this point.

  5. If you choose to create a new User Security Model entry, from the User Security Model region, ensure the Create New option is chosen.

    1. Specify a Username that uniquely identifies the credential. SNMP V3 allows multiple usernames to be set in an SNMP Agent as well as SNMP Manager applications.

    2. Select a Security Level from the drop-down menu. Available parameters become available depending on the security level. There are three levels from which to choose:

      AuthPriv (Authentication + Privacy:) The sender's identity must be confirmed by the receiver (authentication). SNMP V3 messages are encrypted by the sender and must be decrypted by the receiver (privacy).

      AuthNoPriv (Authentication only): The receiver must authenticate the sender's identity before accepting the message.

      NoAuthNoPriv (no security): Neither sender identity confirmation nor message encryption is used.

    3. For AuthPriv and AuthNoPriv security levels, choose a the desired Authentication Protocol. Two authentication protocols are available:

      Secure Hash Algorithm (SHA)

      Message Digest algorithm (MD5)

      The authentication protocols are used to build the message digest when the message is authenticated.

      Privacy Protocol (used for the AuthPriv security level) is used to encrypt/decrypt messages. USM uses the Data Encryption Standard (DES). The Privacy Password is used in conjunction with the Privacy Protocol. the privacy password on both the SNMP Agent and SNMP Manager must match in order for encryption/decryption to succeed.

    If you have already have predefined User Security Model entries, choose the Use Existing option and select one of the USM entries from the drop-down menu. USM entries are listed by username.

    Important:

    Ensure that the USM credentials are identical in OMS and the external trap receiver. If they do not match, Enterprise Manager will still send the SNMP trap, but the trap will not be received. If the USM credentials are invalid, Enterprise Manager will still send the SNMP trap, however, the trap will not be received as the incorrect credentials will result in an authentication error at the SNMP receiver. This type of authentication error will not be apparent from the Enterprise Manager console.
  6. Once you have entered the requisite Notification Method and USM parameters, click Save. The newly created notification method appears in the Notification Method region of the Configuration page.

    newly created notification method

Note:

Once you have defined the SNMP V3 Trap notification method, you must add it to a rule. See "Creating a Rule to Send SNMP Traps to Third Party Systems" for instructions.

4.5.2.3 Editing a User Security Model Entry

You can add USM entries at any time.

  1. From the Setup menu, select Notifications, and then SNMP V3 Traps.

  2. Click on the Configurations tab.

  3. From the User Security Model Entries region, click Create. The User Security Model Entries dialog displays.

    User Security Model Entries dialog
  4. Specify a Username that uniquely identifies the credential. SNMP V3 allows multiple usernames to be set in an SNMP Agent as well as SNMP Manager applications.

  5. Select a Security Level from the drop-down menu. Available parameters become available depending on the security level. There are three levels from which to choose:

    AuthPriv (Authentication + Privacy:) The sender's identity must be confirmed by the receiver (authentication). SNMP V3 messages are encrypted by the sender and must be decrypted by the receiver (privacy).

    AuthNoPriv (Authentication only): The receiver must authenticate the sender's identity before accepting the message.

    NoAuthNoPriv (no security): Neither sender identity confirmation nor message encryption is used.

  6. For AuthPriv and AuthNoPriv security levels, choose a the desired Authentication Protocol. Two authentication protocols are available:

    Secure Hash Algorithm (SHA)

    Message Digest algorithm (MD5)

    The authentication protocols are used to build the message digest when the message is authenticated.

    Privacy Protocol (used for the AuthPriv security level) is used to encrypt/decrypt messages. USM uses the Data Encryption Standard (DES). The Privacy Password is used in conjunction with the Privacy Protocol. the privacy password on both the SNMP Agent and SNMP Manager must match in order for encryption/decryption to succeed.

  7. Click OK.

    The new USM username will appear in the User Security Model Entries table. When creating new SNMP V3 Trap notification methods, the USM username will appear as a selectable option from the Existing Entries drop-down menu.

    After editing the USM, you should verify the change via the notification methods that use it.

4.5.2.4 Viewing Available SNMP V3 Trap Notification Methods

To view available SNMP V3 Trap notification methods:

  1. From the Setup menu, select Notifications, and then SNMP V3 Traps.

  2. Click on the Configurations tab.

  3. The Notification Methods region displays existing SNMP V3 Trap notification methods.

4.5.2.5 Deleting an SNMP V3 Trap Notification Method

To delete available SNMP V3 Trap notification methods:

  1. From the Setup menu, select Notifications, and then SNMP V3 Traps.

  2. Click on the Configurations tab.

  3. From the Notification Methods region, select an existing SNMP V3 Trap notification method.

  4. Click Delete.

4.5.3 Creating an SNMP V1 Trap

Step 1: Define a new notification method based on an SNMP trap.

Log in to Enterprise Manager as a Super Administrator. From the Setup menu, select Notifications and then select Notification Method to access the Notification Methods page. From this page you can add a new method based on an SNMP trap.

You must provide the name of the host (machine) on which the SNMP master agent is running and other details as shown in the following example. As shown in, the SNMP host will receive your SNMP traps.

Figure 4-7 SNMP Trap Required Information

SNMP Trap Required Information

Note:

A Test SNMP Trap button exists for you to test your setup.

Metric severity information will be passed as a series of variables in the SNMP trap.

Step 2: Assign the notification method to a rule.

You can edit an existing rule (or create a new incident rule), then add an action to the rule that subscribes to the advanced notification method. For instructions on setting up incident rules using SNMP traps, see "Creating a Rule to Send SNMP Traps to Third Party Systems".

Example SNMP Trap Implementation

In this scenario, you want to identify the unique issues from the SNMP traps that are sent. Keep in mind that all events that are related to the same issue are part of the same event sequence. Each event sequence has a unique identification number.

An event sequence is a sequence of related events that represent the life of a specific issue from the time it is detected and an event is raised to the time it is fixed and a corresponding clear event is generated. For example, a warning metric alert event is raised when the CPU utilization of a host crosses 80%. This starts the event sequence representing the issue CPU Utilization of the host is beyond normal level. Another critical event is raised for the same issue when the CPU utilization goes above 90% and the event is added to the same event sequence. After a period of time, the CPU utilization returns to a normal level and a clear event is raised. At this point, the issue is resolved and the event sequence is closed.

The SNMP trap sent for this scenario is shown in Example 4-15. Each piece of information is sent as a variable embedded in the SNMP Trap.

Example 4-15 SNMP Trap

**************V1 TRAP***[1]*****************
Community : public
Enterprise :1.3.6.1.4.1.111.15.2
Generic :6
Specific :3
TimeStamp :67809
Agent adress :10.240.36.109
1.3.6.1.4.1.111.15.3.1.1.2.1: NOTIF_NORMAL
1.3.6.1.4.1.111.15.3.1.1.3.1: CPU Utilization is 92.658%, crossed warning (80) or critical (90) threshold.
1.3.6.1.4.1.111.15.3.1.1.4.1: https://sampleserver.oracle.com:5416/em/redirect?pageType=sdk-core-event-console-detailEvent&issueID=C77AE9E578F00773E040F00A6D242F90
1.3.6.1.4.1.111.15.3.1.1.5.1: Critical
1.3.6.1.4.1.111.15.3.1.1.6.1: CRITICAL
1.3.6.1.4.1.111.15.3.1.1.7.1: 0
1.3.6.1.4.1.111.15.3.1.1.8.1:
1.3.6.1.4.1.111.15.3.1.1.9.1:
1.3.6.1.4.1.111.15.3.1.1.10.1: Aug 17, 2012 3:26:36 PM PDT
1.3.6.1.4.1.111.15.3.1.1.11.1: Capacity
1.3.6.1.4.1.111.15.3.1.1.12.1: Capacity
1.3.6.1.4.1.111.15.3.1.1.13.1: Metric Alert
1.3.6.1.4.1.111.15.3.1.1.14.1: Load:cpuUtil
1.3.6.1.4.1.111.15.3.1.1.15.1: 281
1.3.6.1.4.1.111.15.3.1.1.16.1:
1.3.6.1.4.1.111.15.3.1.1.17.1: No
1.3.6.1.4.1.111.15.3.1.1.18.1: New
1.3.6.1.4.1.111.15.3.1.1.19.1: None
1.3.6.1.4.1.111.15.3.1.1.20.1: 0
1.3.6.1.4.1.111.15.3.1.1.21.1: sampleserver.oracle.com
1.3.6.1.4.1.111.15.3.1.1.22.1: https://sampleserver.oracle.com:5416/em/redirect?pageType=TARGET_HOMEPAGE&targetName=sampleserver.oracle.com&targetType=host
1.3.6.1.4.1.111.15.3.1.1.23.1: Host
1.3.6.1.4.1.111.15.3.1.1.24.1: sampleserver.oracle.com
1.3.6.1.4.1.111.15.3.1.1.25.1: SYSMAN
1.3.6.1.4.1.111.15.3.1.1.26.1:
1.3.6.1.4.1.111.15.3.1.1.27.1: 5.8.0.0.0
1.3.6.1.4.1.111.15.3.1.1.28.1: Operating System=Linux, Platform=x86_64,
1.3.6.1.4.1.111.15.3.1.1.29.1:
1.3.6.1.4.1.111.15.3.1.1.30.1:
1.3.6.1.4.1.111.15.3.1.1.31.1:
1.3.6.1.4.1.111.15.3.1.1.32.1:
1.3.6.1.4.1.111.15.3.1.1.33.1:
1.3.6.1.4.1.111.15.3.1.1.34.1:
1.3.6.1.4.1.111.15.3.1.1.35.1:
1.3.6.1.4.1.111.15.3.1.1.36.1:
1.3.6.1.4.1.111.15.3.1.1.37.1:
1.3.6.1.4.1.111.15.3.1.1.38.1:
1.3.6.1.4.1.111.15.3.1.1.39.1: SnmpNotifRuleset
1.3.6.1.4.1.111.15.3.1.1.40.1: SnmpNotifRuleset,SnmpNotifEvent
1.3.6.1.4.1.111.15.3.1.1.41.1: SYSMAN
1.3.6.1.4.1.111.15.3.1.1.42.1: C77AE9E578F00773E040F00A6D242F90
1.3.6.1.4.1.111.15.3.1.1.43.1:
1.3.6.1.4.1.111.15.3.1.1.44.1:
1.3.6.1.4.1.111.15.3.1.1.45.1:
1.3.6.1.4.1.111.15.3.1.1.46.1: CPU Utilization is 92.658%, crossed warning (80) or critical (90) threshold., Incident created by rule (Name = Incident management Ruleset for all targets, Incident creation Rule for metric alerts.; Owner = <SYSTEM>).
1.3.6.1.4.1.111.15.3.1.1.61.1: Metric GUID=0C71A1AFAC2D7199013837DA35522C08
1.3.6.1.4.1.111.15.3.1.1.62.1: Severity GUID=C77AE9E578EC0773E040F00A6D242F90
1.3.6.1.4.1.111.15.3.1.1.63.1: Cycle GUID=C77AE9E578EC0773E040F00A6D242F90
1.3.6.1.4.1.111.15.3.1.1.64.1: Collection Name=LoadLinux
1.3.6.1.4.1.111.15.3.1.1.65.1: Metric Group=Load
1.3.6.1.4.1.111.15.3.1.1.66.1: Metric=CPU Utilization (%)
1.3.6.1.4.1.111.15.3.1.1.67.1: Metric Description=
1.3.6.1.4.1.111.15.3.1.1.68.1: Metric value=92.658
1.3.6.1.4.1.111.15.3.1.1.69.1: Key Value=
1.3.6.1.4.1.111.15.3.1.1.70.1:
1.3.6.1.4.1.111.15.3.1.1.71.1:
1.3.6.1.4.1.111.15.3.1.1.72.1:
1.3.6.1.4.1.111.15.3.1.1.73.1:
1.3.6.1.4.1.111.15.3.1.1.74.1:
1.3.6.1.4.1.111.15.3.1.1.75.1:
1.3.6.1.4.1.111.15.3.1.1.76.1:
1.3.6.1.4.1.111.15.3.1.1.77.1:
1.3.6.1.4.1.111.15.3.1.1.78.1:
1.3.6.1.4.1.111.15.3.1.1.79.1:
1.3.6.1.4.1.111.15.3.1.1.80.1:
1.3.6.1.4.1.111.15.3.1.1.81.1:
1.3.6.1.4.1.111.15.3.1.1.82.1:
1.3.6.1.4.1.111.15.3.1.1.83.1:
1.3.6.1.4.1.111.15.3.1.1.84.1: Number of keys=0
1.3.6.1.4.1.111.15.3.1.1.85.1:
**************END V1 TRAP******************

This following example illustrates how OIDs are used during the lifecycle of an event. Here, for one event (while the event is open), the event sequence OID remains the same even though the event severity changes.

The OID for the event sequence is:

1.3.6.1.4.1.111.15.3.1.1.42.1: C77AE9E578F00773E040F00A6D242F90

The OID for the event severity code is:

1.3.6.1.4.1.111.15.3.1.1.6.1: CRITICAL

When the event clears, these OIDs show the same event sequence with a different severity code:

The OID for the event sequence is:

1.3.6.1.4.1.111.15.3.1.1.42.1: C77AE9E578F00773E040F00A6D242F90

The OID for the event severity code is:

1.3.6.1.4.1.111.15.3.1.1.6.1: CLEAR

The length of the SNMP OID value is limited to 2560 bytes by default. Configure the emoms property oracle.sysman.core.notification.snmp.max_oid_length to change the default limit.

4.5.4 SNMP Traps: Moving from Previous Enterprise Manager Releases to 12c

Note:

When you upgrade from a pre-Enterprise Manager 12c release to 12c, SNMP advanced notification methods defined using previous versions of Enterprise Manager (pre-12c) will continue to function without modification.

For Enterprise Manager 11g and earlier, there were two types of SNMP traps:

  • Alerts

  • Job Status

For Enterprise Manager 12c there is now a single, comprehensive SNMP trap type that covers all available event types such as metric alerts, target availability, compliance standard violations, or job status changes. For more information about pre-12c to 12c SNMP trap mappings, see Appendix C, "SNMP Trap Mappings." Traps will conform to the older Enterprise Manager MIB definition. Hence, pre-Enterprise Manager 12c traps will continue to be sent. See Appendix C, "SNMP Trap Mappings" for more information.

Also, for Enterprise Manager 12c, size of SNMP trap has increased in order to accommodate all event types and provide more comprehensive information. By default, the maximum SNMP packet size is 5120 bytes. If the third party system has a limit in the size of SNMP trap it can receive, you can change the default size of SNMP trap that Enterprise Manager sends. To change the default packet size, set this emoms oracle.sysman.core.notification.snmp_packet_length parameter, and then bounce the OMS.

Note:

When limiting the SNMP trap packet size, Oracle recommends not setting the oracle.sysman.core.notification.snmp_packet_length parameter any lower than 3072 bytes (3K).

The Enterprise Manager 12c MIB includes all pre-Enterprise Manager 12c MIB definitions. Hence, if you have an Enterprise Manager 12c MIB in your third party system, you can receive SNMP traps from both pre-Enterprise Manager 12c as well as Enterprise Manager 12c sites. For detailed information on version mapping, see Appendix C, "SNMP Trap Mappings."

4.6 Management Information Base (MIB)

Enterprise Manager Cloud Control can send SNMP Traps to third-party, SNMP-enabled applications. Details of the trap contents can be obtained from the management information base (MIB) variables. The following sections discuss Enterprise Manager MIB variables in detail.

4.6.1 About MIBs

A MIB is a text file, written in ASN.1 notation, which describes the variables containing the information that SNMP can access. The variables described in a MIB, which are also called MIB objects, are the items that can be monitored using SNMP. There is one MIB for each element being monitored. Each monolithic or subagent consults its respective MIB in order to learn the variables it can retrieve and their characteristics. The encapsulation of this information in the MIB is what enables master agents to register new subagents dynamically — everything the master agent needs to know about the subagent is contained in its MIB. The management framework and management applications also consult these MIBs for the same purpose. MIBs can be either standard (also called public) or proprietary (also called private or vendor).

The actual values of the variables are not part of the MIB, but are retrieved through a platform-dependent process called "instrumentation". The concept of the MIB is very important because all SNMP communications refer to one or more MIB objects. What is transmitted to the framework is, essentially, MIB variables and their current values.

4.6.2 MIB Definition

You can find the SNMP MIB file at the following location:

OMS_HOME/network/doc/omstrap.v1

Note:

The omstrap.v1 file is compatible with both SNMP V1 and SNMP V3.

The file omstrap.v1 is the OMS MIB.

For more information, see Appendix A, "Interpreting Variables of the Enterprise Manager MIB."

A hardcopy version of omstrap.v1 can be found in Appendix B, "Enterprise Manager MIB Definition."

The length of the SNMP OID value is limited to 2560 bytes by default. Configure emoms property oracle.sysman.core.notification.snmp.max_oid_length to change the default limit.

For Enterprise Manager 12c, SNMP traps are delivered for event notifications only. SNMP trap notifications are not supported for incidents or problems.

Note:

SNMP advanced notification methods defined using previous versions of Enterprise Manager (pre-12c) will continue to function without modification. Traps will conform to the older Enterprise Manager MIB definition.

4.6.3 Reading the MIB Variable Descriptions

This section covers the format used to describe MIB variables. Note that the STATUS element of SNMP MIB definition, Version 1, is not included in these MIB variable descriptions. Since Oracle has implemented all MIB variables as CURRENT, this value does not vary.

4.6.3.1 Variable Name

Syntax

Maps to the SYNTAX element of SNMP MIB definition, Version 1.

Max-Access

Maps to the MAX-ACCESS element of SNMP MIB definition, Version 1.

Status

Maps to the STATUS element of SNMP MIB definition, Version 1.

Explanation

Describes the function, use and precise derivation of the variable. (For example, a variable might be derived from a particular configuration file parameter or performance table field.) When appropriate, incorporates the DESCRIPTION part of the MIB definition, Version 1.

Typical Range

Describes the typical, rather than theoretical, range of the variable. For example, while integer values for many MIB variables can theoretically range up to 4294967295, a typical range in an actual installation will vary to a lesser extent. On the other hand, some variable values for a large database can actually exceed this "theoretical" limit (a "wraparound"). Specifying that a variable value typically ranges from 0 to 1,000 or 1,000 to 3 billion will help the third-party developer to develop the most useful graphical display for the variable.

Significance

Describes the significance of the variable when monitoring a typical installation. Alternative ratings are Very Important, Important, Less Important, or Not Normally Used. Clearly, the DBA will want to monitor some variables more closely than others. However, which variables fall into this category can vary from installation to installation, depending on the application, the size of the database, and on the DBA's objectives. Nevertheless, assessing a variable's significance relative to the other variables in the MIB can help third-party developers focus their efforts on those variables of most interest to the most DBAs.

Related Variables

Lists other variables in this MIB, or other MIBs implemented by Oracle, that relate in some way to this variable. For example, the value of this variable might derive from that of another MIB variable. Or perhaps the value of this variable varies inversely to that of another variable. Knowing this information, third-party developers can develop useful graphic displays of related MIB variables.

Suggested Presentation

Suggests how this variable can be presented most usefully to the DBA using the management application: as a simple value, as a gauge, or as an alarm, for example.

4.7 Passing Corrective Action Status Change Information

Passing corrective action status change attributes (such as new status, job name, job type, or rule owner) to PL/SQL procedures or OS commands/scripts allows you to customize automated responses to status changes. For example, you many want to call an OS script to open a trouble ticket for an in-house support trouble ticket system if a critical corrective action fails to run. In this case, you will want to pass status (for example, Problems or Aborted) to the script to open a trouble ticket and escalate the problem.

4.7.1 Passing Corrective Action Execution Status to an OS Command or Script

The notification system passes information to an OS script or executable via system environment variables. Conventions used to access environmental variables vary depending on the operating system:

  • UNIX: $ENV_VARIABLE

  • MS Windows: %ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The notification system will set the environment variable $NOTIF_TYPE = NOTIF_CA for Corrective Action Execution. The script can then use any or all of these variables within the logic of the script.

Following table lists the environment variables for corrective action, they are populated when a corrective action is completed for an event.

Table 4-13 Corrective Action Environment Variables

Environment Variable Description

CA_JOB_STATUS

Corrective action job execution status.

CA_JOB_NAME

Name of the Corrective Action.

CA_JOB_OWNER

Owner of Corrective Action.

CA_JOB_STEP_OUTPUT

The value will be the text output from the Corrective Action execution.

CA_JOB_TYPE

Corrective Action Job type


4.7.2 Passing Corrective Action Execution Status to a PLSQL Procedure

The notification system passes corrective action status change information to PL/SQL procedure - PROCEDURE p(event_msg IN gc$notif_event_msg). The instance gc$notif_corrective_action_job object is defined in event_msg.event_payload. corrective_action if event_msg. msg_info. notification_type is equal to GC$NOTIFICATIONNOTIF_CA. When a corrective action executes, the notification system calls the PL/SQL procedure associated with the incident rule and passes the populated object to the procedure. The procedure is then able to access the fields of the object that has been passed to it. See Table 4-44, "Corrective Action Job-Specific Attributes" for details.

The following status codes are possible values for the job_status field of the MGMT_NOTIFY_CORRECTIVE_ACTION object.

Table 4-14 Corrective Action Status Codes

Name Datatype Value

SCHEDULED_STATUS

NUMBER(2)

1

EXECUTING_STATUS

NUMBER(2)

2

ABORTED_STATUS

NUMBER(2)

3

FAILED_STATUS

NUMBER(2)

4

COMPLETED_STATUS

NUMBER(2)

5

SUSPENDED_STATUS

NUMBER(2)

6

AGENTDOWN_STATUS

NUMBER(2)

7

STOPPED_STATUS

NUMBER(2)

8

SUSPENDED_LOCK_STATUS

NUMBER(2)

9

SUSPENDED_EVENT_STATUS

NUMBER(2)

10

SUSPENDED_BLACKOUT_STATUS

NUMBER(2)

11

STOP_PENDING_STATUS

NUMBER(2)

12

SUSPEND_PENDING_STATUS

NUMBER(2)

13

INACTIVE_STATUS

NUMBER(2)

14

QUEUED_STATUS

NUMBER(2)

15

FAILED_RETRIED_STATUS

NUMBER(2)

16

WAITING_STATUS

NUMBER(2)

17

SKIPPED_STATUS

NUMBER(2)

18

REASSIGNED_STATUS

NUMBER(2)

20


4.8 Passing Job Execution Status Information

Passing job status change attributes (such as new status, job name, job type, or rule owner) to PL/SQL procedures or OS commands/scripts allows you to customize automated responses to status changes. For example, you many want to call an OS script to open a trouble ticket for an in-house support trouble ticket system if a critical job fails to run. In this case you will want to pass status (for example, Problems or Aborted) to the script to open a trouble ticket and escalate the problem. The job execution status information is one of event type - job_status_change event, and its content is in OS command and PL/SQL payload as described in Section 4.3, "Sending Notifications Using OS Commands and Scripts" and Section 4.4, "Sending Notifications Using PL/SQL Procedures".

4.8.1 Passing Job Execution Status to a PL/SQL Procedure

The notification system passes job status change information to a PL/SQL procedure via the event_msg.event_payload object where event_type is equal to job_status_change. An instance of this object is created for every status change. When a job changes status, the notification system calls the PL/SQL p(event_msg IN gc$notif_event_msg) procedure associated with the incident rule and passes the populated object to the procedure. The procedure is then able to access the fields of the event_msg.event_payload object that has been passed to it.

Table 4-15 lists all corrective action status change attributes that can be passed:

Table 4-15 Job Status Attributes

Attribute Datatype Additional Information

event_msg.event_payload.source.source_name

VARCHAR2(128)

The job name.

event_msg.event_payload.source.source_owner

VARCHAR2(256)

The owner of the job.

event_msg.event_payload.source.source_sub_type

VARCHAR2(32)

The type of the job.

event_msg.event_payload. event_attrs(i).value where event_attrs(i).name=' execution_status'

NUMBER

The new status of the job.

event_msg.event_payload. event_attrs(i).value where event_attrs(i).name='state_change_guid'

RAW(16)

The GUID of the state change record.

event_msg.event_payload.source.source_guid

RAW(16)

The unique id of the job.

event_msg target.event_payload. event_attrs(i).value where event_attrs(i).name=' execution_id'

RAW(16)

The unique id of the execution.

event_msg.event_payload.target

gc$notif_target

Target Information object..

event_msg.msg_info.rule_owner

VARCHAR2(64)

The name of the notification rule that cause the notification to be sent.

event_msg.msg_info.rule_name

VARCHAR2(132)

The owner of the notification rule that cause the notification to be sent.

event_msg.event_payload. reported_date

DATE

The time and date when the status change happened.


When a job status change occurs for the job, the notification system creates an instance of the event_msg.event_payload. event_attrs(i).value where event_attrs(i).name=' execution_status' object and populates it with values from the status change. The following status codes have been defined as constants in the MGMT_JOBS package and can be used to determine the type of status in the job_status field of the event_msg.event_payload. event_attrs(i).value where event_attrs(i).name=' execution_status' object.

Table 4-16 Job Status Codes

Name Datatype Value

SCHEDULED_STATUS

NUMBER(2)

1

EXECUTING_STATUS

NUMBER(2)

2

ABORTED_STATUS

NUMBER(2)

3

FAILED_STATUS

NUMBER(2)

4

COMPLETED_STATUS

NUMBER(2)

5

SUSPENDED_STATUS

NUMBER(2)

6

AGENTDOWN_STATUS

NUMBER(2)

7

STOPPED_STATUS

NUMBER(2)

8

SUSPENDED_LOCK_STATUS

NUMBER(2)

9

SUSPENDED_EVENT_STATUS

NUMBER(2)

10

SUSPENDED_BLACKOUT_STATUS

NUMBER(2)

11

STOP_PENDING_STATUS

NUMBER(2)

12

SUSPEND_PENDING_STATUS

NUMBER(2)

13

INACTIVE_STATUS

NUMBER(2)

14

QUEUED_STATUS

NUMBER(2)

15

FAILED_RETRIED_STATUS

NUMBER(2)

16

WAITING_STATUS

NUMBER(2)

17

SKIPPED_STATUS

NUMBER(2)

18

REASSIGNED_STATUS

NUMBER(2)

20


Example 4-16 PL/SQL Procedure Using a Status Code (Job)

CREATE  TABLE job_log (jobid RAW(16), status_code NUMBER(2), occured DATE);
 
CREATE OR REPLACE PROCEDURE LOG_JOB_STATUS_CHANGE(event_msg IN GC$NOTIF_EVENT_MSG)
IS
  l_attrs gc$notif_event_attr_array;
  exec_status_code NUMBER(2) := NULL;
  occured_date DATE := NULL;
  job_guid RAW(16) := NULL;
 
BEGIN
  IF event_msg.event_payload.event_type = 'job_status_change'
  THEN
    l_attrs := event_msg.event_payload.event_attrs;
    IF l_attrs IS NOT NULL
    THEN
      FOR i IN 1..l_attrs.COUNT
      LOOP
        IF l_attrs(i).name = 'exec_status_code'
        THEN
          exec_status_code := TO_NUMBER(l_attrs(i).value);
        END IF;
      END LOOP;
    END IF;
 
    occured_date := event_msg.event_payload.reported_date;
    job_guid := event_msg.event_payload.source.source_guid;
    -- Log all jobs' status
    BEGIN
      INSERT INTO job_log (jobid, status_code, occured)
      VALUES (job_guid, exec_status_code, occured_date);
    EXCEPTION
    WHEN OTHERS
    THEN
      -- If there are any problems then get the notification retried
      RAISE_APPLICATION_ERROR(-20000, 'Please retry');
    END;
    COMMIT;
 
  ELSE
    null; -- it is not a job_status_change event, ignore
  END IF; 
END LOG_JOB_STATUS_CHANGE;
/

4.8.2 Passing Job Execution Status to an OS Command or Script

The notification system passes job execution status information to an OS script or executable via system environment variables. Conventions used to access environmental variables vary depending on the operating system:

  • UNIX: $ENV_VARIABLE

  • MS Windows: %ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.

Table 4-17 Environment Variables

Environment Variable Description

SOURCE_OBJ_NAME

The name of the job.

SOURCE_OBJ_OWNE

The owner of the job.

SOURCE_OBJ_SUB_TYPE

The type of job.

EXEC_STATUS_CODE

The job status.

EVENT_REPORTED_TIME

Time when the severity occurred.

TARGET_NAME

The name of the target.

TARGET_TYPE

The type of the target.

RULE_NAME

Name of the notification rule that resulted in the severity.

RULE_OWNER

Name of the Enterprise Manager administrator who owns the notification rule.


4.9 Passing User-Defined Target Properties to Notification Methods

Enterprise Manager allows you to define target properties (accessed from the target home page) that can be used to store environmental or usage context information specific to that target. Target property values are passed to custom notification methods where they can be processed using conditional logic or simply passed as additional alert information to third-party devices, such as ticketing systems. By default, Enterprise Manager passes all defined target properties to notification methods.

Note:

Target properties are not passed to notification methods when short e-mail format is used.

Figure 4-8 Host Target Properties

Description of Figure 4-8 follows
Description of "Figure 4-8 Host Target Properties"

4.10 Notification Reference

This section contains the following reference material:

4.10.1 EMOMS Properties

EMOMS properties can be used for controlling the size and format of the short e-mail. The following table lists emoms properties for Notification System.

Table 4-18 emoms Properties for Notifications

Property Name Default Value Description

oracle.sysman.core.notification.emails_per_minute

250

Email delivery limits per minute. The Notification system uses this value to throttle number of Email delivery per minutes. Customer should set the value lower if doesn't want to over flow the Email server, or set the value higher if the Email server can handle high volume of Emails.

oracle.sysman.core.notification.cmds_per_minute

100

OS Command delivery limits per minute. The Notification system uses this value to throttle number of OS Command delivery per minutes.

oracle.sysman.core.notification.os_cmd_timeout

30

OS Command delivery timeout in seconds. This value indicates how long to allow OS process to execute the OS Command delivery. Set this value higher if the OS command script requires longer time to complete execution.

oracle.sysman.core.notification.plsql_per_minute

250

PL/SQL delivery limits per minute. The Notification system uses this value to throttle number of PL/SQL delivery per minutes.

em.notification.java_per_minute

500

JAVA delivery limits per minute. The Notification system uses this value to throttle number of Java delivery per minutes.

em.notification.ticket_per_minute

250

Ticket delivery limits per minute. The Notification system uses this value to throttle number of Ticket delivery per minutes.

oracle.sysman.core.notification.traps_per_minute

250

SNMP delivery limits per minute. The Notification system uses this value to control the number of SNMP trap per minutes.

oracle.sysman.core.notification.locale.plsql

OMS Locale

This property specifies the Locale delivered by advanced PL/SQL notification. The customer can define this property to overwrite the default Locale where the OMS is installed.

oracle.sysman.core.notification.locale.email

OMS Locale

This property specifies the Locale delivered by Email. Customer can define this property to overwrite the default Locale where the OMS is installed.

oracle.sysman.core.notification.locale.oscmd

OMS Locale

This property specifies the Locale delivered by OS Command. Customer can define this property to overwrite the default Locale where the OMS is installed.

oracle.sysman.core.notification.locale.snmp

OMS Locale

This property specifies the Locale delivered by SNMP trap. Customer can define this property to overwrite the default Locale where the OMS is installed.

oracle.sysman.core.notification.oscmd.max_env_var_length

512

The maximum length of OS Common environment variable value.

oracle.sysman.core.notification.snmp.max_oid_length

2560

The maximum length of SNMP OID value.

oracle.sysman.core.notification.min_delivery_threads

6

The minimum number of active threads in the thread pool initially and number of active threads are running when system is in low activities. Setting the value higher will use more system resources, but will deliver more notifications.

oracle.sysman.core.notification.max_delivery_threads

24

The maximum number of active threads in the thread pool when the system is in the high activities. This value should greater than em.notification.min_delivery_threads. Setting the value higher will use more system resources and deliver more notifications.

oracle.sysman.core.notification.short_format_length

>=1 (155)

The size limit of the total number of characters in short email format. The customer should modify this property value to fit their email or pager limit content size. The email subject is restricted to a maximum of 80 characters for short email format notifications.

oracle.sysman.core.notification.snmp_packet_length

<=1 (5120)

The maximum size of SNMP Protocol Data unit.

oracle.sysman.core.notification.email_content_transfer_encoding

8-bit, 7-bit(QP), 7-bit(BASE64) (8-bit)

The character set that can encode the Email. Oracle supports three character sets : 8-bit, 7-bit(QP), and7-bit(BASE64).

oracle.sysman.core.notification.emails_per_connection

>=1 (20)

The maximum number of emails delivered to same email gateway before switching to the next available email gateway (assumes customers have configured multiple email gateways). This property is used for email gateway load balance.

oracle.sysman.core.notification.short_format

both, subject, body (both)

Use short format on both subject and body, subject only, or body only..


You must establish the maximum size your device can support and whether the message is sent in subject, body or both.

You can modify the emoms properties by using the Enterprise Manager command line control emctl get/set/delete/list property command.

Note:

The following commands require an OMS restart in order for the changes to take place.

Get Property Command

emctl get [-sysman_pwd "sysman password"]-name oracle.sysman.core.notification.short_format_length

Set Property Command

emctl set  property -name oracle.sysman.core.notification.short_format_length -value 155

Emoms Properties Entries for a Short E-mail Format

emctl set  property -name oracle.sysman.core.notification.short_format_length -value 155
emctl set property -name oracle.sysman.core.notification.short_format -value both

4.10.2 Passing Event, Incident, Problem Information to an OS Command or Script

The notification system passes information to an OS script or executable using system environment variables.

Conventions used to access environmental variables vary depending on the operating system:

  • UNIX: $ENV_VARIABLE

  • Windows:%ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.

4.10.2.1 Environment Variables Common to Event, Incident and Problem

Table 4-19 Generic Environment Variables

Environment Variable Description

NOTIF_TYPE

Type of notification and possible values

NOTIF_NORMAL,

NOTIF_RETRY,

NOTIF_DURATION,

NOTIF_REPEAT,

NOTIF_CA,

NOTIF_RCA

REPEAT_COUNT

How many times the notification has been sent out

before this notification.

RULESET_NAME

The name of the ruleset that triggered this notification.

RULE_NAME

The name of the rule that triggered this notification.

RULE_OWNER

The owner of the ruleset that triggered this notification.

MESSAGE

The message of the event, incident, or problem.

MESSAGE_URL

EM console URL for this message.


Table 4-20 Category-Related Environment Variables

Environment Variable Description

CATEGORIES_COUNT

Number of categories in this notification. This value is equal to1 if one category is associated with event, incident or problem. It is equal to 0 if no category associated with event, incident or problem.

CATEGORY_CODES_COUNT

Number of category codes in this notification.

CATEGORY_n

Category is translated based on locale defined in OMS server. Valid values for the suffix "_n" are between 1.. $CATEGORIES_COUNT

CATEGORY_CODE_n

Codes for the categories. Valid values for the suffix "_n" are between 1..$CATEGORY_CODES_COUNT


Table 4-21 lists the common environment variables for User Defined Target Properties. They will be populated under the following cases: (a) When an event has a related target, (b) When an incident or a problem have single event source and have a related target.

Table 4-21 User-Defined Target Property Environment Variables

Environment Variable Description

ORCL_GTP_COMMENT

Comment

ORCL_GTP_CONTACT

Contact

ORCL_GTP_COST_CENTER

Cost Center

ORCL_GTP_DEPARTMENT

Department

ORCL_GTP_DEPLOYMENT_TYPE

Deployment type

ORCL_GTP_LINE_OF_BUS

Line of Business

ORCL_GTP_LOCATION

Location


4.10.2.2 Event Notification-Specific Environment Variables

Table 4-22 Event Notification-Specific Environment Variables

Environment Variable Description

EVENT_NAME

Event Name.

EVENT_REPORTED_TIME

Event reported date.

EVENT_SOURCE_COUNT

Number of Sources associated with this event.

EVENT_TYPE

Event type.

EVENT_OCCURRENCE_TIME

Event occurrence time.

EVENT_TYPE_ATTRS

The list of event type specific attributes.

EVENT_CONTEXT_ATTRS

Event context data.

LAST_UPDATED_TIME

Last updated time

SEQUENCE_ID

The unique event sequence identifier. An event sequence may consist of one or more events. All events in this sequence have the same event sequence ID.

SEVERITY

Severity of event, it is translated.

SEVERITY_CODE

Code for event severity. Possible values are the following.

FATAL, CRITICAL, WARNING, MINOR_WARNING, INFORMATIONAL, and CLEAR

ACTION_MSG

Message describing the action to take for resolving the event.

TOTAL_OCCURRENCE_COUNT

Total number of duplicate occurrences

RCA_DETAILS

If RCA is associated with this events.

CURRENT_OCCURRENCE_COUNT

Total number of occurrences of the event in the current collection period. This attribute only applies to de-duplicated events.

CURRENT_FIRST_OCCUR_DATE

Time stamp when the event first occurred in the current collection period. This attribute only applies to de-duplicated events.

CURRENT_LAST_OCCUR_DATE_DESC

Time stamp when the e vent last occurred in the current collection period. This attribute only applies to de-duplicated events.


Table 4-23 lists the environment variables for the incident associated with an event. They are populated when the event is associated with an incident.

Table 4-23 Associated Incident Environment Variables

Environment Variable Description

ASSOC_INCIDENT_ACKNOWLEDGED_BY_OWNER

Set to yes, if associated incident was acknowledged by owner

ASSOC_INCIDENT_ACKNOWLEDGED_DETAILS

The details of associated incident acknowledgement. For example:No - if not acknowledged Yes By userName - if acknowledged

ASSOC_INCIDENT_STATUS

Associated Incident Status

ASSOC_INCIDENT_ID

Associated Incident ID

ASSOC_INCIDENT_PRIORITY

Associated Incident priority. Supported value are Urgent, Very High, High, Medium,Low, None.

ASSOC_INCIDENT_OWNER

Associated Incident Owner if it is existed.

ASSOC_INCIDENT_ESCALATION_LEVEL

Escalation level of the associated incident has a value between 0 to 5.


Table 4-24 lists the common environment variables related to the Source Object. They are populated when $SOURCE_OBJ_TYPE is not TARGET.

Table 4-24 Source Object-Related Environment Variables

Environment Variable Description

SOURCE_OBJ_TYPE

Type of the Source object. For example, JOB, TEMPLATE.

SOURCE_OBJ_NAME

Source Object Name.

SOURCE_OBJ_NAME_URL

Source's event console URL.

SOURCE_OBJ_SUB_TYPE

Sub-type of the Source object. For example, it provides the underlying job type for job status change events.

SOURCE_OBJ_OWNER

Owner of the Source object.


Table 4-25 lists the common environment variables for the target, associated with the given issue. They are populated when the issue is related to a target.

Table 4-25 Target-Related Environment Variables

Environment Variable Description

TARGET_NAME

Name of Target

TARGET_TYPE

Type of Target

TARGET_OWNER

Owner of Target

HOST_NAME

The name of the host on which the target is deployed upon.

TARGET_URL

Target's Enterprise Manager Console URL.

TARGET_LIFECYCLE_STATUS

Life Cycle Status of the target.

Possible values: Production, Mission Critical, Stage, Test, and Development.

It is null if not defined.

TARGET_VERSION

Target Version of the target


4.10.2.3 Environment Variables Specific to Event Types

Events are classified into multiple types. For example, the mertc_alert event type is used for modeling metric alerts. You can use SQL queries to list the event types in your deployment as well as their event-specific payload. The following SQL example can be used to list all internal event type names that are registered in Enterprise Manager.

Select event_class as event_type, upper(name) as env_var_name
from em_event_class_attrs
where notif_order != 0
and event_class is not null
union
select event_class as event_type, upper(name) || '_NLS' as env_var_name
from em_event_class_attrs
where notif_order != 0
and event_class is not null
and is_translated = 1
order by event_type, env_var_name; 

The environment variable payload specific to each event type can be accessed via the OS scripts. The following tables list notification attributes for the most critical event types.

Table 4-26 Environment Variables Specific to Metric Alert Event Type

Environment Variable Description

COLL_NAME

The name of the collection collecting the metric.

COLL_NAME_NLS

The translated name of the collection collecting the metric

KEY_COLUMN_X

Internal name of Key Column X where X is a number between 1 and 7.

KEY_COLUMN_X_NLS

Translated name of Key Column X where X is a number between 1 and 7.

KEY_COLUMN_X_VALUE

Value of Key Column X where X is a number between 1 and 7.

KEY_VALUE

Monitored object for the metric corresponding to the Metric Alert event.

METRIC_COLUMN

The name of the metric column

METRIC_COLUMN_NLS

The translated name of the metric column.

METRIC_DESCRIPTION

Brief description of the metric.

METRIC_DESCRIPTION_NLS

Translated brief description of the metric.

METRIC_GROUP

The name of the metric.

METRIC_GROUP_NLS

The translated name of the metric

NUM_KEYS

The number of key metric columns in the metric.

SEVERITY_GUID

The GUID of the severity record associated with this metric alert.

CYCLE_GUID

A unique identifier for a metric alert cycle, which starts from the time the metric alert is initially generated until the time it is clear.

VALUE

Value of the metric when the event triggered.


Table 4-27 Environment variables specific to Target Availability Event Type

Environment Variable Description

AVAIL_SEVERITY

The transition severity that resulted in the status of the target to change to the current availability status.

Possible Values for AVAIL_SEVERITY

  • 15 (Target Up)

  • 25 (Target Down)

  • 115 (Agent Unreachable, Cleared)

  • 125 (Agent Unreachable)

  • 215 (Blackout Ended)

  • 225 (Blackout Started)

  • 315 (Collection Error Cleared)

  • 325 (Collection Error)

  • 425 (No Beacons Available)

  • 515 (Status Unknown)

AVAIL_SUB_STATE

The substatus of a target for the current status.

CYCLE_GUID

A unique identifier for a metric alert cycle, which starts from the time the metric alert is initially generated until the time it is clear.

METRIC_GUID

Metric GUID of response metric.

SEVERITY_GUID

The GUID of the severity record associated with this availability status.

TARGET_STATUS

The current availability status of the target.

TARGET_STATUS_NLS

The translated current availability status of the target.


Table 4-28 Environment variables specific to Job Status Change event type

Environment Variable Description

EXECUTION_ID

Unique ID of the job execution..

EXECUTION_LOG

The job output of the last step executed.

EXECUTION_STATUS

The internal status of the job execution.

EXECUTION_STATUS_NLS

The translated status of the job execution.

EXEC_STATUS_CODE

Execution status code of job execution. For possible values, see Table 4-16, "Job Status Codes".

STATE_CHANGE_GUID

Unique ID of last status change


You can use SQL queries to list the deployed event types in your deployment and the payload specific to each one of them. The following SQL can be used to list all internal event type names which are registered in the Enterprise Manager.

select class_name as event_type_name from em_event_class;

Following SQL lists environment variables specific to metric_alert event type.

select env_var_name
  from
    ( Select event_class as event_type, upper(name) as env_var_name
     from em_event_class_attrs
    where notif_order != 0
    and event_class is not null
    union
    select event_class as event_type, upper(name) || '_NLS' as env_var_name
    from em_event_class_attrs
    where notif_order != 0
    and event_class is not null
    and is_translated = 1)
    where event_type = 'metric_alert';

You can also obtain the description of notification attributes specific to an event type directly from the Enterprise Manager console:

  1. From the Setup menu, select Notifications, then select Customize Email Formats.

  2. Select the event type.

  3. Click Customize.

  4. Click Show Predefined Attributes.

Environment variables, ending with the suffix _NLS, provide the translated value for given attribute. For example, METRIC_COLUMN_NLS environment variable will provide the translated value for the metric column attribute. Translated values will be in the locale of the OMS.

4.10.2.4 Environment Variables Specific to Incident Notifications

Table 4-29 Incident-Specific Environment Variables

Environment Variable Description

SEVERITY

Incident Severity, it is translated. Possible Values: Fatal, Critical, Warning, Informational, Clear

SEVERITY_CODE

Code for Severity.

Possible values are the

FATAL, CRITICAL, WARNING, MINOR_WARNING, INFORMATIONAL, and CLEAR

INCIDENT_REPORTED_TIME

Incident reported time

INCIDENT_ACKNOWLEDGED_BY_OWNER

Set yes, if incident is acknowledged by owner.

INCIDENT_ID

Incident ID

INCIDENT_OWNER

Incident Owner

ASSOC_EVENT_COUNT

The number events associated with this incident.

INCIDENT_STATUS

Incident status. There are two internal fixed resolution status.

NEW

CLOSED

Users can define additional statuses.

ESCALATED

Is Incident escalated

ESCALATED_LEVEL

The escalated level of incident.

PRIORITY

Incident priority. It is the translated priority name. Possible Values: Urgent, Very High, High, Medium, Low, None

PRIOTITY_CODE

Incident priority code

It is the internal value defined in EM.

PRIORITY_URGENT

PRIORITY_VERY_HIGH

PRIORITY_HIGH

PRIORITY_MEDIUM

PRIORITY_LOW

PRIORITY_NONE

TICKET_STATUS

Status of external ticket, if it exists.

TICKET_ID

ID of external ticket, if it exists.

LAST_UPDATED_TIME

Incident last update time.

ADR_INCIDENT_ID

Automatic Diagnostic Reposito ry (ADR) Incident ID: A unique numeric identifier for the ADR Incident. An ADR I ncident is an occurrence of a Problem.

ADR_IMPACT

Impact of the Automatic Diagnostic Repository (ADR) Incident.

ADR_ECID

Execution Context ID (ECID) associated with the associated Automatic Diagnostic Repository (ADR) incident. An ECID i s a globally unique identifier used to tag and track a single call through the Oracle software stack. It is used to correlate problems that could occur across multiple tiers of the stack.

ASSOC_PROBLEM_KEY

Problem key associated with the Automatic Diagnostic Repository (ADR) incident. Problems are critical error s in an Oracle product. The Problem key is a text string that describes the prob lem. It includes an error code and in some cases, other error-specific values.


Table 4-30 lists the associated problem's environment variables, when the incident is associated with a problem.

Table 4-30 Associated Problem Environment Variables for Incidents

Environment Variable Description

ASSOC_PROBLEM_ACKNOWLEDGED_BY_OWNER

Set to yes, if this problem was acknowledged by owner

ASSOC_PROBLEM_STATUS

Associated Problem Status

ASSOC_PROBLEM_ID

Associated Problem ID

ASSOC_PROBLEM_PRIORITY

Associated Problem priority

ASSOC_PROBLEM_OWNER

Associated Problem Owner if it is existed.

ASSOC_PROBLEM_ESCALATION_LEVEL

Escalation level of the associated Problem has a value between 0 to 5.

   

4.10.2.5 Environment Variables Specific to Problem Notifications

Table 4-31 Problem-Specific Environment Variables

Environment Variable Description

SEVERITY

Problem Severity, it is translated.

SEVERITY_CODE

Code for Severity.

Possible values are :

FATAL, CRITICAL, WARNING, MINOR_WARNING, INFORMATIONAL, and CLEAR

PROBLEM_REPORTED_TIME

Problem reported time.

PROBLEM_ACKNOWLEDGED_BY_OWNER

Set yes, if problem is acknowledged by owner.

PROBLEM_ID

Problem ID

PROBLEM_KEY

Problem Key

PROBLEM_OWNER

Problem Owner

ASSOC_INCIDENT_COUNT

The number incident associated with this problem..

PROBLEM_STATUS

Incident status. They are

STATUS_NEW

STATUS_CLOSED

Any other user defined status.

ESCALATED

Is Incident escalated. Yes if it is escalated, otherwise No.

ESCALATED_LEVEL

The escalated level of incident.

PRIORITY

Incident priority. It is the translated priority name..

PRIOTITY_CODE

Incident priority code

It is the internal value defined in Enterprise Manager.

PRIORITY_URGENT

PRIORITY_VERY_HIGH

PRIORITY_HIGH

PRIORITY_MEDIUM

PRIORITY_LOW

PRIORITY_NONE

LAST_UPDATED_TIME

Last updated time

SR_ID

Oracle Service Request Id, if it exists.

BUD_ID

Oracle Bug ID, if an associated bug exists.


4.10.2.6 Environment Variables Common to Incident and Problem Notifications

An incident or problem may be associated with multiple event sources. An event source can be a Target, a Source Object, or both.

4.10.2.6.1 Environment Variables Related to Event Sources

The number of event sources is set by the EVENT_SOURCE_COUNT environment variable. Using the EVENT_SOURCE_COUNT information, a script can be written to loop through the relevant environment variables to fetch the information about multiple event sources. Environment variables for all event sources are prefixed with EVENT_SOURCE_. Environment variables for source objects are suffixed with SOURCE_<attribute_name> . For example, EVENT_SOURCE_1_SOURCE_TYPE provides the source object type of first event source. Environment variables for a target are suffixed with TARGET_<attribute_name>. For example, EVENT_SOURCE_1_TARGET_NAME provides the target name of first event source.

The following table lists the environment variables for source object of x-th Event Source.

Table 4-32 Source Object of the x-th Event Source

Environment Variable Description

EVENT_SOURCE_x_SOURCE_GUID

Source Object GUID.

EVENT_SOURCE_x_SOURCE_TYPE

Source Object Type

EVENT_SOURCE_x_SOURCE_NAME

Source Object Name.

EVENT_SOURCE_x_SOURCE_OWNER

Source Object Owner.

EVENT_SOURCE_x_SOURCE_SUB_TYPE

Source Object Sub-Type.

EVENT_SOURCE_x_SOURCE_URL

Source Object URL to EM console.


Table 4-33 lists the environment variables for a target of xth Event Source.

Table 4-33 Target of x-th Event Source

Environment Variable Description

EVENT_SOURCE_x_TARGET_GUID

Target GUID

EVENT_SOURCE_x_TARGET_NAME

Target name

EVENT_SOURCE_x_TARGET_OWNER

Target Owner

EVENT_SOURCE_x_TARGET_VERSION

Target version

EVENT_SOURCE_x_TARGET_LIFE_CYCLE_STATUS

Target life cycle status

EVENT_SOURCE_x_TARGET_TYPE

Target Type

EVENT_SOURCE_x_HOST_NAME

Target Host Name

EVENT_SOURCE_x_TARGET_URL

Target URL to EM Console.


4.10.3 Passing Information to a PL/SQL Procedure

Passing event, incident, and problem information (payload) to PL/SQL procedures allows you to customize automated responses to these conditions. All three types of notification payloads have a common element: gc$notif_msg_info. It provides generic information that applies to all types of notifications. In addition, each of the three payloads have one specific element that provides the payload specific to the given issue type.

gc$notif_event_msg (payload for event notifications)

gc$notif_event_msg contains two objects - event payload object and message information object.

Table 4-34 Event Notification Payload

Attribute Datatype Additional Information

EVENT_PAYLOAD

gc$notif_event_payload

Event notification payload. See gc$notif_event_payload type definition for detail.

MSG_INFO

gc$notif_msg_info

Notification message. See gc$notif_msg_info definition for detail.


gc$notif_incident_msg (payload for incident notifications)

gc$notif_incident_msg type contains two objects - incident payload and message information. This object represents the delivery payload for Incident notification message, contains all data associated with Incident notification, and can be accessed by user's custom PL/SQL procedures.

Table 4-35 Incident Notification Payload

Attribute Datatype Additional Information

INCIDENT_PAYLOAD

gc$notif_incident_payload

Incident notification payload. See gc$notif_incident_payload type definition for detail.

MSG_INFO

gc$notif_msg_info

Envelope level notification information. See gc$notif_msg_info type definition for detail.


gc$notif_problem_msg (payload for problem notifications)

This object represents the delivery payload for Problem notification message, contains all data associated with problem notification, and can be accessed by a user's custom PL/SQL procedures.

Table 4-36 Problem Notification Payload

Attribute Datatype Additional Information

PROBLEM_PAYLOAD

gc$notif_problem_payload

Problem notification payload. See gc$notif_problem_payload type definition for detail.

MSG_INFO

gc$notif_msg_info

Notification message. See gc$notif_msg_info type definition for detail.


gc$notif_msg_info (common for event/incident/problem payloads)

This object contains the generic notification information including notification_type, rule set and rule name, etc. for Event, Incident or Problem delivery payload.

Table 4-37 Event, Incident, Problem Common Payload

Attribute Datatype Description

NOTIFICATION_TYPE

VARCHAR2(32)

Type of notification, can be one of the following values.

GC$NOTIFICATION.NOTIF_NORMAL GC$NOTIFICATION.NOTIF_RETRY GC$NOTIFICATION.NOTIF_REPEAT GC$NOTIFICATION.NOTIF_DURATION

GC$NOTIFICATION.NOTIF_CA

GC$NOTIFICATION.NOTIF_RCA

REPEAT_COUNT

NUMBER

Repeat notification count

RULESET_NAME

VARCHAR2(256)

Name of the rule set that triggered the notification

RULE_NAME

VARCHAR2(256)

Name of the rule that triggered the notification

RULE_OWNER

VARCAH2(256)

EM User who owns the rule set

MESSAGE

VARCHAR2(4000)

Message about event/incident/problem.

MESSAGE_URL

VARCHAR2(4000)

Link to the Enterprise Manager console page that provides the details of the event/incident/problem.


gc$notif_event_payload (payload specific to event notifications)

This object represents the payload specific to event notifications.

Table 4-38 Common Payloads for Events, Incidents, and Problems

Attribute Datatype Additional Information

EVENT_INSTANCE_GUID

RAW(16)

Event instance global unique identifier.

EVENT_SEQUENCE_GUID

RAW(16)

Event sequence global unique identifier.

TARGET

gc$notif_target

Related Target Information object. See gc$notif_target type definition for detail.

SOURCE

gc$notif_source

Related Source Information object, that is not a target. See gc$notif_source type definition for detail.

EVENT_ATTRS

gc$notif_event_attr_array

The list of event specified attributes. See gc$notif_event_attr type definition for detail.

CORRECTIVE_ACTION

gc$notif_corrective_action_job

Corrective action information, optionally populated when corrective action job execution has completed.

EVENT_TYPE

VARCHAR2(20)

Event type - example: Metric Alert.

EVENT_NAME

VARCHAR2(512)

Event name.

EVENT_MSG

VARCHAR2(4000)

Event message.

REPORTED_DATE

DATE

Event reported date.

OCCURRENCE_DATE

DATE

Event occurrence date.

SEVERITY

VARCHAR2(128)

Event Severity. It is the translated severity name.

SEVERITY_CODE

VARCHAR2(32)

Event Severity code. It is the internal severity name used in Enterprise Manager.

ASSOC_INCIDENT

gc$notif_issue_summary

Summary of associated incident. It is populated if the event is associated with an incident. See gc$notif_issue_summary type definition for detail

ACTION_MSG

VARCHAR2(4000)

Message describing the action to take for resolving the event.

RCA_DETAIL

VARCHAR2(4000)

Root cause analysis detail. The size of RCA details output is limited to 4000 characters long.

EVENT_CONTEXT_DATA

gc$notif_event_context_array

Event context data. See gc$notif_event_context type definition for detail.

CATEGORIES

gc$category_string_array

List of categories that the event belongs to. Category is translated based on locale defined in OMS server. Notification system sends up to 10 categories.

CATEGORY_CODES

gc$category_string_array

Codes for the categories. The size of array is up to 10.


gc$notif_incident_payload (payload specific to incident notifications)

Contains the incident specific attributes, associated problem and ticket information.

Table 4-39 Incident Notification Payloads

Attribute Datatype Additional Information

INCIDENT_ATTRS

gc$notif_issue_attrs

Incident specific attributes. See gc$notif_issue_attrs type definition for detail.

ASSOC_EVENT_COUNT

NUMBER

The total number of events associated with this incident.

TICKET_STATUS

VARCHAR2(64)

The status of external Ticket, if it exists.

TICKET_ID

VARCHAR2(128)

The ID of external Ticket, if it exists.

TICKET_URL

VARCHAR2(4000)

The URL for external Ticket, if it exists.

ASSOC_PROBLEM

gc$notif_issue_summary

Summary of the problem, if it has an associated problem. See gc$notif_issue_summary type definition for detail.


gc$notif_problem_payload (payload specific to problems)

Contains problem specific attributes, key, Service Request(SR) and Bug information.

Table 4-40 Problem Payload

Attribute Datatype Additional Information

PROBLEM_ATTRS

gc$notif_issue_attrs

Problem specific attributes. See gc$notif_issue_attrs type definition for detail.

PROBLEM_KEY

VARCHAR2(850)

Problem key if it is generated.

ASSOC_INCIDENT_COUNT

NUMBER

Number of incidents associated with this problem.

SR_ID

VARCHAR2(64)

Oracle Service Request Id, if it exists.

SR_URL

VARCHAR2(4000)

URL for Oracle Service Request, if it exists.

BUG_ID

VARCHAR2(64)

Oracle Bug ID, if an associated bug exists.


gc$notif_issue_attrs (payload common to incidents and problems)

Provides common details for incident and problem. It contains details such as id, severity, priority, status, categories, acknowledged by owner, and source information with which it is associated.

Table 4-41 Payload Common to Incidents and Problems

Attribute Datatype Additional Information

ID

NUMBER(16)

ID of the incident or problem.

SEVERITY

VARCHAR2(128)

Issue Severity. It is the translated.

SEVERITY_CODE

VARCHAR2(32)

Issue Severity Code.The possible values are defined in descending order of severity:

GC$EVENT.FATAL

GC$EVENT.CRITICAL

GC$EVENT.WARNING

GC$EVENT.MINOR_WARNING

GC$EVENT.INFORMATIONAL

GC$EVENT.CLEAR

PRIORITY

VARCHAR2(128)

Issue Priority. It is the translated priority name.

PRIORITY_CODE

VARCHAR2(32)

Issue Priority. It is the internal value defined in EM. The possible values are defined in descending order of priority:

GC$EVENT.PRIORITY_URGENT GC$EVENT.PRIORITY_VERY_HIGH

GC$EVENT.PRIORITY_HIGH GC$EVENT.PRIORITY_MEDIUM

GC$EVENT.PRIORITY_LOW

GC$EVENT.PRIORITY_NONE

STATUS

VARCHAR2(32)

Status of Issue. The possible values are

GC$EVENT.STATUS_NEW

GC$EVENT.STATUS_CLOSED

Any other user defined status.

ESCALATION_LEVEL

NUMBER(1)

Escalation level of the issue, has a value between 0 to 5.

OWNER

VARCHAR(256)

Issue Owner. Set to NULL if no owner exists.

ACKNOWLEDGED_BY_OWNER

NUMBER(1)

Set to 1, if this issue was acknowledged by owner.

CREATION_DATE

DATE

Issue creation date.

CLOSED_DATE

DATE

Issue closed date, null if not closed.

CATEGORIES

gc$category_string_array

List of categories that the event belongs to. Category is translated based on locale defined in OMS server. Notification system sends up to 10 categories.

CATEGORY_CODES

gc$category_string_array

Codes for the categories. Notification system sends up to 10 category codes.

SOURCE_INFO_ARR

gc$notif_source_info_array

Array of source information associated with this issue. See $gcnotif_source_info type definition for detail.

LAST_MODIFIED_BY

VARCHAR2(256)

Last modified by user.

LAST_UPDATED_DATE

DATE

Last updated date.


gc$notif_issue_summary (common to incident and problem payloads)

Represents the associated incident summary in the event payload, or associated problem summary in the incident payload, respectively.

Table 4-42 Payload

Attribute Datatype Additional Information

ID

NUMBER

Issue Id, either Incident Id or Problem Id.

SEVERITY

VARCHAR(128)

The severity level of an issue. It is translated severity name.

SEVERITY_CODE

VARCHAR2(32)

Issue Severity Code, has one of the following values.

GC$EVENT.FATAL

GC$EVENT.CRITICAL

GC$EVENT.WARNING

GC$EVENT.MINOR_WARNING

GC$EVENT.INFORMATIONAL

GC$EVENT.CLEAR

PRIORITY

VARCHAR2(128)

Current priority. It is the translated priority name.

PRIORITY_CODE

VARCHAR2(32)

Issue priority code, has one of the following values. GC$EVENT.PRIORITY_URGENT

GC$EVENT.PRIORITY_VERY_HIGH

GC$EVENT.PRIORITY_HIGH

GC$EVENT.PRIORITY_MEDIUM

GC$EVENT.PRIORITY_LOW

GC$EVENT.PRIORITY_NONE

STATUS

VARCHAR2(64)

Status of issue. The possible values are

GC$EVENT.STATUS_NEW

GC$EVENT.STATUS_CLOSED

GC$EVENT.WIP (work in progress)

GC$EVENT.RESOLVED

any other user defined status

ESCALATION_LEVEL

VARCHAR2(2)

Issue escalation level range from 0 to 5, default 0.

OWNER

VARCHAR2(256)

Issue Owner. Set to NULL if no owner exists.

ACKNOWLEDGED_BY_OWNER

NUMBER(1)

Set to 1, if this issue was acknowledged by owner.


gc$category_string_array

gc$category_string_array is an array of string containing the categories which event, incident or problem is associated with. The notification system delivers up to 10 categories.

gc$notif_event_context_array

gc$notif_event_context_array provides information about the additional diagnostic data that was captured at event detection time. Note that notification system delivers up to 200 elements from the captured event context. Each element of this array is of the type gc$notif_event_context.

gc$notif_event_context: This object represents the detail of event context data which is additional contextual information captured by the source system at the time of event generation that may have diagnostic value. The context for an event should consist of a set of keys and values along with data type (Number or String only).

Table 4-43 Event Context Type

Attribute Datatype Additional Information

NAME

VARCHAR2(256)

The event context name.

TYPE

NUMBER(1)

The data type of the value, which is stored

(0) - for numeric data

(1) - for string data.

VALUE

NUMBER

The numerical value.

STRING_VALUE

VARCHAR2(4000)

The string value.


gc$notif_corrective_action_job

Provides information about the execution of a corrective action job. Note that the corrective actions are supported for metric alert and target availability events only.

Table 4-44 Corrective Action Job-Specific Attributes

Attribute Datatype Additional Information

JOB_GUID

RAW(16)

Corrective Action Job global unique identifier.

JOB_NAME

VARCHAR2(128)

The value will be the name of the Corrective Action. It applies to Metric Alert and Target Availability Events.

JOB_OWNER

VARCHAR2(256)

Corrective action job owner.

JOB_TYPE

VARCHAR2(256)

Corrective action job type.

JOB_STATUS

VARCHAR2(64)

Corrective action job execution status.

JOB_STATUS_CODE

NUMBER

Corrective action job execution status code. It is the internal value defined in Enterprise Manager. For more information on status codes, see Table 4-14, "Corrective Action Status Codes".

JOB_STEP_OUTPUT

VARCHAR2(4000)

The value will be the text output from the Corrective Action execution. This will be truncated to last 4000 characters.

JOB_EXECUTION_GUID

RAW(16)

Corrective Action Job execution global unique identifier.

JOB_STATE_CHANGE_GUID

RAW(16)

Corrective Action Job change global unique identifier.

OCCURRED_DATE

DATE

Corrective action job occurred date.


gc$notif_source_info_array

Provides access to the multiple sources to which an incident or a problem could be related. NOTE: The notification system delivers up to 200 sources associated with an incident or a problem.

CREATE OR REPLACE TYPE gc$notif_source_info_array AS VARRAY(200) OF gc$notif_source_info;

gc$notif_source_info

Notification source information which is used for referencing source information containing either target or source, or both.

Table 4-45 Source Information Type

Attribute Datatype Additional Information

TARGET

gc$notif_target

It is populated when the event is related to a target. See gc$notif_target type definition for detail.

SOURCE

gc$notif_source

It is populated when the event is related to a (non-target) source. See gc$notif_source type definition for detail.


gc$notif_source

Used for referencing source objects other than a job target.

Table 4-46 Payload

Attribute Datatype Additional Information

SOURCE_GUID

RAW(16)

Source's global unique identifier.

SOURCE_TYPE

VARCHAR2(120)

Type of the Source object, e.g., TARGET, JOB, TEMPLATE, etc.

SOURCE_NAME

VARCHAR2(256)

Source Object Name.

SOURCE_OWNER

VARCHAR2(256)

Owner of the Source object.

SOURCE_SUB_TYPE

VARCHAR2(256)

Sub-type of the Source object, for example, within the TARGET these would be the target types like Host, Database etc.

SOURCE_URL

VARCHAR2(4000)

Source's event console URL.


gc$notif_target

Target information object is used for providing target information.

Table 4-47 Target Information

Attribute Datatype Additional Information

TARGET_GUID

RAW(16)

Target's global unique identifier.

TARGET_NAME

VARCHAR2(256)

Name of target.

TARGET_OWNER

VARCHAR2(256)

Owner of target.

TARGET_LIFECYCLE_STATUS

VARCHAR2(1024)

Life Cycle Status of the target.

TARGET_VERSION

VARCHAR2(64)

Target Version of the target.

TARGET_TYPE

VARCHAR2(128)

Type of a target.

TARGET_TIMEZONE

VARCHAR2(64)

Target's regional time zone.

HOST_NAME

VARCHAR2(256)

The name of the host on which the target is deployed upon.

TARGET_URL

VARCHAR2(4000)

Target's EM Console URL.

UDTP_ARRAY

gc$notif_udtp_array

The list of user defined target properties. It is populated for events that are associated with a target. It is populated for incidents and problems, when they are associated with a single source (gc$notif_source_info).


gc$notif_udtp_array

Array of gc$notif_udtp type with a maximum size of 20.

CREATE OR REPLACE TYPE gc$notif_udtp_array AS VARRAY(20) OF gc$notif_udtp;

gc$notif_udtp

Used for referencing User-defined target properties. UDTP should consist of a set of property key names and property values.

Table 4-48 Payload

Attribute Datatype Additional Information

NAME

VARCHAR2(64),

The name of property.

VALUE

VARCHAR2(1024)

Property value.

LABEL

VARCHAR(256)

Property label.

NLS_ID

VARCHAR(64)

Property nls id


4.10.3.1 Notification Payload Elements Specific to Event Types

gc$notif_event_attr_array

Array of gc$notif_event_attr is used for referencing event-specific attributes. The array has a maximum size of 25. Each element of the array is of type gc$notif_event_attr (used for referencing event type-specific attributes).

Table 4-49 Event Attribute Type

Attribute Datatype Additional Information

NAME

VARCHAR2(64)

The internal name of event type specific attribute.

VALUE

VARCHAR2(4000)

value.

NLS_VALUE

VARCHAR2(4000)

Translated value for the attribute.


You can use SQL queries to list the deployed event types in your deployment and the payload specific to each. The following SQL can be used to list all internal event type names which are registered in the Enterprise Manager.

Select event_class as event_type, upper(name) as env_var_name
from em_event_class_attrs
where notif_order != 0
and event_class is not null
order by event_type, env_var_name;

You should convert the attribute name to upper case before using the name for comparison.

There is an attribute variable payload specific to each event type that can be accessed from a gc$notif_event_attr_array database type. The following tables list notification attributes for the most critical event types. You should convert the attribute name to uppercase before using the name for comparison.

Table 4-50 Environment variables specific to Metric Alert Event Type

Environment Variable Description

COLL_NAME

The name of the collection collecting the metric.

KEY_COLUMN_X

Internal name of Key Column X where X is a number between 1 and 7.

KEY_COLUMN_X_VALUE

Value of Key Column X where X is a number between 1 and 7.

KEY_VALUE

Monitored object for the metric corresponding to the Metric Alert event.

METRIC_COLUMN

The name of the metric column

METRIC_DESCRIPTION

Brief description of the metric.

METRIC_GROUP

The name of the metric.

NUM_KEYS

The number of key metric columns in the metric.

SEVERITY_GUID

The GUID of the severity record associated with this metric alert.

VALUE

Value of the metric when the event triggered.


Table 4-51 Environment variables specific to Target Availability Event Type

Environment Variable Description

AVAIL_SEVERITY

The transition severity (0-6) that resulted in the status of the target to change to the current availability status.

Possible Values for AVAIL_SEVERITY

  • 0 (Target Down)

  • 1 (Target Up)

  • 2 (Target Status Error)

  • 3 (Agent Down)

  • 4 (Target Unreachable)

  • 5 (Target Blackout)

  • 6 (Target Status Unknown)

AVAIL_SUB_STATE

The substatus of a target for the current status.

CYCLE_GUID

A unique identifier for a metric alert cycle, which starts from the time the metric alert is initially generated until the time it is clear.

METRIC_GUID

Metric GUID of response metric.

SEVERITY_GUID

The GUID of the severity record associated with this availability status.

TARGET_STATUS

The current availability status of the target.


Table 4-52 Environment variables specific to Job Status Change event type

Environment Variable Description

EXECUTION_ID

Unique ID of the job execution..

EXECUTION_LOG

The job output of the last step executed.

EXECUTION_STATUS

The internal status of the job execution.

EXEC_STATUS_CODE

Execution status code of job execution. For possible values, see Table 4-16, "Job Status Codes".

STATE_CHANGE_GUID

Unique ID of last status change


Example 4-17 PL/SQL Script: Event Type Payload Elements

-- log_table table is created by following DDL to demostrate how to access 
-- event notification payload GC$NOTIF_EVENT_MSG.

CREATE TABLE log_table (message VARCHAR2(4000)) ;

-- Define PL/SQL notification method for Events 
CREATE OR REPLACE PROCEDURE log_table_notif_proc(s IN GC$NOTIF_EVENT_MSG)
IS
  l_categories gc$category_string_array;
  l_category_codes gc$category_string_array;
  l_attrs gc$notif_event_attr_array;
  l_ca_obj gc$notif_corrective_action_job;
BEGIN
  INSERT INTO log_table VALUES ('notification_type: ' || s.msg_info.notification_type);
  INSERT INTO log_table VALUES ('repeat_count: ' || s.msg_info.repeat_count);
  INSERT INTO log_table VALUES ('ruleset_name: ' || s.msg_info.ruleset_name);
  INSERT INTO log_table VALUES ('rule_name: ' || s.msg_info.rule_name);
  INSERT INTO log_table VALUES ('rule_owner: ' || s.msg_info.rule_owner);
  INSERT INTO log_table VALUES ('message: ' || s.msg_info.message);
  INSERT INTO log_table VALUES ('message_url: ' || s.msg_info.message_url);
  INSERT INTO log_table VALUES ('event_instance_guid: ' || s.event_payload.event_instance_guid);
  INSERT INTO log_table VALUES ('event_type: ' || s.event_payload.event_type);
  INSERT INTO log_table VALUES ('event_name: ' || s.event_payload.event_name);
  INSERT INTO log_table VALUES ('event_msg: ' || s.event_payload.event_msg);
  INSERT INTO log_table VALUES ('source_obj_type: ' || s.event_payload.source.source_type);
  INSERT INTO log_table VALUES ('source_obj_name: ' || s.event_payload.source.source_name);
  INSERT INTO log_table VALUES ('source_obj_url: ' || s.event_payload.source.source_url);
  INSERT INTO log_table VALUES ('target_name: ' || s.event_payload.target.target_name);
  INSERT INTO log_table VALUES ('target_url: ' || s.event_payload.target.target_url);
  INSERT INTO log_table VALUES ('severity: ' || s.event_payload.severity);
  INSERT INTO log_table VALUES ('severity_code: ' || s.event_payload.severity_code);
  INSERT INTO log_table VALUES ('event_reported_date: ' || to_char(s.event_payload.reported_date, 'D MON DD HH24:MI:SS'));

  IF s.event_payload.target.TARGET_LIFECYCLE_STATUS IS NOT NULL
  THEN
    INSERT INTO log_table VALUES ('target lifecycle_status: ' || s.event_payload.target.TARGET_LIFECYCLE_STATUS);
  END IF;

  -- Following block illustrates the list of category codes to which the event
  -- belongs.

  l_category_codes := s.event_payload.category_codes;
  IF l_categories IS NOT NULL 
  THEN
    FOR c IN 1..l_category_codes.COUNT
    LOOP
      INSERT INTO log_table VALUES ('category_code ' || c || ' - ' || l_category_codes(c));
    END LOOP;
  END IF;

  --
  -- Each event type has a specific set of attributes modeled. Examples of
  -- event types include metric_alert, target_availability, job_status_change.
  -- Following block illustrates how to access the attributes for job_status change
  -- event type
  --
  IF s.event_payload.event_type = 'job_staus_chage'
  THEN
    l_attrs := s.event_payload.event_attrs;
    IF l_attrs IS NOT NULL
    THEN
      FOR c IN 1..l_attrs.COUNT
      LOOP
        INSERT INTO log_table VALUES ('EV.ATTR name=' || l_attrs(c).name || ' value=' || l_attrs(c).value || ' nls_value=' || l_attrs(c).nls_value);
      END LOOP;
    END IF;
  END IF;

  -- Following block illustrates how to access corrective action job's attributes  IF s.msg_info.notification_type = GC$NOTIFICATION.NOTIF_CA AND s.event_payload.corrective_action IS NOT NULL
  THEN
    l_ca_obj := s.event_payload.corrective_action;
    INSERT INTO log_table VALUES ('CA JOB_GUID: ' || l_ca_obj.JOB_GUID);
    INSERT INTO log_table VALUES ('CA JOB_NAME: ' || l_ca_obj.JOB_NAME);
    INSERT INTO log_table VALUES ('CA JOB_OWNER: ' || l_ca_obj.JOB_OWNER);
    INSERT INTO log_table VALUES ('CA JOB_TYPE: ' || l_ca_obj.JOB_TYPE);
    INSERT INTO log_table VALUES ('CA JOB_STATUS: ' || l_ca_obj.JOB_STATUS);
    INSERT INTO log_table VALUES ('CA JOB_STATUS_CODE: ' || l_ca_obj.JOB_STATUS_CODE);
    INSERT INTO log_table VALUES ('CA JOB_STEP_OUTPUT: ' || l_ca_obj.JOB_STEP_OUTPUT);
    INSERT INTO log_table VALUES ('CA JOB_EXECUTION_GUID: ' || l_ca_obj.JOB_EXECUTION_GUID);
    INSERT INTO log_table VALUES ('CA JOB_STATE_CHANGE_GUID: ' || l_ca_obj.JOB_STATE_CHANGE_GUID);
    INSERT INTO log_table VALUES ('CA OCCURRED_DATE: ' || l_ca_obj.OCCURRED_DATE);  END IF;

COMMIT ;
END ;
/

4.10.4 Troubleshooting Notifications

To function properly, the notification system relies on various components of Enterprise Manager and your IT infrastructure. For this reason, there can be many causes of notification failure. The following guidelines and suggestions can help you isolate potential problems with the notification system.

4.10.4.1 General Setup

The first step in diagnosing notification issues is to ensure that you have properly configured and defined your notification environment.

OS Command, PL/SQL and SNMP Trap Notifications

Make sure all OS Command, PLSQL and SNMP Trap Notification Methods are valid by clicking the Test button. This will send a test notification and show any problems the OMS has in contacting the method. Make sure that your method was called, for example, if the OS Command notification is supposed to write information to a log file, check that it has written information to its log file.

E-mail Notifications

  • Make sure an e-mail gateway is set up under the Notification Methods page of Setup. The Sender's e-mail address should be valid. Clicking the Test button will send an e-mail to the Sender's e-mail address. Make sure this e-mail is received. Note that the Test button ignores any Notification Schedule.

  • Make sure an e-mail address is set up. Clicking the Test button will send an e-mail to specified address and you should make sure this e-mail is received. Note that the Test button ignores any Notification Schedule.

  • Make sure an e-mail schedule is defined. No e-mails will be sent unless a Notification Schedule has been defined.

  • Make sure a incident rule is defined that matches the states you are interested and make sure e-mail and notification methods are assigned to the rule.

4.10.4.2 Notification System Errors

For any alerts involving problems with notifications, check the following for notification errors.

  • Any serious errors in the Notification System are logged as system errors in the MGMT_SYSTEM_ERROR_LOG table. From the Setup menu, select Management Services and Repository to view these errors.

  • Check for any delivery errors. You can view them from Incident Manager. From the Enterprise menu, select Monitoring, then select Incident Manager. The details will give the reason why the notification was not delivered.

4.10.4.3 Notification System Trace Messages

The Notification System can produce trace messages in sysman/log/emoms.trc file.

Tracing is configured by setting the log4j.category.oracle.sysman.em.notification property flag using the emctl set property command. You can set the trace level to INFO, WARN, DEBUG. For example,

emctl set property -name log4j.category.oracle.sysman.em.notification -value
DEBUG -module logging

Note: The system will prompt you for the SYSMAN password.

Trace messages contain the string "em.notification". If you are working in a UNIX environment, you can search for messages in the emoms.trc and emoms_pbs.trc files using the grep command. For example,

grep em.notification emoms.trc emoms_pbs.trc

What to look for in the trace file.

The following entries in the emoms.trc file are relevant to notifications.

Normal Startup Messages

When the OMS starts, you should see these types of messages.

2011-08-17 13:50:29,458 [EventInitializer] INFO  em.notification init.167 - Short format maximum length is 155
2011-08-17 13:50:29,460 [EventInitializer] INFO  em.notification init.185 - Short format is set to both subject and body
2011-08-17 13:50:29,460 [EventInitializer] INFO  em.notification init.194 - Content-Transfer-Encoding is 8-bit
2011-08-17 13:50:29,460 [EventInitializer] DEBUG em.notification registerAdminMsgCallBack.272 - Registering notification system message call back
2011-08-17 13:50:29,461 [EventInitializer] DEBUG em.notification registerAdminMsgCallBack.276 - Notification system message callback is registered successfully
2011-08-17 13:50:29,713 [EventInitializer] DEBUG em.notification upgradeEmailTemplates.2629 - Enter upgradeEmailTemplates
2011-08-17 13:50:29,735 [EventInitializer] INFO  em.notification upgradeEmailTemplates.2687 - Email template upgrade is not required since no customized templates exist.
2011-08-17 13:49:28,739 [EventCoordinator] INFO  events.EventCoordinator logp.251 - Creating event worker thread pool: min = 4 max = 15
2011-08-17 13:49:28,791 [[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] INFO emdrep.pingHBRecorder initReversePingThreadPool.937 - Creating thread pool for reverse ping : min = 10 max = 50
2011-08-17 13:49:28,797 [[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG emdrep.HostPingCoordinator logp.251 - Creating thread pool of worker thread  for host ping: min = 1 max = 10
2011-08-17 13:49:28,799 [[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] DEBUG emdrep.HostPingCoordinator logp.251 - Creating thread pool for output of worker's  output for host ping: min = 2 max = 20
2011-08-17 13:49:30,327 [ConnectorCoordinator] INFO  connector.ConnectorPoolManager logp.251 - Creating Event thread pool: min = 3 max = 10
2011-08-17 13:51:48,152 [NotificationMgrThread] INFO  notification.pbs logp.251 - Creating thread pool: min = 6 max = 24
2011-08-17 13:51:48,152 [NotificationMgrThread] INFO  em.rca logp.251 - Creating RCA thread pool: min = 3 max = 20

Notification Delivery Messages

2006-11-08 03:18:45,387 [NotificationMgrThread] INFO  em.notification run.682 - Notification ready on EMAIL1

2006-11-08 03:18:46,006 [DeliveryThread-EMAIL1] INFO  em.notification run.114 - Deliver to SYSMAN/admin@oracle.com

2006-11-08 03:18:47,006 [DeliveryThread-EMAIL1] INFO  em.notification run.227 - Notification handled for SYSMAN/admin@oracle.com

Notification System Error Messages

2011-08-17 14:02:23,905 [NotificationMgrThread] DEBUG notification.pbs logp.251 - Notification ready on EMAIL1
2011-08-17 14:02:23,911 [NotificationMgrThread] DEBUG notification.pbs logp.251 - Notification ready on PLSQL4
2011-08-17 14:02:23,915 [NotificationMgrThread] DEBUG notification.pbs logp.251 - Notification ready on OSCMD14
2011-08-17 14:02:19,057 [DeliveryThread-EMAIL1] INFO  notification.pbs logp.251 - Deliver to To: my.admin@oracle.com; issue type: 1; notification type: 1
2011-08-17 14:02:19,120 [DeliveryThread-OSCMD14] INFO  notification.pbs logp.251 - Deliver to SYSMAN, OSCMD, 8; issue type: 1; notification type: 1
2011-08-17 14:02:19,346 [DeliveryThread-PLSQL4] INFO  notification.pbs logp.251 - Deliver to SYSMAN, LOG_JOB_STATUS_CHANGE, 9; issue type: 1; notification type: 1
2011-08-17 14:02:19,977 [DeliveryThread-PLSQL4] DEBUG notification.pbs logp.251 - Notification handled for SYSMAN, LOG_JOB_STATUS_CHANGE, 9
2011-08-17 14:02:20,464 [DeliveryThread-EMAIL1] DEBUG notification.pbs logp.251 - Notification handled for To: my.admin@oracle.com
2011-08-17 14:02:20,921 [DeliveryThread-OSCMD14] DEBUG notification.pbs logp.251 - Notification handled for SYSMAN, OSCMD, 8

4.10.4.4 E-mail Errors

The SMTP gateway is not set up correctly:

Failed to send e-mail to my.admin@oracle.com: For e-mail notifications to be sent, your Super Administrator must configure an Outgoing Mail (SMTP) Server within Enterprise Manager. (SYSMAN, myrule)

Invalid host name:

Failed to connect to gateway: badhost.oracle.com: Sending failed;
nested exception is:
javax.mail.MessagingException: Unknown SMTP     host: badhost.example.com;

Invalid e-mail address:

Failed to connect to gateway: rgmemeasmtp.oraclecorp.com: Sending failed;
nested exception is:
javax.mail.MessagingException: 550 5.7.1        <smpemailtest_ie@example.com>... Access denied

Always use the Test button to make sure the e-mail gateway configuration is valid. Check that an e-mail is received at the sender's e-mail address

4.10.4.5 OS Command Errors

When attempting to execute an OS command or script, the following errors may occur. Use the Test button to make sure OS Command configuration is valid. If there are any errors, they will appear in the console.

Invalid path or no read permissions on file:

Could not find /bin/myscript (machineb10.oracle.com_Management_Service) (SYSMAN, myrule )

No execute permission on executable:

Error calling /bin/myscript: java.io.IOException: /bin/myscript: cannot execute (machineb10.oracle.com_Management_Service) (SYSMAN, myrule ) 

Timeout because OS Command ran too long:

Timeout occurred running /bin/myscript (machineb10.oracle.com_Management_Service) (SYSMAN, myrule )

Any errors such as out of memory or too many processes running on OMS machine will be logged as appropriate.

Always use the Test button to make sure OS Command configuration is valid.

4.10.4.6 SNMP Trap Errors

Use the Test button to make sure SNMP Trap configuration is valid.

The OMS will not report an error if the SNMP trap cannot reach the third party SNMP console as this is sent via UDP. If the SNMP trap encounters problems when trying to reach the third party SNMP console, possible SNMP trap problems include: invalid host name, port, community for a machine running an SNMP Console or a network issue such as a firewall problem.

Other possible SNMP trap problems include: invalid host name, port, or community for a machine running an SNMP Console.

4.10.4.7 PL/SQL Errors

When attempting to execute an PL/SQL procedure, the following errors may occur. Use the Test button to make sure the procedure is valid. If there are any errors, they will appear in the console.

Procedure name is invalid or is not fully qualified. Example: SCOTT.PKG.PROC

Error calling PL/SQL procedure plsql_proc: ORA-06576: not a valid function or procedure name (SYSMAN, myrule) 

Procedure is not the correct signature. Example: PROCEDURE event_proc(s IN GC$NOTIF_EVENT_MSG)

Error calling PL/SQL procedure plsql_proc: ORA-06553: PLS-306: wrong number or types of arguments in call to 'PLSQL_PROC' (SYSMAN, myrule)

Procedure has bug and is raising an exception.

Error calling PL/SQL procedure plsql_proc: ORA-06531: Reference to uninitialized collection (SYSMAN, myrule)

Care should be taken to avoid leaking cursors in your PL/SQL. Any exception due to this condition will result in delivery failure with the message being displayed in the Details section of the alert in the Cloud Control console.

Always use the Test button to make sure the PL/SQL configuration is valid.