PK OXEoa,mimetypeapplication/epub+zipPKOXEiTunesMetadata.plistW artistName Oracle Corporation book-info cover-image-hash 92961928 cover-image-path OEBPS/dcommon/oracle-logo.jpg package-file-hash 311863924 publisher-unique-id E16790-04 unique-id 242384224 genre Oracle Documentation itemName Oracle® Enterprise Manager Administrator's Guide, 11g Release 1 (11.1.0.1) releaseDate 2013-03-12T04:16:26Z year 2013 PK\WPKOXEMETA-INF/container.xml PKYuPKOXEOEBPS/extending.htm Extending Enterprise Manager

15 Extending Enterprise Manager

Enterprise environments consist of a wide variety of components: OS platforms, hardware, software, network, and storage devices. All of these components work in concert to deliver critical information and functionality required to keep enterprise operations performing optimally and providing information to make important business decisions. While Oracle Enterprise Manager Grid Control allows you to monitor and manage a variety of components out-of-box, you may want to monitor third party components or custom applications specific to your environment. For example, with this release, you can seamlessly monitor WebLogic and WebSphere application servers. Additional plug-ins are being developed and will be announced as they become available.

In addition, you can use the same mechanism used by Oracle and partners to extend Enterprise Manager to monitor custom components via modular Management Plug-ins. Once a plug-in is defined, you use the Enterprise Manager Grid Control console to deploy the new plug-in throughout your enterprise environment.

Benefits of Extending Enterprise Manager

Extending Enterprise Manager for monitoring additional components provides the following benefits:

More Extensibility Information

Complete Management Plug-in development information can be found in the Oracle Enterprise Manager Extensibility Guide.

The latest information on all aspects of Oracle Enterprise Manager extensibility can be found at the Oracle Enterprise Manager Grid Control Extensions Exchange located on the Oracle Technology Network web site.

http://www.oracle.com/technology/products/oem/extensions/index.html

PKS PKOXEOEBPS/group_management.htmu@ Group Management

5 Group Management

This chapter introduces the concept of group management and contains the following sections:

Introduction to Groups

Today's IT operations can be responsible for managing a great number of components, such as databases, application servers, hosts, or other components, which can be time consuming and impossible to manage individually. The Enterprise Manager Grid Control group management system lets you combine components (called targets in Enterprise Manager) into logical sets, called groups. This enables you to organize, manage, and effectively monitor the potentially large number of targets in your enterprise.

Enterprise Manager Groups can include:


Note:

An Enterprise Manager "System," used specifically to group the components on which a service runs, is a special kind of Enterprise Manager group. Many of the functions and capabilities for groups and systems are similar.

Typically you can gather together targets that you want to manage as a group. If you use the target properties (for example, Line of Business or Deployment Type) to put operational information about your targets in Enterprise Manager, you can use these properties when creating groups to locate targets. For example, you could search for all databases of Deployment Type = Production and belonging to Line of Business 'HCM'. You can also create a group hierarchy and use nested groups.

Managing Groups

By combining targets in a group, Enterprise Manager offers a wealth of management features that enable you to efficiently manage these targets as one group. Using the Group pages, you can:

You can also customize the console to provide direct access to group management pages.

Group Home Page

The Group Home page, shown in Figure 5–1, is the central location for monitoring information. The Group Home page provides the following features:

  • Availability pie chart that provides at-a-glance information on the current status across all members so you can easily assess the percentage of members that are up, and the percentage of members that are unavailable. You can quickly drill down for information if any member target is down.

  • Roll-up of open alerts and policy violations, categorized by severity, so you can quickly focus on the most critical problems first. Alerts and violations that have occurred within the last 24 hours highlight problems that recently occurred.

  • Access to the Policy Trend Overview page, which provides a comprehensive view of the group for compliance with policy rules over a period of time. Using policy charts, you can assess trends such as increased or decreased number of policy violations, changes in the overall group compliance score, and the percentage of members in compliance with your enterprise's policy rules.

  • Access to the Security at a Glance page, which provides an overview of the security health of the group. This shows statistics about security policy violations and critical security patches that have not been applied.

  • Summary of recent configuration changes across all members in the group, so you can easily determine if a new problem is related to any recent changes.

  • Summary of Critical Patch Advisories for Oracle homes within the group.

Figure 5-1 Group Home Page

This is the Enterprise Manager Group Home page

Figure 5–2 shows the Policy Trend Overview page that you can access from the Group Home page.

Figure 5-2 Policy Trend Overview Page

This is the Enterprise Manager Policy Trend Overview page.


See Also:

"Group Home Page" in the Enterprise Manager online help

Group Charts Page

The Group Charts page, shown in Figure 5–3, enables you to monitor the collective performance of the group. Out-of-box performance charts are provided based on the type of members in the group. For example, when databases are part of the group, a Wait Time (%) chart is provided that shows the top databases with the highest wait time percentage values. You can view this performance information over the last 24 hours, last 7 days, or last 31 days. You can also add your own custom charts to the page.

Figure 5-3 Group Charts Page

This is the Enterprise Manager Group Charts page.


See Also:

"Group Charts Page" in the Enterprise Manager online help

Group Administration Page

The Group Administration page, shown in Figure 5–4, provides a central point for performing common administrative tasks for the group. For example, you can:

  • Run jobs or find out the status of currently running jobs against the group.

  • Define planned outage windows, called blackouts, on the members of the group to perform maintenance tasks.

  • Run SQL commands collectively against the database member targets of the group.

  • View the most recent backup for each database in the group.

  • View the last 100,000 bytes of the alert log for all databases in the group.

  • Use a deployment summary to easily obtain hardware and software inventory information across all member targets.

Figure 5-4 Group Administration Page

This is the Enterprise Manager Group Administration page.


See Also:

"Group Administration Page" in the Enterprise Manager online help

Group Members Page

The Group Members page, shown in Figure 5–5, summarizes information about the member targets in the group. It includes information on their current availability status, roll-up of open alerts and policy violations, and key performance metrics based on the type of targets in the group.

You can visually assess availability and relative performance across all member targets. You can sort on any of the columns to rank members by a certain criterion (for example, database targets in order of decreasing wait time percentage). Default key performance metrics are displayed based on the targets you select, but you can customize these to include additional metrics that are important for managing your group.

Figure 5-5 Group Members Page

This is the Enterprise Manager Group Members page.


See Also:

"Group Members Page" in the Enterprise Manager online help

System Dashboard

The System Dashboard, shown in Figure 5–6, enables you to proactively monitor the status and alerts in the group as they occur. The color-coded interface is designed to highlight problem areas using the universal colors of alarm—targets that are down are highlighted in red, metrics in critical severity are shown as red dots, metrics in warning severity are shown as yellow dots, and metrics operating within normal boundary conditions are shown as green dots.

Using these colors, you can easily spot the problem areas for any target and drill down for details as needed. An alert table is also included to provide a summary for all open alerts in the group. The alerts in the table are presented in reverse chronological order to show the most recent alerts first, but you can also click on any column in the table to change the sort order.You can customize the dashboard according to your needs. You can specify which key metrics should be included in the dashboard and the display names to be used for these metrics. You can also customize the refresh interval to ensure that you always receive timely information about alerts as they are detected.You can launch the System Dashboard in context from any Group Home page. However, using reporting framework features, you can also make the System Dashboard publicly available for any user that has access to a web browser and the Enterprise Manager Reports Web site.

Figure 5-6 System Dashboard

This is the Enterprise Manager System Dashboard.

Out-of-Box Reports

Enterprise Manager provides several out-of-box reports for groups as part of the reporting framework, called Information Publisher. These reports display important administrative information, such as hardware and operating system summaries across all hosts within a group, and monitoring information, such as outstanding alerts and policy violations for a group.

You can access these reports from the Reports link in the Related Links section of all Group pages. Figure 5–7 shows the Availability History report for a specified group over the last 31 days.

Figure 5-7 Availability History Report

This is the Enterprise Manager Availability History page.

Redundancy Groups

A redundancy group is a group that contains members of the same type that function collectively as a unit. A type of redundancy group functions like a single logical target that supports a status (availability) metric. A redundancy group is considered up (available) if at least one of the member targets is up.

You can create and administer a redundancy group from the All Targets page. Redundancy groups support all group management features previously discussed.

When you define the Redundancy Group, you must choose the member type for the members in the Redundancy Group.

You can define the options for how availability of the redundancy group is calculated by selecting either Number or Percentage:

Figure 5-8 shows the Create Redundancy Group page while defining the group using Percentage availability.

Figure 5-8 Redundancy Group Example

Redundancy Group

Do not use redundancy groups if the group you want to model is an Oracle Real Application Clusters database, host cluster, HTTP server high availability group, or OC4J high availability group. Instead, you can use the following specialized target types for this purpose:

Privilege Propagating Groups

Privilege propagating groups enable administrators to grant privileges to other administrators in a manner in which new administrators get the same privileges as its member targets. For example, granting operator privilege on a group to an Administrator grants him the operator privilege on its member targets and also to any members that will be added in the future. Privilege propagating groups can contain individual targets or other privilege propagating groups.

Privileges on the group can be granted to an Enterprise Manager user or a role. Use a role if the privileges you want to grant are to be granted to a group of EM users. See Figure 5-9, "Granting Privileges On a Group To a Role".

Figure 5-9 Granting Privileges On a Group To a Role

PPG Role

For example, suppose you create a large privilege propagating group and grant a privilege to a role which is then granted to administrators. If new targets are later added to the privilege propagating group, then the administrators receive the privileges on the target automatically. Additionally, when a new administrator is hired, you only need to grant the role to the administrator for the administrator to receive all the privileges on the targets automatically.

Creating Privilege Propagating Groups

The privilege propagating group creation function is a privileged activity. The privilege propagating group feature contains two new privileges:

  • Create Privilege Propagating Group

    This privileged activity allows the administrators to create the privilege propagating groups. Administrators with this privilege can create propagating groups and delegate the group administration activity to other users.

  • Group Administration

    This privilege can be granted to administrators on specific group targets and is used to delegate the group administration activities to other administrators. It is granted to both conventional and privilege propagating groups.

Using the Group Administration Privilege

The Group Administration Privilege is available for both Privilege Propagating Groups and conventional groups. If you are granted this privilege, you can grant access to the group to other Enterprise Manager users without having to be the SuperAdministrator to grant the privilege.

Adding Members to Privilege Propagating Groups

The target privileges granted on a propagating group are propagated to member targets. The administrator grants target objects scoped to another administrator, the grantee maintains the same privileges on member targets. The propagating groups maintain the following features:

  • The administrator with a Create Privilege Propagating Group privilege will be able to create a propagating group

  • To add a target as a member of a propagating group, the administrator must have Full target privileges on the target

You can add any non-aggregated target as the member of a privilege propagating group. For aggregated targets in Grid Control version 10.2.0.5, cluster and RAC databases and other propagating groups can be added as members (cluster and RAC databases must be added via the emcli verb). There is no support for this through the Enterprise Manager interface in version 10.2.0.5. Grid Control version 11.1, however, supports more aggregated target types, such as redundancy groups, systems and services. These, along with cluster and RAC databases, can be added in version 11.1 via the Grid Control Console.

If you are not the group creator, you must have at least the Full target privilege on the group to add a target to the group.

Converting Conventional Groups to Privilege Propagating Groups

In Enterprise Manager version 11.1, you can convert conventional groups to privilege propagating groups (and vice-versa) through the use of the specified EMCLI verb. Two new parameters have been added in the modify_group EMCLI verb:

  • privilege_propagation

    This parameter is used to modify the privilege propagation behavior of the group. The possible value of this parameter is either true or false.

  • drop_existing_grants

    This parameter indicates whether existing privilege grants on that group are to be revoked at the time of converting a group from privilege propagation to normal (or vice versa). The possible values of this parameter are yes or no. The default value of this parameter is yes.

These same enhancements have been implemented on the following EMCLI verbs: modify_system, modify_redundancy_group, and modify_aggregrate_service.

The EMCLI verb is listed below:

emcli modify_group
   -name="name"
   [-type=<group>]
   [-add_targets="name1:type1;name2:type2;..."]...
   [-delete_targets="name1:type1;name2:type2;..."]...
   [-privilege_propagation = true/false]
   [-drop_existing_grants = Yes/No]  

For more information about this verb and other EMCLI verbs, see the EMCLI Reference Manual.

PK{uuPKOXE OEBPS/udm.htm User-Defined Metrics

4 User-Defined Metrics

User-defined metrics allow you to extend the reach of Enterprise Manager's monitoring to conditions specific to particular environments via custom scripts or SQL queries and function calls. Once defined, user-defined metrics will be monitored, aggregated in the repository and trigger alerts like regular metrics.

This chapter covers the following topics:

Extending Monitoring Capability

There are two types of user-defined metrics:

Once a user-defined metric is created, all other monitoring features, such as alerts, notifications, historical collections, and corrective actions are automatically available to it.

Administrators who already have their own library of custom monitoring scripts can leverage these monitoring features by integrating their scripts with Enterprise Manager via user-defined metrics. Likewise, existing SQL queries or function calls currently used to monitor database conditions can be easily integrated into Enterprise Manager's monitoring framework using the SQL-based user-defined metric.

Creating OS-Based User-Defined Metrics

Creating an OS-based user-defined metric involves two steps:

Create Your OS Monitoring Script

Using a scripting language of your choice, create a script that contains logic to check for the condition being monitored. For example, scripts that check for disk space or memory usage. All scripts to be run with user-defined metrics should be placed in a directory to which the Management Agent has full access privileges. Scripts themselves must have the requisite permissions set so that they can be executed by the Management Agent. The script runtime environment must also be configured: If your script requires an interpreter, such as a Perl interpreter, this must be installed on that host as well.

All monitoring scripts should contain code to perform the following basic functions:

Code to check the status of monitored objects

Define logic in the code that checks the condition being monitored such as determining the amount of free space on a particular file system or level of memory usage.

After checking the monitored condition, the script should return the value associated with the monitored object.

When you choose to have the script return a specific value from the monitored object (for example, current disk space usage), you can also have Enterprise Manager evaluate the object's current value against specific warning and critical thresholds. You specify these warning and critical thresholds from the Grid Control console at the time you create the user-defined metric. Based on the evaluation of the metric's value against the thresholds, an alert may be triggered at one of the following severity levels:

Table 4-1 Metric Severity Levels

Severity LevelStatus

Script Failure

The script failed to run properly.

Clear

No problems with the object monitored; status is clear. If thresholds were specified for the metric, then it means the thresholds were not reached.

Warning

The value of the monitored object reached the warning threshold.

Critical

The value of the monitored object reached the critical threshold.


Code to return script results to Enterprise Manager

After checking the monitored condition, the script should return the value associated with the monitored object. The script returns values back to Enterprise Manager by sending formatted information to standard output (stdout) using the syntax that is consistent with the scripting language (the "print" statement in Perl, for example). Enterprise Manager then checks the standard output of a script for this formatted information; specifically it checks for the tags: em_result and em_message and the values assigned to these tags.

The script must assign the value of the monitored object to the tag em_result. The output must be written as a string delimited by new line characters. For example, if the value of the monitored object is 200, your script can return this to Enterprise Manager as shown in this Perl statement:

print "em_result=200\n"

You can also have Enterprise Manager evaluate the returned value against specified warning and critical thresholds. You specify these warning and critical thresholds when you register your script as a user-defined metric in the console.

If the comparison between the warning or critical threshold holds true, a warning or critical alert will be generated. The default message for this alert will be:

"The value is $em_result".

You can choose to override this default message with a custom message by assigning the string to be used to the tag em_message.

For example, if you want your alert message to say 'Disk usage is high", your script can return this custom message as follows:

print "em_message=Disk usage is high\n"


Important:

Script output tags must be lower-case in order for Enterprise Manager to recognize the script output as valid user-defined metric feedback. Messages or values associated with each tag can be mixed case.
  • Valid tag output: em_result=My Value\n

  • Invalid tag output: Em_Result=My Value\n


For a successful script execution, the script output must start with the "em_result=" string in a new line. The message must start with the "em_message=" string in a new line.

The following table summarizes the script output tags.

Table 4-2 Script Output Information Tags

TagDefinition

em_result

Use this tag to return script result values. Exactly one em_result tag must be found in STDOUT. If more than one em_result tag is found, the first tag encountered will be used; subsequent em_result tags will be ignored.

Example:

print "em_result=200\n" 

Returns 200 as the value of the monitored object.

em_message

Use this tag to specify a message with the script result value in STDOUT. For OS-based user-defined metrics, only one em_message tag is permitted. If you submit more than one em_message tag, only the first tag is used. Subsequent tags are ignored.

Example:

print "em_result=200\nem_message=Disk usage is high\n" 

Returns 200 as the value of the monitored object in addition to the message "Disk usage is high".

If you want to include the value of em_result in the message, you can use the placeholder $em_result.

Example:

print "em_message=Disk usage is at $em_result.\n"

If script execution is successful AND it does not contain a em_message string, a default em_message string is automatically generated. The following message format is used:

em_message=The value is $em_result 

Example:

print "em_result=200\n"

Returns 200 as the value of the monitored object and the generated message "The value is 200"


The output of the user-defined monitoring script must be either em_result or em_message. In the event of system error, such as Perl aborting and writing information to STDERR pertaining to invalid commands, the script returns:

  • Non-zero value

  • STDOUT and STDERR messages are concatenated and sent to STDERR

This error situation results in a metric error for this user-defined metric. You can view metric errors in the Errors page of the Alerts tab in the Enterprise Manager console.

OS Script Location

Oracle recommends that user-defined metric OS scripts reside in a location outside the Agent Oracle Home. Doing so isolates scripts from any changes that may occur as a result of an Agent upgrade and ensures your scripts remain operational. When registering your script in the Grid Control console, you must specify the full path to the script. Do not use Available Properties (for example, %scriptsDir% or %emdRoot%) as part of the path specification.

Script Runtime Environment

When the user-defined metric is evaluated, it executes the script using the credentials (user name and password) specified at the time the user-defined metric was registered in the Enterprise Manager console. See "Register the Script as a User-Defined Metric". Ensure that the user name and password you specify for the user-defined metric is an active account (on that machine) possessing the requisite permissions to run the script.

Register the Script as a User-Defined Metric

Once you have created the monitoring script, you are ready to add this monitoring functionality to Enterprise Manager as a user-defined metric.


Important:

: For OS-based user-defined metrics, make sure the Management Agent is up and running on the machine where the monitoring script resides before creating the user-defined metric. Operator privilege or higher is required on the host target.

Creating an OS-Based User-Defined Metric

  1. From the home page of the Host that has your OS monitoring script (Related Links), choose User-Defined Metrics. The User-Defined Metrics summary page appears containing a list of previously defined User-Defined Metrics. From this page, you perform edit, view, delete, or create like functions on existing User-Defined Metrics.

  2. Click Create. The Create User-Defined Metric page appears.

  3. Enter the requisite metric definition, threshold, and scheduling information. For the Command Line field, enter the full path to your script, including any requisite shell or interpreters. For example, /bin/sh myscript. See the following section for more details.

  4. Click OK. The User-Defined Metric summary page appears with the new User-Defined Metric appended to the list.

If the user-defined metric has been created and the severity has not been updated recently, it is possible that there are metric errors associated with the user-defined metric execution. In this situation, access the Errors subtab under Alerts tab to check.

Create User-Defined Metric Page (OS-based User-Defined Metric)

The Create User-Defined Metric page allows you to specify the metric information required to successfully register the metric with Enterprise Manager. The page is divided into functional categories:

  • Definition: You define the operational and environmental parameters required to run your script, in addition to the name of the script itself.

  • Operating System Credentials: You enter the credentials used to run the monitoring script. See Enterprise Manager online help for more details on Response Actions. This functional area appears when creating OS-based user-defined metrics.

  • Thresholds: To have the value returned by your script compared with set threshold values, enter the requisite threshold information. The value of the monitored metric returned by your script (as specified by em_result) will be compared against the thresholds you specify. If the comparison holds true, a warning or critical alert may be generated.

  • Schedule: Specify the start time and frequency at which the user-defined script should be run. The time zone used is that of the Agent running on the monitored host.

The following figures show the Create User-Defined Metric pages for an OS-based user-defined metric. When accessing this page from any Host home page, the Create User-Defined Metric page appears as shown in Figure 4-1.

Figure 4-1 Create User-Defined Metric Page (OS-Based)

Description of Figure 4-1 follows

Key elements of this page are described in the following tables.

Table 4-3 Create User-Defined Metric Page: Definition

User-Interface ElementDescription

Metric Name

Metric name identifying the user-defined metric in the Enterprise Manager user interface. This name must be unique for all User-Defined Metrics created on that host.

Metric Type

Type of the value returned by the user-defined script. Choose "NUMBER" if your script returns a number. Choose "STRING" if your script returns an alphanumeric text string.

Command Line

Enter the complete command line entry required to execute the user-defined script. You must enter the full command path as well as full path to the script location. For example, to run a Perl script, you might enter something like the following in the Command Line entry field:

/u1/bin/perl /u1/scripts/myScript.pl

The content of the Command Line is passed as a literal string, so you may use any syntax, special characters, or parameters allowed by your operating system.

Environment

Optional. Enter any environmental variable(s) required to run the user-defined script. A list of predefined properties that can be passed to your script as variables is listed in the Available Properties box. You may also specify your own environment variables. Multiple variables can be defined as a space-separated list.

Example: If your script uses three variables (var1, var2, var3) where var1 is the location of the Perl directory (predefined), var2 is the directory where your Perl scripts are stored (predefined), and var3 is an Oracle home, your entry in the Environment text entry field would appear as follows:

var1=%perlBin% var2=%scriptsDir% var3=/u1/orahome10


Table 4-4 Create User-Defined Metric Page: Operating System

User-Interface ElementDescription

User Name

Enter the user name for a valid operating system account on the machine where the script is to be run. Make sure the specified account has the requisite privileges to access the script directory and execute the script.

Password

Enter the password associated with the User Name.


Table 4-5 Create User-Defined Metric Page: Threshold

User-Interface ElementDescription

Comparison Operator

Select the comparison method Enterprise Manager should use to compare the value returned by the user-defined script to the threshold values.

Available Comparison Operators

Operator Value Metric Type Description

= Number equal to

> Number greater than

< Number less than

>= Number greater than or equal to

<= Number less than or equal to

!= Number not equal to

CONTAINS String contains at least

MATCH String exact match

Warning

The value returned by the script is compared to the Warning threshold value using the specified comparison operator. If this comparison holds true, an alert triggers at the warning severity level.

Specifically, an alert triggers at the warning severity level if the following comparison is true:

<script_value> <comparison_operator> <warning_threshold>

and if the consecutive occurrences preceding notification has been reached.

Critical

The value returned by the script is compared to the Critical threshold value using the specified comparison operator. If this comparison holds true, an alert triggers at the critical severity level.

Specifically, an alert triggers at the critical severity level if the following comparison is true:

<script_value> <comparison_operator> <critical_threshold>

and if the consecutive occurrences preceding notification has been reached.

Consecutive Occurrences Preceding Notification

Consecutive number of times a returned value reaches either the warning or critical thresholds before an alert triggers at a warning or critical severity. This feature is useful when monitoring for sustained conditions. For example, if your script monitors the CPU load for a particular machine, you do not want to be notified at every CPU load spike. Instead, you are only concerned if the CPU load remains at a particular threshold (warning or critical) level for a consecutive number of monitoring cycles.

Response Action

Optional. Specify a script or command that will be executed if the user-defined metric generates a warning or critical alert. The script or command will be executed using the credentials of the Agent owner. Important: Only an Enterprise Manager Super Administrator can create/edit response actions for metrics.

For example, the Management Agent executes the response action if:

The Alert severity is Warning or Critical

AND

There is a change in severity (for example, warning -> critical, critical --> warning, clear --> warning or critical)

For more information, see Enterprise Manager online help.


The User-Defined Metric Schedule interface lets you specify when the Management Agent should start monitoring and the frequency at which it should monitor the condition using your OS script.

OS-Based User-Defined Metric Example

The sample Perl script used in this example monitors the 5-minute load average on the system. The script performs this function by using the 'uptime' command to obtain the average number of jobs in the run queue over the last 5 minutes.

The script is written in Perl and assumes you have Perl interpreter located in /usr/local/bin on the monitored target.

This script, called udmload.pl, is installed in a common administrative script directory defined by the user. For example, /u1/scripts.


Important:

Do not store user-defined metric monitoring scripts in the same location as Enterprise Manager system scripts.

Full text of the script:

#!/usr/local/bin/perl


# Description: 5-min load average.

# Sample User Defined Event monitoring script.


$ENV{PATH} = "/bin:/usr/bin:/usr/sbin"; 


$DATA = `uptime`; 

$DATA =~ /average:\s+([\.\d]+),\s+([\.\d]+),\s+([\.\d]+)\s*$/; 

  


if (defined $2) { 

   print "em_result=$2\n"; 

} else { 

   die "Error collecting data\n"; 

} 

  1. Copy the script (udmload.pl) to the monitored target. For example: /u1/scripts. Make sure you have an Enterprise Manager 10g Management Agent running on this machine.

  2. Edit the script, if necessary, to point to the location of the Perl interpreter on the monitored target. By default, the script assumes the Perl interpreter is in /usr/local/bin.

  3. As a test, run the script: udmload.pl You may need to set its file permissions so that it runs successfully. You should see output of this form:

    em_result=2.1 
    
  4. In Create User-Defined Metric page, create a new user-defined metric as follows:

    1. Definition Settings

      • Metric Name: Test User-Defined Metric

      • Metric Type: Number

      • Command Line: %perlBin%/perl /u1/scripts/udmload.pl

      • Environment: leave blank

      • Operating System User Name: <OS user able to execute the script>

      • Password: ******

    2. Threshold Settings

      • Comparison Operator: >=

      • Critical Threshold: 0.005

      • Warning Threshold: 0.001

      • Consecutive Occurrences Preceding Notification: 1

      In this example, we want the metric to trigger an alert at a Warning level if the 5-minute load average on the machine reaches 0.001, and trigger an alert at a Critical level if the 5-minute load average reaches 0.005. Feel free to change these thresholds depending on your system.

    3. Schedule Settings:

      • Start: Immediately after creation

      • Frequency: Repeat every 5 minutes. You must specify at least a 5 minute interval.

Setting Up the Sample Script as a User-Defined Metric

When the 5-minute load reaches at least 0.001, you should see the metric trigger an alert.

Creating a SQL-Based User-Defined Metric

You can also define new metrics using custom SQL queries or function calls supported against single instance databases and instances on Real Application Clusters (RAC). To create this type of user-defined metric, you must have Enterprise Manager Operator privileges on the database:

  1. From the Related Links area of any Database home page, choose User-Defined Metrics. The User-Defined Metrics summary page appears containing a list of previously defined user-defined metrics. From this page, you perform edit, view, delete, or create like functions on existing user-defined metrics.

  2. Click Create. The Create User-Defined Metric page appears.

  3. Enter the requisite metric definition, threshold, and scheduling information. For the SQL Query field, enter the query or function call. See the following section for more information.

    Click Test to verify that the SQL query or function call can be executed successfully using the credentials you have specified

  4. Click OK. The User-Defined Metric summary page appears with the new user-defined metric appended to the list.

If the user-defined metric has been created and the severity has not been updated recently, it is possible that there are metric errors associated with the user-defined metric execution. In this situation, access the Errors subtab under Alerts tab to check.

Create User-Defined Metric Page (SQL-Based User-Defined Metric)

The Create User-Defined Metric page allows you to specify the metric information required to successfully register the metric with Enterprise Manager. The page is divided into functional categories:

The following figures show the Create User-Defined Metric pages for a SQL-based user-defined metric. When accessing this page from any Database home page, the Create User-Defined Metric page appears as shown in Figure 4-2.

Figure 4-2 Create User-Defined Metric Page (SQL-Based)

Description of Figure 4-2 follows

Key elements of this page are described in the following tables.

Table 4-6 Create User-Defined Metric Page: Definition

User-Interface ElementDescription

Metric Name

Metric name identifying the user-defined metric in the Enterprise Manager user interface.

Metric Type

Type of the value returned by the user-defined script. Choose "NUMBER" if your script returns a number. Choose "STRING" if your script returns an alphanumeric text string.

SQL Query Output

Specify whether the SQL script is to return a single value (one column) or a multiple rows (two columns).

  • Single Value: Query is one of the following types.

    A SELECT statement returning a single value. Example: SELECT sal FROM emp WHERE empno=7369

    A function call returning a single value. Example: myfunc(123, 'abc')

  • Two Columns: Query is a SELECT statement that returns two columns and possibly multiple rows. Example: SELECT ename, sal FROM emp. Each entry in the first column (the key column) must be a unique string. The second column (the value column) must be of the selected Metric Type.

SQL Query

Enter a SQL query or function call that returns values of the appropriate type (STRING or NUMBER). The SQL statement must return one or two column. If your SQL statement only returns one column, only one row can be returned. If you want multiple rows returned, your SQL statement must return two columns.


Table 4-7 Create User-Defined Metric Page: Database Credentials

User-Interface ElementDescription

User Name

Enter the user name for a valid database account on the database where the SQL query is to be run. Make sure that the specified account has the requisite privileges to run the SQL query.

Password

Enter the password associated with the User Name.


Table 4-8 Create User-Defined Metric Page: Threshold

User-Interface ElementDescription

Comparison Operator

Select the comparison method Enterprise Manager should use to compare the value returned by the SQL query or function call to the threshold values. When the query returns two columns, the second column (value column) will be used for comparison against threshold values.

Available Comparison Operators

Operator Value Metric Type Description

= Number equal to

> Number greater than

< Number less than

>= Number greater than or equal to

<= Number less than or equal to

!= Number not equal to

CONTAINS String contains at least

MATCH String exact match

Warning

The value returned by the SQL query or function call is compared to the Warning threshold value using the specified comparison operator. If this comparison holds true, an alert triggers at the warning severity level.

Specifically, an alert triggers at the warning severity level if the following comparison is true:

<query_value> <comparison_operator> <warning_threshold>

and if the consecutive occurrences preceding notification has been reached.

Critical

The value returned by the SQL query or function call is compared to the Critical threshold value using the specified comparison operator. If this comparison holds true, an alert triggers at the critical severity level.

Specifically, an alert triggers at the critical severity level if the following comparison is true:

<query_value> <comparison_operator> <critical_threshold>

and if the consecutive occurrences preceding notification has been reached.

Warning Thresholds by Key

and

Critical Thresholds by Key

For queries returning two columns (the first column is the key and the second column is the value), you can specify thresholds on a per key basis. The following example uses the following query:

SELECT ename FROM emp

Threshold settings for this example are shown.

Use the format key:value . Keys are case-sensitive.

  • Warning:500

  • Critical:300

  • Comparison Operator: <

  • Warning threshold by key: SMITH:250;JONES:400;CLARK:900

    The warning threshold is set to 250 for SMITH, 400 for JONES, and 900 for CLARK.

  • Critical threshold by key: SMITH:100;JONES:200;CLARK:500

    The critical threshold is set to 100 for SMITH, 200 for JONES, and 500 for CLARK.

    All other keys will use the threshold values specified in the Warning and Critical fields.

Consecutive Occurrences Preceding Notification

Consecutive number of times a returned value reaches either the warning or critical thresholds before an alert triggers at a warning or critical severity. This feature is useful when monitoring for sustained conditions. For example, if your script monitors the CPU load for a particular machine, you do not want to be notified at every CPU load spike. Instead, you are only concerned if the CPU load remains at a particular threshold (warning or critical) level for a consecutive number of monitoring cycles.

Response Action

Optional. Specify a script or command that will be executed if the user-defined metric generates a warning or critical alert. The script or command will be executed using the credentials of the Agent owner. Important: Only an Enterprise Manager Super Administrator can create/edit response actions for metrics.

For example, the Management Agent executes the response action if:

The Alert severity is Warning or Critical

AND

There is a change in severity (for example, warning -> critical, critical --> warning, clear --> warning or critical)

For more information, see Enterprise Manager online help.

Alert Message

Enter a custom message (up to 400 characters) to be used when an alert is sent. The default message uses %Key% and %value% variables to display the metric key and its returned value. The %Key% and %value% variables are case-sensitive.

For example, a payroll system alert for underpayment of salary might be defined as:

Underpaid Employee: %Key% has salary of %value%

If the SQL query returns 2 columns, you can use the %Key% variable to represent the key value and the %value% variable to represent the return value.

If the SQL query returns 1 column, only the %value% variable is applicable in the alert message.


The User-Defined Metric Schedule interface lets you specify the frequency at which the SQL query or function should be run.

SQL-Based User-Defined Metric Examples

For a database version 9i and higher, you can run the example queries as dbsnmp, which is the default monitoring user account for the Management Agent. On a 8.1.7 database (which does not have SELECT ANY DICTIONARY system privilege), you must grant dbsnmp the following privileges in order for the queries to run successfully:

For example #1:

grant select on sys.dba_tablespaces to dbsnmp;

grant select on sys.dba_data_files to dbsnmp;

grant select on sys.dba_free_space to dbsnmp;

For example #2:

grant select on sys.dba_extents to dbsnmp;

The above grant statements can be run as SYSDBA after logging in via "connect internal". The queries can also be run by any user who has been granted the DBA role.

Example 1: Query Returning Tablespace Name and Percent Used

This sample user-defined metric monitors the percentage of space used for dictionary managed permanent tablespaces. A DBA can use this as a reference on when to add datafiles for the tablespace.Oracle recommends setting a polling frequency of 30 minutes, warning threshold at 75, and critical threshold at 85.

Example 1 SQL

SELECT d.tablespace_name,

     round(((a.bytes - NVL(f.bytes,0))*100/a.maxbytes),2) used_pct

FROM   sys.dba_tablespaces d,

      (select tablespace_name, sum(bytes) bytes, sum(greatest(maxbytes,bytes)) maxbytes

         from sys.dba_data_files group by tablespace_name) a,

      (select tablespace_name, sum(bytes) bytes

         from sys.dba_free_space group by tablespace_name) f

WHERE d.tablespace_name = a.tablespace_name(+)

 AND d.tablespace_name = f.tablespace_name(+)

 AND NOT (d.extent_management = 'LOCAL' AND d.contents = 'TEMPORARY') 

Example 2: Query Returning Segment Name/Type and Extent Count

This sample user-defined metric checks for non-system table and index segments that are reaching a high number of extents. A high number of extents could indicate a segment with fragmentation and/or performance problems. A DBA can use this as a reference on when to call Segment Shrink or the Reorganization Wizard in Enterprise Manager.Oracle recommends setting a polling frequency of 24 hours, warning threshold at 1000, and critical threshold at 2000.

Example 2 SQL

SELECT decode(nvl(partition_name, ' '),

             ' ',   owner || '.' || segment_name || ' ' || segment_type,

             owner || '.' || segment_name || '.' || partition_name || ' ' || segment_type) as segment,

      count(extent_id) as extent_count

FROM dba_extents

WHERE (segment_type like 'TABLE%' OR segment_type like 'INDEX%') AND

     (owner != 'SYSTEM' AND owner != 'SYS')

GROUP BY owner, segment_name, partition_name, segment_type

ORDER BY EXTENT_COUNT DESC 

Example 3: Embed a Long SQL statement in a PL/SQL Routine

In situations where the SQL statement forming the SQL user-defined metric exceeds 1024 characters, you must embed the SQL statement in a PL/SQL routine. This must be carried out in three step. In this example, a long SQL statement is used to track tablespaces & free space in them and raise alerts if the free space falls below a user specified threshold. A 2-column SQLUDM is crated using this query.

Example 4-1 of a long (more than1024 characters) SQL statement that returns two values: tablespace_name (key) and free_mb(value)

Example 4-1 Long SQL Statement

select Tablespace, case when (MxAvail <= 15) and (MxFreeMB < 20000) then 'CRITICAL, '||MxUsed||'%' when (MxAvail <= 20) and (MxFreeMB < 20000) then 'WARNING, '||MxUsed||'%' else 'OK' end Error_Level, MxAvail,MxUsed,MxFreeMB,MxExdMB from (select nvl(b.tablespace_name, nvl(a.tablespace_name,'UNKNOWN')) as Tablespace, data_files as NumDBFs, mbytes_alloc as AllocMB, Round( mbytes_alloc-nvl(mbytes_free,0),0) as UsedMB, Round(nvl(mbytes_free,0),0) "AllocFreeMB", Round((( mbytes_alloc-nvl(mbytes_free,0))/ mbytes_alloc)*100,0) as AllocUsed, MaxSize_Mbytes as MxExdMB, Round(MaxSize_Mbytes - (mbytes_alloc-nvl(mbytes_free,0)),0) as MxFreeMB, Round((mbytes_alloc/MaxSize_Mbytes*100),0) as MxUsed, Round((MaxSize_Mbytes - (mbytes_alloc-nvl(mbytes_free,0)))/MaxSize_Mbytes *100,0) as MxAvail from ( select sum(bytes)/1024/1024 mbytes_free, max(bytes)/1024/1024 largest,tablespace_name from sys.dba_free_space group by tablespace_name ) a, ( select sum(bytes)/1024/1024 mbytes_alloc, sum(decode(maxbytes,0,bytes,maxbytes))/1024/1024 MaxSize_Mbytes, tablespace_name,count(file_id) data_files from sys.dba_data_files group by tablespace_name )b where a.tablespace_name (+) = b.tablespace_name order by 1)

Because a 2-column SQL UDM is being created (tablespace, free_space_in_MB), an array must created for the data being returned from this query, as shown in

Example 4-2 Creating an Array of Returned Values

CREATE OR REPLACE TYPE tablespace_obj as OBJECT

(

    tablespace_name VARCHAR2(256),

    free_mb NUMBER);

/


CREATE OR REPLACE TYPE tablespace_array AS TABLE OF tablespace_obj;

/

The next step is to embed the long SQL statement shown in Example 4-1 in a PL/SQL routine as shown in Example 4-3

Example 4-3 Embedded SQL in a PL/SQL Routine

CREATE OR REPLACE FUNCTION calc_tablespace_free_mb

RETURN tablespace_array

IS

    tablespace_data TABLESPACE_ARRAY := TABLESPACE_ARRAY();

BEGIN

    SELECT tablespace_obj(tablespace, mxfreemb)

        BULK COLLECT INTO tablespace_data

    FROM

(

select Tablespace, case when (MxAvail <= 15) and (MxFreeMB < 20000) then

 'CRITICAL, '||MxUsed||'%' when (MxAvail <= 20) and (MxFreeMB < 20000) then

 'WARNING, '||MxUsed||'%' else 'OK' end Error_Level,

 MxAvail,MxUsed,MxFreeMB,MxExdMB from (select nvl(b.tablespace_name,

 nvl(a.tablespace_name,'UNKNOWN')) as Tablespace, data_files as NumDBFs, mbytes

_alloc as AllocMB, Round( mbytes_alloc-nvl(mbytes_free,0),0) as UsedMB,

 Round(nvl(mbytes_free,0),0) "AllocFreeMB", Round((( mbytes_alloc-nvl(mbytes

_free,0))/ mbytes_alloc)*100,0) as AllocUsed, MaxSize_Mbytes as MxExdMB,

 Round(MaxSize_Mbytes - (mbytes_alloc-nvl(mbytes_free,0)),0) as MxFreeMB,

 Round((mbytes_alloc/MaxSize_Mbytes*100),0) as MxUsed, Round((MaxSize_Mbytes -

 (mbytes_alloc-nvl(mbytes_free,0)))/MaxSize_Mbytes *100,0) as MxAvail from (

 select sum(bytes)/1024/1024 mbytes_free, max(bytes)/1024/1024 largest,tablespace

_name from sys.dba_free_space group by tablespace_name ) a, ( select

 sum(bytes)/1024/1024 mbytes_alloc,

 sum(decode(maxbytes,0,bytes,maxbytes))/1024/1024 MaxSize_Mbytes, tablespace

_name,count(file_id) data_files from sys.dba_data_files group by tablespace_name

 )b where a.tablespace_name (+) = b.tablespace_name order by 1)

) CUSTOMER_QUERY;


  RETURN tablespace_data;

END calc_tablespace_free_mb;

/

The final step in the process is to create the 2-column UDM with the following:

  • Metric Type: NUMBER

  • SQL Query Output: Two Columns

  • SQL Query:

    SELECT tablespace_name, free_mb
    
    FROM TABLE(CAST(calc_tablespace_free_mb as TABLESPACE_ARRAY))
    
    

Notifications, Corrective Actions, and Monitoring Templates

User-Defined Metrics, because they are treated like other metrics, can take advantage of Enterprise Manager's notification system, corrective actions and monitoring templates.


Note:

Corrective actions and monitoring templates support both OS user-defined metrics and SQL-based user-defined metrics that return single scalar values.

Getting Notifications for User-Defined Metrics

As with regular metrics, you can receive e-mail notifications when user-defined metric critical or warning alert severities are reached. Assuming you have already defined your e-mail addresses and notification schedule, the remaining task is to set up a notification rule for the user-defined metric.

To set up notification rules:

  1. Click Preferences.

  2. From the vertical navigation bar, click Rules if you are a Super Administrator or My Rules if you are a regular Enterprise Manager administrator.

  3. Click Create to define a new notification rule. The Create Notification Rule pages appear.

  4. From the General page, enter the required rule definition information and choose Target Type Host for OS-based user-defined metrics or choose Database Instance or Cluster Database for SQL-based user-defined metrics.

  5. On the Metrics page, click Add. A list of available metrics appears. To view all metrics simultaneously, choose Show All from the drop-down menu.

  6. Select User-Defined Numeric Metric or User-Defined String Metric based on the type of value returned by your user-defined metric.

  7. In the Objects column, choose whether you want to receive notification for all user-defined metrics (All Objects) or specific user-defined metrics (Select).

    image shows the Add Metric page for a UDM

    When choosing the Select option, enter the name of the user-defined metric, or specify multiple user-defined metrics separated by commas. You can use the wildcard character (%) to match patterns for specific user-defined metrics.

    You can search for available user-defined metrics using the search function (flashlight icon). However, search results will only show user-defined metrics that have at least one collected data point. For metrics that have not yet collected at least one data point, as may be the case for a newly created user-defined metric, you must specify them in the Select text entry field.

    The format used to specify individual user-defined metrics in the Select text entry field depends on the type of value(s) returned. The following three examples illustrate how to specify user-defined metrics for three possible return value situations.

    Formats shown in the following table summarize how to specify multiple user-defined metrics returning single and double values. Specific examples follow.

    Return ValuesSelect Text Entry Field Format
    SingleMY_UDM1,MY_UDM3,MY_UDM5
    Double (2-column), All Key ValuesMY_UDM1;%,MY_UDM2;%,MY_UDM4;%

    User-defined metrics returning two columns are specified as <UDM Name>;% where "%" is a wildcard signifying ALL key values associated with that user-defined metric.

    Note: 2-column values apply to SQL user-defined metrics only.

    Double (2-column), Specific Key ValuesMY_UDM1;K1,MY_UDM1_K2

    Note: 2-column values apply to SQL user-defined metrics only.


    Example 1: Host User-Defined Metrics Returning a Single Value

    In this example, you will receive notifications for two host user-defined metrics: MY_UDM9 and MY_UDM11. Both metrics return numeric values. To receive notifications for all host user-defined metrics, select the All Objects (Script) option where Script refers to the user-defined metric name.

    Graphic shows a host UDM retruning a single value.

    Example 2: SQL User-Defined Metric Returning a Single Value

    In this example, you will receive notifications for four SQL user-defined metrics: MY_UDM1 and MY_UDM3 (numeric value returned) and MY_UDM5 and MY_UDM7 (string value returned). To receive notifications for all SQL user-defined metrics, select the All Objects (Script) option where Script refers to the user-defined metric name.

    Graphic shows a SQL UDM returning a single value.

    Example 3: SQL User-Defined Metrics Returning Two Values

    To receive notifications for SQL user-defined metrics returning two values, the syntax is slightly different. In this situation, you specify the user-defined metric using the following format: <Metric ID;Key> where Metric ID is the name of the user-defined metric and Key is the specific key value you want returned. To receive notifications for all SQL user-defined metrics, select the All Objects (Metric ID;Key) option.

    graphic shows 2-column udm for all keys and specific keys

    For SQL user-defined metrics returning two values, you have the option of receiving notifications for all key values or just specific values for a given metric. Use a wildcard character (%) to specify all key values. Example: MY_UDM1;%, MY_UDM2;%.

    To receive notifications for specific key values, specify the exact key value to be returned. Example: MY_UDM5;K1,MY_UDM5;K2 (only key values K1 and K2 are returned from MY_UDM5).

  8. Select the severity or corrective action state for which you would like to receive the notification and then click Continue.

  9. If you want to receive e-mail for the specified user-defined metric, go to the Notification Rule and check the "Send me E-mail" option.

  10. Click OK to create the new notification rule. If you made the notification rule public, other administrators can subscribe to the same rule.

Setting Corrective Actions for User-Defined Metrics

Corrective actions allow you to specify automated responses to alerts ensuring that routine responses to alerts are automatically executed. Corrective actions can be defined for both SQL and OS-based user-defined metrics.

To set up corrective actions:

  1. From a target home page, click Metric and Policy Settings from Related Links.

  2. Locate and edit the user-defined metric.

  3. From the Edit Advanced Settings page, click Add under Corrective Actions for the Critical or Warning alert severity and define the corrective action. Corrective actions can be defined for one or both alert severities.

Deploying User-Defined Metrics Across Many Targets Using Monitoring Templates

Monitoring Templates simplify the task of standardizing monitoring settings across your enterprise by allowing you to specify the monitoring settings once and applying them to your monitored targets. You can thus use Monitoring Templates as a way to propagate user-defined metrics across a large number of targets.

Assuming you have created the user-defined metric on the host or database target, you can use Monitoring Templates to propagate the user-defined metric to other hosts or database targets.

To create a Monitoring Template for the user-defined metric:

  1. Click Setup.

  2. From the vertical navigation bar, click Monitoring Templates

  3. Click Create. The Copy Target Settings page appears.

  4. Specify the host or database on which you defined the user-defined metric and click Continue.

  5. Fill in the requisite information on the General page.

  6. On the Metric Thresholds page, you can choose to keep or remove the other metrics that have been copied over from the target.

  7. You can also edit the user-defined metric's thresholds, collection schedule, and corrective actions.

  8. On the Policies page, you can choose to keep or remove any policy rules that have been copied over from the target.

  9. Click OK to save the template settings to the Management Repository.

Once the template containing the user-defined metric has been created, you can propagate the user-defined metric by applying the template to other hosts or databases.


Important:

For OS-based user-defined metrics, you will first need to separately deploy the OS Script used by the user-defined metric to all destination hosts. The OS Script should reside in the same location across all host targets.

For SQL-based user-defined metrics, if the SQL query specified is a function call, then you need to create this function across all databases on which the SQL-based user-defined metric will be created.


To apply the monitoring template:

  1. On the Monitoring Templates page, select the monitoring template and click Apply.

  2. On the Apply Monitoring Template page, add the targets on which the user-defined metric should be created.

  3. If a two-column SQL-based user-defined metric is part of the template, the Metric with Multiple Thresholds option is applied according one of the following situations:

    • Situation One: The target to which the template will be applied does not contain the two-column SQL user-defined metric defined in the template. In this situation, regardless of which Metric with Multiple Thresholds option is chosen, the user-defined metric is copied to the target when you apply the template.

    • Situation Two: The target to which the template will be applied does contain the two-column SQL user-defined metric. The name (case insensitive) and return value (numeric, scalar, or two column) of both the target and template user-defined metrics must match. In this situation, you must select one of the Metric with Multiple Thresholds options:

      • Apply threshold settings for monitored objects common to both template and target: For only those keys which the target has in common with the template, the target threshold values will be set to the values defined in the template. This option is chosen by default and is recommended for most situations.

      • Duplicate threshold settings on target: For keys which are common between target and template, the thresholds will be set to the values defined in the template. Any extra keys (and their thresholds) that exist in the template but not on the target will be copied to the target in anticipation that these keys will be created in the target at some point in the future. Any extra keys (and their thresholds) that exist on the target but not in the template will be deleted from the target.

    Important: If there are other metrics in the monitoring template, refer first to the Enterprise Manager online help for implications of what this option means for these other metrics.

  4. Click Continue.

  5. On the subsequent page, specify the credentials that should be used when running the user-defined metric on the destination targets.

  6. Click Finish.

  7. When you return back to the Monitoring Templates page, check that "Pending Apply Operations" count for your template i?!s zero. This indicates the number of template apply operations that could be pending. Once they are all complete, the count should be zero.

Deploying User-Defined Metrics Using Scripts

An alternate method of deploying user-defined metrics to large numbers of targets is to use the Enterprise Manager Command Line Interface (EMCLI). Using the EMCLI "apply_template" verb, you can deploy user-defined metrics via custom scripts. For more information about the "apply_template" verb, see the Oracle Enterprise Manager Command Line Interface manual.

Deleting User-Defined Metrics Across Many Targets Using Monitoring Templates

Just as templates can be used to deploy user-defined metrics across targets, templates can also be used to delete these metrics across targets should these metrics no longer be in use.

To create a Monitoring Template for the user-defined metric:

  1. Click Setup.

  2. From the vertical navigation bar, click Monitoring Templates

  3. Click Create. The Copy Target Settings page appears.

  4. Specify the host or database on which there is a user-defined metric that needs to be deleted and click Continue.

  5. Fill in the requisite information on the General page.

  6. On the Metric Thresholds page, remove all metrics from the template except the user-defined metric to be deleted.

  7. Click the pencil icon to access the Edit Advanced Settings page.

  8. Check the Mark for Delete option and click Continue. The Mark for Deletion icon now appears next to the user-defined metric on the Metric Thresholds page.

  9. On the Policies page, remove all policies.

  10. Click OK to save the template settings to the Management Repository.

To apply the monitoring template:

  1. On the Monitoring Templates page, select the monitoring template and click Apply.

  2. On the Apply Monitoring Template page, add the targets on which the user-defined metric should be deleted.

  3. Click Continue.

  4. On the subsequent page, specify the credentials that should be used when running the user-defined metric on the destination targets.

  5. Click Finish.

  6. On the Monitoring Templates page, check that "Pending Apply Operations" count for your template is zero. This indicates the number of template apply operations that could be pending. Once they are all complete, the count should be zero.

Enterprise manager will delete all user-defined metrics found on the selected target that match the following criteria:

  • Name of the user-defined metrics (case insensitive)

  • Return value of the user-defined metric (numeric or scalar)

  • For SQL-based user-defined metrics, the output of the query (single value or two columns). The match does not take into consideration the actual script used by the user-defined metric. For this reason, even though the script on the target user-defined metric may be different from that of the template user-defined metric, the target user-defined metric will still be deleted.

  • Host User-Define Metrics using scripts: You must delete the script from the host on which you want the UDM deleted.

  • SQL-based user-defined metrics using function calls: You must delete the function from the database on which you want the user-defined metric deleted.

Changing User-Defined Metric Credentials

As discussed earlier, user-defined metrics require valid credentials (username and password) in order to execute monitoring scripts/SQL queries. For this reason, both the monitored target's password and the password defined in the user-defined metric must match. This can be problematic if target passwords are changed frequently. For environments with a large number of targets, you can use the Enterprise Manager Command Line Interface (EMCLI) to change the target password and user-defined metric password simultaneously using scripts. Use the "update_password" verb to change the target password. This password change is then propagated to all features of Enterprise Manager that use the specified username, which includes preferred credentials, corrective actions, jobs, and both host and SQL-based user-defined metrics.

The following example changes the password associated with the OS user sysUser from sysUserOldPassword to sysUserNewPassword.

Example 4-4 Host Password Change

update_password   -target_type=host  -target_name=MyHost  -credential_type=HostCreds  -key_column=HostUserName:sysUser-non_key_column=HostPassword:sysUserOldPassword:sysUserNewPassword

The next example changes the password associated with the database user sys from sysPassword to sysNewPassword.

Example 4-5 Database Password Change

update_password -target_type=oracle_database -target_name=ORCL -credential_type=DBCreds -key_column=DBUserName:sys-non_key_column=DBPassword:sysPassword:sysNewPassword:DBAROLE

For more information about EMCLI, see the Oracle Enterprise Manager Command Line Interface guide.

PKzzS?PKOXEOEBPS/dcommon/oracle.gifJGIF87aiyDT2F'G;Q_oKTC[ 3-Bq{ttsoGc4I)GvmLZ).1)!ꑈ53=Z]'yuLG*)g^!8C?-6(29K"Ĩ0Яl;U+K9^u2,@@ (\Ȱ Ë $P`lj 8x I$4H *(@͉0dа8tA  DсSP v"TUH PhP"Y1bxDǕ̧_=$I /& .)+ 60D)bB~=0#'& *D+l1MG CL1&+D`.1qVG ( "D2QL,p.;u. |r$p+5qBNl<TzB"\9e0u )@D,¹ 2@C~KU 'L6a9 /;<`P!D#Tal6XTYhn[p]݅ 7}B a&AƮe{EɲƮiEp#G}D#xTIzGFǂEc^q}) Y# (tۮNeGL*@/%UB:&k0{ &SdDnBQ^("@q #` @1B4i@ aNȅ@[\B >e007V[N(vpyFe Gb/&|aHZj@""~ӎ)t ? $ EQ.սJ$C,l]A `8A o B C?8cyA @Nz|`:`~7-G|yQ AqA6OzPbZ`>~#8=./edGA2nrBYR@ W h'j4p'!k 00 MT RNF6̙ m` (7%ꑀ;PKl-OJPKOXEOEBPS/dcommon/oracle-logo.jpgqEJFIFC    $.' ",#(7),01444'9=82<.342C  2!!22222222222222222222222222222222222222222222222222'7" }!1AQa"q2#BR$3br %&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz w!1AQaq"2B #3Rbr $4%&'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz ?( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (QEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQE!KEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEzE7V%ȣOΏ9??:a"\fSrğjAsKJ:nOzO=}E1-I)3(QEQEQEQEQEQEQE֝Hza<["2"pO#f8M[RL(,?g93QSZ uy"lx4h`O!LŏʨXZvq& c՚]+: ǵ@+J]tQ]~[[eϸ (]6A&>ܫ~+כzmZ^(<57KsHf妬Ϧmnẁ&F!:-`b\/(tF*Bֳ ~V{WxxfCnMvF=;5_,6%S>}cQQjsOO5=)Ot [W9 /{^tyNg#ЄGsֿ1-4ooTZ?K Gc+oyڙoNuh^iSo5{\ܹ3Yos}$.nQ-~n,-zr~-|K4R"8a{]^;I<ȤL5"EԤP7_j>OoK;*U.at*K[fym3ii^#wcC'IIkIp$󿉵|CtĈpW¹l{9>⪦׺*ͯj.LfGߍԁw] |WW18>w.ӯ! VӃ :#1~ +މ=;5c__b@W@ +^]ևՃ7 n&g2I8Lw7uҭ$"&"b eZ":8)D'%{}5{; w]iu;_dLʳ4R-,2H6>½HLKܹR ~foZKZ࿷1[oZ7׫Z7R¢?«'y?A}C_iG5s_~^ J5?œ tp]X/c'r%eܺA|4ծ-Ե+ْe1M38Ǯ `|Kյ OVڅu;"d56, X5kYR<̭CiطXԮ];Oy)OcWj֩}=܅s۸QZ*<~%뺃ȶp f~Bðzb\ݳzW*y{=[ C/Ak oXCkt_s}{'y?AmCjޓ{ WRV7r. g~Q"7&͹+c<=,dJ1V߁=T)TR՜*N4 ^Bڥ%B+=@fE5ka}ędܤFH^i1k\Sgdk> ֤aOM\_\T)8靠㡮3ģR: jj,pk/K!t,=ϯZ6(((((((49 xn_kLk&f9sK`zx{{y8H 8b4>ÇНE|7v(z/]k7IxM}8!ycZRQ pKVr(RPEr?^}'ðh{x+ՀLW154cK@Ng C)rr9+c:׹b Жf*s^ fKS7^} *{zq_@8# pF~ [VPe(nw0MW=3#kȵz晨cy PpG#W:%drMh]3HH<\]ԁ|_W HHҡb}P>k {ZErxMX@8C&qskLۙOnO^sCk7ql2XCw5VG.S~H8=(s1~cV5z %v|U2QF=NoW]ո?<`~׮}=ӬfԵ,=;"~Iy7K#g{ñJ?5$y` zz@-~m7mG宝Gٱ>G&K#]؃y1$$t>wqjstX.b̐{Wej)Dxfc:8)=$y|L`xV8ߙ~E)HkwW$J0uʟk>6Sgp~;4֌W+חc"=|ř9bc5> *rg {~cj1rnI#G|8v4wĿhFb><^ pJLm[Dl1;Vx5IZ:1*p)إ1ZbAK(1ׅ|S&5{^ KG^5r>;X׻K^? s fk^8O/"J)3K]N)iL?5!ƾq:G_=X- i,vi2N3 |03Qas ! 7}kZU781M,->e;@Qz T(GK(ah(((((((Y[×j2F}o־oYYq $+]%$ v^rϭ`nax,ZEuWSܽ,g%~"MrsrY~Ҿ"Fت;8{ѰxYEfP^;WPwqbB:c?zp<7;SBfZ)dϛ; 7s^>}⍱x?Bix^#hf,*P9S{w[]GF?1Z_nG~]kk)9Sc5Ո<<6J-ϛ}xUi>ux#ţc'{ᛲq?Oo?x&mѱ'#^t)ϲbb0 F«kIVmVsv@}kҡ!ˍUTtxO̧]ORb|2yԵk܊{sPIc_?ħ:Ig)=Z~' "\M2VSSMyLsl⺿U~"C7\hz_ Rs$~? TAi<lO*>U}+'f>7_K N s8g1^CeКÿE ;{+Y\ O5|Y{/o+ LVcO;7Zx-Ek&dpzbӱ+TaB0gNy׭ 3^c T\$⫫?F33?t._Q~Nln:U/Ceb1-im WʸQM+VpafR3d׫é|Aү-q*I P7:y&]hX^Fbtpܩ?|Wu󭏤ʫxJ3ߴm"(uqA}j.+?S wV ~ [B&<^U?rϜ_OH\'.;|.%pw/ZZG'1j(#0UT` Wzw}>_*9m>󑓀F?EL3"zpubzΕ$+0܉&3zڶ+jyr1QE ( ( ( ( ( ( ( (UIdC0EZm+]Y6^![ ԯsmܶ捆?+me+ZE29)B[;я*wGxsK7;5w)}gH~.Ɣx?X\ߚ}A@tQ(:ͧ|Iq(CT?v[sKG+*רqҍck <#Ljα5݈`8cXP6T5i.K!xX*p&ќZǓϘ7 *oƽ:wlຈ:Q5yIEA/2*2jAҐe}k%K$N9R2?7ýKMV!{W9\PA+c4w` Wx=Ze\X{}yXI Ү!aOÎ{]Qx)#D@9E:*NJ}b|Z>_k7:d$z >&Vv󃏽WlR:RqJfGإd9Tm(ҝEtO}1O[xxEYt8,3v bFF )ǙrPNE8=O#V*Cc𹾾&l&cmCh<.P{ʦ&ۣY+Gxs~k5$> ӥPquŽўZt~Tl>Q.g> %k#ú:Kn'&{[yWQGqF}AЅ׮/}<;VYZa$wQg!$;_ $NKS}“_{MY|w7G!"\JtRy+贾d|o/;5jz_6fHwk<ѰJ#]kAȎ J =YNu%dxRwwbEQEQEQEQEQEQEQEQEQE'fLQZ(1F)hQ@X1KEQE-Q@ 1KE3h=iPb(((1GjZ(-ʹRPbR@ 1KE7`bڒyS0(-&)P+ ڎԴP11F)h&:LRmQ@Q@Š(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((?l:ϊw "{{-3j3%{Ə77}>ٻګ=Vf&-P7h)Y3>eր.QX>,ާy43/c'Vg%|#Ee$+|NU0>޺F.ݼFm[zWA<7V\[J6 dGhJ+/R.\-3X&+2qF}\㼰L $0? ESմ>P[G4ʍ3d ',r`zZ7tmPiszlZuAhHlm NF9Ƞ J*z66h)[ELrɜ/$wNŞ#wu&vCr;`p آˏĺ \ڤZޚ|/[C73=ǭ\,伿?8ԀvWqox Ҵ˝Vc",lʲ Nq?7yKIoQ%z>kj6v|$[qGOQRi۵ƗZ_@QeP#8 P*g%|#Ee$;:}Avf\$Q@U{=N; .dɠHA#'n.%"BI#TP2I'4%c wxJ;!9 8Prp?lPESԵm7F[SP"u2ĥNbp DZAKT[_OKvHbF|x' J*8'+yc P_-q# *iͻ\iz KYU pJ3=qj03Y.X ,@84rԼK okzm;䌀dcZQEQEQEQEQEQEQEQEQEQEQEQEQEQEd_}A/ekX8W<8-}E\ _H3O \\K0E$QDOk?; z a?^F_i~.cՃ)jXu8c?~~!&E-.ǒ52˞W:3 .N3ח{Nş A{[ /{94y.UICԱFWb0@0Ad/\^%E\#/)Uf`۷-Xsk R$IVBI[wy'wK_{6KD2ǘ0vsz\h myJo`|9*ݱdV?$P**'7~xC%3'eЂr䞊?*4/SzgGwwϽuֳ6!J#O jPRdh,(n$X,U @8<3I/~?ͻvx{+Y>ѭDgr$? cs?x[ܝ~ q)+zpr p~}Lyt Gӊ"® 3T0Ba@l>&|Etx : 8%nFq8eo.cW]m> si:įyAT]ЬMmSkyg{h]zԎe%|=NJ>Z{<rc2)#$H9S:>[ZoK"g%[/@>X q2Ȯ<?t>T W yC^D Jf1ƈ,9;A|$;lxN A /iyahWYjpDHX+cs6C 9U&fo.bv'KIoQ%X|=oOj.Pnb8U_Ht/x҉('h%ԮoIjڅ<1\DY@FFPT\fx?ڤq +0,. 9fʧR9+o\kSi2^ ^0-+0eh|,V4+Jj|1"d>wpp0';@=ow:oú_Ϫ_E(w Hc]O`HpzHuw҈' *E-y^Ï@ѧ:>HFP~bǸkuu,t]=ZK*v!R@F#7: *O.M7ş A'"ie+1mэ`:r-5 4LEt^bnkv#МpGE~G6$]\=H`YX9y;rNMe]Gkt jiG===jEE@~ŧ_k5ItxNUA]1bvcL׆'0i"Cdftii9'h#~k C7χ!X^wTU( 2O*p@ '߉>ү -[I@,2 : v OD~i̒DADM#˜8ž.nXē'ZP,N@O CIfbIOu6fmagkkC n'j(FO'b t]HwI,6PI9cM*=)*nF˰8$( }c6wuY>T @LO,!#v*zu]dzN6hɲ[Dpόr ߅oXhvdS3'e'ʶ( oDӼGϤhoMX0H#XXu~]I )<:Պ(ԾUY<5h V{uIce+sм)fmt=2 ($r͍'( gCC=ad3F=UN`k?>;|9șY&CG(T8ֻ(8 +{x$qơU 8IEqm%~i 5!A~uE^SӮl/#-nxfMnF0dҫ& 'I=ŏ,I[}d[7m`Õ @<acoiqv$0vdpV( { h~(,{{H0\y6! ۨ5i)M֬c.cXt*ASd#5Eq7_ t(I$ Q#0V 0zdž-xGNC%<ד.@=~UEy?|Ew'-6_iPo&d ¬`d<>_;a\]ryCjjIg%~-[򿵴'>_]qgLд}ҬlYwtj!/8aEުp0k\?lϿ{=8px4}w^ChzKMiگ`|yqHKO i:5Uխ# `w"x V< ӟPе+G)d`<n0)" 1mW j?eûOoxnw W_§FƭZ.6)fKy#RZ6X `0p@Q^|)kGeާ^/w;ylHlO+=G_7 mS%-`yc|& j۪ڣ*.{Wy_2|&M·ZwCӭĸuЛ[5p uu'7g xK84[olУu*@:+O|&|Qwzph١:rB cM{YM/?9{*'x헕T,]oh u>$%em XQY TbIn صO(ѼbSCs!2K`G˸{!'G7l/^=ksI6:gT֭#k$Pl 3`QqеmJo7\Z4bFoʦp܁תV=kO] =B<;m'hvח]&χ-k3~ ĺl6` n`@+ς3 opi}gB(#N^ɤͧh6Wr^OooR\ɝ2I9$g?x׍n<+OڠkFgF|h$.f`?sOP_S0(s{`8>٭ksDuV 13><oE <]{6[7yr#gڬXw[ƾ5|Cj] nj6NQ!\߅5o#,Xy ڌ$HOE=(%O`;`xGд[o"WM{s ;£FQ:*?ր=:uhܖ֨|# Vcd`|4$m|C_DqpU,`XA%wG` '~o~zqV_5UNF qu?C~W~1|R'5(01A<,A4v'I╞Y yUi7 @냹e_ S mYbĠ $#s*Vˍ*{k*6ue#*# k%JU#GBhKsKӴ95#!1 m*Fwg#rkjP>*[W ;pHKV4k=„|9(uWaOh u$οڲBq` 7t>,T zxW_);/i><|agPξPP#Q9'65m6gFEKy2u*H#8>8_Ht/x҉+%OH#MI䵵߱`\vsX~$VgB/g} Wm|$f1Ў8?PTz>6!J#x+MFj>qe+l>T3GSW|mCۿGTC/!g+9mnl3g;Ո$j~ u_tH]*rTSydttmKi +xi,U(' z 񟁴_iicE 1>n !e񝤂0@x=@ ?ß}ʷUz"38?$ g'dQVZFaZ0 TUl<7z4^=Kp NmU޴g~%Kic]#,72ӆ u POmֻOz5[Ӭ̺hTKnP%BH|𷌵կM󥴜7  k#MI䵵߱`\vsXo,|GLHM>Z2jpʻn$?B5k՞maUNWg=S$a? Kƚ?*rIɭ1˦?dI7t88b(\g4_§FװW7Oi(t-R{n>lv܍ʒFckO VxcRO:#ixʬaxN3#=%r,{Y.)9#6 0 x |$wt;izFK0.I'Lȡwc maϢ|4QH56'IWE > yoc\ #DVtR!($ qUE81@W7]W>Onޙ+1F ]j[ItWB]4/c_\v.vԂxn⸷9$l]H #׍wߵ~wǑLnw#c ȫ>UxS~eMN9mOy8\ \?2*{] O\ٖ18^6icTPO(.eX6#AEn:GON(/V⯈Ȱ^q`:*?ֻ x Gni&uՕe(#XTmQIe6ǚ"X c*6?^?F|)~4ăE[wXg1>%1( I܅6#;g9g5#:[iՌwvbTYH*z8$t&yUs.{0Op J"x#+RYW RdX/obd0WyFp}Mr ^.JK7w}\"#3"dg)-yVUHX,!cUQF0scŞ мk-i[#aUprp(<ko-ƯH^I$U Ikm{u ۋe:R"mUA\PH"{V4k=„|9(uWiZUevv(I9$I$@VZ8fWh[$a9S#U?%xĚw^k,1x 0;rBQEQEQEQEQEQEQEQEQEQEQEQ\ߍo|WcC/MCP7 EtTE. :$Oæ>6OZMRJP[};+xγ/w۫G "*UI;vT(({c⟋exHv!@ nsR'O7zKI,k3V()al_[zueI4/8<:Պ(i<־ żC˿_iZWswQXI]]iL@8 zVQ^W[RGŝ+Z>wgg'۵-fhׂ"$l2zOV].FCyBy>#ƾ%ꚞAupZ mcY\`':袹F_閚nd邨ke]9ux(ψ7 )/•˴Ձ xl+7.Ք<$]ǨP!xR6)kO20)r>@V~{CeM+[r@,N(B+Y hWzVIjۆ0It~!k7#g K".Z/k1-@Wڀ; ( ( ( ( ( ( ( ( ( OR7U!n(P'Tۊ[cHU I9y(*?mzG,n5?qWZ}07;FFO$T~=#߆EӌcyF@Tp[~f,@bBIoxHU{mZ&x6wfEFj7 ,,-cKGވTzpGP@i V|+h?[n9o)RE.}g8=( o;OCj[̒n,cr~%:~X5GVDĉ$l()'= G!IҬoTkIvfY8=\Xt=.x2=+V<`F!W9$߻ =CZl7t.i쬡$d@q+s?|C}6Z|gՕk2bt VS0 ~ӵUAd-æHq28>/XkoxwSntb8™263Ƞ]oVxM޻nI1\ 8#,U^pƩ儼ЧH$&ԫeOxgWV|TfT)$r$` 1]hc.10E޼wqUdY$1듞'/$:D:~(Km?~s+.dP2a|CI^AzƯ[xúvs[9ɔUam\Euz~#|:Ěo-uݥHzYf 8u|va[P uewMq,z@gL!2<^q~u|AwS.o.6 nX;8Lޯ~>Yp$kpd9ެA$O|'o>a7pKjt'տ~w+|6;c8=묞n巸9 I]H#TP'/m.I]^K<4i4E^[C^K|e> F-f}#]iKaH*ι2HPQ@GmJ&Oj|l2z} ՛}M=!82yHc1SS:'xB@g>~n׵s9OQYHbRgqR A,`0e#P[RI|G"`,`Am c޸9HÚEφg]$[rX$# yJ(~GUl^V`'\g-t+c4 xI״Ju)Ky{$]pF&BoKm=E]IH?bOTyNO?ZΫ6HKOg5P}z%Z܅w;9$5| }?Q^3t [- r6ms{Yo떞 յW]x@ۇʤVa|p1Q@xjvrYNndgIAS>h4~F|̉F[ilxG׼9wXj7Sl{wŇV8):Ex{P\hݡl&lʃ=+W5 3^8Եݢ[kQ)7H )?]y6۝ogu ռG4I#C+ qM/ n|X] P693|o#Y. V{ـ@*},3OLҊ#y7ksnpqvQ@/}GĚw[Y1DlFTσ+CBIMQ%cc,M,oy mj(g16Z d3vұxRo x9.5#kZIeܪ1` ówPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPPKX?TqqPKOXEOEBPS/dcommon/cpyr.htmD Oracle Legal Notices

Oracle Legal Notices

Copyright Notice

Copyright © 1994-2014, Oracle and/or its affiliates. All rights reserved.

Trademark Notice

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group.

License Restrictions Warranty/Consequential Damages Disclaimer

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

Warranty Disclaimer

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

Restricted Rights Notice

If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable:

U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government.

Hazardous Applications Notice

This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.

Third-Party Content, Products, and Services Disclaimer

This software or hardware and documentation may provide access to or information on content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services.

Alpha and Beta Draft Documentation Notice

If this document is in preproduction status:

This documentation is in preproduction status and is intended for demonstration and preliminary use only. It may not be specific to the hardware on which you are using the software. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to this documentation and will not be responsible for any loss, costs, or damages incurred due to the use of this documentation.

Oracle Logo

PK0hPKOXEOEBPS/dcommon/blafdoc.cssc@charset "utf-8"; /* Copyright 2002, 2011, Oracle and/or its affiliates. All rights reserved. Author: Robert Crews Version: 2011.8.12 */ body { font-family: Tahoma, sans-serif; /* line-height: 125%; */ color: black; background-color: white; font-size: small; } * html body { /* http://www.info.com.ph/~etan/w3pantheon/style/modifiedsbmh.html */ font-size: x-small; /* for IE5.x/win */ f\ont-size: small; /* for other IE versions */ } h1 { font-size: 165%; font-weight: bold; border-bottom: 1px solid #ddd; width: 100%; text-align: left; } h2 { font-size: 152%; font-weight: bold; text-align: left; } h3 { font-size: 139%; font-weight: bold; text-align: left; } h4 { font-size: 126%; font-weight: bold; text-align: left; } h5 { font-size: 113%; font-weight: bold; display: inline; text-align: left; } h6 { font-size: 100%; font-weight: bold; font-style: italic; display: inline; text-align: left; } a:link { color: #039; background: inherit; } a:visited { color: #72007C; background: inherit; } a:hover { text-decoration: underline; } a img, img[usemap] { border-style: none; } code, pre, samp, tt { font-family: monospace; font-size: 110%; } caption { text-align: center; font-weight: bold; width: auto; } dt { font-weight: bold; } table { font-size: small; /* for ICEBrowser */ } td { vertical-align: top; } th { font-weight: bold; text-align: left; vertical-align: bottom; } li { text-align: left; } dd { text-align: left; } ol ol { list-style-type: lower-alpha; } ol ol ol { list-style-type: lower-roman; } td p:first-child, td pre:first-child { margin-top: 0px; margin-bottom: 0px; } table.table-border { border-collapse: collapse; border-top: 1px solid #ccc; border-left: 1px solid #ccc; } table.table-border th { padding: 0.5ex 0.25em; color: black; background-color: #f7f7ea; border-right: 1px solid #ccc; border-bottom: 1px solid #ccc; } table.table-border td { padding: 0.5ex 0.25em; border-right: 1px solid #ccc; border-bottom: 1px solid #ccc; } span.gui-object, span.gui-object-action { font-weight: bold; } span.gui-object-title { } p.horizontal-rule { width: 100%; border: solid #cc9; border-width: 0px 0px 1px 0px; margin-bottom: 4ex; } div.zz-skip-header { display: none; } td.zz-nav-header-cell { text-align: left; font-size: 95%; width: 99%; color: black; background: inherit; font-weight: normal; vertical-align: top; margin-top: 0ex; padding-top: 0ex; } a.zz-nav-header-link { font-size: 95%; } td.zz-nav-button-cell { white-space: nowrap; text-align: center; width: 1%; vertical-align: top; padding-left: 4px; padding-right: 4px; margin-top: 0ex; padding-top: 0ex; } a.zz-nav-button-link { font-size: 90%; } div.zz-nav-footer-menu { width: 100%; text-align: center; margin-top: 2ex; margin-bottom: 4ex; } p.zz-legal-notice, a.zz-legal-notice-link { font-size: 85%; /* display: none; */ /* Uncomment to hide legal notice */ } /*************************************/ /* Begin DARB Formats */ /*************************************/ .bold, .codeinlinebold, .syntaxinlinebold, .term, .glossterm, .seghead, .glossaryterm, .keyword, .msg, .msgexplankw, .msgactionkw, .notep1, .xreftitlebold { font-weight: bold; } .italic, .codeinlineitalic, .syntaxinlineitalic, .variable, .xreftitleitalic { font-style: italic; } .bolditalic, .codeinlineboldital, .syntaxinlineboldital, .titleinfigure, .titleinexample, .titleintable, .titleinequation, .xreftitleboldital { font-weight: bold; font-style: italic; } .itemizedlisttitle, .orderedlisttitle, .segmentedlisttitle, .variablelisttitle { font-weight: bold; } .bridgehead, .titleinrefsubsect3 { font-weight: bold; } .titleinrefsubsect { font-size: 126%; font-weight: bold; } .titleinrefsubsect2 { font-size: 113%; font-weight: bold; } .subhead1 { display: block; font-size: 139%; font-weight: bold; } .subhead2 { display: block; font-weight: bold; } .subhead3 { font-weight: bold; } .underline { text-decoration: underline; } .superscript { vertical-align: super; } .subscript { vertical-align: sub; } .listofeft { border: none; } .betadraft, .alphabetanotice, .revenuerecognitionnotice { color: #f00; background: inherit; } .betadraftsubtitle { text-align: center; font-weight: bold; color: #f00; background: inherit; } .comment { color: #080; background: inherit; font-weight: bold; } .copyrightlogo { text-align: center; font-size: 85%; } .tocsubheader { list-style-type: none; } table.icons td { padding-left: 6px; padding-right: 6px; } .l1ix dd, dd dl.l2ix, dd dl.l3ix { margin-top: 0ex; margin-bottom: 0ex; } div.infoboxnote, div.infoboxnotewarn, div.infoboxnotealso { margin-top: 4ex; margin-right: 10%; margin-left: 10%; margin-bottom: 4ex; padding: 0.25em; border-top: 1pt solid gray; border-bottom: 1pt solid gray; } p.notep1 { margin-top: 0px; margin-bottom: 0px; } .tahiti-highlight-example { background: #ff9; text-decoration: inherit; } .tahiti-highlight-search { background: #9cf; text-decoration: inherit; } .tahiti-sidebar-heading { font-size: 110%; margin-bottom: 0px; padding-bottom: 0px; } /*************************************/ /* End DARB Formats */ /*************************************/ @media all { /* * * { line-height: 120%; } */ dd { margin-bottom: 2ex; } dl:first-child { margin-top: 2ex; } } @media print { body { font-size: 11pt; padding: 0px !important; } a:link, a:visited { color: black; background: inherit; } code, pre, samp, tt { font-size: 10pt; } #nav, #search_this_book, #comment_form, #comment_announcement, #flipNav, .noprint { display: none !important; } body#left-nav-present { overflow: visible !important; } } PKr.hcPKOXEOEBPS/dcommon/doccd_epub.jsM /* Copyright 2006, 2012, Oracle and/or its affiliates. All rights reserved. Author: Robert Crews Version: 2012.3.17 */ function addLoadEvent(func) { var oldOnload = window.onload; if (typeof(window.onload) != "function") window.onload = func; else window.onload = function() { oldOnload(); func(); } } function compactLists() { var lists = []; var ul = document.getElementsByTagName("ul"); for (var i = 0; i < ul.length; i++) lists.push(ul[i]); var ol = document.getElementsByTagName("ol"); for (var i = 0; i < ol.length; i++) lists.push(ol[i]); for (var i = 0; i < lists.length; i++) { var collapsible = true, c = []; var li = lists[i].getElementsByTagName("li"); for (var j = 0; j < li.length; j++) { var p = li[j].getElementsByTagName("p"); if (p.length > 1) collapsible = false; for (var k = 0; k < p.length; k++) { if ( getTextContent(p[k]).split(" ").length > 12 ) collapsible = false; c.push(p[k]); } } if (collapsible) { for (var j = 0; j < c.length; j++) { c[j].style.margin = "0"; } } } function getTextContent(e) { if (e.textContent) return e.textContent; if (e.innerText) return e.innerText; } } addLoadEvent(compactLists); function processIndex() { try { if (!/\/index.htm(?:|#.*)$/.test(window.location.href)) return false; } catch(e) {} var shortcut = []; lastPrefix = ""; var dd = document.getElementsByTagName("dd"); for (var i = 0; i < dd.length; i++) { if (dd[i].className != 'l1ix') continue; var prefix = getTextContent(dd[i]).substring(0, 2).toUpperCase(); if (!prefix.match(/^([A-Z0-9]{2})/)) continue; if (prefix == lastPrefix) continue; dd[i].id = prefix; var s = document.createElement("a"); s.href = "#" + prefix; s.appendChild(document.createTextNode(prefix)); shortcut.push(s); lastPrefix = prefix; } var h2 = document.getElementsByTagName("h2"); for (var i = 0; i < h2.length; i++) { var nav = document.createElement("div"); nav.style.position = "relative"; nav.style.top = "-1.5ex"; nav.style.left = "1.5em"; nav.style.width = "90%"; while (shortcut[0] && shortcut[0].toString().charAt(shortcut[0].toString().length - 2) == getTextContent(h2[i])) { nav.appendChild(shortcut.shift()); nav.appendChild(document.createTextNode("\u00A0 ")); } h2[i].parentNode.insertBefore(nav, h2[i].nextSibling); } function getTextContent(e) { if (e.textContent) return e.textContent; if (e.innerText) return e.innerText; } } addLoadEvent(processIndex); PKo"nR M PKOXEOEBPS/security3.htm Enterprise Manager Security

2 Enterprise Manager Security

This chapter describes how to configure Oracle Enterprise Manager Security. Specifically, this chapter contains the following sections:

About Oracle Enterprise Manager Security

Oracle Enterprise Manager provides tools and procedures to help you ensure that you are managing your Oracle environment in a secure manner. The goals of Oracle Enterprise Manager security are:

  • To be sure that only users with the proper privileges have access to critical monitoring and administrative data.

    This goal is met by requiring username and password credentials before users can access the Enterprise Manager consoles and appropriate privileges for accessing the critical data.

  • To be sure that all data transferred between Enterprise Manager components is transferred in a secure manner and that all data gathered by each Oracle Management Agent can be transferred only to the Oracle Management Service for which the Management Agent is configured.

    This goal is met by enabling Enterprise Manager Framework Security. Enterprise Manager Framework Security automates the process of securing the Enterprise Manager components installed and configured on your network.

  • To be sure that sensitive data such as credentials used to access target servers are protected.

    This goal is met by Enterprise Manager's encryption support. The sensitive data is encrypted with an emkey. By following the best practice, even the repository owner and the SYSDBA will not be able to access the sensitive data.

  • To be sure that access to managed targets is controlled through user authentication and privilege delegation.

    This goal is met by configuring the Management Agent with PAM and LDAP for user authentication and using privilege delegation tools like Sudo and PowerBroker.

Enterprise Manager Authentication

Grid Control Authentication is the process of determining the validity of the user accessing Enterprise Manager Grid Control. The authentication feature is available across the different user interfaces such as Enterprise Manager Grid Control console and Enterprise Manager Command Line Interface.

The following authentication schemes are available:

  • Repository-Based Authentication: This is the default authentication option. An Enterprise Manager administrator is also a repository (database) user. By using this option, you can take advantage of all the benefits that this authentication method provides like password control via password profile, enforced password complexity, password life time, number of failed attempts allowed and controls. During the password grace period, the administrator is prompted to change the password but when the password has expired, it must be changed.For more details, refer to Repository-Based Authentication.

  • SSO-Based Authentication: The single sign-on based authentication provides strengthened and centralized user identity management across the enterprise. After you have configured Enterprise Manager to use the Oracle Application Server Single Sign-On, you can register any single sign-on user as an Enterprise Manager administrator. You can then enter your single sign-on credentials to access the Oracle Enterprise Manager Grid Control console. For more details, refer to Single Sign-On Based Authentication.

  • Enterprise User Security Based Authentication: The Enterprise User Security (EUS) option enables you to create and store enterprise users and roles for the Oracle database in an LDAP-compliant directory server. Once the repository is configured with EUS, you can configure Enterprise Manager to use EUS as its authentication mechanism as described in Enterprise User Security Based Authentication. You can register any EUS user as an Enterprise Manager administrator.

    EUS helps centralize the administration of users and roles across multiple databases. If the managed databases are configured with EUS, the process of logging into these databases is simplified. When you drill down to a managed database, Enterprise Manager will attempt to connect to the database using Enterprise Manager Grid Control credentials. If successful, Enterprise Manager will directly connect you to the database without displaying a login page.

Repository-Based Authentication

Enterprise Manager Grid Control allows you to create and manage new administrator accounts. Each administrator account includes its own login credentials as well as a set of roles and privileges that are assigned to the account. You can also assign a password profile to the administrator. To create, edit, or view an administrator account:

  1. From Enterprise Manager Grid Control, click Setup.

  2. Click Administrators in the vertical navigation bar.

  3. Click the appropriate task button on the Administrators page. The following screen is displayed:

Figure 2-1 Create / Edit Administrator

Creating and Editing Administrators

On this page, you can specify the type of administrator account being created, select the password profile, and the password expiry period. The password cannot be changed by the administrator if the Prevent Password Change checkbox is selected.

If you select the Expire Password Now checkbox, the password for administrator account will be set to an expired state. If the password has expired, when you login the next time, the following screen is displayed and you are prompted to change the password.

Figure 2-2 Password Expiry Page

Surrounding text describes Figure 2-2 .

Enter your current password and the new password and click Apply. You can now start using Enterprise Manager Grid Control.

Single Sign-On Based Authentication

If you are currently using Oracle Application Server Single Sign-On to control access and authorization for your enterprise, you can extend those capabilities to the Grid Control Console.

By default, when you navigate to the Grid Control Console, Enterprise Manager displays the Enterprise Manager login page. However, you can configure Enterprise Manager so it uses Oracle Application Server Single Sign-On to authenticate your Grid Control Console users. Instead of seeing the Enterprise Manager login page, Grid Control Console users will see the standard Oracle Application Server Single Sign-On login page. From the login page, administrators can use their Oracle Application Server Single Sign-On credentials to access the Oracle Enterprise Manager 10g Grid Control Console.


Note:

  • You can configure Enterprise Manager Grid Control to either use Oracle Application Server Single Sign-On or the Enterprise User Security features. You cannot use both options at the same time.

  • When Enterprise Manager is configured to use Single Sign-On with Server Load Balancer, make sure that the correct monitoring settings have been defined. For details, refer to the chapter on Grid Control Common Configurations.


The following sections describe how to configure Enterprise Manager as an OracleAS Single Sign-On Partner Application:

Registering Enterprise Manager as a Partner Application

To register Enterprise Manager as a partner application manually, follow these steps:

  1. Enter the following URL to navigate to the SSO Administration page.

    http://sso_host:sso_port/pls/orasso
    
  2. Login as orcladmin user and click SSO Administration.

  3. Click Administer Partner Applications and then click Add Partner Application.

  4. Enter the following information on the Add Partner Application page.

    Name: <EMPartnerName>
    Home URL: protocol://em_host:em_port
    Success URL: protocol://em_host:em_port/osso_login_success 
    Logout URL: protocol://em_host:em_port/osso_logout_success
    Administrator Email: user@host.com
    

    where host, port, and protocol refer to the EM Host, port and the protocol (http or https) used.

  5. After entering these details, click Edit <EMPartnerName> and enter the following parameters to generate the osso.txt. Sample values for these parameters are shown below:

    sso_server_version: v1.2
    cipher_key: <EncryptionKeyValue>
    site_id: <IDValue>
    site_token: <TokenValue>
    login_url: protocol://sso_host:sso_port/pls/orasso/orasso.wwsso_app_admin.ls_login
    logout_url=protocol://sso_host:sso_port/pls/orasso/orasso.wwsso_app_admin.ls_logout
    cancel_url=protocol://em_host:em_port
    sso_timeout_cookie_name=SSO_ID_TIMEOUT
    sso_timeout_cookie_key=9E231B3C1A3A808A
    
  6. Enter the following command to generate the osso.conf file:

    WEBTIER_HOME/ohs/bin/iasobf osso.txt osso.conf root 
    
  7. Use the osso.conf file and configure it as necessary using the emctl command as follows:

    emctl config oms sso -ossoconf <ossoconf file> -dasurl <dasurl> [-unsecure] [-sysman_pwd <pwd>] [-domain <domain>]
    

    where:

    • -ossoconf is the path to the osso.conf file

    • -dasurl is the URL specifying the host and port for the Delegated Administration Service (DAS). Generally, the DAS host name and port are the same as the host name and port of the Oracle Application Server Single Sign-On server. For example:

      http://mgmthost1.acme.com:7777

    • -unsecure is used to register the http port with the single sign-on server.

    • -sysman_pwd is the sysman user password. If this parameter is not specified, you will be prompted to enter it.

    • -domain is the name of the host domain. This parameter needs to be specified if the fully qualified name of the host is not available.

    The sample output for this command is shown below:

    emctl config oms sso -ossoconf /tmp/osso.conf -dasurl http://somehost.domain.com:7777
    Oracle Enterprise Manager 11g Release 1 Grid Control
    Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
    Enter SYSMAN user password :
    SSO Configuration done successfully, Please restart OMS.
    
  8. Restart WebTier and OMS as follows:

    emctl stop oms
    emctl start oms
    

Removing Single Sign-On Configuration

To remove the single sign-on configuration, run the following command:

emctl config oms sso -remove [-sysman_pwd <pwd>]

where -sysman_pwd is the sysman repository password.

Example 2-1 Sample Output of the emctl config oms -remove command

emctl config oms sso -remove 
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Enter SYSMAN user password :
SSO Configuration removed successfully, Please restart OMS.
Restart OMS using
emctl stop oms
emctl start oms

Registering Single Sign-On Users as Enterprise Manager Administrators

After you have configured Enterprise Manager to use the Single Sign-On logon page, you can register any Single Sign-On user as an Enterprise Manager administrator. You can register single sign-on users using:

  • Enterprise Manager Grid Control Graphical User Interface

  • Enterprise Manager Grid Control Command Line Interface

Registering Single Sign-On Users Using the Graphical User Interface

You can use the graphical user interface to register single sign-on users by following these steps:

  1. Go the Enterprise Manager Grid Control Console URL.

    For example:

    http://mgmthost1.acme.com:7777/em
    

    The browser is redirected to the standard Single Sign-On Logon page.

  2. Enter the credentials for a valid Single Sign-On user.

    If the Single Sign-On user is not an Enterprise Manager administrator, the browser is redirected to a modified version of the Enterprise Manager logon page (Figure 2-3).

  3. Log in to Enterprise Manager as a Super Administrator.

  4. Click Setup and then click Administrators to display the Administrators page.


    See Also:

    "Creating, Editing, and Viewing Administrators" in the Enterprise Manager online Help

    Because Enterprise Manager has been configured to use Single Sign-On, the first page in the Create Administrator wizard now offers you the option of creating an administrator based on a registered Oracle Internet Directory user.

  5. Select Oracle Internet Directory and advance to the next page in the wizard.

  6. Enter the name and e-mail address of the Oracle Internet Directory user, or click the flashlight icon to search for a user name in the Oracle Internet Directory.

  7. Use the rest of the wizard pages to define the roles, system privileges, and other characteristics of the Enterprise Manager administrator and then click Finish.

    Enterprise Manager displays a summary page that lists the characteristics of the administrator account.

  8. Click Finish to create the new Enterprise Manager administrator.

    The OID user is now included in the list of Enterprise Manager administrators. You can now verify the account by logging out of the Grid Control Console and logging back in using the OID user credentials on the Single Sign-On logon page.

Figure 2-3 Modified Enterprise Manager Logon Page When Configuring SSO

Modified Enterprise Manager Logon Page When Configuring SSO

Figure 2-4 Create Administrator Page When SSO Support Is Enabled

Create Administrator Page When SSO Support Is Enabled

Registering Single Sign-On Users Using EMCLI

s

You can use the following EMCLI command to create Single Sign-On users:

emcli create_user -name=ssouser -type=EXTERNAL_USER

This command creates a user with the name ssouser who is authenticated against the single sign-on user.

ArgumentDescription
-nameName of the administrator.
-typeThe type of user. The default value for this parameter is EM_USER. The other possible values are:
  • EXTERNAL_USER: Used for single-sign-on based authentication.

  • DB_EXTERNAL_USER: Used for enterprise user based security authentication.

-passwordThe password for the administrator.
-rolesThe list of roles that can be granted to this administrator.
-emailThe list of email addresses for this administrator.
-privilegeThe system privileges that can be granted to the administrator. This option can be specified more than once.
-profileThe name of the database profile. This is an optional parameter. The default profile used is DEFAULT.
-descThe description of the user being added.
-expiredThis parameter is used to set the password to "expired" status. This is an optional parameter and is set to False by default.
-prevent_change_passwordWhen this parameter is set to True, the user cannot change the password. This is an optional parameter and is set to False by default.
input_fileThis parameter allows the administrator to provide the values for any of these arguments in an input file. The format of value is name_of_argument:file_path_with_file_name.

Example 1

emcli create_user         -name="new_admin"         -password="oracle"         -email="first.last@oracle.com;joe.shmoe@shmoeshop.com"         -roles="public"         -privilege="view_job;923470234ABCDFE23018494753091111"         -privilege="view_target;<host>.com:host" 

This example creates an Enterprise Manager administrator named new_admin. This administrator has two privileges: the ability to view the job with ID 923470234ABCDFE23018494753091111 and the ability to view the target <host>.com:host. The administrator new_admin is granted the PUBLIC role.

Example 2

   emcli create_user         -name="User1"         -type="EXTERNAL_USER"         -input_file="privilege:/home/user1/priv_file"         Contents of priv_file are:           view_target;<host>.com:host

This example makes user1 which has been created externally as an Enterprise Manager user. user1 will have view privileges on <host>.com:host.

Example 3

   emcli create_user         -name="User1"         -desc="This is temp hire."
         -prevent_change_password="true"         -profile="MGMT_ADMIN_USER_PROFILE

This example sets user1 as an Enterprise Manager user with some description. The prevent_change_password is set to true to indicate that the password cannot be changed by user1 and the profile is set to MGMT_ADMIN_USER_PROFILE.

Example 4

   emcli create_user         -name="User1"         -desc="This is temp hire."         -expire="true" 

This example sets user1 as an Enterprise Manager with some description. Since the password is set to expire immediately, when the user logs in for the first time, he is prompted to change the password.

Grid Control as a Single Sign-On Partner Application

The emctl config oms sso command adds the Oracle Enterprise Manager Grid Control Console as an Oracle Application Server Single Sign-On partner application. Partner applications are those applications that have delegated authentication to the Oracle Application Server Single Sign-On Server.

To see the list of partner applications, navigate to the following URL:

http://hostname:port/pls/orasso/orasso.home

For example:

http://ssohost1.acme.com:7777/pls/orasso/orasso.home

Bypassing the Single Sign-On Logon Page

After you configure Enterprise Manager to use the Single Sign-On logon page, you can bypass the Single Sign-On page at any time and go directly to the Enterprise Manager logon page by entering the following URL:

http://hostname.domain:port/em/console/logon/logon

For example:

http://mgmthost1.acme.com:7777/em/console/logon/logon

Enterprise User Security Based Authentication

Enterprise User Security enables you to create and store Oracle database information as directory objects in an LDAP-compliant directory server. For example, an administrator can create and store enterprise users and roles for the Oracle database in the directory, which helps centralize the administration of users and roles across multiple databases.


See Also:

Enterprise User Security Configuration Tasks and Troubleshooting in the Oracle Database Advanced Security Administrator's Guide

If you currently use Enterprise User Security for all your Oracle databases, you can extend this feature to Enterprise Manager. Configuring Enterprise Manager for use with Enterprise User Security simplifies the process of logging in to database targets you are managing with the Oracle Enterprise Manager Grid Control Console.

To configure Enterprise Manager for use with Enterprise User Security:

  1. Ensure that you have enabled Enterprise User Security for your Oracle Management Repository database, as well as the database targets you will be managing with the Grid Control Console. Refer to Oracle Database Advanced Security Administrator's Guide for details.

  2. Using the emctl set property command, set the following properties:

    oracle.sysman.emSDK.sec.DirectoryAuthenticationType=EnterpriseUser
    oracle.sysman.emSDK.sec.eus.Domain=<ClientDomainName> (For example:mydomain.com)
    oracle.sysman.emSDK.sec.eus.DASHostUrl=<das_url> (For example:
    oracle.sysman.emSDK.sec.eus.DASHostUrl=http://my.dashost.com:7777 )
    

    For example:

    emctl set property -name oracle.sysman.emSDK.sec.DirectoryAuthenticationType -value EnterpriseUser
    
  3. Change directory to the ORACLE_HOME/sysman/config directory and open the emoms.properties file with your favorite text editor.

  4. Add the following entries in the emoms.properties file:

    oracle.sysman.emSDK.sec.DirectoryAuthenticationType=EnterpriseUser
    oracle.sysman.emSDK.sec.eus.Domain=<ClientDomainName> (For example: mydomain.com)
    oracle.sysman.emSDK.sec.eus.DASHostUrl=<das_url> (For example: oracle.sysman.emSDK.sec.eus.DASHostUrl=http://my.dashost.com:7777 )
    
  5. Save and close the emoms.properties file.

  6. Stop the Oracle Management Service.

  7. Start the Management Service.

The next time you use the Oracle Enterprise Manager Grid Control Console to drill down to a managed database, Enterprise Manager will attempt to connect to the database using Enterprise User Security. If successful, Enterprise Manager will connect you to the database without displaying a login page. If the attempt to use Enterprise User Security fails, Enterprise Manager will prompt you for the database credentials.

Registering Enterprise Users as Enterprise Manager Users

After you have configured Enterprise Manager to use Enterprise Users, you can register existing enterprise users as Enterprise Manager Users and grant them the necessary privileges so that they can manage Enterprise Manager effectively.

You can register existing enterprise users by using:

  • Enterprise Manager Grid Control Graphic User Interface

  • Enterprise Manager Command Line Interface

Registering Enterprise Users Using the Graphical User Interface

You can use the graphical user interface to register enterprise users by following these steps:

  1. Log into Enterprise Manager as a Super Administrator.

  2. Click Setup and then click Administrators to display the Administrators page. Since Enterprise Manager has been configured to use Enterprise Users, the first page of the Create Administrator wizard will provide the option to create an administrator based on a registered Oracle Internet Directory user or a normal database user.

  3. Select Oracle Internet Directory and click Continue to go to the next page in the wizard.

  4. Enter the name and e-mail address of the Oracle Internet Directory user or click the flashlight icon to search for a user name in the Oracle Internet Directory.

  5. Use the rest of the wizard pages to define the roles, system privileges, and other characteristics of the Enterprise Manager administrator and then click Finish. Enterprise Manager displays a summary page that lists the characteristics of the administrator account.

  6. Click Finish to create the new Enterprise Manager administrator.

    The OID user is now included in the list of Enterprise Manager administrators. You can now verify the account by logging out of the Grid Control Console and logging back in using the OID user credentials on the Single Sign-On logon page.

Registering Enterprise Users Using the Command Line Interface

To register Enterprise Users as Enterprise Manager users using EMCLI, enter the following command:

emcli create_user -name=eususer -type=DB_EXTERNAL_USER

This command registers the eususer as an Enterprise Manager user where eususer is an existing Enterprise User. For more details, refer to Registering Single Sign-On Users Using EMCLI.

Enterprise Manager Authorization

System security is a major concern of any corporation. Giving the same level of access to all systems to all administrators is dangerous, but individually granting access to tens, hundreds, or even thousands of targets to every new member of the group is time consuming. With Enterprise Manager's administrator privileges and roles feature, this task can be performed within seconds, instead of hours. Authorization controls the access to the secure resources managed by Enterprise Manager via system, target, and object level privileges and roles.

This section describes Enterprise Manager's Authorization model including user classes, roles, and privileges assigned to each user class. The following topics are described:

  • Classes of Users

  • Privileges and Roles

Classes of Users

Oracle Enterprise Manager supports different classes of Oracle users, depending upon the environment you are managing and the context in which you are using Oracle Enterprise Manager Grid Control.

The Enterprise Manager administrators you create and manage in the Grid Control Console are granted privileges and roles to log in to the Grid Control Console and to manage specific target types and to perform specific management tasks. The default super administrator for the Grid Control Console is the SYSMAN user, which is a database user associated with the Oracle Management Repository. You define the password for the SYSMAN account during the Enterprise Manager installation procedure.

By restricting access to privileged users and providing tools to secure communications between Oracle Enterprise Manager 11g components, Enterprise Manager protects critical information in the Oracle Management Repository.

The Management Repository contains management data that Enterprise Manager Grid Control uses to help you monitor the performance and availability of your entire enterprise. This data provides you with information about the types of hardware and software you have deployed, as well as the historical performance and specific characteristics of the applications, databases, applications servers, and other targets that you manage. The Management Repository also contains information about the Enterprise Manager administrators who have the privileges to access the management data.

You can create and manage Enterprise Manager administrator accounts. Each administrator account includes its own login credentials, as well as a set of roles and privileges that are assigned to the account. There are three administrator access categories:

  • Super Administrator: Powerful Enterprise Manager administrator with full access privileges to all targets and administrator accounts within the Enterprise Manager environment. The Super Administrator, SYSMAN is created by default when Enterprise Manager is installed. The Super Administrator can create other administrator accounts.

  • Administrator: Regular Enterprise Manager administrator.

  • Repository Owner: Database administrator for the Management Repository. This account cannot be modified, duplicated, or deleted.

The types of management tasks that the administrator can perform and targets that he can access depends on the roles, system privileges, and target privileges that he is granted. The Super Administrator can choose to let certain administrators perform only certain management tasks, or access only certain targets, or perform certain management tasks on certain targets. In this way, the Super Administrator can divide the workload among his administrators.

Privileges and Roles

User privileges provide a basic level of security in Enterprise Manager. They are designed to control user access to data and to limit the kinds of SQL statements that users can execute. When creating a user, you grant privileges to enable the user to connect to the database, to run queries and make updates, to create schema objects, and more.

When Enterprise Manager is installed, the SYSMAN user (super administrator) is created by default. The SYSMAN Super Administrator then creates other administrator accounts for daily administration work. The SYSMAN account should only be used to perform infrequent system wide, global configuration tasks.The Super Administrator divides workload among his administrators by filtering target access, or filtering access to management task, or both through the roles, System Privileges, and Target Privileges he grants them. For example, he can allow some administrators to view any target and to add any target in the enterprise and other administrators to only perform specific operations such as maintaining and cloning on a target for which they are responsible.

A role is a collection of Enterprise Manager system privileges, or target privileges, or both, which you can grant to administrators or to other roles. These roles can be based upon geographic location (for example, a role for Canadian administrators to manage Canadian systems), line of business (for example, a role for administrators of the human resource systems or the sales systems), or any other model. Administrators do not want to perform the task of individually granting access to tens, hundreds, or even thousands of targets to every new member of their group.By creating roles, an administrator needs only to assign the role that includes all the appropriate privileges to his team members instead of having to grant many individual privileges. He can divide workload among his administrators by filtering target access, or filtering access to management task, or both.

Public Role: Enterprise Manager creates one role by default called Public. This role is unique in that it is automatically assigned to all new non-super administrators when they are created. By default it has no privileges assigned to it. The Public role should be used to define default privileges you expect to assign to a majority of non-super administrators you create. Privileges need not be assigned to Public initially - they can be added at any time. The role may be deleted if your enterprise does not wish to use it. If deleted, it can be added back in later if you later decide to implement it.

Granting Privileges

A privilege is a right to perform management actions within Enterprise Manager. Privileges can be divided into three categories:

  • System Privileges

  • Target Privileges

  • Object Privileges

System Privileges: These privileges allow a user to perform system wide operations. To set the System Privileges, click the Setup link to navigate to the Setup Overview page and click on the Administrators option in the left panel. Select an administrator from the list and click Edit. The Edit Administrator wizard is displayed. Click Next to navigate through the wizard to see the System Privileges page:

Figure 2-5 System Privileges

Surrounding text describes Figure 2-5 .

Table 2-1 System Privileges

System PrivilegeDescription

USE ANY BEACON

Allows the administrator to use any Beacon on any monitored host to monitor transactions, URLs, and network components.

ADD ANY TARGET

Allows the administrator to add any target to Enterprise Manager for monitoring, administration and management.

VIEW ANY TARGET

Allows the administrator to view any target on the system, including Oracle Management Agents and Management Services.Whenever the VIEW ANY TARGET privilege is granted, the MONITOR ENTERPRISE MANAGER privilege is also granted by default.

CREATE PRIVILEGE PROPAGATING GROUP

Allows the administrator to create privilege propagating groups. Privileges granted to such groups will be automatically granted to all members of the group.

MONITOR ENTERPRISE MANAGER

Allows the administrator to monitor the availability and performance of Enterprise Manager itself, and grants the administrator access to the following targets: the database used for the Management Repository, the Management Service and Management Repository, and all Oracle Management Agents in the global enterprise.

PUBLISH REPORT

Allows the administrator to publish reports for public use.

JVM Diagnostics Administrator

Allows the administrator to manage JVM Diagnostics operations.

JVM Diagnostics User

Allows the user to view JVM Diagnostics data.

Request Monitoring Administrator

Allows the user to manage E2E administrator operations.

Request Monitoring User

Allows the user to view E2E data.


Select the check box to select the system privilege to be granted to the administrator and click Next. The Target Privileges page is displayed.

Target Privileges: These privileges allow an administrator to perform operations on a target. The Target Privileges page shows a list of targets for which privileges can be granted. Select a target from the list and click the pencil icon in the Privilege column. The following screen is displayed.

Figure 2-6 Target Privileges

Surrounding text describes Figure 2-6 .

Select the check box to specify the privileges that are to be granted and click Continue. For more details on setting these privileges, see the Enterprise Manager Online Help.


Note:

If a target has certain privileges, *** need info ***

Table 2-2 Target Privileges

Target PrivilegeDescription

FULL

Implicitly grants all the target privileges and allows the administrator to delete the target from the Enterprise Manager system.

OPERATOR

Allows the administrator to perform normal administrative operations on a target such as configure a blackout, or edit the properties.

BLACKOUT TARGET

Allows the administrator to create, edit, schedule, and stop blackout on a target.

MANAGE TARGET ALERTS

Allows the administrator to clear stateless alerts, manually re-evaluate alerts and acknowledge alerts for the target.

CONFIGURE TARGET

Allows the administrator to edit target properties and modify monitoring configurations.

MANAGE TARGET METRICS

Allows the administrator to edit thresholds for metric and policy settings, apply monitoring templates and manage user defined metrics.

VIEW

Allows the administrator to view properties, inventory and monitor information about a target.


Object Privileges: These privileges allow an administrator to perform a particular action on a specific schema object. Different object privileges are available for different types of schema objects.

Table 2-3 Object Privileges

Target PrivilegeDescription

VIEW JOB

Provides the administrator with the ability to view the job and its definition.

FULL JOB

Provides the administrator with the ability to view, edit, submit, and delete the job.

VIEW REPORT

Provides the administrator with the ability to view the report.

VIEW TEMPLATE

The ability to view the template definition.

FULL TEMPLATE

The ability to edit the template definition.


Configuring Security for Grid Control

This section contains the following topics:

About Enterprise Manager Framework Security

Enterprise Manager Framework Security provides safe and secure communication channels between the components of Enterprise Manager. For example, Framework Security provides secure connections between your Oracle Management Service and its Management Agents.


See Also:

Oracle Enterprise Manager Concepts for an overview of Enterprise Manager components

The following figure shows how Enterprise Manager Framework Security provides security for the connections between the Enterprise Manager components.

Figure 2-7 Enterprise Manager Framework Security

Enterprise Manager Framework Security

Enterprise Manager Framework Security implements the following types of secure connections between the Enterprise Manager components:

  • HTTPS and Public Key Infrastructure (PKI) components, including signed digital certificates, for communications between the Management Service and the Management Agents.


    See Also:

    Oracle Security Overview for an overview of Public Key Infrastructure features, such as digital certificates and public keys

  • Oracle Advanced Security for communications between the Management Service and the Management Repository.


    See Also:

    Oracle Database Advanced Security Administrator's Guide

Overview of the Steps Required to Enable Enterprise Manager Framework Security

To enable Enterprise Manager Framework Security, you must configure each of the Enterprise Manager components in a specific order. The following list outlines the process for securing the Management Service and the Management Agents that upload data to the Management Service:


Note:

The Enterprise Manager components are configured during installation. You can use the following commands if you want to reconfigure any of the components.

  1. Use the emctl stop oms command to stop the OMS and the WebTier.

  2. Use emctl secure oms to enable security for the Management Service.

  3. Restart the OMS and the WebTier using the emctl start oms command.

  4. For each Management Agent, stop the Management Agent, use the emctl secure agent command to enable security for the Management Agent, and restart the Management Agent.

  5. After security is enabled for all the Management Agents, use the emctl secure lock command to restrict HTTP Access to the Management Service. This will ensure that Management Agents for which security has not been enabled will not be able upload data to the Management Service.

The following sections describe how to perform each of these steps in more detail.


Note:

To resolve errors from emctl secure operations, refer to EM_INSTANCE_HOME/sysman/log/secure.log for more details.

Enabling Security for the Oracle Management Service

To enable Enterprise Manager Framework Security for the Management Service, you use the emctl secure oms utility, which is located in the following subdirectory of the Management Service home directory:

ORACLE_HOME/bin

The emctl secure oms utility performs the following actions:

  • Generates a Root Key within your Management Repository. The Root Key is used during distribution of Oracle Wallets containing unique digital certificates for your Management Agents.

  • Modifies your WebTier to enable an HTTPS channel between your Management Service and Management Agents, independent from any existing HTTPS configuration that may be present in your WebTier.

  • Enables your Management Service to accept requests from Management Agents using Enterprise Manager Framework Security.

To run the emctl secure oms utility you must first choose an Agent Registration Password. The Agent Registration password is used to validate that future installation sessions of Oracle Management Agents and Oracle Management Services are authorized to load their data into this Enterprise Manager installation.

To enable Enterprise Manager Framework Security for the Oracle Management Service:

  1. Stop the Management Service, the WebTier, and the other application server components using the following command:

    OMS_ORACLE_HOME/bin/emctl stop oms
    
  2. Enter the following command:

    OMS_ORACLE_HOME/bin/emctl secure oms
    
  3. You will be prompted for the Enterprise Manager Root Password. Enter the SYSMAN password.

  4. You will be prompted for the Agent Registration Password, which is the password required for any Management Agent attempting to secure with the Management Service. Specify an Agent Registration Password for the Management Service.

  5. When the operation is complete, restart the WebLogic Server and the deployed Enterprise Manager application.

    OMS_ORACLE_HOME/bin/emctl start oms
    
  6. After the Management Service restarts, test the secure connection to the Management Service by browsing to the following secure URL using the HTTPS protocol:

    https://hostname.domain:https_upload_port/em
    

    For example:

    https://mgmthost1.acme.com:1159/em
    

    If the Management Service security has been enabled, your browser displays the Enterprise Manager Login page.


Note:

The 1159 port number is the default secure port used by the Management Agents to upload data to the Management Service. This port number may vary if the default port is unavailable.

Example 2-2 Sample Output of the emctl secure oms Command

emctl secure oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Securing OMS... Started.Securing OMS... Successful

Alternatively, you can enter the emctl secure oms command all on one line, but if you enter the command on one line, the passwords you enter will be displayed on the screen as you type the command.

Example 2-3 Usage of the emctl secure oms Command (II)

emctl secure oms [-sysman_pwd <sysman password>] [-reg_pwd <registration password>] [-host <hostname>] [-slb_port <slb port>] [-slb_console_port <slb console port>] [-reset] [-console] [-lock] [-lock_console] [-secure_port <secure_port>] [-upload_http_port <upload_http_port>] [-root_dc <root_dc>] [-root_country <root_country>] [-root_email <root_email>] [-root_state <root_state>] [-root_loc <root_loc>] [-root_org <root_org>] [-root_unit <root_unit>] [-wallet <wallet_loc> -trust_certs_loc <certs_loc>] [-wallet_pwd <pwd>] [-key_strength <strength>] [-cert_validity <validity>] [-protocol <protocol>] 
Valid values for <protocol> are the allowed values for Apache's SSLProtocol directive

The parameters are explained below:

  • sysman_pwd - Oracle Management Repository user password.

  • reg_pwd - The Management Agent registration password.

  • host - The host name to be used in the certificate used by the Oracle Management Service. You may need to use the SLB host name if there is an SLB before the Management Service.

  • reset - A new certificate authority will be created. All the Agents and Oracle Management Services need to be resecured.

  • secure_port - The port to be used for secure communication.

  • upload_http_port - The port used for unsecure upload communications.

  • slb_port - This parameter is required when Server Load Balancer is used. It specifies the secure upload port configured in the Server Load Balancer.

  • slb_console_port - This parameter is required when Server Load Balancer is used. It specifies the secure console port configured in the Server Load Balancer.

  • root_dc - The domain component used in the root certificate. The default value is com.

  • root_country - The country to be used in the root certificate. The default value is US.

  • root_state - The state to be used in the root certificate. The default value is CA.

  • root_loc - The location to be used in the root certificate. The default value is EnterpriseManager on <hostname>.

  • root_org - The organization name to be used in the root certificate. The default value is EnterpriseManager on <hostname>.

  • root_unit - The organizational unit to be used in the root certificate. The default value is EnterpriseManager on <hostname>.

  • root_email - The email address to be used in the root certificate. The default value is EnterpriseManager@<hostname>.

  • wallet: This is the location of the wallet containing the third party certificate. This parameter should be specified while configuring third party certificates.

  • trust_certs_loc - The location of the trusted_certs.txt (required when third party certificates are used).

  • key_strength: The strength of the key to be used. Valid values are 512, 1024, 2048, and 4096.

  • cert_validity: The number of days for which the self-signed certificate is valid. The valid range is between 1 to 3650.

  • protocol: This parameter is used to configure Oracle Management Service in TLSv1-only or SSLv3-only or mixed mode (default). Valid values are the allowed values as per Apache's SSLProtocol directive.


Note:

The key_strength and cert_validity parameters are applicable only when the -wallet option is not used.

Checking the Security Status

You can check whether security has been enabled for the Management Service by entering the emctl status oms -secure command.

Example 2-4 Sample Output of the emctl status oms -details Command

emctl status oms -details [-sysman_pwd <pwd>]
Oracle Enterprise Manager 11g Release 1 Grid Control  
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password : 
Console Server Host : omshost.mydomain.com
HTTP Console Port   : 7788
HTTPS Console Port  : 7799
HTTP Upload Port    : 4889
HTTPS Upload Port   : 1159
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1

Creating a New Certificate Authority

You may need to create a new Certificate Authority (CA) if the current CA is expiring or if you want to change the key strength. A unique identifier is assigned to each CA. For instance, the CA created during installation may have an identifier as ID 1, subsequent CAs will have the IDs 2,3, and so on. At any given time, the last created CA is active and issues certificates for OMSs and Agents.

Example 2-5 Creating a New Certificate Authority

emctl secure createca [-sysman_pwd <pwd>] [-host <hostname>] [-key_strength<strength>] [-cert_validity <validity>] [-root_dc <root_dc>] [-root_country <root_country>] [-root_email <root_email>] [-root_state <root_state>] [-root_loc <root_loc>] [-root_org <root_org>] [-root_unit <root_unit>] 
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Creating CA... Started.
Successfully created CA with ID 2

Example 2-6 Viewing Information about a Certificate Authority

emcli get_ca_info -ca_id="1;2" -details
Info about CA with ID: 1
CA is not configured
DN: CN=myhost.mydomain.com, C=US
Serial# : 3423643907115516586
Valid From: Tue Mar 16 11:06:20 PDT 2010
Valid Till: Sat Mar 14 11:06:20 PDT 2020
Number of Agents registered with CA ID 1 is 1
myhost.mydomain.com:3872

Info about CA with ID: 2
CA is configured
DN: CN=myhost.mydomain.com, C=US, ST=CA
Serial# : 1182646629511862286
Valid From: Fri Mar 19 05:17:15 PDT 2010
Valid Till: Tue Mar 17 05:17:15 PDT 2020
There are no Agents registered with CA ID 2

Note:

The WebLogic Administrator and Node Manager passwords are stored in the Administration Credentials Wallet. This is present in the EM_INSTANCE_HOME/sysman/config/adminCredsWallet directory. To recreate Administrator Credentials wallet, run the following command on each machine on which the Management Service is running:

emctl secure create_admin_creds_wallet [-admin_pwd <pwd>] [-nodemgr_pwd <pwd>]


Viewing the Security Status and OMS Port Information

To view the security status and OMS port information, use the following command

Example 2-7 emctl status oms -details

$ emctl status oms -details [-sysman_pwd welcome1]
Oracle Enterprise Manager 11g Release 1 Grid Control  
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Console Server Host : myhost.mydomain.com
HTTP Console Port   : 7788
HTTPS Console Port  : 7799
HTTP Upload Port    : 4889
HTTPS Upload Port   : 1159
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1

Configuring Transparent Layer Security

The Oracle Management Service can be configured in the following modes:

  • TLSv1-only mode: To configure the OMS to use only TLSv1 connections, do the following:

    1. Stop the OMS by entering the following command:

      OMS_ORACLE_HOME/bin/emctl stop oms
      
    2. Enter the following command:

      emctl secure oms -protocol TLSv1
      
    3. Append -Dweblogic.security.SSL.protocolVersion=TLS1 to JAVA_OPTIONS in Domain_Home/bin/startEMServer.sh. If this property already exists, update the value to TLS1.

    4. Restart the OMS with the following command:

      OMS_ORACLE_HOME/bin/emctl start oms
      

      Note:

      If the OMS is configured in the TLSv1 only mode, the 10.2.x Agents cannot communicate with the OMS since those Agents do not support the TLS mode.

  • SSLv3 Only Mode: To configure the OMS to use SSLv3 connections only, do the following:

    1. Stop the OMS by entering the following command:

      OMS_ORACLE_HOME/bin/emctl stop oms
      
    2. Enter the following command:

      emctl secure oms -protocol SSLv3
      
    3. Append -Dweblogic.security.SSL.protocolVersion=SSL3 to JAVA_OPTIONS in Domain_Home/bin/startEMServer.sh or startEMServer.cmd on Windows. If this property already exists, update the value to SSL3.

    4. Restart the OMS with the following command:

      OMS_ORACLE_HOME/bin/emctl start oms
      
  • Mixed Mode: To configure the OMS to use both SSLv3 and TLSv1 connections, do the following:

    1. Stop the OMS by entering the following command:

      OMS_ORACLE_HOME/bin/emctl stop oms
      
    2. Enter the following command:

      emctl secure oms
      
    3. Append -Dweblogic.security.SSL.protocolVersion=ALL to JAVA_OPTIONS in Domain_Home/bin/startEMServer.sh. If this property already exists, update the value to ALL.

    4. Restart the OMS with the following command:

      OMS_ORACLE_HOME/bin/emctl start oms
      

Note:

By default, the OMS is configured to use the Mixed Mode. To configure the Management Agent in TLSv1 only mode, set allowTLSOnly=true in the emd.properties file and restart the Agent.

Enabling Security for the Oracle Management Agent

When you install the Management Agent on a host, you must identify the Management Service that will be used by the Management Agent. If the Management Service you specify has been configured to take advantage of Enterprise Manager Framework Security, you will be prompted for the Agent Registration Password and Enterprise Manager Framework Security will be enabled for the Management Agent during the installation.

Otherwise, if the Management Service has not been configured for Enterprise Manager Framework Security or if the Registration Password was not specified during installation, then security will not be enabled for the Management Agent. In those cases, you can later enable Enterprise Manager Framework Security for the Management Agent.

To enable Enterprise Manager Framework Security for the Management Agent, use the emctl secure agent utility, which is located in the following directory of the Management Agent home directory:

AGENT_HOME/bin (UNIX)
AGENT_HOME\bin (Windows)

The emctl secure agent utility performs the following actions:

  • Obtains an Oracle Wallet from the Management Service that contains a unique digital certificate for the Management Agent. This certificate is required in order for the Management Agent to conduct SSL communication with the secure Management Service.

  • Obtains an Agent Key for the Management Agent that is registered with the Management Service.

  • Configures the Management Agent so it is available on your network over HTTPS and so it uses the Management Service HTTPS upload URL for all its communication with the Management Service.

To enable Enterprise Manager Framework Security for the Management Agent:

  1. Ensure that your Management Service and the Management Repository are up and running.

  2. Change directory to the following directory:

    AGENT_HOME/bin (UNIX)
    AGENT_HOME\bin (Windows)
    
  3. Stop the Management Agent:

    emctl stop agent
    
  4. Enter the following command:

    emctl secure agent (UNIX)
    emctl secure agent (Windows)
    

    The emctl secure agent utility prompts you for the Agent Registration Password, authenticates the password against the Management Service, and reconfigures the Management Agent to use Enterprise Manager Framework Security.


    Note:

    Alternatively, you can enter the command all on one line, but if you enter the command on one line, the password you enter will be displayed on the screen as you type:
    emctl secure agent agent_registration_pwd (UNIX)
    emctl secure agent agent_registration_pwd (Windows)
    

    shows sample output of the emctl secure agent utility.

  5. Restart the Management Agent:

    emctl start agent
    
  6. Confirm that the Management Agent is secure by checking the Management Agent home page.


    Note:

    You can also check if the Agent Management is secure by running the emctl status agent -secure command, or by checking the Agent and Repository URLs in the output of the emctl status agent command.

    In the General section of the Management Agent home page (Figure 2-8), the Secure Upload field indicates whether or not Enterprise Manager Framework Security has been enabled for the Management Agent.


    See Also:

    "Checking the Status of an Oracle Management Agent" in the Enterprise Manager online Help

Example 2-8 Sample Output of the emctl secure agent Utility

emctl secure agent
Oracle Enterprise Manager 11g Release 1 Grid Control.
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Securing agent...   Started
Securing agent...   Successful.

Example 2-9 Sample Output of the emctl status agent secure Command

emctl status agent -secure
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Checking the security status of the Agent at location set in 
/private/home/oracle/product/102/em/agent10g/sysman/config/emd.properties...  
Done.
Agent is secure at HTTPS Port 3872.
Checking the security status of the OMS at 
http://gridcontrol.oraclecorp.com:4889/em/upload/...  Done.
OMS is secure on HTTPS Port 4888

Figure 2-8 Secure Upload Field on the Management Agent Home Page

Secure Upload Field on the Management Agent Home Page

Enabling Security with Multiple Management Service Installations

If you already have a secure Management Service running and you install an addition7al Management Service that uses the same Management Repository, you will need to enable Enterprise Manager Framework Security for the new Management Service. This task is executed using the same procedure that you used to secure the first Management Service, by running the emctl secure oms utility.

Because you have already established at least one Agent Registration Password and a Root Key in your Management Repository, they must be used for your new Management Service. Your secure Management Agents can then operate against either Management Service.

All the registration passwords assigned to the current Management Repository are listed on the Registration Passwords page in the Oracle Enterprise Manager 11g Grid Control Console.

If you install a new Management Service that uses a new Management Repository, the new Management Service is considered to be a distinct enterprise. There is no way for the new Management Service to partake in the same security trust relationship as another Management Service that uses a different Management Repository. Secure Management Agents of one Management Service will not be able to operate against the other Management Service.

Restricting HTTP Access to the Management Service

By default, when you enable Enterprise Manager Framework Security on your Oracle Management Service there are no default restrictions on HTTP access. The Grid Control Console can also be accessed over HTTP and the Oracle Management Agents will be able to upload over HTTP as well as HTTPS.

However, it is important that only secure Management Agent installations that use the Management Service HTTPS channel are able to upload data to your Management Repository and Grid Control console is accessible via HTTPS only.

To restrict access so Management Agents can upload data to the Management Service only over HTTPS:

  1. Stop the Management Service, the WebTier, and the other application server components:

    cd ORACLE_HOME/opmn/bin
    emctl stop oms
    
  2. Change directory to the following location in the Management Service home:

    ORACLE_HOME/bin
    
  3. Enter the following command to prevent Management Agents from uploading data to the Management Service over HTTP:

    emctl secure lock -upload
    

    • Note:

      • To lock the console and prevent HTTP access to the console, enter the following command:

        emctl secure lock -console
        
      • To lock both, enter either of the following commands:

        emctl secure lock or 
        emctl secure lock -upload -console
        
      • To lock both the console access and uploads from Agents while enabling security on the Management Service, enter the following command:

        emctl secure oms -lock [other options]
        

  4. Restart the Management Service, the WebTier, and the other application server components:

    cd ORACLE_HOME/opmn/bin
    emctl start oms
    
  5. Verify that you cannot access the Management Agent upload URL using the HTTP protocol:

    For example, navigate to the following URL:

    http://hostname.domain:4889/em/upload
    

    You should receive an error message similar to the following:

    ForbiddenYou are not authorised to access this resource on the server.
    
  6. Verify that you can access the Management Agent Upload URL using the HTTPS protocol:

    For example, navigate to the following URL:

    https://hostname.domain:4888/em/upload
    

    You should receive the following message, which confirms the secure upload port is available to secure Management Agents:

    Http XML File receiverHttp Recceiver Servlet active!
    

To allow the Management Service to accept uploads from unsecure Management Agents, use the following command:

emctl secure unlock -upload

Note:

  • To unlock the console and allow HTTP access to the console, enter the following command:

    emctl secure unlock -console
    
  • To unlock both, enter either of the following command:

    emctl secure unlock
    emctl secur unlock -console -upload
    

Example 2-10 Sample Output of the emctl secure lock Command

emctl secure lock
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
OMS Console is locked. Access the console over HTTPS ports.
Agent Upload is locked. Agents must be secure and upload over HTTPS port.

Example 2-11 Sample Output of the emctl secure unlock Command

emctl secure unlock
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
OMS Console is unlocked. HTTP ports too can be used to access console.
Agent Upload is unlocked. Unsecure Agents may upload over HTTP.

To restrict HTTP access to the Oracle Enterprise Manager 11g Grid Control Console, use the emctl secure lock -console command.

Managing Agent Registration Passwords

Enterprise Manager uses the Agent Registration password to validate that installations of Oracle Management Agents are authorized to load their data into the Oracle Management Service.

The Agent Registration password is created during installation when security is enabled for the Oracle Management Service.


Note:

To avoid new Agents from being registered with the Oracle Management Service, you must delete all registration passwords.

Using the Grid Control Console to Manage Agent Registration Passwords

You can use the Grid Control Console to manage your existing registration passwords or create additional registration passwords:

  1. Click Setup at the top of any Grid Control Console page.

  2. Click Registration Passwords.

    Enterprise Manager displays the Registration Passwords page (Figure 2-9). The registration password you created when you ran the emctl secure oms command appears in the Registration Passwords table.

  3. Use the Registration Passwords page to change your registration password, create additional registration passwords, or remove registration passwords associated with the current Management Repository.

Figure 2-9 Managing Registration Passwords in the Grid Control Console

Managing Registration Passwords in the Grid Control Console

When you create or edit an Agent Registration Password on the Registration Passwords page, you can determine whether the password is persistent and available for multiple Management Agents or to be used only once or for a predefined period of time.

For example, if an administrator requests to install a Management Agent on a particular host, you can create a one-time-only password that the administrator can use to install and configure one Management Agent.

On the other hand, you can create a persistent password that an administrator can use for the next two weeks before it expires and the administrator must ask for a new password.

Using emctl to Add a New Agent Registration Password

To add a new Agent Registration Password, use the following emctl command on the machine on which the Management Service has been installed:

emctl secure setpwd [-sysman_pwd] [new registration pwd]

The emctl secure setpwd command requires that you provide the password of the Enterprise Manager super administrator user, sysman, to authorize the addition of the Agent Registration Password.

If you change the Agent Registration Password, you must communicate the new password to other Enterprise Manager administrators who need to install new Management Agents, enable Enterprise Manager Framework Security for existing Management Agents, or install additional Management Services.

As with other security passwords, you should change the Agent Registration Password on a regular and frequent basis to prevent it from becoming too widespread.

Enabling Security with a Server Load Balancer

When you deploy a Management Service that is available behind a Server Load Balancer (SLB), special attention must be given to the DNS host name over which the Management Service will be available. Although the Management Service may run on a particular local host, for example myhost.mycompany.com, your Management Agents will access the Management Service using the host name that has been assigned to the Server Load Balancer. For example, oracleoms.mycompany.com.

As a result, when you enable Enterprise Manager Framework Security for the Management Service, it is important to ensure that the Server Load Balancer host name is embedded into the Certificate that the Management Service uses for SSL communications. To do so, enter the following commands:

This may be done by using emctl secure oms and specifying the host name in the with an extra -host parameter as follows:

  • Enable security on the Management Service by entering the following command:

    emctl secure oms -host <slb_hostname> [-slb_console_port <slb UI port>] [-slb_port <slb upload port>] [other params]

  • Create virtual servers and pools on the Server Load Balancer.

  • Verify that the console can be accessed using the following URL:

    https://slbhost:slb_console_port/em

  • Re-secure the Agents with Server Load Balancer by using the following command:

    emctl secure agent -emdWalletSrcUrl <SLB Upload or UI URL>

    For example:

    Agent_Home/bin/emctl secure agent -emdWalletSrcUrl https://slbost:slb_upload_port/em https://slbost:slb_upload_port/em

Enabling Security for the Management Repository Database

This section describes how to enable Security for the Oracle Management Repository. This section includes the following topics:

About Oracle Advanced Security and the sqlnet.ora Configuration File

You enable security for the Management Repository by using Oracle Advanced Security. Oracle Advanced Security ensures the security of data transferred to and from an Oracle database.


See Also:

Oracle Database Advanced Security Administrator's Guide

To enable Oracle Advanced Security for the Management Repository database, you must make modifications to the sqlnet.ora configuration file. The sqlnet.ora configuration file is used to define various database connection properties, including Oracle Advanced Security parameters.

The sqlnet.ora file is located in the following subdirectory of the Database home:

ORACLE_HOME/network/admin

After you have enabled Security for the Management Repository and the Management Services that communicate with the Management Repository, you must also configure Oracle Advanced Security for the Management Agent by modifying the sqlnet.ora configuration file in the Management Agent home directory.

It is important that both the Management Service and the Management Repository are configured to use Oracle Advanced Security. Otherwise, errors will occur when the Management Service attempts to connect to the Management Repository. For example, the Management Service might receive the following error:

ORA-12645: Parameter does not exist

To correct this problem, be sure both the Management Service and the Management Repository are configured as described in the following sections.


Note:

The procedures in this section describe how to manually modify the sqlnet.ora configuration file to enable Oracle Advanced Security. Alternatively, you can make these modifications using the administration tools described in the Oracle Database Advanced Security Administrator's Guide.

Configuring the Management Service to Connect to a Secure Management Repository Database

If you have enabled Oracle Advanced Security for the Management Service database—or if you plan to enable Oracle Advanced Security for the Management Repository database—use the following procedure to enable Oracle Advanced Security for the Management Service:

  1. Stop the Management Service:

    ORACLE_HOME/bin/emctl stop oms
    
  2. Set the emoms.properties by using the emctl set property command

  3. Restart the Management Service.

    ORACLE_HOME/bin/emctl start oms
    

Table 2-4  Oracle Advanced Security Properties in the Enterprise Manager Properties File

PropertyDescription

oracle.sysman.emRep.dbConn.enableEncryption

Defines whether or not Enterprise Manager will use encryption between Management Service and Management Repository.Possible values are TRUE and FALSE. The default value is TRUE.For example:

oracle.sysman.emRep.dbConn. enableEncryption=true

oracle.net.encryption_client

Defines the Management Service encryption requirement.Possible values are REJECTED, ACCEPTED, REQUESTED, and REQUIRED.The default value is REQUESTED. In other words, if the database supports secure connections, then the Management Service uses secure connections, otherwise the Management Service uses insecure connections.

For example:

oracle.net. encryption_client=REQUESTED

oracle.net.encryption_types_client

Defines the different types of encryption algorithms the client supports.Possible values should be listed within parenthesis. The default value is ( DES40C ).

For example:

oracle.net. encryption_types_client=( DES40C )

oracle.net.crypto_checksum_client

Defines the Client's checksum requirements.

Possible values are REJECTED, ACCEPTED, REQUESTED, and REQUIRED.

The default value is REQUESTED. In other words, if the server supports checksum enabled connections, then the Management Service uses them, otherwise it uses normal connections.

For example:

oracle.net. crypto_checksum_client=REQUESTED

oracle.net.crypto_checksum_types_client

This property defines the different types of checksums algorithms the client supports.

Possible values should be listed within parentheses. The default value is ( MD5 ).

For example:

oracle.net. crypto_checksum_types_client=( MD5 )


Enabling Oracle Advanced Security for the Management Repository

To be sure your database is secure and that only encrypted data is transferred between your database server and other sources, review the security documentation available in the Oracle Database documentation library.


See Also:

Oracle Database Advanced Security Administrator's Guide

The following instructions provide an example of how you can confirm that Oracle Advanced Security is enabled for your Management Repository database and its connections with the Management Service:

  1. Locate the sqlnet.ora configuration file in the following directory of the database Oracle Home:

    ORACLE_HOME/network/admin
    
  2. Using a text editor, look for the following entries (or similar entries) in the sqlnet.ora file:

    SQLNET.ENCRYPTION_SERVER = REQUESTED
    SQLNET.CRYPTO_SEED = "abcdefg123456789"
    

    See Also:

    "Configuring Network Data Encryption and Integrity for Oracle Servers and Clients in the Oracle Application Server 10g Administrator's Guide

  3. Save your changes and exit the text editor.

Enabling Security for a Management Agent Monitoring a Secure Management Repository or Database

After you have enabled Oracle Advanced Security for the Management Repository, you must also enable Advanced Security for the Management Agent that is monitoring the Management Repository:

  1. Locate the sqlnet.ora configuration file in the following directory inside the home directory for the Management Agent that is monitoring the Management Repository:

    AGENT_HOME/network/admin (UNIX)
    AGENT_HOME\network\admin (Windows)
    
  2. Using a text editor, add the following entry to the sqlnet.ora configuration file:

    SQLNET.CRYPTO_SEED = "abcdefg123456789"
    

    The SQLNET.CRYPTO_SEED can be any string between 10 to 70 characters.


    See Also:

    "Configuring Network Data Encryption and Integrity for Oracle Servers and Clients in the Oracle Application Server Administrator's Guide

  3. Save your changes and exit the text editor.

  4. Restart the Management Agent.

Configuring Third Party Certificates

You can configure third party certificates for:

  • HTTPS Upload Virtual Host

  • HTTPS Console Virtual Host


Note:

Only Single Sign-On wallets are supported.

Configuring Third Party Certificate for HTTPS Upload Virtual Host

You can configure the third party certificate for the HTTPS Upload Virtual Host in two ways:

Method I

  1. Create a wallet for each OMS in the grid.

  2. While creating the wallet, specify the host name of the machine where the OMS is installed or the Load Balancer Name if the OMS is behind the Load Balancer for Common Name.

  3. Write the certificates of all the Certificate Authorities in the certificate chain (like the Root Certificate Authority, Intermediate Certificate Authority) into a file named trusted_certs.txt.

  4. Download or copy the trusted_certs.txt file to the host machines on which each Agent that is communicating with the OMS is running.

  5. Restart the Agent after running the add_trust_cert command.

    emctl secure add_trust_cert -trust_certs_loc <location of the trusted_certs.txt file>
    
  6. Secure the OMS and restart it.

    emctl secure oms -wallet <location of wallet> -trust_certs_loc <loc of trusted_certs.txt> [any other options]
    

Method 2

  1. Create a wallet for each OMS in the grid.

  2. Specify the host name of the machine where the OMS is installed or the Load Balancer Name if the OMS is behind the Server Load Balancer for Common Name (CN).

  3. Write the certificates of all the Certificate Authorities in the certificate chain (like the Root Certificate Authority, Intermediate Certificate Authority) into a file named trusted_certs.txt.

  4. Restart the OMS after it has been secured.

    emctl secure oms -wallet <location of wallet> -trust_certs_loc <loc of trusted_certs.txt> [any other options]
    
  5. Either re-secure the Agent by running the emctl secure agent command or import the trust points by running the emctl secure add_trust_cert -trust_certs_loc <location of the trusted_certs.txt file> command. The -trust_certs_loc parameter must contain the path and the filename of the trusted_certs.txt file. This file must only contain certificates in base64 format and no special characters or empty lines.

Configuring Third Party Certificate for HTTPS WebTier Virtual Host

To configure the third party certificate for HTTPS WebTier Virtual Host:

  1. Create a wallet for each OMS in the grid. Specify the host name of the machine where the OMS is installed or the Load Balancer Name if the OMS is behind the Server Load Balancer for Common Name.

  2. Run the following command on each OMS:

    emctl secure console -wallet <location of wallet>
    

    Note:

    Only single-sign-on wallets are supported.

Accessing Managed Targets

The following topics are discussed in this section:

  • Credential Subsystem

  • Pluggable Authentication Modules (PAM) Support

  • Sudo and Powerbroker Support

Credential Subsystem

Credentials like user names and passwords are typically required to access targets such as databases, application servers, and hosts. Credentials are encrypted and stored in Enterprise Manager. By using appropriate credentials, you can:

  • Collect metrics in the background as well as real-time

  • Perform jobs like backup, patching, cloning etc.

  • Perform real-time target administration like start, stop etc.

  • Connect to My Oracle Support

Based on their usage, credentials can be classified into the following categories:

  • Job Credentials: The job system uses the credential subsystem to retrieve the credentials required to submit a job on the targets. The administrator can define their preferred and default credentials in the preference section of EM console. The user can override the default credentials by specifying different credentials while submitting the job.


    Note:

    If the user chooses to use preferred credentials, these credentials will be used when the user submits the job. If the preferred credentials are not available, the default credentials will be used. If default credentials are not present, the job cannot be submitted.

  • Monitoring Credentials: These credentials are used by the Management Agent to monitor certain types of targets. For example, most database monitoring involves connecting to the database, which requires a username, password, and optionally, a role. Monitoring credentials, if stored in the repository, can also be potentially used by management applications to connect directly to the target from the OMS.

  • Collection Credentials: These credentials are associated with user-defined metrics.

To simplify the usage and management of credentials, the following features are available in Enterprise Manager:

  • Preferred Credentials: Preferred credentials are used to simplify access to managed targets by storing target login credentials in the Management Repository. With preferred credentials set, users can access an Enterprise Manager target that recognizes those credentials without being prompted to log into the target. Preferred credentials are set on a per user basis, thus ensuring the security of the managed enterprise environment.

    • Default Credentials: Default credentials can be set for a particular target type and will be available for all the targets of the target type. It will be overridden by target preferred credentials.

    • Target Credentials: Target credentials are preferred credentials set for a particular target. They could be used by applications such as the job system, patch, etc. For example, if the user chooses to use preferred credentials while submitting a job, then the preferred credentials set for the target (target credentials) will be used. If the target credentials are not present, the default credentials (for the target type) will be used. If the default credentials are not present, the job will fail. If not specified, by default, preferred credentials refer to preferred target credentials"

For example, to set the host preferred credentials, click Preferences to navigate to the Preferences page. Click the Preferred Credentials link in the right panel. In the Preferred Credentials page, click the Set Credentials icon for the host. The Host and Cluster Preferred Credentials is displayed.

Figure 2-10 Host and Cluster Preferred Credentials

Surrounding text describes Figure 2-10 .

On this page, you can set both default and explicit preferred credentials for the host and cluster target types. For more details on setting preferred credentials, see the Enterprise Manager Online Help.

Managing Credentials Using EMCLI

You can manage passwords using EMCLI verbs. Using EMCLI, you can:

  • Change the database user password in both the target database and Enterprise Manager.

    emcli update_db_password -change_at_target=Yes|No -change_all_reference=Yes|No
    
  • Update a password which has already been changed at the host target.

    emcli update_host_password -change_all_reference=Yes|No
    
  • Set preferred credentials for given users.

    emcli set_credential
          -target_type="ttype"
          [-target_name="tname"]
          -credential_set="cred_set"
          [-user="user"]
          -columns="col1:newval1;col2:newval2;PDP:SUDO/POWERBROKER;RUNAS:oracle;          PROFILE:user1..."
          [-input_file="tag1:file_path1;tag2:file_path2;..."]
          [-oracle_homes="home1;home2"]
          [-monitoring]
    

For detailed descriptions of these verbs, refer to Enterprise Manager Command Line Interface guide.

Pluggable Authentication Modules (PAM) Support for Hosts

Pluggable authentication modules, or PAM, is a mechanism to integrate multiple low-level authentication schemes into a high-level application programming interface (API). It allows programs that rely on authentication to be written independently of the underlying authentication scheme. By using PAM, instead of using the local password file to authenticate the user accessing the host, you can take advantage of other authentication mechanisms such as LDAP, RADIUS and Kerberors. If your host authentication is configured over PAM, the Management Agent needs to be configured accordingly to enable PAM Authentication. Refer to note 422073.1 for deployment details.


Note:

The local password file (usually /etc/passwd) will be checked and used first. This should be synchronized with the LDAP password if it is being used. If this fails, the Management Agent will switch to the external authentication module.

Configuring PAM for RHEL4 Users

For users on RHEL4, the PAM file configuration is as follows:

#%PAM-1.0
auth required pam_ldap.so
account required pam_ldap.so
password required pam_ldap.so
session required pam_ldap.so

For more details, see https://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/ref-guide/s1-pam-format.html

Configuring PAM for AIX Users

For AIX users, use the edit/etc/pam.conf file and add the following lines:

emagent auth    required        /usr/lib/security/pam_aix
emagent account required        /usr/lib/security/pam_aix
emagent password  required      /usr/lib/security/pam_aix
emagent session required        /usr/lib/security/pam_aix

After editing the file, apply patch 5527130 and run root.sh

Sudo and PowerBroker Support

Privilege delegation allows a logged-in user to perform an activity with the privileges of another user. Sudo and PowerBroker are privilege delegation tools that allow a logged-in user to be assigned these privileges. Typically, the privileges that are granted to a specific user are administered centrally. For example, the sudo command can be used to run a script that requires root access:

sudo root root.sh

In the invocation of sudo in the example above, an administrator can use the sudo command to run a script as root provided he has been granted the appropriate privileges by the system administrator. Enterprise Manager preferred credentials allow you to use two types of privilege delegation tools: Sudo and PowerBroker. You can use EMCLI or the Manage Privilege Delegation Settings page to set/edit privilege delegation settings for a host. See the Enterprise Manager Command Line Interface guide for more information on using the command line.

Sudo: sudo allows a permitted user to execute a command as the super user or another user, as specified in the sudoers file. If the invoking user is root or if the target user is the same as the invoking user, no password is required. Otherwise, sudo requires that users authenticate themselves with a password by default. Once a user has been authenticated, a timestamp is updated and the user may then use sudo without a password for a short period of time (5 minutes unless overridden in sudoers). sudo determines who is an authorized user by consulting the file /etc/sudoers file. For more information, see the manual page on sudo (man sudo) on Unix. Enterprise Manager authenticates the user using sudo, and executes the script as sudo. For example, if the command to be executed is foo -arg1 -arg2, it will be executed as sudo -S foo -arg1 -arg2.

PowerBroker: Symark PowerBroker enables UNIX system administrators to specify the circumstances under which other people may run certain programs such as root (or other important accounts). The result is that responsibility for such actions as adding user accounts, fixing line printer queues, and so on, can be safely assigned to the appropriate people, without disclosing the root password. The full power of root is thus protected from potential misuse or abuse-for example, modifying databases or file permissions, erasing disks, or more subtle damage. Symark PowerBroker can access existing programs as well as its own set of utilities that execute common system administration tasks. Utilities being developed to run on top of Symark PowerBroker can manage passwords, accounts, backups, line printers, file ownership or removal, rebooting, logging people out, killing their programs, deciding who can log in to where from where, and so on. They can also provide TCP/IP, Load Balancer, cron, NIS, NFS, FTP, rlogin, and accounting subsystem management. Users can work from within a restricted shell or editor to access certain programs or files as root. See your Sudo or PowerBroker documentation for detailed setup and configuration information.

Creating a Privilege Delegation Setting

Enterprise Manager allows you to create privilege delegation settings either by creating the setting directly on a host target, or by creating a PDP setting template that you can apply to multiple hosts.

To create a privilege delegation setting directly on a host:

  1. Login to Enterprise Manager and navigate to the Setup page. Click Manage Privilege Delegation Settings on the left panel. The following screen is displayed:

    Figure 2-11 Manage Privilege Delegation Settings

    Manage Privilege Delegation Settings
  2. For any host target appearing in the table, click Edit. Enterprise Manager takes you to the Host Privilege Delegation Setting page.

  3. Select a privilege delegation type (Sudo or PowerBroker).

  4. Enter the privilege delegation command to be used and, in the case of PowerBroker, the optional Password Prompt.

  5. Click Update to apply the settings to the host. The following figure shows the Host Privilege Delegation Setting window that you can use to create a PowerBroker setting.

    Figure 2-12 Host Privilege Delegation Setting - PowerBroker

    Host Privilege Delegation Setting - PowerBroker

    Once you have created a privilege delegation setting, you must apply this setting to selected targets. This setting can be applied to one more hosts or to a composite (Group) target (the group must contain at least one host target). You can apply a Privilege Delegation setting using the Grid Control console by clicking Setup on the Enterprise Manager Home page and then choosing Manage Privilege Delegation Settings from the left menu panel.

Cryptographic Support

To protect the integrity of sensitive data in Enterprise Manager, a signing on verification method known as the emkey is used. Encryption key is the master key that is used to encrypt/decrypt sensitive data, such as passwords and preferred credentials that are stored in the Repository. The key is originally in stored in repository. It is removed from repository and copied to the Credential Store during installation of the first OMS. (the emkey is secured out-of-the-box). A backup is created in OMS_ORACLE_HOME/sysman/config/emkey.ora. It is recommended to create a backup of this file on some other machine. When starting up, OMS reads the emkey from Credential Store and repository. If the emkey is not found or is corrupted, it fails to start. By storing the key separately from Enterprise Manager schema, we ensure that the sensitive data such as Preferred Credentials in the Repository remain inaccessible to the schema owner and other SYSDBA users (Privileged users who can perform maintenance tasks on the database) in the Repository. Moreover, keeping the key from the schema will ensure that sensitive data remain inaccessible while Repository backups are accessed. Further, the schema owner should not have access to the OMS/Repository Oracle homes.

Configuring the emkey

The emkey is an encryption key that is used to encrypt and decrypt sensitive data in Enterprise Manager such as host passwords, database passwords and others. By default, the emkey is stored in the ORACLE_HOME/sysman/config/emkey.ora file. The location of this file can be changed.


WARNING:

If the emkey.ora file is lost or corrupted, all the encrypted data in the Management Repository becomes unusable. Maintain a backup copy of this file on another system.


During startup, the Oracle Management Service checks the status of the emkey. If the emkey has been properly configured, it uses it encrypting and decrypting data. If the emkey has not been configured properly, the following error message is displayed.

Example 2-12 emctl start oms Command

Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
emctl start omsStarting HTTP Server ...Starting Oracle Management Server ...Checking Oracle Management Server Status ...Oracle Management Server is not functioning because of the following reason:The Em Key is not configured properly. Run "emctl status emkey" for more details.

emctl Commands

The emctl commands related to emkey are given below:

  • emctl status emkey [-sysman_pwd <pwd>]

  • emctl config emkey -copy_to_credstore [-sysman_pwd <pwd>]

  • emctl config emkey -copy_to_repos [-sysman_pwd <pwd>]

  • emctl config emkey -remove_from_repos [-sysman_pwd <pwd>]

  • emctl config emkey -copy_to_file_from_credstore -admin_host <host> -admin_port <port> -admin_user <username> [-admin_pwd <pwd>] [-repos_pwd <pwd>] -emkey_file <emkey file>

  • emctl config emkey -copy_to_file_from_repos (-repos_host <host> -repos_port <port> -repos_sid <sid> | -repos_conndesc <conn desc>) -repos_user <username> [-repos_pwd <pwd>] [-admin_pwd <pwd>] -emkey_file <emkey file>

  • emctl config emkey -copy_to_credstore_from_file -admin_host <host> -admin_port <port> -admin_user <username> [-admin_pwd <pwd>] [-repos_pwd <pwd>] -emkey_file <emkey file>

  • emctl config emkey -copy_to_repos_from_file (-repos_host <host> -repos_port <port> -repos_sid <sid> | -repos_conndesc <conn desc>) -repos_user <username> [-repos_pwd <pwd>] [-admin_pwd <pwd>] -emkey_file <emkey file>

emctl status emkey

This command shows the health or status of the emkey. Depending on the status of the emkey, the following messages are displayed:

  • When the emkey has been correctly configured in the Credential Store, the following message is displayed.

    Example 2-13 emctl status emkey - Example 1

    Oracle Enterprise Manager 11g Release 1 Grid Control
    Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
    The EmKey is configured properly, but is not secure. Secure the EmKey by running "emctl config emkey -remove_from_repos"
    
  • When the emkey has been correctly configured in the Credential Store and has been removed from the Management Repository, the following message is displayed.

    Example 2-14 emctl status emkey - Example 2

    Oracle Enterprise Manager 11g Release 1 Grid Control
    Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
    The EMKey exists in the Management Repository, but is not configured properly or is corrupted in the credential store.
    Configure the EMKey by running "emctl config emkey -copy_to_credstore".
    
  • When the emkey is corrupted in the Credential Store and removed from the Management Repository, the following message is displayed.

    Example 2-15 emctl status emkey - Example 3

    Oracle Enterprise Manager 11g Release 1 Grid Control
    Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
    The EMKey is not configured properly or is corrupted in the credential store and does not exist in the Management Repository. To correct the problem:1) Get the backed up emkey.ora file.
    2) Configure the emkey by running "emctl config emkey -copy_to_credstore_from_file"
    

emctl config emkey -copy_to_credstore

This command copies the emkey from the Management Repository to the Credential Store.

Example 2-16 Sample Output of the emctl config emkey -copy_to_credstore Command

emctl config emkey -copy_to_credstore [-sysman_pwd <pwd>]
Oracle Enterprise Manager 11g Release 1 Grid Control  
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
The EMKey has been copied to the Credential Store.

emctl config emkey -copy_to_repos

This command copies the emkey from the Credential Store to Management Repository.

Example 2-17 Sample Output of the emctl config emkey -copy_to_repos Command

emctl config emkey -copy_to_repos [-sysman_pwd <pwd>]Oracle Enterprise Manager 11g Release 1 Grid Control  Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".

emctl config emkey -copy_to_file_from_credstore

This command copies the emkey from the Credential Store to a specified file.

Example 2-18 Sample Output of the emctl config emkey -copy_to_file_from_credstore Command

emctl config emkey -copy_to_file_from_credstore -admin_host <host> -admin_port
<port> -admin_user <username> [-admin_pwd <pwd>] [-repos_pwd <pwd>] -emkey_file
<emkey file>
Oracle Enterprise Manager 11g Release 1 Grid Control  
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
The EMKey has been copied to file.

emctl config emkey -copy_to_file_from_repos

This command copies the emkey from the Management Repository to a specified file.

Example 2-19 Sample Output of the emctl config emkey -copy_to_file_from_repos Command

emctl config emkey -copy_to_file_from_repos (-repos_host <host> -repos_port <port>
-repos_sid <sid> | -repos_conndesc <conn desc>) -repos_user <username> [-repos_pwd
<pwd>] [-admin_pwd <pwd>] -emkey_file <emkey file>Oracle Enterprise Manager 11g Release 1 Grid Control  Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.The EMKey has been copied to file.

emctl config emkey -copy_to_credstore_from_file

This command copies the emkey from a specified file to the Credential Store.

Example 2-20 Sample Output of the emctl config emkey -copy_to_credstore_from_file Command

emctl config emkey -copy_to_credstore_from_file -admin_host <host> -admin_port <port> -admin_user <username> [-admin_pwd <pwd>] [-repos_pwd <pwd>] -emkey_file <emkey file>
Oracle Enterprise Manager 11g Release 1 Grid Control  
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
The EMKey has been copied to the Credential Store.

emctl config emkey -copy_to_repos_from_file

This command copies the emkey from a specified file to the repository.

Example 2-21 Sample Output of the emctl config emkey -copy_to_repos_from_file Command

emctl config emkey -copy_to_repos_from_file (-repos_host <host> -repos_port <port>
-repos_sid <sid> | -repos_conndesc <conn desc>) -repos_user <username> [-repos_pwd
<pwd>] [-admin_pwd <pwd>] -emkey_file <emkey file>
Oracle Enterprise Manager 11g Release 1 Grid Control  
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
The EMKey has been copied to the Management Repository. This operation will cause
the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".

emctl config emkey -remove_from_repos

This command removes the emkey from the repository.

Example 2-22 Sample Output of emctl config emkey -remove_from_repos Command

emctl config emkey -remove_from_repos [-sysman_pwd <pwd>]
Oracle Enterprise Manager 11g Release 1 Grid Control  
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
The EMKey has been removed from the Management Repository.

Note:

If the emkey is corrupted, you cannot remove it from the Management Repository.

Install and Upgrade Scenarios

This section explains the install and upgrade scenarios for emkey.

Installing the Management Repository

A new emkey is generated as a strong random number when the Management Repository is installed.

Installing the First Oracle Management Service

When the Oracle Management Service is installed, the Installer copies the emkey to Credential Store and removes it from repository (emkey is secured out-of-box).

Upgrading from 10.2 to 11.1

The Management Repository is upgraded as usual. When upgrading the OMS, the omsca (OMS Configuration Assistant) copies the emkey to Credential Store and removes from repository. If the emkey is already secured before upgrade or has been removed from repository, then omsca reads the emkey from emkey.ora file present in old OMS Oracle Home and copies it to Credential Store.


  • Note:

    After all the Oracle Management Service have been upgraded, you can secure the emkey, that is, remove it from the Management Repository by running the following command:

    emctl config emkey -remove_from_repos


Recreating the Management Repository

When the Management Repository is recreated, a new emkey is generated. This new key will not be in synchronization with the existing emkey Credential Store.

  1. Copy the new emkey to Credential Store by using the emctl config emkey -copy_to_credstore command.

  2. Take a backup by entering the emctl config emkey -copy_to_file_from_repos command or the emctl config emkey -copy_to_file_from_credstore command.

  3. Secure the emkey by using the emctl config emkey -remove_from_repos command.

Setting Up the Auditing System for Enterprise Manager

All operations performed by Enterprise Manager users such as creating users, granting privileges, starting a remote job like patching or cloning, need to be audited to ensure compliance with the Sarbanes-Oxley Act of 2002 (SAS 70). This act defines standards an auditor must use to assess the contracted internal controls of a service organization. Auditing an operation enables an administrator to monitor, detect, and investigate problems and enforce enterprise wide security policies.

Irrespective of how the user has logged into Enterprise Manager, if auditing is enabled, each user action is audited and the audit details are stored in a record.

Configuring the Enterprise Manager Audit System

You can configure the Enterprise Manager Audit System by using the following emcli commands:

  • enable_audit: Enables auditing for all user operations.

  • disable_audit: Disables auditing for all user operations.

  • show_operations_list: Shows a list of the user operations being audited.

  • show_audit_settings: Shows the audit status, operation list, externalization service details, and purge period details.

Configuring the Audit Data Export Service

Audit data needs to be protected and maintained for several years. The volume of audit data may become very large and impact the performance of the system. To limit the amount of data stored in the repository, the audit data must be externalized or archived at regular intervals. The archived audit data is stored in an XML file complying with the ODL format. To externalize the audit data, the EM_AUDIT_EXTERNALIZATION API is used. Records of the format <file-prefix>.NNNNN.xml, where NNNN is a number are generated. The numbers start with 00001 and continue to 99999.

You can set up the audit externalization service for exporting audit data into the file system by using the update_audit_setting -externalization_switch command.

Updating the Audit Settings

The update_audit_settings command updates the current audit settings in the repository and restarts the Management Service.

Example 2-23 Usage of the update_audit_setting command

emcli update_audit_settings
      -audit_switch="ENABLE/DISABLE"
      -operations_to_enable="name of the operations to enable, for all oprtations
       use ALL"
      -operations_to_disable="name of the operations to disable, for all
       oprtations use ALL"
      -externalization_switch="ENABLE/DISABLE"
      -directory_name="directory_name (DB Directory)"
      -file_prefix="file_prefix"
      -file_size="file_size (Bytes)"
      -data_retention_period="data_retention_period (Days)"
  • -audit_switch: Enables auditing across Enterprise Manager. The possible values are ENABLE/DISABLE. Default value is DISABLE.

  • -operations_to_disable: Enables auditing for specified operations. Enter All to enable all operations.

  • -operations_to_disable: Disables auditing for specified operations. Enter All to disable all operations.

  • -externalization_switch: Enables the audit data export service. The possible values are ENABLE/DISABLE. Default value is DISABLE.

  • -directory: The database directory that is mapped to the OS directory where the export service archives the audit data files.

  • -file_prefix: The file prefix to be used by the export service to create the file in which audit data is to be stored.

  • -file_size: The size of the file on which the audit data is to be stored. The default value is 5000000 bytes.

  • data_retention_period: The period for which the audit data is to be retained inside the repository. The default value is 365 days.

Searching the Audit Data

You can search for audit data that has been generated over a specified period. You can also search for the following:

  • Audit details of a specific user operation or all user operations.

  • Audit details of operations with a Success or Failure status or All operations.

To view the audit data, click the Setup option. On the Setup page, click the Management Services and Repository tab. The Overview page is displayed. Click the Audit Data link under the Audit section. The Audit Data page is displayed. Specify the search criteria in the fields and click Go. The results are displayed in the Summary table.

Figure 2-13 Audit Data Search Page

Audit Data Search Page

To view the details of each record that meets the search criteria, select Detailed in the View drop-down list. To drill down to the full record details, click on the Timestamp. The Audit Record page is displayed.

Figure 2-14 Audit Record Details Page

Audit Record Details Page
Field NameDescription
General
Operation TimestampThe date and time on which the operation took place.
AdministratorThe id of the administrator who has logged into Enterprise Manager.
OperationThe type of operation being audited.
StatusThe status of the operation which can be success or failure.
MessageA descriptive message indicating the status of the operation.
Normalized TimestampThis is the UTC timestamp.
Client Information
SessionThis can either be the HTTP Session ID or the DBMS Session ID.
IP AddressThe IP address of the client's host machine.
HostnameThe name of the client's host machine.
Upstream Component TypeThe type of client, Console, Web Service, EMCLI, being used.
Authentication TypeThe nature of the session (HTTP Session, DB Session).
Upstream Component NameThe name of the client being used.
OMS Information
HostnameThe host name of the Oracle Management Service.
IP AddressThe IP address of the Oracle Management Service.
Instance IDThe Instance ID of the Oracle Management Service.
Operation Specific Information
Object NameThe operation being performed on an object

List of Operations Audited

The following table lists the names of operation and their description.

Table 2-5 List of Operations Audited

Operation NameDescription

change_password

Change Password

create_user

Create User

delete_user

Delete User

logon

Login

logoff

Logout

grant_role

Grant Role

grant_target_priv

Grant Target Privilege

revoke_role

Revoke Role

revoke_target_priv

Revoke Target Privilege

submit_job

Submit Job

edit_job

Edit Job

delete_job

Delete Job

change_pref_cred

Change Preferred Credential

modify_user

Modify User

grant_system_priv

Grant System Privilege

grant_job_priv

Grant Job Privilege

revoke_system_priv

Revoke System Privilege

revoke_job_priv

Revoke Job Privilege

remote_op

Remote Operation Job

get_file

Get File

put_file

Put File

file_transfer

File Transfer

create_role

Create Role

delete_role

Delete Role

modify_role

Modify Role

job_output

Job Output

suspend_job

Suspend Job

agent_resync

Agent Re synchronization Operation

repository_resync

Repository Re synchronization Operation

remove_privilege_delegation_setting

Remove Privilege Delegation Setting

set_privilege_delegation_setting

Set Privilege Delegation Setting

add_agent_registration_password

Add Registration Password

edit_agent_registration_password

Edit Registration Password

delete_agent_registration_password

Delete Registration Password

agent_registration_password_usage

Registration Password Usage

audit_settings

Enable or Disable Auditing

audit_export_settings

Externalize Audit Data Settings

create_template

Create Template

edit_template

Edit Template

delete_template

Delete Template

apply_template

Apply Template

save_monitoring_settings

Save Monitoring Settings

modify_metric_settings

Modify Metric Settings

modify_policy_settings

Modify Policy Settings

create_udp

Create User Defined Policy

edit_udp

Edit User Defined Policy

delete_udp

Delete User Defined Policy

evaluate_udp

Evaluate User Defined Policy

import_udp

Import User Defined Policy

create_udpg

Create User Defined Policy Group

edit_udpg

Edit User Defined Policy Group

delete_udpg

Delete User Defined Policy Group

delete_pg_eval

Delete Policy Group Evaluation Results

create_pg_sched

Create Policy Group Schedule

edit_pg_sched

Edit Policy Group Schedule

delete_pg_sched

Delete Policy Group Schedule

db_login

Audit Database User Login

db_logout

Audit Database User Logout

db_start

Audit Database Startup

db_shutdown

Audit Database Shutdown

db_restart

Audit Database Restart


Additional Security Considerations

After you enable security for the Enterprise Manager components and framework, there are additional security considerations. This section provides the following topics:

Changing the SYSMAN and MGMT_VIEW Passwords

This section describes the commands used to change the SYSMAN and MGMT_VIEW passwords.

Changing the SYSMAN User Password

To change the password of the SYSMAN user, enter the following command:

 emctl config oms -change_repos_pwd [-change_in_db] [-old_pwd <old_pwd>] [-new_pwd <new_pwd>] [-use_sys_pwd [-sys_pwd <sys_pwd>]]

You must run this command on each Management Service in your environment.

ParameterDescription
-change_in_dbThis parameter is optional and is used to change the SYSMAN password in the repository. If there are multiple Management Services running, this parameter must be set to true for at least one Management service.

If this parameter is not specified, the emoms.properties file will be updated with the new SYSMAN password.

-old_pwdThis is the current SYSMAN password.
-new_pwdThis is the new password.
-use_sys_pwdThis parameter is optional and is used to connect to the database as a SYS user.
-sys_pwdThis is the password for the SYS user.

Changing the MGMT_VIEW User Password

To change the password of the MGMT_VIEW user, enter the following command:

emctl config oms -change_view_user_pwd [-sysman_pwd <sysman_pwd>] [-user_pwd <user_pwd>] [-auto_generate]
ParameterDescription
-sysman_pwdThe password for the SYSMAN user.
-user_pwdThe new password for theMGMT_VIEW user.This is an optional parameter and if it is not specified, the password is auto generated.
-auto_generateIf this option is specified, the password is auto generated.

Responding to Browser-Specific Security Certificate Alerts

This section describes how to respond to browser-specific security alert dialog boxes when you are using Enterprise Manager in a secure environment.

The security alert dialog boxes described in this section should appear only if you have enabled Enterprise Manager Framework Security, but you have not completed the more extensive procedures to secure your WebTier properly.

This section contains the following topics:

Responding to the Internet Explorer Security Alert Dialog Box

If you enable security for the Management Service, but do not enable the more extensive security features of your WebTier, you will likely receive a Security Alert dialog box similar to the one shown in Figure 2-15 when you first attempt to display the Grid Control Console using the HTTPS URL in Internet Explorer.


Note:

The instructions in this section apply to Internet Explorer 5.5. The instructions may vary for other supported browsers.

Figure 2-15 Internet Explorer Security Alert Dialog Box

Description of Figure 2-15 follows

When Internet Explorer displays the Security Alert dialog box, use the following instructions to install the certificate and avoid viewing this dialog box again in future Enterprise Manager sessions:

  1. In the Security Alert dialog box, click View Certificate.

    Internet Explorer displays the Certificate dialog box.

  2. Click the Certificate Path tab and select the first entry in the list of certificates as shown in Figure 2-16.

  3. Click View Certificate to display a second Certificate dialog box.

  4. Click Install Certificate to display the Certificate Import wizard.

  5. Accept the default settings in the wizard, click Finish when you are done, and then click Yes in the Root Certificate Store dialog box.

    Internet Explorer displays a message box indicating that the Certificate was imported successfully.

  6. Click OK to close each of the security dialog boxes and click Yes on the Security Alert dialog box to continue with your browser session.

    You should no longer receive the Security Alert dialog box in any future connections to Enterprise Manager when you use this browser.

Figure 2-16 Certificate Path Tab on the Internet Explorer Certificate Dialog Box

Description of Figure 2-16 follows

Responding to the Netscape Navigator New Site Certificate Dialog Box

If you enable security for the Management Service, but you do not enable the more extensive security features of your WebTier, you will likely receive a New Site Certificate dialog box similar to the one shown in Figure 2-17 when you first attempt to display the Grid Control Console using the HTTPS URL in Netscape Navigator.


Note:

The instructions in this section apply to Netscape Navigator 4.79. The instructions may vary for other supported browsers.

When Netscape Navigator displays the New Site Certificate dialog box, use the following instructions to install the certificate and avoid viewing this dialog box again in future Enterprise Manager sessions:

  1. Review the instructions and information on each wizard page; click Next until you are prompted to accept the certificate.

  2. Select Accept this certificate forever (until it expires) from the list of options.

  3. On the last screen of the wizard, click Finish to close the wizard and continue with your browser session.

    You should no longer receive the New Site Certificate dialog box when using the current browser.

Figure 2-17 Netscape Navigator New Site Certificate Dialog Box

Description of Figure 2-17 follows

Preventing the Display of the Internet Explorer Security Information Dialog Box

After you enable Security for the Management Service, you may receive a dialog box similar to the one shown in Figure 2-18 whenever you access certain Enterprise Manager pages.


Note:

The instructions in this section apply to Internet Explorer 6.0. The instructions may vary for other supported browsers.

Figure 2-18 Internet Explorer Security Information Dialog Box

Description of Figure 2-18 follows

To stop this dialog box from displaying:

  1. Select Internet Options from the Internet Explorer Tools menu.

  2. Click the Security tab.

  3. Select Internet and then click Custom Level.

    Internet Explorer displays the Security Settings dialog box.

  4. Scroll down to Miscellaneous settings and enable the Display Mixed Content option.

Configuring Beacons to Monitor Web Applications Over HTTPS

Oracle Beacons provide application performance availability and performance monitoring. They are part of the Application Service Level Management features of Enterprise Manager.


See Also:

"About Application Service Level Management" in the Enterprise Manager Online Help

When a Beacon is used to monitor a URL over Secure Sockets Layer (SSL) using an HTTPS URL, the Beacon must be configured to recognize the Certificate Authority that has been used by the Web site where that URL resides.


See Also:

"The Public Key Infrastructure Approach to Security" in the Oracle Security Overview for an overview of Public Key Infrastructure features, such as Certificate Authorities

The Beacon software is preconfigured to recognize most commercial Certificate Authorities that are likely to be used by a secure Internet Web Site. However, you may encounter Web Sites that, although available over HTTPS, do not have a Certificate that has been signed by a commercial Certificate Authority recognized by the Beacon. The following are out-of-box certificates recognized by Beacons:

  • Class 1 Public Primary Certification Authority by VeriSign, Inc.

  • Class 2 Public Primary Certification Authority by VeriSign, Inc.

  • Class 3 Public Primary Certification Authority by VeriSign, Inc.

  • Secure Server Certification Authority by RSA Data Security, Inc.

  • GTE CyberTrust Root by GTE Corporation

  • GTE CyberTrust Global Root by GTE CyberTrust Solutions, Inc.

  • Entrust.net Secure Server Certification Authority by Entrust.net ((c) 1999

  • Entrust.net Limited, www.entrust.net/CPS incorp. by ref. (limits liab.))

  • Entrust.net Certification Authority (2048) by Entrust.net ((c) 1999

  • Entrust.net Limited, www.entrust.net/CPS_2048 incorp. by ref. (limits liab.))

  • Entrust.net Secure Server Certification Authority by Entrust.net ((c) 2000

  • Entrust.net Limited, www.entrust.net/SSL_CPS incorp. by ref. (limits liab.))

In those cases, for example, if you attempt to use the Test section of the Beacon Performance page to test the HTTP Response of the secure URL, the following error appears in the Status Description column of the Response Metrics table on the URL Test Page:

javax.net.ssl.SSLException: SSL handshake failed: X509CertChainIncompleteErr--https://mgmtsys.acme.com/OracleMyPage.Home

See Also:

"Using Beacons to Monitor Remote URL Availability" in the Enterprise Manager online help

To correct this problem, you must allow the Beacon to recognize the Certificate Authority that was used by the Web Site to support HTTPS. You must add the Certificate of that Certificate Authority to the list of Certificate Authorities recognized by Beacon.

To configure the Beacon to recognize the Certificate Authority:

  1. Obtain the Certificate of the Web Site's Certificate Authority, as follows:

    1. In Microsoft Internet Explorer, connect to the HTTPS URL of the Web Site you are attempting to monitor.

    2. Double-click the lock icon at the bottom of the browser screen, which indicates that you have connected to a secure Web site.

      The browser displays the Certificate dialog box, which describes the Certificate used for this Web site. Other browsers offer a similar mechanism to view the Certificate detail of a Web Site.

    3. Click the Certificate Path tab and select the first entry in the list of certificates as shown in Figure 2-16.

    4. Click View Certificate to display a second Certificate dialog box.

    5. Click the Details tab on the Certificate window.

    6. Click Copy to File to display the Certificate Manager Export wizard.

    7. In the Certificate Manager Export wizard, select Base64 encoded X.509 (.CER) as the format you want to export and save the certificate to a text file with an easily-identifiable name, such as beacon_certificate.cer.

    8. Open the certificate file using a text editor.

      The content of the certificate file will look similar to the content shown in .

  2. Update the list of Beacon Certificate Authorities as follows:

    1. Locate the b64InternetCertificate.txt file in the following directory of Agent Home of the Beacon host:

      agent_home/sysman/config/
      

      This file contains a list of Base64 Certificates.

    2. Edit the b64InternetCertificate.txt file and add the contents of the Certificate file you just exported to the end of the file, taking care to include all the Base64 text of the Certificate including the BEGIN and END lines.

  3. Restart the Management Agent.

    After you restart the Management Agent, the Beacon detects your addition to the list of Certificate Authorities recognized by Beacon and you can successfully monitor the availability and performance of the secure Web site URL.

Example 2-24 Sample Content of an Exported Certificate

-----BEGIN CERTIFICATE----- 
MIIDBzCCAnCgAwIBAgIQTs4NcImNY3JAs5edi/5RkTANBgkqhkiG9w0BAQQFADCB
... base64 certificate content...
-----END CERTIFICATE-----

Using ORACLE _HOME Credentials

Oracle Enterprise Manager 11g Release 1 introduces the concept of ORACLE_HOME credentials to designate the owner of the ORACLE_HOME with special credentials for the ORACLE_HOME. The operating system user who installs the software will also need to perform the patching. In Oracle Enterprise Manager 11g Release 1, one can explicitly set the ORACLE_HOME credentials and store it in the Management Repository. While patching, the user can use existing operating system credentials or override it under special circumstances. The user can specify ORACLE_HOME credentials and in the same interface choose to store it in the Management Repository for future use.

The Enterprise Manager Command line interface (EMCLI) also provides a facility to set ORACLE_HOME credentials. This is useful in cases where the Super Administrator sets the credentials and the user who initiates the patching job is unaware of the actual credentials. For auditing in security-hardened data centers, the owner of the software is usually different from the user who initiates the patching job. The patching application internally switches the user context to the owner of the software and patches the software. To emulate such a case, the patch administrator will set the ORACLE_HOME credentials to the owner of the ORACLE_HOME. The Grid Control user who executes the patching job will be unaware of the credentials. The patching job will internally execute as the owner of the ORACLE_HOME. Grid Control will audit the patching job and capture the name of the Grid Control user who initiated the job. For example, if the owner of the ORACLE_HOME is "X", the patch super administrator in Grid Control is "Y" and the target administrator in Grid Control is "Y". "Y" will set the ORACLE_HOME credential to "X" with the password, using EMCLI. "Z" will submit the patching job using the already stored preferred credentials. Grid Control will audit the job as submitted by "Z".

The following is an example for setting the Oracle Home credentials using command line:

emcli set_credential -target_type=host -target_name=val1 -credential_set=OHCreds -column="OHUsername:val2;OHPassword:val3"
-oracle_homes="val4"

where:

val1 = Hostname

val2 = Oracle Home user name

val3 = Oracle Home password

val4 = Oracle Home location

You can also set credentials for multiple Oracle Homes on the same host using the following command:

emcli set_credential -target_type=host -target_name=val1 -credential_set=OHCreds -column="OHUsername:val2;OHPassword:val3" 
-oracle_homes="val4;val5

where

val1 = Hostname

val2 = Oracle Home user name

val3 = Oracle Home password

val4 = Oracle Home location 1

val5 = Oracle Home location 2


Note:

Only one host can be passed to the verb.* If one wants multiple Oracle Home credentials on multiple hosts, then you will need Shell or Perl script to read lines, one at a time, from a file containing the host, credential values, and home location, and call the emcli set_credential verb for each row in the file.

The emcli set_credential command sets preferred credentials for given users. The following table describes the input values to the emcli set_credential command.

Table 2-6  emcli set_credential Parameters

ParameterInput ValueDescription

-target_type

-target_type ="ttype"

Type of target. Must be "host" in case the "-oracle_homes" parameter is specified.

-target_name

[-target_name="tname"]

Name of target. Omit this argument to set enterprise preferred credentials. Must be hostname in case "-oracle_homes" parameter is specified

-credential_set

-credential_set="cred_set"

Credential set affected.

-user

[-user="user"]

Enterprise Manager user whose credentials are affected. If omitted, the current user's credentials are affected.

-columns

-columns="col1:newval1;col2:newval2;..."

The name and new value of the column(s) to set. Every column of the credential set must be specified. Alternatively, a tag from the -input_file argument may be used so that the credential values are not seen on the command line. This argument may be specified more than once.

-input_file

[-input_file="tag1:file_path1;tag2:file_path2;..."]

Path of file that has -columns argument(s). This option is used to hide passwords. Each path must be accompanied by a tag which is referenced in the -columns argument. This argument may be specified more than once.

-oracle_homes

[-oracle_homes="home1;home2"]

Name of Oracle Homes on the target host. Credentials will be added/updated for all specified home


Patching Oracle Homes When the User is Locked

To patch an Oracle Home used by a user "Oracle" and the user is locked:

  1. Edit the default patching script and prepend sudo or sudo -u or pbrun -u to the default patching step. You need to set a policy (by editing the sudoers file) to allow the user submitting the job (who must be a valid operating system user) to be able to run sudo or pbrun without being prompted for password.


Note:

You cannot patch Oracle Homes without targets. This must be done by using the Patching wizard.

Cloning Oracle Homes

The cloning application is wizard-driven. The source of the Oracle Home being cloned may be either an installed Oracle Home or a Software Library. Following are the steps in the cloning process:

  1. If the source is an installed Oracle Home, then, after selecting the Oracle Home, a user will need to specify the Oracle Home credentials. These credentials once specified for an Oracle Home are stored in the repository. The next time a user clones the same Oracle Home, these credentials are automatically populated. Other parameters queried from the user at this point is a temporary location (on the source computer) and the list of files to be excluded from the Oracle Home. If the cloning source is a Software Library, the source Oracle Home credentials will not be queried for.

  2. The user needs to specify the target location and provide the required credentials for each target location. These credentials will be the Oracle Home credentials for each of these target locations. Subsequently, if a user selects any of these cloned Oracle Homes as a source, the Oracle Home credentials are automatically populated.

  3. Depending on the product being cloned, the user can view the Enterprise Manager page where query parameters required for the particular product being cloned are displayed.

  4. The user can, then, view the execution of user-supplied pre-cloning and post-cloning scripts and the root.sh script. The root.sh script will always be run with sudo privileges, but the user has the option to decide if the pre-cloning and post-cloning scripts run with sudo privileges.Finally, the user can schedule the cloning job at a convenient time.

For more information about cloning, refer to the Enterprise Manager Online Help.

PK? PKOXEOEBPS/repository.htm Maintaining and Troubleshooting the Management Repository

10 Maintaining and Troubleshooting the Management Repository

This chapter describes maintenance and troubleshooting techniques for maintaining a well-performing Management Repository.

Specifically, this chapter contains the following sections:

Management Repository Deployment Guidelines

To be sure that your management data is secure, reliable, and always available, consider the following settings and configuration guidelines when you are deploying the Management Repository:

  • Install a RAID-capable Logical Volume Manager (LVM) or hardware RAID on the system where the Management Repository resides. At a minimum the operating system must support disk mirroring and stripping. Configure all the Management Repository data files with some redundant configuration.

  • Use Real Application Clusters to provide the highest levels of availability for the Management Repository.

  • If you use Enterprise Manager to alert administrators of errors or availability issues in a production environment, be sure that the Grid Control components are configured with the same level of availability. At a minimum, consider using Oracle Data Guard to mirror the Management Repository database. Configure the Data Guard environment for no data loss.


    See Also:

    Oracle Database High Availability Architecture and Best Practices

    Oracle Data Guard Concepts and Administration


  • Oracle strongly recommends that archive logging be turned on and that a comprehensive backup strategy be in place prior to an Enterprise Manager implementation going live in a production environment. The backup strategy should include both incremental and full backups as required.


    See Also:

    Oracle Enterprise Manager Grid Control Installation and Basic Configuration for information about the database initialization parameters required for the Management Repository

Management Repository Data Retention Policies

When the various components of Enterprise Manager are configured and running efficiently, the Oracle Management Service gathers large amounts of raw data from the Management Agents running on your managed hosts and loads that data into the Management Repository. This data is the raw information that is later aggregated, organized, and presented to you in the Grid Control Console.

After the Oracle Management Service loads information into the Management Repository, Enterprise Manager aggregates and purges the data over time.

The following sections describe:

  • The default aggregation and purging policies used to maintain data in the Management Repository.

  • How you can modify the length of time the data is retained before it is aggregated and then purged from the Management Repository.

Management Repository Default Aggregation and Purging Policies

Enterprise Manager aggregates your management data by hour and by day to minimize the size of the Management Repository. Before the data is aggregated, each data point is stored in a raw data table. Raw data is rolled up, or aggregated, into a one-hour aggregated metric table. One-hour records are then rolled up into a one-day table.

After Enterprise Manager aggregates the data, the data is then considered eligible for purging. A certain period of time has to pass for data to actually be purged. This period of time is called the retention time.

The raw data, with the highest insert volume, has the shortest default retention time, which is set to 7 days. As a result, 7 days after it is aggregated into a one-hour record, a raw data point is eligible for purging.

One-hour aggregate data records are purged 31 days after they are rolled up to the one-day data table. The highest level of aggregation, one day, is kept for 365 days.

The default data retention policies are summarized in Table 10-1.

Table 10-1 Default Repository Purging Policies

Aggregate LevelRetention Time

Raw metric data

7 days

One-hour aggregated metric data

31 days

One-day aggregated metric data

365 days


If you have configured and enabled Application Performance Management, Enterprise Manager also gathers, saves, aggregates, and purges response time data. The response time data is purged using policies similar to those used for metric data. The Application Performance Management purging policies are shown in Table 10-2.

Table 10-2 Default Repository Purging Policies for Application Performance Management Data

Aggregate LevelRetention Time

Raw response time data

24 hours

One-hour aggregated response time data

7 days

One-hour distribution response time data

24 hours

One-day aggregated response time data

31 days

One-day distribution aggregated response time data

31 days


Management Repository Default Aggregation and Purging Policies for Other Management Data

Besides the metric data and Application Performance Monitoring data, other types of Enterprise Manager data accumulates over time in the Management Repository.

For example, the last availability record for a target will also remain in the Management Repository indefinitely, so the last known state of a target is preserved.

Modifying the Default Aggregation and Purging Policies

The Enterprise Manager default aggregation and purging policies were designed to provide the most available data for analysis while still providing the best performance and disk-space requirements for the Management Repository. As a result, you should not modify these policies to improve performance or increase your available disk space. Modifying these default policies can affect the performance of the Management Repository and have adverse reactions on the scalability of your Enterprise Manager installation.

However, if you plan to extract or review the raw or aggregated data using data analysis tools other than Enterprise Manager, you may want to increase the amount of raw or aggregated data available in the Management Repository. You can accomplish this by increasing the retention times for the raw or aggregated data.

To modify the default retention time for each level of management data in the Management Repository, you must insert additional rows into the MGMT_PARAMETERS table in the Management Repository database. Table 10-3 shows the parameters you must insert into the MGMT_PARAMETERS table to modify the retention time for each of the raw data and aggregate data tables.

Table names that contain "_RT_" indicate tables used for Application Performance Monitoring response time data. In the Table Name column, replace datatype with one of the three response time data types: DOMAIN, IP, or URL.

Table 10-3 Parameters for Modifying Default Data Retention Times in the Management Repository

Table NameParameter in MGMT_PARAMETERS TableDefault Retention Value

MGMT_METRICS_RAW

mgmt_raw_keep_window

7 days

MGMT_METRICS_1HOUR

mgmt_hour_keep_window

31 days

MGMT_METRICS_1DAY

mgmt_day_keep_window

365 days

MGMT_RT_METRICS_RAW

mgmt_rt_keep_window

24 hours

MGMT_RT_datatype_1HOUR

mgmt_rt_hour_keep_window

7 days

MGMT_RT_datatype_1DAY

mgmt_rt_day_keep_window

31 days

MGMT_RT_datatype_DIST_1HOUR

mgmt_rt_dist_hour_keep_window

24 hours

MGMT_RT_datatype_DIST_1DAY

mgmt_rt_dist_day_keep_window

31 days



Note:

If the first three tables listed in Table 8-3 are not partitioned, the Default Retention Value for each is 1, 7, and 31 days respectively, rather than the 7, 31, and 365 days listed for partitioned tables.

For example, to change the default retention time for the table MGMT_METRICS_RAW from seven days to 14 days:

  1. Use SQL*Plus to connect to the Management Repository database as the Management Repository user.

    The default Management Repository user is sysman.

  2. Enter the following SQL to insert the parameter and change the default value:

    INSERT INTO MGMT_PARAMETERS (PARAMETER_NAME, PARAMETER_VALUE)  VALUES ('mgmt_raw_keep_window','14');
    

Similarly, to change from the default retention time for all of the MGMT_RT_datatype_1DAY tables from 31 days to 100 days:

  1. Use SQL*Plus to connect to the Management Repository database as the Management Repository user.

    The default Management Repository user is sysman.

  2. Enter the following SQL to insert the parameter and change the default value:

    INSERT INTO MGMT_PARAMETERS (PARAMETER_NAME, PARAMETER_VALUE)  VALUES ('mgmt_rt_day_keep_window', '100');
    

Modifying Data Retention Policies When Targets Are Deleted

By default, when you delete a target from the Grid Control console, Enterprise Manager automatically deletes all target data from the Management Repository.

However, deleting raw and aggregated metric data for database and other data-rich targets is a resource consuming operation. Targets can have hundreds of thousands of rows of data and the act of deleting this data can degrade performance of Enterprise Manager for the duration of the deletion, especially when several targets are deleted at once.To avoid this resource-consuming operation, you can prevent Enterprise Manager from performing this task each time you delete a target. When you prevent Enterprise Manager from performing this task, the metric data for deleted targets is not purged as part of target deletion task; instead, it is purged as part of the regular purge mechanism, which is more efficient.

In addition, Oracle strongly recommends that you do not add new targets with the same name and type as the deleted targets within 24 hours of target deletion. Adding a new target with the same name and type will result in the Grid Control console showing data belonging to the deleted target for the first 24 hours.

To disable raw metric data deletion:

  1. Use SQL*Plus to connect to the Management Repository as the Management Repository user.

    The default Management Repository user is SYSMAN. For example:

    SQL> connect sysman/sysman_password;
    
  2. To disable metric deletion, run the following SQL command.

    SQL> EXEC MGMT_ADMIN.DISABLE_METRIC_DELETION();
    SQL> COMMIT;
    

To enable metric deletion at a later point, run the following SQL command:

  1. Use SQL*Plus to connect to the Management Repository as the Management Repository user.

    The default Management Repository user is SYSMAN. For example:

    SQL> connect sysman/oldpassword;
    
  2. To enable metric deletion, run the following SQL command.

    SQL> EXEC MGMT_ADMIN.ENABLE_METRIC_DELETION();
    SQL> COMMIT;
    

How to Modify the Retention Period of Job History

Enterprise Manager Grid Control has a default purge policy which removes all finished job details which are older than 30 days. This section provides details for modifying this default purge policy.

The actual purging of completed job history is implemented via a DBMS job that runs once a day in the repository database. When the job runs, it looks for finished jobs that are 'n' number of days older than the current time (value of sysdate in the repository database) and deletes these jobs. The value of 'n' is, by default, set to 30 days.

The default purge policy cannot be modified via the Enterprise Manager console, but it can be changed using SQL*Plus.

To modify this purge policy, follow these steps:

  1. Log in to the repository database as the SYSMAN user, via SQL*Plus

  2. Check the current values for the purge policies using the following command:

    SQL> select * from mgmt_job_purge_policies;

    POLICY_NAME                      TIME_FRAME
    -------------------------------- ----------
    SYSPURGE_POLICY                          30
    REFRESHFROMMETALINKPURGEPOLICY            7
    FIXINVENTORYPURGEPOLICY                   7
    OPATCHPATCHUPDATE_PAPURGEPOLICY           7
    

    The purge policy responsible for the job deletion is called SYSPURGE_POLICY. As seen above, the default value is set to 30 days.

  3. To change the time period, you must drop and re-create the policy with a different time frame:

    SQL> execute MGMT_JOBS.drop_purge_policy('SYSPURGE_POLICY');

    PL/SQL procedure successfully completed.
    

    SQL> execute MGMT_JOBS.register_purge_policy('SYSPURGE_POLICY', 60, null);

    PL/SQL procedure successfully completed.
    

    SQL> COMMIT;

    Commit complete.
    

    SQL> select * from mgmt_job_purge_policies;

    POLICY_NAME                      TIME_FRAME
    -------------------------------- ---------- 
    SYSPURGE_POLICY                          60
    ....
    

The above commands increase the retention period to 60 days. The timeframe can also be reduced below 30 days, depending on the requirement.

You can check when the purge job will be executed next. The actual time that the job runs may vary with each Enterprise Manager installation. To determine this time in your setup follow these steps:

  1. Login to the Repository database using the SYSMAN account

  2. Execute the following command:

    SQL> alter session set nls_date_format='mm/dd/yy hh:mi:ss pm';

    SQL> select what, next_date from user_jobs where what like '%JOB_ENGINE%';

    WHAT
    ------------------------------------------------------------------------------
    NEXT_DATE
    --------------------
    MGMT_JOB_ENGINE.apply_purge_policies();
    09/23/08 10:26:17 am
    

    In this example, the purge policy DBMS job will run every day at 10:26:17 AM, repository time.

Changing the SYSMAN Password

The SYSMAN account is the default super user account used to set up and administer Enterprise Manager. It is also the database account that owns the objects stored in the Oracle Management Repository. From this account, you can set up additional administrator accounts and set up Enterprise Manager for use in your organization.

The SYSMAN account is created automatically in the Management Repository database during the Enterprise Manager installation. You also provide a password for the SYSMAN account during the installation.


See Also:

Oracle Enterprise Manager Grid Control Installation and Basic Configuration for information about installing Enterprise Manager

If you later need to change the SYSMAN database account password, use the following procedure:

  1. Shut down all the Oracle Management Service instances that are associated with the Management Repository.

  2. Stop the agent that is monitoring the target OMS and Repository.

    Failure to do this will result in the agent attempting to connect to the target with a wrong password once it is changed with SQL*Plus. This may also result in the SYSMAN account being locked which can subsequently prevent logins to the Grid Control console to change the password of the target OMS and Repository.

  3. Change the password of the SYSMAN database account using the following SQL*Plus commands:

    SQL>connect sysman/oldpassword;
    SQL>alter user sysman identified by newpassword;
    
  4. For each Management Service associated with the Management Repository, locate the emoms.properties configuration file.

    The emoms.properties file can be found in the following directory of the Oracle Application Server Home where the Oracle Management Service is installed and deployed:

    IAS_HOME/sysman/config/

  5. Locate the following entries in the emoms.properties file:

    oracle.sysman.eml.mntr.emdRepPwd=ece067ffc15edc4f
    oracle.sysman.eml.mntr.emdRepPwdEncrypted=TRUE
    
  6. Enter your new password in the first entry and enter FALSE in the second entry.

    For example:

    oracle.sysman.eml.mntr.emdRepPwd=new_password
    oracle.sysman.eml.mntr.emdRepPwdEncrypted=FALSE
    
  7. Save and exit the emoms.properties file and restart each Management Service associated with the Management Repository.

  8. In the Grid Control console, click the Targets tab and then click All Targets on the sub tab.

  9. Select the Management Services and Repository target and click Configure. Enterprise Manager displays the Monitoring Configurations page.

  10. Enter the new password in the Repository password field and click OK.

  11. After the Management Service has started, you can check the contents of the emoms.properties file to be sure the password you entered has been encrypted.

    For example, the entries should appear as follows:

    oracle.sysman.eml.mntr.emdRepPwd=ece067ffc15edc4f
    oracle.sysman.eml.mntr.emdRepPwdEncrypted=TRUE
    

Overview of the MGMT_VIEW User

During repository creation, the MGMT_VIEW user is created. This view is used by Grid Control for the reporting framework to execute queries for Table from SQL and Chart from SQL report elements. The OMS is the only entity that uses the account so there is no need to know the password. However, you can still change the password if you choose, which requires that you bounce the OMS. To change the password, you can use either a PL/SQL call or an EMCTL command:

PL/SQL:

SQL> exec mgmt_view_priv.change_view_user_password('<random pwd>');

EMCTL command:

emctl config oms –change_view_user_pwd [-sysman_pwd <pwd>] [-user_pwd <pwd>] [-autogenerate]

Dropping and Recreating the Management Repository

This section provides information about dropping the Management Repository from your existing database and recreating the Management Repository after you install Enterprise Manager.

Dropping the Management Repository

To recreate the Management Repository, you first remove the Enterprise Manager schema from your Management Repository database. You accomplish this task using the -action drop argument to the RepManager script, which is described in the following procedure.

To remove the Management Repository from your database:

  1. Locate the RepManager script in the following directory of the Oracle Application Server Home where you have installed and deployed the Management Service:

    IAS_HOME/sysman/admin/emdrep/bin
    
  2. At the command prompt, enter the following command:

    $PROMPT> RepManager repository_host repository_port repository_SID 
    -sys_password password_for_sys_account -action drop
    

    In this syntax example:

    • repository_host is the machine name where the Management Repository database is located

    • repository_port is the Management Repository database listener port address, usually 1521 or 1526

    • repository_SID is the Management Repository database system identifier

    • password_for_sys_account is the password of the SYS user for the database. For example, change_on_install.

    • -action drop indicates that you want to drop the Management Repository.

Alternatively, you can use a connect descriptor to identify the database on the RepManager command line. The connect descriptor identifies the host, port, and name of the database using a standard Oracle database syntax.

For example, you can use the connect descriptor as follows to create the Management Repository:

$PROMPT> ./RepManager -connect "(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)
(HOST=host1)(PORT=1521)) (CONNECT_DATE=(SERVICE_NAME=servicename)))"
-sys_password efkl34lmn -action drop

See Also:

"Establishing a Connection and Testing the Network" in the Oracle Database Net Services Administrator's Guide for more information about connecting to a database using connect descriptors

Recreating the Management Repository

The preferred method for creating the Management Repository is to create the Management Repository during the Enterprise Manager installation procedure, which is performed using Oracle Universal Installer.


See Also:

Oracle Enterprise Manager Grid Control Installation and Basic Configuration for information about installing Enterprise Manager

However, if you need to recreate the Management Repository in an existing database, you can use the RepManager script, which is installed when you install the Management Service. Refer to the following sections for more information:

Using the RepManager Script to Create the Management Repository

To create a Management Repository in an existing database:

  1. Review the hardware and software requirements for the Management Repository as described in Oracle Enterprise Manager Grid Control Installation and Basic Configuration. and review the section "Management Repository Deployment Guidelines".

  2. Locate the RepManager script in the following directory of the Oracle Management Service home directory:

    ORACLE_HOME/sysman/admin/emdrep/bin
    
  3. At the command prompt, enter the following command:

    $PROMPT> ./RepManager repository_host repository_port repository_SID  -sys_password password_for_sys_account -action create
    

In this syntax example:

  • repository_host is the machine name where the Management Repository database is located

  • repository_port is the Management Repository database listener port address, usually 1521 or 1526

  • repository_SID is the Management Repository database system identifier

  • password_for_sys_account is the password of the SYS user for the database. For example, change_on_install.

Enterprise Manager creates the Management Repository in the database you specified in the command line.

Using a Connect Descriptor to Identify the Management Repository Database

Alternatively, you can use a connect descriptor to identify the database on the RepManager command line. The connect descriptor identifies the host, port, and name of the database using a standard Oracle database syntax.

For example, you can use the connect descriptor as follows to create the Management Repository:

$PROMPT> ./RepManager -connect "(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=host1)(PORT=1521)) (CONNECT_DATA=(SERVICE_NAME=servicename)))" -sys_password efkl34lmn -action create

See Also:

"Establishing a Connection and Testing the Network" in the Oracle Database Net Services Administrator's Guide for more information about connecting to a database using a connect descriptor

The ability to use a connect string allows you to provide an address list as part of the connection string. The following example shows how you can provide an address list consisting of two listeners as part of the RepManager command line. If a listener on one host becomes unavailable, the second listener can still accept incoming requests:

$PROMPT> ./RepManager -connect "(DESCRIPTION=
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=host1)(PORT=1521)
(ADDRESS=(PROTOCOL=TCP)(HOST=host2)(PORT=1521)
(CONNECT_DATE=(SERVICE_NAME=servicename)))"
-sys_password efkl34lmn -action create

See Also:

Oracle Database High Availability Architecture and Best Practices

Oracle Enterprise Manager Grid Control Installation and Basic Configuration


Troubleshooting Management Repository Creation Errors

Oracle Universal Installer creates the Management Repository using a configuration step at the end of the installation process. If the repository configuration tool fails, note the exact error messages displayed in the configuration tools window, wait until the other configuration tools have finished, exit from Universal Installer, and then use the following sections to troubleshoot the problem.

Package Body Does Not Exist Error While Creating the Management Repository

If the creation of your Management Repository is interrupted, you may receive the following when you attempt to create or drop the Management Repository at a later time:

SQL> ERROR:
ORA-00604: error occurred at recursive SQL level 1
ORA-04068: existing state of packages has been discarded
ORA-04067: not executed, package body "SYSMAN.MGMT_USER" does not exist
ORA-06508: PL/SQL: could not find program unit being called
ORA-06512: at "SYSMAN.SETEMUSERCONTEXT", line 5
ORA-06512: at "SYSMAN.CLEAR_EMCONTEXT_ON_LOGOFF", line 4
ORA-06512: at line 4

To fix this problem, see "General Troubleshooting Techniques for Creating the Management Repository".

Server Connection Hung Error While Creating the Management Repository

If you receive an error such as the following when you try to connect to the Management Repository database, you are likely using an unsupported version of the Oracle Database:

Server Connection Hung

To remedy the problem, upgrade your database to the supported version as described in Oracle Enterprise Manager Grid Control Installation and Basic Configuration.

General Troubleshooting Techniques for Creating the Management Repository

If you encounter an error while creating the Management Repository, drop the repository by running the -drop argument to the RepManager script.

If the RepManager script drops the repository successfully, try creating the Management Repository again.

If you encounter errors while dropping the Management Repository, do the following:

  1. Connect to the database as SYSDBA using SQL*Plus.

  2. Check to see if the SYSMAN database user exists in the Management Repository database.

    For example, use the following command to see if the SYSMAN user exists:

    prompt> SELECT username FROM DBA_USERS WHERE username='SYSMAN';
    
  3. If the SYSMAN user exists, drop the user by entering the following SQL*Plus command:

    prompt> DROP USER SYSMAN CASCADE;
    
  4. Check to see if the following triggers exist:

    SYSMAN.EMD_USER_LOGOFF
    SYSMAN.EMD_USER_LOGON
    

    For example, use the following command to see if the EMD_USER_LOGOFF trigger exists in the database:

    prompt> SELECT trigger_name FROM ALL_TRIGGERS 
            WHERE trigger_name='EMD_USER_LOGOFF';
    
  5. If the triggers exist, drop them from the database using the following commands:

    prompt> DROP TRIGGER SYSMAN.EMD_USER_LOGOFF;
    prompt> DROP TRIGGER SYSMAN.EMD_USER_LOGON;
    

Cross Platform Enterprise Manager Repository Migration

There are user requirements for migrating an Enterprise Manager repository across servers - same and cross platforms.

The Enterprise Manager repository migration process is not exactly the same as database migration. In case of Enterprise Manager Repository migration you must take care of Enterprise Manager specific data, options, and pre-requisites for the repository move. You should make sure data integrity is maintained from both the Enterprise Manager and Oracle database perspective.

This brings up need for defining the process that can be followed by end users for successful and reliable migration of repository in minimum time and with maximum efficiency.

The overall strategy for migration depends on:

  • The source and target database version

  • The amount of data/size of repository

  • Actual data to migrate [selective/full migration]

Cross platform transportable tablespace along with data pump (for metadata) is the fastest and best approach for moving large Enterprise Manager Grid Control repository from one platform to another. Other option that can be considered for migration is to use Data Pump for both data and metadata moves but this would require more time than the cross platform transportable tablespace approach for the same amount of data. The advantage to using the data pump approach is that it provides granular control over options and the overall process, as in the case of selective data being migrated and not the whole of source data. If the source and target is not on version 10g then export/import is the only way to get the data migrated cross platform.

More details on cross platform transportable tablespace, data pump, and export/import options can be found at the Oracle Technology Network (OTN) or in the Oracle Database Administrator's Guide.

Common Prerequisites

The following lists the common prerequisites for a repository migration:

  • Source and target database must use the same character set and should be at same version.

  • Source and target database should meet all the pre-requisites mentioned for Enterprise Manager Repository software requirements mentioned in Enterprise Manager install guide.

  • If source and target database are NOT on 10g - only Export/Import can be used for cross platform migration

  • If Source and target database are on 10g - either of three options Cross platform transportable tablespaces migration, Data Pump or Export/Import can be used for cross platform repository migration

  • You cannot transport a tablespace to a target database in which a tablespace with the same name already exists. However, you can rename either the tablespace to be transported or the destination tablespace before the transport operation.

  • To plug a transportable tablespace set into an Oracle Database on a different platform, both databases must have compatibility set to at least 10.0.

  • Most of the platforms(but not all) are supported for cross-platform tablespace transport. You can query the V$TRANSPORTABLE_PLATFORM view to see the platforms that are supported, and to determine their platform IDs and their endian format (byte ordering).

  • Source and Destination host should have EM agent running and configured to the instance which is to be migrated

  • If target database has EM repository installed, it should be first dropped using RepManager before target database related steps are carried out.

Methodologies

The following sections discuss the methodologies of a repository migration.

Cross Platform Transportable Tablespaces

Oracle's transportable tablespace feature allows users to quickly move a user tablespace across Oracle databases. It is the most efficient way to move bulk data between databases. Prior to Oracle Database 10g, if you want to transport a tablespace, both source and target databases need to be on the same platform. Oracle Database 10g adds the cross platform support for transportable tablespaces. With the cross platform transportable tablespace, you can transport tablespaces across platforms.

Cross platform transportable tablespaces allows a database to be migrated from one platform to another (use with Data Pump or Import/Export).

Preparation for Transportable Tablespaces

Use these steps to prepare for transportable tablespaces:

  1. Prepare set of user tablespaces and Check for containment violation

    execute DBMS_TTS.TRANSPORT_SET_CHECK('MGMT_TABLESPACE,MGMT_ECM_DEPOT_TS', TRUE);

    select * FROM transport_set_violations;

  2. Shutdown OMS instances and prepare for migration

    Shutdown OMS, set job queue_processes to 0 and run

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_remove_dbms_jobs.sql

  3. Make the tablespaces to be transported read only

    alter tablespace MGMT_TABLESPACE read only;

    alter tablespace MGMT_ECM_DEPOT_TS read only;

Extract metadata

Extract Metadata for transportable tablespaces using Data Pump Utility:

  1. Create data pump directory

    create directory data_pump_dir as '/scratch/gachawla/EM102/ttsdata';

  2. Extract the metadata using data pump (or export )

    expdp DUMPFILE=ttsem102.dmp TRANSPORT_TABLESPACES=MGMT_TABLESPACE,MGMT_ECM_DEPOT_TS TRANSPORT_FULL_CHECK=Y

  3. Extract other objects ( packages, procedures, functions, temporary tables etc - Not contained in user tablespaces)

    expdp SCHEMAS=SYSMAN CONTENT=METADATA_ONLY EXCLUDE=INDEX,CONSTRAINT DUMPFILE=data_pump_dir:postexp.dmp LOGFILE=data_pump_dir:postexp.log JOB_NAME=expmet

Endian check and conversion

Run Endian check and convert the datafiles if endian is different between source and destination:

  1. For Endian check, run this on both source and destination database

    SELECT endian_format

    FROM v$transportable_platform tp, v$database d

    WHERE tp.platform_name = d.platform_name;

    If the source platform and the target platform are of different endianness, then an additional step must be done on either the source or target platform to convert the tablespace being transported to the target format. If they are of the same endianness, then no conversion is necessary and tablespaces can be transported as if they were on the same platform.

    Example:

    Source Endian 
    Linux IA (32-bit) - Little
     
    Destination Endian  
    Solaris[tm] OE (32-bit) - Big
    
  2. Ship datafiles, metadata dump to target and Convert datafiles using RMAN

    Ship the datafiles and the metadata dump to target and On target convert all datafiles to destination endian:

    CONVERT DATAFILE
    '/d14/em10g/oradata/em102/mgmt.dbf',
    '/d14/em10g/oradata/em102/mgmt_ecm_depot1.dbf'
    FROM PLATFORM 'Linux IA (32-bit)';
    

    Conversion via RMAN can be done either on source or target (For more details refer RMAN doc). Parallelism can be used to speed up the process if the user tablespaces contains multiple datafiles.

Import metadata and plugin tablespaces

Use the following steps to import metadata and plugin tablespaces:

  1. Run RepManager to drop target repository (if target database has EM repository installed)

    RepManager repository_host repository_port repository_SID -sys_password password_for_sys_account -action drop

  2. Run pre import steps to create sysman user and grant privs on target database

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_repos_user.sql

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_pre_import.sql

  3. Invoke Data Pump utility to plug the set of tablespaces into the target database.

    impdp DUMPFILE=ttsem102.dmp DIRECTORY=data_pump_dir

    TRANSPORT_DATAFILES=/d14/em10g/oradata/em102/mgmt.dbf,/d14/em10g/oradata/em102/mgmt_ecm_depot1.dbf

  4. Import other objects (packages, procedures, functions etc)

    impdp CONTENT=METADATA_ONLY EXCLUDE=INDEX,CONSTRAINT DUMPFILE=data_pump_dir:postexp.dmp LOGFILE=data_pump_dir:postexp.log

Post Plug In Steps

Follow these post plug in steps:

  1. Run post plugin steps to recompile any invalids, create public synonyms, create other users, enable VPD policy, repin packages

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_synonyms.sql

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_post_import.sql

    Check for invalid objects - compare source and destination schemas for any discrepancy in counts and invalids.

  2. Bring user tablespaces back to read write mode

    alter tablespace MGMT_TABLESPACE read write;

    alter tablespace MGMT_ECM_DEPOT_TS read write;

  3. Submit EM dbms jobs

    Reset back job_queue_processes to original value and run

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_submit_dbms_jobs.sql

  4. Update OMS properties and startup OMS

    Update emoms.properties to reflect the migrated repository. Update host name - oracle.sysman.eml.mntr.emdRepServer and port with the correct value and start the OMS.

  5. Relocate Management Services and Repository target

    If Management Services and repository target needs to be migrated to the destination host, run em_assoc. handle_relocated_target to relocate the target or recreate the target on the target host.

  6. Discover/relocate Database and database Listener targets

    Discover the target database and listener in EM or relocate the targets from source agent to destination agent.

Data Pump

Oracle Data Pump technology enables high-speed, parallel movement of bulk data and metadata from one database to another. Data Pump uses APIs to load and unload data instead of usual SQL commands. Data pump operations can be run via EM interface and is very useful for cross platform database migration.

The migration of the database using the Data Pump export and Data Pump import tools comprises these steps: export the data into a dump file on the source server with the expdp command; copy or move the dump file to the target server; and import the dump file into Oracle on the target server by using the impdp command; and run post import EM specific steps.

Tuning parameters that were used in original Export and Import, such as BUFFER and RECORDLENGTH, are neither required nor supported by Data Pump Export and Import

Prepare for Data Pump

Use the following steps to prepare for data pump:

  1. Pre-requisite for using Data pump for EM repository

    Impdp fails for EM repository because of data pump bug - Bug 4386766 - IMPDP WITH COMPRESSED INDEXES FAILS WITH ORA-14071 AND ORA-39083. This bug is fixed in 10.2. Backport is available for 10.1.0.4. This RDBMS patch has to be applied to use expdp/impdp for EM repository migration or workaround is to use exp/imp for extract and import.

  2. Create data pump directory

    Create directory data_pump_dir as '/scratch/gachawla/EM102/ttsdata';

  3. Shutdown OMS instances and prepare for migration

    Shutdown OMS, set job queue_processes to 0 and run @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_remove_dbms_jobs.sql

    To improve throughput of a job, PARALLEL parameter should be used to set a degree of parallelism that takes maximum advantage of current conditions. In general, the degree of parallelism should be set to more than twice the number of CPUs on an instance.

    All data pump actions are performed by multiple jobs (server processes not DBMS_JOB jobs). These jobs are controlled by a master control process which uses Advanced Queuing. At runtime an advanced queue table, named after the job name, is created and used by the master control process. The table is dropped on completion of the data pump job. The job and the advanced queue can be named using the JOB_NAME parameter.

    DBMS_DATAPUMP APIs can also be used to do data pump export/import. Please refer to Data pump section in 10g administration manual for all the options.

Data Pump Export

Use these steps to run data pump export:

  1. Run data pump export:

    expdp FULL=y DUMPFILE=data_pump_dir:dpfull1%U.dmp, data_pump_dir:dpfull2%U.dmp PARALLEL=4 LOGFILE=data_pump_dir:dpexpfull.log JOB_NAME=dpexpfull
    Verify the logs for any errors during export
    

    Data pump direct path export sometimes fails for mgmt_metrics_raw and raises ORA 600. This is due to Bug 4221775 (4233303). This bug is fixed in 10.2. Workaround: if using expdp data pump for mgmt_metrics_raw , run expdp with ACCESS_METHOD+EXTERNAL_TABLE parameter.

    expdp directory=db_export dumpfile=exp_st2.dmp logfile=exp_st2.log tables=sysman.mgmt_metrics_raw access_method=external_table

Data Pump Import

Use these steps to run data pump import:

  1. Run RepManager to drop target repository (if target database has EM repository installed)

    RepManager repository_host repository_port repository_SID -sys_password password_for_sys_account -action drop

  2. Prepare the target database

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_tablespaces.sql

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_repos_user.sql

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_pre_import.sql

  3. Run data pump import

    Impdp FULL=y DUMPFILE=data_pump_dir:dpfull1%U.dmp, data_pump_dir:dpfull2%U.dmp PARALLEL=4 LOGFILE=data_pump_dir:dpimpfull.log JOB_NAME=dpimpfull

    Verify the logs for any issues with the import.

Post Import EM Steps

Use the following steps for post import Enterprise Manager steps:

  1. Run post plugin steps to recompile any invalids, create public synonyms, create other users, enable VPD policy, repin packages

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_synonyms.sql

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_post_import.sql

    Check for invalid objects - compare source and destination schemas for any discrepancy in counts and invalids.

  2. Submit EM dbms jobs

    Reset back job_queue_processes to original value and run

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_submit_dbms_jobs.sql

  3. Update OMS properties and startup OMS

    Update emoms.properties to reflect the migrated repository. Update host name - oracle.sysman.eml.mntr.emdRepServer and port with the correct value and start the OMS.

  4. Relocate Management Services and Repository target

    If Management Services and repository target needs to be migrated to the destination host, run em_assoc. handle_relocated_target to relocate the target or recreate the target on the target host.

  5. Discover/relocate Database and database Listener targets

    Discover the target database and listener in EM or relocate the targets from source agent to destination agent.

Export/Import

If the source and destination database is non-10g, then export/import is the only option for cross platform database migration.

For performance improvement of export/import, set higher value for BUFFER and RECORDLENGTH . Do not export to NFS as it will slow down the process considerably. Direct path can be used to increase performance. Note - As EM uses VPD, conventional mode will only be used by Oracle on tables where policy is defined.

Also User running export should have EXEMPT ACCESS POLICY privilege to export all rows as that user is then exempt from VPD policy enforcement. SYS is always exempted from VPD or Oracle Label Security policy enforcement, regardless of the export mode, application, or utility that is used to extract data from the database.

Prepare for Export/Import

Use the following steps to prepare for Export/Import:

  1. Mgmt_metrics_raw partitions check

    select table_name,partitioning_type type,
    partition_count count, subpartitioning_type subtype from
    dba_part_tables where table_name = 'MGMT_METRICS_RAW'
    

    If MGMT_METRICS_RAW has more than 3276 partitions please see Bug 4376351 - This is Fixed in 10.2 . Workaround is to export mgmt_metrics_raw in conventional mode.

  2. Shutdown OMS instances and prepare for migration

    Shutdown OMS, set job queue_processes to 0 and run @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_remove_dbms_jobs.sql

Export

Follow these steps for export:

  1. Export data

    exp full=y constraints=n indexes=n compress=y file=fullem102_1.dmp log=fullem102exp_1.log

  2. Export without data and with constraints

    exp full=y constraints=y indexes=y rows=n ignore=y file=fullem102_2.dmp log=fullem102exp_2.log

Import

Follow these steps to import:

  1. Run RepManager to drop target repository (if target database has EM repository installed)

    RepManager repository_host repository_port repository_SID -sys_password password_for_sys_account -action drop

  2. Pre-create the tablespaces and the users in target database

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_tablespaces.sql

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_repos_user.sql

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_pre_import.sql

  3. Import data

    imp full=y constraints=n indexes=n file=fullem102_1.dmp log=fullem102imp_1.log

  4. Import without data and with constraints

    imp full=y constraints=y indexes=y rows=n ignore=y file=fullem102_2.dmp log=fullem102imp_2.log

Post Import EM Steps

Follow these steps for post import EM steps:

  1. Run post plugin steps to recompile any invalids, create public synonyms, create other users, enable VPD policy, repin packages

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_synonyms.sql

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_post_import.sql

    Check for invalid objects - compare source and destination schemas for any discrepancy in counts and invalids.

  2. Submit EM dbms jobs

    Reset back job_queue_processes to original value and run

    @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_submit_dbms_jobs.sql

  3. Update OMS properties and startup OMS

    Update emoms.properties to reflect the migrated repository. Update host name, oracle.sysman.eml.mntr.emdRepServer and port with the correct value and start the OMS.

  4. Relocate Management Services and Repository target

    If Management Services and repository target needs to be migrated to the destination host, run em_assoc. handle_relocated_target to relocate the target or recreate the target on the target host.

  5. Discover/relocate Database and database Listener targets

     n

    Discover the target database and listener in EM or relocate the targets from source agent to destination agent.

Post Migration Verification

These verification steps should be carried out post migration to ensure that the migration was completely successful:

  • Verify any discrepancy in objects by comparing source and target databases through EM

  • Verify migrated database through EM whether database is running without any issues

  • Verify repository operations, dbms jobs and whether any management system errors reported

  • Verify all EM functionalities are working fine after migration

  • Make sure Management Services and Repository target is properly relocated by verifying through EM

Improving the Login Performance of the Console Home Page

Oracle Enterprise Manager now provides an option that will more quickly display the Console Home page even in a scenario where the Management Repository is very large. Normally, factors such as the number of alerts, errors, policies, and critical patches can contribute to delayed displayed times. Since there is no single factor nor any simple way to scale the SQL or user interface, a simple option flag has been added that removes the following page elements for all users.

When the emoms.properties flag, LargeRepository=, is set to true (when normally the default is false), the SQL for the following items is not executed and thus the items will not be displayed on the Console page.

  1. Three sections from the Overview Page segment:

    • All Target Alerts

      • Critical

      • Warning

      • Errors

    • All Target Policy Violations

      • Critical

      • Warning

      • Informational

    • All Target Jobs

      • Problem Executions (last 7 days)

      • Suspended Executions (last 7 days)

  2. The page segment which includes Security Patch Violations and Critical Patch Advisories.

The Deployment Summary section would move up to fill in the vacated space.

PKr  PKOXEOEBPS/req_monit_setup.htm( Setting Up Request Monitoring

12 Setting Up Request Monitoring

The Request Monitoring feature provides end-to-end visibility into requests for Java applications deployed to WebLogic Server. It establishes a clear view of a request flowing through a multi-tier java environment. This chapter describes the procedure to setup and enable the Request Monitoring feature in Enterprise Manager Grid Control. This chapter covers the following:

Prerequisites

Before you enable Request Monitoring, you must do the following:

  • Identify the WebLogic Server targets and J2EE applications that are to be monitored.

  • Ensure that the necessary hardware for the ADP Manager, ADP database, and JVM Diagnostics Manager is available.

Request Monitoring Architecture

The following diagram shows the request monitoring architecture.

Figure 12-1 Request Monitoring Architecture

Surrounding text describes Figure 12-1 .

The JVM Diagnostics Agent and the ADP Agent are deployed on the WebLogic Server targets that are being monitored. These agents collect JVM Diagnostics and Application metrics and periodically report them to the JVM Diagnostics and ADP Managers. The Management Agent (EM Agent) on the ADP Manager machine, periodically runs a Fetchlet to retrieve the metrics from the ADP Manager and uploads them to the Management Repository. The Request Monitoring framework processes these metrics and uses them to monitor and analyze requests.

Setting Up Request Monitoring

To setup end-user request tracing, follow these steps:

  1. Deploy the Oracle Management Service and the Management Repository database. For details, see the Enterprise Manager Basic Installation and Configuration Guide.

  2. Deploy the ADP Manager(s) on the designated machines. For details on deploying the ADP Manager, see the Getting Started with SOA Management Pack Plus Guide. The ADP Manager must be installed as the same user as the Management Agent (as described in the step below).

  3. Deploy the Management Agent on all the machines on which the ADP Managers have been installed.

  4. Deploy the Management Agent on the machine on which the WebLogic server target to be monitored is running. In Enterprise Manager Grid Control, click Targets and Middleware. On the Middleware page, check if the WebLogic Server target to be monitored has been discovered and registered as a target. If it has not been registered, add the WebLogic Server as a target and register it. For details on discovering and registering the WebLogic Server target, see the Oracle Fusion Middleware Administration Guide. If this WebLogic server target is configured to connect to a database running on a remote machine, ensure that the Management Agent is deployed on the remote machine, and confirm that this database is listed as a target in Enterprise Manager Grid Control.

  5. Register the WebLogic Server target with ADP Manager. To do so, click the Application Dependency and Performance link on the Middleware page. Click the Registration tab on the right panel. The following screen is displayed.

    Figure 12-2 RMI Configuration Page

    RMI Configuration Page

    Enter the name of the ADP Manager to be registered and the fully qualified host name of the machine that the ADP Manager is running on. The default port 51099 is pre-populated for you. Click Test Connect. If you can successfully connect to the host machine, click Add to complete the ADP Manager registration. For more details on registering the ADP Manager, see the Getting Started with SOA Management Pack Plus Guide.

  6. Install the JVM Diagnostics Manager and deploy the JVM Diagnostics Agent on WebLogic Server targets. For more details, see the Setting Up the JVM Diagnostics Manager chapter.

  7. To validate that the JVM Diagnostics Manager is communicating with the JVM Diagnostics Agent, click the JVM Diagnostics link on the Middleware page. Click on a target in the left hand panel, for example __Farm01_medrec__medrec. In the right hand panel, click Pool > Threads > Real-Time Analysis. The following screen is displayed:

    Figure 12-3 JVM Diagnostics All Threads Page

    Surrounding text describes Figure 12-3 .

    On this page, you can view the threads for each connected JVM.

  8. Restart the ADP Manager and the WebLogic Server targets.

  9. Click Application Dependency and Performance link on the Middleware page. Click the Monitoring tab on the right panel and select the CAMM option from the menu. The following screen is displayed.

    Figure 12-4 Agent Information Page

    Agent Information Page

    You can validate that the ADP Agent has discovered the Java EE target by checking the Agent Status on this page.

  10. Click the Request Monitoring link on the Middleware page. Click the Target Setup tab. A list of ADP Managers available for data collection is displayed as shown in the following screen.

    Figure 12-5 Target Setup Page

    Target Setup Page

    On this page, do the following:

    • Select an ADP Manager from the list.

    • Select the Management Agent from the drop down list.

    • Enter the Key Store and Trust Store Password. The default password is acseramanager. For security reasons, it is recommended that you change this password. For more details, see the Getting Started with SOA Management Pack Plus Guide.

    • Click Create Target. The ADP Manager is now registered as a target, and starts collecting data for Request Monitoring.

  11. Request Monitoring setup is now complete. To view and configure requests, grant the Request Monitoring Administrator privilege which provides full access to the Request Monitoring pages and allows the user to modify the request configuration settings. For richer Request Monitoring functionality, we recommend that you grant the following privileges:

    • View any Target privilege which allows user to drill down to the Target Home page and to view the target metrics in the Topology view.

    • JVM Diagnostics privilege which allows user to drill down to the JVM Diagnostics pages from the Topology page.

  12. After the appropriate privileges have been granted, click the Request Monitoring link on the Middleware page and then click the Request Definitions tab. The following screen is displayed.

    Figure 12-6 Request Definitions Page

    Request Definitions Page

You can now view requests and their performance, group requests, and set request response time violation thresholds (Critical and Warning).

PK9S( ((PKOXEOEBPS/part2.htmF Advanced Topics

Part II

Advanced Topics

This section of the guide discusses the application of Enterprise Manager functionality to monitor and maintain your entire ecosystem.

Part II contains the following chapters:

PK}KFPKOXE OEBPS/toc.ncx Oracle® Enterprise Manager Administrator's Guide, 11g Release 1 (11.1.0.1) Cover Title and Copyright Information Contents List of Examples List of Figures List of Tables Preface Part I Basic Administration 1 Monitoring 2 Enterprise Manager Security 3 Notifications 4 User-Defined Metrics 5 Group Management 6 Job System 7 Starting and Stopping Enterprise Manager Components 8 Information Publisher 9 Sizing Your Enterprise Manager Deployment 10 Maintaining and Troubleshooting the Management Repository Part II Advanced Topics 11 Setting Up the JVM Diagnostics Manager 12 Setting Up Request Monitoring 13 Managing Compliance 14 Configuring Services 15 Extending Enterprise Manager Part III Enterprise Manager High Availability 16 High Availability Solutions 17 High Availability: Single Resource Configurations 18 High Availability: Multiple Resource Configurations 19 Management Agent and Management Services Index Copyright PKaZ9PKOXEOEBPS/content.opf5[ Oracle® Enterprise Manager Administrator's Guide, 11g Release 1 (11.1.0.1) en-US E16790-04 Oracle Corporation Oracle Corporation Oracle® Enterprise Manager Administrator's Guide, 11g Release 1 (11.1.0.1) 2013-03-12T04:16:26Z Oracle® Enterprise Manager Administrator's Guide, 11g Release 1 (11.1.0.1) PK55PKOXE OEBPS/lot.htm] List of Tables

List of Tables

PKGMPKOXE OEBPS/lof.htmD List of Figures

List of Figures

PK IDPKOXEOEBPS/index.htm Index

Index

A  B  C  D  E  F  G  H  I  J  L  M  N  O  P  R  S  T  U  V  W 

A

accessing
compliance management pages, 13.2.1
adaptive alert thresholds, purpose of, 1.2.1.1.2
ADDM, 1.1
purpose of, 1.1
Agent Registration Password, 2.4.3
changing, 2.4.7
Agent Upload Problems
default notification rule, 3.1.2.3
AGENT_HOME/network/admin, 2.4.9.4, 2.4.9.4
Agents Unreachable
default notification rule, 3.1.2.3
aggregration and purging policies
See data retention policies
Alert Details page, picture of, 1.5
alerts
automated responses, 1.2.4
corrective actions, 1.2.4
notification methods, 1.2.3
notifications for, 1.2.3
purpose of, 1.2.2
server-generated, 1.1
Warning Alerts page, 1.5
analyzing
job activity, 6.4
Application Performance Management, 2.8.3
Application Server Availability and Critical/Warning States
default notification rule, 3.1.2.3
Application Server Control
starting and stopping on Windows systems, 7.2.2
archive logging
for Management Repository database, 10.1
asynchronous I/O, 9.2.4.6
automated
responses to alerts, 1.2.4
Availability History Report, picture of, 5.3, 8.2

B

Backup, 9.3.1
baseline normalized views, 1.2.1.1.2
baselines, 9.2.2
adaptive thresholds, 1.2.1.1.2
for metrics, 1.2.1.1.2
normalized views, 1.2.1.1.2
BEA WebLogic
Oracle ecosystem and, 1.2.1
Beacons, 9.2.4.7
introduction, 1.2
monitoring Web Applications over HTTPS, 2.8.3
benefits of
extending Enterprise Manager, 15.1
Information Publisher, 8.1
blackouts
command-line interface and, 1.2.5
controlling with emctl, 7.7.4
examples, 7.7.4
functionality of, 1.2.5
retroactive, 1.2.5
buffer cache, 9.2.4.3

C

capacity
predicting, 9.2
Certificate dialog box
Internet Explorer, 2.8.2.1, 2.8.3
Checkpoint Firewall, Oracle ecosystem and, 1.2.1
Command Line Interface (EMCLI)
blackouts and, 1.2.5
Common Configurations
overview, 17.1
common configurations
deploying a remote management repository, 18.1
deploying Grid Control on a single host, 17.2
firewalls and other security considerations, 17.1
high availability configurations, 18.3
managing multiple hosts, 18.1
using multiple Management Services, 18.2
Compare Monitoring Template feature, 1.3
Compare Targets, picture of, 1.5
compliance
management pages, accessing, 13.2.1
violations, investigating, 13.2.2
compliance evaluation
scheduling, 13.3.1
setting up, 13.3
configuring
blackouts, functionality of, 1.2.5
monitoring templates, 1.3
Configuring Services, 14
Availability, 14.3, 14.4.1
Beacons, 14.3
Key Beacons, 14.3
Local Beacon, 14.3
Service Test-Based, 14.3, 14.4.1
System-Based, 14.3, 14.4.1
Command Line Interface, 14.13
Create, 14.3, 14.9
End-User Performance Monitoring, 14.1, 14.8
Access Log Format, 14.8.2.1
chronos_setup.sh, 14.8.2.3.1, 14.8.2.3.3
EUM, 14.8.1
Manage Web Server Data Collection, 14.8.1, 14.8.2.2
Set URLs, 14.8.1
Standalone Web Cache, 14.8.2.4
Unprocessed Samples, 14.8.6
URL Pattern, 14.8.2.1
Web Cache Log Format, 14.8.2.2
Web Cache Manager, 14.8.2.2
Interactive Transaction Tracing, 14.1
J2EE Server Activity, 14.1
Metrics
Performance, 14.3
Usage, 14.4.3
Usage Metrics, 14.3
Monitoring Settings, 14.6
Beacon Overrides, 14.6
Collection Settings, 14.6
Data Granularity, 14.6
Frequency, 14.6
Monitoring Templates, 14.11
Beacons, 14.11.1
Service Tests, 14.11.1
Service Tests and Beacons, 14.11.1
Variables, 14.11.1
Oracle Application Server 10g (9.0.4), 14.1
Performance, 14.3
Performance Metrics, 14.4.2
Aggregation Function, 14.4.2
Recording Transactions, 14.1, 14.5, 14.9.1
Request Performance, 14.1, 14.1, 14.10
Correlate Requests, 14.1
Database Connections, 14.10.2
Enterprise Java Beans (EJBs), 14.10.2
JDeveloper, 14.10.4
Manage OC4J Data Collection, 14.10.2
OC4J Cluster, 14.10
OC4J Instances, 14.10
OC4J Tracing, 14.10.3
Oracle Application Server 10g (9.0.4), 14.1
Oracle User Interface XML (UIX), 14.10.4
Service Tests and Beacons, 14.10.2
Tracing Properties, 14.10.2
Root Cause Analysis, 14.1, 14.4.1, 14.4.6
Component Tests, 14.4.6
Topology, 14.4.6
Topology Viewer, 14.1
Service Level Rules, 14.12
Actual Service Level, 14.12
Availability, 14.12
Business Hours, 14.12
Expected Service Level, 14.12
Information Publisher, 14.12.2
Performance Criteria, 14.12
Services Dashboard, 14.12.2
Service Tests and Beacons, 14.4.5
Configuring Dedicated Beacons, 14.4.5.1
SSL Certificate, 14.4.5.1
Tests, 14.4.5
Web Proxy, 14.4.5.1
Service-Test Based Availability
Key Service Tests, 14.4.1
System, 14.2
Key Components, 14.3
System-Based Availability
Key Components, 14.4.1
Test Performance, 14.1
Thresholds
Critical, 14.3
Warning, 14.3
Time Zone, 14.3
Types
Aggregate Service, 14.7
Types of Service
Generic Service, 14.3
Types of Services
Aggregate Service, 14.3
OCS Service, 14.3
Web Application, 14.3
connect descriptor
using to identify the Management Repository database, 10.4.1, 10.4.2.2
corrective actions
alerts and, 1.2.4
policies, 13.3.4.1
privileges required for, 1.2.4
creating
custom reports, 8.3.1
report definitions, 8.2
user-defined metric, 1.4
creating a monitoring script, 4.2.1
Critical URL Monitoring, as substitute for Management Agent, 1.2
custom reports, 8.3
customizing
notifications, 1.2.3.1
policies, 13.3.4

D

dashboard
groups, 5.2.5
Data Guard
configuring Enterprise Manager availability, 10.1
data retention policies
for Application Performance Management data, 10.2.1
for other Management data, 10.2.2
modifying default, 10.2.3
of the Management Repository, 10.2
when targets are deleted, 10.2.4
Database Availability and Critical/Warning States
default notification rule, 3.1.2.3
Database Control
starting on UNIX, 7.4.1
stopping on UNIX, 7.4.2
DBSNMP database user, 7.7.2
setting the password for, 7.7.2
deleting targets
data retention policies when, 10.2.4
diagnosing
policy violations, 13.2.2
Disaster Recovery, 9.3.3
disk mirroring and stripping
Management Repository guideline, 10.1
disk space management
controlling Management Agent disk space, 19.1.4
documentation, getting from OTN, 1.5
dropping the Management Repository, 10.4

E

E2E monitoring, 9.2.4.7
em_message, 4.2.1.2
em_result, 4.2.1.2
EMADM0400, 2
E-mail Customization, 3.1.4.1
e-mail notifications
upper limits, 3.1.2.1
e-mails, formats of, 1.2.3.1
emctl
controlling blackouts, 7.7.4
listing targets on a managed host, 7.7.3
security commands, 2.4.3
setting monitoring credentials, 7.7.2.2
starting, stopping, and checking the Management Service, 7.2
emctl config agent credentials, 7.7.2.2
emctl config agent listtargets, 7.7.3, 7.7.3
emctl config oms
sample output, 2.6.1
emctl istop, 7.1.2
emctl reload, 7.7.1
emctl secure agent, 2.4.4
sample output, 2.4.4
emctl secure oms, 2.4.3
sample output, 2.4.3
emctl secure setpwd, 2.4.7.2
emctl secure unlock, 2.4.6
emctl start agent, 7.1.1
emctl start blackout, 7.7.4, 7.7.4
emctl start dbconsole, 7.4.1
emctl start oms, 7.2.1.1
emctl status agent, 7.1.1, 7.1.1
emctl status blackout, 7.7.4
emctl status oms, 7.2.1.1
emctl stop agent, 7.1.1
emctl stop blackout, 7.7.4
emctl stop dbconsole, 7.4.2
emctl stop oms, 7.2.1.1
emctl upload, 7.7.1
EMD_URL
property in the emd.properties file, 19.1.3
emd.properties, 19.1.1, 19.1.3, 19.1.4
EMD_URL, 19.1.3
emdWalletSrcUrl, 19.1.1
REPOSITORY_URL, 17.2, 18.1, 19.1.1
UploadMaxBytesXML, 19.1.4
UploadMaxDiskUsedPct, 19.1.4
emdRepPort
property in the emoms.properties file, 19.2.1.1
emdRepPwd
property in the emoms.properties file, 19.2.1.1
emdRepServer
property in the emoms.properties file, 19.2.1.1
emdRepSID
property in the emoms.properties file, 19.2.1.1
emdRepUser
property in the emoms.properties file, 19.2.1.1
emdWalletSrcUrl
property in emd.properties, 19.1.1
em.notification.emails_per_minute
property in emoms.properties, 3.1.1
em.notification.os_cmd_timeout
property in emoms.properties, 3.2.1.1
emoms.properties, 10.3
emdRepPort, 19.2.1.1
emdRepPwd, 19.2.1.1
emdRepServer, 19.2.1.1
emdRepSID, 19.2.1.1
emdRepUser, 19.2.1.1
em.notification.emails_per_connection, 3.1.1
property in emoms.properties, 3.1.1
em.notification.emails_per_minute, 3.1.1
em.notification.os_cmd_timeout, 3.2.1.1
oracle.net.crypto_checksum_client, 2.4.9.2
oracle.net.crypto_checksum_types_client, 2.4.9.2
oracle.net.encryption_client, 2.4.9.2
oracle.net.encryption_types_client, 2.4.9.2
oracle.sysman.eml.mntr.emdRepPwd, 10.3
oracle.sysman.eml.mntr.emdRepPwdEncrypted, 10.3
oracle.sysman.emRep.dbConn.enableEncryption, 2.4.9.2
oracle.sysman.emSDK.sec.DirectoryAuthenticationType, 2.2.3
emwd watchdog script
in the AGENT_HOME/bin directory, 19.1.5
End-User Performance Monitoring
Web Server
Apache HTTP Server 2.0, 14.1, 14.1, 14.8, 14.8.1
Oracle Application Server Web Cache, 14.1, 14.1, 14.8.2
Oracle HTTP Server, 14.8, 14.8.1
Oracle HTTP Server Based on Apache 2.0, 14.1, 14.1, 14.8
OracleAS Web Cache, 14.8, 14.8
Enterprise Manager
blackouts, functionality of, 1.2.5
monitoring templates, 1.3
Enterprise Manager Console, picture of, 1.5
Enterprise Manager Framework Security
about, 2.4.1
configuring, 2.4
enabling for Management Repository, 2.4.9
enabling for multiple Management Services, 2.4.5
enabling for the Management Agent, 2.4.4
overview of steps required, 2.4.2
restricting HTTP access, 2.4.6
types of secure connections, 2.4.1
Extended Network, as substitute for Management Agent, 1.2
extending Enterprise Manager
benefits of, 15.1

F

figures
Availability History Report, 5.3, 8.2
Enterprise Manager Console, 1.5
Enterprise Manager Grid Control Console, 1.5
Group Administration Page, 5.2.3
Group Charts Page, 5.2.2
Group Dashboard, 5.2.5
Group Home Page, 5.2.1
Group Members Page, 5.2.4
Policy Trend Overview Page, 5.2.1
Summary of Target Jobs, 6.4
Warning Alert Metric Details, 1.5
Warning Alerts Page, 1.5

G

generating HTML reports, 8.2
Grid Control
architecture overview, 9.1
components, 9.1
console home page, 1.5
deploying on a single host, 17.2
sizing, 9.2, 9.2
starting, 7.6.1
starting all components of, 7.6.1
stopping, 7.6.2
stopping all components of, 7.6.2
Group Administration page, picture of, 5.2.3
Group Charts page, picture of, 5.2.2
Group Home page, picture of, 5.2.1
Group Members page, picture of, 5.2.4
groups
administrative tasks, 5.2.3
central monitoring location, 5.2.1
dashboard, 5.2.5
description and purpose, 5.1
management features, 5.2
member targets, 5.2.4
monitoring collective performance, 5.2.2
policy trends, 5.2.1
redundancy, 5.4
guidelines
for deploying the Management Repository, 10.1

H

Host Availability and Critical/Warning States
default notification rule, 3.1.2.3
HTTP access
restricting, 2.4.6
HTTP Server Availability and Critical/Warning States
default notification rule, 3.1.2.3
HTTPS, 2.4.1
Hyper-Threading, 9.2.4.1

I

IBM WebSphere
Oracle ecosystem and, 1.2.1
Information Publisher
Create Like function, 8.3
generating HTML reports, 8.2
overview of, 8
predefined report definitions, 8.2
predefined reports, 5.3
report definitions, 8.2
report elements, 8.3.3
reporting framework, 8
sharing reports, 8.5
viewing reports, 8.5
initialization parameter
adjusting when using multiple Management Services, 18.2
Installing a New Grid Control
installation type, 17.2
Internet Explorer
Certificate dialog box, 2.8.2.1, 2.8.3
security alert dialog box, 2.8.2.1
Security Information dialog box, 2.8.2.3
I/O Channels
monitoring, 9.2.4.6
istop
emctl command, 7.1.2

J

javax.net.ssl.SSLException
SSL handshake failed, 2.8.3
Job Activity page, 6.1
jobs
analyzing job activity, 6.4
definition of, 6.1
Job Activity page, 6.1
job executions, 6.1.1
job runs, 6.1.1
multitask, 6.3.4.1
notification rules for e-mail, 6.3.2
operations on runs and executions, 6.1.2
predefined tasks, 6.2.1
privileges for sharing job responsibilities, 6.2.3
purpose of, 6.1

L

Listener Availability
default notification rule, 3.1.2.3
load balancer switches
BIG-IP, Oracle ecosystem and, 1.2.1
Loader, 9.2.4.2
loader threads, 9.2.4.2
LVM (Logical Volume Manager), 10.1

M

Management Agent, 9.1, 9.2.1.1, 9.2.2
additional Management Agent commands, 7.7
checking the status on UNIX, 7.1.1
checking the status on Windows, 7.1.3
configuring trust points, 19.1.7
Critical URL Monitoring as substitute, 1.2
Extended Network as substitute, 1.2
purpose of, 1.2
reinstalling, 9.3.3.3
starting and stopping on UNIX, 7.1.1
starting and stopping on Windows, 7.1.2
Management Information Base (MIB), 3.9
definition, 3.9.1
MIB variable descriptions, 3.9.2
Management Plug-ins
extending Enterprise Manager with, 15
Management Repository
introduction of, 1.2.1
See Oracle Management Repository
Management Server, 9.2.4.1
Management Servers
adding, 9.2.4.5
Management Service, 9.1, 9.2.1.1
starting and stopping on Windows systems, 7.2.2
using a server load balancer, 18.3.4.1
managing
compliance, 13.2
groups, 5.2
policies, 13.2
master agent
Oracle Peer SNMP Master Agent service, 7.1.2
metric
baselines, 1.2.1.1.2
snapshots, 1.2.1.1.1
threshold value and alerts, 1.2.2
thresholds, 1.2.1.1
metric baselines
adaptive thresholds, 1.2.1.1.2
normalized views, 1.2.1.1.2
metrics
creating user-defined, 1.4
snapshots, 1.2.1.1.1
threshold values, 1.2.1
thresholds, 1.2.1.1
user-defined, creating, 1.4
user-defined, types of, 1.4
warning alert details page, 1.5
MGMT_ADMIN.DISABLE_METRIC_DELETION, 10.2.4
MGMT_ADMIN.ENABLE_METRIC_DELETION, 10.2.4
MGMT_METRICS_1DAY table, 10.2.3
MGMT_METRICS_1HOUR table, 10.2.3
MGMT_METRICS_RAW table, 10.2.3
MGMT_PARAMETERS table, 10.2.3
MGMT_RT_datatype_1DAY table, 10.2.3
MGMT_RT_datatype_1HOUR table, 10.2.3
MGMT_RT_datatype_DIST_1DAY table, 10.2.3
MGMT_RT_datatype_DIST_1HOUR table, 10.2.3
MGMT_RT_METRICS_RAW table, 10.2.3
MIB
See Management Information Base (MIB)
monitoring
alerts as they occur, 5.2.5
basics of, 1.2
information, accessing, 1.5
systems, 1
templates
function of, 1.3
monitoring credentials
defined, 7.7.2
example of setting, 7.7.2.2
setting, 7.7.2
setting in Grid Control, 7.7.2.1
setting with emctl, 7.7.2.2
monitoring script creation, 4.2.1
monitoring template, comparing, 1.3
monitoring templates, 4.4.3
multitask jobs, 6.3.4.1

N

NetApp Filers
Oracle ecosystem and, 1.2.1
Netscape Navigator
New Site Certificate dialog box, 2.8.2.2
network/admin, 2.4.9.1, 2.4.9.3, 2.4.9.4, 2.4.9.4
New Site Certificate dialog box
Netscape Navigator, 2.8.2.2
Notification Methods, 3.2
notification methods
based on a PL/SQL Procedure, 3.2.1.2
based on an SNMP trap, 3.2.1.3
based on operating system commands, 3.2.1.1
notification rules
definition, 3.1.2.3
out-of-box, 3.1.2.3
out-of-the-box notification rules, 3.1.2.1
subscribing to, 3.1.2.3
notification schedules, 3.1.2.1
notification system
e-mail errors, 3.10.4
notification system errors, 3.10.2
notification system, trace messages, 3.10.3
notifications
alerts, 1.2.3
assigning methods to rules, 3.6
assigning rules to methods, 3.7
customizing, 1.2.3.1
defining multiple mail servers, 3.1.1
for jobs, 6.3.2
long e-mail notifications, 3.1.2.1
mail server settings, 3.1.1
mail server settings in emoms.properties, 3.1.1
management information base (MIB), 3.9
methods, 1.2.3
notification method, 1.2.3
notification schedules, 3.1.2.1
sample Operating System command script, 3.2.1.1
setting up, 3.1
short email notifications, 3.1.2.1
using custom notfication methods, 3.2.1
Notifiction Rules
Custom, 3.1.2.3

O

OC4J Availability and Critical/Warning States
default notification rule, 3.1.2.3
operating system
user-defined metrics, 1.4
Operating System command
sample notification method for, 3.2.1.1
sample script, 3.2.1.1
Operating System scripts, 3.2, 3.2
while creating notification methods, 3.2.1
ORA-12645
Parameter does not exist, 2.4.9.1
Oracle
ecosystem, 1.2.1
Technology Network (OTN), 1.5
Oracle Advanced Security, 2.4.1, 2.4.9.1
enabling for Management Repository, 2.4.9.3
enabling for the Management Agent, 2.4.9.4
Oracle Application Server Web Cache
Web Cache Manager, 14.8.2.2
Oracle Enterprise Manager
components, 9.2.1.1
rollup process, 9.2.4.3
Oracle Enterprise Manager 10g Grid Control
See Grid Control
Oracle Enterprise Manger
tuning, 9.2.4
Oracle Internet Directory, 2.2.2.3.1
Oracle Management Agent
changing the port, 19.1.3
controlling disk space used by, 19.1.4
enabling security for, 2.4.4, 2.4.9.4
reconfiguring to use a new Management Service, 19.1.1
starting and stopping, 7.1
Watchdog process, 19.1.5
Oracle Management Repository, 9.1
changing the Management Repository password, 19.2.1.2
configuring for high availability, 18.3.1
data retention policies, 10.2
deploying on a remote host, 18.1
deployment guidelines, 10.1
dropping, 10.4
enabling Oracle Advanced Security, 2.4.9.3
enabling security for, 2.4.9
identifying with a connect descriptor, 10.4.1, 10.4.2.2
recreating, 10.4, 10.4.2
reloading data, 7.7.1
restoring, 9.3.3.1
starting the Management Repository database, 7.6.1
troubleshooting, 10.5.3
uploading data, 7.7.1
Oracle Management Service, 9.3.3.2
adjusting the PROCESSES initialization parameter, 18.2
configuring to use a new Repository, 19.2.1
enabling security for, 2.4.3
enabling security for multiple Management Services, 2.4.5
modifying monitoring credentials, 7.7.2
reconfiguring, 19.2
restoring, 9.3.3.2
starting, stopping, and checking, 7.2
using multiple management services, 18.2
Oracle Process Management and Notification (OPMN)
using to start and stop the Management Service, 7.2.2
Oracle Technology Network (OTN), 14.8.2.4.1
ORACLE_HOME/network/admin, 2.4.9.1, 2.4.9.3
oracle.net.crypto_checksum_client
property in emoms.properties, 2.4.9.2
oracle.net.crypto_checksum_types_client
property in emoms.properties, 2.4.9.2
oracle.net.encryption_client
property in emoms.properties, 2.4.9.2
oracle.net.encryption_types_client
property in emoms.properties, 2.4.9.2
oracle.sysman.eml.mntr.emdRepPwd
property in emoms.properties, 10.3
oracle.sysman.eml.mntr.emdRepPwdEncrypted
property in emoms.properties, 10.3
oracle.sysman.emRep.dbConn.enableEncryption
entry in emoms.properties, 2.4.9.2
oracle.sysman.emSDK.sec.DirectoryAuthenticationType
property in emoms.properties, 2.2.3
OS scripts
See Operating System scripts
OTN (Oracle Technology Network), 14.8.2.4.1
OTN and documentation, 1.5
out-of-box
monitoring, 1.2.1
policies, 13.3.3
reports, 8.2

P

password
changing the Management Repository password, 19.2.1.2
changing the SYSMAN password, 10.3
peer encapsulator service
SNMP, 7.1.2
Performance Metrics
Beacon Aggregation Function
Average, 14.4.2, 14.4.2
Maximum, 14.4.2
Minimum, 14.4.2, 14.4.2
Sum, 14.4.2, 14.4.2
System Aggregation Function
Maximum, 14.4.2
planning outage periods, blackouts, 1.2.5
PL/SQL procedures, 3.2
sample, 3.2.1.2
while creating a notification method, 3.2.1.2
while creating notfication methods, 3.2.1
policies
assessing security policies, 13.2.3
corrective actions, defining, 13.3.4.1
customizing, 13.3.4
investigating violations, 13.2.2
managing, 13.2
out-of-box, 13.3.3
reports on violations, 13.2.5
using templates for monitoring, 13.3.4.2
violations reports, 13.2.5
policy group
Oracle database, 13.4.1
Oracle Listener, 13.4.3
RAC, 13.4.2
Policy Groups reports, 13.2.5
policy management, 13.2
Policy Trend Overview page, picture of, 5.2.1
Policy Violations reports, 13.2.5
ports
changing the Management Agent port, 19.1.3
default port for the Management Agent upload URL, 17.2
predefined
job tasks, 6.2.1
privileges
for corrective actions, 1.2.4
for sharing job responsibilities, 6.2.3
PROCESSES, 18.2
ProcessManager
service used to control the Management Service on Windows systems, 7.2.2
Public Key Infrastructure (PKI), 2.4.1, 2.8.3
purging policies
See data retention policies

R

RAID-capable disk
Management Repository guideline, 10.1
Recovery, 9.3.1
redundancy groups, 5.4
Repeat Notifications, 3.1.1.1
Repeat Notifications for Rules, 3.1.1.1
RepManager script, 10.4.1, 10.4.2.1
reports
creating custom reports, 8.3.1
custom, 8.3
definitions, Information Publisher, 8.2
e-mailing, 8.4.3
generating HTML report, 8.2
Information Publisher, 8
out-of-box, Information Publisher, 8.2
policy violations, 13.2.5
predefined, 5.3
predefined definitions, 8.2
predefined report definitions, 8.2
report elements, 8.3.3
scheduling, 8.4, 8.4
sharing, 8.5, 8.5
storing and purging, 8.4.2
viewing, 8.5
Repository Operations Availability
default notification rule, 3.1.2.3
REPOSITORY_URL
property in emd.properties, 17.2, 18.1
property in the emd.properties file, 19.1.1
retroactive blackouts, 1.2.5
rollup process, 9.2.4.3
Root Cause Analysis
Mode
Automatic, 14.4.6
Manual, 14.4.6
root password
See also SYSMAN
when enabling security for the Management Service, 2.4.3

S

scheduled maintenance with blackouts, 1.2.5
scheduling
reports, 8.4
reports, flexibility, 8.4.1
script registration, UDM, 4.2.2
script results, returning, 4.2.1.2
security
about Enterprise Manager security, 2.1
overview of steps required to enable Enterprise Manager Framework Security, 2.4.2
policies and, 13.2.3
See also Enterprise Manager Framework Security
security alert dialog box
Internet Explorer, 2.8.2.1
Security At a Glance feature, policies and, 13.2.3
security certificate alerts
responding to, 2.8.2
security features
See Enterprise Manager Framework Security
Security Information dialog box
Internet Explorer, 2.8.2.3
self-monitoring
feature of the Management Agent, 19.1.5
Server Connection Hung
error while creating the repository, 10.5.2
Server Load Balancer, 2.4.8
server load balancer, 18.3.4.1
using with Management Services, 18.3.4.1
server-generated alerts, 1.1
Service Tests and Beacons
Tests
DNS, 14.4.5
FTP, 14.4.5
SOAP, 14.4.5
Web Transaction, 14.4.5
Services control panel
using to start and stop the Management Agent, 7.4.3
using to start the Management Service, 7.2.2
setting
metric threshold values, 1.2.1
sharing reports, 8.5, 8.5
snapshots of metrics, 1.2.1.1.1
SNMP
Oracle Peer SNMP Master Agent service, 7.1.2
Oracle SNMP Peer Encapsulator service, 7.1.2
SNMP traps, 3.2, 3.2.1, 3.2.1.3
sample, 3.2.1.3
SQL
user-defined metrics, 1.4
SQL UDM, character limit, 4.3.1.3
SQL UDM, long statements, 4.3.1.3
SQLNET.CRYPTO_SEED
entry in sqlnet.ora, 2.4.9.3, 2.4.9.4
SQLNET.ENCRYPTION_SERVER
entry in sqlnet.ora, 2.4.9.3
sqlnet.ora, 2.4.9.1, 2.4.9.1
SQLNET.CRYPTO_SEED, 2.4.9.3, 2.4.9.4
SQLNET.ENCRYPTION_SERVER, 2.4.9.3
state directory
in the Management Agent home, 19.1.1
Status Codes, Corrective Actions, 3.3.2, 3.4.1
support of third-party components, 1.1
SYSMAN
changing the SYSMAN password, 10.3
checking for existence of, 10.5.3
entering SYSMAN password when enabling security, 2.4.3
System Dashboard, picture of, 5.2.5
system errors, notification, 3.10.2
systems
monitoring, 1

T

target
definition of, 1.2
target monitoring credentials
defined, 7.7.2
example of setting, 7.7.2.2
setting, 7.7.2
setting in Grid Control, 7.7.2.1
targets
listing targets on a managed host, 7.7.3
templates, policies and, 13.3.4.2
third-party
components, support of, 1.1
thresholds, 9.2.2, 9.2.2
adaptive, 1.2.1.1.2
adaptive alert, 1.2.1.1.2
alerts and, 1.2.2
definition of, 1.2.1
for metrics, 1.2.1.1, 1.2.1.1
troubleshooting
general techniques while creating the Management Repository, 10.5.3
while creating the Management Repository, 10.5
Troubleshooting Service Tests, 14.14
Forms Transactions, 14.14.1
troubleshooting, notifications, 3.10
trust points
Management Agent Configuration, 19.1.7

U

upload directory
in the Management Agent home, 19.1.1, 19.1.4
UploadMaxBytesXML
property in the emd.properties file, 19.1.4
UploadMaxDiskUsedPct
property in the emd.properties file, 19.1.4
Usage Metrics
Aggregation Function
Average, 14.4.3, 14.4.4
Maximum, 14.4.3, 14.4.4
Minimum, 14.4.3, 14.4.4
Sum, 14.4.3, 14.4.4
user-defined
metrics, 1.4
metrics, creating, 1.4
User-defined metric page
Command Line, 4.2.2
Comparison Operator, 4.2.2, 4.3
Consecutive Occurrences Preceding Notification, 4.2.2, 4.3
Critical, 4.2.2, 4.3
Environment, 4.2.2
Metric Name, 4.2.2, 4.3
Metric Type, 4.2.2, 4.3
Operating System User Name and Password, 4.2.2
Response Action, 4.2.2, 4.3
Warning, 4.2.2, 4.3
User-defined metric page, Central Console, 4.2.2, 4.3
user-defined metric, example, 4.2.3
User-defined metrics, 4
user-defined metrics, 4.4.3

V

viewing
reports, 8.5

W

Warning Alert Metric Details page, picture of, 1.5
Warning Alerts page, picture of, 1.5
watchdog process
for the Management Agent, 19.1.5
Web Application
Source
Step, 14.4.2
Step Group, 14.4.2
Transaction, 14.4.2
Web Applications
monitoring over HTTPS, 2.8.3
Web Cache Availability and Critical/Warning States
default notification rule, 3.1.2.3
PKW+&8 8PKOXEOEBPS/sizing.htm Sizing Your Enterprise Manager Deployment

9 Sizing Your Enterprise Manager Deployment

Oracle Enterprise Manager 10g Grid Control has the ability to scale for hundreds of users and thousands of systems and services on a single Enterprise Manager implementation.

This chapter describes techniques for achieving optimal performance using the Oracle Enterprise Manager application. It can also help you with capacity planning, sizing and maximizing Enterprise Manager performance in a large scale environment. By maintaining routine housekeeping and monitoring performance regularly, you insure that you will have the required data to make accurate forecasts of future sizing requirements. Receiving good baseline values for the Enterprise Manager Grid Control vital signs and setting reasonable warning and critical thresholds on baselines allows Enterprise Manager to monitor itself for you.

This chapter also provides practical approaches to backup, recovery, and disaster recovery topics while addressing different strategies when practical for each tier of Enterprise Manager.

This chapter contains the following sections:

Oracle Enterprise Manager Grid Control Architecture Overview

The architecture for Oracle Enterprise Manager 10g Grid Control exemplifies two key concepts in application performance tuning: distribution and parallelization of processing. Each component of Grid Control can be configured to apply both these concepts.

The components of Enterprise Manager Grid Control include:

  • The Management Agent - A process that is deployed on each monitored host and that is responsible for monitoring all services and components on the host. The Management Agent is also responsible for communicating that information to the middle-tier Management Service and for managing and maintaining the system and its services.

  • The Management Service - A J2EE Web application that renders the user interface for the Grid Control Console, works with all Management Agents to process monitoring and jobs information, and uses the Management Repository as its data store.

  • The Management Repository - The schema is an Oracle Database that contains all available information about administrators, services, and applications managed within Enterprise Manager.

Figure 9-1 Overview of Enterprise Manager Architecture Components

Illustration of Enterprise Manager architecture components

For more information about the Grid Control architecture, see the Oracle Enterprise Manager 10g documentation:

The Oracle Enterprise Manager 10g documentation is available at the following location on the Oracle Technology Network (OTN):

http://otn.oracle.com/documentation/oem.html

Enterprise Manager Grid Control Sizing and Performance Methodology

An accurate predictor of capacity at scale is the actual metric trend information from each individual Enterprise Manager Grid Control deployment. This information, combined with an established, rough, starting host system size and iterative tuning and maintenance, produces the most effective means of predicting capacity for your Enterprise Manager Grid Control deployment. It also assists in keeping your deployment performing at an optimal level.

Here are the steps to follow to enact the Enterprise Manager Grid Control sizing methodology:

  1. If you have not already installed Enterprise Manager Grid Control 10g, choose a rough starting host configuration as listed in Table 9-1.

  2. Periodically evaluate your site's vital signs (detailed later).

  3. Eliminate bottlenecks using routine DBA/Enterprise Manager administration housekeeping.

  4. Eliminate bottlenecks using tuning.

  5. Extrapolate linearly into the future to plan for future sizing requirements.

Step one need only be done once for a given deployment. Steps two, three, and four must be done, regardless of whether you plan to grow your Enterprise Manager Grid Control site, for the life of the deployment on a regular basis. These steps are essential to an efficient Enterprise Manager Grid Control site regardless of its size or workload. You must complete steps two, three, and four before you continue on to step five. This is critical. Step five is only required if you intend to grow the deployment size in terms of monitored targets. However, evaluating these trends regularly can be helpful in evaluating any other changes to the deployment.

Step 1: Choosing a Starting Platform Grid Control Deployment

If you have not yet installed Enterprise Manager Grid Control on an initial platform, this step helps you choose a rough approximation based on experiences with real world Enterprise Manager Grid Control deployments. If you have already installed Enterprise Manager Grid Control, proceed to Step 2. Three typical deployment sizes are defined: small, medium, and large. The number and type of systems (or targets) it monitors largely defines the size of an Enterprise Manager Grid Control deployment.

Table 9-1 Management Server

Deployment SizeHostsCPUs/HostsMemory/Host (GB)

Small (100 monitored targets)

1

1 (3 GHz)

4

Medium (1,000 monitored targets)

1

2 (3 GHz)

Greater than or equal to 4

Large (10,000 monitored targets)

2

2 (3 GHz) 2

Greater than or equal to 6


In any OMS host box, OPMN processes, Admin Server Process, Node Manager processes, and/or DB processes will be running, so the minimum memory requirement is 4 GB per OMS host.

Table 9-2 Management Repository

Deployment SizeHostsCPUs/HostMemory/Host (GB)

Small

Shares host with Management Server

Shares host with Management Server

Shares host with Management Server

Medium

1

2

4

Large

2

4

6


Table 9-3 Total Management Repository Storage

Deployment SizeMinimum Tablespace Sizes*
SYSTEM**MGMT_TABLESPACEMGMT_ECM_DEPOT_TSMGMT_AD4J_TSTEMP

*These are strictly minimum values and are intended as rough guidelines only. The actual size of the MGMT_TABLESPACE could vary widely from deployment to deployment due to variations in target type distribution, user customization, and several other factors. These tablespaces are defined with AUTOEXTEND set to ON by default to help mitigate space constraint issues. On raw file systems Oracle recommends using more than the minimum size to help prevent space constraint issues.
**The SYSTEM and TEMP tablespace sizes are minimums for Enterprise Manager only repositories. If Enterprise Manager is sharing the repository database with other application(s), these minimums may be too low.
Note: You cannot monitor tablespaces through the use of alerts with auto extended files in version 10g of Enterprise Manager. You can either set up TABLESPACE FULL alerts generate if you want to have greater control over the management of your tablespaces, or you can allow Oracle to grow your database and not alert you through the AUTOEXTEND feature. Therefore to exercise greater control of the TABLESPACE FULL alerts, you can turn off autoextend.

Small

600 MB

50 GB

1 GB

100 MB

10 GB

Medium

600 MB

200 GB

4 GB

200 MB

20 GB

Large

600 MB

300 GB

Greater than 4 GB

400 MB

40 GB


The previous tables show the estimated minimum hardware requirements for each deployment size. Management Servers running on more than one host, as portrayed in the large deployment above, will divide work amongst themselves.

Deploying multiple Management Servers also provides basic fail-over capabilities, with the remaining servers continuing to operate in the event of the failure of one. Use of a Server Load Balancer, or SLB, provides transparent failover for Enterprise Manager UI clients in the event of a Management Server host failure, and it also balances the request load between the available Management Servers. SLBs are host machines dedicated for load balancing purposes. SLBs can be clustered to provide fail-over capability.

Using multiple hosts for the Management Repository assumes the use of Oracle Real Application Clusters (RAC). Doing so allows the same Oracle database to be accessible on more than one host system. Beyond the storage required for the Management Server, Management Repository storage may also be required. Management Server storage is less impacted by the number of management targets. The numbers suggested in the Enterprise Manager Grid Control documentation should be sufficient in this regard.

Network Topology Considerations

A critical consideration when deploying Enterprise Manager Grid Control is network performance between tiers. Enterprise Manager Grid Control ensures tolerance of network glitches, failures, and outages between application tiers through error tolerance and recovery. The Management Agent in particular is able to handle a less performant or reliable network link to the Management Service without severe impact to the performance of Enterprise Manager as a whole. The scope of the impact, as far as a single Management Agent's data being delayed due to network issues, is not likely to be noticed at the Enterprise Manager Grid Control system wide level.

The impact of slightly higher network latencies between the Management Service and Management Repository will be substantial, however. Implementations of Enterprise Manager Grid Control have experienced significant performance issues when the network link between the Management Service and Management Repository is not of sufficient quality. The following diagram that displays the Enterprise Manager components and their connecting network link performance requirements. These are minimum requirements based on larger real world Enterprise Manager Grid Control deployments and testing.

Figure 9-2 Network Links Related to Enterprise Manager Components

Surrounding text describes Figure 9-2 .

You can see in Figure 9-2 that the bandwidth and latency minimum requirements of network links between Enterprise Manager Grid Control components greatly impact the performance of the Enterprise Manager application.

Step 2: Periodically Evaluate the Vital Signs of Your Site

This is the most important step of the five. Without some degree of monitoring and understanding of trends or dramatic changes in the vital signs of your Enterprise Manager Grid Control site, you are placing site performance at serious risk. Every monitored target sends data to the Management Repository for loading and aggregation through its associated Management Agent. This adds up to a considerable volume of activity that requires the same level of management and maintenance as any other enterprise application.

Enterprise Manager has "vital signs" that reflect its health. These vital signs should be monitored for trends over time as well as against established baseline thresholds. You must establish realistic baselines for the vital signs when performance is acceptable. Once baselines are established, you can use built-in Oracle Enterprise Manager Grid Control functionality to set baseline warning and critical thresholds. This allows you to be notified automatically when something significant changes on your Enterprise Manager site. The following table is a point-in-time snapshot of the Enterprise Manager Grid Control vital signs for two sites:

ModuleMetricsEM Site 1EM Site 2
Site URL
emsite1.acme.comemsite2.acme.com




Target CountsDatabase Targets192 (45 not up)1218 (634 not up)

Host Targets833 (12 not up)1042 (236 not up)





Total Targets2580 (306 not up)12293 (6668 not up)




Loader StatisticsLoader Threads616

Total Rows/Hour1,692,0002,736,000

Rows/hour/load/thread282,000171,000

Rows/second/load thread475187

Percent of Hour Run1544




Rollup StatisticsRows per Second2,267417

Percent of Hour Run519




Job StatisticsJob Dispatchers24

Job Steps/second/dispatcher3210




Notification StatisticsNotifications per Second81

Percent of Hour Run113




Alert StatisticsAlerts per Hour5361,100




Management Service Host StatisticsAverage % CPU (Host 1)9 (emhost01)13 (emhost01)

Average % CPU (Host 2)6 (emhost02)17 (emhost02)

Average % CPU (Host 3)N/A38 (em6003)

Average % CPU (Host 4)N/A12 (em6004)

Number of CPUs per host2 X 2.8 (Xeon)4 X 2.4 (Xeon)

Memory per Host (GB)66




Management Repository Host StatisticsAverage % CPU (Host 1)12 (db01rac)32 (em6001rac)

Average % CPU (Host 2)


Average % CPU (Host 3)


Average % CPU (Host 4)


Number of CPUs per host


Buffer Cache Size (MB)


Memory per Host (GB)612

Total Management Repository Size (GB)5698

RAC Interconnect Traffic (MB/s)14

Management Server Traffic (MB/s)44

Total Management Repository I/O (MB/s)627




Enterprise Manager UI Page Response/SecHome Page36

All Host Page330+

All Database Page630+

Database Home Page22

Host Home Page22

The two Enterprise Manager sites are at the opposite ends of the scale for performance.

EM Site 1 is performing very well with high loader rows/sec/thread and high rollup rows/sec. It also has a very low percentage of hours run for the loader and the rollup. The CPU utilization on both the Management Server and Management Repository Server hosts are low. Most importantly, the UI Page Response times are excellent. To summarize, Site 1 is doing substantial work with minimal effort. This is how a well configured, tuned and maintained Oracle Enterprise Manager Grid Control site should look.

Conversely, EM Site 2 is having difficulty. The loader and rollup are working hard and not moving many rows. Worst of all are the UI page response times. There is clearly a bottleneck on Site 2, possibly more than one.

The following table outlines metric guidelines for the different modules based on tests that were run with the configurations outlined. It can serve as a reference point for you to extrapolate information and data based on the metrics and test environment used in the specified environment.

Table 9-4 Metric Guidelines for Modules Based On Test Environments

ModuleMetricsValueTest Environment

Loader Statistics

Loader Threads

10

OMS Details


# of OMS Hosts = 2
# of CPU Per Host = 4 Intel Xeon
Memory = 6 GB

Repository Details


# of Repository Nodes = 2
# of CPU per host = 4 Intel Xeon
Memory = 6 GB

EM Details


Shared Recv Directory = Yes
# of Agents = 867
# of Hosts = 867
Total Targets = 1803

The Metrics are collected for 5 hours after 2 OMSs were started and each agent had 50 MB of upload backlog files.

Total Rows/Hour

4,270,652

Rows/Hour/loaderthread

427,065

Rows/second/loaderthread

120

Rollup Statistics

Rows per second



Job Statistics

Job Dispatchers

1 x Number of OMSs


Job Steps/second/dispatcher



Notification Statistics

Notifications per second

16

OMS Details


# of OMS Hosts = 1
# of CPU Per Host = 4 Intel Xeon
Memory = 6 GB

Repository Details


# of Repository Nodes = 1
# of CPU per host = 4 Intel Xeon
Memory = 6 GB

EM Details


# of OMSs = 1
# of Repository Nodes = 1
# of Agents = 2474
# of Hosts = 2474
DB Total Targets = 8361

Alert Statistics

Alerts per hour

7200

OMS Details


# of OMS Hosts = 1
# of CPU Per Host = 4 Intel Xeon

Memory = 6 GB

Repository Details


# of Repository Nodes = 1
# of CPU per host = 4 Intel Xeon

Memory = 6 GB

EM Details


# of OMSs = 1
# of Repository Nodes = 1
# of Agents = 2474
# of Hosts = 2474

DB Total Targets = 8361

Management Service Host Statistics

Average % CPU (Host 1)

31%

OMS Details


# of OMS Hosts = 2
# of CPU Per Host = 4 Intel Xeon

Memory = 6 GB

Repository Details


# of Repository Nodes = 2
# of CPU per host = 4 Intel Xeon

Memory = 6 GB

EM Details

Shared Recv Directory = Yes


# of Agents = 867
# of Hosts = 867

Total Targets = 1803

The Metrics are collected for 5 hours after 2 OMSs were started and each agent had 50 MB of upload backlog files.

Average % CPU (Host 2)

34%

Number of CPUs per host

4 (Xeon)

Memory per Host (GB)

6

Management Repository Host Statistics

Average % CPU (Host 1)

32%

OMS Details


# of OMS Hosts = 2
# of CPU Per Host = 4 Intel Xeon

Memory = 6 GB

Repository Details


# of Repository Nodes = 2
# of CPU per host = 4 Intel Xeon

Memory = 6 GB

EM Details

Shared Recv Directory = Yes


# of Agents = 867
# of Hosts = 867

Total Targets = 1803

The Metrics are collected for 5 hours after 2 OMSs were started and each agent had 50 MB of upload backlog files.

Average % CPU (Host 2)

26%

Number of CPUs per host

4

SGA Target

2 GB

Memory per Host (GB)

6

Total Management Repository Size (GB)

94

RAC Interconnect Traffic (MB/s)

1

Management Server Traffic (MB/s)


Total Management Repository I/O (MB/s)


Enterprise Manager UI Page Response/Sec

Home Page

9.1 secs

OMS Details


# of OMS Hosts = 1
# of CPU Per Host = 4 Intel Xeon

Memory = 6 GB

Repository Details


# of Repository Nodes = 1
# of CPU per host = 4 Intel Xeon

Memory = 6 GB

EM Details


# of OMSs = 1
# of Repository Nodes = 1
# of Agents = 2474
# of Hosts = 2474

DB Total Targets = 8361

All Host Page

9.8 secs

All Database Page

5.7 secs

Database Home Page

1.7 secs

Host Home Page

< 1 sec


These vital signs are all available from within the Enterprise Manager interface. Most values can be found on the All Metrics page for each host, or the All Metrics page for Management Server. Keeping an eye on the trends over time for these vital signs, in addition to assigning thresholds for warning and critical alerts, allows you to maintain good performance and anticipate future resource needs. You should plan to monitor these vital signs as follows:

  • Take a baseline measurement of the vital sign values seen in the previous table when the Enterprise Manager Grid Control site is running well.

  • Set reasonable thresholds and notifications based on these baseline values so you can be notified automatically if they deviate substantially. This may require some iteration to fine-tune the thresholds for your site. Receiving too many notifications is not useful.

  • On a daily (or weekly at a minimum) basis, watch for trends in the 7-day graphs for these values. This will not only help you spot impending trouble, but it will also allow you to plan for future resource needs.

The next step provides some guidance of what to do when the vital sign values are not within established thresholds. Also, it explains how to maintain your site's performance through routine housekeeping.

Step 3: Use DBA and Enterprise Manager Tasks To Eliminate Bottlenecks Through Housekeeping

It is critical to note that routine housekeeping helps keep your Enterprise Manager Grid Control site running well. The following are lists of housekeeping tasks and the interval on which they should be done.

Online Weekly Tasks

Analyze the three major tables in the Management Repository: MGMT_METRICS_RAW, MGMT_METRICS_1HOUR, and MGMT_METRICS_1DAY. If your Management Repository is in an Oracle 10g database, then these tables are automatically analyzed weekly and you can skip this task. If your Management Repository is in an Oracle version 9 database, then you will need to ensure that the following commands are run weekly:

  • exec dbms_stats.gather_table_stats('SYSMAN', 'MGMT_METRICS_RAW', null, .2, false, 'for all indexed columns', null, 'global', true, null, null, null);

  • exec dbms_stats.gather_table_stats('SYSMAN', 'MGMT_METRICS_1HOUR', null, .2, false, 'for all indexed columns', null, 'global', true, null, null, null);

  • exec dbms_stats.gather_table_stats('SYSMAN', 'MGMT_METRICS_1DAY', null, .2, false, 'for all indexed columns', null, 'global', true, null, null, null);

  • exec dbms_stats.gather_table_stats('SYSMAN', 'MGMT_STRING_METRIC_HISTORY', null, .2, false, 'for all indexed columns', null, 'global', true, null, null, null);

Offline Monthly Tasks

Rebuild and defragment indexes and reorganize tables as required. You may not actually need to rebuild any indexes or tables on a monthly basis. All you should do monthly is evaluate the Management Repository for tables and indexes that have grown significantly and been purged back down to a fraction of their allocated size. Unnecessarily building tables and indexes causes the Management Repository to work harder than necessary to reallocate needed space. The tables that require reorganization are easily identifiable. These tables will be large in allocated size with a relatively small number of rows, or actual size. In a real Management Repository, you may see one table that is approximately 800MB in size but only contains 6 rows. If the table is this badly oversized, it requires reorganization. Tables can be reorganized and rebuilt using commands similar to the following example:

  • exec dbms_redefinition.start_redef_table('SYSMAN','MGMT_VIOLATIONS','temp_viol');

  • CREATE TABLE temp_viol AS

    SELECT * FROM mgmt_violations WHERE 1=2;

  • exec dbms_redefinition.start_redef_table

    ('SYSMAN','MGMT_VIOLATIONS','temp_viol');

  • DROP table temp_viol;

These commands rebuild the table and returns its physical structure to its clean initial state. The 800 MB table is an extreme case. Smaller disparities between actual size and row count may also indicate the need for reorganization. The Management Server(s) must be down when reorganizing a table. If you reorganize the table, the indexes must also be rebuilt. This helps make index range scans more efficient again. Indexes can be reorganized using a command similar to the following example:

  • ALTER INDEX SEVERITY_PRIMARY_KEY REBUILD;

There are a few tables (along with their indexes) that may require rebuilding more frequently than others based on the higher volume of inserts and deletes they typically see. These tables are:

  • MGMT_VIOLATIONS

  • MGMT_CURRENT_VIOLATION

  • MGMT_SYSTEM_ERROR_LOG

  • MGMT_METRIC_ERRORS

  • MGMT_CURRENT_METRIC_ERRORS

  • MGMT_JOB_OUTPUT

These Management Repository tables can be prone to uneven growth patterns and can have large areas of allocated space that are unused due to infrequent, large spikes in data volume.

Step 4: Eliminate Bottlenecks Through Tuning

The most common causes of performance bottlenecks in the Enterprise Manager Grid Control application are listed below (in order of most to least common):

  1. Housekeeping that is not being done (far and away the biggest source of performance problems)

  2. Hardware or software that is incorrectly configured

  3. Hardware resource exhaustion

When the vital signs are routinely outside of an established threshold, or are trending that way over time, you must address two areas. First, you must ensure that all previously listed housekeeping is up to date. Secondly, you must address resource utilization of the Enterprise Manager Grid Control application. The vital signs listed in the previous table reflect key points of resource utilization and throughput in Enterprise Manager Grid Control. The following sections cover some of the key vital signs along with possible options for dealing with vital signs that have crossed thresholds established from baseline values.

High CPU Utilization

When you are asked to evaluate a site for performance and notice high CPU utilization, there are a few common steps you should follow to determine what resources are being used and where.

  1. The Management Server is typically a very minimal consumer of CPU. High CPU utilization in the Enterprise Manager Grid Control almost always manifests as a symptom at the Management Repository.

  2. Use the Processes display on the Enterprise Manager Host home page to determine which processes are consuming the most CPU on any Management Service or Management Repository host that has crossed a CPU threshold.

  3. Once you have established that Enterprise Manager is consuming the most CPU, use Enterprise Manager to identify what activity is the highest CPU consumer. Typically this manifests itself on a Management Repository host where most of the Management Service's work is performed. It is very rare that the Management Service itself is the source of the bottleneck. Here are a few typical spots to investigate when the Management Repository appears to be using too many resources.

    1. Click on the CPU Used database resource listed on the Management Repository's Database Performance page to examine the SQL that is using the most CPU at the Management Repository.

    2. Check the Database Locks on the Management Repository's Database Performance page looking for any contention issues.

High CPU utilization is probably the most common symptom of any performance bottleneck. Typically, the Management Repository is the biggest consumer of CPU, which is where you should focus. A properly configured and maintained Management Repository host system that is not otherwise hardware resource constrained should average roughly 40 percent or less total CPU utilization. A Management Server host system should average roughly 20 percent or less total CPU utilization. These relatively low average values should allow sufficient headroom for spikes in activity. Allowing for activity spikes helps keep your page performance more consistent over time. If your Enterprise Manager Grid Control site interface pages happen to be responding well (approximately 3 seconds) while there is no significant (constant) loader backlog, and it is using more CPU than recommended, you may not have to address it unless you are concerned it is part of a larger upward trend.

The recommended path for tracking down the root cause of high Management Repository CPU utilization is captured under step 3.b above. This allows you to start at the Management Repository Performance page and work your way down to the SQL that is consuming the most CPU in its processing. This approach has been used very successfully on several real world sites.

If you are running Enterprise Manager on Intel based hosts, the Enterprise Manager Grid Control Management Service and Management Repository will both benefit from Hyper-Threading (HT) being enabled on the host or hosts on which they are deployed. HT is a function of certain late models of Intel processors, which allows the execution of some amount of CPU instructions in parallel. This gives the appearance of double the number of CPUs physically available on the system. Testing has proven that HT provides approximately 1.5 times the CPU processing power as the same system without HT enabled. This can significantly improve system performance. The Management Service and Management Repository both frequently have more than one process executing simultaneously, so they can benefit greatly from HT.

Loader Vital Signs

The vital signs for the loader indicate exactly how much data is continuously coming into the system from all the Enterprise Manager Agents. The most important items here are the percent of hour runs and rows/second/thread. The (Loader) % of hour run indicates whether the loader threads configured are able to keep pace with the incoming data volume. As this value approaches 100%, it becomes apparent that the loading process is failing to keep pace with the incoming data volume. The lower this value, the more efficiently your loader is running and the less resources it requires from the Management Service host. Adding more loader threads to your Management Server can help reduce the percent of hour run for the loader.

Rows/Second/Thread is a precise measure of each loader thread's throughput per second. The higher this number, the better. Rows/Second/Thread as high as 1200 have been observed on some smaller, well configured and maintained Enterprise Manager Grid Control sites. If you have not increased the number of loader threads and this number is trending down, it may indicate a problem later. One way to overcome a decreasing rows/second/thread is to add more loader threads.

The number of Loader Threads is always set to one by default in the Management Server configuration file. Each Management Server can have a maximum of 10 loader threads. Adding loader threads to a Management Server typically increases the overall host CPU utilization by 2% to 5% on a Enterprise Manager Grid Control site with many Management Agents configured. Customers can change this value as their site requires. Most medium size and smaller configurations will never need more than one loader thread. Here is a simple guideline for adding loader threads:

Max total (across all Management Servers) loader threads = 2 X number of Management Repository host CPUs

There is a diminishing return when adding loader threads. You will not yield 100% capacity from the second, or higher, thread. There should be a positive benefit, however. As you add loader threads, you should see rows/second/thread decrease, but total rows/hour throughput should increase. If you are not seeing significant improvement in total rows/hour, and there is a constantly growing loader file backlog, it may not be worth the cost of the increase in loader threads. You should explore other tuning or housekeeping opportunities in this case.

To add more loader threads, you can change the following configuration parameter where n is a positive integer [1-10]:

em.loader.threadPoolSize=n

The default is 1 and any value other than [1-10] will result in the thread pool size defaulting to 1. This property file is located in the {ORACLE_HOME}/sysman/config directory. Changing this parameter will require a restart of the Management Service to be reloaded with the new value.

The following two parameters are set for the Receiver module which receives files from agents.

  1. em.loader.maxDataRecvThreads=n (Default 75)

    Where n is a positive integer and default value is 75. This is used to limit the maximum number of concurrent data file receiver threads. So at the peak time only 75 receiver threads will be receiving files and an extra request will be rejected with a Server Busy error. These rejected requests will be resent by the agent after the default retry time.

    Care should be taken while settting this value as too high a value will put an increased load on the OMS machine and shared receiver directory box. If too low a value is set then data file receive throughput will be low.

  2. oracle.sysman.emRep.dbConn.maxConnForReceiver=n (Default 25)

    Where n is a positive integer and default value is 25. This is used to set the maximum number of Repository Database connections for the receive threads. Oracle recommends you set this value equal to em.loader.maxDataRecvThreads, as each Receiver thread gets one DB session and there will be no wait for DB connections.

Rollup Vital Signs

The rollup process is the aggregation mechanism for Enterprise Manager Grid Control. Once an hour, it processes all the new raw data loaded into the Management Repository table MGMT_METRICS_RAW, calculates averages and stores them in the tables MGMT_METRICS_1HOUR and MGMT_METRICS_1DAY. The two vital signs for the rollup are the rows/second and % of hour run. Due to the large volume of data rows processed by the rollup, it tends to be the largest consumer of Management Repository buffer cache space. Because of this, the rollup vital signs can be great indicators of the benefit of increasing buffer cache size.

Rollup rows/second shows exactly how many rows are being processed, or aggregated and stored, every second. This value is usually around 2,000 (+/- 500) rows per second on a site with a decent size buffer cache and reasonable speedy I/O. A downward trend over time for this value may indicate a future problem, but as long as % of hour run is under 100 your site is probably fine.

If rollup % of hour run is trending up (or is higher than your baseline), and you have not yet set the Management Repository buffer cache to its maximum, it may be advantageous to increase the buffer cache setting. Usually, if there is going to be a benefit from increasing buffer cache, you will see an overall improvement in resource utilization and throughput on the Management Repository host. The loader statistics will appear a little better. CPU utilization on the host will be reduced and I/O will decrease. The most telling improvement will be in the rollup statistics. There should be a noticeable improvement in both rollup rows/second and % of hour run. If you do not see any improvement in any of these vital signs, you can revert the buffer cache to its previous size. The old Buffer Cache Hit Ratio metric can be misleading. It has been observed in testing that Buffer Cache Hit Ratio will appear high when the buffer cache is significantly undersized and Enterprise Manager Grid Control performance is struggling because of it. There will be times when increasing buffer cache will not help improve performance for Grid Control. This is typically due to resource constraints or contention elsewhere in the application. Consider using the steps listed in the High CPU Utilization section to identify the point of contention. Grid Control also provides advice on buffer cache sizing from the database itself. This is available on the database Memory Parameters page.

One important thing to note when considering increasing buffer cache is that there may be operating system mechanisms that can help improve Enterprise Manager Grid Control performance. One example of this is the "large memory" option available on Red Hat Linux. The Linux OS Red Hat Advanced Server™ 2.1 (RHAS) has a feature called big pages. In RHAS 2.1, bigpages is a boot up parameter that can be used to pre-allocate large shared memory segments. Use of this feature, in conjunction with a large Management Repository SGA, can significantly improve overall Grid Control application performance. Starting in Red Hat Enterprise Linux™ 3, big pages functionality is replaced with a new feature called huge pages, which no longer requires a boot-up parameter.

Rollup Process

The Rollup process introduces the concept of rollup participating instance; where rollup processing will be distributed among all participating instances. To add a candidate instance to the participating EMROLLUP group, the parameter instance_groups should be set on the instance level as follows:

  • Add EMROLLUP_1 to the instance_group parameter for node 1

    Add EMROLLUP_2 to the instance_group parameter for node 2

  • Introduce the PQ and PW parallel processing modes where:

    • PQ is the parallel query/parallel dml mode. In this mode, each participating instance will have one worker utilizing the parallel degree specified.

    • PW is the parallel worker mode. In this mode, each participating instance will have a number of worker jobs equal to the parallel level specified

  • Distribute the work load for all participating RAC instances as follows:

    • Each participating instance will be allocated equal number of targets. So for (n) number of participating instances with total workload (tl), each instance will be allocated (tl/n)

    • Each worker on any participating instance will be allocated equal number of targets of that instance workload. So for (il) number of targets per instance with (w) number of workers, each worker will be allocated (il/w)

    • For each worker, the load is further divided into batches to control the number of times the rollup SQL is executed. The number of rows per batch will be the total number of rows allocated for the worker divided by the number of batches.

Use the following recommendations as guidelines during the Rollup process:

  • Use the parallel worker (PW) mode, and utilize the participating EMROLLUP_xx instance group.

  • The recommendation is to use the parallel worker mode

  • Splitting the work among more workers will improve the performance and scalability until a certain point where the diminishing returns rule will apply. This is dependent on the number of CPUs available on each RAC node. In this test case, running with 10 workers was the optimal configuration, balancing the response time, machine CPU and IO utilization

  • It is important to set a proper batch size (10 recommended). The optimal run was the one with 10 batches, attributed to balancing the number of executions of the main SQL (calling EMD_1HOUR_ROLLUP) and the sort space needed for each individual execution

  • Start by setting the number of batches to 10 bearing in mind the number of batches can be changed based on the data distribution

The recommendations above will yield the following results. Using the multi-instance parallel worker (8 PW) mode (with the redesigned code described earlier) improves the performance by a factor of 9-13 when utilizing two participating RAC instances.

Rollup row count (in millions) in MGMT_METRICS_1HOURTime (min)WorkersBatch Size
29.53081
9.45810
** For the entire test there were 15779 distinct TARGET_GUID

** The test produced “29.5 Million” new rollup rows in MGMT_METRICS_1HOUR


Run **Rows/WorkersBatches/WorkersRows/BatchTime (min)
8 PW /1 instance39453945140
8 PW /2 instances19731973130

Job, Notification, and Alert Vital Signs

Jobs, notifications, and alerts are indicators of the processing efficiency of the Management Service(s) on your Enterprise Manager Grid Control site. Any negative trends in these values are usually a symptom of contention elsewhere in the application. The best use of these values is to measure the benefit of running with more than one Management Server. There is one job dispatcher in each Management Server. Adding Management Servers will not always improve these values. In general, adding Management Servers will improve overall throughput for Grid Control when the application is not otherwise experiencing resource contention issues. Job, Notification, and Alert vital signs can help measure that improvement.

I/O Vital Signs

Monitoring the I/O throughput of the different channels in your Enterprise Manager Grid Control deployment is essential to ensuring good performance. At minimum, there are three different I/O channels on which you should have a baseline and alert thresholds defined:

  • Disk I/O from the Management Repository instance to its data files

  • Network I/O between the Management Server and Management Repository

  • RAC interconnect (network) I/O (on RAC systems only)

You should understand the potential peak and sustained throughput I/O capabilities for each of these channels. Based on these and the baseline values you establish, you can derive reasonable thresholds for warning and critical alerts on them in Grid Control. You will then be notified automatically if you approach these thresholds on your site. Some Grid Control site administrators can be unaware or mistaken about what these I/O channels can handle on their sites. This can lead to Enterprise Manager Grid Control saturating these channels, which in turn cripples performance on the site. In such an unfortunate situation, you would see that many vital signs would be impacted negatively.

To discover whether the Management Repository is involved, you can use Grid Control to check the Database Performance page. On the Performance page for the Management Repository, click on the wait graph showing the largest amount of time spent. From this you can continue to drill down into the actual SQL code or sessions that are waiting. This should help you to understand where the bottleneck is originating.

Another area to check is unexpected I/O load from non-Enterprise Manager Grid Control sources like backups, another application, or a possible data-mining co-worker who engages in complex SQL queries, multiple Cartesian products, and so on.

Total Repository I/O trouble can be caused by two factors. The first is a lack of regular housekeeping. Some of the Grid Control segments can be very badly fragmented causing a severe I/O drain. Second, there can be some poorly tuned SQL statements consuming much of the site I/O bandwidth. These two main contributors can cause most of the Grid Control vital signs to ptjlummet. In addition, the lax housekeeping can cause the Management Repository's allocated size to increase dramatically.

One important feature of which to take advantage is asynchronous I/O. Enabling asynchronous I/O can dramatically improve overall performance of the Grid Control application. The Sun Solaris™ and Linux operating systems have this capability, but may be disabled by default. The Microsoft Windows™ operating system uses asynchronous I/O by default. Oracle strongly recommends enabling of this operating system feature on the Management Repository hosts and on Management Service hosts as well.

Automatic Storage Management (ASM) is recommended for Enterprise Manager Grid Control repository database storage.

The Oracle Enterprise Manager Performance Page

There may be occasions when Enterprise Manager user interface pages are slow in the absence of any other performance degradation. The typical cause for these slow downs will be an area of Enterprise Manager housekeeping that has been overlooked. The first line of monitoring for Enterprise Manger page performance is the use of Enterprise Manager Beacons. These functionalities are also useful for web applications other than Enterprise Manager.

Beacons are designed to be lightweight page performance monitoring targets. After defining a Beacon target on an Management Agent, you can then define UI performance transactions using the Beacon. These transactions are a series of UI page hits that you will manually walk through once. Thereafter, the Beacon will automatically repeat your UI transaction on a specified interval. Each time the Beacon transaction is run, Enterprise Manager will calculate its performance and store it for historical purposes. In addition, alerts can be generated when page performance degrades below thresholds you specify.

When you configure the Enterprise Manager Beacon, you begin with a single predefined transaction that monitors the home page you specify during this process. You can then add as many transactions as are appropriate. You can also set up additional Beacons from different points on your network against the same web application to measure the impact of WAN latency on application performance. This same functionality is available for all Web applications monitored by Enterprise Manager Grid Control.

After you are alerted to a UI page that is performing poorly, you can then use the second line of page performance monitoring in Enterprise Manager Grid Control. This new end-to-end (or E2E) monitoring functionality in Grid Control is designed to allow you to break down processing time of a page into its basic parts. This will allow you to pinpoint when maintenance may be required to enhance page performance. E2E monitoring in Grid Control lets you break down both the client side processing and the server side processing of a single page hit.

The next page down in the Middle Tier Performance section will break out the processing time by tier for the page. By clicking on the largest slice of the Processing Time Breakdown pie chart, which is JDBC time above, you can get the SQL details. By clicking on the SQL statement, you break out the performance of its execution over time.

The JDBC page displays the SQL calls the system is spending most of its page time executing. This SQL call could be an individual DML statement or a PL/SQL procedure call. In the case of an individual SQL statement, you should examine the segments (tables and their indexes) accessed by the statement to determine their housekeeping (rebuild and reorg) needs. The PL/SQL procedure case is slightly more involved because you must look at the procedure's source code in the Management Repository to identify the tables and associated indexes accessed by the call.

Once you have identified the segments, you can then run the necessary rebuild and reorganization statements for them with the Management Server down. This should dramatically improve page performance. There are cases where page performance will not be helped by rebuild and reorganization alone, such as when excessive numbers of open alerts, system errors, and metric errors exist. The only way to improve these calls is to address (for example, clean up or remove) the numbers of these issues. After these numbers are reduced, then the segment rebuild and reorganization should be completed to optimize performance. These scenarios are covered in Step 3: Use DBA and Enterprise Manager Tasks To Eliminate Bottlenecks Through Housekeeping. If you stay current, you should not need to analyze UI page performance as often, if at all.

Step 5: Extrapolating Linearly Into the Future for Sizing Requirements

Determining future storage requirements is an excellent example of effectively using vital sign trends. You can use two built-in Grid Control charts to forecast this: the total number of targets over time and the Management Repository size over time.

Both of the graphs are available on the All Metrics page for the Management Service. It should be obvious that there is a correlation between the two graphs. A straight line applied to both curves would reveal a fairly similar growth rate. After a target is added to Enterprise Manager Grid Control for monitoring, there is a 31-day period where Management Repository growth will be seen because most of the data that will consume Management Repository space for a target requires approximately 31 days to be fully represented in the Management Repository. A small amount of growth will continue for that target for the next year because that is the longest default data retention time at the highest level of data aggregation. This should be negligible compared with the growth over the first 31 days.

When you stop adding targets, the graphs will level off in about 31 days. When the graphs level off, you should see a correlation between the number of targets added and the amount of additional space used in the Management Repository. Tracking these values from early on in your Enterprise Manager Grid Control deployment process helps you to manage your site's storage capacity proactively. This history is an invaluable tool.

The same type of correlation can be made between CPU utilization and total targets to determine those requirements. There is a more immediate leveling off of CPU utilization as targets are added. There should be no significant increase in CPU cost over time after adding the targets beyond the relatively immediate increase. Introducing new monitoring to existing targets, whether new metrics or increased collections, would most likely lead to increased CPU utilization.

Oracle Enterprise Manager Backup, Recovery, and Disaster Recovery Considerations

Enterprise Manager incorporates a portable browser-based interface to the Grid Control console, as well as the Oracle application server technology, to serve as the middle-tier Management Service tool. The foundation of the tool remains rooted in database server technology to manage both the Management Repository and historical data. This section provides practical approaches to these high availability topics and discusses different strategies when practical for each tier of Enterprise Manager.

Best Practices for Backup

For the Oracle database, the best backup practice is to use the standard database tools and do the following:

  1. Have the database in archivelog mode

  2. Perform regular online backups using the Oracle Suggested Backup strategy option available through Grid Control. This strategy uses Recovery Manager (RMAN).

This strategy creates a full backup and then creates incremental backups on each subsequent run. The incremental changes are then rolled up into the baseline, creating a new full backup baseline.

Using the Oracle Suggested Backup strategy also takes advantage of the capabilities of Grid Control to execute the backups. Backup jobs are automatically scheduled through the Grid Control Job subsystem. The history of the backups is available for review and the status of the backup displays in the Job Activity section of the database target's home page.

Use of this job along with archiving and flashback technologies provides a restore point in the event of the loss of any part of the Management Repository. This backup, along with archive and online logs, allows the Management Repository to be recovered to the last completed transaction.

To enable archiving and flashback technologies, use the Recovery Settings page and enable:

  1. Archive Logging

    Bounce the database and restart all Management Service processes

  2. Flashback Database

    Bounce the database and restart all Management Service processes

  3. Block Change Tracking feature to speed up backup operations.

A summary of how to configure backups using Enterprise Manager is available in the Oracle Database 2 Day DBA manual.

For additional information on database high availability best practices, review the Oracle Database High Availability Architecture and Best Practices manual.

You can set the frequency of the backup job depending on how much data is generated in the Grid Control environment and how much outage time you can tolerate if a restore is required. If the outage window is small and the Service Level Agreement can not be satisfied by restoring the database, consider additional strategies for Management Repository availability such a Real Application Cluster (RAC) or Data Guard database. Additional High Availability options for the Management Repository are documented in the Configuring Enterprise Manager for High Availability paper available from the Maximum Availability Architecture (MAA) page on the Oracle Technology Network (OTN) at http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm

Best Practices for Recovery

For the Oracle Database, the best practice for recovery is to be prepared. Because in some situations the Management Repository, Management Service, and Management Agents will not have access to Grid Control, you will need to use the command-line interface to enter the RMAN commands.

Recovering the Management Repository

If something happens to affect the Management Repository, Grid Control will not be available to provide the management interface to RMAN.

A sample syntax for database recovery using RMAN follows. For detailed information, review the information on database recovery in the Oracle Database Backup and Recovery User's Guide.

RMAN> STARTUP MOUNT;

RMAN> RESTORE DATABASE;

RMAN> RECOVER DATABASE;

RMAN> ALTER DATABASE OPEN;

When considering recovery of the Management Repository, there are two cases to consider:

  • Full recovery of the Management Repository is possible

    There are no special considerations for Enterprise Manager. When the database is recovered, restart the database and Management Service processes. Management Agents will then upload pending files to the Management Repository.

  • Only point in time and incomplete recovery is possible

    Management Agents will be unable to communicate to the Management Repository correctly until they are reset. You must perform the following steps manually:

    1. Shut down the Management Agent

    2. Delete the agntstmp.txt and lastupld.xml files in the $AGENT_HOME/sysman/emd directories

    3. Go the /state and /upload subdirectories and clear their contents

    4. Restart the Management Agent.

    You must repeat these steps for each Management Agent.

In the case of incomplete recovery, Management Agents may not be able to upload data until the previous steps are completed. Additionally, there is no immediate indication in the interface that the Management Agents cannot communicate with the Management Service after this type of recovery. This information would be available from the Management Agent logs or command line Management Agent status. If incomplete recovery is required, you must perform this procedure for each Management Agent.

Recovering the Oracle Management Service

Because the Management Service is stateless, the task is to restore the binaries and configuration files in the shortest time possible. There are two alternatives in this case.

  • Backup the entire software directory structure. You can restore the directory structure to the same directory path should a Management Service failure occur. At the same time, backup the Management Agent associated with this Management Service install. You will need to restore this Management Agent should a restore of the Management Service be required.

  • Reinstall from the original media.

For any highly available Management Service install, it is a recommended practice that you ensure that the /recv directory is protected with a mirroring technology. The /recv directory is the directory the Management Service uses to stage files it receives from Management Agents before writing their contents to the database Management Repository.

After the Management Agent finishes transmitting its XML files to the Management Service, the Management Agent deletes its copy of the XML files. Metric data sent from the Management Agents would then be lost.

Recovering the Oracle Management Agent

The recovery of the Management Agent is similar to the Management Service recovery except that the Management Agent is not stateless. There are two strategies that can be used:

  • If the host name has changed, and you are using an SLB to manage connections, you have to modify the connection pools in the SLB to drop the old host name and add the new name. If you are not using an SLB, each agent that previously pointed to the old OMS host must have its emd. properties file modified to point to the new OMS host name. You can use this procedure to handle a case where you need to bring up a new OMS on a new host because the former machine has crashed.

    Assuming the host name has not changed, a disk backup and restore is sufficient.

    1. Delete the agntstmp.txt and the lastupld.xml files from the /sysman/emd directory.

    2. Clear the /state and /upload subdirectories of all entries before restarting the Management Agent.

    3. Start the Management Agent. This step forces a rediscovery of the targets on the host.

  • Reinstall the Management Agent from the original media.

As with the Management Service, it is recommended you protect the /state and /upload directories with a mirroring technology.

Preventing Data Loss and Down Time While Switching From a Non-shared File System to a Shared File System

Data loss and down time can be prevented while switching from a Non-shared File System to a Shared File System by switching each OMS to a Shared File System in rolling fashion and ensuring that there is no backlog in the receive directory. To prevent data loss, follow these steps for each OMS:

  1. Shutdown the http server on the OMS.

  2. Wait for the existing backlog to get processed. To determine whether the existing backlog has been processed, continue to monitor the Loader receive directory. Wait until all the files in the receive directory are uploaded.

  3. Run emctl config oms loader -shared yes -dir <sharedfs>. If there is any backlog, this command prompts you to clear the backlog.

  4. Bounce the OMS.

Best Practice for Disaster Recovery (DR)

In the event of a node failure, you can restore the database using RMAN or OS commands. To speed this process, implement Data Guard to replicate the Management Repository to a different hardware node.

Management Repository

If you are restoring the Management Repository to a new host, restore a backup of the database and modify the emoms.properties file for each Management Service manually to point to the new Management Repository location. In addition, you must update the targets.xml file for each Management Service to reflect the new Management Repository location. If there is a data loss during recovery, see Recovering the Management Repository for information.

To speed Management Repository reconnection from the Management Service in the event of a single Management Service failure, configure the Management Service with a Transparent Application Failover (TAF) aware connect string. You can configure the Management Service with a TAF connect string in the emoms.properities file that will automatically redirect communications to another node using the FAILOVER syntax. An example follows:

EM=

(description=

(failover=on)

(address_list=

(failover=on)

(address=(protocol=tcp)(port=1522)(host=EMPRIM1.us.oracle.com))

(address=(protocol=tcp)(port=1522)(host=EMPRIM2.us.oracle.com)))

(address_list=

(failover=on)

(address=(protocol=tcp)(port=1522)(host=EMSEC1.us.oracle.com))

(address=(protocol=tcp)(port=1522)(host=EMSEC2.us.oracle.com)))

(connect_data=(service_name=EMrep.us.oracle.com)))

Oracle Management Service

Preinstall the Management Service and Management Agent on the hardware that will be used for Disaster Recovery. This eliminates the step of restoring a copy of the Enterprise Manager binary files from backup and modifying the Management Service and Management Agent configuration files.


Note:

In the event of a disaster, do not restore the Management Service and Management Agent binaries from an existing backup to a new host because there are host name dependencies. Always do a fresh install.

Management Agent

In the event of a true disaster recovery, it is easier to reinstall the Management Agent and allow it to do a clean discovery of all targets running on the new host.

PKSmtPKOXEOEBPS/ha_single_resource.htm High Availability: Single Resource Configurations

17 High Availability: Single Resource Configurations

Single resource configurations consist of a single instance Enterprise Manager configuration utilizing some form of protected storage to protect both the repository database and the software installation. As one of the most common installation types, implementing high availability for single resource configurations is cost effective from both time and resource standpoints as the objective is to leverage the technology that is already available, such as Recovery Manager, Flashback, Ultrasafe, and Automated Storage Management.

This chapter covers the following topics:

About Single Resource Configurations

The configurations described in this chapter are provided as examples only. The actual Grid Control configurations that you deploy in your own environment will vary depending upon the needs of your organization. For example, in a production environment you will likely want to implement firewalls and other security considerations. For specific information on implementing firewalls and security protocols, you should refer to the following Enterprise Manager documentation:

Besides providing a description of common configurations, this chapter can also help you understand the architecture and flow of data among the Grid Control components. Based on this knowledge, you can make better decisions about how to configure Grid Control for your specific management requirements.

The Grid Control architecture consists of the following software components:

  • Oracle Management Agent

  • Oracle Management Service

  • Oracle Management Repository

  • Oracle Enterprise Manager 11g Grid Control Console


    See Also:

    Oracle Enterprise Manager Concepts for more information about each of the Grid Control components

Deploying Grid Control Components on a Single Host

Figure 17-1 shows how each of the Grid Control components are configured to interact when you install Grid Control on a single host. This is the default configuration that results when you use the Installing a New Grid Control installation type.

In Enterprise Manager release 11g, the installation does not create a new database. You must install a database which should be on the same host as Enterprise Manager.

Figure 17-1 Grid Control Components Installed on a Single Host

Description of Figure 17-1 follows

When you install all the Grid Control components on a single host, the management data travels along the following paths:

  1. Administrators use the Grid Control console to monitor and administer the managed targets that are discovered by the Management Agents on each host. The Grid Control console uses the following URL to connect to the Oracle HTTP Server:

    https://host1.acme.com:7799/em

    The Management Service retrieves data from the Management Repository as it is requested by the administrator using the Grid Control console.

  2. The Management Agent loads its data (which includes management data about all the managed targets on the host, including the Management Service and the Management Repository database) by way of the Oracle HTTP Server upload URL. The Management Agent uploads data directly to Oracle HTTP Server. The default port for the upload URL is 1159 (if it is available during the installation procedure). The upload URL is defined by the REPOSITORY_URL property in the following configuration file in the Management Agent home directory:

    AGENT_HOME/sysman/config/emd.properties (UNIX)
    AGENT_HOME\sysman\config\emd.properties (Windows)
    

    See Also:

    For more information about the Oracle Enterprise Manager directory structure (AGENT_HOME directory in particular), see the Oracle® Enterprise Manager Grid Control Advanced Installation and Configuration Guide.

  3. The Management Service uses JDBC connections to load data into the Management Repository database and to retrieve information from the Management Repository so it can be displayed in the Grid Control console. The Management Repository connection details can be listed and changed by using the following emctl commands:

    emctl config oms -list_repos_details
    emctl config oms -store_repos_details
    

    See Also:

    "Reconfiguring the Oracle Management Service" for more information on modifying the Management Repository connection information.

  4. The Management Service sends data to the Management Agent by way of HTTP. The Management Agent software includes a built-in HTTP listener that listens on the Management Agent URL for messages from the Management Service. As a result, the Management Service can bypass the Oracle HTTP Server and communicate directly with the Management Agent. If the Management Agent is on a remote system, no Oracle HTTP Server is required on the Management Agent host.

    The Management Service uses the Management Agent URL to monitor the availability of the Management Agent, submit Enterprise Manager jobs, and other management functions.

    The Management Agent URL can be identified by the EMD_URL property in the following configuration file in the Management Agent home directory:

    AGENT_HOME/sysman/config/emd.properties (UNIX)
    AGENT_HOME\sysman\config\emd.properties (Windows)
    

    For example:

    EMD_URL=https://host1.acme.com:3872/emd/main
    

    In addition, the name of the Management Agent as it appears in the Grid Control console consists of the Management Agent host name and the port used by the Management Agent URL.

Backup and Recovery

Although Enterprise Manager functions as a single entity, technically, it is built on a distributed, multi-tier software architecture composed of the following software components:

  • Oracle Management Services (OMS)

  • Oracle Management Agent (Agent)

  • Oracle Management Repository (Repository)

Each component, being uniquely different in composition and function, requires different approaches to backup and recovery. For this reason, the backup and recovery strategies are discussed on a per-tier basis in this chapter. For an overview of Enterprise Manager architecture, refer to the Oracle® Enterprise Manager Grid Control Basic Installation Guide.

Oracle Configuration Manager

Oracle Configuration Manager (OCM) is used to collect client configuration information and upload it to the Oracle repository. When the client configuration data is uploaded on a regular basis, customer support representatives can analyze this data and provide better service to the customers.

When installing Oracle software, the installer provides an option to setup and configure OCM. In recovery scenarios where software is installed in software-only mode, OCM can be configured manually by running the following from OMS and agent Oracle Homes:

<OracleHome>/ccr/bin/setupCCR

The Oracle Configuration Manager client is installed into the ORACLE_HOME directory. Once installed, OCM collects configuration data related to the ORACLE_HOME directory and the host on which it is installed. In addition to collecting and uploading configuration data, it also checks if any software updates to the Oracle Configuration Manager client are available. If updates are available, it downloads them and updates the Oracle Configuration Manager software installed on the customer's system.

Repository Backup and Recovery

The Repository is the storage location where all the information collected by the Agent gets stored. It consists of objects such as database jobs, packages, procedures, views, and tablespaces. Because it is configured in an Oracle Database, the backup and recovery strategies for the repository are essentially the same as those for the Oracle Database. Backup procedures for the database are well established standards and can be implemented using the RMAN backup utility, which can be accessed via the Enterprise Manager console.

Repository Backup

Oracle recommends using High Availability Best Practices for protecting the Repository database against unplanned outages. As such, use the following standard database backup strategies.

  • Database should be in archivelog mode. Not running the repository database in archivelog mode leaves the database vulnerable to being in an unrecoverable condition after a media failure.

  • Perform regular hot backups with RMAN using the Recommended Backup Strategy option via the Enterprise Manager console. Other utilities such as DataGuard and RAC can also be used as part of a comprehensive strategy to prevent data loss.

Adhering to these strategies will create a full backup and then create incremental backups on each subsequent run. The incremental changes will then be rolled up into the baseline, creating a new full backup baseline.

Using the Recommended Backup Strategy also takes advantage of the capabilities of Enterprise Manager to execute the backups: Jobs will be automatically scheduled through the Job sub-system of Enterprise Manager. The history of the backups will then be available for review and the status of the backup will be displayed on the repository database target home page. This backup job along with archiving and flashback technologies will provide a restore point in the event of the loss of any part of the repository. This type of backup, along with archive and online logs, allows the repository to be recovered to the last completed transaction.

You can view when the last repository backup occurred on the Management Services and Repository Overview page under the Repository details section.

Setting Up the Backup

First, navigate to the Enterprise Manager Recovery Settings page (Target-->Database--><Repository Database Target>-->Availability-->Recovery Settings) and enable Archive Logging then Flashback Database as shown in Figure 17-2.

Figure 17-2 Recovery Settings Page

Recovery settings page

Next, navigate to the Backup Policies page (Target-->Database--><Repository Database Target>-->Availability-->Backup Settings-->Policy) and enable Block Change Tracking to speed up backup operations as shown in Figure 17-3.

Figure 17-3 Backup Policy Page

Description of Figure 17-3 follows

Figure 17-4 Backup Policy Page

backup policy page

A thorough summary of how to configure backups using Enterprise Manager is available in the Oracle Database 2 Day DBA guide. For additional information on Database high availability best practices, review the Oracle Database High Availability Best Practices documentation.

Repository Recovery

Recovery of the Repository database must be performed using RMAN since Grid Control will not be available when the repository database is down. There are two recovery cases to consider:

  • Full Recovery: No special consideration is required for Enterprise Manager.

  • Point-in-Time/Incomplete Recovery: Recovered repository may be out of sync with Agents because of lost transactions. In this situation, some metrics may show up incorrectly in the Grid Control console unless the repository is synchronized with the latest state available on the Agents.

A repository resync feature (Enterprise Manager version 10.2.0.5 and later) allows you to automate the process of synchronizing the Enterprise Manager repository with the latest state available on the Agents.


Note:

resync requires Agents version 10.2.0.5 or later. Older Agents must be synchronized manually. See "Manually Resynchronizing Agents".

To resynchronize the repository with the Agents, you use Enterprise Manager Command-line utility (emctl) resync repos command:

emctl resync repos -full -name "<descriptive name for the operation>"

You must run this command from the OMS Oracle Home after restoring the repository but BEFORE starting the OMS. After submitting the command, start up all OMS's and monitor the progress of repository resychronization from the Enterprise Manager console's Repository Resynchronization page, as shown in Figure 17-5.

Figure 17-5 Repository Synchronization Page

Description of Figure 17-5 follows

Repository recovery is complete when the resynchronization jobs complete on all Agents.

Oracle strongly recommends that the repository database be run in archivelog mode so that in case of failure, the database can be recovered to the latest transaction. If the database cannot be recovered to the last transaction, Repository Synchronization can be used to restore monitoring capabilities for targets that existed when the last backup was taken. Actions taken after the backup will not be recovered automatically. Some examples of actions that will not be recovered automatically by Repository Synchronization are:

  • Notification Rules

  • Preferred Credentials

  • Groups, Services, Systems

  • Jobs/Deployment Procedures

  • Custom Reports

  • New Agents

Manually Resynchronizing Agents

The Enterprise Manager Repository Synchronization feature can only be used for Agents 10.2.0.5 or later. Older Agents must be resynchronized manually by shutting down the Agent using the following procedure:

  1. Shut down the Agent.

  2. Delete the agentstmp.txt, lastupld.xml, state/* and upload/* files from the <AGENT_HOME>/sysman/emd directory.

  3. Restart the Agent.

Recovery Scenarios

A prerequisite for repository (or any database) recovery is to have a valid, consistent backup of the repository. Using Enterprise Manager to automate the backup process ensures regular, up-to-date backups are always available if repository recovery is ever required. Recovery Manager (RMAN) is a utility that backs up, restores, and recovers Oracle Databases. The RMAN recovery job syntax should be saved to a safe location. This allows you to perform a complete recovery of the Enterprise Manager repository database. In its simplest form, the syntax appears as follows:

run {

restore database;

recover database;

}

Actual syntax will vary in length and complexity depending on your environment. For more information on extracting syntax from an RMAN backup and recovery job, or using RMAN in general, see the Oracle Database Backup and Recovery User's Guide.

The following scenarios illustrate various repository recovery situations along with the recovery steps.

Full Recovery on the Same Host

Repository database is running in archivelog mode. Recent backup, archive log files and redo logs are available. The repository database disk crashes. All datafiles and control files are lost.

Resolution:

  1. Stop the OMS(s) using emctl stop oms.

  2. Recover the database using RMAN

  3. Bring the site up using the command emctl start oms on all OMS(s).

  4. Verify that the site is fully operational.

Incomplete Recovery on the Same Host

Repository database is running in noarchivelog mode. Full offline backup is available. The repository database disk crashes. All datafiles and control files are lost.

Resolution:

  1. Stop the OMS(s) using emctl stop oms.

  2. Recover the database using RMAN.

  3. Initiate Repository Resync using emctl resync repos -full -name "<resync name>" from one of the OMS Oracle Home.

  4. Start the OMS(s) using emctl start oms.

  5. Manually fix any pre-10.2.0.5 Agent by shutting down the Agent, deleting the agentstmp.txt, lastupld.xml, state/* and upload/* files under the <AGENT_HOME>/sysman/emd directory, and then restarting the Agents.

  6. Log into Grid Control. Navigate to Management Services and Repository Overview page. Click on Repository Synchronization under Related Links. Monitor the status of resync jobs. Resubmit failed jobs, if any, after fixing the error.

  7. Verify that the site is fully operational.

Full Recovery on a Different Host

The repository database is running on host "A" in archivelog mode. Recent backup, archive log files and redo logs are available. The repository database crashes. All datafiles and control files are lost.

Resolution:

  1. Stop the OMS(s) using the command emctl stop oms.

  2. Recover the database using RMAN on a different host (host "B").

  3. Correct the connect descriptor for the repository in credential store by running

    $emctl config oms –store_repos_details -repos_conndesc <connect descriptor> -repos_user sysman
    
  4. Start the OMS(s) using the command emctl start oms.

  5. Relocate the repository database target to the Agent running on host "B" by running the following command from the OMS:

    $emctl config repos -host <hostB> -oh <OH of repository on hostB>  -conn_desc "<TNS connect descriptor>"
    

    Note:

    This command can only be used to relocate the repository database under the following conditions:
    • An Agent is already running on this machine.

    • No database on host "B" has been discovered.

    If a new Agent had been installed on host "B", you must ensure there are NO previously discovered database targets.


  6. Change the monitoring configuration for the OMS and Repository target: by running the following command from the OMS:

    $emctl config emrep -conn_desc "<TNS connect descriptor>"
    
  7. Verify that the site is fully operational.

Incomplete Recovery on a Different Host

The repository database is running on host "A" in noarchivelog mode. Full offline backup is available. Host "A" is lost due to hardware failure. All datafiles and control files are lost.

Resolution:

  1. Stop the OMS(s) using emctl stop oms.

  2. Recover the database using RMAN on a different host (host "B").

  3. Correct the connect descriptor for the repository in credential store.

    $emctl config oms –store_repos_details -repos_conndesc <connect descriptor> -repos_user sysman
    
  4. Initiate Repository Resync:

    emctl resync repos -full -name "<resync name>"

    from one of the OMS Oracle Homes.

  5. Start the OMS(s) using the command emctl start oms.

  6. Run the command to relocate the repository database target to the Agent running on host "B":

    emctl config repos -agent <agent on host B> -host <hostB> -oh <OH of repository on hostB> -conn_desc "<TNS connect descriptor>"

  7. Run the command to change monitoring configuration for the OMS and Repository target:

    emctl config emrep -conn_desc "<TNS connect descriptor>"

  8. Manually fix all pre-10.2.0.5 Agents by shutting down the Agents, deleting the agentstmp.txt, lastupld.xml, state/* and upload/* files under the <AGENT_HOME>/sysman/emd directory and then restarting the Agents.

  9. Log in to Grid Control. Navigate to Management Services and Repository Overview page. Click on Repository Synchronization under Related Links. Monitor the status of resync jobs. Resubmit failed jobs, if any, after fixing the error mentioned.

  10. Verify that the site is fully operational.

Oracle Management Service Backup and Recovery

The Oracle Management Service (OMS) orchestrates with Management Agents to discover targets, monitor and manage them, and store the collected information in a repository for future reference and analysis. The OMS also renders the Web interface for the Enterprise Manager console. For Enterprise Manager version 11.1, the OMS architecture has changed.

Backing Up the OMS

The OMS is generally stateless. Some transient and configuration data is stored on the OMS file system. The shared loader “recv”directory stores metric data uploaded from Agents temporarily before the data is loaded into the repository. If an OMS goes down, other surviving OMS(s) upload the data stored in the shared loader location. In a High Availability (HA) configuration, the shared loader receive directory should be protected using an HA storage technology, such as a redundant disk.

A snapshot of OMS configuration can be taken using the emctl exportconfig oms command.

emctl exportconfig oms [-sysman_pwd <sysman password>]

      [-dir <backup dir>]     Specify directory to store backup file

      [-keep_host]            Specify this parameter if the OMS was installed

                              using a virtual hostname. 

                              For example: ORACLE_HOSTNAME

Note:

The exportconfig oms command is only available with Enterprise Manager version 10.2.0.5 or newer.

Running exportconfig captures a snapshot of the OMS at a given point in time, thus allowing you to back up the most recent OMS configuration on a regular basis. If required, the most recent snapshot can then be restored on a fresh OMS installation on the same or different host.

Backup strategies for the OMS components are as follows:

  • Software Homes

    Composed of three WebLogic components – Middeware Home, the OMS Oracle Home and the WebTier (OHS) Oracle Home. Software Homes only change when patches or patchsets are applied. For this reason, filesystem-level backups should be taken after each patch/patchset application. You should back up the Oracle inventory files along with the Software Homes. .


    Important:

    Beginning with Enterprise Manager version 11.1, the location of the OMS Oracle Home must be the same for all OMS's in your monitored environment.

  • Instance Home

    Composed of WebLogic, OMS and WebTier configuration files. The Instance Home can be backed up using the emctl exportconfig oms command.

  • Software Library

    Composed of components used by Enterprise Manager patching and provisioning functions. Oracle Database Filesystem (DBFS) is recommended for software library backup. DBFS technology allows an Oracle database tablespace to be exposed to applications as a mounted filesystem. Internally, all the files are stored as secure files in the Oracle database. Storing the software library in the Enterprise Manager repository database using DBFS lets you leverage the comprehensive capabilities of the Oracle database to take consistent backups of the software library along with the Enterprise Manager repository. For more information about DBFS, see the Oracle® Database SecureFiles and Large Objects Developer's Guide.

  • Shared Loader RECV Directory

    The shared loader receive (RECV) directory temporarily stores metric data uploaded from Agents before the data is loaded into the repository. Use a high availability storage technology to protect the receive directory.

  • AdminServer

    Beginning with Enterprise Manager version 11.1, the OMS's WebLogic architecture introduces the concept of an AdminServer. The AdminServer operates as the central control entity for the configuration of the entire OMS(s) domain. The AdminServer is an integral part of the first OMS installed in your Grid Control deployment and shares the Software Homes and Instance Home.

Recovering the OMS

If an OMS is lost, it should be reinstalled using “Installing Software Only and Configuring Later". This is an additional Management Service option documented in the Oracle Enterprise Manager Grid Control Installation and Basic Configuration guide. The OMS configuration can then be restored with the OMS Configuration Assistant using the following command:

omsca recovery -BACKUP_FILE <file>

Use the export file generated by the emctl exportconfig command shown in the previous section.

Recovering an OMS essentially consists of two steps, recovering the Software Homes and then configuring the Instance Home. When restoring on the same host, the software homes can be restored from filesystem backup. In case a backup does not exist, the software homes can be reconstructed using the software-only installation of WebLogic and OMS, software-only installation of add-ons (if any) and all patches that were applied before the crash. As stated earlier, the location of the OMS Oracle Home is fixed and cannot be changed. Hence, ensure that the OMS Oracle Home is restored in the same location that was used previously.

Once the Software Homes are recovered, the instance home can be reconstructed using the omsca command in recovery mode.

OMS Recovery Scenarios

The following scenarios illustrate various OMS recovery situations along with the recovery steps.


Important:

A prerequisite for OMS recovery is to have recent, valid OMS configuration backups available. Oracle recommends that you back up the OMS using the emctl exportconfig oms command whenever an OMS configuration change is made. This command must be run on the primary OMS running the WebLogic AdminServer.

Alternatively, you can run this command on a regular basis using the Enterprise Manager Job system.


Single OMS, No Server Load Balancer (SLB), OMS Restored on the same Host

Site hosts a single OMS. No SLB is present. The OMS configuration was backed up using the emctl exportconfig oms command on the primary OMS running the AdminServer. The OMS Oracle Home is lost.

Resolution:

  1. Ensure that loader receive directory and software library locations are still accessible.

  2. Restore the software homes from filesystem backup taken earlier. Alternately, if a backup does not exist, use the software-only install method to reconstruct the WebLogic and OMS Oracle Home, add-ons that were installed earlier need to be reinstalled in software-only mode and all patches that were applied earlier need to be reapplied. Remember that the location of OMS Oracle Home needs to be the same as one used before.

  3. Run omsca in recovery mode specifying the export file taken earlier to configure the OMS:

    omsca recover –as –ms –backup_file <file>
    

    Note: The -backup_file to be passed must be the latest file generated from emctl exportconfig oms command.

  4. Configure agent:

    >agentca -f
    >emctl secure agent -emdWalletSrcUrl <oms url>
    
  5. At this point, two possibilities exist depending upon the port used by the reinstalled agent that comes along with the OMS:

    Option A: Agent uses the same port as the previous installation.

    • OMS automatically blocks the Agent. Resync the Agent from Agent homepage

    Option B: Agent uses a different port.

    • Run the command to relocate the OMS and Repository target to reinstalled Agent:

      emctl config emrep -agent <reinstalled agent>

      Example: emctl config emrep -agent foo.us.oracle.com:3872

  6. Locate duplicate targets from the Management Services and Repository Overview page. Relocate duplicate targets from the old agent to the reinstalled Agent. Delete the old Agent.

  7. Verify that the site is fully operational.

Single OMS, No SLB, OMS Restored on a Different Host

Site hosts a single OMS. The OMS is running on host "A." No SLB is present. The OMS configuration was backed up using the emctl exportconfig oms command. Host "A" is lost.

Resolution:

  1. Ensure that loader receive directory and software library locations are accessible from Host "B".

  2. Usually filesystem restore does not work across hosts. Use the software only install method to reconstruct the WebLogic and OMS Oracle home, add-ons that were installed earlier need to re-installed in software only mode and all patches that were applied earlier need to be reapplied. Remember that the location of OMS Oracle Home must be the same as one used before.

  3. Run omsca in recovery mode specifying the OMS configuration backup file generated earlier to configure the new OMS:

    omsca recover –as –ms –backup_file <file>
    
  4. Configure agent:

    agentca -f
    emctl secure agent -emdWalletSrcUrl <oms url>
    
  5. Change the OMS to which all Agents point and then resecure all Agents

    • Make all Agents in the deployment point to new OMS running on Host "B". On each Agent, run the following command

      emctl secure agent -emdWalletUrlSrc "http://hostB:<httpport>/em"

    • Run the command to relocate OMS and Repository target to Agent "B":

      emctl config emrep -agent <agent on host "B">.


      Note:

      Because the new machine is using a different hostname from the one originally hosting the OMS, all Agents in your monitored environment must be told where to find the new OMS.

  6. Locate duplicate targets from the Management Services and Repository Overview page of the Enterprise Manager console. Click the Duplicate Targets link to access the Duplicate Targets page. To resolve duplicate target errors, the duplicate target must be renamed on the conflicting Agent. Relocate duplicate targets from Agent "A" to Agent "B".

  7. Verify that the site is fully operational.

Single OMS, No SLB, OMS Restored on a Different Host using the Original Hostname

Site hosts a single OMS. The OMS is running on host "A." No SLB is present. The OMS configuration was backed up using the emctl exportconfig oms command. Host "A" is lost.

Resolution:

  1. Ensure that loader receive directory and software library locations are accessible from Host "B".

  2. Usually filesystem restore does not work across hosts. Use the software-only install method to reconstruct the WebLogic and OMS Oracle home, add-ons that were installed earlier need to re-installed in software-only mode and all patches that were applied earlier need to be reapplied. Remember that the location of OMS Oracle home needs to be the same as one used before.

  3. Modify the network configuration such that HostB also responds to hostname Host "A". Specific instructions on how to configure this are beyond the scope of this document. However, some general configuration suggestions are:

    • Modify your DNS server such that both Host "B" and Host "A" network addresses resolve to the physical IP of Host "B".

    • Multi-home Host "B". Configure an additional IP of Host "A" on Host "B". For example, on Host "B" run the following commands:

      > ifconfig eth0:1 <IP of HostA> netmask <netmask>  
      > /sbin/arping -q -U -c 3 -I eth0 <IP of HostA>
      
  4. Run omsca in recovery mode specifying the export file taken earlier to configure the OMS:

    omsca recover –as –ms –backup_file <file>
    
  5. Resecure the OMS:

    emctl secure oms host <Host A>

  6. Configure agent:

    > agentca -f followed by

    > emctl secure agent -emdWalletSrcUrl <oms url>

  7. Run the command to relocate Management Services and Repository target to Agent "B":

    emctl config emrep -agent <agent on host B>

  8. Locate duplicate targets from the Management Services and Repository Overview page of the Enterprise Manager console. Click the Duplicate Targets link to access the Duplicate Targets page. To resolve duplicate target errors, the duplicate target must be renamed on the conflicting Agent. Relocate duplicate targets from Agent "A" to Agent "B".

  9. Verify that the site is fully operational.

Multiple OMS, Server Load Balancer, Primary OMS Recovered on the Same Host

Site hosts multiple OMSs. All OMSs are fronted by a Server Load Balancer. OMS configuration backed up using the emctl exportconfig oms command on the primary OMS running the WebLogic AdminServer. The primary OMS is lost.

Resolution:

  1. Ensure that shared loader receive directory and shared software library locations are still accessible.

  2. Restore the software homes from filesystem backup taken earlier. Alternately if backup does not exist, use the software only install method to reconstruct the WebLogic and OMS Oracle home, add-ons that were installed earlier need to re-installed in software only mode and all patches that were applied earlier need to be reapplied. Remember that the location of OMS Oracle home needs to be the same as one used before.

  3. Run omsca in recovery mode specifying the export file taken earlier to configure the OMS:

    omsca recover -as -ms -backup_file <file>

  4. Resecure the Agent that gets installed along with OMS.

    emctl secure agent -emdWalletSrcUrl "http://slb:<httpport>/em"

  5. At this point, two possibilities exist depending upon the port used by the reinstalled agent that comes along with the OMS:

    • Option A: Agent gets the same port as earlier--OMS automatically blocks the agent. Resync the agent from agent homepage.

    • Option B: Agent gets a different port--Run the command to relocate Management Services and Repository target to reinstalled agent:

      emctl config emrep -agent <reinstalled agent>

      Locate duplicate targets from the Management Services and Repository Overview page. Relocate duplicate targets from old agent to reinstalled agent. Delete the old agent.

  6. Re-enroll the additional OMS, if any, with the recovered Administration Server by running emctl enroll oms on each additional OMS.

  7. Verify that the site is fully operational.

Multiple OMS, Server Load Balancer configured, Primary OMS Recovered on a Different Host

Site hosts multiple OMSs. OMSs fronted by a Server Load Balancer. OMS Configuration Backed Up Using emctl exportconfig oms command. Primary OMS on host "A" is lost and needs to be recovered on Host "B".

  1. Ensure that shared loader receive directory and shared software library locations are accessible from the new OMS host (host "B")

  2. Filesystem restore typically does not work across hosts. Use the software-only install method to reconstruct the WebLogic and OMS Oracle home. Add-ons that were installed earlier need to re-installed in software-only mode and all patches that were applied earlier need to be reapplied. Remember that the location of OMS Oracle home needs to be the same as one used before.

  3. Run omsca in recovery mode specifying the export file taken earlier to configure the OMS:

    omsca recover -as -ms -backup_file <file>

  4. Configure agent:

    agentca -f followed by

    emctl secure agent -emdWalletSrcUrl "http://slb:<httpport>/em"

  5. Add the new OMS to the SLB

  6. Relocate the OMS and Repository target to reinstalled Agent:

    emctl config emrep -agent <agent on Host B>

  7. Locate duplicate targets from the Management Services and Repository Overview page. Relocate duplicate targets from the Agent on Host "A" to the Agent on Host "B". Delete the Agent on Host "A".

  8. Re-enroll the additional OMS, if any, with the recovered Administration Server

    emctl enroll oms -as_host <HostB> -as_port <admin secure port>

    Run this command on each additional OMS.

  9. Verify that the site is fully operational.

Multiple OMS, SLB configured, additional OMS recovered on same or different host

Multi OMS site. OMSs fronted by SLB. OMS configuration backed up using emctl exportconfig oms command on the first OMS. Additional OMS is lost and needs to be recovered on same or different host.

  1. Ensure that shared loader receive directory and shared software library locations are accessible.

  2. If recovering on same host, restore the Software Homes from a filesystem backup. Alternatively, if a backup does not exist, or when recovering on a different host, use the software-only install method to reconstruct the WebLogic and OMS Oracle Home. Add-ons that were installed earlier need to reinstalled in software-only mode and all patches that were applied earlier need to be reapplied. The location of the restored OMS Oracle home needs to be the same as the previous.

  3. Run omsca in recovery mode specifying the export file taken earlier to configure the OMS:

    omsca recover –ms –backup_file <file>

  4. Configure agent:

    agentca -f followed by emctl secure agent -emdWalletSrcUrl "http://slb:<httpport>/em"

  5. Add the new OMS (if recovered on a different host) to the SLB

  6. At this point, three possibilities exist depending upon the port used by the reinstalled agent that comes along with the OMS:

    • Option A: OMS installed on same host and agent gets the same port as earlier

      OMS automatically blocks the agent. Resync the agent from agent homepage.

    • Option B: OMS installed on same host and agent gets a different port

      Locate duplicate targets from the Management Services and Repository Overview page. Relocate duplicate targets from old agent to reinstalled agent. Delete the old agent.

    • Option C: OMS installed on different host

      Locate duplicate targets from the Management Services and Repository Overview page. Relocate duplicate targets from old agent to reinstalled agent.

  7. Verify that the site is fully operational.

Agent Backup and Recovery

The Agent is an integral software component that is deployed on each monitored host. It is responsible for monitoring all the targets running on those hosts, communicating that information to the middle-tier OMS and managing and maintaining the hosts and its targets.

Backing Up Agents

There are no special considerations for backing up Agents. As a best practice, reference Agent installs should be maintained for different platforms and kept up-to-date in terms of customizations in the emd.properties file and patches applied. Use Deployment options from the Grid Control console to install and maintain reference Agent installs.

Recovering Agents

If an Agent is lost, it should be reinstalled by cloning from a reference install. Cloning from a reference install is often the fastest way to recover an Agent install as it is not necessary to track and reapply customizations and patches. Care should be taken to reinstall the Agent using the same port. Using the Enterprise Manager's Agent Resynchronization feature, a reinstalled Agent can be reconfigured using target information present in the repository. When the Agent is reinstalled using the same port, the OMS detects that it has been re-installed and blocks it temporarily to prevent the auto-discovered targets in the re-installed Agent from overwriting previous customizations.


Blocked Agents:

A Blocked Agent is a condition where the OMS rejects all heartbeat or upload requests from the blocked Agent. Hence, a blocked Agent will not be able to upload any alerts or metric data to the OMS. However, blocked Agents continue to collect monitoring data.

The Agent can be resynchronized and unblocked from the Agent homepage by clicking on the Resynchronize Agent button. Resynchronization pushes all targets from the repository to the Agent and then unblocks the Agent.

Agent Recovery Scenarios

The following scenarios illustrate various Agent recovery situations along with the recovery steps. Agent recovery is supported for Agent versions 10.2.0.5 and later. The Agent resynchronization feature requires that a reinstalled Agent use the same port as the previous Agent that crashed.

Agent Reinstall Using the Same Port

An Agent is monitoring multiple targets. The Agent installation is lost.

  1. Deinstall Agent OracleHome using the Oracle Universal Installer.

  2. Install a new Agent or use the Agent clone option to reinstall the Agent though Enterprise Manager. Specify the same port as used by the crashed Agent. The location of install need not be same as previous install.

    The OMS detects that Agent has been re-installed and blocks the Agent.

  3. Initiate Agent Resynchronization from the Agent homepage.

    All targets in the repository are pushed to the new Agent. The Agent is instructed to clear backlogged files and then do a clearstate. Agent is unblocked.

  4. Reconfigure User-defined Metrics if the location of User-defined Metric scripts have changed.

  5. Verify that the Agent is operational and all target configurations have been restored.

Agent Restore from Filesystem Backup

An Agent is monitoring multiple targets. File system backup for the Agent OracleHome exists. The Agent install is lost.

  1. Deinstall Agent OracleHome using OUI.

  2. Restore the Agent from file system backup. Start the Agent.

    OMS detects that Agent has been restored from backup and blocks the Agent.

  3. Initiate Agent Resynchronization from the Agent homepage.

    All targets in the repository are pushed to the new Agent. The Agent is instructed to clear backlogged files and performs a clearstate. The Agent is unblocked.

  4. Verify that the Agent is functional and all target configurations have been restored using the following emctl commands:

    emctl status agent
    
    emctl upload agent 
    

    There should be no errors and no XML files in the backlog.

Recovering from a Simultaneous OMS-Repository Failure

When both OMS and repository fail simultaneously, the recovery situation becomes more complex depending upon factors such as whether the OMS and repository recovery has to be performed on the same or different host, or whether there are multiple OMSs fronted by an SLB. In general, the order of recovery for this type of compound failure should be repository first, followed by OMS(s) following the steps outlined in the appropriate recovery scenarios discussed earlier. The following scenarios illustrate two OMS-Repository failures and the requisite recovery steps.

Collapsed Configuration: Incomplete Repository Recovery, Primary OMS on the Same Host

Repository and the primary OMS are installed on same host (host "A"). The repository database is running in noarchivelog mode. Full cold backup is available. A recent OMS backup file exists ( emctl exportconfig oms). The repository, OMS and the Agent crash.

  1. Follow the repository recovery procedure shown in Incomplete Recovery on the Same Host with the following exception:

    Since the OMS OracleHome is not available and repository resynchronization has to be initiated before starting an OMS against the restored repository, submit "resync" via the following PL/SQL block. Log into the repository as SYSMAN using SQLplus and run:

    begin emd_maintenance.full_repository_resync('<resync name>'); end;
    
  2. Follow the OMS recovery procedure shown in Single OMS, No Server Load Balancer (SLB), OMS Restored on the same Host

  3. Verify that the site is fully operational.

Distributed Configuration: Incomplete Repository Recovery, Primary OMS and additional OMS on Different Hosts, SLB Configured

The Repository, primary OMS, and additional OMS all reside on the different hosts. Repository database was running in noarchivelog mode. OMS backup file from a recent backup exists (emctl exportconfig oms). Full cold backup of the database exists. All three hosts are lost.

  1. Follow the repository recovery procedure shown in Incomplete Recovery on the Same Host with the following exception:

    Since OMS OracleHome is not yet available and Repository resync has to be initiated before starting an OMS against the restored repository, submit resync via the following PL/SQL block. Log into the repository as SYSMAN using SQLplus and run the following:

    begin emd_maintenance.full_repo*sitory_resync('resync name'); end;
    
  2. Follow the OMS recovery procedure shown in Multiple OMS, Server Load Balancer configured, Primary OMS Recovered on a Different Host with the following exception:

    Override the repository connect description present in the backup file by passing the additional omsca parameter: “-REPOS_CONN_STR <restored repos descriptor>”. This needs to be added along with other parameters listed in Multiple OMS, Server Load Balancer configured, Primary OMS Recovered on a Different Host.

  3. Follow the OMS recovery procedure shown in Multiple OMS, SLB configured, additional OMS recovered on same or different host with the following exception:

    Override the repository connect description and AdminServer details present in the backup file by passing the additional omsca parameters:

    “-REPOS_CONN_STR <restored repos descriptor>” –AS_HOST <recovered admin host> -AS_HTTPS_PORT <recovered admin port>
    

    This must be added along with other parameters listed in Multiple OMS, SLB configured, additional OMS recovered on same or different host.

  4. Verify that the site is fully operational.

EMCTL High Availability Commands

The Enterprise Manager command-line utility (emctl) adds many new commands that allow you to perform requisite backup and recovery operations for all major components.

exportconfig oms

Exports a snapshot of the OMS configuration to the specified directory.

Usage:

emctl exportconfig oms [-sysman_pwd <sysman password>]

      [-dir <backup dir>]     Specify the directory used to store the backup file

      [-keep_host]            Specify to back up hostname if no SLB is defined

                              (Use this option only if recovery will be performed

                               on the machine that responds to this hostname)

importconfig oms

Imports the OMS configuration from the specified backup file.

Usage:

emctl importconfig oms [-sysman_pwd <sysman password>] [-reg_pwd <registration password>]

      -file <backup file>     Required backup file to import from

      [-key_only]             Specify to import emkey only

      [-no_resecure]          Specify not to resecure the oms after import

                              (default is to resecure after import)

config emrep

Configures the OMS and repository target. The command is used to change the monitoring Agent for the target and/or the connection string used to monitor this target.

Usage:

emctl config emrep [-sysman_pwd <sysman password>]

      [-agent <new agent>]    Specify a new destination Agent for emrep target

      [-conn_desc [<jdbc connect descriptor>]]

                      Update the Connect Descriptor with value if specified,

                      else from value stored in the emoms.properties file.

config repos

Configures the repository database target. The command is used to change the monitoring Agent for the target and/or the monitoring properties (hostname, Oracle Home and connection string used to monitor this target).

Usage:

emctl config repos [-sysman_pwd <sysman password>]

      [-agent <new agent>]    Specify new destination agent for repository target

      [-host <new host>]      Specify new hostname for repository target

      [-oh <new oracle home>] Specify new OracleHome for repository target

      [-conn_desc [<jdbc connect descriptor>]]

                       Update the Connect Descriptor with the specified value,

                       else from the value stored in emoms.properties

resync repos

Submits a repository resynchronization operation. When the –full option is specified, all agents are instructed to upload the latest state to the repository. A list of agents can be specified using the –agentlist option to resync with a given list of agents.

Usage:

emctl resync repos (-full|-agentlist "agent names") [-name "resync name"] [-sysman_pwd "sysman password"]

abortresync repos

Aborts the currently running repository resynchronization operation. Use the –full option to stop a full repository resynchronization. Use the –agentlist option to stop resync on a list of agents.

Usage:

emctl abortresync repos (-full|-agentlist "agent names") -name "resync name" [-sysman_pwd "sysman password"]

 

statusresync repos

Lists the status of given repository resynchronization operation.

Usage:

emctl statusresync repos -name "resync name" 

create service

Valid on Windows only. The command creates a service for the Oracle Management Services on Windows. You use this command to manage the Windows service for the OMS on a failover host in a Cold Failover Cluster setup.

Usage:

emctl create service [-user <username>] [-pwd <password>]

      -name <servicename>     Name of service to be created  

delete service

Valid on Windows only. Deletes the service for the Oracle Management Services on Windows.

Usage:

emctl delete service

      -name <servicename>     Name of service to be deleted  

resyncAgent

Resynchronizes a restored or reinstalled Agent by pushing all target configuration from the repository.

Usage:

emcli resyncAgent -agent="Agent Name"

        [-keep_blocked]

PK(n~+*PKOXEOEBPS/notification.htm Notifications

3 Notifications

The notification system allows you to notify Enterprise Manager administrators of alerts, policy violations, and the status changes of job executions. In addition to notifying administrators, the notification system can perform actions such as executing operating system commands (including scripts) and PL/SQL procedures when an alert is triggered. This capability allows you to implement automatically specific IT practices under particular alert conditions. For example, if an alert is generated when monitoring the operational (up/down) status of a database, you may want the notification system to automatically open an in-house trouble-ticket using an OS script so that the appropriate IT staff can respond in a timely manner.

By using Simple Network Management Protocol (SNMP) traps, the Enterprise Manager notification system also allows you to send traps to SNMP-enabled third-party applications such as HP OpenView. Some administrators may want to send third-party applications a notification when a certain metric has exceeded a threshold.

This chapter covers the following:

  • Setting Up Notifications

  • Extending Notification Beyond E-mail

  • Passing Corrective Action Status Change Information

  • Passing Job Execution Status Information

  • Passing User-Defined Target Properties to Notification Methods

  • Assigning Methods to Rules

  • Assigning Rules to Methods

  • Notification Coverage

  • Management Information Base (MIB)

  • Troubleshooting Notifications

Setting Up Notifications

All Enterprise Manager administrators can set up e-mail notifications for themselves. Super Administrators also have the ability to set up notifications for other Enterprise Manager administrators.

Setting Up a Mail Server for Notifications

Before Enterprise Manager can send e-mail notifications, you must first specify the Outgoing Mail (SMTP) servers to be used by the notification system. Once set, you can then define e-mail notifications for yourself or, if you have Super Administrator privileges, other Enterprise Manager administrators.

You specify the Outgoing Mail (SMTP) server on the Notification Methods page (Figure 3-1). Display the Notification Methods page by clicking Setup on any page in the Grid Control console and clicking Notification Methods in the vertical navigation bar.


Note:

You must have Super Administrator privileges in order to set up SMTP servers.

Specify one or more outgoing mail server names, the mail server authentication credentials (User Name, Password, and Confirm Password), if required, the name you want to appear as the sender of the notification messages, and the e-mail address you want to use to send your e-mail notifications. This address, called the Sender's Mail Address, must be a valid address on each mail server that you specify. A message will be sent to this e-mail address if any problem is encountered during the sending of an e-mail notification. Example 3-1 shows sample notification method entries.

Example 3-1 Mail Server Settings

  • Outgoing Mail (SMTP) Server - smtp01.mycorp.com:587, smtp02.mycorp.com

  • User Name - myadmin

  • Password - ******

  • Confirm Password - ******

  • Identify Sender As - Enterprise Manager

  • Sender's E-mail Address - mgmt_rep@mycorp.com

  • Use Secure Connection - No: E-mail is not encrypted. SSL: E-mail is encrypted using the Secure Sockets Layer protocol. TLS, if available: E-mail is encrypted using the Transport Layer Security protocol if the mail server supports TLS. If the server does not support TLS, the e-mail is automatically sent as plain text.

Figure 3-1 Defining a Mail Server

Definition entry fields on Notification Methods page.


Note:

The e-mail address you specify on this page is not the e-mail address to which the notification is sent. You will have to specify the e-mail address (where notifications will be sent) from the Preferences General page.

After configuring the e-mail server, click Test Mail Servers to verify your e-mail setup. You should verify that an e-mail message was received by the e-mail account specified in the Sender's E-mail Address field.

Defining multiple mail servers will improve the reliability of e-mail notification delivery and spread the load across multiple systems. The Management Service makes use of each mail server to send e-mails and the behavior is controlled by the following parameters found in the $ORACLE_HOME/sysman/config/emoms.properties file.

Example 3-2 Management Service Parameters

# The maximum number of emails that can be sent in a single connection to an

# email server

# em.notification.emails_per_connection=20

#

# The maximum number of emails that can be sent in a minute

# em.notification.emails_per_minute=250

Based on the defaults in Example 3-2, the first mail server is used to send 20 e-mails before the Management Service switches to the next mail server which is used to send another 20 e-mails before switching to the next mail server. This prevents one mail server from becoming overloaded and should improve overall reliability and throughput.

Setting Up Repeat Notifications

Repeat notifications allow administrators to be notified repeatedly until an alert is either acknowledged or the number of Maximum Repeat Notifications has been reached. Enterprise Manager supports repeat notification for all notification methods (e-mail, OS command, PL/SQL procedure, and SNMP trap). To enable this feature for a notification method, select the Send Repeat Notifications option. In addition to setting the maximum number of repeat notifications, you can also set the time interval at which the notifications are sent.


Important:

If the Grid Control Repository database version is 9.2, the aq_tm_processes init.ora parameter must be set to at least 1 to enable repeat notification functionality.

Repeat Notifications for Rules

Setting repeat notifications globally at the notification method level may not be provide sufficient flexibility. For example, you may want to have different repeat notification settings based on the metric type and/or alert severity. Enterprise Manager accomplishes this by allowing you to set repeat notifications for individual notification rules. Repeat notifications set at the rule level take precedence over those defined at the notification method level.


Important:

Repeat notifications for rules will only be sent if the Send Repeat Notifications option is enabled in the Notification Methods page.

For PL/SQL, OS command, and SNMP trap notification methods, you must enable each method to support repeat notifications. You can select Supports Repeat Notifications option when adding a new notification method or by editing an existing method.

Figure 3-2 Enabling Repeat Notification for an OS Command Notification Method

Description of Figure 3-2 follows

Setting Up E-mail for Yourself

If you want to receive notifications by e-mail, you will need to specify your e-mail address(s) in the General page under the Preferences link in the Grid Control console. In addition to defining notification e-mail addresses, you associate the notification message format (long or short) to be used for your e-mail address.

Setting up e-mail involves three steps:

Step 1: Define e-mail addresses.

Step 2: Set up a Notification Schedule.

Step 3: Subscribe to receive e-mail for notification rules.

Defining E-mail Addresses

An e-mail address can have up to 128 characters. There is no upper limit with the number of e-mail addresses.

To add an e-mail address:

  1. From the Grid Control console, click Preferences. By default the General page is selected.

  2. Click Add Another Row to create a new e-mail entry field in the E-mail Addresses table.

  3. Specify the e-mail associated with your Enterprise Manager account. All e-mail notifications you receive from Enterprise Manager will be sent to the e-mail addresses you specify.

    For example, user1@oracle.com

    Select the message format for your e-mail address. The Long Format sends a HTML formatted e-mail that contains detailed information. Example 3-3 shows a typical notification that uses the long format.

    The Short Format (Example 3-4) sends a concise, text e-mail that is limited to a configurable number of characters, thereby allowing the e-mail be received as an SMS message or page. The content of the message can be sent entirely in the subject, entirely in the body or split across the subject and body. For example, in the last case, the subject could contain the severity type (for example, Critical) and the target name. The body could contain the time the severity occurred and the severity message. Since the message length is limited, some of this information may be truncated. If truncation has occurred there will be an ellipsis end of the message.

  4. Click Apply to save your e-mail address.

Example 3-3 Long E-mail Notification for Alerts

Name=myhost.com

Type=Host

Host=myhost.com

Metric=Filesystem Space Available (%)

Mount Point =/usr

Timestamp=06-OCT-2006 16:27:05 US/Pacific

Severity=Warning

Message=Filesystem / has only 76.07% available space

Rule Name=Host Availability and Critical States

Rule Owner=SYSMAN

Example 3-4 Short E-mail Notification for Alerts

Subject is :

EM:Unreachable Start:myhost

Body is :

Nov 16, 2006 2:02:19 PM EST:Agent is Unreachable (REASON = Connection refused) 

but the host is UP

More about Short E-mail Format

Enterprise Manager does not directly support message services such as paging or SMS, but instead relies on external gateways to, for example, perform the conversion from e-mail to page.

Entries in the emoms.properties file define the size and format of the short e-mail.

You must establish the maximum size your device can support and whether the message is sent in subject, body or both.

emoms.properties Entries for a Short E-mail Format

# The maximum size of a short format email

# em.notification.short_format_length=155

# The format of the short email. It can be set to subject, body or both.

#

# When set to subject the entire message is sent in the subject i.e.

#  EM:<severity>:<target>:<message>:<timestamp>

# When set to body the entire message is sent in the body i.e.

#  EM:<severity>:<target>:<message>:<timestamp>

# When set to both the message is split i.e. the subject contains

#  EM:<severity>:<target>

# and the body contains

#  <message>:<timestamp>

# In all cases the message is truncated to the length specified in the

# em.notification.short_format_length parameter

# em.notification.short_format=both

Setting Up a Notification Schedule

Once you have defined your e-mail notification addresses, you will need to define a notification schedule. For example, if your e-mail addresses are user1@oracle.com, user2@oracle.com, user3@oracle.com, you can choose to use one or more of these e-mail addresses for each time period in your notification schedule.


Note:

When you enter e-mail addresses for the first time, a 24x7 weekly notification schedule is set automatically. You can then review and modify the schedule to suit your monitoring needs.

A notification schedule is a repeating schedule used to specify your on-call schedule—the days and time periods and e-mail addresses that should be used by Enterprise Manager to send notifications to you. Each administrator has exactly one notification schedule. When a notification needs to be sent to an administrator, Enterprise Manager consults that administrator's notification schedule to determine the e-mail address to be used. Depending on whether you are Super Administrator or a regular Enterprise Manager administrator, the process of defining a notification schedule differs slightly.

If you are a regular Enterprise Manager administrator and are defining your own notification schedule:

  1. From the Enterprise Manager Grid Control, click Preferences at the top of the page. By default the General page is selected.

  2. Click Notification Schedule in the vertical navigation bar. Your Notification Schedule page appears.

  3. Follow the directions on the Notification Schedule page to specify when you want to receive e-mails.

Subscribe to Receive E-mail for Notification Rules

A notification rule is a user-defined rule that defines the criteria by which notifications should be sent for alerts, policy violations, corrective action execution status, and job execution status. Specifically, in each rule, you can specify the criteria you are interested in and the notification methods (such as e-mail) that should be used for sending these notifications. For example, you can set up a rule that when any database goes down or any database backup job fails, e-mail should be sent and the "log trouble ticket" notification method should be called. Or you can define another rule such that when the CPU or Memory Utilization of any host reach critical severities, SNMP traps should be sent to another management console. During notification rule creation, you specify criteria such as the targets you are interested in, their monitored metrics, associated alert severity conditions (clear, warning, critical), policy violations, corrective action execution status, or job execution status, and the associated notification method.

To subscribe to a notification rule you create, while creating the rule, go to the Actions page and check the Send Me E-mail option.

Out-of-Box Notification Rules

Enterprise Manager Grid control comes with out-of-box notification rules that cover the most common alert conditions. When you install the Oracle Management Service, you are given the option to receive e-mail notifications for critical alerts. If you choose this option, and if an e-mail address for the SYSMAN user was specified, then some default notification rules are created that cover the availability and critical states for common target types and would also be configured to send e-mail notifications to the SYSMAN e-mail address for the conditions defined in the notification rules.

You can access the out-of-box notification rules by clicking on Preferences on any page in the Enterprise Manager console and clicking Public Rules in the vertical navigation bar. If the conditions defined in the out-of-box notification rules meet your requirements, you can simply subscribe to receive e-mail notifications for the conditions defined in the rule by clicking on Subscribe column in the row of the Public Rules table that corresponds to the notification rule that you are interested in. Click Apply to save your changes.

Table 3-1 lists all default notification rules. These are all owned by the SYSMAN user and are public rules.

Table 3-1 Default Notification Rules

NameDescriptionApplies to Targets of the TypeSend Notification on the Following Availability StatesSend Notification if Any of the Metrics is at CRITICAL Alert Severity

Agent Upload Problems

System-generated notification rule for monitoring Agents that may have problems uploading data to the Management Service.

Oracle Management Service and Repository

N/A

Count of targets not uploading data

Agents Unreachable

System-generated notification rule for monitoring Agents that lose contact with the Management Service due to network problems, host problems or Agents going down.

Agents

Agent UnreachableAgent Unreachable Resolved

N/A

Application Server Availability and Critical States

System-generated notification rule for monitoring Application Servers' availability, and critical metric statuses.

Application Servers

Down

CPU Usage (%)

Database Availability and Critical States

System-generated notification rule for monitoring Databases' availability, and critical metric statuses.

Databases (single instance only)

Down

Process Limit Usage (%)

Session Limit Usage (%)

Blocking Session Count All Objects

Archiver Hung Alert Log Error Status

Data Block Corruption Alert Log Error Status

Generic Alert Log Error Status

Media Failure Alert Log Error Status

Session Terminated Alert Log Error Status

Archive Area Used (%) All Objects

Segments Not Able to Extend Count All Objects

Segments Approaching Maximum Extents Count All Objects

Tablespace Space Used (%) All Objects

Wait Time (%)

HTTP Server Availability and Critical States

System-generated notification rule for monitoring HTTP Server's availability, and critical metric statuses.

Oracle HTTP Server

Down

CPU Usage (%)

Percentage of Busy Processes

Active HTTP Connections

Request Processing Time (seconds)

Host Availability and Critical States

System-generated notification rule for monitoring Hosts' availability, and critical metric statuses.

Hosts

Agent Unreachable

Agent Unreachable Resolved

Average Disk I/O Service Time (ms)

Disk Device Busy (%)

Filesystem Space Available (%)

CPU in I/O Wait (%)

Run Queue Length (5 minute average)

CPU Utilization (%)

Memory Utilization (%)

Memory Page Scan Rate (per second)

Swap Utilization (%)

Network Interface Combined Utilization (%)

Listener Availability

System-generated notification rule for monitoring database Listeners' availability, and critical metric statuses.

Listeners

Down

N/A

Misconfigured Agents

System-generated notification rule for misconfigured agents.

Agent

Agent Unreachable

Agent Unreachable Resolved

Consecutive severity upload failure count Consecutive heartbeat failure count

MS Agent time skew (mins)Consecutive metadata upload failure count

OC4J Availability and Critical States

System-generated notification rule for monitoring OC4J instance's availability, and critical metric statuses.

OC4J

Down

CPU Usage (%)

OC4J Instance - Request Processing Time (seconds)

OC4J Instance - Active Sessions

OMS Service Initialization Errors

System-generated notification rule for monitoring OMS service initialization errors.

OMS and Repository

N/A

Service Status

PAF Status Notification

System-generated notification rule for Provisioning Advisor Framework: Notifies the instance creator of any status updates.

N/A

Up

Down

Corrective Actions on Target Down

Agent Unreachable

Agent Unreachable Resolved

Metric Error Detected

Metric Error ResolvedBlackout Started

Blackout Ended

N/A

Repository Operations Availability

System-generated notification rule for monitoring the availability of the DBMS jobs that are part of the Management Repository.

OMS and Repository

Critical

DBMS Job UpDown

Violation Notification for Database Security Policies

System-generated notification rule for monitoring the secureness of the database configuration.

Databases

Critical

N/A

Web Cache Availability and Critical States

System-generated notification rule for monitoring Web Cache's instance's availability, and critical metric statuses.

Oracle Web Cache

Down

Hits (% of requests)

Web Cache CPU Usage (%)


Creating Your Own Notification Rules

If you find that the default notification rules do not meet your needs, you can define your own custom rules. The following procedure documents the process of notification rule creation for non-Super Administrators.

To create your own notification rule:

  1. From the Enterprise Manager Grid Control, click Preferences.

  2. Click My Rules in the vertical navigation bar.

    If you are not logged in as an administrator with Super Administrator privileges, you will see a link for My Rules instead of Rules as in the case of an administrator with Super Administrator privileges.

  3. Click Create.

    Enterprise Manager displays the Create Notification Rule pages. Enter the requisite information on each page to create your notification rule.

    When you specify the notification rule properties, check Make Public in the General page if you want other non-privileged users to be able to view and share that rule. For example, it allows other administrators to later specify that they should receive e-mail for this rule.

    When you specify the notification rule, you will only be able to choose from e-mail and SNMP traps. Specifying custom commands and PL/SQL procedures is an option that is only available to Super Administrators. To receive e-mail notifications for conditions defined in the rule, go to the Actions page and check the Send Me E-Mail option.

Specifying Additional Alert Duration Criteria

You can set additional alert duration criteria for a notification rule and have the rule apply only to alerts that have been open for at least a certain amount of time and have not been acknowledged. These criteria apply only to Target Down, Agent Unreachable, Metric alerts, Policy Violations, Blackout Started and Metric Error Start alerts.

Typical scenarios where you would use additional alert criteria are:

  • For log alerts that have been open for at least 7 days, clear the alerts

  • For alerts that have been open for at least 48 hours and have not been acknowledged, send e-mail to the DBA Manager

To specify additional alert duration criteria:

  1. Create or edit a notification rule. Additional alert criteria can be added from the Availability, Metrics, or Policy tab.

  2. From one of the aforementioned tabs, go to the Additional Alert Criteria section and click Add. The Additional Alert Criteria page appears.

  3. Specify the alert duration criteria and click Continue.

Setting Up E-mail for Other Administrators

If you have Super Administrator privileges, you can set up e-mail notifications for other Enterprise Manager administrators. To set up e-mail notifications for other Enterprise Manager administrators, you need to:

Step 1: Ensure Each Administrator Account has an Associated E-mail Address

Each administrator to which you want to send e-mail notifications must have a valid e-mail address.

  1. Click Setup.

  2. Click Administrators from the vertical navigation bar.

  3. For each administrator, define an e-mail address. This sets up a 24x7 notification schedule for this user that uses all the e-mail addresses specified.

Enterprise Manager also allows you to specify an administrator address when editing an administrator's notification schedule.

Step 2: Define Administrators' Notification Schedules

Once you have defined e-mail notification addresses for each administrator, you will need to define their respective notification schedules. Although a default 24x7 notification schedule is created when you specify an e-mail address for the first time, you should review and edit the notification schedule as needed.

  1. Click Setup.

  2. From the vertical navigation bar, click Schedules (under Notification). The Notification Schedule page appears.

  3. Specify the administrator who's notification schedule you wish to edit and click Change.

  4. Click Edit Schedule Definition. The Edit Schedule Definition: Time Period page appears. If necessary, modify the rotation schedule.

  5. Click Continue. The Edit Schedule Definition: E-mail Addresses page appears.

  6. Follow the directions on the Edit Schedule Definition: E-mail Addresses page to modify the notification schedule.

  7. Click Finish when you are done.

  8. Repeat steps three through seven for each administrator.

Step 3: Assign Notification Rules to Administrators

With the notification schedules set, you now need to assign the appropriate notification rules for each designated administrator.

  1. Click Setup.

  2. From the vertical navigation bar, click Administrators.

  3. Select the desire administrator.

  4. Click Subscribe to Rules. The Subscribe <administrator> to Public Notification Rules page appears.

  5. Select the desired notification rules and click Subscribe.

  6. Click OK when you are finished.

  7. Repeat steps three through six for each administrator.

E-mail Customization

Enterprise Manager allows Super Administrators to customize global e-mail notifications for the four alert types (Metric Alert, Target Availability, Policy Violation, and Job Status Change). Using predefined building blocks (called attributes and labels) contained within a simple script, Super Administrators can customize alert e-mails by selecting from a wide variety of information content.

To customize an e-mail:

  1. Access the E-mail Customization page. Setup-->E-mail Customization

  2. Choose the Alert Type and Format.

  3. Click Edit. The Edit E-mail Template page is displayed.

From the Edit E-mail Template page, you can modify the content of the e-mail template Enterprise Manager uses to generate e-mail notifications. Extensive information on script formatting, syntax, and options is available from the Edit E-mail Template page via imbedded assistance and online help.

Figure 3-3 E-mail Customization

Description of Figure 3-3 follows

E-mail Customization Reference

The following reference summarizes the semantics and component syntax of the pseudo-language used to define e-mails. The pseudo-language provides you with a simple, yet flexible way to customize e-mail notifications. The following is a summary of pseudo-language conventions/limitations:

  • You can add comments (or any free-form text) using separate lines beginning with "--" or at end of lines.

  • You can use attributes.

  • You can use IF & ELSE & ENDIF control structures. You can also use multiple conditions using "AND" or "OR". Nested IF statements are not supported.

  • You can insert spaces for formatting purposes. Spaces at the beginning of a line will be ignored in the actual e-mail. To insert spaces at the beginning of a line, use the [SP] attribute.

  • Use "/" to escape and "[" or "]" if you want to add attribute names, operators, or IF clauses to the actual e-mail.

  • HTML is not supported.

Reserved Words and Operators

The following table lists all reserved words and operators used when modifying e-mail scripts.

Table 3-2 Reserved Words and Operators

Reserved Word/OperatorDescription

IF, ELSIF, ENDIF, ELSE

Used in IF-ELSE constructs.

AND, OR

Boolean operators – used in IF-ELSE constructs only.

NULL

To check NULL value for attributes - used in IF-ELSE constructs only.

|


Pipe operator – used to show the first non-NULL value in a list of attributes.

For example:

METRIC_NAME|POLICY_NAME

EQ, NEQ

Equal and Not-Equal operators – applicable to NULL, STRING and NUMERIC values.

/


Escape character – used to escape reserved words and operators. Escape characters signify that what follows the escape character takes an alternative interpretation.

[ , ]

Delimiters used to demarcate attribute names and IF clauses.


Syntax Elements

Literal Text

You can specify any text as part of the e-mail content. The text will be displayed in the e-mail and will not be translated if the Oracle Management Services (OMS) language setting is changed. For example, 'my Oracle Home' appears as 'my Oracle Home' in the generated e-mail.

Predefined Attributes

Predefined attributes/labels will be substituted with actual values in a specific context. To specify a predefined attribute/label, use the following syntax:

[PREDEFINED_ATTR]

Attribute names can be in either UPPER or LOWER case. The parsing process is case-insensitive.

A pair of square brackets is used to distinguish predefined attributes from literal text. For example, for a job e-mail notification, the actual job name will be substituted for [JOB_NAME]. For a metric e-mail notification, the actual metric column name will be substituted for [METIRC_COLUMN].

You can use the escape character “/” to specify words and not have them interpreted as predefined labels/attributes. For example, "/[NEW/]” will not be considered as the predefined attribute [NEW] when parsed.

Operators

EQ, NEQ – for text and numeric values

NULL- for text and numeric values

GT, LT, GE, LE – for numeric values

Control Structures

The following table lists acceptable script control structures.

Table 3-3 Control Structures

Control StructureDescription

Pipe "|"

Two or more attributes can be separated by '|' character. For example,

[METRIC_NAME|POLICY_NAME]

In this example, only the applicable attribute within the current alert context will be used (replaced by the actual value) in the e-mail. If more than one attributes are applicable, only the left-most attribute is used.

IF

Allows you to make a block of text conditional. Only one level of IF and ELSIF is supported. Nested IF constructs are not supported.

All attributes can be used in IF or ELSIF evaluation using EQ/NEQ operators on NULL values. Other operators are allowed for “SEVERITY” and “REPEAT_COUNT” only.

Inside the IF block, the values need to be contained within quotation marks “ ”. Enterprise Manager will extract the attribute name and its value based on the position of “EQ” and other key words such as “and”, “or”. For example,

[IF REPEAT_COUNT EQ “1” AND SEVERITY EQ “CRITICAL” THEN]

The statement above will be true when the attributes of the alert match the following condition:

  • Attribute Name: REPEAT_COUNT

  • Attribute Value: 1

  • Attribute Name: SEVERITY

  • Attribute Value: CRITICAL

Example IF Block:

[IF JOB_NAME NEQ NULL]

       [JOB_NAME_LABEL]=[JOB_NAME] 

       [JOB_OWNER_LABEL]=[JOB_OWNER] 

[ENDIF] 

 

[IF SEVERITY EQ CRITICAL ]

       [MTRIC_NAME_LABEL]=[METRIC_NAME] 

       [METRIC_VALUE_LABEL]=[METRIC_VALUE] 

       [TARGET_NAME_LABEL]=[TARGET_NAME] 

       [KEY_VALUES] 

[ENDIF] 

Example IF and ELSEIF Block:

[IF SEVERITY EQ CRITICAL]           statement1[ELSIF SEVERITY EQ WARNING]           statement2[ELSIF SEVERITY EQ CLEAR]           statement3[ELSE]           statement4[ENDIF]


Comments

You can add comments to your script by prefacing a single line of text with two hyphens "--". For example,

 -- Code added on 8/3/2009     [IF REPEAT_COUNT NEQ NULL]     . . . 

Comments may also be placed at the end of a line of text.

[IF SEVERITY_SHORT EQ W] -- for Warning alert

HTML Tags in Customization Content

Use of HTML tags is not supported.

When Enterprise Manager parses the e-mail script, it will convert the “<” and “>” characters of HTML tags into encoded format (&lt; and &gt;). This ensures that the HTML tag is not treated as HTML by the destination system.

Examples

E-mail customization template scripts support three main operators.

  • Comparison operators: EQ/NEQ/GT/LT/GE/LELogic operators: AND/ORPipeline operator: |

Extending Notification Beyond E-mail

Notification Methods are the mechanisms by which alerts are sent. Enterprise Manager Super Administrators can set up e-mail notifications by configuring the 'e-mail' notification method. Most likely this would already have been setup as part of the Oracle Management Service installation.Enterprise Manager Super Administrators can also define other custom notification methods. For example, alerts may need to be forwarded to a 3rd party trouble-ticketing system. Assuming APIs to the third-party trouble-ticketing system are available, a custom notification method can be created to call a custom OS script that has the appropriate APIs. The custom notification method can be named in a user-friendly fashion, for example, "Log trouble ticket". Once that is defined, any time an administrator needs to send alerts to the trouble-ticketing system, he merely needs to invoke the now globally available notification method called "Log trouble ticket". Custom notification methods can be defined based on any custom OS script, any custom PL/SQL procedure, or by sending SNMP traps. A fourth type of notification method (Java Callback) exists to support Oracle internal functionality and cannot be created or edited by Enterprise Manager administrators.

Only Super Administrators can define OS Command, PL/SQL, and SNMP Trap notification methods. However, any Enterprise Manager administrator can add these notification methods (once defined by the Super Administrator) to their notification rules.

Through the Notification Methods page, you can:

  • Set up the outgoing mail servers if you plan to send e-mail notifications through notification rules

  • Create other custom notification methods using OS and PL/SQL scripts and SNMP traps.

  • Set global repeat notifications.

Custom Notification Methods Using Scripts and SNMP Traps

You can create other custom notification methods based on OS scripts, PL/SQL procedures, or SNMP traps. Any administrator can then use these methods in Notification Rules.

Adding a Notification Method based on an OS Command or Script

Complete the following four steps to define a notification method based on an OS command or script.


Note:

Notification methods based on OS commands must be configured by an administrator with Super Administrator privileges.

Step 1: Define your OS command or script.

You can specify an OS command or script that will be called by the notification system. You can use target and alert or policy violation context information, corrective action execution status and job execution status within the body of the script. Passing metric severity attributes (severity level, type, notification rule, rule owner, or rule owner, and so on) or policy violation information to OS commands/scripts allows you to customize automated responses to alerts or policy violations. For example, if an OS script opens a trouble ticket for an in-house support trouble ticket system, you will want to pass severity levels (critical, warning, and so on) to the script to open a trouble ticket with the appropriate details and escalate the problem. For more information on passing specific types of information to OS Command or Scripts, see

Step 2: Deploy the script on each Management Service host.

You must deploy the OS Command or Script on each Management Service host machine that connects to the Management Repository. The OS Command is run as the user who started the Management Service.

The OS Command or Script should be deployed on the same location on each Management Service host machine. The OS Command should be an absolute path, for example, /u1/bin/logSeverity.sh. The command is run by the user who started the Management Service. If an error is encountered during the running of the OS Command, the Notification System can be instructed to retry the sending of the notification to the OS Command by returning an exit code of 100. The procedure is initially retried after one minute, then two minutes, then three minutes and so on, until the notification is a day old, at which point it will be purged.

Example 3-5 shows the parameter in emoms.properties that controls how long the OS Command can execute without being killed by the Management Service. This is to prevent OS Commands from running for an inordinate length of time and blocking the delivery of other notifications. By default the command is allowed to run for 30 seconds before it is killed.

Example 3-5 Parameter in emoms.properties File

# The amount of time in seconds after which an OS Command started by the

# Notification System will be killed if it has not exited

# em.notification.os_cmd_timeout=30

Step 3: Register your OS Command or Script as a new Notification Method.

Add this OS command as a notification method that can be called in Notification Rules. Log in as a Super Administrator, click Setup and then Notification Methods from the vertical navigation bar. From this page, you can define a new notification based on the 'OS Command' type. See "Adding a Notification Method based on an OS Command or Script" .

The following information is required for each OS command notification method:

  • Name

  • Description

    Both Name and Description should be clear and intuitive so that the function of the method is clear to other administrators.

  • OS Command

You must enter the full path of the OS command or script in the OS command field (for example, /u1/bin/myscript.sh). For environments with multiple Management Services, the path must be exactly the same on each machine that has a Management Service. Command line parameters can be included after the full path (for example, /u1/bin/myscript.sh arg1 arg2).

Example 3-6 shows information required for the notification method.

Example 3-6 OS Command Notification Method

Name Trouble Ticketing

Description Notification method to log trouble ticket for a severity occurrence

OS Command /private/mozart/bin/logTicket.sh


Note:

There can be more than one OS Command configured per system.

Step 4: Assign the notification method to a rule.

You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a single rule, see "Assigning Rules to Methods" or "Assigning Methods to Rules".

Passing Alert and Policy Violation Information to an OS Command or Script

The notification system passes severity information to an OS script or executable using system environment variables.

Conventions used to access environmental variables vary depending on the operating system:

  • UNIX: $ENV_VARIABLE

  • Windows:%ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.

Table 3-4 Environment Variables

Environment VariableDescription

TARGET_NAME

Name of the target on which the severity occurred.

TARGET_TYPE

Type of target on which the severity occurred. Targets are defined as any monitorable entity, such as Host, Database, Listener, or Oracle HTTP Server. You can view the type of a monitored target on the All Targets page.

HOST

Name of the machine on which the target resides.

METRIC

Metric generating the severity. This variable is not set for policy violations.

METRIC_VALUE

The value of the metric when the threshold was exceeded. Not set for policy violations

CYCLE_GUID

A unique identifier for a metric alert cycle, which starts from the time the metric alert is initially generated until the time it is clear.

POLICY_RULE

The name of the policy when the threshold was exceeded. Not set for metric severities

KEY_VALUE

For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity.

KEY_VALUE_NAME

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

VIOLATION_CONTEXT

A comma-separated list of name-value pairs that shows the alert context for a policy violation.

TIMESTAMP

Time when the severity occurred.

SEVERITY

Type of severity. For example, severity for a target's (availability) status metric are:

  • UP

  • DOWN

  • UNREACHABLE CLEAR

  • UNREACHABLE START

  • BLACKOUT END

  • BLACKOUT START

Other metrics can have any of the following severities:

  • WARNING

  • CRITICAL

  • CLEAR

  • METRIC ERROR CLEAR

  • METRIC ERROR START

MESSAGE

Message for the alert that provides details about what triggered the condition.

RULE_NAME

Name of the notification rule to which the OS Command notification method was assigned.

RULE_OWNER

Name of the Enterprise Manager administrator who owns the notification rule.


Your script may reference some or all of these variables.

The sample OS script shown in Example 3-7 appends environment variable entries to a log file. In this example, the script logs a severity occurrence to a file server. If the file server is unreachable then an exit code of 100 is returned to force the Oracle Management Service Notification System to retry the notification

Example 3-7 Sample OS Command Script

#!/bin/ksh


LOG_FILE=/net/myhost/logs/severity.log

if test -f $LOG_FILE

then

echo $TARGET_NAME $MESSAGE $TIMESTAMP >> $LOG_FILE

else

   exit 100

fi

Example 3-8 shows an OS script that logs alert information to the file 'alertmsg.txt'. The file is saved to the /u1/results directory.

Example 3-8 Alert Logging Script

#!/usr/bin/sh

echo "Alert logged:" > /u1/results/alertmsg.txt

echo "\n" >> /u1/results/alertmsg.txt

echo "target name is " $TARGET_NAME >> /u1/results/alertmsg.txt

echo "target type is " $TARGET_TYPE >> /u1/results/alertmsg.txt

echo "target is on host " $HOST &gt;> /u1/results/alertmsg.txt

echo "metric in alert is " $METRIC >> /u1/results/alertmsg.txt

echo "metric index is " $KEY_VALUE >> /u1/results/alertmsg.txt

echo "timestamp is " $TIMESTAMP >> /u1/results/alertmsg.txt

echo "severity is " $SEVERITY >> /u1/results/alertmsg.txt

echo "message is " $MESSAGE >> /u1/results/alertmsg.txt

echo "notification rule is " $RULE_NAME >> /u1/results/alertmsg.txt

echo "rule owner is " $RULE_OWNER >> /u1/results/alertmsg.txt

exit 0

Example 3-9 shows a script that sends an alert to an HP OpenView console from Enterprise Manager Grid Control. When a metric alert is triggered, the Enterprise Manager Grid Control displays the alert. The HP OpenView script is then called, invoking opcmsg and forwarding the information to the HP OpenView management server.

Example 3-9 HP OpenView Script

/opt/OV/bin/OpC/opcmsg severity="$SEVERITY" app=OEM msg_grp=Oracle msg_text="$MESSAGE" object="$TARGET"

Adding a Notification Method Based on a PL/SQL Procedure

Complete the following four steps to define a notification method based on a PL/SQL procedure.

Step 1: Define the PL/SQL procedure.

The procedure must have one of the following signatures depending on the type of notification that will be received.

For alerts and policy violations:

PROCEDURE p(severity IN MGMT_NOTIFY_SEVERITY)

For job execution status changes:

PROCEDURE p(job_status_change IN MGMT_NOTIFY_JOB)

For corrective action status changes:

PROCEDURE p(ca_status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION)


Note:

The notification method based on a PL/SQL procedure must be configured by an administrator with Super Administrator privileges before a user can select it while creating/editing a notification rule.

For more information on passing specific types of information to scripts or PL/SQL procedures, see the following sections:

"Passing Alert and Policy Violation Information to a PL/SQL Procedure"

"Passing Corrective Action Status Change Information"

"Passing Job Execution Status Information"

Step 2: Create the PL/SQL procedure on the Management Repository.

Create the PL/SQL procedure on the repository database using one of the following procedure specifications:

PROCEDURE p(severity IN MGMT_NOTIFY_SEVERITY)

PROCEDURE p(job_status_change IN MGMT_NOTIFY_JOB)

PROCEDURE p(ca_status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION)

The PL/SQL procedure must be created on the repository database using the database account of the repository owner (such as SYSMAN)

If an error is encountered during the running of the procedure, the Notification System can be instructed to retry the sending of the notification to the procedure by raising a user defined exception that uses the error code -20000. See Example 3-11, "PL/SQL Procedure Using a Severity Code". The procedure initially retried after one minute, then two minutes, then three minutes and so on, until the notification is a day old, at which point it will be purged.

Step 3: Register your PL/SQL procedure as a new notification method.

Log in as a Super Administrator, click Setup and then Notification Methods from the vertical navigation bar. From this page, you can define a new notification based on 'PL/SQL Procedure'. See "Adding a Notification Method Based on a PL/SQL Procedure".

Make sure to use a fully qualified name that includes the schema owner, package name and procedure name. The procedure will be executed by the repository owner and so the repository owner must have execute permission on the procedure.

Create a notification method based on your PL/SQL procedure. The following information is required when defining the method:

  • Name

  • Description

  • PL/SQL Procedure

You must enter a fully qualified procedure name (for example, OWNER.PKGNAME.PROCNAME) and ensure that the owner of the Management Repository has execute privilege on the procedure.

An example of the required information is shown in Example 3-10.

Example 3-10 PL/SQL Procedure Required Information

Name Open trouble ticket

Description Notification method to open a trouble ticket in the event

PLSQL Procedure ticket_sys.ticket_ops.open_ticket

Step 4: Assign the notification method to a rule.

You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a single rule, see "Assigning Rules to Methods" or "Assigning Methods to Rules".

There can be more than one PL/SQL-based method configured for your Enterprise Manager environment.

Information about the severity types that relate to a target's availability, and how metric severity and policy violation information is passed to the PLSQL procedure is covered in the next section.

Passing Alert and Policy Violation Information to a PL/SQL Procedure

Passing metric severity attributes (severity level, type, notification rule, rule owner, or rule owner, and so on) or policy violation information to PL/SQL procedures allows you to customize automated responses to alerts or policy violations.

The notification system passes information about metric severities or policy violations to a PL/SQL procedure using the MGMT_NOTIFY_SEVERITY object. An instance of this object is created for every alert or policy violation. When an alert or policy violation occurs, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_SEVERITY object that has been passed to it.

The following table lists all metric severity attributes that can be passed:

Table 3-5 Metric Severity Attributes

AttributeDatatypeAdditional Information

TARGET_NAME

VARCHAR2(256)

Name of the target on which the severity occurred.

TARGET_TYPE

VARCHAR2(64)

Type of target on which the severity occurred. Targets are defined as any monitorable service.

TIMEZONE

VARCHAR2(64)

The target's regional timezone

HOST_NAME

VARCHAR2(128)

Name of the machine on which the target resides.

METRIC_NAME

VARCHAR2(64)

Metric or policy generating the severity.

METRIC_DESCRIPTION

VARCHAR2(128)

Meaningful description of the metric that can be understood by other administrators.

METRIC_COLUMN

VARCHAR2(64)

For table metrics, the metric column contains the name of the column in the table that is being defined. If the metric that is being defined is not a table metric, the value in this column is a single space. This attribute is not used for policy violations.

METRIC_VALUE

VARCHAR2(1024)

The value of the metric.

KEY_VALUE

VARCHAR2(1290)

For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity.

KEY_VALUE_NAME

VARCHAR2(512)

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

KEY_VALUE_GUID

VARCHAR2(256)

GUID associated with a composite key value name.

CTXT_LIST

MGMT_NOTIFY_COLUMNS

Details of the alert context.

COLLECTION_TIMESTAMP

DATE

The time when the target status change was last detected and logged in the management repository.

SEVERITY_CODE

NUMBER

Numeric code identifying the severity level. See Severity Code table below.

MESSAGE

VARCHAR2(4000)

An optional message that is generated when the alert is created that provides additional information about the alert condition.

SEVERITY_GUID

RAW(16)

Severity global unique identifier.

METRIC_GUID

RAW(16)

Metric global unique identifier.

TARGET_GUID

RAW(16)

Target global unique identifier.

RULE_OWNER

VARCHAR2(64)

Name of the Enterprise Manager administrator who owns the rule.

RULE_NAME

VARCHAR2(132)

Name of the notification rule that resulted in the severity.


When a severity occurs for the target, the notification system creates an instance of the MGMT_NOTIFY_SEVERITY object and populates it with values from the severity. The severity codes in Table 3-6 have been defined as constants in the MGMT_GLOBAL package and can be used to determine the type of severity in the severity_code field of the MGMT_NOTIFY_SEVERITY object.

Table 3-6 Severity Codes

NameDatatypeValue

G_SEVERITY_COMMENT

NUMBER(2)

10

G_SEVERITY_CLEAR

NUMBER(2)

15

G_SEVERITY_WARNING

NUMBER(2)

20

G_SEVERITY_CRITICAL

NUMBER(2)

25

G_SEVERITY_UNREACHABLE_CLEAR

NUMBER(3)

115

G_SEVERITY_UNREACHABLE_START

NUMBER(3)

125

G_SEVERITY_BLACKOUT_END

NUMBER(3)

215

G_SEVERITY_BLACKOUT_START

NUMBER(3)

225

G_SEVERITY_ERROR_END

NUMBER(3)

315

G_SEVERITY_ERROR_START

NUMBER(3)

325

G_SEVERITY_NO_BEACONS

NUMBER(3)

425

G_SEVERITY_UNKNOWN

NUMBER(3)

515


Example 3-11 PL/SQL Procedure Using a Severity Code

CREATE TABLE alert_log (target_name VARCHAR2(64),

alert_msg VARCHAR2(4000),

occured DATE);


PROCEDURE LOG_CRITICAL_ALERTS(severity IN MGMT_NOTIFY_SEVERITY)

IS

BEGIN

-- Log all critical severities

   IF severity.severity_code = MGMT_GLOBAL.G_SEVERITY_CRITICAL

   THEN

         BEGIN

                 INSERT INTO alert_log (target_name, alert_msg, occured)

                 VALUES (severity.target_name, severity.message,

                                 severity.collection_timestamp);

         EXCEPTION

         WHEN OTHERS

         THEN

         -- If there are any problems then get the notification retried

                 RAISE_APPLICATION_ERROR(-20000, 'Please retry');

                 END;

                 COMMIT;

   END IF;

END LOG_CRITICAL_ALERTS;

Adding a Notification Method Based on an SNMP Trap

Enterprise Manager supports integration with third-party management tools through the SNMP. For example, you can use SNMP to notify a third-party application that a selected metric has exceeded its threshold.

The trap is an SNMP Version 1 trap and is described by the MIB definition shown at the end of this chapter. See "Management Information Base (MIB)".

For more comprehensive configuration information, see the documentation specific to your platform; SNMP configuration differs from platform to platform.


Note:

Notification methods based on SNMP traps must be configured by an administrator with Super Administrator privileges before any user can then choose to select one or more of these SNMP trap methods while creating/editing a notification rule.

Step 1: Define a new notification method based on an SNMP trap.

Log in to Enterprise Manager as a Super Administrator. Click Setup and then click Notification Method from the vertical navigation bar to access the Notification Methods page. From this page you can add a new method based on an SNMP trap.

You must provide the name of the host (machine) on which the SNMP master agent is running and other details as shown in the following example. In Example 3-12, the SNMP host will receive your SNMP traps.

Example 3-12 SNMP Trap Required Information

Name HP OpenView Console

Description Notification method to send trap to HP OpenView console

SNMP Trap Host Name machine1.us.oracle.com

SNMP Host Port 162

SNMP Community public

This SNMP host will receive your SNMP traps.


Note:

A Test Trap button exists for you to test your setup.

Metric severity information will be passed as a series of variables in the SNMP trap.

An example SNMP Trap is shown in Example 3-13. Each piece of information is sent as a variable embedded in the SNMP Trap.

Example 3-13 SNMP Trap

Tue Oct 28 05:00:02 2006


Command: 4

   Enterprise: 1.3.6.1.4.1.111.15.2

   Agent: 138.1.6.200

   Generic Trap: 6

   Specific Trap: 1

   Time Stamp: 8464:39.99

   Count: 11


Name: 1.3.6.1.4.1.111.15.1.1.1.2.1

   Kind: OctetString

   Value: "mydatabase"


Name: 1.3.6.1.4.1.111.15.1.1.1.3.1

   Kind: OctetString

   Value: "Database"


Name: 1.3.6.1.4.1.111.15.1.1.1.4.1

  Kind: OctetString

  Value: "myhost.com"


Name: 1.3.6.1.4.1.111.15.1.1.1.5.1

   Kind: OctetString

   Value: "Owner's Invalid Object Count"


Name: 1.3.6.1.4.1.111.15.1.1.1.6.1

   Kind: OctetString

   Value: "Invalid Object Owner"


Name: 1.3.6.1.4.1.111.15.1.1.1.7.1

   Kind: OctetString

   Value: "SYS"


Name: 1.3.6.1.4.1.111.15.1.1.1.8.1

   Kind: OctetString

   Value: "28-OCT-2006 04:59:10 (US/Eastern GMT)"


Name: 1.3.6.1.4.1.111.15.1.1.1.9.1

   Kind: OctetString

   Value: "Warning"


Name: 1.3.6.1.4.1.111.15.1.1.1.10.1

   Kind: OctetString

   Value: "12 object(s) are invalid in the SYS schema."


Name: 1.3.6.1.4.1.111.15.1.1.1.11.1

   Kind: OctetString

   Value: "Database Metrics"


Name: 1.3.6.1.4.1.111.15.1.1.1.12.1

   Kind: OctetString

   Value: "SYSMAN"

Step 2: Assign the notification method to a rule.

You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a rule, see "Assigning Methods to Rules" or "Assigning Rules to Methods".

Passing Corrective Action Status Change Information

Passing corrective action status change attributes (such as new status, job name, job type, notification rule, or rule owner) to PL/SQL procedures or OS commands/scripts allows you to customize automated responses to status changes. For example, you many want to call an OS script to open a trouble ticket for an in-house support trouble ticket system if a critical corrective action fails to run. In this case, you will want to pass status (for example, Problems or Aborted) to the script to open a trouble ticket and escalate the problem.

Passing Corrective Action Execution Status to an OS Command or Script

The notification system passes information to an OS script or executable via system environment variables. Conventions used to access environmental variables vary depending on the operating system:

  • UNIX: $ENV_VARIABLE

  • MS Windows: %ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.

Table 3-7 Environment Variables

Environment VariableDescription

JOB_NAME

The name of the corrective action.

JOB_OWNER

The owner of the corrective action.

JOB_TYPE

The type of corrective action.

JOB_STATUS

The corrective action status.

TIMESTAMP

Time when the severity occurred.

NUM_TARGETS

The number of targets.

TARGET_NAMEn

The name of the nth target on which the corrective action ran. Example: TARGET_NAME1, TARGET_NAME2.

METRIC

The name of the metric in the alert that caused the corrective action to run. Not set for policy violations.

POLICY_RULE

The name of the policy rule in the alert that caused the corrective action to run. Not set for metric severities.

METRIC_VALUE

The value of the metric column in the alert that caused the corrective action to run.

VIOLATION_CONTEXT

A comma-separated list of name-value pairs that show the policy violation context.

KEY_VALUE_NAME

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

KEY_VALUE

For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity.

SEVERITY

Type of alert severity. For example, severity types that relate to a target's availability are:

  • UP

  • DOWN

  • UNREACHABLE CLEAR

  • UNREACHABLE START

  • BLACKOUT END

  • BLACKOUT START

Other metrics can have any of the following severities:

  • WARNING

  • CRITICAL

  • CLEAR

  • METRIC ERROR CLEAR

  • METRIC ERROR START

RULE_NAME

Name of the notification rule that resulted in the execution of the corrective action.

RULE_OWNER

Name of the Enterprise Manager administrator who owns the notification rule.


Passing Corrective Action Execution Status to a PLSQL Procedure

The notification system passes corrective action status change information to a PL/SQL procedure via the MGMT_NOTIFY_CORRECTIVE_ACTION object. An instance of this object is created for every status change. When a corrective action executes, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_CORRECTIVE_ACTION object that has been passed to it.

Table 3-8 lists all corrective action status change attributes that can be passed:

Table 3-8 Corrective Action Status Attributes

AttributeDatatypeAdditional Information

JOB_NAME

VARCHAR2(128)

The corrective action name.

JOB_OWNER

VARCHAR(256)

The owner of the corrective action.

JOB_TYPE

VARCHAR2(32)

The type of the corrective action.

JOB_STATUS

NUMBER

The new status of the corrective action. See Table 3-9, "Corrective Action Status Codes" for a list of possible status conditions.

STATE_CHANGE_GUID

RAW(16)

The GUID of the state change record.

JOB_GUID

RAW(16)

The unique id of the corrective action.

EXECUTION_ID

RAW(16)

The unique id of the corrective action execution.

TARGETS

SMP_EMD_NVPAIR_ARRAY

An array of the target name/target type pairs that the corrective action runs on.

METRIC_NAME

VARCHAR2(256)

The name of the metric/policy rule in the alert that caused the corrective action to run.

METRIC_COLUMN

VARCHAR2(64)

The name of the metric in the alert that caused the corrective action to run. This is not used for policy violations.

METRIC_VALUE

VARCHAR2(1024)

The value of the metric in the alert that caused the corrective action to run.

SEVERITY_CODE

NUMBER

The severity code of the alert that caused the corrective action to run. See Table 3-6, "Severity Codes".

KEY_VALUE_NAME

VARCHAR2(512)

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

KEY_VALUE

VARCHAR2(1290)

For table metrics, this column contains the value of the key column for the row in the table whose thresholds are being defined. If the thresholds are not for a table metric, or the thresholds apply for all rows in the metric column, then the value in this column will contain a single space.

KEY_VALUE_GUID

RAW(16)

GUID associated with a composite key value name.

CTXT_LIST

MGMT_NOTIFY_COLUMNS

Details of the corrective action status change context.

RULE_OWNER

VARCHAR2(64)



The owner of the notification rule that caused the PL/SQL notification to be sent.

RULE_NAME

VARCHAR2(132)

The name of the notification rule that caused the PL/SQL notification method to be invoked.

OCCURRED_DATE

DATE

The time and date when the status change happened.


The following status codes are possible values for the job_status field of the MGMT_NOTIFY_CORRECTIVE_ACTION object.

Table 3-9 Corrective Action Status Codes

NameDatatypeValue

SCHEDULED_STATUS

NUMBER(2)

1

EXECUTING_STATUS

NUMBER(2)

2

ABORTED_STATUS

NUMBER(2)

3

FAILED_STATUS

NUMBER(2)

4

COMPLETED_STATUS

NUMBER(2)

5

SUSPENDED_STATUS

NUMBER(2)

6

AGENTDOWN_STATUS

NUMBER(2)

7

STOPPED_STATUS

NUMBER(2)

8

SUSPENDED_LOCK_STATUS

NUMBER(2)

9

SUSPENDED_EVENT_STATUS

NUMBER(2)

10

SUSPENDED_BLACKOUT_STATUS

NUMBER(2)

11

STOP_PENDING_STATUS

NUMBER(2)

12

SUSPEND_PENDING_STATUS

NUMBER(2)

13

INACTIVE_STATUS

NUMBER(2)

14

QUEUED_STATUS

NUMBER(2)

15

FAILED_RETRIED_STATUS

NUMBER(2)

16

WAITING_STATUS

NUMBER(2)

17

SKIPPED_STATUS

NUMBER(2)

18

REASSIGNED_STATUS

NUMBER(2)

20


Example 3-14 PL/SQL Procedure Using a Status Code

CREATE TABLE ca_log (jobid RAW(16),                     occured DATE);


CREATE OR REPLACE PROCEDURE LOG_PROBLEM_CAS(status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION)

IS

BEGIN

 -- Log all failed corrective actions

   IF status_change.job_status = MGMT_JOBS.FAILED_STATUS

   THEN

         BEGIN

                 INSERT INTO ca_log (jobid, occured)

                 VALUES (status_change.job_guid, SYSDATE);

         EXCEPTION

         WHEN OTHERS

         THEN

         -- If there are any problems then get the notification retried

                 RAISE_APPLICATION_ERROR(-20000, 'Please retry');

                 END;

                 COMMIT;

   END IF;

END LOG_PROBLEM_CAS;

Passing Job Execution Status Information

Passing job status change attributes (such as new status, job name, job type, notification rule, or rule owner) to PL/SQL procedures or OS commands/scripts allows you to customize automated responses to status changes. For example, you many want to call an OS script to open a trouble ticket for an in-house support trouble ticket system if a critical job fails to run. In this case you will want to pass status (for example, Problems or Aborted) to the script to open a trouble ticket and escalate the problem.

Passing Job Execution Status to a PLSQL Procedure

The notification system passes job status change information to a PL/SQL procedure via the MGMT_NOTIFY_JOB object. An instance of this object is created for every status change. When a job changes status, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_JOB object that has been passed to it.

Table 3-10 lists all corrective action status change attributes that can be passed:

Table 3-10 Job Status Attributes

AttributeDatatypeAdditional Information

job_name

VARCHAR2(128)

The job name.

job_owner

VARCHAR2(256)

The owner of the job.

job_type

VARCHAR2(32)

The type of the job.

job_status

NUMBER

The new status of the job.

state_change_guid

RAW(16)

The GUID of the state change record.

job_guid

RAW(16)

The unique id of the job.

execution_id

RAW(16)

The unique id of the execution.

targets

SMP_EMD_NVPAIR_ARRAY

An array of the target name/target type pairs that the job runs on.

rule_owner

VARCHAR2(64)

The name of the notification rule that cause the notification to be sent.

rule_name

VARCHAR2(132)

The owner of the notification rule that cause the notification to be sent.

occurred_date

DATE

The time and date when the status change happened.


When a job status change occurs for the job, the notification system creates an instance of the MGMT_NOTIFY_JOB object and populates it with values from the status change. The following status codes have been defined as constants in the MGMT_JOBS package and can be used to determine the type of status in the job_status field of the MGMT_NOTIFY_JOB object.

Table 3-11 Job Status Codes

NameDatatypeValue

SCHEDULED_STATUS

NUMBER(2)

1

EXECUTING_STATUS

NUMBER(2)

2

ABORTED_STATUS

NUMBER(2)

3

FAILED_STATUS

NUMBER(2)

4

COMPLETED_STATUS

NUMBER(2)

5

SUSPENDED_STATUS

NUMBER(2)

6

AGENTDOWN_STATUS

NUMBER(2)

7

STOPPED_STATUS

NUMBER(2)

8

SUSPENDED_LOCK_STATUS

NUMBER(2)

9

SUSPENDED_EVENT_STATUS

NUMBER(2)

10

SUSPENDED_BLACKOUT_STATUS

NUMBER(2)

11

STOP_PENDING_STATUS

NUMBER(2)

12

SUSPEND_PENDING_STATUS

NUMBER(2)

13

INACTIVE_STATUS

NUMBER(2)

14

QUEUED_STATUS

NUMBER(2)

15

FAILED_RETRIED_STATUS

NUMBER(2)

16

WAITING_STATUS

NUMBER(2)

17

SKIPPED_STATUS

NUMBER(2)

18

REASSIGNED_STATUS

NUMBER(2)

20


Example 3-15 PL/SQL Procedure Using a Status Code (Job)

CREATE TABLE job_log (jobid RAW(16),                    occured DATE);


CREATE OR REPLACE PROCEDURE LOG_PROBLEM_JOBS(status_change IN MGMT_NOTIFY_JOB)

IS

BEGIN

 -- Log all failed jobs

   IF status_change.job_status = MGMT_JOBS.FAILED_STATUS

   THEN

         BEGIN

                 INSERT INTO job_log (jobid, occured)

                 VALUES (status_change.job_guid, SYSDATE);

         EXCEPTION

         WHEN OTHERS

         THEN

         -- If there are any problems then get the notification retried

                 RAISE_APPLICATION_ERROR(-20000, 'Please retry');

                 END;

                 COMMIT;

   END IF;

END LOG_PROBLEM_JOBS;

Passing Job Execution Status to an OS Command or Script

The notification system passes job execution status information to an OS script or executable via system environment variables. Conventions used to access environmental variables vary depending on the operating system:

  • UNIX: $ENV_VARIABLE

  • MS Windows: %ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.

Table 3-12 Environment Variables

Environment VariableDescription

JOB_NAME

The name of the job.

JOB_OWNER

The owner of the job.

JOB_TYPE

The type of job.

JOB_STATUS

The job status.

TIMESTAMP

Time when the severity occurred.

NUM_TARGETS

The number of targets.

TARGET_NAMEn

The name of the nth target. For example, TARGET_NAME1, TARGET_NAME2.

TARGET_TYPEn

The type of the nth target. For example TARGET_TYPE1, TARGET_TYPE2.

RULE_NAME

Name of the notification rule that resulted in the severity.

RULE_OWNER

Name of the Enterprise Manager administrator who owns the notification rule.


Passing User-Defined Target Properties to Notification Methods

Enterprise Manager allows you to define target properties (accessed via Related Links on the target home page) that can be used to store environmental or usage context information specific to that target. Target property values are passed to custom notification methods where they can be processed using conditional logic or simply passed as additional alert information to third-party devices, such as ticketing systems. By default, Enterprise Manager passes all defined target properties to notification methods.


Note:

Target properties are not passed to notification methods when short e-mail format is used.

Figure 3-4 Host Target Properties

Description of Figure 3-4 follows

Assigning Methods to Rules

For each notification rule, you can assign one or more notification methods to be called when any of the criteria in the notification rule is met.

  1. From the Enterprise Manager Grid Control, click Preferences at the top of the page.

  2. Click Notification Rules in the vertical navigation bar.

    The Enterprise Manager Grid Control displays the Notification Rules page. Any notification rules already created are listed in the Notification Rules table.

  3. Click Assign Methods to Multiple Rules.

  4. Perform your assignments.

Figure 3-5 Assigning Methods to Rules

This graphic shows the Assign Methods to Multiple rule page

Assigning Rules to Methods

For each notification method, you can associate one or more notification rules that will use that method to send notifications.

  1. From the Enterprise Manager Grid Control, click Preferences at the top of the page.

  2. Click Notification Rules in the vertical navigation bar.

    The Enterprise Manager Grid Control displays the Notification Rules page. Any notification rules already created are listed in the Notification Rules table.

  3. Click Assign Methods to Multiple Rules.

  4. From the View menu, select By Method.

  5. Perform your assignments.

Figure 3-6 Assign Rules to Methods

graphic shows the notification method page

Notification Coverage

To reduce the likelihood of an alert triggering and no administrator being notified because there was no notification rule covering that condition, you can use the Information Publisher (Enterprise Manager Report system) to view, for each target, the metrics monitored for that target and associated notification rules. Information Publisher provides an out-of-box report specifically designed for this purpose. You can run this report from the Report Definitions page (Reports tab) under Monitoring-->Alerts and Policy Violations--> Notification Rule Coverage for Metric Alerts and Availabilities (Target)

Management Information Base (MIB)

Enterprise Manager Grid Control can send SNMP Traps to third-party, SNMP-enabled applications. Details of the trap contents can be obtained from the management information base (MIB) variables. The following sections discuss Enterprise Manager MIB variables in detail.

About MIBs

A MIB is a text file, written in ASN.1 notation, which describes the variables containing the information that SNMP can access. The variables described in a MIB, which are also called MIB objects, are the items that can be monitored using SNMP. There is one MIB for each element being monitored. Each monolithic or subagent consults its respective MIB in order to learn the variables it can retrieve and their characteristics. The encapsulation of this information in the MIB is what enables master agents to register new subagents dynamically — everything the master agent needs to know about the subagent is contained in its MIB. The management framework and management applications also consult these MIBs for the same purpose. MIBs can be either standard (also called public) or proprietary (also called private or vendor).

The actual values of the variables are not part of the MIB, but are retrieved through a platform-dependent process called "instrumentation". The concept of the MIB is very important because all SNMP communications refer to one or more MIB objects. What is transmitted to the framework is, essentially, MIB variables and their current values.

Reading the MIB Variable Descriptions

This section covers the format used to describe MIB variables. Note that the STATUS element of SNMP MIB definition, Version 2, is not included in these MIB variable descriptions. Since Oracle has implemented all MIB variables as CURRENT, this value does not vary.

Variable Name

Syntax

Maps to the SYNTAX element of SNMP MIB definition, Version 2.

Max-Access

Maps to the MAX-ACCESS element of SNMP MIB definition, Version 2.

Status

Maps to the STATUS element of SNMP MIB definition, Version 2.

Explanation

Describes the function, use and precise derivation of the variable. (For example, a variable might be derived from a particular configuration file parameter or performance table field.) When appropriate, incorporates the DESCRIPTION part of the MIB definition, Version 2.

Typical Range

Describes the typical, rather than theoretical, range of the variable. For example, while integer values for many MIB variables can theoretically range up to 4294967295, a typical range in an actual installation will vary to a lesser extent. On the other hand, some variable values for a large database can actually exceed this "theoretical" limit (a "wraparound"). Specifying that a variable value typically ranges from 0 to 1,000 or 1,000 to 3 billion will help the third-party developer to develop the most useful graphical display for the variable.

Significance

Describes the significance of the variable when monitoring a typical installation. Alternative ratings are Very Important, Important, Less Important, or Not Normally Used. Clearly, the DBA will want to monitor some variables more closely than others. However, which variables fall into this category can vary from installation to installation, depending on the application, the size of the database, and on the DBA's objectives. Nevertheless, assessing a variable's significance relative to the other variables in the MIB can help third-party developers focus their efforts on those variables of most interest to the most DBAs.

Related Variables

Lists other variables in this MIB, or other MIBs implemented by Oracle, that relate in some way to this variable. For example, the value of this variable might derive from that of another MIB variable. Or perhaps the value of this variable varies inversely to that of another variable. Knowing this information, third-party developers can develop useful graphic displays of related MIB variables.

Suggested Presentation

Suggests how this variable can be presented most usefully to the DBA using the management application: as a simple value, as a gauge, or as an alarm, for example.

MIB Definition

Example 3-16 shows a typical MIB definition used by Enterprise Manager.

Example 3-16 MIB Definition

ORACLE-ENTERPRISE-MANAGER-4-MIB DEFINITIONS ::= BEGIN

IMPORTS

    TRAP-TYPE

        FROM RFC-1215

    DisplayString

        FROM RFC1213-MIB

    OBJECT-TYPE

        FROM RFC-1212

    enterprises

        FROM RFC1155-SMI;

oracle OBJECT IDENTIFIER ::= { enterprises  111 }

oraEM4 OBJECT IDENTIFIER ::= { oracle  15 }

oraEM4Objects OBJECT IDENTIFIER ::= { oraEM4  1 }

oraEM4AlertTable OBJECT-TYPE

    SYNTAX  SEQUENCE OF OraEM4AlertEntry

    ACCESS  not-accessible

    STATUS  mandatory

    DESCRIPTION

     "Information on alerts generated by Oracle Enterprise Manager. This table is

 not queryable; it exists only to document the variables included in the

 oraEM4Alert trap.  Each trap contains a single instance of each variable in the

 table."

    ::= { oraEM4Objects  1 }

oraEM4AlertEntry OBJECT-TYPE

    SYNTAX  OraEM4AlertEntry

    ACCESS  not-accessible

    STATUS  mandatory

    DESCRIPTION

     "Information about a particular Oracle Enterprise Manager alert."

    INDEX   { oraEM4AlertIndex }

    ::= { oraEM4AlertTable  1 }

OraEM4AlertEntry ::=

    SEQUENCE {

        oraEM4AlertIndex

            INTEGER,

        oraEM4AlertTargetName

       DisplayString,

        oraEM4AlertTargetType

       DisplayString,

        oraEM4AlertHostName

       DisplayString,

        oraEM4AlertMetricName

       DisplayString,

        oraEM4AlertKeyName

       DisplayString,

        oraEM4AlertKeyValue

       DisplayString,

        oraEM4AlertTimeStamp

       DisplayString,

        oraEM4AlertSeverity

       DisplayString,

        oraEM4AlertMessage

       DisplayString,

        oraEM4AlertRuleName

       DisplayString

        oraEM4AlertRuleOwner

       DisplayString

    oraEM4AlertMetricValue

           DisplayString,

        oraEM4AlertContext

           DisplayString

    }

oraEM4AlertIndex OBJECT-TYPE

    SYNTAX  INTEGER

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "Index of a particular alert, unique only at the moment an alert is generated."

    ::= { oraEM4AlertEntry  1 }

oraEM4AlertTargetName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the target to which this alert applies."

    ::= { oraEM4AlertEntry  2 }

oraEM4AlertTargetType OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The type of the target to which this alert applies."

    ::= { oraEM4AlertEntry  3 }

oraEM4AlertHostName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the host on which this alert originated."

    ::= { oraEM4AlertEntry  4 }

oraEM4AlertMetricName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the metric or policy which generated this alert."

    ::= { oraEM4AlertEntry  5 }

oraEM4AlertKeyName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the key-column, if present, for the metric which generated this alert."

    ::= { oraEM4AlertEntry  6 }

oraEM4AlertKeyValue OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The value of the key-column, if present, for the metric which generated this alert."

    ::= { oraEM4AlertEntry  7 }

oraEM4AlertTimeStamp OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The time at which this alert was generated."

    ::= { oraEM4AlertEntry  8 }

oraEM4AlertSeverity OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The severity of the alert e.g. Critical."

    ::= { oraEM4AlertEntry  9 }

oraEM4AlertMessage OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The message associated with the alert."

    ::= { oraEM4AlertEntry  10 }

oraEM4AlertRuleName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the notification rule that caused this notification."

    ::= { oraEM4AlertEntry  11 }

oraEM4AlertRuleOwner OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The owner of the notification rule that caused this notification."

    ::= { oraEM4AlertEntry  12 }

oraEM4AlertMetricValue OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The value of the metric which caused this alert to be generated."

    ::= { oraEM4AlertEntry  13 }

oraEM4AlertContext OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "A comma separated list of metric column names and values associated with the metric that caused this alert to be generated."

    ::= { oraEM4AlertEntry  14 }

oraEM4Traps OBJECT IDENTIFIER ::= { oraEM4  2 }

oraEM4Alert TRAP-TYPE

    ENTERPRISE  oraEM4Traps

    VARIABLES   { oraEM4AlertTargetName, oraEM4AlertTargetType,

                  oraEM4AlertHostName, oraEM4AlertMetricName,

                  oraEM4AlertKeyName, oraEM4AlertKeyValue, oraEM4AlertTimeStamp,

                  oraEM4AlertSeverity, oraEM4AlertMessage,

                  oraEM4AlertRuleName, oraEM4AlertRuleOwner,

                  oraEM4AlertMetricValue, oraEM4AlertContext }

    DESCRIPTION

     "The variables included in the oraEM4Alert trap."

    ::= 1

oraEM4JobAlertTable OBJECT-TYPE

    SYNTAX  SEQUENCE OF OraEM4JobAlertEntry

    ACCESS  not-accessible

    STATUS  mandatory

    DESCRIPTION

     "Information on alerts generated by Oracle Enterprise Manager. This table is

 not queryable; it exists only to document the variables included in the

 oraEM4JobAlert trap.  Each trap contains a single instance of each variable in

 the table."

    ::= { oraEM4Objects  2 }

oraEM4JobAlertEntry OBJECT-TYPE

    SYNTAX  OraEM4AlertEntry

    ACCESS  not-accessible

    STATUS  mandatory

    DESCRIPTION

     "Information about a particular Oracle Enterprise Manager alert."

    INDEX   { oraEM4JobAlertIndex }

    ::= { oraEM4JobAlertTable  1 }

OraEM4JobAlertEntry ::=

    SEQUENCE {

        oraEM4JobAlertIndex

            INTEGER,

        oraEM4JobAlertJobName

       DisplayString,

        oraEM4JobAlertJobOwner

       DisplayString,

        oraEM4JobAlertJobType

       DisplayString,

        oraEM4JobAlertJobStatus

       DisplayString,

        oraEM4JobAlertTargets

       DisplayString,

        oraEM4JobAlertTimeStamp

       DisplayString,

        oraEM4JobAlertRuleName

       DisplayString

        oraEM4JobAlertRuleOwner

       DisplayString,

    oraEM4JobAlertMetricName

           DisplayString,

    oraEM4JobAlertMetricValue

           DisplayString,

        oraEM4JobAlertContext

           DisplayString,

        oraEM4JobAlertKeyName

       DisplayString,

        oraEM4JobAlertKeyValue

       DisplayString,

        oraEM4JobAlertSeverity

       DisplayString

    }

oraEM4JobAlertIndex OBJECT-TYPE

    SYNTAX  INTEGER

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "Index of a particular alert, unique only at the moment an alert is

 generated."

    ::= { oraEM4JobAlertEntry  1 }

oraEM4JobAlertJobName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the job to which this alert applies."

    ::= { oraEM4JobAlertEntry  2 }

oraEM4JobAlertJobOwner OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The owner of the job to which this alert applies."

    ::= { oraEM4JobAlertEntry  3 }

oraEM4JobAlertJobType OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The type of the job to which this alert applies."

    ::= { oraEM4JobAlertEntry  4 }

oraEM4JobAlertJobStatus OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The status of the job to which this alert applies."

    ::= { oraEM4JobAlertEntry  5 }

oraEM4JobAlertTargets OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "A comma separated list of target to which this alert applies."

    ::= { oraEM4JobAlertEntry  6 }

oraEM4JobAlertTimeStamp OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The time at which this job status changed causing this alert."

    ::= { oraEM4JobAlertEntry  7 }

oraEM4JobAlertRuleName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the notification rule that caused this notification."

    ::= { oraEM4JobAlertEntry  8 }

oraEM4JobAlertRuleOwner OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The owner of the notification rule that caused this notification."

    ::= { oraEM4JobAlertEntry  9 }

oraEM4JobAlertMetricName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the metric or policy which caused the Corrective Action to run

 that caused this alert."

    ::= { oraEM4JobAlertEntry  10 }

oraEM4JobAlertMetricValue OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The value of the metric which caused the Corrective Action to run that

 caused this alert."

    ::= { oraEM4JobAlertEntry  11 }

oraEM4JobAlertContext OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "A comma separated list of metric column names and values associated with the

 metric which caused the Corrective Action to run that caused this alert."

    ::= { oraEM4JobAlertEntry  12 }

oraEM4JobAlertKeyName OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The name of the key-column, if present, for the metric which caused the

 Corrective Action to run that generated this alert."

    ::= { oraEM4JobAlertEntry  13 }

oraEM4JobAlertKeyValue OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The value of the key-column, if present, for the metric which caused the

 Corrective Action to run that generated this alert."

    ::= { oraEM4JobAlertEntry  14 }

oraEM4JobAlertSeverity OBJECT-TYPE

    SYNTAX  DisplayString

    ACCESS  read-only

    STATUS  mandatory

    DESCRIPTION

     "The severity of the metric which caused the Corrective Action to run that

 generated this alert e.g. Critical."

    ::= { oraEM4JobAlertEntry  15 }

oraEM4JobAlert TRAP-TYPE

    ENTERPRISE  oraEM4Traps

    VARIABLES   { oraEM4JobAlertJobName, oraEM4JobAlertJobOwner,

                  oraEM4JobAlertJobType, oraEM4JobAlertJobStatus,

                  oraEM4JobAlertTargets, oraEM4JobAlertTimeStamp,

                  oraEM4JobAlertRuleName, oraEM4JobAlertRuleOwner,

                  oraEM4JobAlertMetricName, oraEM4JobAlertMetricValue,

                  oraEM4JobAlertContext, oraEM4JobAlertKeyName,

                  oraEM4JobAlertKeyValue, oraEM4JobAlertSeverity }

    DESCRIPTION

     "The variables included in the oraEM4JobAlert trap."

    ::= 2

END

Troubleshooting Notifications

To function properly, the notification system relies on various components of Enterprise Manager and your IT infrastructure. For this reason, there can be many causes of notification failure. The following guidelines and suggestions can help you isolate potential problems with the notification system.

General Setup

The first step in diagnosing notification issues is to ensure that you have properly configured and defined your notification environment.

OS Command, PL/SQL and SNMP Trap Notifications

Make sure all OS Command, PLSQL and SNMP Trap Notification Methods are valid by clicking the Test button. This will send a test notification and show any problems the OMS has in contacting the method. Make sure that your method was called, for example, if the OS Command notification is supposed to write information to a log file, check that it has written information to its log file.

E-mail Notifications

  • Make sure an e-mail gateway is set up under the Notification Methods page of Setup. The Sender's e-mail address should be valid. Clicking the Test button will send an e-mail to the Sender's e-mail address. Make sure this e-mail is received. Note that the Test button ignores any Notification Schedule.

  • Make sure an e-mail address is setup under General page of Preferences. Clicking the Test button will send an e-mail to specified address and you should make sure this e-mail is received. Note that the Test button ignores any Notification Schedule.

  • Make sure an e-mail schedule is defined under the Schedule page of Preferences. No e-mails will be sent unless a Notification Schedule has been defined.

  • Make sure a Notification Rule is defined to match the target, metric, severity and availability states you are interested and make sure e-mail and notification methods are assigned to the rule. A summary of the notification rule can be checked by going to the Rules page under Setup and clicking the rule name.

Notification System Errors

For any alerts involving problems with notifications, check the following for notification errors.

  • Any serious errors in the Notification System are logged as system errors in the MGMT_SYSTEM_ERROR_LOG table. These errors can be seen in the Errors page under Management Services and Repository under Setup.

  • Check for any delivery errors. From the Alerts section of a target home page, click on the alert message to access the metric details page. In the Alert History section, click on the Details icon for more information about the alert. The details will give the reason why the notification was not delivered. Delivery errors are stored in MGMT_NOTIFICATION_LOG with the DELIVERED column set to 'N'.

  • Severities will not be displayed in the Grid Control console if no metric values have been loaded for the metric associated with the severity.

Notification System Trace Messages

The Notification System can produce trace messages in sysman/log/emoms.trc file.

Tracing is configured by setting the log4j.em.notification property flag using the emctl set property command. You can set the trace level to INFO, WARN, DEBUG. For example,

./emctl set property -sysman_pwd your_sysman_password -name log4j.em.notification -value DEBUG

Trace messages contain the string "em.notification". If you are working in a UNIX environment, you can search for messages in the emoms.trc file using the grep command. For example,

grep em.notification emoms.trc

What to look for in the trace file.

The following entries in the emoms.trc file are relevant to notifications.

Normal Startup Messages

When the OMS starts, you should see these types of messages.

2006-11-08 03:18:45,385 [Orion Launcher] INFO  em.notification init.1279 - Short format maximum length is 155


2006-11-08 03:18:45,386 [Orion Launcher] INFO  em.notification init.1297 - Short format is set to both subject and body


2006-11-08 03:18:45,387 [NotificationMgrThread] INFO  em.notification run.1010 - Waiting for connection to EM Repository...


2006-11-08 03:18:46,006 [NotificationMgrThread] INFO  em.notification run.1041 - Registering for Administrative Queue Name...


2006-11-08 03:18:46,250 [NotificationMgrThread] INFO  em.notification run.1078 - Administrative Queue is ADM21


2006-11-08 03:18:46,250 [NotificationMgrThread] INFO  em.notification run.1089 - Creating thread pool: min = 6 max = 24


2006-11-08 03:18:48,206 [NotificationMgrThread] INFO  em.notification handleAdminNotification.655 - Handling notifications for EMAIL1

Notification Delivery Messages

2006-11-08 03:18:45,387 [NotificationMgrThread] INFO  em.notification run.682 - Notification ready on EMAIL1


2006-11-08 03:18:46,006 [DeliveryThread-EMAIL1] INFO  em.notification run.114 - Deliver to SYSMAN/admin@oracle.com


2006-11-08 03:18:47,006 [DeliveryThread-EMAIL1] INFO  em.notification run.227 - Notification handled for SYSMAN/admin@oracle.com