Oracle Enterprise Manager Administration, 10g Release 5 (10.2.0.5)

10g Release 5 (10.2.0.5)

E14586-03

Contents

List of Examples

List of Figures

List of Tables

Title and Copyright Information

Preface

- Audience

- Documentation Accessibility

- Related Documents

- Conventions

Part I Basic Administration

1 Monitoring

- Systems Monitoring: Breadth and Depth

- Monitoring Basics

- Out-of-Box Monitoring

- Metric Thresholds

- Alerts

- Notifications

- Customizing Notifications

- Corrective Actions

- Blackouts

- Monitoring Templates

- User-Defined Metrics

- Accessing Monitoring Information

2 Setting Up Enterprise Manager

- Grid Control Setup

- Roles

- Administrators

- Super Administrator Account

- Administrator Account

- Notification Methods

- Patching Setup

- Blackouts

- Registration Passwords

- Management Pack Access

- Monitoring Templates

- Corrective Action Library

- Management Plug-ins

- Client System Analyzer in Grid Control

- Grid Control Preferences

- General

- Preferred Credentials

- Notification

- Notification Rules

- Notification Schedule

- Target Subtabs

3 Enterprise Manager Security

- About Oracle Enterprise Manager Security

- Enterprise Manager Authentication

- Repository-Based Authentication

- Single Sign-On Based Authentication

- Configuring Enterprise Manager to Use the Single Sign-On Logon Page

- Registering HTTP Port With Single Sign On Server

- Configuring Enterprise Manager to Use Single Sign-On with the osso.conf File

- Registering Single Sign-On Users as Enterprise Manager Administrators

- Grid Control as a Single Sign-On Partner Application

- Bypassing the Single Sign-On Logon Page

- Enterprise User Security Based Authentication

- Registering Enterprise Users as Enterprise Manager Users

- Enterprise Manager Authorization

- Classes of Users

- Leveraging Oracle Application Server Security Services

- Privileges and Roles

- Granting Privileges

- Configuring Security for Grid Control

- About Enterprise Manager Framework Security

- Overview of the Steps Required to Enable Enterprise Manager Framework Security

- Enabling Security for the Oracle Management Service

- Checking the Security Status

- Enabling Security for the Oracle Management Agent

- Enabling Security with Multiple Management Service Installations

- Restricting HTTP Access to the Management Service

- Managing Agent Registration Passwords

- Using the Grid Control Console to Manage Agent Registration Passwords

- Using emctl to Add a New Agent Registration Password

- Enabling Security with a Server Load Balancer

- Enabling Security for the Management Repository Database

- About Oracle Advanced Security and the sqlnet.ora Configuration File

- Configuring the Management Service to Connect to a Secure Management Repository Database

- Enabling Oracle Advanced Security for the Management Repository

- Enabling Security for a Management Agent Monitoring a Secure Management Repository or Database

- Configuring Third Party Certificates

- Configuring Third Party Certificate for HTTPS Upload Virtual Host

- Configuring Third Party Certificate for HTTPS Apache Virtual Host

- Configuring Security for the Database Control

- Accessing Managed Targets

- Credential Subsystem

- Managing Credentials Using EMCLI

- Pluggable Authentication Modules (PAM) Support for Hosts

- Configuring PAM for RHEL4 Users

- Configuring PAM for AIX Users

- Sudo and PowerBroker Support

- Creating a Privilege Delegation Setting

- Cryptographic Support

- Configuring the emkey

- Generating the emkey

- emctl Commands

- emctl status emkey

- emctl config emkey -repos

- emctl config emkey -emkeyfile

- emctl config emkey -emkey

- emctl config emkey -remove_from_repos

- emctl config emkey -copy_to_repos

- Install and Upgrade Scenarios

- Installing the Management Repository

- Installing the First Oracle Management Service

- Installing Additional Oracle Management Service

- Upgrading from 10.1 to 10.2

- Recreating the Management Repository

- Setting Up the Auditing System for Enterprise Manager

- Configuring the Enterprise Manager Audit System

- Enabling and Disabling Auditing Using emcli Commands

- Enabling and Disabling Auditing Using PL/SQL

- Configuring the Audit Data Export Service

- Searching the Audit Data

- Operation Codes

- Additional Security Considerations

- Changing the SYSMAN and MGMT_VIEW Passwords

- Changing the SYSMAN User Password

- Changing the MGMT_VIEW User Password

- Responding to Browser-Specific Security Certificate Alerts

- Responding to the Internet Explorer Security Alert Dialog Box

- Responding to the Netscape Navigator New Site Certificate Dialog Box

- Preventing the Display of the Internet Explorer Security Information Dialog Box

- Configuring Beacons to Monitor Web Applications Over HTTPS

- Using ORACLE _HOME Credentials

- Patching Oracle Homes When the User is Locked

- Cloning Oracle Homes

4 Notifications

- Setting Up Notifications

- Setting Up a Mail Server for Notifications

- Setting Up Repeat Notifications

- Setting Up E-mail for Yourself

- Defining E-mail Addresses

- Setting Up a Notification Schedule

- Subscribe to Receive E-mail for Notification Rules

- Setting Up E-mail for Other Administrators

- E-mail Customization

- E-mail Customization Reference

- Extending Notification Beyond E-mail

- Custom Notification Methods Using Scripts and SNMP Traps

- Adding a Notification Method based on an OS Command or Script

- Adding a Notification Method Based on a PL/SQL Procedure

- Adding a Notification Method Based on an SNMP Trap

- Passing Corrective Action Status Change Information

- Passing Corrective Action Execution Status to an OS Command or Script

- Passing Corrective Action Execution Status to a PLSQL Procedure

- Passing Job Execution Status Information

- Passing Job Execution Status to a PLSQL Procedure

- Passing Job Execution Status to an OS Command or Script

- Passing User-Defined Target Properties to Notification Methods

- Assigning Methods to Rules

- Assigning Rules to Methods

- Notification Coverage

- Management Information Base (MIB)

- About MIBs

- Reading the MIB Variable Descriptions

- Variable Name

- MIB Definition

- Troubleshooting Notifications

- General Setup

- Notification System Errors

- Notification System Trace Messages

- E-mail Errors

- OS Command Errors

- SNMP Trap Errors

- PL/SQL Errors

5 User-Defined Metrics

- Extending Monitoring Capability

- Creating OS-Based User-Defined Metrics

- Create Your OS Monitoring Script

- Code to check the status of monitored objects

- Code to return script results to Enterprise Manager

- Script Runtime Environment

- Register the Script as a User-Defined Metric

- OS-Based User-Defined Metric Example

- Creating a SQL-Based User-Defined Metric

- SQL-Based User-Defined Metric Examples

- Example 1: Query Returning Tablespace Name and Percent Used

- Example 2: Query Returning Segment Name/Type and Extent Count

- Example 3: Embed a Long SQL statement in a PL/SQL Routine

- Notifications, Corrective Actions, and Monitoring Templates

- Getting Notifications for User-Defined Metrics

- Setting Corrective Actions for User-Defined Metrics

- Deploying User-Defined Metrics Across Many Targets Using Monitoring Templates

- Deleting User-Defined Metrics Across Many Targets Using Monitoring Templates

- Changing User-Defined Metric Credentials

6 Group Management

- Introduction to Groups

- Managing Groups

- Group Home Page

- Group Charts Page

- Group Administration Page

- Group Members Page

- System Dashboard

- Out-of-Box Reports

- Redundancy Groups

7 Job System

- What Is a Job?

- What Are Job Executions and Job Runs?

- Differences Between Job Executions and Job Runs

- Using and Defining Jobs

- Analyzing Job Activity

- Jobs and Groups

- Sharing Job Responsibilities

- Job Library

- Job Notifications

- Multitask Jobs

8 Starting and Stopping Enterprise Manager Components

- Controlling the Oracle Management Agent

- Starting, Stopping, and Checking the Status of the Management Agent on UNIX

- Starting and Stopping the Management Agent on Windows

- Checking the Status of the Management Agent on Windows

- Controlling the Oracle Management Service

- Controlling the Management Service on UNIX

- Using OPMN to Start and Stop the Management Service

- Using emctl to Start, Stop, and Check the Status of the Oracle Management Service

- Starting and Stopping Oracle Application Server Web Cache

- Controlling the Management Service on Windows

- Controlling the Application Server Control

- Starting and Stopping the Application Server Control on UNIX

- Starting and Stopping the Application Server Control on Windows

- Controlling the Database Control on UNIX

- Starting the Database Control on UNIX

- Stopping the Database Control on UNIX

- Starting and Stopping the Database Control on Windows

- Guidelines for Starting Multiple Enterprise Manager Components on a Single Host

- Starting and Stopping Oracle Enterprise Manager 10g Grid Control

- Starting Grid Control and All Its Components

- Stopping Grid Control and All Its Components

- Additional Management Agent Commands

- Uploading and Reloading Data to the Management Repository

- Specifying New Target Monitoring Credentials

- Using the Grid Control console to Modify the Monitoring Credentials

- Using the Enterprise Manager Command Line to Modify the Monitoring Credentials

- Listing the Targets on a Managed Host

- Controlling Blackouts

- Changing the Management Agent Time Zone

- Reevaluating Metric Collections

- emctl Commands

- Using emctl.log File

9 Backup, Recovery, and Disaster Recovery

- Backup and Recovery of Enterprise Manager

- Repository Backup and Recovery

- Repository Backup

- Repository Recovery

- Recovery Scenarios

- Full Recovery on the Same Host

- Incomplete Recovery on the Same Host

- Full Recovery on a Different Host

- Incomplete Recovery on a Different Host

- OMS Backup and Recovery

- Backing Up the OMS

- Recovering the OMS

- OMS Recovery Scenarios

- Single OMS with No Server Load Balancer (SLB). OMS Restored on the same Host

- Single OMS, No SLB, OMS Restored on a Different Host

- Multiple OMS, Server Load Balancer configured, OMS restored on the same host

- Multiple OMS, Server Load Balancer configured, OMS restored on a different host

- Agent Backup and Recovery

- Backing Up Agents

- Recovering Agents

- Agent Recovery Scenarios

- Agent reinstall, same port.

- Agent restore from filesystem backup

- Recovering from a Compound OMS-Repository Failure

- Collapsed configuration, recovery on the same host, incomplete recovery of repository

- Distributed configuration, Multi-OMS with SLB, recovery on different hosts, incomplete recovery of repository

- EMCTL High Availability Commands

10 Information Publisher

- About Information Publisher

- Out-of-Box Report Definitions

- Custom Reports

- Creating Custom Reports

- Report Parameters

- Report Elements

- Scheduling Reports

- Flexible Schedules

- Storing and Purging Report Copies

- E-mailing Reports

- Sharing Reports

11 Sizing Your Enterprise Manager Deployment

- Oracle Enterprise Manager Grid Control Architecture Overview

- Enterprise Manager Grid Control Sizing and Performance Methodology

- Step 1: Choosing a Starting Platform Grid Control Deployment

- Network Topology Considerations

- Step 2: Periodically Evaluate the Vital Signs of Your Site

- Step 3: Use DBA and Enterprise Manager Tasks To Eliminate Bottlenecks Through Housekeeping

- Online Weekly Tasks

- Offline Monthly Tasks

- Step 4: Eliminate Bottlenecks Through Tuning

- High CPU Utilization

- Loader Vital Signs

- Rollup Vital Signs

- Job, Notification, and Alert Vital Signs

- I/O Vital Signs

- The Oracle Enterprise Manager Performance Page

- Step 5: Extrapolating Linearly Into the Future for Sizing Requirements

- Oracle Enterprise Manager Backup, Recovery, and Disaster Recovery Considerations

- Best Practices for Backup

- Best Practices for Recovery

- Recovering the Management Repository

- Recovering the Oracle Management Service

- Recovering the Oracle Management Agent

- Best Practice for Disaster Recovery (DR)

- Management Repository

- Oracle Management Service

- Management Agent

12 Maintaining and Troubleshooting the Management Repository

- Management Repository Deployment Guidelines

- Management Repository Data Retention Policies

- Management Repository Default Aggregation and Purging Policies

- Management Repository Default Aggregation and Purging Policies for Other Management Data

- Modifying the Default Aggregation and Purging Policies

- Modifying Data Retention Policies When Targets Are Deleted

- How to Modify the Retention Period of Job History

- Changing the SYSMAN Password

- Overview of the MGMT_VIEW User

- Dropping and Recreating the Management Repository

- Dropping the Management Repository

- Recreating the Management Repository

- Using the RepManager Script to Create the Management Repository

- Using a Connect Descriptor to Identify the Management Repository Database

- Troubleshooting Management Repository Creation Errors

- Package Body Does Not Exist Error While Creating the Management Repository

- Server Connection Hung Error While Creating the Management Repository

- General Troubleshooting Techniques for Creating the Management Repository

- Cross Platform Enterprise Manager Repository Migration

- Common Prerequisites

- Methodologies

- Cross Platform Transportable Tablespaces

- Data Pump

- Export/Import

- Post Migration Verification

- Improving the Login Performance of the Console Home Page

Part II Advanced Topics

13 Managing Compliance

- Compliance Overview

- Compliance Management

- Accessing Compliance Management Pages in Grid Control

- Investigating Compliance Violations

- Assessing Security

- Viewing Compliance Evaluation Results

- Compliance Violations Reports

- Setting Up Compliance Evaluations

- Scheduling an Evaluation

- Out-of-Box Policies and Policy Groups

- Customizing Policies

- Defining Corrective Actions

- Using Templates for Monitoring

- Policy Groups

- Secure Configuration for Oracle Database

- Secure Configuration for Oracle Real Application Cluster

- Secure Configuration for Oracle Listener

14 Configuring Services

- Summary of Service Management Tasks

- Setting up the System

- Creating a Service

- Configuring a Service

- Availability Definition

- Performance Metrics

- Usage Metrics

- Business Metrics

- Service Tests and Beacons

- Configuring the Beacons

- Configuring Windows Beacons for Web Transaction (Browser) Playback

- Root Cause Analysis Configuration

- Getting the Most From Root Cause Analysis

- Recording Web Transactions

- Monitoring Settings

- Configuring Aggregate Services

- Configuring End-User Performance Monitoring

- Configuring End-User Performance Monitoring Using Oracle HTTP Server Based on Apache 2.0 or Apache HTTP Server 2.0

- Setting up the Third Party Apache Server

- Configuring End-User Performance Monitoring Using Oracle Application Server Web Cache

- Configuring Oracle Application Server Web Cache 10.1.2

- Configuring Oracle Application Server Web Cache 9.0.4

- Configuring End-User Performance Monitoring Using Earlier Versions of Oracle Application Server Web Cache

- Configuring End-User Performance Monitoring Using Standalone Oracle Application Server Web Cache

- Configuring End-User Performance Monitoring for Web Page Extensions

- Configuring End-User Performance Monitoring for Web Pages Having the Same URI

- Starting and Stopping End-User Performance Monitoring

- Verifying and Troubleshooting End-User Performance Monitoring

- Enabling End-User Performance Monitoring for Third-Party Application Servers

- Managing Forms Applications

- Recording and Monitoring Forms Transactions

- Setting the Permissions of the .java.policy File

- Using a Trusted Enterprise Manager Certificate

- Adding a Forms Certificate to the Enterprise Manager Agent

- Configuring the Forms Server

- Installing the Transaction Recorder to Record and Play Back Forms Transactions

- Monitoring the End-User Performance of Forms Applications

- Configuring the Forms Server for End-User Performance Monitoring

- Configuring the OracleAS Web Cache

- Configuring the Oracle HTTP Server / Apache HTTP Server

- Starting and Stopping End-User Performance Monitoring

- Configuring OC4J for Request Performance Diagnostics

- Selecting OC4J Targets for Request Performance Diagnostics

- Configuring Interactive Transaction Tracing

- Configuring OC4J Tracing for Request Performance Data

- Additional Configuration for Monitoring UIX Applications

- Setting Up Monitoring Templates

- Configuring Service Tests and Beacons

- Configuring Service Levels

- Defining Service Level Rules

- Viewing Service Level Details

- Configuring a Service Using the Command Line Interface

- Troubleshooting Service Tests

- Verifying and Troubleshooting Forms Transactions

- Troubleshooting Forms Transaction Playback

- Troubleshooting Forms Transaction Recording

- Troubleshooting End-User Performance of Forms Transactions

- Verifying and Troubleshooting Web Transactions

15 Extending Enterprise Manager

- Benefits of Extending Enterprise Manager

- Developing Management Plug-ins

- About the Oracle Management Agent

- Designing your Management Plug-in

- Developing Your Management Plug-in

- Validating the Plug-in

- Packaging the Plug-in

- Deploying the Management Plug-in

- Management Plug-in User Interface

- Enhancing Default Target Home Pages with Real-Time Reports

Index

List of Examples

- 3-1 Sample Output of the emctl config oms sso Command

- 3-2 Sample Output of the emctl secure oms Command

- 3-3 Usage of the emctl secure oms Command (II)

- 3-4 Sample Output of the emctl secure status oms Command

- 3-5 Sample Output of the emctl secure agent Utility

- 3-6 Sample Output of the emctl secure status agent Command

- 3-7 Sample Output of the emctl secure lock Command

- 3-8 Sample Output of the emctl secure unlock Command

- 3-9 Sample Output from the emctl secure em Command

- 3-10 emctl start oms Command

- 3-11 emctl start oms Command

- 3-12 emctl status emkey - Example 1

- 3-13 emctl status emkey - Example 2

- 3-14 emctl status emkey - Example 3

- 3-15 emctl status emkey - Example 4

- 3-16 Sample Output of the emctl config emkey -repos Command

- 3-17 Sample Output of emctl config emkey -emkeyfile Command

- 3-18 Sample Output of emctl config emkey -emkey Command

- 3-19 Sample Output of emctl config emkey -remove_from_repos Command

- 3-20 Sample Output of emctl config emkey_copy_to_repos Command

- 3-21 Sample Content of an Exported Certificate

- 4-1 Mail Server Settings

- 4-2 Management Service Parameters

- 4-3 Long E-mail Notification for Alerts

- 4-4 Short E-mail Notification for Alerts

- 4-5 Parameter in emoms.properties File

- 4-6 OS Command Notification Method

- 4-7 Sample OS Command Script

- 4-8 Alert Logging Script

- 4-9 HP OpenView Script

- 4-10 PL/SQL Procedure Required Information

- 4-11 PL/SQL Procedure Using a Severity Code

- 4-12 SNMP Trap Required Information

- 4-13 SNMP Trap

- 4-14 PL/SQL Procedure Using a Status Code

- 4-15 PL/SQL Procedure Using a Status Code (Job)

- 4-16 MIB Definition

- 5-1 Long SQL Statement

- 5-2 Creating an Array of Returned Values

- 5-3 Embedded SQL in a PL/SQL Routine

- 5-4 Host Password Change

- 5-5 Database Password Change

- 8-1 Checking the Status of the Management Agent

- 8-2 Modifying the Database Monitoring Credentials

- 8-3 Listing the Targets on a Managed Host

- 15-1 Target Type Metadata File Excerpt

- 15-2 Target Type Default Collections File

- 15-3 Using the EMCLI to create a Management Plug-in Archive

List of Figures

- 1-1 Enterprise Manager Console

- 1-2 Warning Alerts Page

- 1-3 Warning Alert: Metric Details

- 1-4 Compare Targets

- 1-5 Alert Details

- 2-1 Grid Control's Overview of Setup Page

- 3-1 Create / Edit Administrator

- 3-2 Modified Enterprise Manager Logon Page When Configuring SSO

- 3-3 Create Administrator Page When SSO Support Is Enabled

- 3-4 Administrators Page

- 3-5 Enterprise Manager Framework Security

- 3-6 Secure Upload Field on the Management Agent Home Page

- 3-7 Managing Registration Passwords in the Grid Control Console

- 3-8 Audit Data Page

- 3-9 Audit Data Search Page

- 3-10 Audit Record Details Page

- 3-11 Internet Explorer Security Alert Dialog Box

- 3-12 Certificate Path Tab on the Internet Explorer Certificate Dialog Box

- 3-13 Netscape Navigator New Site Certificate Dialog Box

- 3-14 Internet Explorer Security Information Dialog Box

- 4-1 Defining a Mail Server

- 4-2 Enabling Repeat Notification for an OS Command Notification Method

- 4-3 E-mail Customization

- 4-4 Host Target Properties

- 4-5 Assigning Methods to Rules

- 4-6 Assign Rules to Methods

- 5-1 Create User-Defined Metric Page (OS-Based)

- 5-2 Create User-Defined Metric Page (SQL-Based)

- 6-1 Group Home Page

- 6-2 Policy Trend Overview Page

- 6-3 Group Charts Page

- 6-4 Group Administration Page

- 6-5 Group Members Page

- 6-6 System Dashboard

- 6-7 Availability History Report

- 7-1 Job Activity Page

- 7-2 Summary of Target Jobs on the Grid Control Console Home Page

- 9-1 Recovery Settings Page

- 9-2 Backup Policy Page

- 9-3 Repository Synchronization Page

- 10-1 Availability History (Group) Report

- 11-1 Overview of Enterprise Manager Architecture Components

- 11-2 Network Links Related to Enterprise Manager Components

- 14-1 Create Service - General Page

- 14-2 Monitoring Configuration Page

- 14-3 Manage Web Server Data Collection

- 15-1 Management Plug-in Import Page

- 15-2 Management Plug-ins Page

- 15-3 Deploy Management Plug-in Page

- 15-4 Default Target Home Page

- 15-5 Default Target Home Page with Real-Time Report

List of Tables

- 3-1 Arguments for the emctl sso Command

- 3-2 System Privileges

- 3-3 Target Privileges

- 3-4 Object Privileges

- 3-5 Oracle Advanced Security Properties in the Enterprise Manager Properties File

- 3-6 Operation Specific Data

- 3-7 emcli set_credential Parameters

- 4-1 Default Notification Rules

- 4-2 Reserved Words and Operators

- 4-3 Control Structures

- 4-4 Environment Variables

- 4-5 Metric Severity Attributes

- 4-6 Severity Codes

- 4-7 Environment Variables

- 4-8 Corrective Action Status Attributes

- 4-9 Corrective Action Status Codes

- 4-10 Job Status Attributes

- 4-11 Job Status Codes

- 4-12 Environment Variables

- 5-1 Metric Severity Levels

- 5-2 Script Output Information Tags

- 5-3 Create User-Defined Metric Page: Definition

- 5-4 Create User-Defined Metric Page: Operating System

- 5-5 Create User-Defined Metric Page: Threshold

- 5-6 Create User-Defined Metric Page: Definition

- 5-7 Create User-Defined Metric Page: Database Credentials

- 5-8 Create User-Defined Metric Page: Threshold

- 8-1 Starting, Stopping, and Checking the Status of the Management Agent

- 8-2 Summary of Services Installed and Configured When You Install the Management Agent on Windows

- 8-3 Starting, Stopping, and Checking the Status of the Management Service

- 8-4 Starting, Stopping, and Checking the Status of Oracle Application Server Web Cache

- 8-5 Summary of Services Installed and Configured When You Install the Oracle Management Service on WIndows

- 8-6 Manually Reloading and Uploading Management Data

- 8-7 Summary of Blackout Commands

- 8-8 emctl Commands

- 8-9 Description of Exit Codes

- 10-1 Predefined Report Definitions

- 10-2 Report Elements

- 11-1 Management Server

- 11-2 Management Repository

- 11-3 Total Management Repository Storage

- 12-1 Default Repository Purging Policies

- 12-2 Default Repository Purging Policies for Application Performance Management Data

- 12-3 Parameters for Modifying Default Data Retention Times in the Management Repository

- 14-1 Summary of Service Management Tasks

- 14-2 Beacon Aggregation Functions

- 14-3 System Aggregation Functions

- 14-4 Aggregation Functions - Usage Metrics

- 14-5 Aggregation Functions - Usage Metrics

- 15-1 Fetchlets

Oracle® Enterprise Manager

Administration

10g Release 5 (10.2.0.5)

E14586-03

December 2009

Oracle Enterprise Manager Administration, 10g Release 5 (10.2.0.5)

E14586-03

Copyright © 2009, Oracle and/or its affiliates. All rights reserved.

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

If this software or related documentation is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable:

U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle USA, Inc., 500 Oracle Parkway, Redwood City, CA 94065.

This software is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications which may create a risk of personal injury. If you use this software in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure the safe use of this software. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software in dangerous applications.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

This software and documentation may provide access to or information on content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services.

Preface

The Enterprise Manager Administrator's Guide is an administrative reference that teaches you how to perform day-to-day Enterprise Manager administrative tasks. The goal of this book is to help you understand the concepts behind Enterprise Manager. It teaches you how to perform all common administration tasks needed to effectively monitor and manage targets within your Enterprise Manager environment.

Audience

This document is intended for all levels of Enterprise Manager administrators, as well as database, network, and application administrators using Enterprise Manager to monitor and maintain their IT infrastructure.

Documentation Accessibility

Our goal is to make Oracle products, services, and supporting documentation accessible to all users, including users that are disabled. To that end, our documentation includes features that make information available to users of assistive technology. This documentation is available in HTML format, and contains markup to facilitate access by the disabled community. Accessibility standards will continue to evolve over time, and Oracle is actively engaged with other market-leading technology vendors to address technical obstacles so that our documentation can be accessible to all of our customers. For more information, visit the Oracle Accessibility Program Web site at http://www.oracle.com/accessibility/.

Accessibility of Code Examples in Documentation

Screen readers may not always correctly read the code examples in this document. The conventions for writing code require that closing braces should appear on an otherwise empty line; however, some screen readers may not always read a line of text that consists solely of a bracket or brace.

Accessibility of Links to External Web Sites in Documentation

This documentation may contain links to Web sites of other companies or organizations that Oracle does not own or control. Oracle neither evaluates nor makes any representations regarding the accessibility of these Web sites.

Deaf/Hard of Hearing Access to Oracle Support Services

To reach Oracle Support Services, use a telecommunications relay service (TRS) to call Oracle Support at 1.800.223.1711. An Oracle Support Services engineer will handle technical issues and provide customer support according to the Oracle service request process. Information about TRS is available at http://www.fcc.gov/cgb/consumerfacts/trs.html, and a list of phone numbers is available at http://www.fcc.gov/cgb/dro/trsphonebk.html.

Related Documents

For more information, see the following manuals in the Oracle Enterprise Manager 10g Release 2 documentation set:

- Oracle Enterprise Manager Grid Control Quick Installation Guide

- Oracle Enterprise Manager Grid Control Installation and Configuration

- Oracle Enterprise Manager Advanced Configuration

- Oracle Enterprise Manager Configuration for Oracle Collaboration Suite

- Oracle Enterprise Manager Policy Reference Manual

- Oracle Enterprise Manager Metric Reference Manual

- Extending Oracle Enterprise Manager

- Oracle Enterprise Manager Command Line Interface

- SNMP Support Reference Guide

The latest versions of this and other Enterprise Manager books can be found at:

http://www.oracle.com/technology/documentation/oem.html

Oracle Enterprise Manager 10g also provides extensive online help. Click Help on any Oracle Enterprise Manager 10g page to display the online help system.

Conventions

The following text conventions are used in this document:

Convention

Meaning

boldface

Boldface type indicates graphical user interface elements associated with an action, or terms defined in text or the glossary.

italic

Italic type indicates book titles, emphasis, or placeholder variables for which you supply particular values.

monospace

Monospace type indicates commands within a paragraph, URLs, code in examples, text that appears on the screen, or text that you enter.

Part I

Basic Administration

This section of the guide covers using Enterprise Manager framework functionality to monitor your managed environment.

Part I contains the following chapters:

- Chapter 1, "Monitoring"

- Chapter 2, "Setting Up Enterprise Manager"

- Chapter 3, "Enterprise Manager Security"

- Chapter 4, "Notifications"

- Chapter 5, "User-Defined Metrics"

- Chapter 6, "Group Management"

- Chapter 7, "Job System"

- Chapter 8, "Starting and Stopping Enterprise Manager Components"

- Chapter 9, "Backup, Recovery, and Disaster Recovery"

- Chapter 10, "Information Publisher"

- Chapter 11, "Sizing Your Enterprise Manager Deployment"

- Chapter 12, "Maintaining and Troubleshooting the Management Repository"

1 Monitoring

Because of the size, complexity, and criticality of today's enterprise IT operations, the challenge for IT professionals is to be able to maintain high levels of component availability and performance for both applications and all components that make up the application's technology stack. Monitoring the performance of these components and quickly correcting problems before they can impact business operations is crucial. Enterprise Manager provides comprehensive, flexible, easy-to-use monitoring functionality that supports the timely detection and notification of impending IT problems across your enterprise.

This chapter covers the following topics:

- Systems Monitoring: Breadth and Depth

- Monitoring Basics

- Monitoring Templates

- User-Defined Metrics

- Accessing Monitoring Information

Systems Monitoring: Breadth and Depth

Enterprise Manager system monitoring features provide increased out-of-box value, automation, and grid monitoring support to enable IT organizations to maximize operational efficiencies and provide high quality services. For applications that are built on Oracle, Enterprise Manager offers the most comprehensive monitoring of the Oracle Grid environment—from Oracle Database instances to Oracle Real Application Clusters to Oracle Application Server Farms and Clusters. To support the myriad and variety of applications built on Oracle, Enterprise Manager expands its monitoring scope to non-Oracle components, such as third-party application servers, hosts, firewalls, server load balancers, and storage.

Enterprise Manager provides the most comprehensive management features for all Oracle products. For example, Enterprise Manager's monitoring functionality is tightly integrated with Oracle Database 10g manageability features such as server-generated alerts. These alerts are generated by the database itself about problems it has self-detected. Server-generated alerts can be managed from the Enterprise Manager console and include recommendations on how problems can be resolved. Performance problems such as poorly performing SQL and corresponding recommendations that are generated by the database's self-diagnostic engine, called Automatic Database Diagnostic Monitor (ADDM), are also captured and exposed through the Enterprise Manager console. This allows Enterprise Manager administrators to implement ADDM recommendations with ease and convenience.

Enterprise Manager also makes it easy to expand the scope of system monitoring beyond individual components. Using Enterprise Manager's group management functionality, you can easily organize monitorable targets into groups, allowing you to monitor and manage many components as one.

Monitoring Basics

System monitoring functionality permits unattended monitoring of your IT environment. Enterprise Manager comes with a comprehensive set of performance and health metrics that allows monitoring of key components in your environment, such as applications, application servers, databases, as well as the back-end components on which they rely (hosts, operating systems, storage, and so on).

The Management Agent on each monitored host monitors the status, health, and performance of all managed components (also referred to as targets) on that host. If a target goes down, or if a performance metric crosses a warning or critical threshold, an alert is generated and sent to Enterprise Manager and to Enterprise Manager administrators who have registered interest in receiving such notifications. Systems monitoring functionality and the mechanisms that support this functionality are discussed in the following sections.

When it is not practical to have a Management Agent present to monitor specific components of your IT infrastructure, as might be the case with an IP traffic controller or remote Web application, Enterprise Manager provides Extended Network and Critical URL Monitoring functionality. This feature allows the Beacon functionality of the Agent to monitor remote network devices and URLs for availability and responsiveness without requiring an Agent to be physically present on that device. You simply select a specific Beacon, and add key network components and URLs to the Network and URL Watch Lists. More information about using this feature is available in the Enterprise Manager online help.

Out-of-Box Monitoring

Enterprise Manager's Management Agents automatically start monitoring their host's systems (including hardware and software configuration data on these hosts) as soon as they are deployed and started. Enterprise Manager provides auto-discovery scripts that enable these Agents to automatically discover all Oracle components and start monitoring them using a comprehensive set of metrics at Oracle-recommended thresholds. This monitoring functionality includes other components of the Oracle ecosystem such as NetApp Filer, BIG-IP load balancers, Checkpoint Firewall, and IBM WebSphere and BEA WebLogic application servers. Metrics from all monitored components are stored and aggregated in the Management Repository, providing administrators with a rich source of diagnostic information and trend analysis data. When critical alerts are detected, notifications are sent to administrators for rapid resolution.

Out-of-box, Enterprise Manager monitoring functionality provides:

- In-depth monitoring with Oracle-recommended metrics and thresholds.

- Access to real-time performance charts.

- Collection, storage, and aggregation of metric data in the Management Repository. This allows you to perform strategic tasks such as trend analysis and reporting.

- E-mail notification for detected critical alerts.

Enterprise Manager can monitor a wide variety of components (such as databases, hosts, and routers) within your IT infrastructure.

Some examples of monitored metrics are:

- Archive Area Used (Database)

- Component Memory Usage (Application Server)

- Segments Approaching Maximum Extents Count (Database)

- Network Interface Total I/O Rate (Host)

Some metrics have associated predefined limiting parameters called thresholds that cause alerts to be triggered when collected metric values exceed these limits. Enterprise Manager allows you to set metric threshold values for two levels of alert severity:

- Warning - Attention is required in a particular area, but the area is still functional.

- Critical - Immediate action is required in a particular area. The area is either not functional or indicative of imminent problems.

Hence, thresholds are boundary values against which monitored metric values are compared. For example, for each disk device associated with the Disk Utilization (%) metric, you might define a warning threshold at 80% disk space used and critical threshold at 95%.

Metric Thresholds

As mentioned earlier, some metric thresholds come predefined out-of-box. While these values are acceptable for most monitoring conditions, your environment may require that you customize threshold values to more accurately reflect the operational norms of your environment. Setting accurate threshold values, however, may be more challenging for certain categories of metrics such as performance metrics.

For example, what are appropriate warning and critical thresholds for the Response Time Per Transaction database metric? For such metrics, it might make more sense to be alerted when the monitored values for the performance metric deviates from normal behavior. Enterprise Manager provides features to enable you to capture normal performance behavior for a target and determine thresholds that are deviations from that performance norm.

Note:

Enterprise Manager administrators must be granted OPERATOR or greater privilege on a target in order to perform any metric threshold changes.

Metric Snapshots

A metric snapshot is a named collection of a target's performance metrics that have been collected at a specific point in time. A metric snapshot can be used as an aid in calculating metric threshold values based on the target's past performance.

The key in defining a metric snapshot for a target is to select a date during which target performance was acceptable under typical workloads. Given this date, actual values of the performance metrics for the target are retrieved and these represent what is normal or expected performance behavior for the target. Using these values, you can then use Enterprise Manager to calculate warning and critical thresholds for the metrics that are a specified percentage 'worse' than the actual metric snapshot values. These represent values which, when crossed, could indicate performance problems. After thresholds are calculated, you can still edit the calculated values if needed.

You can define a metric snapshot for a target based on a date and (optionally) time. If you only specify a date, the metric snapshot is the set of average daily values of the target's performance metrics for that date. If you also specify an hour within the date, then the metric snapshot is the set of Low and High metric values for the preceding hour.

Metric snapshots apply to all monitored targets except 10.2 or higher databases, Services, and Web applications. For these targets, the Metric Baseline feature is supported.

Metric Baselines

Metric baselines are statistical characterizations of system performance over well-defined time periods. Metric baselines can be used to implement adaptive alert thresholds for certain performance metrics as well as provide normalized views of system performance. Adaptive alert thresholds are used to detect unusual performance events. Baseline normalized views of metric behavior help administrators explain and understand such events.Metric baselines are well defined time intervals (baseline periods) over which Enterprise Manager has captured system performance metrics. The underlying assumption of metric baselines is that systems with relatively stable performance should exhibit similar metric observations (that is, values) over times of comparable workload. Two types of baseline periods are supported: moving window baseline periods and static baseline periods. Moving window baseline periods are defined as some number of days prior to the current date (example: Last 7 days). This allows comparison of current metric values with recently observed history. Moving window baselines are useful for operational systems with predictable workload cycles (example: OLTP days and batch nights).Static baselines are periods of time that you define that are of particular interest to you (example: end of the fiscal year). These baselines can be used to characterize workload periods for comparison against future occurrences of that workload (example: compare end of the fiscal year from one calendar year to the next).

Adaptive Thresholds

Once metric baselines are defined, they can be used to establish alert thresholds that are statistically significant and adapt to expected variations across time. For example, you can define alert thresholds to be generated based on significance level, such as the HIGH significance level thresholds are values that occur 5 in 100 times. Alternatively, you can generate thresholds based on a percentage of the maximum value observed within the baseline period. These can be used to generate alerts when performance metric values are observed to exceed normal peaks within that period.

Baseline Normalized Views

Enterprise Manager provides charts which graphically display the values of observed performance and workload metrics normalized against the baseline. Using these charts, statistically significant values are easily seen as 'blips' in the charts. These allow administrators to easily perform time-correlation of events. For example, performance events can be related to significantly increased demand or significantly unusual workload.

Metric baselines are supported for databases (version 10.2 or higher) and for Services and Web Application target types.

Alerts

When a metric threshold value is reached, an alert is generated. An alert indicates a potential problem; either a warning or critical threshold for a monitored metric has been crossed. An alert can also be generated for various target availability states, such as:

- Target is down.

- Oracle Management Agent monitoring the target is unreachable.

When an alert is generated, you can access details about the alert from the Enterprise Manager console. See "Accessing Monitoring Information" on page 1E8 for more information on viewing alert information.

Enterprise Manager provides various options to respond to alerts. Administrators can be automatically notified when an alert triggers and/or corrective actions can be set up to automatically resolve an alert condition.

Notifications

When a target becomes unavailable or if thresholds for performance are crossed, alerts are generated in the Enterprise Manager console and notifications are sent to the appropriate administrators. Enterprise Manager supports notifications via e-mail (including e-mail-to-page systems), SNMP traps, and/or by running custom scripts.

Enterprise Manager supports these various notification mechanisms via notification methods. A notification method is used to specify the particulars associated with a specific notification mechanism, for example, which SMTP gateway(s) to use for e-mail, which OS script to run to log trouble-tickets, and so on. Super Administrators perform a one-time setup of the various types of notification methods available for use. Once defined, other administrators can create notification rules that specify the set of criteria that determines when a notification should be sent and how it should be sent. The criteria defined in notification rules include the targets, metrics and severity states (clear, warning or critical) and the notification method that should be used when an alert occurs that matches the criteria. For example, you can define a notification rule that specifies e-mail should be sent to you when CPU Utilization on any host target is at critical severity, or another notification rule that creates a trouble-ticket when any database is down. Once a notification rule is defined, it can be made public for sharing across administrators. For example, administrators can subscribe to the same rule if they are interested in receiving alerts for the same criteria defined in the rule. Alternatively, an Enterprise Manager Super Administrator can assign notification rules to other administrators such that they receive notifications for alerts as defined in the rule.

Notifications are not limited to alerting administrators. Notification methods can be extended to execute any custom OS script or PL/SQL procedure, and thus can be used to automate any type of alert handling. For example, administrators can define notification methods that call into a trouble ticketing system, invoke third-party APIs to share alert information with other monitoring systems, or log a bug against a product.

Customizing Notifications

Notifications that are sent to Administrators can be customized based on message type and on-call schedule. Message customization is useful for administrators who rely on both e-mail and paging systems as a means for receiving notifications. The message formats for these systems typically vary—messages sent to e-mail can be lengthy and can contain URLs, and messages sent to a pager are brief and limited to a finite number of characters. To support these types of mechanisms, Enterprise Manager allows administrators to associate a long or short message format with each e-mail address. E-mail addresses that are used to send 'regular' e-mails can be associated with the 'long' format; e-mail addresses that are used to send pages can be associated with the 'short' format. The 'long' format contains full details about the alert; the 'short' format contains the most critical pieces of information.

Notifications can also be customized based on an administrator's on-call schedule. An administrator who is on-call might want to be contacted by both his pager and work e-mail address during business hours and only by his pager address during off hours. Enterprise Manager offers a flexible notification schedule to support the wide variety of on-call schedules. Using this schedule, an administrator defines his on-call schedule by specifying the e-mail addresses by which they should be contacted when they are on-call. For periods where they are not on-call, or do not wish to receive notifications for alerts, they simply leave that part of the schedule blank. All alerts that are sent to an administrator automatically adhere to his specified schedule.

Corrective Actions

Corrective actions allow you to specify automated responses to alerts. Corrective actions ensure that routine responses to alerts are automatically executed, thereby saving administrator time and ensuring problems are dealt with before they noticeably impact users. For example, if Enterprise Manager detects that a component, such as the SQL*Net listener is down, a corrective action can be specified to automatically start it back up. A corrective action is thus any task you specify that will be executed when a metric triggers a warning or critical alert severity. By default, the corrective action runs on the target on which the alert has triggered. Administrators can also receive notifications for the success or failure of corrective actions.

A corrective action can also consist of multiple tasks, with each task running on a different target. For example, if an Oracle Application Server's J2EE container (called an OC4J container) triggers a warning alert indicating it is approaching its limit on the number of requests it can handle, a corrective action can be defined to automatically start up another OC4J container on another host, thereby sharing application load among different containers. As shown by this example, corrective actions can be used to dynamically allocate resources as demand increases, thereby preventing performance bottlenecks before they impact overall application availability.

Corrective actions for a target can be defined by all Enterprise Manager administrators who have been granted OPERATOR or greater privilege on the target. For any metric, you can define different corrective actions when the metric triggers at warning severity or at critical severity.

Corrective actions must run using the credentials of a specific Enterprise Manager administrator. For this reason, whenever a corrective action is created or modified, the credentials that the modified action will run with must be specified.

Blackouts

Blackouts allow you to support planned outage periods to perform emergency or scheduled maintenance. When a target is put under blackout, monitoring is suspended, thus preventing unnecessary alerts from being sent when you bring down a target for scheduled maintenance operations such as database backup or hardware upgrade. Blackout periods are automatically excluded when calculating a target's overall availability.

A blackout period can be defined for individual targets, a group of targets or for all targets on a host. The blackout can be scheduled to run immediately or in the future, and to run indefinitely or stop after a specific duration. Blackouts can be created on an as-needed basis, or scheduled to run at regular intervals. If, during the maintenance period, you discover that you need more (or less) time to complete maintenance tasks, you can easily extend (or stop) the blackout that is currently in effect. Blackout functionality is available from both the Enterprise Manager console as well as via the Enterprise Manager command-line interface (EMCLI). The EMCLI is often useful for administrators who would like to incorporate the blacking out of a target within their maintenance scripts. When a blackout ends, the Management Agent automatically re-evaluates all metrics for the target to provide current status of the target post-blackout.

If an administrator inadvertently performs scheduled maintenance on a target without first putting the target under blackout, these periods would be reflected as target downtime instead of planned blackout periods. This has an adverse impact on the target's availability records. In such cases, Enterprise Manager allows Super Administrators to go back and define the blackout period that should have happened at that time. The ability to create these retroactive blackouts provides Super Administrators with the flexibility to define a more accurate picture of target availability.

Monitoring Templates

Monitoring templates simplify the task of standardizing monitoring settings across your enterprise by allowing you to specify the monitoring settings once and apply them to your monitored targets. This makes it easy for you to apply specific monitoring settings to specific classes of targets throughout your enterprise. For example, you can define one monitoring template for test databases and another monitoring template for production databases.

A monitoring template defines all Enterprise Manager parameters you would normally set to monitor a target, such as:

- Target type to which the template applies.

- Metrics (including user-defined metrics), thresholds, metric collection schedules, and corrective actions.

When a change is made to a template, you can reapply the template across affected targets in order to propagate the new changes. You can reapply the monitoring templates as often as needed. For any target, you can preserve custom monitoring settings by specifying metric settings that can never be overwritten by a template.

Comparing Differences Between Targets and Monitoring Templates

Deciding how and when to apply a template is simplified by using the Compare Monitoring Template feature. This feature allows you to see at a glance how metric and policy settings defined in a template differ from those defined on the destination target. Compare Monitoring Template is especially useful when working with aggregate targets such as groups and systems. For example, after you apply a Monitoring Template to a group, you want to verify that the group members now have the same monitoring settings as the template. The Compare Monitoring Template feature makes checking simple. You can also schedule this as a report, allowing you to check periodically if the group members still follow the template settings.

User-Defined Metrics

User-defined metrics allow you to extend the reach of Enterprise Manager's monitoring to conditions specific to particular environments via custom scripts or SQL queries and function calls. Once a user-defined metric is defined, it will be monitored, aggregated in the repository, and can trigger alerts like any other metric in Enterprise Manager. There are two types of user-defined metrics: Operating System and SQL.

- Operating System (OS) User-Defined Metrics: Accessed from Host target home pages, these user-defined metrics allow you to implement custom monitoring functions via OS scripts.

- SQL User-Defined Metrics: Accessed from the Database target home pages, these user-defined metrics allow you to implement custom database monitoring using SQL queries or function calls.

Creating a User-Defined Metric

To monitor a particular condition (example: check successful completion of monthly system maintenance routines), you can write a custom OS script to monitor that condition, then register it as a user-defined metric in Enterprise Manager. Each time the metric is evaluated by Enterprise Manager, it uses this script to evaluate the condition. SQL user-defined metrics do not use external scripts: you enter SQL directly into the Enterprise Manager console at the time of metric creation. Once a user-defined metric is defined, all other monitoring features, such as alerts, notifications, historical collections, and corrective actions are automatically available to it.

If you already have your own library of custom monitoring scripts, you can leverage Enterprise Manager's monitoring features by integrating these scripts with Enterprise Manager as OS user-defined metrics. Likewise, existing SQL queries or function calls currently used to monitor database conditions can be easily integrated into Enterprise Manager's monitoring framework as SQL user-defined metrics. For more information about user-defined metrics, see Oracle Enterprise Manager Advanced Configuration.

Accessing Monitoring Information

All monitoring information is accessed via the Enterprise Manager console, providing quick views into the health of your monitored environment.

Enterprise Manager Console Home Page

The Enterprise Manager console home page shown in Figure 1-1 gives you an at-a-glance view of the overall status of your monitored environment. As shown in the following figure, the home page summarizes key monitoring areas such as availability across all managed targets, open alerts, policy violations, and recent problems with job executions. Links on this page allow you to drill down to detailed performance information.

The Resource Center is your central access point to Enterprise Manager documentation as well as the comprehensive technical resources of the Oracle Technology Network (OTN).

Figure 1-1 Enterprise Manager Console

Description of "Figure 1-1 Enterprise Manager Console"

From the home page, you can easily access alert information. For example, you can click on the Down link in the All Targets Status legend to determine which targets are currently down. Under All Target Alerts, you can click on the Warning alerts value to access a list of warning alerts for all monitored targets (Figure 1-2).

Figure 1-2 Warning Alerts Page

The most recent alerts are listed first. You can change the sorting methodology by clicking on the appropriate column header. By clicking on a specific alert message, you can drill down to explicit details about the metric in alert ().

Figure 1-3 Warning Alert: Metric Details

Description of "Figure 1-3 Warning Alert: Metric Details"

By default, metric values shown on this page reflect the last 24 hours of collected data. You can also select another time period or specify a custom time period with which to view metric data and easily assess if the problem occurred recently or across a long time period. Because Enterprise Manager collects and aggregates metric data in the Management Repository, you can click on the Compare Targets related link to display metric data for more than one target simultaneously, thus allowing you to compare performance across multiple targets (Figure 1-4).

Figure 1-4 Compare Targets

Description of "Figure 1-4 Compare Targets"

The Alert History table shows alerts generated over the selected time period. You can view explicit details about a specific alert in this table by clicking on the eyeglasses icon in the Details column. Figure 1-5 shows the Alert Details page.

Figure 1-5 Alert Details

Description of "Figure 1-5 Alert Details"

The Alert Details page shows all notifications for an alert, any corrective actions that have been executed, and any custom notifications, for example, the opening of a case ticket for an alert. On this page, you also have the option of adding annotations or comments for other administrators to see.

2 Setting Up Enterprise Manager

Before you begin using Enterprise Manager, establish standard administrative settings in the Setup and Privileges pages, such as roles, notifications, passwords, templates, plug-ins, and more. These settings will help you manage user administration, communication, notification, standard actions and access, and security.

This chapter contains the following sections:

- Grid Control Setup

- Grid Control Preferences

Grid Control Setup

The Grid Control Setup area allows you to configure the following:

- Roles

- Administrators

- Notification Methods

- Patching Setup

- Blackouts

- Registration Passwords

- Management Pack Access

- Monitoring Templates

- Corrective Action Library

- Management Plug-ins

- Client System Analyzer in Grid Control

Figure 2-1 Grid Control's Overview of Setup Page

Description of "Figure 2-1 Grid Control's Overview of Setup Page"

Roles

Roles allow you to group Enterprise Manager system and target privileges, and grant these to administrators or to other roles. Privileges give the administrator rights to perform management actions within Enterprise Manager. Using roles and privileges allows you to manage security across all functional areas of Enterprise Manager.

Creating roles is an easy way to grant a predefined set of privileges to a group of administrators. If you change a role, the changes are automatically propagated to all administrators who are assigned that role.

Roles can be based on:

- Geographic location. For example, you can define a role for UK administrators to manage UK systems.

- Line of business. For example, you can define a role for administrators of the human resource systems.

- Any other IT model.

Administrators

Administrators are database users defined in the Management Repository who can log in to Enterprise Manager to perform management tasks. The breadth of management tasks available in Enterprise Manager depends on the privileges and roles assigned to the administrators.

There are two types of accounts:

- Super Administrator Account

- Administrator Account

Super Administrator Account

Enterprise Manager is installed with a default super administrator account named SYSMAN. During the installation, you are prompted provide a password for SYSMAN. You use the SYSMAN account for the initial login to Enterprise Manager. The super administrator account cannot be deleted or renamed.

The super administrator account can manage all other administrator accounts and set up all administrator credentials. The super administrator can:

- Create Enterprise Manager privileges and roles

- Perform the initial setup of Enterprise Manager, for example, defining e-mail configurations and defining global notifications rules

- Add targets to Enterprise Manager

- Perform any action on any target in the system

Oracle recommends that you create an administrator account for each individual user. The super administrator account should not be used to conduct daily administration tasks.

Administrator Account

Administrator accounts provide users permission to perform administrative tasks and access administrative information. You can set up each administrator account to have its own:

- E-mail address

- Notification rules

- Privileges

- Role, or set of privileges

See Also:

"About Administrators and Roles" in the Enterprise Manager online help

Notification Methods

Notification Methods allow you to define different mechanisms for sending notifications. These include e-mail, SNMP traps, or running custom scripts—or all three. Once defined, these methods can then be used with Notification Rules for sending notifications to administrators as a result of alert occurrences.

See Also:

"Notifications" in Chapter 4, "Notifications"

Patching Setup

On the Patching Setup page, specify parameters that allow you to configure and use Grid Control's patching features. These features simplify the staging and application of Oracle patches and patch sets to any host on which the Oracle Management Agent is running. They also allow you to collect information about Oracle critical patch advisories that are relevant to your enterprise.

You can use this page to set your MetaLink credentials, set patch cache size, turn on patch set validation, configure proxy and connection details, as well as offline patching settings.

Blackouts

Blackouts allow Enterprise Manager administrators to suspend any data collection activity on one or more monitored targets, thus allowing Enterprise Manager administrators to perform scheduled maintenance on targets. For example, you can stop data collections during periods where a managed target is undergoing routine maintenance, such as a database backup or hardware upgrade. If you continue monitoring during these periods, the collected data will show trends and other monitoring information that are not the result of normal day-to-day operations. To get a more accurate, long-term picture of a target's performance, you can use blackouts to exclude these special-case situations from data analysis.

A blackout can be defined for individual target(s), a group of multiple targets that reside on different hosts, or for all targets on a host. The blackout can be scheduled to run immediately or in the future, and to run indefinitely or stop after a specific duration. Blackouts can be created on an as-needed basis, or scheduled to run at regular intervals. If, during the maintenance period, the administrator discovers that he needs more (or less) time to complete his maintenance tasks, he can easily extend (or stop) the blackout that is currently in effect.

See Also:

"Blackouts" in Chapter 1, "Monitoring"

Registration Passwords

Set registration passwords to secure Management Agents against any Management Service that uses the central Management Repository.

When defining registration passwords, you may specify an expire date after which the password will be invalid, or you may specify that the password can only be used one time, after which it is deleted automatically from the repository.

Management Pack Access

Oracle offers a number of management options, or "Management Packs," that bring together a set of technologies related to a specific area of Enterprise Manager.

Each pack has several premium features bundled as part of that pack. Having a pack licensed or unlicensed on a target has a very significant impact on the user experience because the corresponding links, which need the pack, are enabled or disabled accordingly.

Monitoring Templates

Monitoring templates simplify the task of setting up monitoring for large numbers of targets by allowing administrators to specify the monitoring settings once, then apply them as often as needed. You can create, edit, and save a monitoring template that contains your enterprise's standards for monitoring—metrics, thresholds, corrective actions and/or policy rules. Once defined, you can propagate these standards by applying the template to managed targets.

See Also:

"Monitoring Templates" in Chapter 1, "Monitoring"

Corrective Action Library

Corrective Actions allow administrators to specify automated responses to alerts or policy violations. Corrective Actions ensure that routine responses to alerts or policy violations are automatically executed, thereby saving administrator time and ensuring problems are dealt with before they noticeably impact end users.

Corrective actions share many features in common with the Job System. By default, the corrective action will run on the target on which the alert or policy violation has triggered. Alternatively, a corrective action can be specified to contain multiple tasks, with each task running on a different target. Administrators can also receive notifications for the success or failure of corrective actions.

Use this page to display and manage corrective actions. The library allows administrators to re-use library corrective actions that have been created.

See Also:

"Defining Corrective Actions" in Chapter 13, "Managing Compliance"

Management Plug-ins

A Management Plug-in is a target type provided by the user or a third party to extend Enterprise Manager's set of predefined target types. This page is used to define new Management Plug-ins, import Management Plug-ins from, or export Management Plug-ins to a Management Plug-in Archive, or to deploy a Management Plug-in into your system.

The Tool Center is the central access point for all of the functionality available to you to operate on your Management Plug-ins. Any Management Plug-ins that have been imported into the Management Repository are listed on this page. Certain operations allow you to operate on multiple Management Plug-ins simultaneously (delete and export); some operations only allow you to operate on a single Management Plug-in at a time (deploy and undeploy); and import allows you to add Management Plug-ins to the table on this page. You also can search through the Management Plug-ins using the search criteria at the top of the page.

Client System Analyzer in Grid Control

The Client System Analyzer (CSA) allows Web server administrators to collect and analyze end-user client data. End-users can directly access CSA or may be redirected to it from other applications.

An instance of the Client System Analyzer application is pre-installed with Grid Control. The application is run by the Oracle Management Services' web servers. It can be used to collect client information for any system that can reach the Oracle Management Services URLs. This option allows administrators to collect client data without setting up a separate web application.

Grid Control Preferences

The Grid Control Preferences area allows you to set options for the following:

- General

- Preferred Credentials

- Notification

- Target Subtabs

General

You can specify one or more e-mail addresses and associated message formats for your Enterprise Manager account.

Specify the e-mail addresses and message formats you want associated with your Enterprise Manager account. All e-mail notifications you receive from Enterprise Manager will be sent to the e-mail addresses you specify.

Preferred Credentials

Preferred credentials simplify access to managed targets by storing target login credentials in the Management Repository. With preferred credentials set, users can access an Enterprise Manager target that recognizes those credentials without being prompted to log into the target. Preferred credentials are set on a per user basis, thus ensuring the security of the managed enterprise environment.

The Preferred Credentials page provides a top-level view of all preferred credentials set for your Enterprise Manager environment. From this page, you can set preferred credentials for any managed target listed in the Preferred Credentials table. Each row of the table provides a credential summary for each target. Targets are organized according to target type.

Notification

From Preferences, you can set up notification rules and the schedule.

Notification Rules

If you want to define a target or set of targets, the target availability states for which you would like to receive notifications, the metrics you want to monitor for thresholds, the policies you want to monitor for thresholds, the job status changes you want to monitor, and the notification action you want to take when a threshold is exceeded or a job's status changes, click the Create button.

Use the Notification Rules page to create, view, or edit notification rules. You can choose the targets and conditions for which you want to receive notifications from Enterprise Manager. The methods for sending notifications include e-mail, SNMP traps, running custom scripts, PL/SQL procedures, or all four.

See Also:

"Notifications" in Chapter 1, "Monitoring"

Notification Schedule

Use the notification schedule page to select a user for which a notification schedule needs to be set. An e-mail address must be defined for the administrator prior to defining a notification schedule. For each administrator for which a notification schedule has been specified, all specifics pertaining to their schedule are displayed.

See Also:

"Notifications" in Chapter 1, "Monitoring"

Target Subtabs

Use the Target Subtabs page to select and order the categories subtabs that appear within the Targets tab. The categories you create appear as subtabs under the Targets page.

3 Enterprise Manager Security

This chapter describes how to configure Oracle Enterprise Manager Security. Specifically, this chapter contains the following sections:

- About Oracle Enterprise Manager Security

- Enterprise Manager Authentication

- Enterprise Manager Authorization

- Configuring Security for Grid Control

- Accessing Managed Targets

- Cryptographic Support

- Setting Up the Auditing System for Enterprise Manager

- Additional Security Considerations

About Oracle Enterprise Manager Security

Oracle Enterprise Manager provides tools and procedures to help you ensure that you are managing your Oracle environment in a secure manner. The goals of Oracle Enterprise Manager security are:

- To be sure that only users with the proper privileges have access to critical monitoring and administrative data.
This goal is met by requiring username and password credentials before users can access the Enterprise Manager consoles and appropriate privileges for accessing the critical data.

- To be sure that all data transferred between Enterprise Manager components is transferred in a secure manner and that all data gathered by each Oracle Management Agent can be transferred only to the Oracle Management Service for which the Management Agent is configured.
This goal is met by enabling Enterprise Manager Framework Security. Enterprise Manager Framework Security automates the process of securing the Enterprise Manager components installed and configured on your network.

- To be sure that sensitive data such as credentials used to access target servers are protected.
This goal is met by Enterprise Manager's encryption support. The sensitive data is encrypted with an emkey. By following the best practice, even the repository owner and the SYSDBA will not be able to access the sensitive data.

- To be sure that access to managed targets is controlled through user authentication and privilege delegation.
This goal is met by configuring the Management Agent with PAM and LDAP for user authentication and using privilege delegation tools like Sudo and PowerBroker.

Enterprise Manager Authentication

Grid Control Authentication is the process of determining the validity of the user accessing Enterprise Manager Grid Control. The authentication feature is available across the different user interfaces such as Enterprise Manager Grid Control console and Enterprise Manager Command Line Interface.

The following authentication schemes are available:

- Repository-Based Authentication: This is the default authentication option. An Enterprise Manager administrator is also a repository (database) user. By using this option, you can take advantage of all the benefits that this authentication method provides like password control via password profile, enforced password complexity, password life time, number of failed attempts allowed and controls. During the password grace period, the administrator is prompted to change the password but when the password has expired, it must be changed.

- SSO-Based Authentication: The single sign-on based authentication provides strengthened and centralized user identity management across the enterprise. After you have configured Enterprise Manager to use the Oracle Application Server Single Sign-On, you can register any single sign-on user as an Enterprise Manager administrator. You can then enter your single sign-on credentials to access the Oracle Enterprise Manager Grid Control console. For more details, refer to Single Sign-On Based Authentication.

- Enterprise User Security Based Authentication: The Enterprise User Security (EUS) option enables you to create and store enterprise users and roles for the Oracle database in an LDAP-compliant directory server. Once the repository is configured with EUS, you can configure Enterprise Manager to use EUS as its authentication mechanism as described in Enterprise User Security Based Authentication. You can register any EUS user as an Enterprise Manager administrator.
EUS helps centralize the administration of users and roles across multiple databases. If the managed databases are configured with EUS, the process of logging into these databases is simplified. When you drill down to a managed database, Enterprise Manager will attempt to connect to the database using Enterprise Manager Grid Control credentials. If successful, Enterprise Manager will directly connect you to the database without displaying a login page.

Repository-Based Authentication

Enterprise Manager Grid Control allows you to create and manage new administrator accounts. Each administrator account includes its own login credentials as well as a set of roles and privileges that are assigned to the account. You can also assign a password profile to the administrator. To create, edit, or view an administrator account:

1. From Enterprise Manager Grid Control, click Setup.

2. Click Administrators in the vertical navigation bar.

3. Click the appropriate task button on the Administrators page. The following screen is displayed:

Figure 3-1 Create / Edit Administrator

On this page, you can specify the type of administrator account being created, select the password profile, and the password expiry period. For more details, refer to the Enterprise Manager Online Help.

Single Sign-On Based Authentication

If you are currently using Oracle Application Server Single Sign-On to control access and authorization for your enterprise, you can extend those capabilities to the Grid Control Console.

By default, when you navigate to the Grid Control Console, Enterprise Manager displays the Enterprise Manager login page. However, you can configure Enterprise Manager so it uses Oracle Application Server Single Sign-On to authenticate your Grid Control Console users. Instead of seeing the Enterprise Manager login page, Grid Control Console users will see the standard Oracle Application Server Single Sign-On login page. From the login page, administrators can use their Oracle Application Server Single Sign-On credentials to access the Oracle Enterprise Manager 10g Grid Control Console.

Note:

- You can configure Enterprise Manager Grid Control to either use Oracle Application Server Single Sign-On or the Enterprise User Security features. You cannot use both options at the same time.

- When Enterprise Manager is configured to use Single Sign-On with Server Load Balancer, make sure that the correct monitoring settings have been defined. For details, refer to the chapter on Grid Control Common Configurations.

The following sections describe how to configure Enterprise Manager as an OracleAS Single Sign-On Partner Application:

- Configuring Enterprise Manager to Use the Single Sign-On Logon Page

- Registering HTTP Port With Single Sign On Server

- Configuring Enterprise Manager to Use Single Sign-On with the osso.conf File

- Registering Single Sign-On Users as Enterprise Manager Administrators

- Grid Control as a Single Sign-On Partner Application

- Bypassing the Single Sign-On Logon Page

Configuring Enterprise Manager to Use the Single Sign-On Logon Page

To configure the Grid Control Console for use with Oracle Application Server Single Sign-On:

1. Set the ORACLE_HOME environment variables to the Management Service home directory.
For example:

$PROMPT> setenv ORACLE_HOME /dev01/oracle/em10g_GridControl

2. Change directory to the bin directory of the Management Service Oracle home:

$PROMPT> cd $ORACLE_HOME/opmn/bin

3. Stop the Management Service, the Oracle HTTP Server, and the other components of the application server:

$PROMPT> ./opmnctl stopall

4. Change directory to the bin directory of the Management Service Oracle home:

$PROMPT> cd $ORACLE_HOME/bin

5. Enter the following command at the operating system prompt:

$PROMPT> ./emctl config oms sso -host ssoHost -port ssoPort -sid ssoSid  -pass ssoPassword -das http://ssohost:port/

For example:

$PROMPT> ./emctl config oms sso -host sshost1.acme.com -port 1521 -sid asdb -pass Ch22x5xt -das http://ssohost1.acme.com:7777

Table 3-1 describes the arguments on the emctl config oms sso command line.

shows the typical output generated by the emctl config oms sso command.

Note:

- You can use the osso.conf file to configure the Grid Control for use with Oracle Application Server Single Sign-On. For more details, refer to Registering HTTP Port With Single Sign On Server.

- By default the emctl config oms sso command registers https URL with the SSO server. You can use the -unsecure option to register the http URL.

6. Restart the Management Service, Oracle HTTP Server, and the other application server components:

$PROMPT> cd $ORACLE_HOME/opmn/bin
$PROMPT> ./opmnctl startall

7. Go the Grid Control Console URL.
For example:

https://mgmthost1.acme.com:7777/em

The browser is redirected to the standard Single Sign-On Logon page.

Note:

You can remove the single sign-on configuration and restore Grid control authentication by using the following command:
emctl config oms sso -remove

Table 3-1  Arguments for the emctl sso Command

Argument

Description

-host

The name of the host computer where the Oracle Application Server Single Sign-On server resides. Be sure to use the fully-qualified host name.

-port

The port for the Oracle Application Server Single Sign-On database, for example, 1521.

-sid

The system identifier (SID) for the Oracle Application Server Single Sign-On database.

-pass

The password for the Oracle Application Server Single Sign-On schema (orasso). The orasso schema password is randomized when the Oracle Application Server infrastructure is installed.

To obtain the password, see "Obtaining the Single Sign-On Schema Password" in the Oracle Application Server Single Sign-On Administrator's Guide.

-das

The URL containing the host and port for the Delegated Administration Service (DAS). Generally, the DAS host name and port are the same as the host name and port of the Oracle Application Server Single Sign-On server. For example:

http://mgmthost1.acme.com:7777

-ossoconf

This is the fully qualified name of the osso.conf file.

-unsecure

This parameter is used to register the http port with the Single Sign-On Server. This is an optional parameter.

-sitename

The sitename that will be used to list the partner applications with the Single Sign-On Server.

-emurl

This parameter is used to specify the URL for the Load Balancer. If this parameter is not specified, the URL is constructed using the default host and port settings.

Example 3-1 Sample Output of the emctl config oms sso Command

$prompt> ./emctl config oms sso -host
<host>.com -port 1521 -sid orcl -pass W5RB9YD3 -das
http://<host>.com:7777 oracle

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.
/scratch/smptest/mm9/oms10g/Apache/Apache/conf/httpd.conf has been modified.
/scratch/smptest/mm9/oms10g/sysman/config/emoms.properties has been
modified.
Registering to SSO server, please wait...
Parameters passed to SSO registration tool :
param0:-oracle_home_path param1:/scratch/smptest/mm9/oms10g param2:-host
param3:
<host>.com param4:-port param5:1521 param6:-sid param7:orcl
param8:
-schema param9:orasso param10:-pass param11:**** param12:-site_name
param13:stam
<host>.com:4889 param14:-success_url
param15:http://<host>.com
:4889/osso_login_success param16:-logout_url
param17:http://<host>.com:4889/osso_logout_success param18:-cancel_url
param19:http://stamt03.us.oracle.
com:4889/ param20:-home_url param21:http://<host>.com:4889/
param22:-
config_mod_osso param23:TRUE param24: param25:oracle
param26:-sso_server_versi
on param27:v1.2  -DinstallType=
-DoldOracleHome=
-DoldOHSUser=root
Check /scratch/smptest/mm9/oms10g/sso/log/ssoreg.log for details of this registration
SSO registration tool finished successfully.
Done!

Registering HTTP Port With Single Sign On Server

By default, the https port is registered with the Single Sign-on server. If you want to register the http port with the Single Sign-on server, you can specify the -unsecure parameter as follows

$prompt>emctl config oms sso -host ssoHost -port ssoPort -sid ssoSid -pass ssoPassword -das dasURL -unsecure

where:

- host ssoHost: is the fully qualified host name.

- port ssoPort: is the listener port for the SSO database.

- sid ssoSid: is the SSO database SID

- pass ssoPassword: is the orasso user password. This parameter is optional.

- das dasURL: stands for the http://host:port where host is the ssoHost, port is the http port for the OIDDAS URL.

- unsecure: is used to register the http port with the Single Sign-On Server.

Configuring Enterprise Manager to Use Single Sign-On with the osso.conf File

In some environments, the Single Sign-On Server may be managed by different administrators who may not share the credentials. The osso.conf file can be used or the necessary parameter values required to create the osso.conf file can be specified.

To register Enterprise Manager as a partner application manually, follow these steps:

1. Enter the following URL to navigate to the SSO Administration page.

http://sso_host:sso_port/pls/orasso

2. Login as orcladmin user and click SSO Administration.

3. Click Administer Partner Applications and then click Add Partner Application.

4. Enter the following information on the Add Partner Application page.

Name: <EMPartnerName>
Home URL: protocol://em_host:em_port
Success URL: protocol://em_host:em_port/osso_login_success
Logout URL: protocol://em_host:em_port/osso_logout_success
Administrator Email: user@host.com

where host, port, and protocol refer to the EM Host, port and the protocol (http or https) used.

5. After entering these details, click Edit <EMPartnerName> and enter the following parameters to generate the osso.txt. Sample values for these parameters are shown below:

sso_server_version: v1.2
cipher_key: <EncryptionKeyValue>
site_id: <IDValue>
site_token: <TokenValue>
login_url: protocol://sso_host:sso_port/pls/orasso/orasso.wwsso_app_admin.lslogin
logout_url=protocol://sso_host:sso_port/pls/orasso/orasso.wwsso_app_admin.ls_logout
cancel_url=protocol://em_host:em_port
sso_timeout_cookie_name=SSO_ID_TIMEOUT
sso_timeout_cookie_key=9E231B3C1A3A808A

6. Enter the following command to generate the osso.conf file:

$ORACLE_HOME/Apache/Apache/bin/iasobf osso.txt osso.conf root

7. Use the osso.conf file for registration as follows:

$emctl config oms sso -ossoconf osso.conf -das http://sso_host:sso_port/

8. Restart Apache and OMS as follows:

Opmnctl stopall
Opmnctl startall

Registering Single Sign-On Users as Enterprise Manager Administrators

After you have configured Enterprise Manager to use the Single Sign-On logon page, you can register any Single Sign-On user as an Enterprise Manager administrator. You can register single sign-on users using:

- Enterprise Manager Grid Control Graphical User Interface

- Enterprise Manager Grid Control Command Line Interface

Registering Single Sign-On Users Using the Graphical User Interface

You can use the graphical user interface to register single sign-on users by following these steps:

1. Go the Enterprise Manager Grid Control Console URL.
For example:

http://mgmthost1.acme.com:7777/em

The browser is redirected to the standard Single Sign-On Logon page.

2. Enter the credentials for a valid Single Sign-On user.
If the Single Sign-On user is not an Enterprise Manager administrator, the browser is redirected to a modified version of the Enterprise Manager logon page (Figure 3-2).

3. Log in to Enterprise Manager as a Super Administrator.

4. Click Setup and then click Administrators to display the Administrators page.

See Also:

"Creating, Editing, and Viewing Administrators" in the Enterprise Manager online Help

Because Enterprise Manager has been configured to use Single Sign-On, the first page in the Create Administrator wizard now offers you the option of creating an administrator based on a registered Oracle Internet Directory user.

5. Select Oracle Internet Directory and advance to the next page in the wizard.

6. Enter the name and e-mail address of the Oracle Internet Directory user, or click the flashlight icon to search for a user name in the Oracle Internet Directory.

7. Use the rest of the wizard pages to define the roles, system privileges, and other characteristics of the Enterprise Manager administrator and then click Finish.
Enterprise Manager displays a summary page that lists the characteristics of the administrator account.

8. Click Finish to create the new Enterprise Manager administrator.
The OID user is now included in the list of Enterprise Manager administrators. You can now verify the account by logging out of the Grid Control Console and logging back in using the OID user credentials on the Single Sign-On logon page.

Figure 3-2 Modified Enterprise Manager Logon Page When Configuring SSO

Description of "Figure 3-2 Modified Enterprise Manager Logon Page When Configuring SSO"

Figure 3-3 Create Administrator Page When SSO Support Is Enabled

Description of "Figure 3-3 Create Administrator Page When SSO Support Is Enabled"

Registering Single Sign-On Users Using EMCLI

s

You can use the following EMCLI command to create Single Sign-On users:

./emcli create_user -name=ssouser -type=EXTERNAL_USER

This command creates a user with the name ssouser who is authenticated against the single sign-on user.

Argument

Description

-name

Name of the administrator.

-type

The type of user. The default value for this parameter is EM_USER. The other possible values are:

- EXTERNAL_USER

- DB_EXTERNAL_USER

-password

The password for the administrator.

-roles

The list of roles that can be granted to this administrator.

-email

The list of email addresses for this administrator.

-privilege

The system privileges that can be granted to the administrator. This option can be specified more than once.

-profile

The name of the database profile. This is an optional parameter. The default profile used is DEFAULT.

-desc

The description of the user being added.

-expired

This parameter is used to set the password to "expired" status. This is an optional parameter and is set to False by default.

-prevent_change_password

When this parameter is set to True, the user cannot change the password. This is an optional parameter and is set to False by default.

input_file

This parameter allows the administrator to provide the values for any of these arguments in an input file. The format of value is name_of_argument:file_path_with_file_name.

Example 1

emcli create_user         -name="new_admin"         -password="oracle"         -email="first.last@oracle.com;joe.shmoe@shmoeshop.com"         -roles="public"         -privilege="view_job;923470234ABCDFE23018494753091111"         -privilege="view_target;<host>.com:host"

This example creates an Enterprise Manager administrator named new_admin. This administrator has two privileges: the ability to view the job with ID 923470234ABCDFE23018494753091111 and the ability to view the target <host>.com:host. The administrator new_admin is granted the PUBLIC role.

Example 2

emcli create_user         -name="User1"         -type="EXTERNAL_USER"         -input_file="privilege:/home/user1/priv_file"         Contents of priv_file are:           view_target;<host>.com:host

This example makes user1 which has been created externally as an Enterprise Manager user. user1 will have view privileges on <host>.com:host.

Example 3

emcli create_user         -name="User1"         -desc="This is temp hire."
-prevent_change_password="true"         -profile="MGMT_ADMIN_USER_PROFILE

This example sets user1 as an Enterprise Manager user with some description. The prevent_change_password is set to true to indicate that the password cannot be changed by user1 and the profile is set to MGMT_ADMIN_USER_PROFILE.

Example 4

emcli create_user         -name="User1"         -desc="This is temp hire."         -expire="true"

This example sets user1 as an Enterprise Manager with some description. Since the password is set to expire immediately, when the user logs in for the first time, he is prompted to change the password.

Grid Control as a Single Sign-On Partner Application

The emctl config oms sso command adds the Oracle Enterprise Manager 10g Grid Control Console as an Oracle Application Server Single Sign-On partner application. Partner applications are those applications that have delegated authentication to the Oracle Application Server Single Sign-On Server.

To see the list of partner applications, navigate to the following URL:

http://hostname:port/pls/orasso/orasso.home

For example:

http://ssohost1.acme.com:7777/pls/orasso/orasso.home

Bypassing the Single Sign-On Logon Page

After you configure Enterprise Manager to use the Single Sign-On logon page, you can bypass the Single Sign-On page at any time and go directly to the Enterprise Manager logon page by entering the following URL:

http://hostname.domain:port/em/console/logon/logon

For example:

http://mgmthost1.acme.com:7777/em/console/logon/logon

Enterprise User Security Based Authentication

Enterprise User Security enables you to create and store Oracle database information as directory objects in an LDAP-compliant directory server. For example, an administrator can create and store enterprise users and roles for the Oracle database in the directory, which helps centralize the administration of users and roles across multiple databases.

See Also:

"Enterprise User Security Configuration Tasks and Troubleshooting" in the Oracle Database Advanced Security Administrator's Guide

If you currently use Enterprise User Security for all your Oracle databases, you can extend this feature to Enterprise Manager. Configuring Enterprise Manager for use with Enterprise User Security simplifies the process of logging in to database targets you are managing with the Oracle Enterprise Manager 10g Grid Control Console.

To configure Enterprise Manager for use with Enterprise User Security:

1. Ensure that you have enabled Enterprise User Security for your Oracle Management Repository database, as well as the database targets you will be managing with the Grid Control Console. Refer to Oracle Database Advanced Security Administrator's Guide for details.

2. Change directory to the ORACLE_HOME/sysman/config directory and open the emoms.properties file with your favorite text editor.

3. Add the following entries in the emoms.properties file:

oracle.sysman.emSDK.sec.DirectoryAuthenticationType=EnterpriseUser
oracle.sysman.emSDK.sec.eus.Domain=<ClientDomainName> (For example: mydomain.com)
oracle.sysman.emSDK.sec.eus.DASHostUrl=<das_url> (For example: oracle.sysman.emSDK.sec.eus.DASHostUrl=http://my.dashost.com:7777 )

4. Save and close the emoms.properties file.

5. Stop the Oracle Management Service.

See Also:

"Controlling the Oracle Management Service"

6. Start the Management Service.

The next time you use the Oracle Enterprise Manager 10g Grid Control Console to drill down to a managed database, Enterprise Manager will attempt to connect to the database using Enterprise User Security. If successful, Enterprise Manager will connect you to the database without displaying a login page. If the attempt to use Enterprise User Security fails, Enterprise Manager will prompt you for the database credentials.

Registering Enterprise Users as Enterprise Manager Users

After you have configured Enterprise Manager to use Enterprise Users, you can register existing enterprise users as Enterprise Manager Users and grant them the necessary privileges so that they can manage Enterprise Manager effectively.

You can register existing enterprise users by using:

- Enterprise Manager Grid Control Graphic User Interface

- Enterprise Manager Command Line Interface

Registering Enterprise Users Using the Graphical User Interface

You can use the graphical user interface to register enterprise users by following these steps:

1. Log into Enterprise Manager as a Super Administrator.

2. Click Setup and then click Administrators to display the Administrators page. Since Enterprise Manager has been configured to use Enterprise Users, the first page of the Create Administrator wizard will provide the option to create an administrator based on a registered Oracle Internet Directory user or a normal database user.

3. Select Oracle Internet Directory and click Continue to go to the next page in the wizard.

4. Enter the name and e-mail address of the Oracle Internet Directory user or click the flashlight icon to search for a user name in the Oracle Internet Directory.

5. Use the rest of the wizard pages to define the roles, system privileges, and other characteristics of the Enterprise Manager administrator and then click Finish. Enterprise Manager displays a summary page that lists the characteristics of the administrator account.

6. Click Finish to create the new Enterprise Manager administrator.
The OID user is now included in the list of Enterprise Manager administrators. You can now verify the account by logging out of the Grid Control Console and logging back in using the OID user credentials on the Single Sign-On logon page.

Registering Enterprise Users Using the Command Line Interface

To register Enterprise Users as Enterprise Manager users using EMCLI, enter the following command:

./emcli create_user -name=eususer -type=DB_EXTERNAL_USER

This command registers the eususer as an Enterprise Manager user where eususer is an existing Enterprise User.

Enterprise Manager Authorization

System security is a major concern of any corporation. Giving the same level of access to all systems to all administrators is dangerous, but individually granting access to tens, hundreds, or even thousands of targets to every new member of the group is time consuming. With Enterprise Manager's administrator privileges and roles feature, this task can be performed within seconds, instead of hours. Authorization controls the access to the secure resources managed by Enterprise Manager via system, target, and object level privileges and roles.

This section describes Enterprise Manager's Authorization model including user classes, roles, and privileges assigned to each user class. The following topics are described:

- Classes of Users

- Privileges and Roles

- Authorization and Access Enforcement

Classes of Users

Oracle Enterprise Manager supports different classes of Oracle users, depending upon the environment you are managing and the context in which you are using Oracle Enterprise Manager Grid Control.

The Enterprise Manager administrators you create and manage in the Grid Control Console are granted privileges and roles to log in to the Grid Control Console and to manage specific target types and to perform specific management tasks. The default super administrator for the Grid Control Console is the SYSMAN user, which is a database user associated with the Oracle Management Repository. You define the password for the SYSMAN account during the Enterprise Manager installation procedure.

By restricting access to privileged users and providing tools to secure communications between Oracle Enterprise Manager 10g components, Enterprise Manager protects critical information in the Oracle Management Repository.

The Management Repository contains management data that Enterprise Manager Grid Control uses to help you monitor the performance and availability of your entire enterprise. This data provides you with information about the types of hardware and software you have deployed, as well as the historical performance and specific characteristics of the applications, databases, applications servers, and other targets that you manage. The Management Repository also contains information about the Enterprise Manager administrators who have the privileges to access the management data.

You can create and manage Enterprise Manager administrator accounts. Each administrator account includes its own login credentials, as well as a set of roles and privileges that are assigned to the account. There are three administrator access categories:

- Super Administrator: Powerful Enterprise Manager administrator with full access privileges to all targets and administrator accounts within the Enterprise Manager environment. The Super Administrator, SYSMAN is created by default when Enterprise Manager is installed. The Super Administrator can create other administrator accounts.

- Administrator: Regular Enterprise Manager administrator.

- Repository Owner: Database administrator for the Management Repository. This account cannot be modified, duplicated, or deleted.

The types of management tasks that the administrator can perform and targets that he can access depends on the roles, system privileges, and target privileges that he is granted. The Super Administrator can choose to let certain administrators perform only certain management tasks, or access only certain targets, or perform certain management tasks on certain targets. In this way, the Super Administrator can divide the workload among his administrators. To create, edit, or view an administrator:

1. Click Setup at the top of any Grid Control Console page.

2. Click Administrators. The Administrators page is displayed.

Figure 3-4 Administrators Page

3. Click the appropriate task button on the Administrators page.
Enterprise Manager displays a wizard page for the task you have chosen. Click Help from the wizard page for more information on administrators.

Note:

The interface for user creation may vary depending on the authentication scheme you have selected.

Leveraging Oracle Application Server Security Services

As a Web-based application, Enterprise Manager relies on industry-standard technologies to provide secure access to the Oracle Enterprise Manager 10g Grid Control Console and Application Server Control Console.

When you configure security for the Oracle Enterprise Manager 10g Grid Control Console, Enterprise Manager Framework Security provides secure communications between the components of your Enterprise Manager installation.

See Also:

"Configuring Security for Grid Control" for more information about the Enterprise Manager Framework Security
Oracle HTTP Server Administrator's Guide for information about configuring security for your Oracle HTTP Server

Enterprise Manager Grid Console application is deployed in a OPMN managed OC4J instance. When you configure security for the Grid Console, Enterprise Manager uses the standard security services of OC4J and OHS to protect your management data.

Privileges and Roles

User privileges provide a basic level of security in Enterprise Manager. They are designed to control user access to data and to limit the kinds of SQL statements that users can execute. When creating a user, you grant privileges to enable the user to connect to the database, to run queries and make updates, to create schema objects, and more.

When Enterprise Manager is installed, the SYSMAN user (super administrator) is created by default. The SYSMAN Super Administrator then creates other administrator accounts for daily administration work. The SYSMAN account should only be used to perform infrequent system wide, global configuration tasks.The Super Administrator divides workload among his administrators by filtering target access, or filtering access to management task, or both through the roles, System Privileges, and Target Privileges he grants them. For example, he can allow some administrators to view any target and to add any target in the enterprise and other administrators to only perform specific operations such as maintaining and cloning on a target for which they are responsible.

A role is a collection of Enterprise Manager system privileges, or target privileges, or both, which you can grant to administrators or to other roles. These roles can be based upon geographic location (for example, a role for Canadian administrators to manage Canadian systems), line of business (for example, a role for administrators of the human resource systems or the sales systems), or any other model. Administrators do not want to perform the task of individually granting access to tens, hundreds, or even thousands of targets to every new member of their group.By creating roles, an administrator needs only to assign the role that includes all the appropriate privileges to his team members instead of having to grant many individual privileges. He can divide workload among his administrators by filtering target access, or filtering access to management task, or both.

Public Role: Enterprise Manager creates one role by default called Public. This role is unique in that it is automatically assigned to all new non-super administrators when they are created. By default it has no privileges assigned to it. The Public role should be used to define default privileges you expect to assign to a majority of non-super administrators you create. Privileges need not be assigned to Public initially - they can be added at any time. The role may be deleted if your enterprise does not wish to use it. If deleted, it can be added back in later if you later decide to implement it.

Granting Privileges

A privilege is a right to perform management actions within Enterprise Manager. Privileges can be divided into three categories:

- System Privileges

- Target Privileges

- Object Privileges

System Privileges: These privileges allow a user to perform system wide operations and are listed below:

Table 3-2 System Privileges

System Privilege

Description

USE ANY BEACON

Allows the administrator to use any Beacon on any monitored host to monitor transactions, URLs, and network components.

ADD ANY TARGET

Allows the administrator to add any target to Enterprise Manager for monitoring, administration and management.

VIEW ANY TARGET

Allows the administrator to view any target on the system, including Oracle Management Agents and Management Services.Whenever the VIEW ANY TARGET privilege is granted, the MONITOR ENTERPRISE MANAGER privilege is also granted by default.

CREATE PRIVILEGE PROPAGATING GROUP

Allows the administrator to create privilege propagating groups. Privileges granted to such groups will be automatically granted to all members of the group.

MONITOR ENTERPRISE MANAGER

Allows the administrator to monitor the availability and performance of Enterprise Manager itself, and grants the administrator access to the following targets: the database used for the Management Repository, the Management Service and Management Repository, and all Oracle Management Agents in the global enterprise.

PUBLISH REPORT

Allows the administrator to publish reports for public use.

Target Privileges: These privileges allow an administrator to perform operations on a target.

Table 3-3 Target Privileges

Target Privilege

Description

FULL

Implicitly grants all the target privileges and allows the administrator to delete the target from the Enterprise Manager system.

OPERATOR

Allows the administrator to perform normal administrative operations on a target such as configure a blackout, or edit the properties.

BLACKOUT TARGET

Allows the administrator to create, edit, schedule, and stop blackout on a target.

MANAGE TARGET ALERTS

Allows the administrator to clear stateless alerts, manually re-evaluate alerts and acknowledge alerts for the target.

CONFIGURE TARGET

Allows the administrator to edit target properties and modify monitoring configurations.

MANAGE TARGET METRICS

Allows the administrator to edit thresholds for metric and policy settings, apply monitoring templates and manage user defined metrics.

VIEW

Allows the administrator to view properties, inventory and monitor information about a target.

Object Privileges: These privileges allow an administrator to perform a particular action on a specific schema object. Different object privileges are available for different types of schema objects.

Table 3-4 Object Privileges

Target Privilege

Description

VIEW JOB

Provides the administrator with the ability to view the job and its definition.

FULL JOB

Provides the administrator with the ability to view, edit, submit, and delete the job.

VIEW REPORT

Provides the administrator with the ability to view the report.

Configuring Security for Grid Control

This section contains the following topics:

- About Enterprise Manager Framework Security

- Overview of the Steps Required to Enable Enterprise Manager Framework Security

- Enabling Security for the Oracle Management Service

- Enabling Security for the Oracle Management Agent

- Enabling Security with Multiple Management Service Installations

- Restricting HTTP Access to the Management Service

- Managing Agent Registration Passwords

- Enabling Security with a Server Load Balancer

- Enabling Security for the Management Repository Database

About Enterprise Manager Framework Security

Enterprise Manager Framework Security provides safe and secure communication channels between the components of Enterprise Manager. For example, Framework Security provides secure connections between your Oracle Management Service and its Management Agents.

See Also:

Oracle Enterprise Manager Concepts for an overview of Enterprise Manager components

Enterprise Manager Framework Security works in concert with—but does not replace—the security features you should enable for your Oracle HTTP Server. Oracle HTTP Server is part of the Oracle Application Server instance that is used to deploy the Management Service J2EE Web application.

See Also:

Oracle Application Server 10g Security Guide

The following figure shows how Enterprise Manager Framework Security provides security for the connections between the Enterprise Manager components.

Figure 3-5 Enterprise Manager Framework Security

Enterprise Manager Framework Security implements the following types of secure connections between the Enterprise Manager components:

- HTTPS and Public Key Infrastructure (PKI) components, including signed digital certificates, for communications between the Management Service and the Management Agents.

See Also:

Oracle Security Overview for an overview of Public Key Infrastructure features, such as digital certificates and public keys

- Oracle Advanced Security for communications between the Management Service and the Management Repository.

See Also:

Oracle Database Advanced Security Administrator's Guide

Overview of the Steps Required to Enable Enterprise Manager Framework Security

To enable Enterprise Manager Framework Security, you must configure each of the Enterprise Manager components in a specific order. The following list outlines the process for securing the Management Service and the Management Agents that upload data to the Management Service:

Note:

The Enterprise Manager components are configured during installation. You can use the following commands if you want to reconfigure any of the components.

1. Use the opmnctl stopall command to stop the Management Service, the Oracle HTTP Server, and the other components of the Oracle Application Server that are used to deploy the Management Service.

2. Use emctl secure oms to enable security for the Management Service.

3. Restart the Management Service, the Oracle HTTP Server, OracleAS Web Cache, and the other application server components using the opmnctl startall command.

4. For each Management Agent, stop the Management Agent, use the emctl secure agent command to enable security for the Management Agent, and restart the Management Agent.

5. After security is enabled for all the Management Agents, use the emctl secure lock command to restrict HTTP Access to the Management Service. This will ensure that Management Agents for which security has not been enabled will not be able upload data to the Management Service.

The following sections describe how to perform each of these steps in more detail.

Note:

To resolve errors from emctl secure operations, refer to $ORACLE_HOME/sysman/log/secure.log for more details.

Enabling Security for the Oracle Management Service

To enable Enterprise Manager Framework Security for the Management Service, you use the emctl secure oms utility, which is located in the following subdirectory of the Management Service home directory:

$ORACLE_HOME/bin

The emctl secure oms utility performs the following actions:

- Generates a Root Key within your Management Repository. The Root Key is used during distribution of Oracle Wallets containing unique digital certificates for your Management Agents.

- Modifies your Oracle HTTP Server to enable an HTTPS channel between your Management Service and Management Agents, independent from any existing HTTPS configuration that may be present in your Oracle HTTP Server.

- Enables your Management Service to accept requests from Management Agents using Enterprise Manager Framework Security.

To run the emctl secure oms utility you must first choose an Agent Registration Password. The Agent Registration password is used to validate that future installation sessions of Oracle Management Agents and Oracle Management Services are authorized to load their data into this Enterprise Manager installation.

To enable Enterprise Manager Framework Security for the Oracle Management Service:

1. Change directory to the following directory in the Management Service home:

ORACLE_HOME/opmn/bin

2. Stop the Management Service, the Oracle HTTP Server, and the other application server components using the following command:

$PROMPT> ./opmnctl stopall

3. Change directory to the following directory in the Management Service home:

ORACLE_HOME/bin

4. Enter the following command:

$PROMPT> ./emctl secure oms

5. You will be prompted for the Enterprise Manager Root Password. Enter the SYSMAN password.

6. You will be prompted for the Agent Registration Password, which is the password required for any Management Agent attempting to secure with the Management Service. Specify an Agent Registration Password for the Management Service.

7. When the operation is complete, restart the Management Service, the Oracle HTTP Server, and OracleAS Web Cache:

$PROMPT> cd $ORACLE_HOME/opmn/bin
$PROMPT> ./opmnctl startall

8. After the Management Service restarts, test the secure connection to the Management Service by browsing to the following secure URL using the HTTPS protocol:

https://hostname.domain:https_upload_port/em

For example:

https://mgmthost1.acme.com:1159/em

If the Management Service security has been enabled, your browser displays the Enterprise Manager Login page.

Note:

The 1159 port number is the default secure port used by the Management Agents to upload data to the Management Service. This port number may vary if the default port is unavailable.

Caution:

While the emctl secure oms command provides immediate HTTPS browser access to the Grid Control Console by using the secure Management Agent upload port, it does not enable security for the default OracleAS Web Cache port that your administrators use to display the Grid Control Console.
To enable security for users who access the Grid Control through OracleAS Web Cache, refer to Oracle Application Server 10g Security Guide.

Example 3-2 Sample Output of the emctl secure oms Command

$PROMPT> ./emctl secure oms
Oracle Enterprise Manager 10g Release 5 Grid ControlCopyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.Securing OMS... Started.Securing OMS... Successful

Alternatively, you can enter the emctl secure oms command all on one line, but if you enter the command on one line, the passwords you enter will be displayed on the screen as you type the command.

Example 3-3 Usage of the emctl secure oms Command (II)

$PROMPT> emctl secure oms [-sysman_pwd <sysman password>][-reg_pwd <registration password>][-host <hostname>][-reset][-secure_port <secure_port>][-upload_http_port <upload_http_port>][-slb_port <slb port>][-slb_console_port <slb console port>][-root_dc <root_dc>][-root_country <root_country>][-root_state <root_state>][-root_loc <root_loc>][-root_org <root_org>][-root_unit <root_unit>]        [-root_email <root_email>][-wallet <wallet_loc> -trust_certs_loc <certs_loc>] [-wallet_pwd <pwd>][-key_strength <strength>][-cert_validity <validity>]

The parameters are explained below:

- sysman_pwd - Oracle Management Repository user password.

- reg_pwd - The Management Agent registration password.

- host - The host name to be used in the certificate used by the Oracle Management Service. You may need to use the SLB host name if there is an SLB before the Management Service.

- reset - If the Oracle Management Service is secured with this option, a new root certificate is generated. All the agents and the Oracle Management Services need to be resecured for use with the new root certificate.

- secure_port - The port to be used for secure communication. The default value is 4888.

- upload_http_port - The port used for upload communications.

- slb_port - This parameter is required when Server Load Balancer is used. It specifies the secure upload port configured in the Server Load Balancer.

- slb_console_port - This parameter is required when Server Load Balancer is used. It specifies the secure upload port configured in the Server Load Balancer.

- trust_certs_loc - The location of the trusted_certs.txt (required when third party certificates are used).

- root_dc - The domain component used in the root certificate. The default value is com.

- root_country - The country to be used in the root certificate. The default value is US.

- root_state - The state to be used in the root certificate. The default value is CA.

- root_loc - The location to be used in the root certificate. The default value is EnterpriseManager on <hostname>.

- root_org - The organization name to be used in the root certificate. The default value is EnterpriseManager on <hostname>.

- root_unit - The organizational unit to be used in the root certificate. The default value is EnterpriseManager on <hostname>.

- root_email - The email address to be used in the root certificate. The default value is EnterpriseManager@<hostname>.

- wallet: This is the directory where the wallet to be used in the https upload port is located.

- wallet_pwd: This is the wallet password and is required only if the wallet is not an SSO wallet.

- key_strength: The strength of the key to be used. Valid values are 512, 1024, 2048, and 4096.

- cert_validity: The number of days for which the self-signed certificate is valid. The valid range is between 1 to 3650.

Note:

The key_strength and cert_validity parameters are applicable only when the -wallet option is not used.

Checking the Security Status

You can check whether security has been enabled for the Management Service by entering the emctl status oms -secure command.

Example 3-4 Sample Output of the emctl secure status oms Command

$prompt> emctl status oms -secure
Oracle Enterprise Manager 10g Release 5 Grid Control
Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.
Checking the security status of the OMS at location set in /OH/oms10g/sysman/config/emoms.properties...  Done.
OMS is secure on HTTPS Port 1159

Enabling Security for the Oracle Management Agent

When you install the Management Agent on a host, you must identify the Management Service that will be used by the Management Agent. If the Management Service you specify has been configured to take advantage of Enterprise Manager Framework Security, you will be prompted for the Agent Registration Password and Enterprise Manager Framework Security will be enabled for the Management Agent during the installation.

Otherwise, if the Management Service has not been configured for Enterprise Manager Framework Security or if the Registration Password was not specified during installation, then security will not be enabled for the Management Agent. In those cases, you can later enable Enterprise Manager Framework Security for the Management Agent.

To enable Enterprise Manager Framework Security for the Management Agent, use the emctl secure agent utility, which is located in the following directory of the Management Agent home directory:

AGENT_HOME/bin (UNIX)
AGENT_HOME\bin (Windows)

The emctl secure agent utility performs the following actions:

- Obtains an Oracle Wallet from the Management Service that contains a unique digital certificate for the Management Agent. This certificate is required in order for the Management Agent to conduct SSL communication with the secure Management Service.

- Obtains an Agent Key for the Management Agent that is registered with the Management Service.

- Configures the Management Agent so it is available on your network over HTTPS and so it uses the Management Service HTTPS upload URL for all its communication with the Management Service.

To enable Enterprise Manager Framework Security for the Management Agent:

1. Ensure that your Management Service and the Management Repository are up and running.

2. Change directory to the following directory:

AGENT_HOME/bin (UNIX)
AGENT_HOME\bin (Windows)

3. Stop the Management Agent:

$PROMPT> ./emctl stop agent

4. Enter the following command:

$PROMPT> ./emctl secure agent (UNIX)
$PROMPT> emctl secure agent (Windows)

The emctl secure agent utility prompts you for the Agent Registration Password, authenticates the password against the Management Service, and reconfigures the Management Agent to use Enterprise Manager Framework Security.

Note:

Alternatively, you can enter the command all on one line, but if you enter the command on one line, the password you enter will be displayed on the screen as you type:

$PROMPT> ./emctl secure agent agent_registration_pwd (UNIX)
$PROMPT> emctl secure agent agent_registration_pwd (Windows)

shows sample output of the emctl secure agent utility.

5. Restart the Management Agent:

$PROMPT> ./emctl start agent

6. Confirm that the Management Agent is secure by checking the Management Agent home page.

Note:

You can also check if the Agent Management is secure by running the emctl status agent -secure command, or by checking the Agent and Repository URLs in the output of the emctl status agent command.

In the General section of the Management Agent home page (Figure 3-6), the Secure Upload field indicates whether or not Enterprise Manager Framework Security has been enabled for the Management Agent.

See Also:

"Checking the Status of an Oracle Management Agent" in the Enterprise Manager online Help

Example 3-5 Sample Output of the emctl secure agent Utility

$PROMPT> ./emctl secure agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.
Securing agent...   Started
Securing agent...   Successful.

Example 3-6 Sample Output of the emctl secure status agent Command

[oracle@stang14 bin]$ ./emctl status agent -secure
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.
Checking the security status of the Agent at location set in
/private/home/oracle/product/102/em/agent10g/sysman/config/emd.properties...
Done.
Agent is secure at HTTPS Port 3872.
Checking the security status of the OMS at
http://gridcontrol.oraclecorp.com:4889/em/upload/...  Done.
OMS is secure on HTTPS Port 4888

Figure 3-6 Secure Upload Field on the Management Agent Home Page

Enabling Security with Multiple Management Service Installations

If you already have a secure Management Service running and you install an additional Management Service that uses the same Management Repository, you will need to enable Enterprise Manager Framework Security for the new Management Service. This task is executed using the same procedure that you used to secure the first Management Service, by running the emctl secure oms utility.

Because you have already established at least one Agent Registration Password and a Root Key in your Management Repository, they must be used for your new Management Service. Your secure Management Agents can then operate against either Management Service.

All the registration passwords assigned to the current Management Repository are listed on the Registration Passwords page in the Oracle Enterprise Manager 10g Grid Control Console.

If you install a new Management Service that uses a new Management Repository, the new Management Service is considered to be a distinct enterprise. There is no way for the new Management Service to partake in the same security trust relationship as another Management Service that uses a different Management Repository. Secure Management Agents of one Management Service will not be able to operate against the other Management Service.

Restricting HTTP Access to the Management Service

By default, when you enable Enterprise Manager Framework Security on your Oracle Management Service there are no default restrictions on HTTP access. The Grid Control Console can also be accessed over HTTP and the Oracle Management Agents will be able to upload over HTTP as well as HTTPS.

However, it is important that only secure Management Agent installations that use the Management Service HTTPS channel are able to upload data to your Management Repository and Grid Control console is accessible via HTTPS only.

To restrict access so Management Agents can upload data to the Management Service only over HTTPS:

1. Stop the Management Service, the Oracle HTTP Server, and the other application server components:

$PROMPT> cd $ORACLE_HOME/opmn/bin
$PROMPT> ./opmnctl stopall

2. Change directory to the following location in the Management Service home:

$ORACLE_HOME/bin

3. Enter the following command to prevent Management Agents from uploading data to the Management Service over HTTP:

$PROMPT> emctl secure lock -upload

- Note:

- To lock the console and prevent HTTP access to the console, enter the following command:

emctl secure lock -console

- To lock both, enter either of the following commands:

emctl secure lock or
emctl secure lock -upload -console

- To lock both the console access and uploads from Agents while enabling security on the Management Service, enter the following command:

emctl secure oms -lock [other options]

4. Restart the Management Service, the Oracle HTTP Server, and the other application server components:

$PROMPT> cd $ORACLE_HOME/opmn/bin
$PROMPT> ./opmnctl startall

5. Verify that you cannot access the Management Agent upload URL using the HTTP protocol:
For example, navigate to the following URL:

http://hostname.domain:4889/em/upload

You should receive an error message similar to the following:

ForbiddenYou don't have permission to access /em/upload on this server

6. Verify that you can access the Management Agent Upload URL using the HTTPS protocol:
For example, navigate to the following URL:

https://hostname.domain:4888/em/upload

You should receive the following message, which confirms the secure upload port is available to secure Management Agents:

Http XML File receiverHttp Recceiver Servlet active!

To allow the Management Service to accept uploads from unsecure Management Agents, use the following command:

$PROMPT> emctl secure unlock -upload

Note:

- To unlock the console and allow HTTP access to the console, enter the following command:

emctl secure unlock -console

- To unlock both, enter either of the following command:

emctl secure unlock
emctl secur unlock -console -upload

Example 3-7 Sample Output of the emctl secure lock Command

$prompt> emctl secure lock
Oracle Enterprise Manager 10g Release 5 Grid Control
Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.
OMS Console is locked. Access the console over HTTPS ports.
Agent Upload is locked. Agents must be secure and upload over HTTPS port.

Example 3-8 Sample Output of the emctl secure unlock Command

$prompt> emctl secure unlock
Oracle Enterprise Manager 10g Release 5 Grid Control
Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.
OMS Console is unlocked. HTTP ports too can be used to access console.
Agent Upload is unlocked. Unsecure Agents may upload over HTTP.

To restrict HTTP access to the Oracle Enterprise Manager 10g Grid Control Console, use the emctl secure lock -console command.

Managing Agent Registration Passwords

Enterprise Manager uses the Agent Registration password to validate that installations of Oracle Management Agents are authorized to load their data into the Oracle Management Service.

The Agent Registration password is created during installation when security is enabled for the Oracle Management Service.

Note:

To avoid new Agents from being installed, you can delete all the registration passwords.

Using the Grid Control Console to Manage Agent Registration Passwords

You can use the Grid Control Console to manage your existing registration passwords or create additional registration passwords:

1. Click Setup at the top of any Grid Control Console page.

2. Click Registration Passwords.
Enterprise Manager displays the Registration Passwords page (Figure 3-7). The registration password you created when you ran the emctl secure oms command appears in the Registration Passwords table.

3. Use the Registration Passwords page to change your registration password, create additional registration passwords, or remove registration passwords associated with the current Management Repository.

Figure 3-7 Managing Registration Passwords in the Grid Control Console

When you create or edit an Agent Registration Password on the Registration Passwords page, you can determine whether the password is persistent and available for multiple Management Agents or to be used only once or for a predefined period of time.

For example, if an administrator requests to install a Management Agent on a particular host, you can create a one-time-only password that the administrator can use to install and configure one Management Agent.

On the other hand, you can create a persistent password that an administrator can use for the next two weeks before it expires and the administrator must ask for a new password.

Using emctl to Add a New Agent Registration Password

To add a new Agent Registration Password, use the following emctl command on the machine on which the Management Service has been installed:

$PROMPT> emctl secure setpwd [sysman pwd] [new registration pwd]

The emctl secure setpwd command requires that you provide the password of the Enterprise Manager super administrator user, sysman, to authorize the resetting of the Agent Registration Password.

If you change the Agent Registration Password, you must communicate the new password to other Enterprise Manager administrators who need to install new Management Agents, enable Enterprise Manager Framework Security for existing Management Agents, or install additional Management Services.

As with other security passwords, you should change the Agent Registration Password on a regular and frequent basis to prevent it from becoming too widespread.

Enabling Security with a Server Load Balancer

When you deploy a Management Service that is available behind a Server Load Balancer (SLB), special attention must be given to the DNS host name over which the Management Service will be available. Although the Management Service may run on a particular local host, for example myhost.mycompany.com, your Management Agents will access the Management Service using the host name that has been assigned to the Server Load Balancer. For example, oracleoms.mycompany.com.

As a result, when you enable Enterprise Manager Framework Security for the Management Service, it is important to ensure that the Server Load Balancer host name is embedded into the Certificate that the Management Service uses for SSL communications. To do so, enter the following commands:

This may be done by using emctl secure oms and specifying the host name in the with an extra -host parameter as follows:

- Specify the -host parameter with the emctl secure oms command as follows:
$PROMPT>emctl secure oms -host <hostname>

- Set UseCanonicalName directive to On in the OMS_Home/Apache/Apache/conf/httpd.conf file.

- Enable security on the Management Service by entering the following command:
$PROMPT>emctl secure oms -host <slb_hostname> [-slb_console_port <slb UI port>] [-slb_port <slb upload port>] [other params]

- Create virtual servers and pools on the Server Load Balancer.

- Verify that the console can be accessed using the following URL:
https://slbhost:slb_console_port/em

- Re-secure the Agents with Server Load Balancer by using the following command:
$PROMPT>emctl secure agent -emdWalletSrcUrl <SLB Upload url>

Enabling Security for the Management Repository Database

This section describes how to enable Security for the Oracle Management Repository. This section includes the following topics:

- About Oracle Advanced Security and the sqlnet.ora Configuration File

- Configuring the Management Service to Connect to a Secure Management Repository Database

- Enabling Oracle Advanced Security for the Management Repository

- Enabling Security for a Management Agent Monitoring a Secure Management Repository or Database

About Oracle Advanced Security and the sqlnet.ora Configuration File

You enable security for the Management Repository by using Oracle Advanced Security. Oracle Advanced Security ensures the security of data transferred to and from an Oracle database.

See Also:

Oracle Database Advanced Security Administrator's Guide

To enable Oracle Advanced Security for the Management Repository database, you must make modifications to the sqlnet.ora configuration file. The sqlnet.ora configuration file is used to define various database connection properties, including Oracle Advanced Security parameters.

The sqlnet.ora file is located in the following subdirectory of the Database home:

ORACLE_HOME/network/admin

After you have enabled Security for the Management Repository and the Management Services that communicate with the Management Repository, you must also configure Oracle Advanced Security for the Management Agent by modifying the sqlnet.ora configuration file in the Management Agent home directory.

See Also:

"Enabling Security for a Management Agent Monitoring a Secure Management Repository or Database"

It is important that both the Management Service and the Management Repository are configured to use Oracle Advanced Security. Otherwise, errors will occur when the Management Service attempts to connect to the Management Repository. For example, the Management Service might receive the following error:

ORA-12645: Parameter does not exist

To correct this problem, be sure both the Management Service and the Management Repository are configured as described in the following sections.

Note:

The procedures in this section describe how to manually modify the sqlnet.ora configuration file to enable Oracle Advanced Security. Alternatively, you can make these modifications using the administration tools described in the Oracle Database Advanced Security Administrator's Guide.

Configuring the Management Service to Connect to a Secure Management Repository Database

If you have enabled Oracle Advanced Security for the Management Service database—or if you plan to enable Oracle Advanced Security for the Management Repository database—use the following procedure to enable Oracle Advanced Security for the Management Service:

1. Stop the Management Service:

$PROMPT> ORACLE_HOME/bin/emctl stop oms

2. Locate the following configuration file in the Management Service home directory:

ORACLE_HOME/sysman/config/emoms.properties

3. Using a text editor, add the entries described in the emoms.properties file.
The entries described in the table correspond to valid parameters you can set when you configure network data encryption for the Oracle Database.

See Also:

"Configuring Network Data Encryption and Integrity for Oracle Servers and Clients" in the Oracle Application Server 10g Administrator's Guide

4. Save your changes and exit the text editor.

5. Restart the Management Service.

$PROMPT> ORACLE_HOME/bin/emctl start oms

Table 3-5  Oracle Advanced Security Properties in the Enterprise Manager Properties File

Property

Description

oracle.sysman.emRep.dbConn.enableEncryption

Defines whether or not Enterprise Manager will use encryption between Management Service and Management Repository.Possible values are TRUE and FALSE. The default value is FALSE.For example:

oracle.sysman.emRep.dbConn. enableEncryption=true

oracle.net.encryption_client

Defines the Management Service encryption requirement.Possible values are REJECTED, ACCEPTED, REQUESTED, and REQUIRED.The default value is REQUESTED. In other words, if the database supports secure connections, then the Management Service uses secure connections, otherwise the Management Service uses insecure connections.

For example:

oracle.net. encryption_client=REQUESTED

oracle.net.encryption_types_client

Defines the different types of encryption algorithms the client supports.Possible values should be listed within parenthesis. The default value is ( DES40C ).

For example:

oracle.net. encryption_types_client=( DES40C )

oracle.net.crypto_checksum_client

Defines the Client's checksum requirements.

Possible values are REJECTED, ACCEPTED, REQUESTED, and REQUIRED.

The default value is REQUESTED. In other words, if the server supports checksum enabled connections, then the Management Service uses them, otherwise it uses normal connections.

For example:

oracle.net. crypto_checksum_client=REQUESTED

oracle.net.crypto_checksum_types_client

This property defines the different types of checksums algorithms the client supports.

Possible values should be listed within parentheses. The default value is ( MD5 ).

For example:

oracle.net. crypto_checksum_types_client=( MD5 )

Enabling Oracle Advanced Security for the Management Repository

To be sure your database is secure and that only encrypted data is transferred between your database server and other sources, review the security documentation available in the Oracle Database 10g documentation library.

See Also:

Oracle Database Advanced Security Administrator's Guide

The following instructions provide an example of how you can confirm that Oracle Advanced Security is enabled for your Management Repository database and its connections with the Management Service:

1. Locate the sqlnet.ora configuration file in the following directory of the database Oracle Home:

ORACLE_HOME/network/admin

2. Using a text editor, look for the following entries (or similar entries) in the sqlnet.ora file:

SQLNET.ENCRYPTION_SERVER = REQUESTED
SQLNET.CRYPTO_SEED = "abcdefg123456789"

See Also:

"Configuring Network Data Encryption and Integrity for Oracle Servers and Clients" in the Oracle Application Server 10g Administrator's Guide

3. Save your changes and exit the text editor.

Enabling Security for a Management Agent Monitoring a Secure Management Repository or Database

After you have enabled Oracle Advanced Security for the Management Repository, you must also enable Advanced Security for the Management Agent that is monitoring the Management Repository:

1. Locate the sqlnet.ora configuration file in the following directory inside the home directory for the Management Agent that is monitoring the Management Repository:

AGENT_HOME/network/admin (UNIX)
AGENT_HOME\network\admin (Windows)

2. Using a text editor, add the following entry to the sqlnet.ora configuration file:

SQLNET.CRYPTO_SEED = "abcdefg123456789"

See Also:

"Configuring Network Data Encryption and Integrity for Oracle Servers and Clients" in the Oracle Application Server 10g Administrator's Guide

3. Save your changes and exit the text editor.

4. Restart the Management Agent.

Configuring Third Party Certificates

You can configure third party certificates for:

- HTTPS Upload Virtual Host

- HTTPS Apache Virtual Host

Configuring Third Party Certificate for HTTPS Upload Virtual Host

You can configure the third party certificate for the HTTPS Upload Virtual Host in two ways:

Method I

1. Create a wallet for each OMS in the grid.

2. While creating the wallet, specify the host name of the machine where the OMS is installed or the Load Balancer Name if the OMS is behind the Load Balancer for Common Name.

3. Write the certificates of all the Certificate Authorities in the certificate chain (like the Root Certificate Authority, Intermediate Certificate Authority) into a file named trusted_certs.txt.

4. Download or copy the trusted_certs.txt file to the host machines on which each Agent that is communicating with the OMS is running.

5. Run the following command on each Agent and restart the Agent:

emctl secure add_trust_cert -trust_certs_loc <location of the trusted_certs.txt file>

6. Run the following command on each OMS:

emctl secure oms -wallet <location of wallet> -trust_certs_loc <loc of trusted_certs.txt> [any other options]

Note:

If the wallet is not a single sign-on wallet, you will be prompted for the password.

Method 2

1. Create a wallet for each OMS in the grid.

2. Specify the host name of the machine where the OMS is installed or the Load Balancer Name if the OMS is behind the Server Load Balancer for Common Name (CN).

3. Write the certificates of all the Certificate Authorities in the certificate chain (like the Root Certificate Authority, Intermediate Certificate Authority) into a file named trusted_certs.txt.

4. Download or copy the trusted_certs.txt file to the host machines on which each Agent that is communicating with the OMS is running.

5. Run the following command on each OMS:

emctl secure oms -wallet <location of wallet> -trust_certs_loc <loc of trusted_certs.txt> [any other options]

Note:

If the wallet is not a single sign-on wallet, you will be prompted for the password.

6. Either re-secure the Agent by running the emctl secure agent command or import the trust points by running the emctl secure add_trust_cert -trust_certs_loc <location of the trusted_certs.txt file> command.

Configuring Third Party Certificate for HTTPS Apache Virtual Host

To configure the third party certificate for HTTPS Apache Virtual Host:

1. Create a wallet for each OMS in the grid. Specify the host name of the machine where the OMS is installed or the Load Balancer Name if the OMS is behind the Server Load Balancer for Common Name.

2. Run the following command on each OMS:

emctl secure console -wallet <location of wallet>

Note:

If the wallet is not a single sign-on wallet, your are prompted for the wallet's password.

Caution:

Before you use the emctl secure agent command to secure the Application Server Control Console, be sure to stop the Application Server Control Console.

To configure security for the Application Server Control, use the following procedure:

1. Stop the Application Server Control Console by entering the following command in the IAS_HOME/bin directory:

$PROMPT> ./emctl stop iasconsole

2. Enter the following command in the ORACLE_HOME/bin directory:

$PROMPT> ./emctl secure em

Enterprise Manager secures the Application Server Control Console. Sample output of the emctl secure em command is shown in .

3. Start the Application Server Control Console by entering the following command in the IAS_HOME/bin directory:

$PROMPT> ./emctl start iasconsole

4. Test the security of the Application Server Control Console by entering the following URL in your Web browser:

https://hostname:port/

For example:

https://mgmthost1:1812/

Example 3-9 Sample Output from the emctl secure em Command

$PROMPT> ./emctl secure em
Oracle Enterprise Manager 9.0.4
Copyright (c) 2002, 2003 Oracle Corporation.  All rights reserved.
Generating Standalone Console Root Key (this takes a minute)...   Done.
Fetching Standalone Console Root Certificate...   Done.
Generating Standalone Console Agent Key...   Done.
Generating Oracle Wallet for the Standalone Console Agent...   Done.
Configuring Agent for HTTPS...   Done.
EMD_URL set in /dsk01/oracle/appserver1/sysman/config/emd.properties
Generating Standalone Console Java Keystore...   Done.
$PROMPT>

Configuring Security for the Database Control

This section describes the architecture and configuration of security for the Oracle Enterprise Manager 10g Grid Control.

See Also:

Oracle strongly recommends that you use the Secure Socket Layer (SSL) protocol and HTTPS for all connections to Enterprise Manager and that you use a valid digital security certificate.

To configure security for the Database Control:

1. Stop the Database Control by entering the following command in the ORACLE_HOME/bin directory (UNIX) or the ORACLE_HOME\bin (Windows):

$PROMPT> ./emctl stop dbconsole (UNIX)
$PROMPT> emctl stop dbconsole (Windows)

See Also:

2. Change directory to the ORACLE_HOME/bin directory or the ORACLE_HOME\bin (Windows) and enter the following emctl command

$PROMPT> ./emctl secure dbconsole (UNIX)
$PROMPT> emctl secure dbconsole (Windows)

Enterprise Manager prompts you for the Enterprise Manager Root Password.

3. Enter the password for the SYSMAN database user.
Enterprise Manager prompts you to specify an Agent Registration Password, which is a new password that will be required for any Management Agents that attempt to connect to the Management Service.

4. Specify an Agent Registration Password for the Management Service.
Enterprise Manager prompts you to confirm the host name of the Management Service.

5. Enter the name of the host where the Management Service resides.
The emctl secure utility reconfigures the Management Service to enable Framework Security. If the Management Service is up and running, Enterprise Manager restarts the Management Service.

When the operation is complete, communications between the Enterprise Manager components is secure.

In addition, you can access the Grid Control Console using the HTTPS protocol.

6. Start the Database Control by entering the following command in the ORACLE_HOME/bin directory or the ORACLE_HOME\bin (Windows):

$PROMPT> ./emctl start dbconsole (UNIX)
$PROMPT> emctl start dbconsole (Windows)

See Also:

7. Test the security of the Database Control by entering the following URL in your Web browser:

https://hostname:port/em

For example:

https://dbhost1:1820/em

Note:

Alternatively, you can enter the emctl secure dbconsole command all on one line, but if you enter the command on one line, the passwords you enter will be displayed on the screen as you type:
$PROMPT> emctl secure dbconsole sysman_pwd agent_reg_pwd

Accessing Managed Targets

The following topics are discussed in this section:

- Credential Subsystem

- Pluggable Authentication Modules (PAM) Support

- Sudo and Powerbroker Support

Credential Subsystem

Credentials like user names and passwords are typically required to access targets such as databases, application servers, and hosts. Credentials are encrypted and stored in Enterprise Manager. By using appropriate credentials, you can:

- Collect metrics in the background as well as real-time

- Perform jobs like backup, patching, cloning etc.

- Perform real-time target administration like start, stop etc.

- Connect to My Oracle Support

Based on their usage, credentials can be classified into the following categories:

- Job Credentials: The job system uses the credential subsystem to retrieve the credentials required to submit a job on the targets. The administrator can define their preferred and default credentials in the preference section of EM console. The user can override the default credentials by specifying different credentials while submitting the job.

Note:

If the user chooses to use preferred credentials, these credentials will be used when the user submits the job. If the preferred credentials are not available, the default credentials will be used. If default credentials are not present, the job cannot be submitted.

- Monitoring Credentials: These credentials are used by the Management Agent to monitor certain types of targets. For example, most database monitoring involves connecting to the database, which requires a username, password, and optionally, a role. Monitoring credentials, if stored in the repository, can also be potentially used by management applications to connect directly to the target from the OMS.

- Collection Credentials: These credentials are associated with user-defined metrics.

To simplify the usage and management of credentials, the following features are available in Enterprise Manager:

- Host Preferred Credentials: Preferred credentials simplify access to managed targets by storing target login credentials in the Management Repository. With preferred credentials set, users can access an Enterprise Manager target that recognizes those credentials without being prompted to log into the target. Preferred credentials are set on a per user basis, thus ensuring the security of the managed enterprise environment.

- Default Credentials: Default credentials can be set for a particular target type and will be available for all the targets of the target type. It will be overridden by target preferred credentials.

- Target Credentials: Target credentials are preferred credentials set for a particular target. They could be used by applications such as the job system, patch, etc. For example, if the user chooses to use preferred credentials while submitting a job, then the preferred credentials set for the target (target credentials) will be used. If the target credentials is not present, the default credentials (for the target type) will be used. If the default credentials are not present, the job will fail.

Managing Credentials Using EMCLI

You can manage passwords using EMCLI verbs. Using EMCLI, you can:

- Change the database user password in both the target database and Enterprise Manager.

emcli update_db_password -change_at_target=Yes|No -change_all_reference=Yes|No

- Update a password which has already been changed at the host target.

emcli update_host_password -change_all_reference=Yes|No

Pluggable Authentication Modules (PAM) Support for Hosts

Pluggable authentication modules, or PAM, is a mechanism to integrate multiple low-level authentication schemes into a high-level application programming interface (API). It allows programs that rely on authentication to be written independently of the underlying authentication scheme.

Note:

The local password file (usually /etc/passwd) will be checked and used first. This should be synchronized with the LDAP password if it is being used. If this fails, the Management Agent will switch to the external authentication module.

The following steps describe the procedure to configure the Management Agent with PAM and LDAP for user authentication.

1. Check for PAM Module: Verify that the Linux PAM and NSS_LDAP packages are installed on the machine.

Example on Linux
$ rpm -qa | grep pam
$ rpm -qa | grep nss_ldap

These packages contain library files that are needed for the LDAP and PAM authentication. Typically, these files are stored in the /usr/lib directory:

$ ls -l /usr/lib/libpam.so
$ ls -l /usr/lib/libnss_ldap.so

If the libpam.so file is not installed in the default location, enter the following commands:

$ find / -name libpam.so      -print 2>/dev/null
$ find / -name libnss_ldap.so -print 2>/dev/null

2. Adjust the Agent Configuration: Edit the $ORACLE_HOME/bin/commonenv file of the Agent and add the directory of libpam.so file as follows:

if [ "$LD_LIBRARY_PATH" = "" ] ; then
LD_LIBRARY_PATH=<path to libpam.so>
else
LD_LIBRARY_PATH=<path to libpam.so>:$LD_LIBRARY_PATH
fi

To verify, run the set command from OMS using execute host command, look for the value of LD_LIBRARY_PATH in the result. Check the value to ensure that it has the PATH to libpam.so file.

3. Setup the emagent PAM File: On Linux, under /etc/pam.d directory, create a file named emagent (/etc/pam.d/emagent) with the following content:

#%PAM-1.0
auth required /lib/security/pam_ldap.so
account required /lib/security/pam_ldap.so
password required /lib/security/pam_ldap.so
session required /lib/security/pam_ldap.so

This assumes that the Linux system uses /etc/pam.d/configuration and the LDAP implementation is located at /lib/security directory.

Configuring PAM for RHEL4 Users

For users on RHEL4, the PAM file configuration is as follows:

#%PAM-1.0
auth required pam_ldap.so
account required pam_ldap.so
password required pam_ldap.so
session required pam_ldap.so

For more details, see https://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/ref-guide/s1-pam-format.html

Configuring PAM for AIX Users

For AIX users, use the edit/etc/pam.conf file and add the following lines:

emagent auth    required        /usr/lib/security/pam_aix
emagent account required        /usr/lib/security/pam_aix
emagent password  required      /usr/lib/security/pam_aix
emagent session required        /usr/lib/security/pam_aix

After editing the file, apply patch 5527130 and run root.sh

Sudo and PowerBroker Support

Privilege delegation allows a logged-in user to perform an activity with the privileges of another user. Sudo and PowerBroker are privilege delegation tools that allow a logged-in user to be assigned these privileges. Typically, the privileges that are granted to a specific user are administered centrally. For example, the sudo command can be used to run a script that requires root access:

sudo root root.sh

In the invocation of sudo in the example above, an administrator can use the sudo command to run a script as root provided he has been granted the appropriate privileges by the system administrator. Enterprise Manager preferred credentials allow you to use two types of privilege delegation tools: Sudo and PowerBroker. You can use EMCLI or the Manage Privilege Delegation Settings page to set/edit privilege delegation settings for a host. See the Enterprise Manager Command Line Interface guide for more information on using the command line.

Sudo: sudo allows a permitted user to execute a command as the super user or another user, as specified in the sudoers file. If the invoking user is root or if the target user is the same as the invoking user, no password is required. Otherwise, sudo requires that users authenticate themselves with a password by default. Once a user has been authenticated, a timestamp is updated and the user may then use sudo without a password for a short period of time (5 minutes unless overridden in sudoers). sudo determines who is an authorized user by consulting the file /etc/sudoers file. For more information, see the manual page on sudo (man sudo) on Unix. Enterprise Manager authenticates the user using sudo, and executes the script as sudo. For example, if the command to be executed is foo -arg1 -arg2, it will be executed as sudo -S foo -arg1 -arg2.

PowerBroker: Symark PowerBroker enables UNIX system administrators to specify the circumstances under which other people may run certain programs such as root (or other important accounts). The result is that responsibility for such actions as adding user accounts, fixing line printer queues, and so on, can be safely assigned to the appropriate people, without disclosing the root password. The full power of root is thus protected from potential misuse or abuse-for example, modifying databases or file permissions, erasing disks, or more subtle damage. Symark PowerBroker can access existing programs as well as its own set of utilities that execute common system administration tasks. Utilities being developed to run on top of Symark PowerBroker can manage passwords, accounts, backups, line printers, file ownership or removal, rebooting, logging people out, killing their programs, deciding who can log in to where from where, and so on. They can also provide TCP/IP, Load Balancer, cron, NIS, NFS, FTP, rlogin, and accounting subsystem management. Users can work from within a restricted shell or editor to access certain programs or files as root. See your Sudo or PowerBroker documentation for detailed setup and configuration information.

Creating a Privilege Delegation Setting

Enterprise Manager allows you to create privilege delegation settings either by creating the setting directly on a host target, or by creating a PDP setting template that you can apply to multiple hosts.

To create a privilege delegation setting directly on a host:

1. Login to Enterprise Manager and navigate to the Setup page. Click Manage Privilege Delegation Settings on the left panel.

2. For any host target appearing in the table, click Edit. Enterprise Manager takes you to the Host Privilege Delegation Setting page.

3. Select a privilege delegation type (Sudo or PowerBroker).

4. Enter the privilege delegation command to be used and, in the case of PowerBroker, the optional Password Prompt.

5. Click Update to apply the settings to the host.

Cryptographic Support

To protect the integrity of sensitive data in Enterprise Manager, a signing on verification method known as the emkey is used. Encryption key is the master key that is used to encrypt/decrypt sensitive data, such as passwords and preferred credentials that are stored in the Repository. The key itself is originally stored in the Repository and it needs to be in the Repository any time a new OMS is being installed. However, once all OMSs have been installed. The key should be removed from the Repository. By storing the key separately from Enterprise Manager schema, we ensure that the sensitive data such as Preferred Credentials in the Repository remain inaccessible to the schema owner and other SYSDBA users (Privileged users who can perform maintenance tasks on the database) in the Repository. Moreover, keeping the key from the schema will ensure that sensitive data remain inaccessible while Repository backups are accessed. Further, the schema owner should not have access to the OMS/Repository Oracle homes.

Configuring the emkey

The emkey is an encryption key that is used to encrypt and decrypt sensitive data in Enterprise Manager such as host passwords, database passwords and others. By default, the emkey is stored in the $ORACLE_HOME/sysman/config/emkey.ora file. The location of this file can be changed.

WARNING:

If the emkey.ora file is lost or corrupted, all the encrypted data in the Management Repository becomes unusable. Maintain a backup copy of this file on another system.

During startup, the Oracle Management Service checks the status of the emkey. If the emkey has been properly configured, it uses it encrypting and decrypting data. If the emkey has not been configured properly, the following error message is displayed.

Example 3-10 emctl start oms Command

$prompt> emctl start omsOracle Enterprise Manager 10g Release 10.2.0.0.0  Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.Starting HTTP Server ...Starting Oracle Management Server ...Checking Oracle Management Server Status ...Oracle Management Server is not functioning because of the following reason:The Em Key is not configured properly. Run "emctl status emkey" for more details.

Generating the emkey

The emkey is an encryption key that is used to encrypt and decrypt sensitive data in Enterprise Manager such as host passwords, database passwords and others. By default, the emkey is stored in the $ORACLE_HOME/sysman/config/emkey.ora file. The location of this file can be changed.

WARNING:

If the emkey.ora file is lost or corrupted, all the encrypted data in the Management Repository becomes unusable. Maintain a backup copy of this file on another system.

During startup, the Oracle Management Service checks the status of the emkey. If the emkey has been properly configured, it uses it encrypting and decrypting data. If the emkey has not been configured properly, the following error message is displayed.

Example 3-11 emctl start oms Command

$prompt> emctl start omsOracle Enterprise Manager 10g Release 10.2.0.0.0  Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.Starting HTTP Server ...Starting Oracle Management Server ...Checking Oracle Management Server Status ...Oracle Management Server is not functioning because of the following reason:The Em Key is not configured properly. Run "emctl status emkey" for more details.

emctl Commands

The emctl commands related to emkey are given below:

- emctl status emkey

- emctl config emkey -repos

- emctl config emkey -emkeyfile

- emctl config emkey -emkey

- emctl config emkey -remove_from_repos

- emctl config emkey -copy_to_repos

The usage of these commands is given below:

$prompt> emctl status emkey [-sysman_pwd <sysman password>]
$prompt> emctl config emkey -repos [-emkeyfile <emkey.ora path>] [-force] [-sysman_pwd <sysman password>]

$prompt> emctl config emkey -emkeyfile <emkey.ora path> [-force] [-sysman_pwd <sysman password>]
$prompt> emctl config emkey -emkey [-emkeyfile <emkey.ora path>] [-force] [-sysman_pwd <sysman password>]
$prompt> emctl config emkey -remove_from_repos [-sysman_pwd <sysman password>]
$prompt> emctl config emkey -copy_to_repos [-sysman_pwd <sysman password>]

emctl status emkey

This command shows the health or status of the emkey. Depending on the status of the emkey, the following messages are displayed:

- When the emkey has been correctly configured in the Management Service but is still present in the Management Repository, the following message is displayed.

Example 3-12 emctl status emkey - Example 1

Oracle Enterprise Manager 10g Release 10.2.0.0.0  Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.The Em Key is configured properly, but is not secure. Secure the Em Key by running "emctl config emkey -remove_from_repos".

- When the emkey has been correctly configured in the Management Service and has been removed from the Management Repository, the following message is displayed.

Example 3-13 emctl status emkey - Example 2

Oracle Enterprise Manager 10g Release 10.2.0.0.0  Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.The Em Key is configured properly.

- When the emkey.ora file is corrupt or missing and is present in the Management Repository, the following message is displayed.

Example 3-14 emctl status emkey - Example 3

Oracle Enterprise Manager 10g Release 10.2.0.0.0  Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.The Em Key exists in the Management Repository, but is not configured properly or is corrupted in the file system.Configure the Em Key by running "emctl config emkey -repos".

- When the emkey.ora file is corrupt or missing and is not present in the Management Repository, the following message is displayed.

Example 3-15 emctl status emkey - Example 4

Oracle Enterprise Manager 10g Release 10.2.0.0.0  Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.The Em Key is not configured properly or is corrupted in the file system and does not exist in the Management Repository. To correct the problem:1) Copy the emkey.ora file from another OMS or backup machine to the OH/sysman/config directory.2) Configure the emkey.ora file by running "emctl config emkey -emkeyfile <emkey.ora file location>".

emctl config emkey -repos

This command copies the emkey from the Management Repository to the emkey.ora file.

Example 3-16 Sample Output of the emctl config emkey -repos Command

$ emctl config emkey -repos -emkeyfile /tmp/emkey.ora.0 -forceOracle Enterprise Manager 10g Release 10.2.0.0.0  Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.Please enter repository password: The Em Key has been configured successfully.

In this example, the emkey is copied from the Management Repository to the /tmp/emkey.ora.0 file. The command configures the oracle.sysman.emkeyfile property in the emoms.properties to point to this file.

Note:

The -force option is required only if the emkey file is already configured.
If the -emkeyfile option is not provided in the Management Repository, the emkey is overwritten to the already configured emkey.ora file.

emctl config emkey -emkeyfile

This command can be used to configure a new emkey.ora file.

Example 3-17 Sample Output of emctl config emkey -emkeyfile Command

$ emctl config emkey -emkeyfile /tmp/emkey.ora.1 -forceOracle Enterprise Manager 10g Release 10.2.0.0.0  Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.Please enter repository password: The Em Key has been configured successfully.

This command configures the /tmp/emkey.ora.1 file as the new emkey.ora file. It also modifies the oracle.sysman.emkeyfile property in emoms.properties to point to this file. The -force option is required only if the emkey.ora file has already been configured.

emctl config emkey -emkey

This command is used to configure a new emkey.

Example 3-18 Sample Output of emctl config emkey -emkey Command

$ emctl config emkey -emkey -emkeyfile /tmp/emkey.ora.2 -forceOracle Enterprise Manager 10g Release 10.2.0.0.0  Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.Please enter repository password: Please enter the em key: The Em Key has been configured successfully.

This command writes the emkey provided as standard input into the /tmp/emkey.ora.2 file and configures it. The -force option is required only if the emkey.ora file has already been configured. If the -emkeyfile option is not provided, the emkey is overwritten to the already configured emkey.ora file.

emctl config emkey -remove_from_repos

This command removes the emkey from the Management Repository.

Example 3-19 Sample Output of emctl config emkey -remove_from_repos Command

$ emctl config emkey -remove_from_reposOracle Enterprise Manager 10g Release 10.2.0.0.0  Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.Please enter repository password: The Em Key has been removed from the Management Repository.Make a backup copy of OH/sysman/config/emkey.ora file and store it on another machine.WARNING: Encrypted data in Enterprise Manager will become unusable if the emkey.ora file is lost or corrupted.

emctl config emkey -copy_to_repos

This command copies the emkey back to the Management Repository.

Example 3-20 Sample Output of emctl config emkey_copy_to_repos Command

$ emctl config emkey -copy_to_reposOracle Enterprise Manager 10g Release 10.2.0.0.0  Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.Please enter repository password: The Em Key has been copied to the Management Repository. This operation will cause the Em Key to become unsecure.

Note:

This command is used during the additional Oracle Management Service install (See Install and Upgrade Scenarios). When you use this command, the emkey will be present in the Management Repository, which is not considered secure. You can secure it after the additional Oracle Management Service install by running the command:emctl config emkey -remove_from_repos

Install and Upgrade Scenarios

This section explains the install and upgrade scenarios for emkey.

Installing the Management Repository

A new emkey is generated as a strong random number when the Management Repository is installed.

Installing the First Oracle Management Service

When the Oracle Management Service is installed, the installer copies the emkey from the Management Repository and stores it in the emkey.ora file.

Note:

After installation, the emkey will be present in the Management Repository. This is not considered secure. The user can secure the emkey by running the emctl command emctl config emkey -remove_from_repos

Installing Additional Oracle Management Service

Similar to the first Oracle Management Service install, the installer will copy the emkey from the Management Repository to the emkey.ora file of the additional Oracle Management Service.

Note:

After the first Oracle Management Service install, you may have removed the emkey from the Management Repository using the emctl command.
Before the additional Oracle Management Service is installed, run the following command from the first Oracle Management Service home to copy the emkey to the Management Repository.

emctl config emkey -copy_to_repos

If the additional Oracle Management Service install is done without the emkey in the Management Repository, the installer will prompt the user to run the command mentioned above.

Upgrading from 10.1 to 10.2

The Management Repository is upgraded as usual. When the Oracle Management Service is upgraded, the upgrade script copies the emkey from the Management Repository to the emkey.ora file of each Oracle Management Service.

- Note:

After all the Oracle Management Service have been upgraded, you can secure the emkey, that is, remove it from the Management Repository by running the following command: emctl config emkey -remove_from_repos

Recreating the Management Repository

When the Management Repository is recreated, a new emkey is generated. This new key will not be in synchronization with the existing emkey.ora in the Oracle Management Service home directory. Enter the emctl config emkey -repos -force command to overwrite the new emkey to the emkey.ora file.

Setting Up the Auditing System for Enterprise Manager

All operations performed by Enterprise Manager users such as creating users, granting privileges, starting a remote job like patching or cloning, need to be audited to ensure compliance with the Sarbanes-Oxley Act of 2002 (SAS 70). This act defines standards an auditor must use to assess the contracted internal controls of a service organization. Auditing an operation enables an administrator to monitor, detect, and investigate problems and enforce enterprise wide security policies.

Irrespective of how the user has logged into Enterprise Manager, if auditing is enabled, each user action is audited and the audit details are stored in a record.

Configuring the Enterprise Manager Audit System

You can configure the Enterprise Manager Audit System by using the following options:

- Enabling and Disabling Auditing Using emcli Commands

- Enabling and Disabling Auditing Using PL/SQL

Enabling and Disabling Auditing Using emcli Commands

You can use the following emcli commands:

- enable_audit: Enables auditing for all user operations.

- disable_audit: Disables auditing for all user operations.

- show_audit_actions_list: Shows a list of the user operations being audited.

- show_audit_settings: Shows the audit status, operation list, externalization service details, and purge period details.

Enabling and Disabling Auditing Using PL/SQL

To set up the audit system in Enterprise Manager:

1. The audit function is turned off by default. Log in to the Enterprise Manager Management Repository as the sysman user. To turn on the audit function, enter the following commands:

SQL> exec mgmt_audit_admin.enable_audit;
SQL> commit;

2. After enabling auditing, you must restart the Oracle Management Service to ensure that this change has taken effect.

3. You can then login to Enterprise Manager and perform other user operations.

Notes:

- You can disable the auditing function by entering the following command:

SQL> exec  mgmt_audit_admin.disable_audit;
SQL> commit;

After disabling auditing, you must restart the Oracle Management Service to ensure that this change has taken effect.

- All the Super Administrators can view the audit data.

- To view the audit data, login into Enterprise Manager and click the Setup option. On the Setup page, click the Management Services and Repository tab. In the Overview page, click the Audit Data link under the Audit section to view the audit data.

Configuring the Audit Data Export Service

Audit data needs to be protected and maintained for several years. The volume of audit data may become very large and impact the performance of the system. To limit the amount of data stored in the repository, the audit data must be externalized or archived at regular intervals. The archived audit data is stored in an XML file complying with the ODL format. To externalize the audit data, the EM_AUDIT_EXTERNALIZATION API is used. Records of the format <file-prefix>.NNNNN.xml, where NNNN is a number are generated. The numbers start with 00001 and continue to 99999.

You can set up the audit externalization service for exporting audit data into the file system by using the following emcli command:

- update_audit_setting -file_prefix=<file_prefix> -directory_name=<directory_name> -file_size = <file size> -data_retention_period=<period in days>: Sets up the externalization service for exporting audit data to the file system.

- file_prefix: The prefix of the file which contains the audit data.

- directory_name: The name of the database directory that is mapped to the OS directory.

- file_size: The file size is the size of the file the data is written to.

- data_retention_period: The period for which the audit data is to be retained inside the repository.

For more details on the EMCLI verbs, refer to Enterprise Manager Command Line Reference.

Searching the Audit Data

You can search for audit data that has been generated over a specified period. You can also search for the following:

- Audit details of a specific user operation or all user operations.

- Audit details of operations with a Success or Failure status or All operations.

To view the audit data, click the Setup option. On the Setup page, click the Management Services and Repository tab. The Overview page is displayed. Click the Audit Data link under the Audit section. The Audit Data page is displayed.

Figure 3-8 Audit Data Page

Specify the search criteria in the fields and click Go. The results are displayed in the Summary table.

Figure 3-9 Audit Data Search Page

To view the details of each record that meets the search criteria, select Detailed in the View drop-down list. To drill down to the full record details, click on the Timestamp. The Audit Record page is displayed.

Figure 3-10 Audit Record Details Page

Field Name

Description

General

Operation Timestamp

The date and time on which the operation took place.

Administrator

The id of the administrator who has logged into Enterprise Manager.

Operation

The type of operation being audited.

Status

The status of the operation which can be success or failure.

Message

A descriptive message indicating the status of the operation.

Normalized Timestamp

This is the UTC timestamp.

Client Information

Session

This can either be the HTTP Session ID or the DBMS Session ID.

IP Address

The IP address of the client's host machine.

Hostname

The name of the client's host machine.

Upstream Component Type

The type of client, Console, Web Service, EMCLI, being used.

Authentication Type

The nature of the session (HTTP Session, DB Session).

Upstream Component Name

The name of the client being used.

OMS Information

Hostname

The host name of the Oracle Management Service.

IP Address

The IP address of the Oracle Management Service.

Instance ID

The Instance ID of the Oracle Management Service.

Operation Specific Information

Object Name

The operation being performed on an object

Operation Codes

The following table lists the names of operation and their description.

Table 3-6 Operation Specific Data

Operation Name

Ordered List

change_password

Change Password

create_user

Create User

delete_user

Delete User

logon

Login

logoff

Logout

grant_role

Grant Role

grant_target_priv

Grant Target Privilege

revoke_role

Revoke Role

revoke_target_priv

Revoke Target Privilege

submit_job

Submit Job

edit_job

Edit Job

delete_job

Delete Job

change_pref_cred

Change Preferred Credential

modify_user

Modify User

grant_system_priv

Grant System Privilege

grant_job_priv

Grant Job Privilege

revoke_system_priv

Revoke System Privilege

revoke_job_priv

Revoke Job Privilege

remote_op

Remote Operation Job

get_file

Get File

put_file

Put File

file_transfer

File Transfer

create_role

Create Role

delete_role

Delete Role

modify_role

Modify Role

job_output

Job Output

suspend_job

Suspend Job

agent_resync

Agent Re synchronization Operation

repository_resync

Repository Re synchronization Operation

remove_privilege_delegation_setting

Remove Privilege Delegation Setting

set_privilege_delegation_setting

Set Privilege Delegation Setting

add_agent_registration_password

Add Registration Password

edit_agent_registration_password

Edit Registration Password

delete_agent_registration_password

Delete Registration Password

agent_registration_password_usage

Registration Password Usage

Additional Security Considerations

After you enable security for the Enterprise Manager components and framework, there are additional security considerations. This section provides the following topics:

- Changing the SYSMAN and MGMT_VIEW Passwords

- Responding to Browser-Specific Security Certificate Alerts

- Configuring Beacons to Monitor Web Applications Over HTTPS

- Using ORACLE _HOME Credentials

- Patching Oracle Homes When the User is Locked

- Cloning Oracle Homes

Changing the SYSMAN and MGMT_VIEW Passwords

This section describes the commands used to change the SYSMAN and MGMT_VIEW passwords.

Changing the SYSMAN User Password

To change the password of the SYSMAN user, enter the following command:

emctl config oms -change_repos_pwd [-change_in_db] [-old_pwd <old_pwd>] [-new_pwd <new_pwd>] [-use_sys_pwd [-sys_pwd <sys_pwd>]]

You must run this command on each Management Service in your environment.

Parameter

Description

-change_in_db

This parameter is optional and is used to change the SYSMAN password in the repository. If there are multiple Management Services running, this parameter must be set to true for at least one Management service.
If this parameter is not specified, the emoms.properties file will be updated with the new SYSMAN password.

-old_pwd

This is the current SYSMAN password.

-new_pwd

This is the new password.

-use_sys_pwd

This parameter is optional and is used to connect to the database as a SYS user.

-sys_pwd

This is the password for the SYS user.

Changing the MGMT_VIEW User Password

To change the password of the MGMT_VIEW user, enter the following command:

emctl config oms -change_view_user_pwd [-sysman_pwd <sysman_pwd>] [-user_pwd <user_pwd>] [-auto_generate]

Parameter

Description

-sysman_pwd

The password for the SYSMAN user.

-user_pwd

The new password for theMGMT_VIEW user.This is an optional parameter and if it is not specified, the password is auto generated.

-auto_generate

If this option is specified, the password is auto generated.

Responding to Browser-Specific Security Certificate Alerts

This section describes how to respond to browser-specific security alert dialog boxes when you are using Enterprise Manager in a secure environment.

The security alert dialog boxes described in this section should appear only if you have enabled Enterprise Manager Framework Security, but you have not completed the more extensive procedures to secure your Oracle HTTP Server properly.

See Also:

Oracle Application Server 10g Security Guide

This section contains the following topics:

- Responding to the Internet Explorer Security Alert Dialog Box

- Responding to the Netscape Navigator New Site Certificate Dialog Box

- Preventing the Display of the Internet Explorer Security Information Dialog Box

Responding to the Internet Explorer Security Alert Dialog Box

If you enable security for the Management Service, but do not enable the more extensive security features of your Oracle HTTP Server, you will likely receive a Security Alert dialog box similar to the one shown in Figure 3-11 when you first attempt to display the Grid Control Console using the HTTPS URL in Internet Explorer.

Note:

The instructions in this section apply to Internet Explorer 5.5. The instructions may vary for other supported browsers.

Figure 3-11 Internet Explorer Security Alert Dialog Box

Description of "Figure 3-11 Internet Explorer Security Alert Dialog Box"

When Internet Explorer displays the Security Alert dialog box, use the following instructions to install the certificate and avoid viewing this dialog box again in future Enterprise Manager sessions:

1. In the Security Alert dialog box, click View Certificate.
Internet Explorer displays the Certificate dialog box.

2. Click the Certificate Path tab and select the first entry in the list of certificates as shown in Figure 3-12.

3. Click View Certificate to display a second Certificate dialog box.

4. Click Install Certificate to display the Certificate Import wizard.

5. Accept the default settings in the wizard, click Finish when you are done, and then click Yes in the Root Certificate Store dialog box.
Internet Explorer displays a message box indicating that the Certificate was imported successfully.

6. Click OK to close each of the security dialog boxes and click Yes on the Security Alert dialog box to continue with your browser session.
You should no longer receive the Security Alert dialog box in any future connections to Enterprise Manager when you use this browser.

Figure 3-12 Certificate Path Tab on the Internet Explorer Certificate Dialog Box

Description of "Figure 3-12 Certificate Path Tab on the Internet Explorer Certificate Dialog Box"

Responding to the Netscape Navigator New Site Certificate Dialog Box

If you enable security for the Management Service, but you do not enable the more extensive security features of your Oracle HTTP Server, you will likely receive a New Site Certificate dialog box similar to the one shown in Figure 3-13 when you first attempt to display the Grid Control Console using the HTTPS URL in Netscape Navigator.

Note:

The instructions in this section apply to Netscape Navigator 4.79. The instructions may vary for other supported browsers.

When Netscape Navigator displays the New Site Certificate dialog box, use the following instructions to install the certificate and avoid viewing this dialog box again in future Enterprise Manager sessions:

1. Review the instructions and information on each wizard page; click Next until you are prompted to accept the certificate.

2. Select Accept this certificate forever (until it expires) from the list of options.

3. On the last screen of the wizard, click Finish to close the wizard and continue with your browser session.
You should no longer receive the New Site Certificate dialog box when using the current browser.

Figure 3-13 Netscape Navigator New Site Certificate Dialog Box

Description of "Figure 3-13 Netscape Navigator New Site Certificate Dialog Box"

Preventing the Display of the Internet Explorer Security Information Dialog Box

After you enable Security for the Management Service, you may receive a dialog box similar to the one shown in Figure 3-14 whenever you access certain Enterprise Manager pages.

Note:

The instructions in this section apply to Internet Explorer 6.0. The instructions may vary for other supported browsers.

Figure 3-14 Internet Explorer Security Information Dialog Box

Description of "Figure 3-14 Internet Explorer Security Information Dialog Box"

To stop this dialog box from displaying:

1. Select Internet Options from the Internet Explorer Tools menu.

2. Click the Security tab.

3. Select Internet and then click Custom Level.
Internet Explorer displays the Security Settings dialog box.

4. Scroll down to Miscellaneous settings and enable the Display Mixed Content option.

Configuring Beacons to Monitor Web Applications Over HTTPS

Oracle Beacons provide application performance availability and performance monitoring. They are part of the Application Service Level Management features of Enterprise Manager.

See Also:

"About Application Service Level Management" in the Enterprise Manager Online Help

When a Beacon is used to monitor a URL over Secure Sockets Layer (SSL) using an HTTPS URL, the Beacon must be configured to recognize the Certificate Authority that has been used by the Web site where that URL resides.

See Also:

"The Public Key Infrastructure Approach to Security" in Oracle Security Overview for an overview of Public Key Infrastructure features, such as Certificate Authorities

The Beacon software is preconfigured to recognize most commercial Certificate Authorities that are likely to be used by a secure Internet Web Site. However, you may encounter Web Sites that, although available over HTTPS, do not have a Certificate that has been signed by a commercial Certificate Authority recognized by the Beacon. The following are out-of-box certificates recognized by Beacons:

- Class 1 Public Primary Certification Authority by VeriSign, Inc.

- Class 2 Public Primary Certification Authority by VeriSign, Inc.

- Class 3 Public Primary Certification Authority by VeriSign, Inc.

- Secure Server Certification Authority by RSA Data Security, Inc.

- GTE CyberTrust Root by GTE Corporation

- GTE CyberTrust Global Root by GTE CyberTrust Solutions, Inc.

- Entrust.net Secure Server Certification Authority by Entrust.net ((c) 1999

- Entrust.net Limited, www.entrust.net/CPS incorp. by ref. (limits liab.))

- Entrust.net Certification Authority (2048) by Entrust.net ((c) 1999

- Entrust.net Limited, www.entrust.net/CPS_2048 incorp. by ref. (limits liab.))

- Entrust.net Secure Server Certification Authority by Entrust.net ((c) 2000

- Entrust.net Limited, www.entrust.net/SSL_CPS incorp. by ref. (limits liab.))

In those cases, for example, if you attempt to use the Test section of the Beacon Performance page to test the HTTP Response of the secure URL, the following error appears in the Status Description column of the Response Metrics table on the URL Test Page:

javax.net.ssl.SSLException: SSL handshake failed: X509CertChainIncompleteErr--https://mgmtsys.acme.com/OracleMyPage.Home

See Also:

"Using Beacons to Monitor Remote URL Availability" in the Enterprise Manager online help

To correct this problem, you must allow the Beacon to recognize the Certificate Authority that was used by the Web Site to support HTTPS. You must add the Certificate of that Certificate Authority to the list of Certificate Authorities recognized by Beacon.

To configure the Beacon to recognize the Certificate Authority:

1. Obtain the Certificate of the Web Site's Certificate Authority, as follows:

1. In Microsoft Internet Explorer, connect to the HTTPS URL of the Web Site you are attempting to monitor.

2. Double-click the lock icon at the bottom of the browser screen, which indicates that you have connected to a secure Web site.
The browser displays the Certificate dialog box, which describes the Certificate used for this Web site. Other browsers offer a similar mechanism to view the Certificate detail of a Web Site.

3. Click the Certificate Path tab and select the first entry in the list of certificates as shown in Figure 3-12.

4. Click View Certificate to display a second Certificate dialog box.

5. Click the Details tab on the Certificate window.

6. Click Copy to File to display the Certificate Manager Export wizard.

7. In the Certificate Manager Export wizard, select Base64 encoded X.509 (.CER) as the format you want to export and save the certificate to a text file with an easily-identifiable name, such as beacon_certificate.cer.

8. Open the certificate file using a text editor.
The content of the certificate file will look similar to the content shown in .

2. Update the list of Beacon Certificate Authorities as follows:

1. Locate the b64InternetCertificate.txt file in the following directory of Agent Home of the Beacon host:

agent_home/sysman/config/

This file contains a list of Base64 Certificates.

2. Edit the b64InternetCertificate.txt file and add the contents of the Certificate file you just exported to the end of the file, taking care to include all the Base64 text of the Certificate including the BEGIN and END lines.

3. Restart the Management Agent.
After you restart the Management Agent, the Beacon detects your addition to the list of Certificate Authorities recognized by Beacon and you can successfully monitor the availability and performance of the secure Web site URL.

Example 3-21 Sample Content of an Exported Certificate

-----BEGIN CERTIFICATE-----
MIIDBzCCAnCgAwIBAgIQTs4NcImNY3JAs5edi/5RkTANBgkqhkiG9w0BAQQFADCB
... base64 certificate content...
-----END CERTIFICATE-----

Using ORACLE _HOME Credentials

Oracle Enterprise Manager 10g Release 2 introduces the concept of ORACLE_HOME credentials to designate the owner of the ORACLE_HOME with special credentials for the ORACLE_HOME. The operating system user who installs the software will also need to perform the patching. In Oracle Enterprise Manager 10g Release 2, one can explicitly set the ORACLE_HOME credential and store it in the Management Repository. While patching, the user can use existing operating system credentials or override it under special circumstances. The user can specify ORACLE_HOME credentials and in the same interface choose to store it in the Management Repository for future use.

The Enterprise Manager Command line interface (EM CLI) also provides a facility to set ORACLE_HOME credentials. This is useful in cases where the Super Administrator sets the credentials and the user who initiates the patching job is unaware of the actual credentials. For auditing in security-hardened data centers, the owner of the software is usually different from the user who initiates the patching job. The patching application internally switches the user context to the owner of the software and patches the software. To emulate such a case, the patch administrator will set the ORACLE_HOME credentials to the owner of the ORACLE_HOME. The Grid Control user who executes the patching job will be unaware of the credentials. The patching job will internally execute as the owner of the ORACLE_HOME. Grid Control will audit the patching job and capture the name of the Grid Control user who initiated the job. For example, if the owner of the ORACLE_HOME is "X", the patch super administrator in Grid Control is "Y" and the target administrator in Grid Control is "Y". "Y" will set the ORACLE_HOME credential to "X" with the password, using EMCLI. "Z" will submit the patching job using the already stored preferred credentials. Grid Control will audit the job as submitted by "Z".

The following is an example for setting the Oracle Home credentials using command line:

./emcli set_credential -target_type=host -target_name=val1 -credential_set=OHCreds -column="OHUsername:val2;OHPassword:val3"
-oracle_homes="val4"

where:

val1 = Hostname

val2 = Oracle Home user name

val3 = Oracle Home password

val4 = Oracle Home location

You can also set credentials for multiple Oracle Homes on the same host using the following command:

./emcli set_credential -target_type=host -target_name=val1 -credential_set=OHCreds -column="OHUsername:val2;OHPassword:val3"
-oracle_homes="val4;val5

where

val1 = Hostname

val2 = Oracle Home user name

val3 = Oracle Home password

val4 = Oracle Home location 1

val5 = Oracle Home location 2

Note:

Only one host can be passed to the verb.* If one wants multiple Oracle Home credentials on multiple hosts, then you will need Shell or Perl script to read lines, one at a time, from a file containing the host, credential values, and home location, and call the emcli set_credential verb for each row in the file.

The emcli set_credential command sets preferred credentials for given users. The following table describes the input values to the emcli set_credential command.

Table 3-7  emcli set_credential Parameters

Parameter

Input Value

Description

-target_type

-target_type ="ttype"

Type of target. Must be "host" in case the "-oracle_homes" parameter is specified.

-target_name

[-target_name="tname"]

Name of target. Omit this argument to set enterprise preferred credentials. Must be hostname in case "-oracle_homes" parameter is specified

-credential_set

-credential_set="cred_set"

Credential set affected.

-user

[-user="user"]

Enterprise Manager user whose credentials are affected. If omitted, the current user's credentials are affected.

-columns

-columns="col1:newval1;col2:newval2;..."

The name and new value of the column(s) to set. Every column of the credential set must be specified. Alternatively, a tag from the -input_file argument may be used so that the credential values are not seen on the command line. This argument may be specified more than once.

-input_file

[-input_file="tag1:file_path1;tag2:file_path2;..."]

Path of file that has -columns argument(s). This option is used to hide passwords. Each path must be accompanied by a tag which is referenced in the -columns argument. This argument may be specified more than once.

-oracle_homes

[-oracle_homes="home1;home2"]

Name of Oracle Homes on the target host. Credentials will be added/updated for all specified home

Patching Oracle Homes When the User is Locked

To patch an Oracle Home used by a user "Oracle" and the user is locked:

1. Edit the default patching script and prepend sudo or sudo -u or pbrun -u to the default patching step. You need to set a policy (by editing the sudoers file) to allow the user submitting the job (who must be a valid operating system user) to be able to run sudo or pbrun without being prompted for password.

Note:

You cannot patch Oracle Homes without targets. This must be done by using the Patching wizard.

Cloning Oracle Homes

The cloning application is wizard-driven. The source of the Oracle Home being cloned may be either an installed Oracle Home or a Software Library. Following are the steps in the cloning process:

1. If the source is an installed Oracle Home, then, after selecting the Oracle Home, a user will need to specify the Oracle Home credentials. These credentials once specified for an Oracle Home are stored in the repository. The next time a user clones the same Oracle Home, these credentials are automatically populated. Other parameters queried from the user at this point is a temporary location (on the source computer) and the list of files to be excluded from the Oracle Home. If the cloning source is a Software Library, the source Oracle Home credentials will not be queried for.

2. The user needs to specify the target location and provide the required credentials for each target location. These credentials will be the Oracle Home credentials for each of these target locations. Subsequently, if a user selects any of these cloned Oracle Homes as a source, the Oracle Home credentials are automatically populated.

3. Depending on the product being cloned, the user can view the Enterprise Manager page where query parameters required for the particular product being cloned are displayed.

4. The user can, then, view the execution of user-supplied pre-cloning and post-cloning scripts and the root.sh script. The root.sh script will always be run with sudo privileges, but the user has the option to decide if the pre-cloning and post-cloning scripts run with sudo privileges.Finally, the user can schedule the cloning job at a convenient time.

For more information about cloning, refer to the Enterprise Manager Online Help.

4 Notifications

The notification system allows you to notify Enterprise Manager administrators of alerts, policy violations, and the status changes of job executions. In addition to notifying administrators, the notification system can perform actions such as executing operating system commands (including scripts) and PL/SQL procedures when an alert is triggered. This capability allows you to implement automatically specific IT practices under particular alert conditions. For example, if an alert is generated when monitoring the operational (up/down) status of a database, you may want the notification system to automatically open an in-house trouble-ticket using an OS script so that the appropriate IT staff can respond in a timely manner.

By using Simple Network Management Protocol (SNMP) traps, the Enterprise Manager notification system also allows you to send traps to SNMP-enabled third-party applications such as HP OpenView. Some administrators may want to send third-party applications a notification when a certain metric has exceeded a threshold.

This chapter covers the following:

- Setting Up Notifications

- Extending Notification Beyond E-mail

- Passing Corrective Action Status Change Information

- Passing Job Execution Status Information

- Passing User-Defined Target Properties to Notification Methods

- Assigning Methods to Rules

- Assigning Rules to Methods

- Notification Coverage

- Management Information Base (MIB)

- Troubleshooting Notifications

Setting Up Notifications

All Enterprise Manager administrators can set up e-mail notifications for themselves. Super Administrators also have the ability to set up notifications for other Enterprise Manager administrators.

Setting Up a Mail Server for Notifications

Before Enterprise Manager can send e-mail notifications, you must first specify the Outgoing Mail (SMTP) servers to be used by the notification system. Once set, you can then define e-mail notifications for yourself or, if you have Super Administrator privileges, other Enterprise Manager administrators.

You specify the Outgoing Mail (SMTP) server on the Notification Methods page (Figure 4-1). Display the Notification Methods page by clicking Setup on any page in the Grid Control console and clicking Notification Methods in the vertical navigation bar.

Note:

You must have Super Administrator privileges in order to set up SMTP servers.

Specify one or more outgoing mail server names, the mail server authentication credentials (User Name, Password, and Confirm Password), if required, the name you want to appear as the sender of the notification messages, and the e-mail address you want to use to send your e-mail notifications. This address, called the Sender's Mail Address, must be a valid address on each mail server that you specify. A message will be sent to this e-mail address if any problem is encountered during the sending of an e-mail notification. Example 4-1 shows sample notification method entries.

Example 4-1 Mail Server Settings

- Outgoing Mail (SMTP) Server - smtp01.mycorp.com:587, smtp02.mycorp.com

- User Name - myadmin

- Password - ******

- Confirm Password - ******

- Identify Sender As - Enterprise Manager

- Sender's E-mail Address - mgmt_rep@mycorp.com

- Use Secure Connection - No: E-mail is not encrypted. SSL: E-mail is encrypted using the Secure Sockets Layer protocol. TLS, if available: E-mail is encrypted using the Transport Layer Security protocol if the mail server supports TLS. If the server does not support TLS, the e-mail is automatically sent as plain text.

Figure 4-1 Defining a Mail Server

Description of "Figure 4-1 Defining a Mail Server"

Note:

The e-mail address you specify on this page is not the e-mail address to which the notification is sent. You will have to specify the e-mail address (where notifications will be sent) from the Preferences General page.

After configuring the e-mail server, click Test Mail Servers to verify your e-mail setup. You should verify that an e-mail message was received by the e-mail account specified in the Sender's E-mail Address field.

Defining multiple mail servers will improve the reliability of e-mail notification delivery and spread the load across multiple systems. The Management Service makes use of each mail server to send e-mails and the behavior is controlled by the following parameters found in the $ORACLE_HOME/sysman/config/emoms.properties file.

Example 4-2 Management Service Parameters

# The maximum number of emails that can be sent in a single connection to an

# email server

# em.notification.emails_per_connection=20

#

# The maximum number of emails that can be sent in a minute

# em.notification.emails_per_minute=250

Based on the defaults in Example 4-2, the first mail server is used to send 20 e-mails before the Management Service switches to the next mail server which is used to send another 20 e-mails before switching to the next mail server. This prevents one mail server from becoming overloaded and should improve overall reliability and throughput.

Setting Up Repeat Notifications

Repeat notifications allow administrators to be notified repeatedly until an alert is either acknowledged or the number of Maximum Repeat Notifications has been reached. Enterprise Manager supports repeat notification for all notification methods (e-mail, OS command, PL/SQL procedure, and SNMP trap). To enable this feature for a notification method, select the Send Repeat Notifications option. In addition to setting the maximum number of repeat notifications, you can also set the time interval at which the notifications are sent.

Important:

If the Grid Control Repository database version is 9.2, the aq_tm_processes init.ora parameter must be set to at least 1 to enable repeat notification functionality.

Repeat Notifications for Rules

Setting repeat notifications globally at the notification method level may not be provide sufficient flexibility. For example, you may want to have different repeat notification settings based on the metric type and/or alert severity. Enterprise Manager accomplishes this by allowing you to set repeat notifications for individual notification rules. Repeat notifications set at the rule level take precedence over those defined at the notification method level.

Important:

Repeat notifications for rules will only be sent if the Send Repeat Notifications option is enabled in the Notification Methods page.

For PL/SQL, OS command, and SNMP trap notification methods, you must enable each method to support repeat notifications. You can select Supports Repeat Notifications option when adding a new notification method or by editing an existing method.

Figure 4-2 Enabling Repeat Notification for an OS Command Notification Method

Description of "Figure 4-2 Enabling Repeat Notification for an OS Command Notification Method"

Setting Up E-mail for Yourself

If you want to receive notifications by e-mail, you will need to specify your e-mail address(s) in the General page under the Preferences link in the Grid Control console. In addition to defining notification e-mail addresses, you associate the notification message format (long or short) to be used for your e-mail address.

Setting up e-mail involves three steps:

Step 1: Define e-mail addresses.

Step 2: Set up a Notification Schedule.

Step 3: Subscribe to receive e-mail for notification rules.

Defining E-mail Addresses

An e-mail address can have up to 128 characters. There is no upper limit with the number of e-mail addresses.

To add an e-mail address:

1. From the Grid Control console, click Preferences. By default the General page is selected.

2. Click Add Another Row to create a new e-mail entry field in the E-mail Addresses table.

3. Specify the e-mail associated with your Enterprise Manager account. All e-mail notifications you receive from Enterprise Manager will be sent to the e-mail addresses you specify.
For example, user1@oracle.com

Select the message format for your e-mail address. The Long Format sends a HTML formatted e-mail that contains detailed information. Example 4-3 shows a typical notification that uses the long format.

The Short Format (Example 4-4) sends a concise, text e-mail that is limited to a configurable number of characters, thereby allowing the e-mail be received as an SMS message or page. The content of the message can be sent entirely in the subject, entirely in the body or split across the subject and body. For example, in the last case, the subject could contain the severity type (for example, Critical) and the target name. The body could contain the time the severity occurred and the severity message. Since the message length is limited, some of this information may be truncated. If truncation has occurred there will be an ellipsis end of the message.

4. Click Apply to save your e-mail address.

Example 4-3 Long E-mail Notification for Alerts

Name=myhost.com

Type=Host

Host=myhost.com

Metric=Filesystem Space Available (%)

Mount Point =/usr

Timestamp=06-OCT-2006 16:27:05 US/Pacific

Severity=Warning

Message=Filesystem / has only 76.07% available space

Rule Name=Host Availability and Critical States

Rule Owner=SYSMAN

Example 4-4 Short E-mail Notification for Alerts

Subject is :

EM:Unreachable Start:myhost

Body is :

Nov 16, 2006 2:02:19 PM EST:Agent is Unreachable (REASON = Connection refused)

but the host is UP

More about Short E-mail Format

Enterprise Manager does not directly support message services such as paging or SMS, but instead relies on external gateways to, for example, perform the conversion from e-mail to page.

Entries in the emoms.properties file define the size and format of the short e-mail.

You must establish the maximum size your device can support and whether the message is sent in subject, body or both.

emoms.properties Entries for a Short E-mail Format

# The maximum size of a short format email

# em.notification.short_format_length=155

# The format of the short email. It can be set to subject, body or both.

#

# When set to subject the entire message is sent in the subject i.e.

#  EM:<severity>:<target>:<message>:<timestamp>

# When set to body the entire message is sent in the body i.e.

#  EM:<severity>:<target>:<message>:<timestamp>

# When set to both the message is split i.e. the subject contains

#  EM:<severity>:<target>

# and the body contains

#  <message>:<timestamp>

# In all cases the message is truncated to the length specified in the

# em.notification.short_format_length parameter

# em.notification.short_format=both

Setting Up a Notification Schedule

Once you have defined your e-mail notification addresses, you will need to define a notification schedule. For example, if your e-mail addresses are user1@oracle.com, user2@oracle.com, user3@oracle.com, you can choose to use one or more of these e-mail addresses for each time period in your notification schedule.

Note:

When you enter e-mail addresses for the first time, a 24x7 weekly notification schedule is set automatically. You can then review and modify the schedule to suit your monitoring needs.

A notification schedule is a repeating schedule used to specify your on-call schedule—the days and time periods and e-mail addresses that should be used by Enterprise Manager to send notifications to you. Each administrator has exactly one notification schedule. When a notification needs to be sent to an administrator, Enterprise Manager consults that administrator's notification schedule to determine the e-mail address to be used. Depending on whether you are Super Administrator or a regular Enterprise Manager administrator, the process of defining a notification schedule differs slightly.

If you are a regular Enterprise Manager administrator and are defining your own notification schedule:

1. From the Enterprise Manager Grid Control, click Preferences at the top of the page. By default the General page is selected.

2. Click Notification Schedule in the vertical navigation bar. Your Notification Schedule page appears.

3. Follow the directions on the Notification Schedule page to specify when you want to receive e-mails.

Subscribe to Receive E-mail for Notification Rules

A notification rule is a user-defined rule that defines the criteria by which notifications should be sent for alerts, policy violations, corrective action execution status, and job execution status. Specifically, in each rule, you can specify the criteria you are interested in and the notification methods (such as e-mail) that should be used for sending these notifications. For example, you can set up a rule that when any database goes down or any database backup job fails, e-mail should be sent and the "log trouble ticket" notification method should be called. Or you can define another rule such that when the CPU or Memory Utilization of any host reach critical severities, SNMP traps should be sent to another management console. During notification rule creation, you specify criteria such as the targets you are interested in, their monitored metrics, associated alert severity conditions (clear, warning, critical), policy violations, corrective action execution status, or job execution status, and the associated notification method.

To subscribe to a notification rule you create, while creating the rule, go to the Actions page and check the Send Me E-mail option.

Out-of-Box Notification Rules

Enterprise Manager Grid control comes with out-of-box notification rules that cover the most common alert conditions. When you install the Oracle Management Service, you are given the option to receive e-mail notifications for critical alerts. If you choose this option, and if an e-mail address for the SYSMAN user was specified, then some default notification rules are created that cover the availability and critical states for common target types and would also be configured to send e-mail notifications to the SYSMAN e-mail address for the conditions defined in the notification rules.

You can access the out-of-box notification rules by clicking on Preferences on any page in the Enterprise Manager console and clicking Public Rules in the vertical navigation bar. If the conditions defined in the out-of-box notification rules meet your requirements, you can simply subscribe to receive e-mail notifications for the conditions defined in the rule by clicking on Subscribe column in the row of the Public Rules table that corresponds to the notification rule that you are interested in. Click Apply to save your changes.

Table 4-1 lists all default notification rules. These are all owned by the SYSMAN user and are public rules.

Table 4-1 Default Notification Rules

Name

Description

Applies to Targets of the Type

Send Notification on the Following Availability States

Send Notification if Any of the Metrics is at CRITICAL Alert Severity

Agent Upload Problems

System-generated notification rule for monitoring Agents that may have problems uploading data to the Management Service.

Oracle Management Service and Repository

N/A

Count of targets not uploading data

Agents Unreachable

System-generated notification rule for monitoring Agents that lose contact with the Management Service due to network problems, host problems or Agents going down.

Agents

Agent UnreachableAgent Unreachable Resolved

N/A

Application Server Availability and Critical States

System-generated notification rule for monitoring Application Servers' availability, and critical metric statuses.

Application Servers

Down

CPU Usage (%)

Database Availability and Critical States

System-generated notification rule for monitoring Databases' availability, and critical metric statuses.

Databases (single instance only)

Down

Process Limit Usage (%)

Session Limit Usage (%)

Blocking Session Count All Objects

Archiver Hung Alert Log Error Status

Data Block Corruption Alert Log Error Status

Generic Alert Log Error Status

Media Failure Alert Log Error Status

Session Terminated Alert Log Error Status

Archive Area Used (%) All Objects

Segments Not Able to Extend Count All Objects

Segments Approaching Maximum Extents Count All Objects

Tablespace Space Used (%) All Objects

Wait Time (%)

HTTP Server Availability and Critical States

System-generated notification rule for monitoring HTTP Server's availability, and critical metric statuses.

Oracle HTTP Server

Down

CPU Usage (%)

Percentage of Busy Processes

Active HTTP Connections

Request Processing Time (seconds)

Host Availability and Critical States

System-generated notification rule for monitoring Hosts' availability, and critical metric statuses.

Hosts

Agent Unreachable

Agent Unreachable Resolved

Average Disk I/O Service Time (ms)

Disk Device Busy (%)

Filesystem Space Available (%)

CPU in I/O Wait (%)

Run Queue Length (5 minute average)

CPU Utilization (%)

Memory Utilization (%)

Memory Page Scan Rate (per second)

Swap Utilization (%)

Network Interface Combined Utilization (%)

Listener Availability

System-generated notification rule for monitoring database Listeners' availability, and critical metric statuses.

Listeners

Down

N/A

Misconfigured Agents

System-generated notification rule for misconfigured agents.

Agent

Agent Unreachable

Agent Unreachable Resolved

Consecutive severity upload failure count Consecutive heartbeat failure count

MS Agent time skew (mins)Consecutive metadata upload failure count

OC4J Availability and Critical States

System-generated notification rule for monitoring OC4J instance's availability, and critical metric statuses.

OC4J

Down

CPU Usage (%)

OC4J Instance - Request Processing Time (seconds)

OC4J Instance - Active Sessions

OMS Service Initialization Errors

System-generated notification rule for monitoring OMS service initialization errors.

OMS and Repository

N/A

Service Status

PAF Status Notification

System-generated notification rule for Provisioning Advisor Framework: Notifies the instance creator of any status updates.

N/A

Up

Down

Corrective Actions on Target Down

Agent Unreachable

Agent Unreachable Resolved

Metric Error Detected

Metric Error ResolvedBlackout Started

Blackout Ended

N/A

Repository Operations Availability

System-generated notification rule for monitoring the availability of the DBMS jobs that are part of the Management Repository.

OMS and Repository

Critical

DBMS Job UpDown

Violation Notification for Database Security Policies

System-generated notification rule for monitoring the secureness of the database configuration.

Databases

Critical

N/A

Web Cache Availability and Critical States

System-generated notification rule for monitoring Web Cache's instance's availability, and critical metric statuses.

Oracle Web Cache

Down

Hits (% of requests)

Web Cache CPU Usage (%)

Creating Your Own Notification Rules

If you find that the default notification rules do not meet your needs, you can define your own custom rules. The following procedure documents the process of notification rule creation for non-Super Administrators.

To create your own notification rule:

1. From the Enterprise Manager Grid Control, click Preferences.

2. Click My Rules in the vertical navigation bar.
If you are not logged in as an administrator with Super Administrator privileges, you will see a link for My Rules instead of Rules as in the case of an administrator with Super Administrator privileges.

3. Click Create.
Enterprise Manager displays the Create Notification Rule pages. Enter the requisite information on each page to create your notification rule.

When you specify the notification rule properties, check Make Public in the General page if you want other non-privileged users to be able to view and share that rule. For example, it allows other administrators to later specify that they should receive e-mail for this rule.

When you specify the notification rule, you will only be able to choose from e-mail and SNMP traps. Specifying custom commands and PL/SQL procedures is an option that is only available to Super Administrators. To receive e-mail notifications for conditions defined in the rule, go to the Actions page and check the Send Me E-Mail option.

Specifying Additional Alert Duration Criteria

You can set additional alert duration criteria for a notification rule and have the rule apply only to alerts that have been open for at least a certain amount of time and have not been acknowledged. These criteria apply only to Target Down, Agent Unreachable, Metric alerts, Policy Violations, Blackout Started and Metric Error Start alerts.

Typical scenarios where you would use additional alert criteria are:

- For log alerts that have been open for at least 7 days, clear the alerts

- For alerts that have been open for at least 48 hours and have not been acknowledged, send e-mail to the DBA Manager

To specify additional alert duration criteria:

1. Create or edit a notification rule. Additional alert criteria can be added from the Availability, Metrics, or Policy tab.

2. From one of the aforementioned tabs, go to the Additional Alert Criteria section and click Add. The Additional Alert Criteria page appears.

3. Specify the alert duration criteria and click Continue.

Setting Up E-mail for Other Administrators

If you have Super Administrator privileges, you can set up e-mail notifications for other Enterprise Manager administrators. To set up e-mail notifications for other Enterprise Manager administrators, you need to:

Step 1: Ensure Each Administrator Account has an Associated E-mail Address

Each administrator to which you want to send e-mail notifications must have a valid e-mail address.

1. Click Setup.

2. Click Administrators from the vertical navigation bar.

3. For each administrator, define an e-mail address. This sets up a 24x7 notification schedule for this user that uses all the e-mail addresses specified.

Enterprise Manager also allows you to specify an administrator address when editing an administrator's notification schedule.

Step 2: Define Administrators' Notification Schedules

Once you have defined e-mail notification addresses for each administrator, you will need to define their respective notification schedules. Although a default 24x7 notification schedule is created when you specify an e-mail address for the first time, you should review and edit the notification schedule as needed.

1. Click Setup.

2. From the vertical navigation bar, click Schedules (under Notification). The Notification Schedule page appears.

3. Specify the administrator who's notification schedule you wish to edit and click Change.

4. Click Edit Schedule Definition. The Edit Schedule Definition: Time Period page appears. If necessary, modify the rotation schedule.

5. Click Continue. The Edit Schedule Definition: E-mail Addresses page appears.

6. Follow the directions on the Edit Schedule Definition: E-mail Addresses page to modify the notification schedule.

7. Click Finish when you are done.

8. Repeat steps three through seven for each administrator.

Step 3: Assign Notification Rules to Administrators

With the notification schedules set, you now need to assign the appropriate notification rules for each designated administrator.

1. Click Setup.

2. From the vertical navigation bar, click Administrators.

3. Select the desire administrator.

4. Click Subscribe to Rules. The Subscribe <administrator> to Public Notification Rules page appears.

5. Select the desired notification rules and click Subscribe.

6. Click OK when you are finished.

7. Repeat steps three through six for each administrator.

E-mail Customization

Enterprise Manager allows Super Administrators to customize global e-mail notifications for the four alert types (Metric Alert, Target Availability, Policy Violation, and Job Status Change). Using predefined building blocks (called attributes and labels) contained within a simple script, Super Administrators can customize alert e-mails by selecting from a wide variety of information content.

To customize an e-mail:

1. Access the E-mail Customization page. Setup-->E-mail Customization

2. Choose the Alert Type and Format.

3. Click Edit. The Edit E-mail Template page is displayed.

From the Edit E-mail Template page, you can modify the content of the e-mail template Enterprise Manager uses to generate e-mail notifications. Extensive information on script formatting, syntax, and options is available from the Edit E-mail Template page via imbedded assistance and online help.

Figure 4-3 E-mail Customization

Description of "Figure 4-3 E-mail Customization"

E-mail Customization Reference

The following reference summarizes the semantics and component syntax of the pseudo-language used to define e-mails. The pseudo-language provides you with a simple, yet flexible way to customize e-mail notrifications. The following is a summary of pseudo-language conventions/limitations:

- You can add comments (or any free-form text) using separate lines beginning with "--" or at end of lines.

- You can use attributes.

- You can use IF & ELSE & ENDIF control structures. You can also use multiple conditions using "AND" or "OR". Nested IF statements are not supported.

- You can insert spaces for formatting purposes. Spaces at the beginning of a line will be ignored in the actual e-mail. To insert spaces at the beginning of a line, use the [SP] attribute.

- Use "/" to escape and "[" or "]" if you want to add attribute names, operators, or IF clauses to the actual e-mail.

- HTML is not supported.

Reserved Words and Operators

The following table lists all reserved words and operators used when modifying e-mail scripts.

Table 4-2 Reserved Words and Operators

Reserved Word/Operator

Description

IF, ELSIF, ENDIF, ELSE

Used in IF-ELSE constructs.

AND, OR

Boolean operators – used in IF-ELSE constructs only.

NULL

To check NULL value for attributes - used in IF-ELSE constructs only.

|

Pipe operator – used to show the first non-NULL value in a list of attributes.

For example:

METRIC_NAME|POLICY_NAME

EQ, NEQ

Equal and Not-Equal operators – applicable to NULL, STRING and NUMERIC values.

/

Escape character – used to escape reserved words and operators. Escape characters signify that what follows the escape character takes an alternative interpretation.

[ , ]

Delimiters used to demarcate attribute names and IF clauses.

Syntax Elements

Literal Text

You can specify any text as part of the e-mail content. The text will be displayed in the e-mail and will not be translated if the Oracle Management Services (OMS) language setting is changed. For example, 'my Oracle Home' appears as 'my Oracle Home' in the generated e-mail.

Predefined Attributes

Predefined attributes/labels will be substituted with actual values in a specific context. To specify a predefined attribute/label, use the following syntax:

[PREDEFINED_ATTR]

Attribute names can be in either UPPER or LOWER case. The parsing process is case-insensitive.

A pair of square brackets is used to distinguish predefined attributes from literal text. For example, for a job e-mail notification, the actual job name will be substituted for [JOB_NAME]. For a metric e-mail notification, the actual metric column name will be substituted for [METIRC_COLUMN].

You can use the escape character “/” to specify words and not have them interpreted as predefined labels/attributes. For example, "/[NEW/]” will not be considered as the predefined attribute [NEW] when parsed.

Operators

EQ, NEQ – for text and numeric values

NULL- for text and numeric values

GT, LT, GE, LE – for numeric values

Control Structures

The following table lists acceptable script control structures.

Table 4-3 Control Structures

Control Structure

Description

Pipe "|"

Two or more attributes can be separated by '|' character. For example,

[METRIC_NAME|POLICY_NAME]

In this example, only the applicable attribute within the current alert context will be used (replaced by the actual value) in the e-mail. If more than one attributes are applicable, only the left-most attribute is used.

IF

Allows you to make a block of text conditional. Only one level of IF and ELSIF is supported. Nested IF constructs are not supported.

All attributes can be used in IF or ELSIF evaluation using EQ/NEQ operators on NULL values. Other operators are allowed for “SEVERITY” and “REPEAT_COUNT” only.

Inside the IF block, the values need to be contained within quotation marks “ ”. Enterprise Manager will extract the attribute name and its value based on the position of “EQ” and other key words such as “and”, “or”. For example,

[IF REPEAT_COUNT EQ “1” AND SEVERITY EQ “CRITICAL” THEN]

The statement above will be true when the attributes of the alert match the following condition:

- Attribute Name: REPEAT_COUNT

- Attribute Value: 1

- Attribute Name: SEVERITY

- Attribute Value: CRITICAL

Example IF Block:

[IF JOB_NAME NEQ NULL]

[JOB_NAME_LABEL]=[JOB_NAME]

[JOB_OWNER_LABEL]=[JOB_OWNER]

[ENDIF]

[IF SEVERITY EQ CRITICAL ]

[MTRIC_NAME_LABEL]=[METRIC_NAME]

[METRIC_VALUE_LABEL]=[METRIC_VALUE]

[TARGET_NAME_LABEL]=[TARGET_NAME]

[KEY_VALUES]

[ENDIF]

Example IF and ELSEIF Block:

[IF SEVERITY EQ CRITICAL]           statement1[ELSIF SEVERITY EQ WARNING]           statement2[ELSIF SEVERITY EQ CLEAR]           statement3[ELSE]           statement4[ENDIF]

Comments

You can add comments to your script by prefacing a single line of text with two hyphens "--". For example,

-- Code added on 8/3/2009     [IF REPEAT_COUNT NEQ NULL]     . . .

Comments may also be placed at the end of a line of text.

[IF SEVERITY_SHORT EQ W] -- for Warning alert

HTML Tags in Customization Content

Use of HTML tags is not supported.

When Enterprise Manager parses the e-mail script, it will convert the “<” and “>” characters of HTML tags into encoded format (&lt; and &gt;). This ensures that the HTML tag is not treated as HTML by the destination system.

Examples

E-mail customization template scripts support three main operators.

- Comparison operators: EQ/NEQ/GT/LT/GE/LELogic operators: AND/ORPipeline operator: |

Extending Notification Beyond E-mail

Notification Methods are the mechanisms by which alerts are sent. Enterprise Manager Super Administrators can set up e-mail notifications by configuring the 'e-mail' notification method. Most likely this would already have been setup as part of the Oracle Management Service installation.Enterprise Manager Super Administrators can also define other custom notification methods. For example, alerts may need to be forwarded to a 3rd party trouble-ticketing system. Assuming APIs to the third-party trouble-ticketing system are available, a custom notification method can be created to call a custom OS script that has the appropriate APIs. The custom notification method can be named in a user-friendly fashion, for example, "Log trouble ticket". Once that is defined, any time an administrator needs to send alerts to the trouble-ticketing system, he merely needs to invoke the now globally available notification method called "Log trouble ticket". Custom notification methods can be defined based on any custom OS script, any custom PL/SQL procedure, or by sending SNMP traps.

Only Super Administrators can define OS Command, PL/SQL, and SNMP Trap notification methods. However, any Enterprise Manager administrator can add these notification methods (once defined by the Super Administrator) to their notification rules.

Through the Notification Methods page, you can:

- Set up the outgoing mail servers if you plan to send e-mail notifications through notification rules

- Create other custom notification methods using OS and PL/SQL scripts and SNMP traps.

- Set global repeat notifications.

Custom Notification Methods Using Scripts and SNMP Traps

You can create other custom notification methods based on OS scripts, PL/SQL procedures, or SNMP traps. Any administrator can then use these methods in Notification Rules.

Adding a Notification Method based on an OS Command or Script

Complete the following four steps to define a notification method based on an OS command or script.

Note:

Notification methods based on OS commands must be configured by an administrator with Super Administrator privileges.

Step 1: Define your OS command or script.

You can specify an OS command or script that will be called by the notification system. You can use target and alert or policy violation context information, corrective action execution status and job execution status within the body of the script. Passing metric severity attributes (severity level, type, notification rule, rule owner, or rule owner, and so on) or policy violation information to OS commands/scripts allows you to customize automated responses to alerts or policy violations. For example, if an OS script opens a trouble ticket for an in-house support trouble ticket system, you will want to pass severity levels (critical, warning, and so on) to the script to open a trouble ticket with the appropriate details and escalate the problem. For more information on passing specific types of information to OS Command or Scripts, see

- "Passing Alert and Policy Violation Information to an OS Command or Script"

- "Passing Corrective Action Execution Status to an OS Command or Script"

- "Passing Job Execution Status to an OS Command or Script"

Step 2: Deploy the script on each Management Service host.

You must deploy the OS Command or Script on each Management Service host machine that connects to the Management Repository. The OS Command is run as the user who started the Management Service.

The OS Command or Script should be deployed on the same location on each Management Service host machine. The OS Command should be an absolute path, for example, /u1/bin/logSeverity.sh. The command is run by the user who started the Management Service. If an error is encountered during the running of the OS Command, the Notification System can be instructed to retry the sending of the notification to the OS Command by returning an exit code of 100. The procedure is initially retried after one minute, then two minutes, then three minutes and so on, until the notification is a day old, at which point it will be purged.

Example 4-5 shows the parameter in emoms.properties that controls how long the OS Command can execute without being killed by the Management Service. This is to prevent OS Commands from running for an inordinate length of time and blocking the delivery of other notifications. By default the command is allowed to run for 30 seconds before it is killed.

Example 4-5 Parameter in emoms.properties File

# The amount of time in seconds after which an OS Command started by the

# Notification System will be killed if it has not exited

# em.notification.os_cmd_timeout=30

Step 3: Register your OS Command or Script as a new Notification Method.

Add this OS command as a notification method that can be called in Notification Rules. Log in as a Super Administrator, click Setup and then Notification Methods from the vertical navigation bar. From this page, you can define a new notification based on the 'OS Command' type. See "Adding a Notification Method based on an OS Command or Script" .

The following information is required for each OS command notification method:

- Name

- Description
Both Name and Description should be clear and intuitive so that the function of the method is clear to other administrators.

- OS Command

You must enter the full path of the OS command or script in the OS command field (for example, /u1/bin/myscript.sh). For environments with multiple Management Services, the path must be exactly the same on each machine that has a Management Service. Command line parameters can be included after the full path (for example, /u1/bin/myscript.sh arg1 arg2).

Example 4-6 shows information required for the notification method.

Example 4-6 OS Command Notification Method

Name Trouble Ticketing

Description Notification method to log trouble ticket for a severity occurrence

OS Command /private/mozart/bin/logTicket.sh

Note:

There can be more than one OS Command configured per system.

Step 4: Assign the notification method to a rule.

You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a single rule, see "Assigning Rules to Methods" or "Assigning Methods to Rules".

Passing Alert and Policy Violation Information to an OS Command or Script

The notification system passes severity information to an OS script or executable using system environment variables.

Conventions used to access environmental variables vary depending on the operating system:

- UNIX: $ENV_VARIABLE

- Windows:%ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.

Table 4-4 Environment Variables

Environment Variable

Description

TARGET_NAME

Name of the target on which the severity occurred.

TARGET_TYPE

Type of target on which the severity occurred. Targets are defined as any monitorable entity, such as Host, Database, Listener, or Oracle HTTP Server. You can view the type of a monitored target on the All Targets page.

HOST

Name of the machine on which the target resides.

METRIC

Metric generating the severity. This variable is not set for policy violations.

METRIC_VALUE

The value of the metric when the threshold was exceeded. Not set for policy violations

POLICY_RULE

The name of the policy when the threshold was exceeded. Not set for metric severities

KEY_VALUE

For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity.

KEY_VALUE_NAME

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

VIOLATION_CONTEXT

A comma-separated list of name-value pairs that shows the alert context for a policy violation.

TIMESTAMP

Time when the severity occurred.

SEVERITY

Type of severity. For example, severity for a target's (availability) status metric are:

- UP

- DOWN

- UNREACHABLE CLEAR

- UNREACHABLE START

- BLACKOUT END

- BLACKOUT START

Other metrics can have any of the following severities:

- WARNING

- CRITICAL

- CLEAR

- METRIC ERROR CLEAR

- METRIC ERROR START

MESSAGE

Message for the alert that provides details about what triggered the condition.

RULE_NAME

Name of the notification rule to which the OS Command notification method was assigned.

RULE_OWNER

Name of the Enterprise Manager administrator who owns the notification rule.

Your script may reference some or all of these variables.

The sample OS script shown in Example 4-7 appends environment variable entries to a log file. In this example, the script logs a severity occurrence to a file server. If the file server is unreachable then an exit code of 100 is returned to force the Oracle Management Service Notification System to retry the notification

Example 4-7 Sample OS Command Script

#!/bin/ksh

LOG_FILE=/net/myhost/logs/severity.log

if test -f $LOG_FILE

then

echo $TARGET_NAME $MESSAGE $TIMESTAMP >> $LOG_FILE

else

exit 100

fi

Example 4-8 shows an OS script that logs alert information to the file 'alertmsg.txt'. The file is saved to the /u1/results directory.

Example 4-8 Alert Logging Script

#!/usr/bin/sh

echo "Alert logged:" > /u1/results/alertmsg.txt

echo "\n" >> /u1/results/alertmsg.txt

echo "target name is " $TARGET_NAME >> /u1/results/alertmsg.txt

echo "target type is " $TARGET_TYPE >> /u1/results/alertmsg.txt

echo "target is on host " $HOST >> /u1/results/alertmsg.txt

echo "metric in alert is " $METRIC >> /u1/results/alertmsg.txt

echo "metric index is " $KEY_VALUE >> /u1/results/alertmsg.txt

echo "timestamp is " $TIMESTAMP >> /u1/results/alertmsg.txt

echo "severity is " $SEVERITY >> /u1/results/alertmsg.txt

echo "message is " $MESSAGE >> /u1/results/alertmsg.txt

echo "notification rule is " $RULE_NAME >> /u1/results/alertmsg.txt

echo "rule owner is " $RULE_OWNER >> /u1/results/alertmsg.txt

exit 0

Example 4-9 shows a script that sends an alert to an HP OpenView console from Enterprise Manager Grid Control. When a metric alert is triggered, the Enterprise Manager Grid Control displays the alert. The HP OpenView script is then called, invoking opcmsg and forwarding the information to the HP OpenView management server.

Example 4-9 HP OpenView Script

/opt/OV/bin/OpC/opcmsg severity="$SEVERITY" app=OEM msg_grp=Oracle msg_text="$MESSAGE" object="$TARGET"

Adding a Notification Method Based on a PL/SQL Procedure

Complete the following four steps to define a notification method based on a PL/SQL procedure.

Step 1: Define the PL/SQL procedure.

The procedure must have one of the following signatures depending on the type of notification that will be received.

For alerts and policy violations:

PROCEDURE p(severity IN MGMT_NOTIFY_SEVERITY)

For job execution status changes:

PROCEDURE p(job_status_change IN MGMT_NOTIFY_JOB)

For corrective action status changes:

PROCEDURE p(ca_status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION)

Note:

The notification method based on a PL/SQL procedure must be configured by an administrator with Super Administrator privileges before a user can select it while creating/editing a notification rule.

For more information on passing specific types of information to scripts or PL/SQL procedures, see the following sections:

"Passing Alert and Policy Violation Information to a PL/SQL Procedure"

"Passing Corrective Action Status Change Information"

"Passing Job Execution Status Information"

Step 2: Create the PL/SQL procedure on the Management Repository.

Create the PL/SQL procedure on the repository database using one of the following procedure specifications:

PROCEDURE p(severity IN MGMT_NOTIFY_SEVERITY)

PROCEDURE p(job_status_change IN MGMT_NOTIFY_JOB)

PROCEDURE p(ca_status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION)

The PL/SQL procedure must be created on the repository database using the database account of the repository owner (such as SYSMAN)

If an error is encountered during the running of the procedure, the Notification System can be instructed to retry the sending of the notification to the procedure by raising a user defined exception that uses the error code -20000. See Example 4-11, "PL/SQL Procedure Using a Severity Code". The procedure initially retried after one minute, then two minutes, then three minutes and so on, until the notification is a day old, at which point it will be purged.

Step 3: Register your PL/SQL procedure as a new notification method.

Log in as a Super Administrator, click Setup and then Notification Methods from the vertical navigation bar. From this page, you can define a new notification based on 'PL/SQL Procedure'. See "Adding a Notification Method Based on a PL/SQL Procedure".

Make sure to use a fully qualified name that includes the schema owner, package name and procedure name. The procedure will be executed by the repository owner and so the repository owner must have execute permission on the procedure.

Create a notification method based on your PL/SQL procedure. The following information is required when defining the method:

- Name

- Description

- PL/SQL Procedure

You must enter a fully qualified procedure name (for example, OWNER.PKGNAME.PROCNAME) and ensure that the owner of the Management Repository has execute privilege on the procedure.

An example of the required information is shown in Example 4-10.

Example 4-10 PL/SQL Procedure Required Information

Name Open trouble ticket

Description Notification method to open a trouble ticket in the event

PLSQL Procedure ticket_sys.ticket_ops.open_ticket

Step 4: Assign the notification method to a rule.

You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a single rule, see "Assigning Rules to Methods" or "Assigning Methods to Rules".

There can be more than one PL/SQL-based method configured for your Enterprise Manager environment.

Information about the severity types that relate to a target's availability, and how metric severity and policy violation information is passed to the PLSQL procedure is covered in the next section.

Passing Alert and Policy Violation Information to a PL/SQL Procedure

Passing metric severity attributes (severity level, type, notification rule, rule owner, or rule owner, and so on) or policy violation information to PL/SQL procedures allows you to customize automated responses to alerts or policy violations.

The notification system passes information about metric severities or policy violations to a PL/SQL procedure using the MGMT_NOTIFY_SEVERITY object. An instance of this object is created for every alert or policy violation. When an alert or policy violation occurs, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_SEVERITY object that has been passed to it.

The following table lists all metric severity attributes that can be passed:

Table 4-5 Metric Severity Attributes

Attribute

Datatype

Additional Information

TARGET_NAME

VARCHAR2(256)

Name of the target on which the severity occurred.

TARGET_TYPE

VARCHAR2(64)

Type of target on which the severity occurred. Targets are defined as any monitorable service.

TIMEZONE

VARCHAR2(64)

The target's regional timezone

HOST_NAME

VARCHAR2(128)

Name of the machine on which the target resides.

METRIC_NAME

VARCHAR2(64)

Metric or policy generating the severity.

METRIC_DESCRIPTION

VARCHAR2(128)

Meaningful description of the metric that can be understood by other administrators.

METRIC_COLUMN

VARCHAR2(64)

For table metrics, the metric column contains the name of the column in the table that is being defined. If the metric that is being defined is not a table metric, the value in this column is a single space. This attribute is not used for policy violations.

METRIC_VALUE

VARCHAR2(1024)

The value of the metric.

KEY_VALUE

VARCHAR2(1290)

For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity.

KEY_VALUE_NAME

VARCHAR2(512)

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

KEY_VALUE_GUID

VARCHAR2(256)

GUID associated with a composite key value name.

CTXT_LIST

MGMT_NOTIFY_COLUMNS

Details of the alert context.

COLLECTION_TIMESTAMP

DATE

The time when the target status change was last detected and logged in the management repository.

SEVERITY_CODE

NUMBER

Numeric code identifying the severity level. See Severity Code table below.

MESSAGE

VARCHAR2(4000)

An optional message that is generated when the alert is created that provides additional information about the alert condition.

SEVERITY_GUID

RAW(16)

Severity global unique identifier.

METRIC_GUID

RAW(16)

Metric global unique identifier.

TARGET_GUID

RAW(16)

Target global unique identifier.

RULE_OWNER

VARCHAR2(64)

Name of the Enterprise Manager administrator who owns the rule.

RULE_NAME

VARCHAR2(132)

Name of the notification rule that resulted in the severity.

When a severity occurs for the target, the notification system creates an instance of the MGMT_NOTIFY_SEVERITY object and populates it with values from the severity. The severity codes in Table 4-6 have been defined as constants in the MGMT_GLOBAL package and can be used to determine the type of severity in the severity_code field of the MGMT_NOTIFY_SEVERITY object.

Table 4-6 Severity Codes

Name

Datatype

Value

G_SEVERITY_COMMENT

NUMBER(2)

10

G_SEVERITY_CLEAR

NUMBER(2)

15

G_SEVERITY_WARNING

NUMBER(2)

20

G_SEVERITY_CRITICAL

NUMBER(2)

25

G_SEVERITY_UNREACHABLE_CLEAR

NUMBER(3)

115

G_SEVERITY_UNREACHABLE_START

NUMBER(3)

125

G_SEVERITY_BLACKOUT_END

NUMBER(3)

215

G_SEVERITY_BLACKOUT_START

NUMBER(3)

225

G_SEVERITY_ERROR_END

NUMBER(3)

315

G_SEVERITY_ERROR_START

NUMBER(3)

325

G_SEVERITY_NO_BEACONS

NUMBER(3)

425

G_SEVERITY_UNKNOWN

NUMBER(3)

515

Example 4-11 PL/SQL Procedure Using a Severity Code

CREATE TABLE alert_log (target_name VARCHAR2(64),

alert_msg VARCHAR2(4000),

occured DATE);

PROCEDURE LOG_CRITICAL_ALERTS(severity IN MGMT_NOTIFY_SEVERITY)

IS

BEGIN

-- Log all critical severities

IF severity.severity_code = MGMT_GLOBAL.G_SEVERITY_CRITICAL

THEN

BEGIN

INSERT INTO alert_log (target_name, alert_msg, occured)

VALUES (severity.target_name, severity.message,

severity.collection_timestamp);

EXCEPTION

WHEN OTHERS

THEN

-- If there are any problems then get the notification retried

RAISE_APPLICATION_ERROR(-20000, 'Please retry');

END;

COMMIT;

END IF;

END LOG_CRITICAL_ALERTS;

Adding a Notification Method Based on an SNMP Trap

Enterprise Manager supports integration with third-party management tools through the SNMP. For example, you can use SNMP to notify a third-party application that a selected metric has exceeded its threshold.

The trap is an SNMP Version 1 trap and is described by the MIB definition shown at the end of this chapter. See "Management Information Base (MIB)".

For more comprehensive configuration information, see the documentation specific to your platform; SNMP configuration differs from platform to platform.

Note:

Notification methods based on SNMP traps must be configured by an administrator with Super Administrator privileges before any user can then choose to select one or more of these SNMP trap methods while creating/editing a notification rule.

Step 1: Define a new notification method based on an SNMP trap.

Log in to Enterprise Manager as a Super Administrator. Click Setup and then click Notification Method from the vertical navigation bar to access the Notification Methods page. From this page you can add a new method based on an SNMP trap.

You must provide the name of the host (machine) on which the SNMP master agent is running and other details as shown in the following example. In Example 4-12, the SNMP host will receive your SNMP traps.

Example 4-12 SNMP Trap Required Information

Name HP OpenView Console

Description Notification method to send trap to HP OpenView console

SNMP Trap Host Name machine1.us.oracle.com

SNMP Host Port 162

SNMP Community public

This SNMP host will receive your SNMP traps.

Note:

A Test Trap button exists for you to test your setup.

Metric severity information will be passed as a series of variables in the SNMP trap.

An example SNMP Trap is shown in Example 4-13. Each piece of information is sent as a variable embedded in the SNMP Trap.

Example 4-13 SNMP Trap

Tue Oct 28 05:00:02 2006

Command: 4

Enterprise: 1.3.6.1.4.1.111.15.2

Agent: 138.1.6.200

Generic Trap: 6

Specific Trap: 1

Time Stamp: 8464:39.99

Count: 11

Name: 1.3.6.1.4.1.111.15.1.1.1.2.1

Kind: OctetString

Value: "mydatabase"

Name: 1.3.6.1.4.1.111.15.1.1.1.3.1

Kind: OctetString

Value: "Database"

Name: 1.3.6.1.4.1.111.15.1.1.1.4.1

Kind: OctetString

Value: "myhost.com"

Name: 1.3.6.1.4.1.111.15.1.1.1.5.1

Kind: OctetString

Value: "Owner's Invalid Object Count"

Name: 1.3.6.1.4.1.111.15.1.1.1.6.1

Kind: OctetString

Value: "Invalid Object Owner"

Name: 1.3.6.1.4.1.111.15.1.1.1.7.1

Kind: OctetString

Value: "SYS"

Name: 1.3.6.1.4.1.111.15.1.1.1.8.1

Kind: OctetString

Value: "28-OCT-2006 04:59:10 (US/Eastern GMT)"

Name: 1.3.6.1.4.1.111.15.1.1.1.9.1

Kind: OctetString

Value: "Warning"

Name: 1.3.6.1.4.1.111.15.1.1.1.10.1

Kind: OctetString

Value: "12 object(s) are invalid in the SYS schema."

Name: 1.3.6.1.4.1.111.15.1.1.1.11.1

Kind: OctetString

Value: "Database Metrics"

Name: 1.3.6.1.4.1.111.15.1.1.1.12.1

Kind: OctetString

Value: "SYSMAN"

Step 2: Assign the notification method to a rule.

You can edit an existing rule (or create a new notification rule), then go to the Methods page. In the list of Advanced Notification Methods, select your notification method and click 'Assign Method to Rule'. To assign multiple rules to a method or methods to a rule, see "Assigning Methods to Rules" or "Assigning Rules to Methods".

Passing Corrective Action Status Change Information

Passing corrective action status change attributes (such as new status, job name, job type, notification rule, or rule owner) to PL/SQL procedures or OS commands/scripts allows you to customize automated responses to status changes. For example, you many want to call an OS script to open a trouble ticket for an in-house support trouble ticket system if a critical corrective action fails to run. In this case, you will want to pass status (for example, Problems or Aborted) to the script to open a trouble ticket and escalate the problem.

Passing Corrective Action Execution Status to an OS Command or Script

The notification system passes information to an OS script or executable via system environment variables. Conventions used to access environmental variables vary depending on the operating system:

- UNIX: $ENV_VARIABLE

- MS Windows: %ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.

Table 4-7 Environment Variables

Environment Variable

Description

JOB_NAME

The name of the corrective action.

JOB_OWNER

The owner of the corrective action.

JOB_TYPE

The type of corrective action.

JOB_STATUS

The corrective action status.

TIMESTAMP

Time when the severity occurred.

NUM_TARGETS

The number of targets.

TARGET_NAMEn

The name of the nth target on which the corrective action ran. Example: TARGET_NAME1, TARGET_NAME2.

METRIC

The name of the metric in the alert that caused the corrective action to run. Not set for policy violations.

POLICY_RULE

The name of the policy rule in the alert that caused the corrective action to run. Not set for metric severities.

METRIC_VALUE

The value of the metric column in the alert that caused the corrective action to run.

VIOLATION_CONTEXT

A comma-separated list of name-value pairs that show the policy violation context.

KEY_VALUE_NAME

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

KEY_VALUE

For metrics that monitor a set of objects, the KEY_VALUE indicates the specific object that triggered the severity. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE is 'USERS' if the USERS tablespace triggered at warning or critical severity.

SEVERITY

Type of alert severity. For example, severity types that relate to a target's availability are:

- UP

- DOWN

- UNREACHABLE CLEAR

- UNREACHABLE START

- BLACKOUT END

- BLACKOUT START

Other metrics can have any of the following severities:

- WARNING

- CRITICAL

- CLEAR

- METRIC ERROR CLEAR

- METRIC ERROR START

RULE_NAME

Name of the notification rule that resulted in the execution of the corrective action.

RULE_OWNER

Name of the Enterprise Manager administrator who owns the notification rule.

Passing Corrective Action Execution Status to a PLSQL Procedure

The notification system passes corrective action status change information to a PL/SQL procedure via the MGMT_NOTIFY_CORRECTIVE_ACTION object. An instance of this object is created for every status change. When a corrective action executes, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_CORRECTIVE_ACTION object that has been passed to it.

Table 4-8 lists all corrective action status change attributes that can be passed:

Table 4-8 Corrective Action Status Attributes

Attribute

Datatype

Additional Information

JOB_NAME

VARCHAR2(128)

The corrective action name.

JOB_OWNER

VARCHAR(256)

The owner of the corrective action.

JOB_TYPE

VARCHAR2(32)

The type of the corrective action.

JOB_STATUS

NUMBER

The new status of the corrective action. See Table 4-9, "Corrective Action Status Codes" for a list of possible status conditions.

STATE_CHANGE_GUID

RAW(16)

The GUID of the state change record.

JOB_GUID

RAW(16)

The unique id of the corrective action.

EXECUTION_ID

RAW(16)

The unique id of the corrective action execution.

TARGETS

SMP_EMD_NVPAIR_ARRAY

An array of the target name/target type pairs that the corrective action runs on.

METRIC_NAME

VARCHAR2(256)

The name of the metric/policy rule in the alert that caused the corrective action to run.

METRIC_COLUMN

VARCHAR2(64)

The name of the metric in the alert that caused the corrective action to run. This is not used for policy violations.

METRIC_VALUE

VARCHAR2(1024)

The value of the metric in the alert that caused the corrective action to run.

SEVERITY_CODE

NUMBER

The severity code of the alert that caused the corrective action to run. See Table 4-6, "Severity Codes".

KEY_VALUE_NAME

VARCHAR2(512)

For metrics that monitor a set of objects, the KEY_VALUE_NAME indicates the type of object monitored. For example for the Tablespace Space Used (%) metric that monitors tablespace objects, the KEY_VALUE_NAME is 'Tablespace Name'.

KEY_VALUE

VARCHAR2(1290)

For table metrics, this column contains the value of the key column for the row in the table whose thresholds are being defined. If the thresholds are not for a table metric, or the thresholds apply for all rows in the metric column, then the value in this column will contain a single space.

KEY_VALUE_GUID

RAW(16)

GUID associated with a composite key value name.

CTXT_LIST

MGMT_NOTIFY_COLUMNS

Details of the corrective action status change context.

RULE_OWNER

VARCHAR2(64)

The owner of the notification rule that caused the PL/SQL notification to be sent.

RULE_NAME

VARCHAR2(132)

The name of the notification rule that caused the PL/SQL notification method to be invoked.

OCCURRED_DATE

DATE

The time and date when the status change happened.

The following status codes are possible values for the job_status field of the MGMT_NOTIFY_CORRECTIVE_ACTION object.

Table 4-9 Corrective Action Status Codes

Name

Datatype

Value

SCHEDULED_STATUS

NUMBER(2)

1

EXECUTING_STATUS

NUMBER(2)

2

ABORTED_STATUS

NUMBER(2)

3

FAILED_STATUS

NUMBER(2)

4

COMPLETED_STATUS

NUMBER(2)

5

SUSPENDED_STATUS

NUMBER(2)

6

AGENTDOWN_STATUS

NUMBER(2)

7

STOPPED_STATUS

NUMBER(2)

8

SUSPENDED_LOCK_STATUS

NUMBER(2)

9

SUSPENDED_EVENT_STATUS

NUMBER(2)

10

SUSPENDED_BLACKOUT_STATUS

NUMBER(2)

11

STOP_PENDING_STATUS

NUMBER(2)

12

SUSPEND_PENDING_STATUS

NUMBER(2)

13

INACTIVE_STATUS

NUMBER(2)

14

QUEUED_STATUS

NUMBER(2)

15

FAILED_RETRIED_STATUS

NUMBER(2)

16

WAITING_STATUS

NUMBER(2)

17

SKIPPED_STATUS

NUMBER(2)

18

REASSIGNED_STATUS

NUMBER(2)

20

Example 4-14 PL/SQL Procedure Using a Status Code

CREATE TABLE ca_log (jobid RAW(16),                     occured DATE);

CREATE OR REPLACE PROCEDURE LOG_PROBLEM_CAS(status_change IN MGMT_NOTIFY_CORRECTIVE_ACTION)

IS

BEGIN

-- Log all failed corrective actions

IF status_change.job_status = MGMT_JOBS.FAILED_STATUS

THEN

BEGIN

INSERT INTO ca_log (jobid, occured)

VALUES (status_change.job_guid, SYSDATE);

EXCEPTION

WHEN OTHERS

THEN

-- If there are any problems then get the notification retried

RAISE_APPLICATION_ERROR(-20000, 'Please retry');

END;

COMMIT;

END IF;

END LOG_PROBLEM_CAS;

Passing Job Execution Status Information

Passing job status change attributes (such as new status, job name, job type, notification rule, or rule owner) to PL/SQL procedures or OS commands/scripts allows you to customize automated responses to status changes. For example, you many want to call an OS script to open a trouble ticket for an in-house support trouble ticket system if a critical job fails to run. In this case you will want to pass status (for example, Problems or Aborted) to the script to open a trouble ticket and escalate the problem.

Passing Job Execution Status to a PLSQL Procedure

The notification system passes job status change information to a PL/SQL procedure via the MGMT_NOTIFY_JOB object. An instance of this object is created for every status change. When a job changes status, the notification system calls the PL/SQL procedure associated with the notification rule and passes the populated object to the procedure. The procedure is then able to access the fields of the MGMT_NOTIFY_JOB object that has been passed to it.

Table 4-10 lists all corrective action status change attributes that can be passed:

Table 4-10 Job Status Attributes

Attribute

Datatype

Additional Information

job_name

VARCHAR2(128)

The job name.

job_owner

VARCHAR2(256)

The owner of the job.

job_type

VARCHAR2(32)

The type of the job.

job_status

NUMBER

The new status of the job.

state_change_guid

RAW(16)

The GUID of the state change record.

job_guid

RAW(16)

The unique id of the job.

execution_id

RAW(16)

The unique id of the execution.

targets

SMP_EMD_NVPAIR_ARRAY

An array of the target name/target type pairs that the job runs on.

rule_owner

VARCHAR2(64)

The name of the notification rule that cause the notification to be sent.

rule_name

VARCHAR2(132)

The owner of the notification rule that cause the notification to be sent.

occurred_date

DATE

The time and date when the status change happened.

When a job status change occurs for the job, the notification system creates an instance of the MGMT_NOTIFY_JOB object and populates it with values from the status change. The following status codes have been defined as constants in the MGMT_JOBS package and can be used to determine the type of status in the job_status field of the MGMT_NOTIFY_JOB object.

Table 4-11 Job Status Codes

Name

Datatype

Value

SCHEDULED_STATUS

NUMBER(2)

1

EXECUTING_STATUS

NUMBER(2)

2

ABORTED_STATUS

NUMBER(2)

3

FAILED_STATUS

NUMBER(2)

4

COMPLETED_STATUS

NUMBER(2)

5

SUSPENDED_STATUS

NUMBER(2)

6

AGENTDOWN_STATUS

NUMBER(2)

7

STOPPED_STATUS

NUMBER(2)

8

SUSPENDED_LOCK_STATUS

NUMBER(2)

9

SUSPENDED_EVENT_STATUS

NUMBER(2)

10

SUSPENDED_BLACKOUT_STATUS

NUMBER(2)

11

STOP_PENDING_STATUS

NUMBER(2)

12

SUSPEND_PENDING_STATUS

NUMBER(2)

13

INACTIVE_STATUS

NUMBER(2)

14

QUEUED_STATUS

NUMBER(2)

15

FAILED_RETRIED_STATUS

NUMBER(2)

16

WAITING_STATUS

NUMBER(2)

17

SKIPPED_STATUS

NUMBER(2)

18

REASSIGNED_STATUS

NUMBER(2)

20

Example 4-15 PL/SQL Procedure Using a Status Code (Job)

CREATE TABLE job_log (jobid RAW(16),                    occured DATE);

CREATE OR REPLACE PROCEDURE LOG_PROBLEM_JOBS(status_change IN MGMT_NOTIFY_JOB)

IS

BEGIN

-- Log all failed jobs

IF status_change.job_status = MGMT_JOBS.FAILED_STATUS

THEN

BEGIN

INSERT INTO job_log (jobid, occured)

VALUES (status_change.job_guid, SYSDATE);

EXCEPTION

WHEN OTHERS

THEN

-- If there are any problems then get the notification retried

RAISE_APPLICATION_ERROR(-20000, 'Please retry');

END;

COMMIT;

END IF;

END LOG_PROBLEM_JOBS;

Passing Job Execution Status to an OS Command or Script

The notification system passes job execution status information to an OS script or executable via system environment variables. Conventions used to access environmental variables vary depending on the operating system:

- UNIX: $ENV_VARIABLE

- MS Windows: %ENV_VARIABLE%

The notification system sets the following environment variables before calling the script. The script can then use any or all of these variables within the logic of the script.

Table 4-12 Environment Variables

Environment Variable

Description

JOB_NAME

The name of the job.

JOB_OWNER

The owner of the job.

JOB_TYPE

The type of job.

JOB_STATUS

The job status.

TIMESTAMP

Time when the severity occurred.

NUM_TARGETS

The number of targets.

TARGET_NAMEn

The name of the nth target. For example, TARGET_NAME1, TARGET_NAME2.

TARGET_TYPEn

The type of the nth target. For example TARGET_TYPE1, TARGET_TYPE2.

RULE_NAME

Name of the notification rule that resulted in the severity.

RULE_OWNER

Name of the Enterprise Manager administrator who owns the notification rule.

Passing User-Defined Target Properties to Notification Methods

Enterprise Manager allows you to define target properties (accessed via Related Links on the target home page) that can be used to store environmental or usage context information specific to that target. Target property values are passed to custom notification methods where they can be processed using conditional logic or simply passed as additional alert information to third-party devices, such as ticketing systems. By default, Enterprise Manager passes all defined target properties to notification methods.

Note:

Target properties are not passed to notification methods when short e-mail format is used.

Figure 4-4 Host Target Properties

Description of "Figure 4-4 Host Target Properties"

Assigning Methods to Rules

For each notification rule, you can assign one or more notification methods to be called when any of the criteria in the notification rule is met.

1. From the Enterprise Manager Grid Control, click Preferences at the top of the page.

2. Click Notification Rules in the vertical navigation bar.
The Enterprise Manager Grid Control displays the Notification Rules page. Any notification rules already created are listed in the Notification Rules table.

3. Click Assign Methods to Multiple Rules.

4. Perform your assignments.

Figure 4-5 Assigning Methods to Rules

Description of "Figure 4-5 Assigning Methods to Rules"

Assigning Rules to Methods

For each notification method, you can associate one or more notification rules that will use that method to send notifications.

1. From the Enterprise Manager Grid Control, click Preferences at the top of the page.

2. Click Notification Rules in the vertical navigation bar.
The Enterprise Manager Grid Control displays the Notification Rules page. Any notification rules already created are listed in the Notification Rules table.

3. Click Assign Methods to Multiple Rules.

4. From the View menu, select By Method.

5. Perform your assignments.

Figure 4-6 Assign Rules to Methods

Description of "Figure 4-6 Assign Rules to Methods"

Notification Coverage

To reduce the likelihood of an alert triggering and no administrator being notified because there was no notification rule covering that condition, you can use the Information Publisher (Enterprise Manager Report system) to view, for each target, the metrics monitored for that target and associated notification rules. Information Publisher provides an out-of-box report specifically designed for this purpose. You can run this report from the Report Definitions page (Reports tab) under Monitoring-->Alerts and Policy Violations--> Notification Rule Coverage for Metric Alerts and Availabilities (Target)

Management Information Base (MIB)

Enterprise Manager Grid Control can send SNMP Traps to third-party, SNMP-enabled applications. Details of the trap contents can be obtained from the management information base (MIB) variables. The following sections discuss Enterprise Manager MIB variables in detail.

About MIBs

A MIB is a text file, written in ASN.1 notation, which describes the variables containing the information that SNMP can access. The variables described in a MIB, which are also called MIB objects, are the items that can be monitored using SNMP. There is one MIB for each element being monitored. Each monolithic or subagent consults its respective MIB in order to learn the variables it can retrieve and their characteristics. The encapsulation of this information in the MIB is what enables master agents to register new subagents dynamically — everything the master agent needs to know about the subagent is contained in its MIB. The management framework and management applications also consult these MIBs for the same purpose. MIBs can be either standard (also called public) or proprietary (also called private or vendor).

The actual values of the variables are not part of the MIB, but are retrieved through a platform-dependent process called "instrumentation". The concept of the MIB is very important because all SNMP communications refer to one or more MIB objects. What is transmitted to the framework is, essentially, MIB variables and their current values.

Reading the MIB Variable Descriptions

This section covers the format used to describe MIB variables. Note that the STATUS element of SNMP MIB definition, Version 2, is not included in these MIB variable descriptions. Since Oracle has implemented all MIB variables as CURRENT, this value does not vary.

Variable Name

Syntax

Maps to the SYNTAX element of SNMP MIB definition, Version 2.

Max-Access

Maps to the MAX-ACCESS element of SNMP MIB definition, Version 2.

Status

Maps to the STATUS element of SNMP MIB definition, Version 2.

Explanation

Describes the function, use and precise derivation of the variable. (For example, a variable might be derived from a particular configuration file parameter or performance table field.) When appropriate, incorporates the DESCRIPTION part of the MIB definition, Version 2.

Typical Range

Describes the typical, rather than theoretical, range of the variable. For example, while integer values for many MIB variables can theoretically range up to 4294967295, a typical range in an actual installation will vary to a lesser extent. On the other hand, some variable values for a large database can actually exceed this "theoretical" limit (a "wraparound"). Specifying that a variable value typically ranges from 0 to 1,000 or 1,000 to 3 billion will help the third-party developer to develop the most useful graphical display for the variable.

Significance

Describes the significance of the variable when monitoring a typical installation. Alternative ratings are Very Important, Important, Less Important, or Not Normally Used. Clearly, the DBA will want to monitor some variables more closely than others. However, which variables fall into this category can vary from installation to installation, depending on the application, the size of the database, and on the DBA's objectives. Nevertheless, assessing a variable's significance relative to the other variables in the MIB can help third-party developers focus their efforts on those variables of most interest to the most DBAs.

Related Variables

Lists other variables in this MIB, or other MIBs implemented by Oracle, that relate in some way to this variable. For example, the value of this variable might derive from that of another MIB variable. Or perhaps the value of this variable varies inversely to that of another variable. Knowing this information, third-party developers can develop useful graphic displays of related MIB variables.

Suggested Presentation

Suggests how this variable can be presented most usefully to the DBA using the management application: as a simple value, as a gauge, or as an alarm, for example.

MIB Definition

Example 4-16 shows a typical MIB definition used by Enterprise Manager.

Example 4-16 MIB Definition

ORACLE-ENTERPRISE-MANAGER-4-MIB DEFINITIONS ::= BEGIN

IMPORTS

TRAP-TYPE

FROM RFC-1215

DisplayString

FROM RFC1213-MIB

OBJECT-TYPE

FROM RFC-1212

enterprises

FROM RFC1155-SMI;

oracle OBJECT IDENTIFIER ::= { enterprises  111 }

oraEM4 OBJECT IDENTIFIER ::= { oracle  15 }

oraEM4Objects OBJECT IDENTIFIER ::= { oraEM4  1 }

oraEM4AlertTable OBJECT-TYPE

SYNTAX  SEQUENCE OF OraEM4AlertEntry

ACCESS  not-accessible

STATUS  mandatory

DESCRIPTION

"Information on alerts generated by Oracle Enterprise Manager. This table is

not queryable; it exists only to document the variables included in the

oraEM4Alert trap.  Each trap contains a single instance of each variable in the

table."

::= { oraEM4Objects  1 }

oraEM4AlertEntry OBJECT-TYPE

SYNTAX  OraEM4AlertEntry

ACCESS  not-accessible

STATUS  mandatory

DESCRIPTION

"Information about a particular Oracle Enterprise Manager alert."

INDEX   { oraEM4AlertIndex }

::= { oraEM4AlertTable  1 }

OraEM4AlertEntry ::=

SEQUENCE {

oraEM4AlertIndex

INTEGER,

oraEM4AlertTargetName

DisplayString,

oraEM4AlertTargetType

DisplayString,

oraEM4AlertHostName

DisplayString,

oraEM4AlertMetricName

DisplayString,

oraEM4AlertKeyName

DisplayString,

oraEM4AlertKeyValue

DisplayString,

oraEM4AlertTimeStamp

DisplayString,

oraEM4AlertSeverity

DisplayString,

oraEM4AlertMessage

DisplayString,

oraEM4AlertRuleName

DisplayString

oraEM4AlertRuleOwner

DisplayString

oraEM4AlertMetricValue

DisplayString,

oraEM4AlertContext

DisplayString

}

oraEM4AlertIndex OBJECT-TYPE

SYNTAX  INTEGER

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"Index of a particular alert, unique only at the moment an alert is generated."

::= { oraEM4AlertEntry  1 }

oraEM4AlertTargetName OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The name of the target to which this alert applies."

::= { oraEM4AlertEntry  2 }

oraEM4AlertTargetType OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The type of the target to which this alert applies."

::= { oraEM4AlertEntry  3 }

oraEM4AlertHostName OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The name of the host on which this alert originated."

::= { oraEM4AlertEntry  4 }

oraEM4AlertMetricName OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The name of the metric or policy which generated this alert."

::= { oraEM4AlertEntry  5 }

oraEM4AlertKeyName OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The name of the key-column, if present, for the metric which generated this alert."

::= { oraEM4AlertEntry  6 }

oraEM4AlertKeyValue OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The value of the key-column, if present, for the metric which generated this alert."

::= { oraEM4AlertEntry  7 }

oraEM4AlertTimeStamp OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The time at which this alert was generated."

::= { oraEM4AlertEntry  8 }

oraEM4AlertSeverity OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The severity of the alert e.g. Critical."

::= { oraEM4AlertEntry  9 }

oraEM4AlertMessage OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The message associated with the alert."

::= { oraEM4AlertEntry  10 }

oraEM4AlertRuleName OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The name of the notification rule that caused this notification."

::= { oraEM4AlertEntry  11 }

oraEM4AlertRuleOwner OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The owner of the notification rule that caused this notification."

::= { oraEM4AlertEntry  12 }

oraEM4AlertMetricValue OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The value of the metric which caused this alert to be generated."

::= { oraEM4AlertEntry  13 }

oraEM4AlertContext OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"A comma separated list of metric column names and values associated with the metric that caused this alert to be generated."

::= { oraEM4AlertEntry  14 }

oraEM4Traps OBJECT IDENTIFIER ::= { oraEM4  2 }

oraEM4Alert TRAP-TYPE

ENTERPRISE  oraEM4Traps

VARIABLES   { oraEM4AlertTargetName, oraEM4AlertTargetType,

oraEM4AlertHostName, oraEM4AlertMetricName,

oraEM4AlertKeyName, oraEM4AlertKeyValue, oraEM4AlertTimeStamp,

oraEM4AlertSeverity, oraEM4AlertMessage,

oraEM4AlertRuleName, oraEM4AlertRuleOwner,

oraEM4AlertMetricValue, oraEM4AlertContext }

DESCRIPTION

"The variables included in the oraEM4Alert trap."

::= 1

oraEM4JobAlertTable OBJECT-TYPE

SYNTAX  SEQUENCE OF OraEM4JobAlertEntry

ACCESS  not-accessible

STATUS  mandatory

DESCRIPTION

"Information on alerts generated by Oracle Enterprise Manager. This table is

not queryable; it exists only to document the variables included in the

oraEM4JobAlert trap.  Each trap contains a single instance of each variable in

the table."

::= { oraEM4Objects  2 }

oraEM4JobAlertEntry OBJECT-TYPE

SYNTAX  OraEM4AlertEntry

ACCESS  not-accessible

STATUS  mandatory

DESCRIPTION

"Information about a particular Oracle Enterprise Manager alert."

INDEX   { oraEM4JobAlertIndex }

::= { oraEM4JobAlertTable  1 }

OraEM4JobAlertEntry ::=

SEQUENCE {

oraEM4JobAlertIndex

INTEGER,

oraEM4JobAlertJobName

DisplayString,

oraEM4JobAlertJobOwner

DisplayString,

oraEM4JobAlertJobType

DisplayString,

oraEM4JobAlertJobStatus

DisplayString,

oraEM4JobAlertTargets

DisplayString,

oraEM4JobAlertTimeStamp

DisplayString,

oraEM4JobAlertRuleName

DisplayString

oraEM4JobAlertRuleOwner

DisplayString,

oraEM4JobAlertMetricName

DisplayString,

oraEM4JobAlertMetricValue

DisplayString,

oraEM4JobAlertContext

DisplayString,

oraEM4JobAlertKeyName

DisplayString,

oraEM4JobAlertKeyValue

DisplayString,

oraEM4JobAlertSeverity

DisplayString

}

oraEM4JobAlertIndex OBJECT-TYPE

SYNTAX  INTEGER

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"Index of a particular alert, unique only at the moment an alert is

generated."

::= { oraEM4JobAlertEntry  1 }

oraEM4JobAlertJobName OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The name of the job to which this alert applies."

::= { oraEM4JobAlertEntry  2 }

oraEM4JobAlertJobOwner OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The owner of the job to which this alert applies."

::= { oraEM4JobAlertEntry  3 }

oraEM4JobAlertJobType OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The type of the job to which this alert applies."

::= { oraEM4JobAlertEntry  4 }

oraEM4JobAlertJobStatus OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The status of the job to which this alert applies."

::= { oraEM4JobAlertEntry  5 }

oraEM4JobAlertTargets OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"A comma separated list of target to which this alert applies."

::= { oraEM4JobAlertEntry  6 }

oraEM4JobAlertTimeStamp OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The time at which this job status changed causing this alert."

::= { oraEM4JobAlertEntry  7 }

oraEM4JobAlertRuleName OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The name of the notification rule that caused this notification."

::= { oraEM4JobAlertEntry  8 }

oraEM4JobAlertRuleOwner OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The owner of the notification rule that caused this notification."

::= { oraEM4JobAlertEntry  9 }

oraEM4JobAlertMetricName OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The name of the metric or policy which caused the Corrective Action to run

that caused this alert."

::= { oraEM4JobAlertEntry  10 }

oraEM4JobAlertMetricValue OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The value of the metric which caused the Corrective Action to run that

caused this alert."

::= { oraEM4JobAlertEntry  11 }

oraEM4JobAlertContext OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"A comma separated list of metric column names and values associated with the

metric which caused the Corrective Action to run that caused this alert."

::= { oraEM4JobAlertEntry  12 }

oraEM4JobAlertKeyName OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The name of the key-column, if present, for the metric which caused the

Corrective Action to run that generated this alert."

::= { oraEM4JobAlertEntry  13 }

oraEM4JobAlertKeyValue OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The value of the key-column, if present, for the metric which caused the

Corrective Action to run that generated this alert."

::= { oraEM4JobAlertEntry  14 }

oraEM4JobAlertSeverity OBJECT-TYPE

SYNTAX  DisplayString

ACCESS  read-only

STATUS  mandatory

DESCRIPTION

"The severity of the metric which caused the Corrective Action to run that

generated this alert e.g. Critical."

::= { oraEM4JobAlertEntry  15 }

oraEM4JobAlert TRAP-TYPE

ENTERPRISE  oraEM4Traps

VARIABLES   { oraEM4JobAlertJobName, oraEM4JobAlertJobOwner,

oraEM4JobAlertJobType, oraEM4JobAlertJobStatus,

oraEM4JobAlertTargets, oraEM4JobAlertTimeStamp,

oraEM4JobAlertRuleName, oraEM4JobAlertRuleOwner,

oraEM4JobAlertMetricName, oraEM4JobAlertMetricValue,

oraEM4JobAlertContext, oraEM4JobAlertKeyName,

oraEM4JobAlertKeyValue, oraEM4JobAlertSeverity }

DESCRIPTION

"The variables included in the oraEM4JobAlert trap."

::= 2

END

Troubleshooting Notifications

To function properly, the notification system relies on various components of Enterprise Manager and your IT infrastructure. For this reason, there can be many causes of notification failure. The following guidelines and suggestions can help you isolate potential problems with the notification system.

General Setup

The first step in diagnosing notification issues is to ensure that you have properly configured and defined your notification environment.

OS Command, PL/SQL and SNMP Trap Notifications

Make sure all OS Command, PLSQL and SNMP Trap Notification Methods are valid by clicking the Test button. This will send a test notification and show any problems the OMS has in contacting the method. Make sure that your method was called, for example, if the OS Command notification is supposed to write information to a log file, check that it has written information to its log file.

E-mail Notifications

- Make sure an e-mail gateway is set up under the Notification Methods page of Setup. The Sender's e-mail address should be valid. Clicking the Test button will send an e-mail to the Sender's e-mail address. Make sure this e-mail is received. Note that the Test button ignores any Notification Schedule.

- Make sure an e-mail address is setup under General page of Preferences. Clicking the Test button will send an e-mail to specified address and you should make sure this e-mail is received. Note that the Test button ignores any Notification Schedule.

- Make sure an e-mail schedule is defined under the Schedule page of Preferences. No e-mails will be sent unless a Notification Schedule has been defined.

- Make sure a Notification Rule is defined to match the target, metric, severity and availability states you are interested and make sure e-mail and notification methods are assigned to the rule. A summary of the notification rule can be checked by going to the Rules page under Setup and clicking the rule name.

Notification System Errors

For any alerts involving problems with notifications, check the following for notification errors.

- Any serious errors in the Notification System are logged as system errors in the MGMT_SYSTEM_ERROR_LOG table. These errors can be seen in the Errors page under Management Services and Repository under Setup.

- Check for any delivery errors. From the Alerts section of a target home page, click on the alert message to access the metric details page. In the Alert History section, click on the Details icon for more information about the alert. The details will give the reason why the notification was not delivered. Delivery errors are stored in MGMT_NOTIFICATION_LOG with the DELIVERED column set to 'N'.

- Severities will not be displayed in the Grid Control console if no metric values have been loaded for the metric associated with the severity.

Notification System Trace Messages

The Notification System can produce trace messages in sysman/log/emoms.trc file.

Tracing is configured by setting the following flag in sysman/config/emomslogging.properties file. You can set the trace level to INFO, WARN, DEBUG. For example,

log4j.em.notification=DEBUG

Trace messages contain the string "em.notification". If you are working in a UNIX environment, you can search for messages in the emoms.trc file using the grep command. For example,

grep em.notification emoms.trc

What to look for in the trace file.

The following entries in the emoms.trc file are relevant to notifications.

Normal Startup Messages

When the OMS starts, you should see these types of messages.

2006-11-08 03:18:45,385 [Orion Launcher] INFO  em.notification init.1279 - Short format maximum length is 155

2006-11-08 03:18:45,386 [Orion Launcher] INFO  em.notification init.1297 - Short format is set to both subject and body

2006-11-08 03:18:45,387 [NotificationMgrThread] INFO  em.notification run.1010 - Waiting for connection to EM Repository...

2006-11-08 03:18:46,006 [NotificationMgrThread] INFO  em.notification run.1041 - Registering for Administrative Queue Name...

2006-11-08 03:18:46,250 [NotificationMgrThread] INFO  em.notification run.1078 - Administrative Queue is ADM21

2006-11-08 03:18:46,250 [NotificationMgrThread] INFO  em.notification run.1089 - Creating thread pool: min = 6 max = 24

2006-11-08 03:18:48,206 [NotificationMgrThread] INFO  em.notification handleAdminNotification.655 - Handling notifications for EMAIL1

Notification Delivery Messages

2006-11-08 03:18:45,387 [NotificationMgrThread] INFO  em.notification run.682 - Notification ready on EMAIL1

2006-11-08 03:18:46,006 [DeliveryThread-EMAIL1] INFO  em.notification run.114 - Deliver to SYSMAN/admin@oracle.com

2006-11-08 03:18:47,006 [DeliveryThread-EMAIL1] INFO  em.notification run.227 - Notification handled for SYSMAN/admin@oracle.com

Notification System Error Messages

2006-11-08 07:26:30,242 [NotificationMgrThread] ERROR  em.notification getConnection.237 - Failed to get a connection Io exception: The Network Adapter could not establish the connection

E-mail Errors

The SMTP gateway is not set up correctly:

Failed to send e-mail to my.admin@oracle.com: For e-mail notifications to be sent, your Super Administrator must configure an Outgoing Mail (SMTP) Server within Enterprise Manager. (SYSMAN, myrule)

Invalid host name:

Failed to connect to gateway: badhost.us.oracle.com: Sending failed;

nested exception is:

javax.mail.MessagingException: Unknown SMTP      host: badhost.us.oracle.com;

Invalid e-mail address:

Failed to connect to gateway: rgmemeasmtp.oraclecorp.com: Sending failed;

nested exception is:

javax.mail.MessagingException: 550 5.7.1         <smpemailtest_ie@oracle.com>... Access denied

Always use the Test button to make sure the e-mail gateway configuration is valid. Check that an e-mail is received at the sender's e-mail address

OS Command Errors

When attempting to execute an OS command or script, the following errors may occur. Use the Test button to make sure OS Command configuration is valid. If there are any errors, they will appear in the console.

Invalid path or no read permissions on file:

Could not find /bin/myscript (stacb10.us.oracle.com_Management_Service) (SYSMAN, myrule )

No execute permission on executable:

Error calling /bin/myscript: java.io.IOException: /bin/myscript: cannot execute (stacb10.us.oracle.com_Management_Service) (SYSMAN, myrule )

Timeout because OS Command ran too long:

Timeout occurred running /bin/myscript (stacb10.us.oracle.com_Management_Service) (SYSMAN, myrule )

Any errors such as out of memory or too many processes running on OMS machine will be logged as appropriate.

Always use the Test button to make sure OS Command configuration is valid.

SNMP Trap Errors

Use the Test button to make sure SNMP Trap configuration is valid.

Other possible SNMP trap problems include: invalid host name, port, or community for a machine running an SNMP Console.

PL/SQL Errors

When attempting to execute an PL/SQL procedure, the following errors may occur. Use the Test button to make sure the procedure is valid. If there are any errors, they will appear in the console.

Procedure name is invalid or is not fully qualified. Example: SCOTT.PKG.PROC

Error calling PL/SQL procedure plsql_proc: ORA-06576: not a valid function or procedure name (SYSMAN, myrule)

Procedure is not the correct signature. Example: PROCEDURE p(s IN MGMT_NOTIFY_SEVERITY)

Error calling PL/SQL procedure plsql_proc: ORA-06553: PLS-306: wrong number or types of arguments in call to 'PLSQL_PROC' (SYSMAN, myrule)

Procedure has bug and is raising an exception.

Error calling PL/SQL procedure plsql_proc: ORA-06531: Reference to uninitialized collection (SYSMAN, myrule)

Care should be taken to avoid leaking cursors in your PL/SQL. Any exception due to this condition will result in delivery failure with the message being displayed in the Details section of the alert in the Grid Control console.

Always use the Test button to make sure the PL/SQL configuration is valid.

5 User-Defined Metrics

User-defined metrics allow you to extend the reach of Enterprise Manager's monitoring to conditions specific to particular environments via custom scripts or SQL queries and function calls. Once defined, user-defined metrics will be monitored, aggregated in the repository and trigger alerts like regular metrics.

This chapter covers the following topics:

- Extending Monitoring Capability

- Creating OS-Based User-Defined Metrics

- Creating a SQL-Based User-Defined Metric

- Notifications, Corrective Actions, and Monitoring Templates

- Changing User-Defined Metric Credentials

Extending Monitoring Capability

There are two types of user-defined metrics:

- OS-Based User-Defined Metrics: Accessed from Host target home pages, these user-defined metrics allow you to define new metrics using custom Operating System (OS) scripts.
To monitor for a particular condition (for example, check successful completion of monthly system maintenance routines), you can write a custom script that will monitor that condition, then create an OS-based user-defined metric that will use your custom script. Each time the metric is evaluated by Enterprise Manager, it will use the specified script, relying on that script to return the value of the condition.

- SQL-Based User-Defined Metrics: Accessed from Database target home pages, these user-defined metrics allow you to implement custom database monitoring using SQL queries or function calls.
SQL-based user-defined metrics do not use external scripts. You enter SQL directly into the Enterprise Manager user interface at the time of metric creation

Once a user-defined metric is created, all other monitoring features, such as alerts, notifications, historical collections, and corrective actions are automatically available to it.

Administrators who already have their own library of custom monitoring scripts can leverage these monitoring features by integrating their scripts with Enterprise Manager via user-defined metrics. Likewise, existing SQL queries or function calls currently used to monitor database conditions can be easily integrated into Enterprise Manager's monitoring framework using the SQL-based user-defined metric.

Creating OS-Based User-Defined Metrics

Creating an OS-based user-defined metric involves two steps:

- Step 1: Create Your OS Monitoring Script

- Step 2: Register the Script as a User-Defined Metric

Create Your OS Monitoring Script

Using a scripting language of your choice, create a script that contains logic to check for the condition being monitored. For example, scripts that check for disk space or memory usage. All scripts to be run with user-defined metrics should be placed in a directory to which the Management Agent has full access privileges. Scripts themselves must have the requisite permissions set so that they can be executed by the Management Agent. The script runtime environment must also be configured: If your script requires an interpreter, such as a Perl interpreter, this must be installed on that host as well.

All monitoring scripts should contain code to perform the following basic functions:

- Code to check the status of monitored objects

- Code to return script results to Enterprise Manager

Code to check the status of monitored objects

Define logic in the code that checks the condition being monitored such as determining the amount of free space on a particular file system or level of memory usage.

After checking the monitored condition, the script should return the value associated with the monitored object.

When you choose to have the script return a specific value from the monitored object (for example, current disk space usage), you can also have Enterprise Manager evaluate the object's current value against specific warning and critical thresholds. You specify these warning and critical thresholds from the Grid Control console at the time you create the user-defined metric. Based on the evaluation of the metric's value against the thresholds, an alert may be triggered at one of the following severity levels:

Table 5-1 Metric Severity Levels

Severity Level

Status

Script Failure

The script failed to run properly.

Clear

No problems with the object monitored; status is clear. If thresholds were specified for the metric, then it means the thresholds were not reached.

Warning

The value of the monitored object reached the warning threshold.

Critical

The value of the monitored object reached the critical threshold.

Code to return script results to Enterprise Manager

After checking the monitored condition, the script should return the value associated with the monitored object. The script returns values back to Enterprise Manager by sending formatted information to standard output (stdout) using the syntax that is consistent with the scripting language (the "print" statement in Perl, for example). Enterprise Manager then checks the standard output of a script for this formatted information; specifically it checks for the tags: em_result and em_message and the values assigned to these tags.

The script must assign the value of the monitored object to the tag em_result. The output must be written as a string delimited by new line characters. For example, if the value of the monitored object is 200, your script can return this to Enterprise Manager as shown in this Perl statement:

print "em_result=200\n"

You can also have Enterprise Manager evaluate the returned value against specified warning and critical thresholds. You specify these warning and critical thresholds when you register your script as a user-defined metric in the console.

If the comparison between the warning or critical threshold holds true, a warning or critical alert will be generated. The default message for this alert will be:

"The value is $em_result".

You can choose to override this default message with a custom message by assigning the string to be used to the tag em_message.

For example, if you want your alert message to say 'Disk usage is high", your script can return this custom message as follows:

print "em_message=Disk usage is high\n"

Important:

Script output tags must be lower-case in order for Enterprise Manager to recognize the script output as valid user-defined metric feedback. Messages or values associated with each tag can be mixed case.

- Valid tag output: em_result=My Value\n

- Invalid tag output: Em_Result=My Value\n

For a successful script execution, the script output must start with the "em_result=" string in a new line. The message must start with the "em_message=" string in a new line.

The following table summarizes the script output tags.

Table 5-2 Script Output Information Tags

Tag

Definition

em_result

Use this tag to return script result values. Exactly one em_result tag must be found in STDOUT. If more than one em_result tag is found, the first tag encountered will be used; subsequent em_result tags will be ignored.

Example:

print "em_result=200\n"

Returns 200 as the value of the monitored object.

em_message

Use this tag to specify a message with the script result value in STDOUT. For OS-based user-defined metrics, only one em_message tag is permitted. If you submit more than one em_message tag, only the first tag is used. Subsequent tags are ignored.

Example:

print "em_result=200\nem_message=Disk usage is high\n"

Returns 200 as the value of the monitored object in addition to the message "Disk usage is high".

If you want to include the value of em_result in the message, you can use the placeholder $em_result.

Example:

print "em_message=Disk usage is at $em_result.\n"

If script execution is successful AND it does not contain a em_message string, a default em_message string is automatically generated. The following message format is used:

em_message=The value is $em_result

Example:

print "em_result=200\n"

Returns 200 as the value of the monitored object and the generated message "The value is 200"

The output of the user-defined monitoring script must be either em_result or em_message. In the event of system error, such as Perl aborting and writing information to STDERR pertaining to invalid commands, the script returns:

- Non-zero value

- STDOUT and STDERR messages are concatenated and sent to STDERR

This error situation results in a metric error for this user-defined metric. You can view metric errors in the Errors page of the Alerts tab in the Enterprise Manager console.

OS Script Location

Oracle recommends that user-defined metric OS scripts reside in a location outside the Agent Oracle Home. Doing so isolates scripts from any changes that may occur as a result of an Agent upgrade and ensures your scripts remain operational. When registering your script in the Grid Control console, you must specify the full path to the script. Do not use Available Properties (for example, %scriptsDir% or %emdRoot%) as part of the path specification.

Script Runtime Environment

When the user-defined metric is evaluated, it executes the script using the credentials (user name and password) specified at the time the user-defined metric was registered in the Enterprise Manager console. See "Register the Script as a User-Defined Metric". Ensure that the user name and password you specify for the user-defined metric is an active account (on that machine) possessing the requisite permissions to run the script.

Register the Script as a User-Defined Metric

Once you have created the monitoring script, you are ready to add this monitoring functionality to Enterprise Manager as a user-defined metric.

Important:

: For OS-based user-defined metrics, make sure the Management Agent is up and running on the machine where the monitoring script resides before creating the user-defined metric. Operator privilege or higher is required on the host target.

Creating an OS-Based User-Defined Metric

1. From the home page of the Host that has your OS monitoring script (Related Links), choose User-Defined Metrics. The User-Defined Metrics summary page appears containing a list of previously defined User-Defined Metrics. From this page, you perform edit, view, delete, or create like functions on existing User-Defined Metrics.

2. Click Create. The Create User-Defined Metric page appears.

3. Enter the requisite metric definition, threshold, and scheduling information. For the Command Line field, enter the full path to your script, including any requisite shell or interpreters. For example, /bin/sh myscript. See the following section for more details.

4. Click OK. The User-Defined Metric summary page appears with the new User-Defined Metric appended to the list.

If the user-defined metric has been created and the severity has not been updated recently, it is possible that there are metric errors associated with the user-defined metric execution. In this situation, access the Errors subtab under Alerts tab to check.

Create User-Defined Metric Page (OS-based User-Defined Metric)

The Create User-Defined Metric page allows you to specify the metric information required to successfully register the metric with Enterprise Manager. The page is divided into functional categories:

- Definition: You define the operational and environmental parameters required to run your script, in addition to the name of the script itself.

- Operating System Credentials: You enter the credentials used to run the monitoring script. See Enterprise Manager online help for more details on Response Actions. This functional area appears when creating OS-based user-defined metrics.

- Thresholds: To have the value returned by your script compared with set threshold values, enter the requisite threshold information. The value of the monitored metric returned by your script (as specified by em_result) will be compared against the thresholds you specify. If the comparison holds true, a warning or critical alert may be generated.

- Schedule: Specify the start time and frequency at which the user-defined script should be run. The time zone used is that of the Agent running on the monitored host.

The following figures show the Create User-Defined Metric pages for an OS-based user-defined metric. When accessing this page from any Host home page, the Create User-Defined Metric page appears as shown in Figure 5-1.

Figure 5-1 Create User-Defined Metric Page (OS-Based)

Description of "Figure 5-1 Create User-Defined Metric Page (OS-Based)"

Key elements of this page are described in the following tables.

Table 5-3 Create User-Defined Metric Page: Definition

User-Interface Element

Description

Metric Name

Metric name identifying the user-defined metric in the Enterprise Manager user interface. This name must be unique for all User-Defined Metrics created on that host.

Metric Type

Type of the value returned by the user-defined script. Choose "NUMBER" if your script returns a number. Choose "STRING" if your script returns an alphanumeric text string.

Command Line

Enter the complete command line entry required to execute the user-defined script. You must enter the full command path as well as full path to the script location. For example, to run a Perl script, you might enter something like the following in the Command Line entry field:

/u1/bin/perl /u1/scripts/myScript.pl

The content of the Command Line is passed as a literal string, so you may use any syntax, special characters, or parameters allowed by your operating system.

Environment

Optional. Enter any environmental variable(s) required to run the user-defined script. A list of predefined properties that can be passed to your script as variables is listed in the Available Properties box. You may also specify your own environment variables. Multiple variables can be defined as a space-separated list.

Example: If your script uses three variables (var1, var2, var3) where var1 is the location of the Perl directory (predefined), var2 is the directory where your Perl scripts are stored (predefined), and var3 is an Oracle home, your entry in the Environment text entry field would appear as follows:

var1=%perlBin% var2=%scriptsDir% var3=/u1/orahome10

Table 5-4 Create User-Defined Metric Page: Operating System

User-Interface Element

Description

User Name

Enter the user name for a valid operating system account on the machine where the script is to be run. Make sure the specified account has the requisite privileges to access the script directory and execute the script.

Password

Enter the password associated with the User Name.

Table 5-5 Create User-Defined Metric Page: Threshold

User-Interface Element

Description

Comparison Operator

Select the comparison method Enterprise Manager should use to compare the value returned by the user-defined script to the threshold values.

Available Comparison Operators

Operator Value Metric Type Description

= Number equal to

> Number greater than

< Number less than

>= Number greater than or equal to

<= Number less than or equal to

!= Number not equal to

CONTAINS String contains at least

MATCH String exact match

Warning

The value returned by the script is compared to the Warning threshold value using the specified comparison operator. If this comparison holds true, an alert triggers at the warning severity level.

Specifically, an alert triggers at the warning severity level if the following comparison is true:

<script_value> <comparison_operator> <warning_threshold>

and if the consecutive occurrences preceding notification has been reached.

Critical

The value returned by the script is compared to the Critical threshold value using the specified comparison operator. If this comparison holds true, an alert triggers at the critical severity level.

Specifically, an alert triggers at the critical severity level if the following comparison is true:

<script_value> <comparison_operator> <critical_threshold>

and if the consecutive occurrences preceding notification has been reached.

Consecutive Occurrences Preceding Notification

Consecutive number of times a returned value reaches either the warning or critical thresholds before an alert triggers at a warning or critical severity. This feature is useful when monitoring for sustained conditions. For example, if your script monitors the CPU load for a particular machine, you do not want to be notified at every CPU load spike. Instead, you are only concerned if the CPU load remains at a particular threshold (warning or critical) level for a consecutive number of monitoring cycles.

Response Action

Optional. Specify a script or command that will be executed if the user-defined metric generates a warning or critical alert. The script or command will be executed using the credentials of the Agent owner. Important: Only an Enterprise Manager Super Administrator can create/edit response actions for metrics.

For example, the Management Agent executes the response action if:

The Alert severity is Warning or Critical

AND

There is a change in severity (for example, warning -> critical, critical --> warning, clear --> warning or critical)

For more information, see Enterprise Manager online help.

The User-Defined Metric Schedule interface lets you specify when the Management Agent should start monitoring and the frequency at which it should monitor the condition using your OS script.

OS-Based User-Defined Metric Example

The sample Perl script used in this example monitors the 5-minute load average on the system. The script performs this function by using the 'uptime' command to obtain the average number of jobs in the run queue over the last 5 minutes.

The script is written in Perl and assumes you have Perl interpreter located in /usr/local/bin on the monitored target.

This script, called udmload.pl, is installed in a common administrative script directory defined by the user. For example, /u1/scripts.

Important:

Do not store user-defined metric monitoring scripts in the same location as Enterprise Manager system scripts.

Full text of the script:

#!/usr/local/bin/perl

# Description: 5-min load average.

# Sample User Defined Event monitoring script.

$ENV{PATH} = "/bin:/usr/bin:/usr/sbin";

$DATA = `uptime`;

$DATA =~ /average:\s+([\.\d]+),\s+([\.\d]+),\s+([\.\d]+)\s*$/;

if (defined $2) {

print "em_result=$2\n";

} else {

die "Error collecting data\n";

}

1. Copy the script (udmload.pl) to the monitored target. For example: /u1/scripts. Make sure you have an Enterprise Manager 10g Management Agent running on this machine.

2. Edit the script, if necessary, to point to the location of the Perl interpreter on the monitored target. By default, the script assumes the Perl interpreter is in /usr/local/bin.

3. As a test, run the script: udmload.pl You may need to set its file permissions so that it runs successfully. You should see output of this form:

em_result=2.1

4. In Create User-Defined Metric page, create a new user-defined metric as follows:

1. Definition Settings

- Metric Name: Test User-Defined Metric

- Metric Type: Number

- Command Line: %perlBin%/perl /u1/scripts/udmload.pl

- Environment: leave blank

- Operating System User Name: <OS user able to execute the script>

- Password: ******

2. Threshold Settings

- Comparison Operator: >=

- Critical Threshold: 0.005

- Warning Threshold: 0.001

- Consecutive Occurrences Preceding Notification: 1

In this example, we want the metric to trigger an alert at a Warning level if the 5-minute load average on the machine reaches 0.001, and trigger an alert at a Critical level if the 5-minute load average reaches 0.005. Feel free to change these thresholds depending on your system.

3. Schedule Settings:

- Start: Immediately after creation

- Frequency: Repeat every 5 minutes. You must specify at least a 5 minute interval.

Setting Up the Sample Script as a User-Defined Metric

When the 5-minute load reaches at least 0.001, you should see the metric trigger an alert.

Creating a SQL-Based User-Defined Metric

You can also define new metrics using custom SQL queries or function calls supported against single instance databases and instances on Real Application Clusters (RAC). To create this type of user-defined metric, you must have Enterprise Manager Operator privileges on the database:

1. From the Related Links area of any Database home page, choose User-Defined Metrics. The User-Defined Metrics summary page appears containing a list of previously defined user-defined metrics. From this page, you perform edit, view, delete, or create like functions on existing user-defined metrics.

2. Click Create. The Create User-Defined Metric page appears.

3. Enter the requisite metric definition, threshold, and scheduling information. For the SQL Query field, enter the query or function call. See the following section for more information.
Click Test to verify that the SQL query or function call can be executed successfully using the credentials you have specified

4. Click OK. The User-Defined Metric summary page appears with the new user-defined metric appended to the list.

If the user-defined metric has been created and the severity has not been updated recently, it is possible that there are metric errors associated with the user-defined metric execution. In this situation, access the Errors subtab under Alerts tab to check.

Create User-Defined Metric Page (SQL-Based User-Defined Metric)

The Create User-Defined Metric page allows you to specify the metric information required to successfully register the metric with Enterprise Manager. The page is divided into functional categories:

- Definition: You define the operational and environmental parameters required to run your script, in addition to the name of the script itself.

- Database Credentials: You enter the user name and password for a valid user account on the database where the SQL is to be run. Make sure the specified user account has the requisite administrative and access privileges to execute the SQL query or function call.

- Thresholds: To have the value returned by your SQL query or function call compared with set threshold values, enter the requisite threshold information. The value of the monitored metric returned by your query or function call will be compared against the thresholds you specify. If the comparison holds true, a warning or critical alert may be generated.

- Schedule: Specify the start time and frequency at which the user-defined SQL query or function call should be executed. The time zone used is that of the Agent running on the monitored machine.

The following figures show the Create User-Defined Metric pages for a SQL-based user-defined metric. When accessing this page from any Database home page, the Create User-Defined Metric page appears as shown in Figure 5-2.

Figure 5-2 Create User-Defined Metric Page (SQL-Based)

Description of "Figure 5-2 Create User-Defined Metric Page (SQL-Based)"

Key elements of this page are described in the following tables.

Table 5-6 Create User-Defined Metric Page: Definition

User-Interface Element

Description

Metric Name

Metric name identifying the user-defined metric in the Enterprise Manager user interface.

Metric Type

Type of the value returned by the user-defined script. Choose "NUMBER" if your script returns a number. Choose "STRING" if your script returns an alphanumeric text string.

SQL Query Output

Specify whether the SQL script is to return a single value (one column) or a multiple rows (two columns).

- Single Value: Query is one of the following types.
A SELECT statement returning a single value. Example: SELECT sal FROM emp WHERE empno=7369

A function call returning a single value. Example: myfunc(123, 'abc')

- Two Columns: Query is a SELECT statement that returns two columns and possibly multiple rows. Example: SELECT ename, sal FROM emp. Each entry in the first column (the key column) must be a unique string. The second column (the value column) must be of the selected Metric Type.

SQL Query

Enter a SQL query or function call that returns values of the appropriate type (STRING or NUMBER). The SQL statement must return one or two column. If your SQL statement only returns one column, only one row can be returned. If you want multiple rows returned, your SQL statement must return two columns.

Table 5-7 Create User-Defined Metric Page: Database Credentials

User-Interface Element

Description

User Name

Enter the user name for a valid database account on the database where the SQL query is to be run. Make sure that the specified account has the requisite privileges to run the SQL query.

Password

Enter the password associated with the User Name.

Table 5-8 Create User-Defined Metric Page: Threshold

User-Interface Element

Description

Comparison Operator

Select the comparison method Enterprise Manager should use to compare the value returned by the SQL query or function call to the threshold values. When the query returns two columns, the second column (value column) will be used for comparison against threshold values.

Available Comparison Operators

Operator Value Metric Type Description

= Number equal to

> Number greater than

< Number less than

>= Number greater than or equal to

<= Number less than or equal to

!= Number not equal to

CONTAINS String contains at least

MATCH String exact match

Warning

The value returned by the SQL query or function call is compared to the Warning threshold value using the specified comparison operator. If this comparison holds true, an alert triggers at the warning severity level.

Specifically, an alert triggers at the warning severity level if the following comparison is true:

<query_value> <comparison_operator> <warning_threshold>

and if the consecutive occurrences preceding notification has been reached.

Critical

The value returned by the SQL query or function call is compared to the Critical threshold value using the specified comparison operator. If this comparison holds true, an alert triggers at the critical severity level.

Specifically, an alert triggers at the critical severity level if the following comparison is true:

<query_value> <comparison_operator> <critical_threshold>

and if the consecutive occurrences preceding notification has been reached.

Warning Thresholds by Key

and

Critical Thresholds by Key

For queries returning two columns (the first column is the key and the second column is the value), you can specify thresholds on a per key basis. The following example uses the following query:

SELECT ename FROM emp

Threshold settings for this example are shown.

Use the format key:value . Keys are case-sensitive.

- Warning:500

- Critical:300

- Comparison Operator: <

- Warning threshold by key: SMITH:250;JONES:400;CLARK:900
The warning threshold is set to 250 for SMITH, 400 for JONES, and 900 for CLARK.

- Critical threshold by key: SMITH:100;JONES:200;CLARK:500
The critical threshold is set to 100 for SMITH, 200 for JONES, and 500 for CLARK.

All other keys will use the threshold values specified in the Warning and Critical fields.

Consecutive Occurrences Preceding Notification

Consecutive number of times a returned value reaches either the warning or critical thresholds before an alert triggers at a warning or critical severity. This feature is useful when monitoring for sustained conditions. For example, if your script monitors the CPU load for a particular machine, you do not want to be notified at every CPU load spike. Instead, you are only concerned if the CPU load remains at a particular threshold (warning or critical) level for a consecutive number of monitoring cycles.

Response Action

Optional. Specify a script or command that will be executed if the user-defined metric generates a warning or critical alert. The script or command will be executed using the credentials of the Agent owner. Important: Only an Enterprise Manager Super Administrator can create/edit response actions for metrics.

For example, the Management Agent executes the response action if:

The Alert severity is Warning or Critical

AND

There is a change in severity (for example, warning -> critical, critical --> warning, clear --> warning or critical)

For more information, see Enterprise Manager online help.

Alert Message

Enter a custom message (up to 400 characters) to be used when an alert is sent. The default message uses %Key% and %value% variables to display the metric key and its returned value. The %Key% and %value% variables are case-sensitive.

For example, a payroll system alert for underpayment of salary might be defined as:

Underpaid Employee: %Key% has salary of %value%

If the SQL query returns 2 columns, you can use the %Key% variable to represent the key value and the %value% variable to represent the return value.

If the SQL query returns 1 column, only the %value% variable is applicable in the alert message.

The User-Defined Metric Schedule interface lets you specify the frequency at which the SQL query or function should be run.

SQL-Based User-Defined Metric Examples

For a database version 9i and higher, you can run the example queries as dbsnmp, which is the default monitoring user account for the Management Agent. On a 8.1.7 database (which does not have SELECT ANY DICTIONARY system privilege), you must grant dbsnmp the following privileges in order for the queries to run successfully:

For example #1:

grant select on sys.dba_tablespaces to dbsnmp;

grant select on sys.dba_data_files to dbsnmp;

grant select on sys.dba_free_space to dbsnmp;

For example #2:

grant select on sys.dba_extents to dbsnmp;

The above grant statements can be run as SYSDBA after logging in via "connect internal". The queries can also be run by any user who has been granted the DBA role.

Example 1: Query Returning Tablespace Name and Percent Used

This sample user-defined metric monitors the percentage of space used for dictionary managed permanent tablespaces. A DBA can use this as a reference on when to add datafiles for the tablespace.Oracle recommends setting a polling frequency of 30 minutes, warning threshold at 75, and critical threshold at 85.

Example 1 SQL

SELECT d.tablespace_name,

round(((a.bytes - NVL(f.bytes,0))*100/a.maxbytes),2) used_pct

FROM   sys.dba_tablespaces d,

(select tablespace_name, sum(bytes) bytes, sum(greatest(maxbytes,bytes)) maxbytes

from sys.dba_data_files group by tablespace_name) a,

(select tablespace_name, sum(bytes) bytes

from sys.dba_free_space group by tablespace_name) f

WHERE d.tablespace_name = a.tablespace_name(+)

AND d.tablespace_name = f.tablespace_name(+)

AND NOT (d.extent_management = 'LOCAL' AND d.contents = 'TEMPORARY')

Example 2: Query Returning Segment Name/Type and Extent Count

This sample user-defined metric checks for non-system table and index segments that are reaching a high number of extents. A high number of extents could indicate a segment with fragmentation and/or performance problems. A DBA can use this as a reference on when to call Segment Shrink or the Reorganization Wizard in Enterprise Manager.Oracle recommends setting a polling frequency of 24 hours, warning threshold at 1000, and critical threshold at 2000.

Example 2 SQL

SELECT decode(nvl(partition_name, ' '),

' ',   owner || '.' || segment_name || ' ' || segment_type,

owner || '.' || segment_name || '.' || partition_name || ' ' || segment_type) as segment,

count(extent_id) as extent_count

FROM dba_extents

WHERE (segment_type like 'TABLE%' OR segment_type like 'INDEX%') AND

(owner != 'SYSTEM' AND owner != 'SYS')

GROUP BY owner, segment_name, partition_name, segment_type

ORDER BY EXTENT_COUNT DESC

Example 3: Embed a Long SQL statement in a PL/SQL Routine

In situations where the SQL statement forming the SQL user-defined metric exceeds 1024 characters, you must embed the SQL statement in a PL/SQL routine. This must be carried out in three step. In this example, a long SQL statement is used to track tablespaces & free space in them and raise alerts if the free space falls below a user specified threshold. A 2-column SQLUDM is crated using this query.

Example 5-1 of a long (more than1024 characters) SQL statement that returns two values: tablespace_name (key) and free_mb(value)

Example 5-1 Long SQL Statement

select Tablespace, case when (MxAvail <= 15) and (MxFreeMB < 20000) then 'CRITICAL, '||MxUsed||'%' when (MxAvail <= 20) and (MxFreeMB < 20000) then 'WARNING, '||MxUsed||'%' else 'OK' end Error_Level, MxAvail,MxUsed,MxFreeMB,MxExdMB from (select nvl(b.tablespace_name, nvl(a.tablespace_name,'UNKNOWN')) as Tablespace, data_files as NumDBFs, mbytes_alloc as AllocMB, Round( mbytes_alloc-nvl(mbytes_free,0),0) as UsedMB, Round(nvl(mbytes_free,0),0) "AllocFreeMB", Round((( mbytes_alloc-nvl(mbytes_free,0))/ mbytes_alloc)*100,0) as AllocUsed, MaxSize_Mbytes as MxExdMB, Round(MaxSize_Mbytes - (mbytes_alloc-nvl(mbytes_free,0)),0) as MxFreeMB, Round((mbytes_alloc/MaxSize_Mbytes*100),0) as MxUsed, Round((MaxSize_Mbytes - (mbytes_alloc-nvl(mbytes_free,0)))/MaxSize_Mbytes *100,0) as MxAvail from ( select sum(bytes)/1024/1024 mbytes_free, max(bytes)/1024/1024 largest,tablespace_name from sys.dba_free_space group by tablespace_name ) a, ( select sum(bytes)/1024/1024 mbytes_alloc, sum(decode(maxbytes,0,bytes,maxbytes))/1024/1024 MaxSize_Mbytes, tablespace_name,count(file_id) data_files from sys.dba_data_files group by tablespace_name )b where a.tablespace_name (+) = b.tablespace_name order by 1)

Because a 2-column SQL UDM is being created (tablespace, free_space_in_MB), an array must created for the data being returned from this query, as shown in

Example 5-2 Creating an Array of Returned Values

CREATE OR REPLACE TYPE tablespace_obj as OBJECT

(

tablespace_name VARCHAR2(256),

free_mb NUMBER);

/

CREATE OR REPLACE TYPE tablespace_array AS TABLE OF tablespace_obj;

/

The next step is to embed the long SQL statement shown in Example 5-1 in a PL/SQL routine as shown in Example 5-3

Example 5-3 Embedded SQL in a PL/SQL Routine

CREATE OR REPLACE FUNCTION calc_tablespace_free_mb

RETURN tablespace_array

IS

tablespace_data TABLESPACE_ARRAY := TABLESPACE_ARRAY();

BEGIN

SELECT tablespace_obj(tablespace, mxfreemb)

BULK COLLECT INTO tablespace_data

FROM

(

select Tablespace, case when (MxAvail <= 15) and (MxFreeMB < 20000) then

'CRITICAL, '||MxUsed||'%' when (MxAvail <= 20) and (MxFreeMB < 20000) then

'WARNING, '||MxUsed||'%' else 'OK' end Error_Level,

MxAvail,MxUsed,MxFreeMB,MxExdMB from (select nvl(b.tablespace_name,

nvl(a.tablespace_name,'UNKNOWN')) as Tablespace, data_files as NumDBFs, mbytes

_alloc as AllocMB, Round( mbytes_alloc-nvl(mbytes_free,0),0) as UsedMB,

Round(nvl(mbytes_free,0),0) "AllocFreeMB", Round((( mbytes_alloc-nvl(mbytes

_free,0))/ mbytes_alloc)*100,0) as AllocUsed, MaxSize_Mbytes as MxExdMB,

Round(MaxSize_Mbytes - (mbytes_alloc-nvl(mbytes_free,0)),0) as MxFreeMB,

Round((mbytes_alloc/MaxSize_Mbytes*100),0) as MxUsed, Round((MaxSize_Mbytes -

(mbytes_alloc-nvl(mbytes_free,0)))/MaxSize_Mbytes *100,0) as MxAvail from (

select sum(bytes)/1024/1024 mbytes_free, max(bytes)/1024/1024 largest,tablespace

_name from sys.dba_free_space group by tablespace_name ) a, ( select

sum(bytes)/1024/1024 mbytes_alloc,

sum(decode(maxbytes,0,bytes,maxbytes))/1024/1024 MaxSize_Mbytes, tablespace

_name,count(file_id) data_files from sys.dba_data_files group by tablespace_name

)b where a.tablespace_name (+) = b.tablespace_name order by 1)

) CUSTOMER_QUERY;

RETURN tablespace_data;

END calc_tablespace_free_mb;

/

The final step in the process is to create the 2-column UDM with the following:

- Metric Type: NUMBER

- SQL Query Output: Two Columns

- SQL Query:

SELECT tablespace_name, free_mb

FROM TABLE(CAST(calc_tablespace_free_mb as TABLESPACE_ARRAY))

Notifications, Corrective Actions, and Monitoring Templates

User-Defined Metrics, because they are treated like other metrics, can take advantage of Enterprise Manager's notification system, corrective actions and monitoring templates.

Note:

Corrective actions and monitoring templates support both OS user-defined metrics and SQL-based user-defined metrics that return single scalar values.

Getting Notifications for User-Defined Metrics

As with regular metrics, you can receive e-mail notifications when user-defined metric critical or warning alert severities are reached. Assuming you have already defined your e-mail addresses and notification schedule, the remaining task is to set up a notification rule for the user-defined metric.

To set up notification rules:

1. Click Preferences.

2. From the vertical navigation bar, click Rules if you are a Super Administrator or My Rules if you are a regular Enterprise Manager administrator.

3. Click Create to define a new notification rule. The Create Notification Rule pages appear.

4. From the General page, enter the required rule definition information and choose Target Type Host for OS-based user-defined metrics or choose Database Instance for SQL-based user-defined metrics.

5. On the Metrics page, click Add. A list of available metrics appears. To view all metrics on simultaneously, choose Show All from the drop-down menu.

6. Select User-Defined Numeric Metric or User-Defined String Metric based on the type of value returned by your user-defined metric.

7. In the Objects column, choose whether you want to receive notification for all user-defined metrics (All Objects) or specific user-defined metrics (Select).
When choosing the Select option, enter the name of the user-defined metric, or specify multiple user-defined metrics separated by commas. You can use the wildcard character (%) to match patterns for specific user-defined metrics.

You can search for available user-defined metrics using the search function (flashlight icon). However, search results will only show user-defined metrics that have at least one collected data point. For metrics that have not yet collected at least one data point, as may be the case for a newly created user-defined metric, you must specify them in the Select text entry field.

8. Select the severity or corrective action state for which you would like to receive the notification and then click Continue.

9. If you want to receive e-mail for the specified user-defined metric, go to the Notification Rule and check the "Send me E-mail" option.

10. Click OK to create the new notification rule. If you made the notification rule public, other administrators can subscribe to the same rule.

Setting Corrective Actions for User-Defined Metrics

Corrective actions allow you to specify automated responses to alerts ensuring that routine responses to alerts are automatically executed. Corrective actions can be defined for both SQL and OS-based user-defined metrics.

To set up corrective actions:

1. From a target home page, click Metric and Policy Settings from Related Links.

2. Locate and edit the user-defined metric.

3. From the Edit Advanced Settings page, click Add under Corrective Actions for the Critical or Warning alert severity and define the corrective action. Corrective actions can be defined for one or both alert severities.

Deploying User-Defined Metrics Across Many Targets Using Monitoring Templates

Monitoring Templates simplify the task of standardizing monitoring settings across your enterprise by allowing you to specify the monitoring settings once and applying them to your monitored targets. You can thus use Monitoring Templates as a way to propagate user-defined metrics across a large number of targets.

Assuming you have created the user-defined metric on the host or database target, you can use Monitoring Templates to propagate the user-defined metric to other hosts or database targets.

To create a Monitoring Template for the user-defined metric:

1. Click Setup.

2. From the vertical navigation bar, click Monitoring Templates

3. Click Create. The Copy Target Settings page appears.

4. Specify the host or database on which you defined the user-defined metric and click Continue.

5. Fill in the requisite information on the General page.

6. On the Metric Thresholds page, you can choose to keep or remove the other metrics that have been copied over from the target.

7. You can also edit the user-defined metric's thresholds, collection schedule, and corrective actions.

8. On the Policies page, you can choose to keep or remove any policy rules that have been copied over from the target.

9. Click OK to save the template settings to the Management Repository.

Once the template containing the user-defined metric has been created, you can propagate the user-defined metric by applying the template to other hosts or databases.

Important:

For OS-based user-defined metrics, you will first need to separately deploy the OS Script used by the user-defined metric to all destination hosts. The OS Script should reside in the same location across all host targets.
For SQL-based user-defined metrics, if the SQL query specified is a function call, then you need to create this function across all databases on which the SQL-based user-defined metric will be created.

To apply the monitoring template:

1. On the Monitoring Templates page, select the monitoring template and click Apply.

2. On the Apply Monitoring Template page, add the targets on which the user-defined metric should be created.

3. If a two-column SQL-based user-defined metric is part of the template, the Metric with Multiple Thresholds option is applied according one of the following situations:

- Situation One: The target to which the template will be applied does not contain the two-column SQL user-defined metric defined in the template. In this situation, regardless of which Metric with Multiple Thresholds option is chosen, the user-defined metric is copied to the target when you apply the template.

- Situation Two: The target to which the template will be applied does contain the two-column SQL user-defined metric. The name (case insensitive) and return value (numeric, scalar, or two column) of both the target and template user-defined metrics must match. In this situation, you must select one of the Metric with Multiple Thresholds options:

- Apply threshold settings for monitored objects common to both template and target: For only those keys which the target has in common with the template, the target threshold values will be set to the values defined in the template. This option is chosen by default and is recommended for most situations.

- Duplicate threshold settings on target: For keys which are common between target and template, the thresholds will be set to the values defined in the template. Any extra keys (and their thresholds) that exist in the template but not on the target will be copied to the target in anticipation that these keys will be created in the target at some point in the future. Any extra keys (and their thresholds) that exist on the target but not in the template will be deleted from the target.

Important: If there are other metrics in the monitoring template, refer first to the Enterprise Manager online help for implications of what this option means for these other metrics.

4. Click Continue.

5. On the subsequent page, specify the credentials that should be used when running the user-defined metric on the destination targets.

6. Click Finish.

7. When you return back to the Monitoring Templates page, check that "Pending Apply Operations" count for your template is zero. This indicates the number of template apply operations that could be pending. Once they are all complete, the count should be zero.

Deploying User-Defined Metrics Using Scripts

An alternate method of deploying user-defined metrics to large numbers of targets is to use the Enterprise Manager Command Line Interface (EMCLI). Using the EMCLI "apply_template" verb, you can deploy user-defined metrics via custom scripts. For more information about the "apply_template" verb, see the Oracle Enterprise Manager Command Line Interface manual.

Deleting User-Defined Metrics Across Many Targets Using Monitoring Templates

Just as templates can be used to deploy user-defined metrics across targets, templates can also be used to delete these metrics across targets should these metrics no longer be in use.

To create a Monitoring Template for the user-defined metric:

1. Click Setup.

2. From the vertical navigation bar, click Monitoring Templates

3. Click Create. The Copy Target Settings page appears.

4. Specify the host or database on which there is a user-defined metric that needs to be deleted and click Continue.

5. Fill in the requisite information on the General page.

6. On the Metric Thresholds page, remove all metrics from the template except the user-defined metric to be deleted.

7. Click the pencil icon to access the Edit Advanced Settings page.

8. Check the Mark for Delete option and click Continue. The Mark for Deletion icon now appears next to the user-defined metric on the Metric Thresholds page.

9. On the Policies page, remove all policies.

10. Click OK to save the template settings to the Management Repository.

To apply the monitoring template:

1. On the Monitoring Templates page, select the monitoring template and click Apply.

2. On the Apply Monitoring Template page, add the targets on which the user-defined metric should be deleted.

3. Click Continue.

4. On the subsequent page, specify the credentials that should be used when running the user-defined metric on the destination targets.

5. Click Finish.

6. On the Monitoring Templates page, check that "Pending Apply Operations" count for your template is zero. This indicates the number of template apply operations that could be pending. Once they are all complete, the count should be zero.

Enterprise manager will delete all user-defined metrics found on the selected target that match the following criteria:

- Name of the user-defined metrics (case insensitive)

- Return value of the user-defined metric (numeric or scalar)

- For SQL-based user-defined metrics, the output of the query (single value or two columns). The match does not take into consideration the actual script used by the user-defined metric. For this reason, even though the script on the target user-defined metric may be different from that of the template user-defined metric, the target user-defined metric will still be deleted.

- Host User-Define Metrics using scripts: You must delete the script from the host on which you want the UDM deleted.

- SQL-based user-defined metrics using function calls: You must delete the function from the database on which you want the user-defined metric deleted.

Changing User-Defined Metric Credentials

As discussed earlier, user-defined metrics require valid credentials (username and password) in order to execute monitoring scripts/SQL queries. For this reason, both the monitored target's password and the password defined in the user-defined metric must match. This can be problematic if target passwords are changed frequently. For environments with a large number of targets, you can use the Enterprise Manager Command Line Interface (EMCLI) to change the target password and user-defined metric password simultaneously using scripts. Use the 'update_password' verb to change the target password. This password change is then propagated to all features of Enterprise Manager that use the specified username, which includes preferred credentials, corrective actions, jobs, and both host and SQL-based user-defined metrics.

The following example changes the password associated with the OS user sysUser from sysUserOldPassword to sysUserNewPassword.

Example 5-4 Host Password Change

update_password   -target_type=host  -target_name=MyHost  -credential_type=HostCreds  -key_column=HostUserName:sysUser-non_key_column=HostPassword:sysUserOldPassword:sysUserNewPassword

The next example changes the password associated with the database user sys from sysPassword to sysNewPassword.

Example 5-5 Database Password Change

update_password -target_type=oracle_database -target_name=ORCL -credential_type=DBCreds -key_column=DBUserName:sys-non_key_column=DBPassword:sysPassword:sysNewPassword:DBAROLE

For more information about EMCLI, see the Oracle Enterprise Manager Command Line Interface guide.

6 Group Management

This chapter introduces the concept of group management and contains the following sections:

- Introduction to Groups

- Managing Groups

- Out-of-Box Reports

- Redundancy Groups

Introduction to Groups

Today's IT operations can be responsible for managing a great number of components, such as databases, application servers, hosts, or other components, which can be time consuming and impossible to manage individually. The Enterprise Manager Grid Control group management system lets you combine components (called targets in Enterprise Manager) into logical sets, called groups. This enables you to organize, manage, and effectively monitor the potentially large number of targets in your enterprise.

Enterprise Manager Groups can include:

- Targets of the same type, such as:

- All hosts in your data center

- All of your production databases

- Targets of different types, such as:

- The database, application server, listener, and host that are used in your application environment

- Targets operating within a particular data center region

Note:

An Enterprise Manager "System," used specifically to group the components on which a service runs, is a special kind of Enterprise Manager group. Many of the functions and capabilities for groups and systems are similar.

Managing Groups

By combining targets in a group, Enterprise Manager offers a wealth of management features that enable you to efficiently manage these targets as one group. Using the Group pages, you can:

- View a summary status of the targets within the group.

- Monitor outstanding alerts and policy violations for the group collectively, rather than individually.

- Monitor the overall performance of the group through performance charts.

- Perform administrative tasks, such as scheduling jobs for the entire group, or blacking out the group for maintenance periods.

You can also customize the console to provide direct access to group management pages.

Group Home Page

The Group Home page, shown in Figure 6–1, is the central location for monitoring information. The Group Home page provides the following features:

- Availability pie chart that provides at-a-glance information on the current status across all members so you can easily assess the percentage of members that are up, and the percentage of members that are unavailable. You can quickly drill down for information if any member target is down.

- Roll-up of open alerts and policy violations, categorized by severity, so you can quickly focus on the most critical problems first. Alerts and violations that have occurred within the last 24 hours highlight problems that recently occurred.

- Access to the Policy Trend Overview page, which provides a comprehensive view of the group for compliance with policy rules over a period of time. Using policy charts, you can assess trends such as increased or decreased number of policy violations, changes in the overall group compliance score, and the percentage of members in compliance with your enterprise's policy rules.

- Access to the Security at a Glance page, which provides an overview of the security health of the group. This shows statistics about security policy violations and critical security patches that have not been applied.

- Summary of recent configuration changes across all members in the group, so you can easily determine if a new problem is related to any recent changes.

- Summary of Critical Patch Advisories for Oracle homes within the group.

Figure 6-1 Group Home Page

Description of "Figure 6-1 Group Home Page"

Figure 6–2 shows the Policy Trend Overview page that you can access from the Group Home page.

Figure 6-2 Policy Trend Overview Page

Description of "Figure 6-2 Policy Trend Overview Page"

See Also:

"Group Home Page" in the Enterprise Manager online help

Group Charts Page

The Group Charts page, shown in Figure 6–3, enables you to monitor the collective performance of the group. Out-of-box performance charts are provided based on the type of members in the group. For example, when databases are part of the group, a Wait Time (%) chart is provided that shows the top databases with the highest wait time percentage values. You can view this performance information over the last 24 hours, last 7 days, or last 31 days. You can also add your own custom charts to the page.

Figure 6-3 Group Charts Page

Description of "Figure 6-3 Group Charts Page"

See Also:

"Group Charts Page" in the Enterprise Manager online help

Group Administration Page

The Group Administration page, shown in Figure 6–4, provides a central point for performing common administrative tasks for the group. For example, you can:

- Run jobs or find out the status of currently running jobs against the group.

- Define planned outage windows, called blackouts, on the members of the group to perform maintenance tasks.

- Run SQL commands collectively against the database member targets of the group.

- View the most recent backup for each database in the group.

- View the last 100,000 bytes of the alert log for all databases in the group.

- Use a deployment summary to easily obtain hardware and software inventory information across all member targets.

Figure 6-4 Group Administration Page

Description of "Figure 6-4 Group Administration Page"

See Also:

"Group Administration Page" in the Enterprise Manager online help

Group Members Page

The Group Members page, shown in Figure 6–5, summarizes information about the member targets in the group. It includes information on their current availability status, roll-up of open alerts and policy violations, and key performance metrics based on the type of targets in the group.

You can visually assess availability and relative performance across all member targets. You can sort on any of the columns to rank members by a certain criterion (for example, database targets in order of decreasing wait time percentage). Default key performance metrics are displayed based on the targets you select, but you can customize these to include additional metrics that are important for managing your group.

Figure 6-5 Group Members Page

Description of "Figure 6-5 Group Members Page"

See Also:

"Group Members Page" in the Enterprise Manager online help

System Dashboard

The System Dashboard, shown in Figure 6–6, enables you to proactively monitor the status and alerts in the group as they occur. The color-coded interface is designed to highlight problem areas using the universal colors of alarm—targets that are down are highlighted in red, metrics in critical severity are shown as red dots, metrics in warning severity are shown as yellow dots, and metrics operating within normal boundary conditions are shown as green dots.

Using these colors, you can easily spot the problem areas for any target and drill down for details as needed. An alert table is also included to provide a summary for all open alerts in the group. The alerts in the table are presented in reverse chronological order to show the most recent alerts first, but you can also click on any column in the table to change the sort order.You can customize the dashboard according to your needs. You can specify which key metrics should be included in the dashboard and the display names to be used for these metrics. You can also customize the refresh interval to ensure that you always receive timely information about alerts as they are detected.You can launch the System Dashboard in context from any Group Home page. However, using reporting framework features, you can also make the System Dashboard publicly available for any user that has access to a web browser and the Enterprise Manager Reports Web site.

Figure 6-6 System Dashboard

Description of "Figure 6-6 System Dashboard"

Out-of-Box Reports

Enterprise Manager provides several out-of-box reports for groups as part of the reporting framework, called Information Publisher. These reports display important administrative information, such as hardware and operating system summaries across all hosts within a group, and monitoring information, such as outstanding alerts and policy violations for a group.

You can access these reports from the Reports link in the Related Links section of all Group pages. Figure 6–7 shows the Availability History report for a specified group over the last 31 days.

Figure 6-7 Availability History Report

Description of "Figure 6-7 Availability History Report"

See Also:

Chapter 10, "Information Publisher"

Redundancy Groups

A redundancy group is a group that contains members of the same type that function collectively as a unit. A type of redundancy group functions like a single logical target that supports a status (availability) metric. A redundancy group is considered up (available) if at least one of the member targets is up.

You can create and administer a redundancy group from the All Targets page. Redundancy groups support all group management features previously discussed.

Do not use redundancy groups if the group you want to model is an Oracle Real Application Clusters database, host cluster, HTTP server high availability group, or OC4J high availability group. Instead, you can use the following specialized target types for this purpose:

- Cluster

- Cluster Database

- HTTP HA Group

- OC4J HA Group

7 Job System

Because today's IT environments are composed of many sets of components, you need to minimize the time needed to support those IT components and eliminate the human error associated with component maintenance. The Enterprise Manager Grid Control Job System provides the capacity to automate routine administrative tasks and synchronize components in your environment so you can manage them more efficiently.

This chapter describes the Job System in the following sections:

- What Is a Job?

- Using and Defining Jobs

- Analyzing Job Activity

- Jobs and Groups

- Sharing Job Responsibilities

- Job Library

- Job Notifications

- Multitask Jobs

What Is a Job?

The Enterprise Manager Job System serves a dual purpose:

- Provides for the automation of many administrative tasks, for example, backup, cloning, and patching

- Allows users to create their own jobs using their own custom OS and SQL scripts

A job is a unit of work that you define to automate commonly-run tasks. One of the advantages of jobs is that you can schedule a job to start immediately or start at a later date and time. You also have the option to have the job run once or at a specific interval, for example, three times every month. Job results are displayed on the target's home page.

The Job Activity page (Figure 7-1) is the hub of the Job System. From this page you can:

- Search for existing job runs and job executions
You can restrict the search by name, owner, status, scheduled start, job type, target type, and target name.

- Create a job

- View, edit, create like, suspend, resume, stop, and delete a run

- View, edit, create like, suspend, resume, retry, stop, and delete an execution

Figure 7-1 Job Activity Page

Description of "Figure 7-1 Job Activity Page"

Job functionality is not restricted to only the Jobs tab. You can access job functionality while you are working on deployments and databases. On the Deployments page you can create a job to clone a database and another job to clone an Oracle home. While you are working with your databases, you can clone a database by clicking Deployments in the Related Links section.

What Are Job Executions and Job Runs?

Job executions are usually associated with one target, for example, a backup job on a particular database. When a job is run against multiple targets, each execution may execute on one target.

Job executions are not always a one-to-one mapping to a target. Some executions have multiple targets, for example, comparing hosts. Other executions have no targets, for example, the RefreshFromMetaLink job.

When you submit a job to many targets, it would be tedious to examine the status of each execution of the job against each target. For example, say that you run a backup job against 100 databases. Typical questions would be: Were all the backups successful? If not, which backups failed? If this backup job runs every week, you would want to know each week which backups were successful and those that failed.

With the Job System, you can easily get these answers viewing the "job run." A job run is the sum of all job executions of a job that ran on a particular scheduled date. Using the backup example, if you have a backup job against 100 databases on November 5th, then you will have a November 5 job run. The job table that shows the job run will provide a roll-up of the status of those executions.

Differences Between Job Executions and Job Runs

In addition to supporting the standard job operations of create, edit, create like, and delete, the Job System allows you to suspend and resume jobs, as well as retry failed executions. For example, you may need to suspend a job if a needed resource was unavailable, or the job needs to be postponed. Once you suspend a job, any scheduled executions do not occur until you decide to resume the job.

When analyzing a failed execution, it is useful to be able to retry a failed execution after the cause of the problem has been determined. This alleviates the need to create a new job for that failed execution. When you use the Retry operation in the Job System, Enterprise Manager provides links from the failed execution to the retried execution and vice versa, should it become useful to retroactively examine the causes of the failed executions.

See Also:

For more information on job executions and runs, refer to Enterprise Manager Grid Control online help.

Using and Defining Jobs

Enterprise Manager provides predefined job tasks for database targets and deployments. A job task is used to contain predefined, unchangeable logic—for example, patch an application, back up a database, and so on.

The predefined database jobs include backup, export, and import. The predefined jobs associated with deployments include patching, cloning Oracle homes, and cloning databases.

In addition to predefined job tasks, you can define your own job tasks by writing code to be included in OS and SQL scripts. The advantages of using these scripts include:

- When defining these jobs, you can use target properties.

- You can submit the jobs against many targets or a group. The job automatically keeps up with the group membership.

- For host command jobs, you can submit to a cluster.

- For SQL jobs you can submit to a Real Application Cluster.

Using the Job System, you can create jobs using the following job types:

- CloneHome: Copies the known state of an Oracle home. For example, after you have an Oracle home in a known state (you have chosen particular install options for it, applied required patches to it, and tested it), you may want to clone this Oracle home to one or more hosts.

- Host Command: Use to execute a user-defined OS script.

- Patch: Finds and applies a patch or patch set.

- Refresh from MetaLink: Use to be notified of critical patch advisories.

- SQL script: Use to execute a user-defined SQL script.

Analyzing Job Activity

After you submit jobs, the status of all job executions across all targets is automatically rolled up and available for review on the Grid Control Console Home page. Figure 7-2 shows the All Targets Jobs information on the Grid Control Console Home page.

Figure 7-2 Summary of Target Jobs on the Grid Control Console Home Page

Description of "Figure 7-2 Summary of Target Jobs on the Grid Control Console Home Page"

This information is particularly important when you are examining jobs that execute against hundreds or thousands of systems. You can determine the job executions that have failed. By clicking the number associated with a particular execution, you can drill down to study the details of the failed job.

Jobs and Groups

In addition to submitting jobs to individual targets, you can submit jobs against a group of targets. Any job that you submit to a group is automatically extended to all its member targets and takes into account the membership of the group as it changes.

For example, if a Human Resources job is submitted to the Payroll group, then if a new host is added to this group, the host automatically becomes part of the Human Resources job. In addition, if the Payroll group is composed of diverse targets, for example, databases, hosts, and application servers, then the job only runs against applicable targets in the group.

By accessing the Group Home page, you can analyze the job activity for that group.

See Also:

Chapter 6, "Group Management"

Sharing Job Responsibilities

To allow you to share job responsibilities, the Job System provides job privileges. These job privileges allow you to share the job with other administrators. Using privileges, you can:

- View access to the administrators who need to see the results of the job

- Give full access to the administrators who may need to edit the job definition

These privileges can be granted on an as-needed basis.

See Also:

"Roles" in Chapter 2, "Setting Up Enterprise Manager"

Job Library

Once you have defined jobs, you can save these jobs to the Job Library. Use the Library as a repository for frequently used jobs. Analogous to active jobs, you can grant View or Full access to specific administrators.

In addition, you can use the Job Library to store:

- Basic definitions of jobs, then add targets and other custom settings before submitting the job

- Jobs for your own reuse or to share with others. You can share jobs using views or giving full access to the jobs.

Job Notifications

The Grid Control Notification system allows you to define a notification rule to send e-mail to the job owner when a job enters a chosen state (Scheduled, Running, Suspended, Completed, or Problems). New functionality has been added to the Notification system (rule creation) that allows you to easily associate specific jobs with a notification rule.

See Also:

Chapter 4, "Notifications"

Multitask Jobs

Multitask jobs allow you to create complex jobs consisting of one or more distinct tasks. Because multitask jobs can run against targets of the same or different type, they can perform ad hoc operations on one or more targets of the same or different type.

You can create a multitask job consisting of two tasks, each a different job type, each operating on two separate (and different) target types. For example,

- Task 1 (OS Command job type) performs an operation on Host 1.

- If Task 1 is successful, run Task2 (SQL Script job type) against Database 1 and Database 2.

The Job System's multitask functionality makes it easy to create extremely complex operations.

8 Starting and Stopping Enterprise Manager Components

This chapter explains how to use the Enterprise Manager command line utility (emctl) to start and stop the Management Service, the Management Agent, the Grid Control console, the Application Server Control Console, and Database Control.

This chapter also explains the various emctl commands, exit codes, and how to use log information to troubleshoot emctl.

Following are the sections in this chapter:

- Controlling the Oracle Management Agent

- Controlling the Oracle Management Service

- Controlling the Application Server Control

- Controlling the Database Control on UNIX

- Guidelines for Starting Multiple Enterprise Manager Components on a Single Host

- Starting and Stopping Oracle Enterprise Manager 10g Grid Control

- Additional Management Agent Commands

- emctl Commands

- Using emctl.log File

Controlling the Oracle Management Agent

The following sections describe how to use the Enterprise Manager command line utility (emctl) to control the Oracle Management Agent:

- Starting, Stopping, and Checking the Status of the Management Agent on UNIX

- Starting and Stopping the Management Agent on Windows

- Checking the Status of the Management Agent on Windows

Starting, Stopping, and Checking the Status of the Management Agent on UNIX

To start, stop, or check the status of the Management Agent on UNIX systems:

1. Change directory to the AGENT_HOME/bin directory.

2. Use the appropriate command described in Table 8-1.
For example, to stop the Management Agent, enter the following commands:

$PROMPT> cd AGENT_HOME/bin

$PROMPT> ./emctl stop agent

Table 8-1 Starting, Stopping, and Checking the Status of the Management Agent

Command

Purpose

emctl start agent

Starts the Management Agent

emctl stop agent

Stops the Management Agent

emctl status agent

If the Management Agent is running, this command displays status information about the Management Agent, including the Agent Home, the process ID, and the time and date of the last successful upload to the Management Repository ().

Example 8-1 Checking the Status of the Management Agent

$PROMPT> ./emctl status agent

Oracle Enterprise Manager 10g Release 10.2.0.5.0

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

-----------------------------------------------------------------

Agent Version     : 10.2.0.5.0

OMS Version       : 10.2.0.5.0

Protocol Version  : 10.2.0.5.0

Agent Home        : /scratch/OracleHomesX/agent10g

Agent binaries    : /scratch/OracleHomesX/agent10g

Agent Process ID  : 17604

Parent Process ID : 17587

Agent URL         : https://stadj32.us.oracle.com:3872/emd/main/

Repository URL    : https://stadj32.us.oracle.com:1159/em/upload

Started at        : 2009-09-13 01:31:11

Started by user   : test

Last Reload       : 2009-09-13 01:31:11

Last successful upload                       : 2009-09-13 01:39:01

Total Megabytes of XML files uploaded so far :     0.28

Number of XML files pending upload           :        0

Size of XML files pending upload(MB)         :     0.00

Available disk space on upload filesystem    :     8.36%

Last successful heartbeat to OMS             : 2009-09-13 01:38:51

---------------------------------------------------------------

Agent is Running and Ready

$PROMPT>

On IBM AIX environment with a large memory configuration where the Management Agent is monitoring a large number of targets, the Agent may not start. To prevent this issue, prior to starting the Management Agent, set the following variables in the shell:

LDR_CNTRL="MAXDATA=0x80000000"@NOKRTL

AIX_THREADSCOPE=S

The LDR_CNTRL variable sets the data segment size and disables loading of run time libraries in kernel space. The AIX_THREADSCOPE parameter changes AIX Threadscope context from the default Processwide 'P' to Systemwide 'S'. This causes less mutex contention.

Starting and Stopping the Management Agent on Windows

When you install the Oracle Management Agent on a Windows system, the installation procedure creates three new services in the Services control panel.

The procedure for accessing the Services control panel varies, depending upon the version of Microsoft Windows you are using. For example, on Windows 2000, locate the Services Control panel by selecting Settings and then Administrative Tools from the Start menu.

Note:

The emctl utility described in Controlling the Management Service on UNIX is available in the bin subdirectory of the Oracle home where you installed the Management Agent; however, Oracle recommends that you use the Services control panel to start and stop the Management Agent on Windows systems.

Table 8-2 describes the Windows services that you use to control the Management Agent.

Table 8-2 Summary of Services Installed and Configured When You Install the Management Agent on Windows

Component

Service Name Format

Description

Oracle Management Agent

Oracle<agent_home>Agent

For example:

OracleOraHome1Agent

Use this to start and stop the Management Agent.

Oracle SNMP Peer Encapsulator

Oracle<oracle_home>SNMPPeerEncapsulator

For example:

OracleOraHome1PeerEncapsulator

Use this service only if you are using the advanced features of the Simple Network Management Protocol (SNMP).

For more information, see the Oracle SNMP Support Reference Guide.

Oracle Peer SNMP Master Agent

Oracle<oracle_home>SNMPPeerMasterAgent

For example:

OracleOraHome1PeerMasterAgent

Use this service only if you are using the advanced features of the Simple Network Management Protocol (SNMP).

For more information, see the Oracle SNMP Support Reference Guide.

Note:

If you are having trouble starting or stopping the Management Agent on a Windows NT system, try stopping the Management Agent using the following emctl command:

$PROMPT> <AGENT_HOME>/bin/emctl istop agent

After stopping the Management Agent using the emctl istop agent command, start the Management Agent using the Services control panel.

This problem and solution applies only to the Windows NT platform, not to other Windows platforms, such as Windows 2000 or Windows XP systems.

Checking the Status of the Management Agent on Windows

To check the status of the Management Agent on Windows systems:

1. Change directory to the following location in the AGENT_HOME directory:

AGENT_HOME/bin

2. Enter the following emctl command to check status of the Management Agent:

$PROMPT> ./emctl status agent

If the Management Agent is running, this command displays status information about the Management Agent, including the Agent Home, the process ID, and the time and date of the last successful upload to the Management Repository ().

Controlling the Oracle Management Service

The following sections describe how to control the Oracle Management Service:

- Controlling the Management Service on UNIX

- Controlling the Management Service on Windows

Controlling the Management Service on UNIX

There are two methods for starting and stopping the Oracle Management Service on UNIX systems. You can use the Oracle Process Management and Notification (OPMN) utility, or you can use a set of emctl commands.

The following sections describe these two approaches to controlling the Management Service, as well as information about starting and stopping OracleAS Web Cache, which is also required by the Grid Control console:

- Using OPMN to Start and Stop the Management Service

- Using emctl to Start, Stop, and Check the Status of the Oracle Management Service

- Starting and Stopping Oracle Application Server Web Cache

Using OPMN to Start and Stop the Management Service

One method of starting and stopping the Management Service by using the Oracle Process Management and Notification (OPMN) utility. The OPMN utility (opmnctl) is a standard command used to start and stop components of the Oracle Application Server instance.

The Management Service is a J2EE application running in an Oracle Application Server Containers for J2EE (OC4J) instance within the Application Server. As a result, the following command will start all the components of the Oracle Application Server instance, including the OC4J_EM instance and the Management Service application:

$PROMPT> cd opmn/bin

$PROMPT> ./opmnctl startall

Similarly, the following command will stop all the components of the Oracle Application Server instance:

$PROMPT> ./opmnctl stopall

If you want to start only the components necessary to run the Management Service, you can use the Enterprise Manager command-line utility.

Using emctl to Start, Stop, and Check the Status of the Oracle Management Service

To start, stop, or check the status of the Management Service with the Enterprise Manager command-line utility:

1. Change directory to the ORACLE_HOME/bin directory in the Management Service home.

2. Use the appropriate command described in Table 8-3.
For example, to stop the Management Service, enter the following commands:

$PROMPT> cd bin

$PROMPT> ./emctl stop oms

Table 8-3 Starting, Stopping, and Checking the Status of the Management Service

Command

Purpose

emctl start oms

Starts the Oracle Application Server components required to run the Management Service J2EE application. Specifically, this command starts OPMN, the Oracle HTTP Server, and the OC4J_EM instance where the Management Service is deployed.

Note: The emctl start oms command does not start Oracle Application Server Web Cache. For more information, see "Starting and Stopping Oracle Application Server Web Cache".

emctl stop oms

Stops the Management Service.

Note that this command does not stop the other processes that are managed by the Oracle Process Manager and Notification Server (OPMN) utility.To stop the other Oracle Application Server components, such as the Oracle HTTP Server and Oracle Application Server Web Cache, see "Starting and Stopping Oracle Enterprise Manager 10g Grid Control".

emctl status oms

Displays a message indicating whether or not the Management Service is running.

Starting and Stopping Oracle Application Server Web Cache

By default, when you install Oracle Enterprise Manager 10g, the Grid Control console is configured to use Oracle Application Server Web Cache.

See Also:

Oracle Application Server Web Cache Administrator's Guide for more information about Oracle Application Server Web Cache

Oracle Application Server Web Cache not only improves the performance of the Grid Control console, but also makes it possible to measure the end-user performance of the Enterprise Manager Web application.

To view the Grid Control console using Oracle Application Server Web Cache, you access the Grid Control console using the standard port number assigned during the Oracle Enterprise Manager 10g installation procedure. You can find this default port number (usually 7777) in the setupinfo.txt file, which is copied to the following directory during the Enterprise Manager installation procedure:

AS_HOME/Apache/Apache

If Oracle Application Server Web Cache is not running, you will receive an error message, such as the following, if you try to access the Grid Control console using the default port number:

HTTP 500 - Internal server error

To start Oracle Application Server Web Cache:

1. Change directory to the ORACLE_HOME/opmn/bin directory in the Management Service home.

2. Use the appropriate command described in Table 8-4.
For example, to stop Oracle Application Server Web Cache, enter the following commands:

$PROMPT> cd opmn/bin

$PROMPT> ./opmnctl stopproc ias-component=WebCache

Table 8-4 Starting, Stopping, and Checking the Status of Oracle Application Server Web Cache

Command

Purpose

opmnctl startproc ias-component=WebCache

Starts Oracle Application Server Web Cache.

opmnctl stopproc ias-component=WebCache

Stops Oracle Application Server Web Cache.

opmnctl status

Displays a message showing the status of all the application server components managed by OPMN, including Oracle Application Server Web Cache.

Controlling the Management Service on Windows

When you install the Oracle Management Service on a Windows system, the installation procedure creates three new services in the Services control panel.

The procedure for accessing the Services control panel varies, depending upon the version of Microsoft Windows you are using. For example, on Windows 2000, locate the Services control panel by selecting Settings and then Administrative Tools from the Start menu.

Note:

The emctl utility described in Controlling the Management Service on UNIX is available in the bin subdirectory of the Oracle home where you installed the Management Service; however, Oracle recommends that you use the Services control panel to start and stop the Management Service on Windows systems.

Table 8-5 describes the Windows services that you use to control the Oracle Management Service.

Table 8-5 Summary of Services Installed and Configured When You Install the Oracle Management Service on WIndows

Component

Service Name Format

Description

Application Server Control

Oracle<oracle_home>ASControl

For example:

OracleOraHome1ASControl

Use this Service to start and stop the Application Server Control for the Oracle Application Server instance that was installed and configured to deploy the Management Service J2EE application.

Oracle Process Management and Notification (OPMN)

Oracle<oracle_home>ProcessManager

For example:

OracleOraHome1ProcessManager

Use this service to start and stop all the components of the Oracle Application Server instance that were installed and configured to deploy the Management Service J2EE application.

Use this service to start and stop the Management Service and all its related components, including OC4J, Oracle HTTP Server, and OracleAS Web Cache, which by default must be running in order for you to access the Grid Control console.

Controlling the Application Server Control

The Application Server Control is a component of Oracle Enterprise Manager 10g that is installed as part of any Oracle Application Server installation. The following sections describe how to start and stop the Application Server Control:

- Starting and Stopping the Application Server Control on UNIX

- Starting and Stopping the Application Server Control on Windows

See Also:

Oracle Application Server 10g Administrator's Guide for more information about:

- Using emctl to control the Application Server Control Console

- Starting and stopping the Application Server Control Console on Windows

- Displaying disabled components of the Application Server

Starting and Stopping the Application Server Control on UNIX

To control the Application Server Control Console on UNIX systems, you use the emctl command line utility that is available in the IAS_HOME/bin directory after you install Oracle Application Server.

To start the Application Server Control Console, change directory to the IAS_HOME/bin directory and then enter the following command:

$PROMPT> ./emctl start iasconsole

To stop the Application Server Control Console, enter the following command:

$PROMPT> ./emctl stop iasconsole

Starting and Stopping the Application Server Control on Windows

To start or stop the Application Server Control on Windows systems:

1. Open the Services control panel.
For example, on Windows NT, select Start, point to Settings, select Control Panel, and then double-click the Services icon.

On Windows 2000, select Start, point to Administrative Tools, and select Services.

2. Locate the Application Server Control in the list of services.
The name of the service is usually consists of "Oracle", followed by the name of the home directory you specified during the installation, followed by the word "ASControl." For example, if you specified AS10g as the Oracle Home, the Service name would be:

OracleAS10gASControl

3. After you locate the service, you can use the Services control panel to start or stop the Application Server Control service.
By default, the Application Server Control service is configured to start automatically when the system starts.

You can also start the Oracle Application Server Control console (iasconsole) on Windows using NET START Oracleoms10gASControl.

To stop the Oracle Application Server Control console (iasconsole) on Windows. use NET STOP Oracleoms10gASControl.

Controlling the Database Control on UNIX

The Oracle Enterprise Manager 10g Database Control Console is a component of Oracle Enterprise Manager 10g that is installed as part of any Oracle Database 10g installation.

To control the Database Control, you use the emctl command-line utility that is available in the ORACLE_HOME/bin directory after you install Oracle Database 10g.

Starting the Database Control on UNIX

To start the Database Control, as well the Management Agent and the Management Service associated with the Database Control:

1. Set the following environment variables to identify the Oracle home and the system identifier (SID) for the database instance you want to manage:

- ORACLE_HOME

- ORACLE_SID

2. Change directory to the ORACLE_HOME/bin directory.

3. Enter the following command:

$PROMPT> ./emctl start dbconsole

Stopping the Database Control on UNIX

To stop the Database Control, as well the Management Agent and the Management Service associated with the Database Control:

1. Set the following environment variables to identify the Oracle home and the system identifier (SID) for the database instance you want to manage:

- ORACLE_HOME

- ORACLE_SID

2. Change directory to the ORACLE_HOME/bin directory.

3. Enter the following command:

$PROMPT> ./emctl stop dbconsole

Starting and Stopping the Database Control on Windows

To start or stop the Database Control on Windows systems:

1. Open the Services control panel.
For example, on Windows NT, select Start, point to Settings, select Control Panel, and then double-click the Services icon.

On Windows 2000, select Start, point to Administrative Tools, and select Services.

2. Locate the Database Control in the list of services.
The name of the service is usually consists of "Oracle", followed by the name of the home directory you specified during the installation and the database system identifier (SID), followed by the word "DBControl." For example, if you specified DBd10g as the Oracle Home, the Service name would be:

OracleDB10gDBControl

3. After you locate the service, you can use the Services control panel to start or stop the Database Control service.
By default, the Database Control service is configured to start automatically when the system starts.

You can also start the Database Control on Windows using emctl start iasconsole as described in Stopping the Database Control on UNIX.

Guidelines for Starting Multiple Enterprise Manager Components on a Single Host

Oracle Enterprise Manager 10g components are used to manage a variety of Oracle software products. For example, each time you install Oracle Application Server 10g (9.0.4) instance, you also install an Application Server Control. Similarly, each time you install Oracle Database 10g, you install a Database Control. In addition, if you want to centrally manage your system with Database Control, the Management Agent is also installed on each host you monitor.

In most cases, in a production environment, you will want to distribute your database and application server instances among multiple hosts to improve performance and availability of your software resources. However, in rare cases where you must install multiple application servers or databases on the same host, consider the following guidelines.

When you start Application Server Control, the Management Agent, or the Database Control, Enterprise Manager immediately begins gathering important monitoring data about the host and its managed targets. Keep this in mind when you develop a process for starting the components on the host.

Specifically, consider staggering the startup process so that each Enterprise Manager process has a chance to start before the next process begins its startup procedure.

For example, suppose you have installed OracleAS Infrastructure 10g, the J2EE and Web Cache application server installation type, and the Management Agent on the same host. When you start up all the components (for example, after a restart of the system), use a process such as the following:

1. Use the opmnctl startall command to start all the OPMN-managed processes in the OracleAS Infrastructure 10g home directory.

2. Wait 15 seconds.

3. Use the emctl start iasconsole command to start the Application Server Control in the OracleAS Infrastructure 10g home directory.

4. Wait 15 seconds.

5. Use the opmnctl startall command to start all the OPMN-managed processes in the J2EE and Web Cache home directory.

6. Wait 15 seconds.

7. Use the emctl start iasconsole command to start the Application Server Control in the J2EE and Web Cache home directory.

8. Wait 15 seconds.

9. Use the emctl start agent command to start the Management Agent for the host.

Using a staggered startup procedure such as the preceding example will ensure that the processes are not in contention for resources during the CPU-intensive startup phase for each component.

Starting and Stopping Oracle Enterprise Manager 10g Grid Control

As described in the previous sections, you use separate commands to control the Oracle Management Service, Oracle Management Agent, and the Oracle Application Server components on which the Grid Control depends.

The following sections describe how to stop and start all the Grid Control components that are installed by the Oracle Enterprise Manager 10g Grid Control Console installation procedure.

You can use this procedure to start all the framework components after a system reboot or to shutdown all the components before bringing the system down for system maintenance.

Starting Grid Control and All Its Components

The following procedure summarizes the steps required to start all the components of the Grid Control. For example, use this procedure if you have restarted the host computer and all the components of the Grid Control have been installed on that host.

To start all the Grid Control components on a host, use the following procedure:

1. If your Oracle Management Repository resides on the host, change directory to the Oracle Home for the database where you installed the Management Repository and start the database and the Net Listener for the database:

1. Set the ORACLE_HOME environment variable to the Management Repository database home directory.

2. Set the ORACLE_SID environment variable to the Management Repository database SID (default is asdb).

3. Start the Net Listener:

$PROMPT> $ORACLE_HOME/bin/lsnrctl start

4. Start the Management Repository database instance:

ORACLE_HOME/bin/sqlplus /nolog

SQL> connect SYS as SYSDBA

SQL> startup

SQL> quit

See Also:

Oracle Database Administrator's Guide for information about starting and stopping an Oracle Database

2. Start the Oracle Management Service:

$PROMPT> ORACLE_HOME/bin/emctl start oms

See Also:

"Controlling the Oracle Management Service"

3. Start OracleAS Web Cache:

$PROPMT> $ORACLE_HOME/opmn/bin/opmnctl startproc ias-component=WebCache

4. Change directory to the home directory for the Oracle Management Agent and start the Management Agent:

$PROMPT> AGENT_HOME/bin/emctl start agent

See Also:

"Controlling the Oracle Management Agent"

Note:

Be sure to run the emctl start agent command in the Oracle Management Agent home directory and not in the Management Service home directory.

5. Optionally, start the Application Server Control Console, which is used to manage the Oracle Application Server instance that is used to deploy the Management Service:

$PROMPT> $ORACLE_HOME/bin/emctl start iasconsole

See Also:

"Controlling the Application Server Control"

Stopping Grid Control and All Its Components

The following procedure summarizes the steps required to stop all the components of the Grid Control. For example, use this procedure if you have installed all the components of the Grid Control on the same host you want to shut down or restart the host computer.

To stop all the Grid Control components on a host, use the following procedure:

1. Stop the Oracle Management Service:

$PROMPT> $ORACLE_HOME/bin/emctl stop oms

See Also:

"Controlling the Oracle Management Service"

2. If necessary, stop the Application Server Control Console, which is used to manage the Oracle Application Server instance used to deploy the Management Service:

$PROMPT> $ORACLE_HOME/bin/emctl stop iasconsole

See Also:

"Controlling the Application Server Control"

3. Stop all the Oracle Application Server components, such as the Oracle HTTP Server the OracleAS Web Cache:

$PROMPT> $ORACLE_HOME/opmn/bin/opmnctl stopall

See Also:

Oracle Application Server 10g Administrator's Guide

4. Change directory to the home directory for the Oracle Management Agent and stop the Management Agent:

$PROMPT> AGENT_HOME/bin/emctl stop agent

See Also:

"Controlling the Oracle Management Agent"

Note:

Be sure to run the emctl stop agent command in the Oracle Management Agent home directory and not in the Oracle Application Server home directory.

5. If your Oracle Management Repository resides on the same host, change directory to the Oracle Home for the database where you installed the Management Repository and stop the database and the Net Listener for the database:

1. Set the ORACLE_HOME environment variable to the Management Repository database home directory.

2. Set the ORACLE_SID environment variable to the Management Repository database SID (default is asdb).

3. Stop the database instance:

$PROMPT> ORACLE_HOME/bin/sqlplus /nolog

SQL> connect SYS as SYSDBA

SQL> shutdown

SQL> quit

See Also:

Oracle Database Administrator's Guide for information about starting and stopping an Oracle Database

4. Stop the Net Listener:

$PROMPT> $ORACLE_HOME/bin/lsnrctl stop

Additional Management Agent Commands

The following sections describe additional emctl commands you can use to control the Management Agent:

- Uploading and Reloading Data to the Management Repository

- Specifying New Target Monitoring Credentials

- Listing the Targets on a Managed Host

- Controlling Blackouts

Uploading and Reloading Data to the Management Repository

Under normal circumstances, the Management Agent uploads information about your managed targets to the Management Service at regular intervals.

However, there are two Enterprise Manager commands that can help you force an immediate upload of data to the Management Service or a reload of the target definitions and attributes stored in the Management Agent home directory.

To use these commands, change directory to the AGENT_HOME/bin directory (UNIX) or the AGENT_HOME\bin directory (Windows) and enter the appropriate command as described in Table 8-6.

Table 8-6 Manually Reloading and Uploading Management Data

Command

Purpose

emctl upload

Use this command to force an immediate upload of the current management data from the managed host to the Management Service. Use this command instead of waiting until the next scheduled upload of the data.

emctl reload

This command can be used to modify the emd.properties file. For example, to change the upload interval, emd.properties can be modified, and emctl reload can then be run. This command can also be used when manual edits are made to the Management Agent configuration (.XML) files. For example, if changes are made to the targets.xml file, which defines the attributes of your managed targets, this command will upload the modified target information to the Management Service, which will then update the information in the Management Repository.

Note: Oracle does not support manual editing of the targets.xml files unless the procedure is explicitly documented or you are instructed to do so by Oracle Support.

Specifying New Target Monitoring Credentials

To monitor the performance of your database targets, Enterprise Manager connects to your database using a database user name and password. This user name and password combination is referred to as the database monitoring credentials.

Note:

The instructions in this section are specific to the monitoring credentials for a database target, but you can use this procedure for any other target type that requires monitoring credentials. For example, you can use this procedure to specify new monitoring credentials for your Oracle Management Service and Management Repository.

When you first add an Oracle9i Database target, or when it is added for you during the installation of the Management Agent, Enterprise Manager uses the DBSNMP database user account and the default password for the DBSNMP account as the monitoring credentials.

When you install Oracle Database 10g, you specify the DBSNMP monitoring password during the database installation procedure.

As a result, if the password for the DBSNMP database user account is changed, you must modify the properties of the database target so that Enterprise Manager can continue to connect to the database and gather configuration and performance data.

Similarly, immediately after you add a new Oracle Database 10g target to the Grid Control, you may need to configure the target so it recognizes the DBSNMP password that you defined during the database installation. Otherwise, the Database Home page may display no monitoring data and the status of the database may indicate that there is a metric collection error.

You can modify the Enterprise Manager monitoring credentials by using the Oracle Enterprise Manager 10g Grid Control Console or by using the Enterprise Manager command line utility (emctl).

Using the Grid Control console to Modify the Monitoring Credentials

To modify the password for the DBSNMP account in the Oracle Enterprise Manager 10g Grid Control Console:

1. Click the Targets tab in the Grid Control console.

2. Click the Database subtab to list the database targets you are monitoring.

3. Select the database and click Configure.
Enterprise Manager displays the Configure Database: Properties page.

4. Enter the new password for the DBSNMP account in the Monitor Password field.

5. Click Test Connection to confirm that the monitoring credentials are correct.

6. If the connection is successful, continue to the end of the Database Configuration wizard and click Submit.

Using the Enterprise Manager Command Line to Modify the Monitoring Credentials

To enter new monitoring credentials with the Enterprise Manager command-line utility:

1. Change directory to the AGENT_HOME/bin directory (UNIX) or the AGENT_HOME\bin directory (Windows).

2. Enter the following command to specify new monitoring credentials:

$PROMPT>./emctl config agent credentials [Target_name[:Target_Type]]

To determine the correct target name and target type, see "Listing the Targets on a Managed Host".

shows an example of the prompts and the output you receive from the command.

Example 8-2 Modifying the Database Monitoring Credentials

$PROMPT>./emctl config agent credentials emrep10.acme.com:oracle_database

Oracle Enterprise Manager 10g Release 10.1.0.2.0

Copyright (c) 2002, 2003 Oracle Corporation.  All rights reserved.

Name = emrep10.us.oracle.com, Type = oracle_database

Want to change for "UserName" (y/n):n

Want to change for "password" (y/n):y

Enter the value for "password" :*******

EMD reload completed successfully

Listing the Targets on a Managed Host

There are times when you need to provide the name and type of a particular target you are managing. For example, you must know the target name and type when you are setting the monitoring credentials for a target.

To list the name and type of each target currently being monitored by a particular Management Agent:

1. Change directory to the AGENT_HOME/bin directory (UNIX) or the AGENT_HOME\bin directory (Windows).

2. Enter the following command to specify new monitoring credentials:

$PROMPT>./emctl config agent listtargets  [AGENT_HOME]

shows the typical output of the command.

Example 8-3 Listing the Targets on a Managed Host

./emctl config agent listtargets

Oracle Enterprise Manager 10g Release 10.1.0.2.0

Copyright (c) 2002, 2003 Oracle Corporation.  All rights reserved.

[usunnab08.us.oracle.com, host]

[LISTENER_usunnab08.us.oracle.com, oracle_listener]

[EnterpriseManager.usunnab08.us.oracle.com_HTTP Server, oracle_apache]

[EnterpriseManager.usunnab08.us.oracle.com_home, oc4j]

[EnterpriseManager.usunnab08.us.oracle.com_Web Cache, oracle_webcache]

[EnterpriseManager.usunnab08.us.oracle.com, oracle_ias]

[EnterpriseManager.usunnab08.us.oracle.com_OC4J_EM, oc4j]

[EnterpriseManager.usunnab08.us.oracle.com_OC4J_Demos, oc4j]

[EM_Repository, oracle_emrep]

[usunnab08.us.oracle.com:1813, oracle_emd]

[EM Website, website]

[emrep10.us.oracle.com, oracle_database]

Controlling Blackouts

Blackouts allow Enterprise Manager users to suspend management data collection activity on one or more managed targets. For example, administrators use blackouts to prevent data collection during scheduled maintenance or emergency operations.

See Also:

The "Systems Monitoring" chapter in Oracle Enterprise Manager Concepts for more information about Enterprise Manager blackouts

You can control blackouts from the Oracle Enterprise Manager 10g Grid Control Console or from the Enterprise Manager command line utility (emctl). However, if you are controlling target blackouts from the command line, you should not attempt to control the same blackouts from the Grid Control console. Similarly, if you are controlling target blackouts from the Grid Control console, do not attempt to control those blackouts from the command line.

See Also:

"Creating, Editing, and Viewing Blackouts" in the Enterprise Manager online help for information about controlling blackouts from the Grid Control console

From the command line, you can perform the following blackout functions:

- Starting Immediate Blackouts

- Stopping Immediate Blackouts

- Checking the Status of Immediate Blackouts

Note:

When you start a blackout from the command line, any Enterprise Manager jobs scheduled to run against the blacked out targets will still run. If you use the Grid Control console to control blackouts, you can optionally prevent jobs from running against blacked out targets.

To use the Enterprise Manager command-line utility to control blackouts:

1. Change directory to the AGENT_HOME/bin directory (UNIX) or the AGENT_HOME\bin directory (Windows).

2. Enter the appropriate command as described in Table 8-7.

Note:

When you start a blackout, you must identify the target or targets affected by the blackout. To obtain the correct target name and target type for a target, see "Listing the Targets on a Managed Host".

Table 8-7 Summary of Blackout Commands

Blackout Action

Command

Set an immediate blackout on a particular target or list of targets

emctl start blackout <Blackoutname> [<Target_name>[:<Target_Type>]].... [-d <Duration>]

Be sure to use a unique name for the blackout so you can refer to it later when you want to stop or check the status of the blackout.

The -d option is used to specify the duration of the blackout. Duration is specified in [days] hh:mm where:

- days indicates number of days, which is optional

- hh indicates number of hours

- mm indicates number of minutes

If you do not specify a target or list of targets, Enterprise Manager will blackout the local host target. All monitored targets on the host are not blacked out unless a list is specified or you use the -nodelevel argument.

If two targets of different target types share the same name, you must identify the target with its target type.

Stop an immediate blackout

emctl stop blackout <Blackoutname>

Set an immediate blackout for all targets on a host

emctl start blackout <Blackoutname> [-nodeLevel] [-d <Duration>]

The -nodeLevel option is used to specify a blackout for all the targets on the host; in other words, all the targets that the Management Agent is monitoring, including the Management Agent host itself. The -nodeLevel option must follow the blackout name. If you specify any targets after the -nodeLevel option, the list is ignored.

Check the status of a blackout

emctl status blackout [<Target_name>[:<Target_Type>]]....

Use the following examples to learn more about controlling blackouts from the Enterprise Manager command line:

- To start a blackout called "bk1" for databases "db1" and "db2," and for Oracle Listener "ldb2," enter the following command:

$PROMPT> emctl start blackout bk1 db1 db2 ldb2:oracle_listener -d 5 02:30

The blackout starts immediately and will last for 5 days 2 hours and 30 minutes.

- To check the status of all the blackouts on a managed host:

$PROMPT> emctl status blackout

- To stop blackout "bk2" immediately:

$PROMPT> emctl stop blackout bk2

- To start an immediate blackout called "bk3" for all targets on the host:

$PROMPT> emctl start blackout bk3 -nodeLevel

- To start an immediate blackout called "bk3" for database "db1" for 30 minutes:

$PROMPT> emctl start blackout bk3 db1 -d 30

- To start an immediate blackout called "bk3" for database "db2" for five hours:

$PROMPT> emctl start blackout bk db2 -d 5:00

Changing the Management Agent Time Zone

The Management Agent may fail to start after the upgrade if it realizes that it is no longer in the same time zone that it was originally configured with.

There were bugs in Enterprise Manager Releases 10.1.0.2 and 10.1.0.3 RAC Management Agent installs that caused the Management Agent to be configured with a UTC timezone.

You can correct the time zone used by the Management Agent using the following command:

emctl resetTZ agent

This command will correct the Management Agent side time zone and specify an additional command to be run against the Management Repository to correct the value there.

IMPORTANT:

Before you change the Management Agent time zone, first check to see if there are any blackouts that are currently running or scheduled to run on any target managed by that Management Agent.

To check for blackouts:

1. In the Grid Control Console, go to the All Targets page under the Targets tab, and locate the Management Agent in the list of targets. Click on the Management Agent's name. This brings you to the Management Agent's home page.

2. The list of targets monitored by the Management Agent are listed in the "Monitored Targets" section.

3. For each of target in the list:

1. Click the target name. This brings you to the target's home page.

2. In the Related Links section of the home page, click the Blackouts link. This allows you to check any currently running blackouts or blackouts that are scheduled in the future for this target.

If such blackouts exist, then:

1. From the Grid Control Console, stop all currently running blackouts on all targets monitored by that Management Agent.

2. From the Grid Control Console, stop all scheduled blackouts on all targets monitored by that Management Agent.

Once you have stopped all currently running and scheduled blackouts, you can run the emctl resetTZ agent command to change the Management Agent's time zone.Once you have changed the Management Agent's time zone, create new blackouts on the targets as needed.

Reevaluating Metric Collections

If you are running a Management Agent Release 10.2, then you can use the following command to perform an immediate reevaluation of a metric collection:

emctl control agent runCollection <targetName>:<targetType> <colletionItemName>

where <collectionItemName> is the name of the Collection Item that collects the metric.

Performing this command causes the reevaluated value of the metric to be uploaded into the Management Repository, and possibly trigger alerts if the metric crosses its threshold.

Related metrics are typically collected together; collectively a set of metrics collected together is called a Metric Collection. Each Metric Collection has its own name. If you want to reevaluate a metric, you first need to determine the name of the Metric Collection to which it belongs, then the CollectionItem for that Metric Collection.

When you run the previous command to reevaluate the metric, all other metrics that are part of the same Metric Collection and Collection Item will also be reevaluated.

Perform the following steps to determine the Metric Collection name and Collection Item name for a metric:

1. Go to $ORACLE_HOME/sysman/admin/metadata directory, where $ORACLE_HOME is the Oracle Home of the Management Agent.

2. Locate the XML file for the target type. For example, if you are interested in the host metric 'Filesystem Space Available(%)' metric, look for the host.xml file.

3. In the xml file, look for the metric in which you are interested. The metric that you are familiar with is actually the display name of the metric. The metric name would be preceded by a tag that started with:
<Label NLSID=

For example, in the host.xml file, the metric 'Filesystem Space Available(%)" would have an entry that looks like this:

<Label NLSID="host_filesys_pctAvailable">Filesystem Space Available (%) </Label>

4. Once you have located the metric in the xml file, you will notice that its entry is part of a bigger entry that starts with:
<Metric NAME=

Take note of the value defined for "Metric NAME". This is the Metric Collection name. For example, for the 'Filesystem Space Available(%)' metric, the entry would look like this:

<Metric NAME="Filesystems"

So for the 'Filesystem Space Available(%)' metric, the Metric Collection name is 'Filesystems'.

5. The Collection Item name for this Metric Collection needs to be determined next. Go to the $ORACLE_HOME/sysman/admin/default_collection directory, where $ORACLE_HOME is the Oracle Home of the Management Agent.

6. In this directory, look for the collection file for the target type. In our example, this would be host.xml.

7. In cases where a Metric Collection is collected by itself, there would be a single Collection Item of the same name in the collection file. To determine if this is the case for your Metric Collection, look for an entry in the collection file that starts with:
<CollectionItem NAME=

where the value assigned to the CollectionItem NAME matches the Metric NAME in step (4).

For the 'Filesystem Space Available(%)' metric, the entry in the collection file would look like:

<CollectionItem NAME = "Filesystems"

8. If you find such an entry, then the value assigned to "CollectionItem NAME" is the collection item name that you can use in the emctl command.

9. Otherwise, this means the Metric Collection is collected with other Metric Collections under a single Collection Item. To find the Collection Item for your Metric Collection, first search for your Metric Collection. It should be preceded by the tag:
<MetricColl NAME=

Once you have located it, look in the file above it for: <CollectionItem NAME=

The value associated with the CollectionItem NAME is the name of the collection item that you should use in the emctl command.

For example if the you want to reevaluate the host metric "Open Ports", using the previous steps, you would do the following:

1. Go to the $ORACLE_HOME/sysman/admin/metadata directory where $ORACLE_HOME is the Oracle Home of the Management Agent. Look for the host.xml file and in that file locate: <Metric NAME="openPorts".

2. Then go to the $ORACLE_HOME/sysman/admin/default_collection directory. Look for the host.xml file and in that file look for <CollectionItem NAME="openPorts".
Failing this, look for <MetricColl NAME="openPorts".

3. Look above this entry in the file to find the <CollectionItem NAME= string and find <CollectionItem NAME="oracle_security".

The CollectionItem NAME oracle_security is what you would use in the emctl command to reevaluate the Open Ports metric.

emctl Commands

This section lists the emctl commands for the Enterprise Manager agent.

Table 8-8 emctl Commands

emctl Command

Description

emctl start | stop agent

Starts or stops agent.

emctl start | stop | status subagent

Starts or stops subagent.

emctl status agent

Lists the status of agent.

emctl status agent -secure [-omsurl <http://<oms-hostname>:<oms-unsecure-port>/em/*>]

Lists the secure status of the agent and the port on which the agent is running in secure mode and also the OMS security status of the agent it points to. This command also gives the OMS secure port. Below is an example output:

bash-3.00$ emctl status agent -secure

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

Checking the security status of the Agent at location set in /ade/aime_cpap4_ag/oracle/sysman/config/emd.properties...  Done.

Agent is secure at HTTPS Port 1838.

Checking the security status of the OMS at http://staxe05.us.oracle.com:7654/em/upload/...  Done.

OMS is secure on HTTPS Port 4473

bash-3.00$

emctl status agent scheduler

Lists all Running, Ready, and Scheduled Collection threads.

emctl status agent jobs

Lists the status of the jobs that are running at present on the agent. The following is an example output:

bash-3.00$ emctl status agent jobs

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

-------------------------------------------------

step id  typ     pid  stat  command line

-------  ---     ---  ----  ------------

-------------------------------------------------

Agent is Running and Ready

emctl status agent target <target name>,<target type>,<metric>

Lists the detailed status of the specified targets in the order of target name, target type. The following is an example of an oracle_database target. You can also provide a particular metric name in the emctl command to get the status of a particular metric of a target.

bash-3.00$ emctl status agent target database,oracle_database

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

-------------------------------------------------

Target Name : database

Target Type : oracle_database

Current severity state

----------------------

Metric        Column name      Key    State   Timestamp

-------------------------------------------------

DeferredTrans errortrans_count  n/a CLEAR 2009-07-09 02:38:07

DeferredTrans deftrans_count    n/a   CLEAR 2009-07-09 02:38:07

ha_recovery missing_media_files n/a   CLEAR 2009-07-09 02:28:57

ha_recovery corrupt_data_blocks n/a  CLEAR 2009-07-09 02:28:57

ha_recovery datafiles_need_recovery n/a CLEAR 2009-07-09 02:28:57

Response     Status        n/a     CLEAR  2009-07-09 02:38:04

Response userLogon         n/a CLEAR  2009-07-09 02:38:04

Response    State          n/a  CLEAR  2009-07-09 02:38:04

OCMInstrumentation  NeedToInstrument n/a CLEAR 2009-07-09 02:31:55

health_check  Status  n/a  CLEAR   2009-07-09 02:40:00

health_check  Unmounted  n/a  CLEAR  2009-07-09 02:40:00

health_check   Mounted  n/a CLEAR  2009-07-09 02:40:00

health_check Unavailable  n/a  CLEAR  2009-07-09 02:40:00

health_check Maintenance n/a  CLEAR  2009-07-09 02:40:00

sql_response       time  n/a  CLEAR  2009-07-09 02:38:50

sga_pool_wastage java_free_pct n/a  CLEAR  2009-07-09 02:28:

58

UserAudit username DBSNMP_staxe05  CLEAR 2009-07-09 02:32:48

-------------------------------------------------------

Agent is Running and Ready

emctl status agent mcache <target name>,<target type>,<metric>

Lists the names of the metrics for which the values are present in the metric cache. See the following example for a simple host target:

bash-3.00$ emctl status agent mcache staxe05.us.oracle.com,host

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

-------------------------------------------------------

Metric cache contains value for following metrics at 2009-07-09 02:54:47

CPUUsage

DiskActivity

FileMonitoring

LPAR Performance on AIX

Load

Network

PagingActivity

-------------------------------------------------------

Agent is Running and Ready

The metrics listed above are the ones whose values are present in the metric cache.

emctl status agent cpu

Dumps the agent Thread CPU usage into a .trc file. This file contains the list of all the threads that are running at present and their CPU usage.Following is the sample output of emctl status agent cpu:

bash-3.00$ emctl status agent cpu

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

-------------------------------------------------------

Agent Thread CPU Snapshot available in file: /ade/aime_cpap4_ag/oracle/sysman/emd/cputrack/emagent_1654792_2009-07-09_02-58-54_cpudiag.trc

-------------------------------------------------------

Agent is Running and Ready

emctl status agent mutex

Gives the detailed status of the agent mutex contention for each thread. It gives the acquired, release, and wait time of mutexes for each thread.Following is a sample output:

bash-3.00$ emctl status agent mutex

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

-------------------------------------------------------

Mutex status at 2009-07-09 03:03:46 (prev : 2009-07-09 03:03:13)

Addr                 Name :  TotAcq  TotRel TotWT   LSAcq   LSRel LSWT MxWt OwnerTid

2346a578            CollState :      14      14    18       0       0    0    1 NULL-thread

232e7628            CollState :       0       0     0       0       0    0    0 NULL-thread

23386638            CollState :       0       0     0       0       0    0    0 NULL-thread

2329fd28            CollState :       0       0     0       0       0    0    0 NULL-thread

239521c8             CollItem :       0       0     0       0       0    0    0 NULL-thread

2328cd98            CollState :       0       0     0       0       0    0    0 NULL-thread

23a54948             CollItem :       0       0     0       0       0    0    0 NULL-thread

232d8b68             CollItem :       1       1     1       0       0    0    1 NULL-thread

2358ce38      MetricCacheItem :       2       2     2       0       0    0    1 NULL-thread

emctl status agent memory

Used for debugging agent memory. You will need to set “enableMemoryTracing=TRUE” in emd.properties for memory profiling agent.

emctl status agent memclean

Clears the memory hash table. Note that by default, the memory tracing will not be enabled by the agent. So, if the memory tracing is not enabled, then emctl status agent memclean will not clear the hashtable. To enable memory tracing, you need to set enableMemoryTracing=true in emd.properties of the agent and then reload the agent.The following is a sample output:

bash-3.00$ emctl status agent memclean

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

-------------------------------------------------------

Memory hashtable cleared.

------------------------------------------------------

Agent is Running and Ready

emctl reload [agent]

Reloads the agent by reading the emd.properties and targets.xml files again. If you have changed any property in emd.properties file, for example, if you have changed the tracing level of collector in the emd.properties by changing tracelevel.collector=DEBUG (default will be WARN) then you need to reload the agent to make the agent takes this change into account. Note that the agent should be up and running for the reload to happen successfully.

emctl reload agent dynamicproperties [<Target_name>:<Target_Type>]...

Recomputes the dynamic properties of a target and generates the dynamic properties for the target.Sample output for oracle_database is as follows:

bash-3.00$ emctl reload agent dynamicproperties database:oracle_database

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

-------------------------------------------------------

EMD recompute dynprops completed successfully

emctl upload

Uploads xml files that are pending to upload to the OMS under the upload directory.

emctl pingOMS [agent]

Pings the OMS to check if the agent is able to connect to the OMS. Agent will wait for the reverse ping from the OMS so that agent can say the pingOMS is successful.

emctl config agent <options>

Configures agent based on the options provided.

emctl config agent updateTZ

Updates the current timezone of the agent in emd.properties file.

emctl config agent getTZ

Prints the current timezone of the agent.

emctl config agent credentials [<Target_name>[:<Target_Type>]]

Provides the option to change the credentials for a particular target. Through this, you can change the user name and password of the target. It will ask you when you run this command whether you want to change the user name or password. If you select yes, then you have to provide the new user name and password for the target which you want to configure. Then it will reload the agent.Sample output for oracle_database is as follows:

bash-3.00$ emctl config agent credentials database

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

Name = database, Type = oracle_database

Want to change for "UserName" (y/n):

Want to change for "password" (y/n):

EMD reload completed successfully

emctl config agent getSupportedTZ

Prints the supported timezones for the agent.

emctl config console <fileloc> [<EM loc>]

Allows you to configure the console based on the configuration entries that you have mentioned in the file <fileloc>.

<EM loc> is optional and can be used to operate on a different Oracle Home.

emctl config [agent] addtarget [-f|-force] <fileloc> [<EM loc>]

Allows you to configure agent.

<fileloc> contains a definition of target to add and where -f or -force allows to overwrite an existing target.If -f or -force is not specified, existing targets cannot be overwritten.

<EM loc> is optional and can be used to operate on a different Oracle Home.

emctl config [agent] addtargets [-f|-force] <fileloc> [<EM loc>]

Allows you to configure agent.

<fileloc> contains a definition of targets to add and -f or -force allows to overwrite existing targets.If -f or -force is not specified, existing targets cannot be overwritten.

<EM loc> is optional and can be used to operate on a different Oracle Home.

emctl config [agent] modifytarget <fileloc> [<EM loc>] [<EM State>] [-mergeProps]

Allows you to configure agent.

<fileloc> contains a definition of target to modify and -mergeProps is used when only target properties are to be updated.

<EM loc> is optional and can be used to operate on a different Oracle Home.

emctl config [agent] deletetarget <type> <name> [<EM loc>]

Allows you to delete target.

<type>,<name> specify target type and name to delete.

<EM loc> is optional and can be used to operate on a different Oracle Home.

emctl config [agent] listtargets [<EM loc>]

Lists all targets present in targets.xml.

<EM loc> is optional and can be used to operate on a different Oracle Home.

emctl config agent listtargetsfully [<EM loc1>] [<EM loc2>] ...

Lists all targets present in targets.xml of the given Enterprise Manager location.

<EM loc> is optional and can be used to operate on a different Oracle Home.

emctl config [agent] listcentralagents [<EM loc>]

Lists the central agents this home is associated with.

The centralagent command does not apply in an agent-only home.

<EM loc> is optional and can be used to operate on a different Oracle Home.

emctl config [agent] addcentralagent <centralAgentHomePath> [<EM loc>]

Associates this home with a new central agent.

The centralagent command does not apply in an agent-only home.

<EM loc> is optional and can be used to operate on a different Oracle Home.

emctl config [agent] removecentralagent <centralAgentHomePath> [<EM loc>]

Removes the association of this home with a central agent.

The centralagent command does not apply in an agent-only home.

<EM loc> is optional and can be used to operate on a different Oracle Home.

emctl config [agent] upgradecentralagent <centralAgentHomePathOld> [<centralAgentHomePathNew>]

Upgrades all product homes being monitored by this central agent.

The centralagent command does not apply in an agent-only home.

emctl config [agent] setcentralagents <centralAgent1> [<centralAgent2> ...]

Sets the list of central agents this home is associated with.

The centralagent command does not apply in an agent-only home.

emctl config agent addTargetsToRepository <uploadFile> <update_on_dup (true | false)>

Adds targets to repository.

uploadFile contains definition for targetsupdate_on_dup decides whether updating duplicate targetsThis function is for central agent.

emctl config agent addAssociationsToRepository <uploadFile>

Adds associations to repository.

uploadFile contains definition(s) for association(s)This function is for central agent.

emctl config agent getLocalHost

Prints the local host where the agent is running.

emctl control agent runCollection <target_name>:<target_type> <metric_name>

Allows to manually run the collections for a particular metric of a target. Sample output is as follows:

bash-3.00$ emctl control agent runCollection staxe05.us.oracle.com:host CPUUsage

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

-------------------------------------------------------

EMD runCollection completed successfully

emctl getcurdir agent

Prints the current working directory you are in (pwd).

emctl resetTZ agent

Resets the timezone of the agent. Stop the agent first and then run this command to change the current timezone to a different timezone. Then start the agent.

emctl resettzhost <hostname> <override_timezone>

Resets the timezone settings of the host where the agent is running.

emctl getversion

Prints the version of the agent. Sample output is as follows:

bash-3.00$ emctll getversion

bash: emctll: command not found

bash-3.00$ emctl getversion

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

--- Standalone agent

Enterprise Manager 10g Agent Version 10.2.0.5.0

emctl dumpstate agent <component> . . .

Generates the dumps for the agent. This command allow you to analyze the memory/cpu issues of the agent. Sample output is as follows:

bash-3.00$ emctl dumpstate agent

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

dump file generated: /ade/aime_cpap4_ag/oracle/sysman/dump/emagent_1654792_20090709042448.diagtrc

bash-3.00$

emctl gensudoprops

Generates the sudo properties of the agent.

emctl clearsudoprops

Clears the sudo properties.

emctl clearstate

Clears the state directory contents. The files that are located under $ORACLE_HOME/sysman/emd/state will be deleted if this command is run. The state files are the files which are ready for the agent to convert them into corresponding xml files.

emctl getemhome

Prints the agent home directory. The sample output is as follows:

bash-3.00$ emctl getemhome

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation.  All rights reserved.

EMHOME=/ade/aime_cpap4_ag/oracle

emctl start blackout <Blackoutname> [-nodeLevel] [<Target_name>[:<Target_Type>]].... [-d <Duration>]

Starts blackout on a target.

<Target_name:Target_type> defaults to local node target if not specified.If -nodeLevel is specified after <Blackoutname>,the blackout will be applied to all targets and any target list that follows will be ignored.Duration is specified in [days] hh:mm

emctl stop blackout <Blackoutname>

Stops the blackout that was started on a particular target. Only those blackouts that are started by the emctl tool can be stopped using emctl. This command cannot stop the blackouts that are started using the Console or emcli.

emctl status blackout [<Target_name>[:<Target_Type>]]....

Provides the status of the blackout of the target. The status includes the type of blackout, whether one time, repeating, or a scheduled blackout. This command also specifies whether the blackout has started or stopped.

emctl secure agent <registration password> [-passwd_file <abs file loc>]

Secures the agent against an OMS. The registration password must be provided.

emctl unsecure agent

Unsecures the agent. This will make the agent unsecure and the agent's port will be changed to http port.

emctl verifykey

Verifies the communication between the OMS and agent by sending pingOMS.

emctl deploy agent [-s <install-password>] [-o <omshostname:consoleSrvPort>] [-S] <deploy-dir> <deploy-hostname>:<port> <source-hostname>

'agent' creates and deploys only the agent.

[-s <password>]: Install password for securing agent.

[-S ]: Password will be provided in STDIN.

[-o <omshostname:consoleSrvPort>]: The OMS Hostname and console servlet port. Choose the unsecured port.

<deploy-dir> : Directory to create the shared (state-only) installation port.

<deploy-hostname:port> : Host name and port of the shared (state-only) installation. Choose unused port.

<source-hostname>: The host name of the source install. Typically the machine where EM is installed. This is searched and replaced in targets.xml by the host name provided in argument <deploy-hostname:port>.

<sid>: The instance of the remote database. Only specified when deploying "dbconsole".

emctl deploy dbconsole [-s <install-password>] <deploy-dir> <deploy-hostname>:<port> <source-hostname> <sid>

'dbconsole' creates and deploys both the agent and the dbconsole.

[-s <password>]: Install password for securing agent.

<deploy-dir> : Directory to create the shared (state-only) installation port.

<deploy-hostname:port> : Host name and port of the shared(state-only) installation. Choose unused port.

<source-hostname>: The host name of the source install. Typically the machine where EM is installed. This is searched and replaced in targets.xml by the host name provided in argument <deploy-hostname:port>.

<sid>: The instance of the remote database. Only specified when deploying "dbconsole".

emctl ilint

Allows ilint support of agent.

Using emctl.log File

The emctl.log file is a file that captures the results of all emctl commands you run. The log file resides in the $ORACLE_HOME/sysman/log directory of the Management Agent, and is updated every time you run an emctl command. If your emctl command fails for some reason, access this log file to diagnose the issue.

For example, run the following command from the Oracle home directory of the Management Agent to check its status:

<Oracle_Home>emctl status agent

After running the command, navigate to the log directory to view the following information in the emctl.log file:

1114306 :: Wed Jun 10 02:29:36 2009::AgentLifeCycle.pm: Processing status agent

1114306 :: Wed Jun 10 02:29:36 2009::AgentStatus.pm:Processing status agent

1114306 :: Wed Jun 10 02:29:37 2009::AgentStatus.pm:emdctl status returned 3

Here, the first column, that is, 1114306, is the thread that was used to check the status. The second column shows the date and time when the command was run. The third column mentions the Perl script that was run for the command. The last column describes the result of the command, where it shows the progress made by the command and the exit code returned for the command. In this case, the exit code is 3, which means that the Management Agent is up and running.

Another example, run the following command from the Oracle home directory of the Management Agent to upload data:

<Oracle_Home>emctl upload agent

After running the command, navigate to the log directory to view the following information in the emctl.log file:

1286220 :: Tue Jun  9 07:13:09 2009::AgentStatus.pm:Processing upload

1286220 :: Tue Jun  9 07:13:10 2009::AgentStatus.pm:emdctl status agent returned 3

1286220 :: Tue Jun  9 07:13:41 2009::AgentStatus.pm: emdctl upload returned with exit code 6

Here, the entries are similar to the entries in the first example, but the exit code returned is 6, which means the upload operation is failing for some reason.

Table 8-9 describes the different exit codes of an emctl command:

Table 8-9 Description of Exit Codes

Exit Code

Description

0

Operation was successful. No action required from you.

1

Failed to connect to the Management Agent. Maybe the Management Agent is not running. Restart the Management Agent, and then try the emctl command again.

2

Timed out while connecting to the Management Agent. Maybe the Management Agent is hanging. Restart the Management Agent, and then try the emctl command again.

3

Management Agent is up and running. No action required from you.

4

Management Agent is up but not ready. Wait for some more time, and then try the emctl command again.

5

Input/output error while sending Management Agent-related request or receiving Management Agent-related response.

6

Unable to upload. Check the status of the Management Agent, and then try the emctl command again.

7

Management Agent is in an abnormal state. Check the status of the Management Agent, and then try the emctl command again.

8

Operation was incomplete. The command might have timed out. Try again.

9

Usage error. Check the command you are running and try again with the correct command.

10

SSL handshake error while communicating with the Management Agent. Secure the Management Agent, and then try the emctl command again.

11

Key mismatch while communicating with the Management Agent. Secure the Management Agent, and then try the emctl command again.

255

Unable to open the temporary file. Try the emctl command again.

9 Backup, Recovery, and Disaster Recovery

Enterprise Manager's distributed architecture necessitates a multi-pronged approach to backup, recovery and disaster recovery planning. This chapter covers practical approaches to implementing an efficient and robust backup and recovery environment and discusses the best practices for backup and recovery of different tiers of Enterprise Manager.

This chapter contains the following sections:

- Backup and Recovery of Enterprise Manager

- Repository Backup and Recovery

- OMS Backup and Recovery

- Agent Backup and Recovery

- Recovering from a Compound OMS-Repository Failure

- EMCTL High Availability Commands

Backup and Recovery of Enterprise Manager

Although Enterprise Manager functions as a single entity, technically, it is built on a distributed, multi-tier software architecture composed of the following software components:

- Oracle Management Services (OMS)

- Oracle Management Agent (Agent)

- Oracle Management Repository (Repository)

Each component, being uniquely different in composition and function, requires different approaches to backup and recovery. For this reason, the backup and recovery strategies are discussed on a per-tier basis in this chapter. For an overview of the Enterprise Manager Architecture, refer to Enterprise Manager Grid Control Installation and Basic Configuration 10g Release 5 (10.2)

Repository Backup and Recovery

The Repository is the storage location where all the information collected by the Agent gets stored. It consists of objects such as database jobs, packages, procedures, views, and tablespaces. Because it is configured in an Oracle Database, the backup and recovery strategies for the repository are essentially the same as those for the Oracle Database. Backup procedures for the database are well established standards and can be implemented using the RMAN backup utility, which can be accessed via the Enterprise Manager console.

Repository Backup

Oracle recommends using High Availability Best Practices for protecting the Repository database against unplanned outages. As such, use the following standard database backup strategies.

- Database should be in archivelog mode. Not running the repository database in archivelog mode leaves the database vulnerable to being in an unrecoverable condition after a media failure.

- Perform regular hot backups with RMAN using the Recommended Backup Strategy option via the Enterprise Manager console. Other utilities such as DataGuard and RAC can also be used as part of a comprehensive strategy to prevent data loss.

Adhering to these strategies will create a full backup and then create incremental backups on each subsequent run. The incremental changes will then be rolled up into the baseline, creating a new full backup baseline.

Using the Recommended Backup Strategy also takes advantage of the capabilities of Enterprise Manager to execute the backups: Jobs will be automatically scheduled through the Job sub-system of Enterprise Manager. The history of the backups will then be available for review and the status of the backup will be displayed on the repository database target home page. This backup job along with archiving and flashback technologies will provide a restore point in the event of the loss of any part of the repository. This type of backup, along with archive and online logs, allows the repository to be recovered to the last completed transaction.

Setting Up the Backup

First, navigate to the Enterprise Manager Recovery Settings page (Target-->Database--><Repository Database Target>-->Availability-->Recovery Settings) and enable Archive Logging then Flashback Database as shown in Figure 9-1.

Figure 9-1 Recovery Settings Page

Description of "Figure 9-1 Recovery Settings Page"

Next, navigate to the Backup Policies page (Target-->Database--><Repository Database Target>-->Availability-->Backup Settings-->Policy) and enable Block Change Tracking to speed up backup operations as shown in Figure 9-2.

Figure 9-2 Backup Policy Page

Description of "Figure 9-2 Backup Policy Page"

A thorough summary of how to configure backups using Enterprise Manager is available in the Oracle Database 2 Day DBA guide. For additional information on Database high availability best practices, review the Oracle Database High Availability Best Practices documentation.

Repository Recovery

Recovery of the Repository database must be performed using RMAN since Grid Control will not be available when the repository database is down. There are two recovery cases to consider:

- Full Recovery: No special consideration is required for Enterprise Manager.

- Point in time/Incomplete Recovery: Recovered repository may be out of sync with Agents because of lost transactions. In this situation, some metrics may show up incorrectly in the Grid Control console unless the repository is synchronized with the latest state available on the Agents.

Beginning with Enterprise Manager version 10.2.0.5, a new repository resync feature allows you to automate the process of synchronizing the Enterprise Manager repository with the latest state available on the Agents. To resynchornize the repository with the Agents, you use Enterprise Manager Command-line utility (emctl) resync repos command:

emctl resync repos -full -name "<descriptive name for the operation>"

You must run this command from the OMS Oracle Home after restoring the repository but BEFORE starting the OMS. After submitting the command, start up all OMS's and monitor the progress of repository resychronization from the Enterprise Manager console's Repository Resynchronization page, as shown in Figure 9-3.

Figure 9-3 Repository Synchronization Page

Description of "Figure 9-3 Repository Synchronization Page"

Repository recovery is complete when the resynchronization jobs complete on all Agents.

Oracle strongly recommends that the repository database be run in archivelog mode so that in case of failure, the database can be recovered to the latest transaction. If the database cannot be recovered to the last transaction, Repository Synchronization can be used to restore monitoring capabilities for targets that existed when the last backup was taken. Actions taken after the backup will not be recovered automatically. Some examples of actions that will not be recovered automatically by Repository Synchronization are:

- Notification Rules

- Preferred Credentials

- Groups, Services, Systems

- Jobs/Deployment Procedures

- Custom Reports

- New Agents

Manually Resynchronizing Agents

The Enterprise Manager Repository Synchronization feature can only be used for Agents 10.2.0.5 or later. Older Agents must be resynchronized manually by shutting down the Agent using the following procedure:

1. Shut down the Agent.

2. Delete the agentstmp.txt, lastupld.xml, state/* and upload/* files from the $AGENT_HOME/sysman/emd directory.

3. Restart the Agent.

Recovery Scenarios

A prerequisite for repository (or any database) recovery is to have a valid, consistent backup of the repository. Using Enterprise Manager to automate the backup process ensures regular, up-to-date backups are always available if repository recovery is ever required. Recovery Manager (RMAN) is a utility that backs up, restores, and recovers Oracle Databases. The RMAN recovery job syntax should be saved to a safe location. This you to perform a complete recovery of the Enterprise Manager repository database. In its simplest form, the syntax would appear as follows:

run {

restore database;

recover database;

}

Actual syntax will vary in length and complexity depending on your environment. For more information on extracting syntax from an RMAN backup and recovery job, or using RMAN in general, see the Oracle Database Backup and Recovery Advanced User's Guide.

The following scenarios illustrate various repository recovery situations along with the recovery steps.

Full Recovery on the Same Host

Repository database is running in archivelog mode. Recent backup, archive log files and redo logs are available. The repository database crashes.

Resolution:

1. Stop the OMS(s) using opmnctl stopall.

2. Recover the database using RMAN

3. Bring the site up using opmnctl startall on all OMS(s)

4. No further action is required.

5. Verify that the site is fully operational.

Incomplete Recovery on the Same Host

Repository database is running in noarchivelog mode. Full offline backup is available. The repository database crashes.

Resolution:

1. Stop the OMS(s) using opmnctl stopall.

2. Recover the database using RMAN

3. Initiate Repository Resync using emctl resync repos -full -name "<resync name>" from one of the OMS Oracle home.

4. Start OMS(s) using opmnctl startall.

5. Manually fix all pre-10.2.0.5 Agents by shutting down the Agent, deleting the agentstmp.txt, lastupld.xml, state/* and upload/* files under the $AGENT_HOME/sysman/emd directory. Restart the Agent.

6. Log into Grid Control. Navigate to Management Services and Repository Overview page. Click on Repository Synchronization under Related Links. Monitor the status of resync jobs. Resubmit failed jobs, if any, after fixing the error.

7. Verify that the site is fully operational.

Full Recovery on a Different Host

The repository database is running on host "A" in archivelog mode. Recent backup, archive log files and redo logs are available. Host "A" is lost due to hardware failure.

Resolution:

1. Stop the OMS(s) using opmnctl stopall.

2. Recover the database using RMAN on a different host (host "B").

3. Change the connect descriptor in each OMS emoms.properties file to point to database on host "B".

4. Start the OMS(s) using opmnctl startall.

5. Relocate the repository database target to the Agent running on host "B" by running the following command from the OMS:

$emctl config repos -host <hostB> -oh <OH of repository on hostB>  -conn_desc "<TNS connect descriptor>"

Note:

This command can only be used to relocate the repository database under the following conditions:

- An Agent is already running on this machine.

- No database on host "B" has been discovered.

If a new Agent had been installed on host "B", you must ensure there are NO previously discovered database targets.

6. Change the monitoring configuration for the OMS and Repository target: by running the following command from the OMS:

$emctl config emrep -conn_desc "<TNS connect descriptor>"

7. Verify that the site is fully operational.

Incomplete Recovery on a Different Host

The repository database is running on host "A" in noarchivelog mode. Full offline backup is available. Host "A" is lost due to hardware failure.

Resolution:

1. Stop the OMS(s) using opmnctl stopall.

2. Recover the database using RMAN on a different host (host "B").

3. Change the connect descriptor in each OMS emoms.properties file to point to the database on host "B".

4. Initiate Repository Resync:
emctl resync repos -full -name "<resync name>"

from one of the OMS Oracle homes.

5. Start the OMS(s) using opmnctl startall

6. Run the command to relocate the repository database target to the Agent running on host "B":
emctl config repos -agent <agent on host B> -host <hostB> -oh <OH of repository on hostB> -conn_desc "<TNS connect descriptor>"

7. Run the command to change monitoring configuration for the OMS and Repository target:
emctl config emrep -conn_desc "<TNS connect descriptor>"

8. Manually fix all pre-10.2.0.5 Agents by shutting down the Agent, deleting the agentstmp.txt, lastupld.xml, state/* and upload/* files under $AGENT_HOME/sysman/emd directory. Restart the Agent.

9. Log in to Grid Control. Navigate to Management Services and Repository Overview page. Click on Repository Synchronization under Related Links. Monitor the status of resync jobs. Resubmit failed jobs, if any, after fixing the error mentioned.

10. Verify that the site is fully operational.

OMS Backup and Recovery

The OMS is a J2EE Web application that orchestrates with Management Agents to discover targets, monitor and manage them, and store the collected information in a repository for future reference and analysis. The OMS also renders the Web interface for the Enterprise Manager console.

Backing Up the OMS

The OMS is generally stateless. Some transient and configuration data is stored on the OMS file system. The shared loader “recv”directory stores metric data uploaded from Agents temporarily before the data is loaded into the repository. If an OMS goes down, other surviving OMS(s) upload the data stored in the shared loader location. In a High Availability (HA) configuration, the shared loader receive directory should be protected using some HA storage technology.

Beginning with Enterprise Manager version 10.2.0.5, a snapshot of OMS configuration can be taken using the emctl exportconfig oms command.

emctl exportconfig oms [-sysman_pwd <sysman password>]

[-dir <backup dir>]     Specify directory to store backup file

[-keep_host]            Specify this parameter if the OMS was installed

using a virtual hostname.

For example: ORACLE_HOSTNAME

Running exportconfig captures a snapshot of the OMS at a given point in time, thus allowing you to back up the most recent OMS configuration on a regular basis. If required, the most recent snapshot can then be restored on a fresh OMS installation on the same or different host.

Recovering the OMS

If an OMS is lost, it should be reinstalled using “Installing Software-Only and Configuring Later". This is an additional Management Service option documented in the Oracle Enterprise Manager Grid Control Installation and Basic Configuration guide. The OMS configuration can then be restored with the following command:

emctl importconfig oms

Use export file backed up earlier. This command is available beginning with Enterprise Manager version 10.2.0.5.

OMS Recovery Scenarios

The following scenarios illustrate various OMS recovery situations along with the recovery steps.

Important:

A prerequisite for recovery is to have recent, valid OMS configuration backups available. Oracle recommends that you back up the OMS using the emctl exportconfig oms command whenever an OMS configuration change is made. Alternatively, you can run this command on a regular basis using the Enterprise Manager job system.

Single OMS with No Server Load Balancer (SLB). OMS Restored on the same Host

Single OMS site. No SLB is present. The OMS configuration was backed up using the emctl exportconfig oms command. OMS is lost.

Resolution:

1. If possible, deinstall the OMS and Agent OracleHomes using the Oracle Universal Installer (OUI).

Note:

This step only applies if there is some remnant of the OMS still exists on the file system. Deinstallation of the OMS using OUI would not be possible if the OMS was lost as a result of disk/media failure.

2. Reinstall the OMS using “Install Software only, configure later” – Additional management service option. Location of the install need not be same as the previous install.

3. Import the OMS configuration:
emctl importconfig oms -file <exportfile>

Important:

When using the importconfig command, the following restrictions apply:

- The OMS configuration import can only be performed between the same operating system types.

- The OMS configuration must be imported into the same version OMS.

- Importing the OMS configuration only copies information which is unique to the installation (for example, emkey). For this reason, you should not use the importconfig command to copy OMS configurations across different Repositories.

At this point two options exist depending upon the port used by the reinstalled Agent that comes along with the OMS:

Option A: Agent uses the same port as the previous installation.

- OMS automatically blocks the Agent. Resync the Agent from Agent homepage

Option B: Agent uses a different port.

- Run the command to relocate the OMS and Repository target to reinstalled Agent:
emctl config emrep -agent <reinstalled agent>

- Locate duplicate targets from the Management Services and Repository Overview page. Relocate duplicate targets from the old agent to the reinstalled Agent. Delete the old Agent.

4. Verify that the site is fully operational.

Single OMS, No SLB, OMS Restored on a Different Host

Single OMS site. The OMS is running on host "A." No SLB is present. The OMS configuration was backed up using the emctl exportconfig oms command. Host "A" is lost.

Resolution:

1. Reinstall the OMS using “Install Software only, configure later” – Additional management service option. Location of install need not be same as earlier install.

2. Import the OMS configuration:
emctl importconfig oms -file <exportfile>

At this point, two possibilities exist depending upon your network setup

3. Option A: Make host "B" reachable with hostname host "A"

- Update the network information on host "B" so that it is reachable using the old host name from host "A." This can be done by multi-homing and adding an additional IP of host "A" to host "B".

- Resecure the OMS
emctl secure oms -host <host A>

- Resecure the Agent on host "B".
emctl secure agent -emdWalletUrlSrc "http://hostA:<httpport>/em"

Note:

Because the new machine uses the same hostname as the old machine, all the Agents in your monitored environment already know where to locate the new OMS.

Option B: Change the OMS to which all Agents point and then resecure all Agents

- Make all Agents in the deployment point to new OMS running on host "B". On each Agent, run the following command
emctl secure agent -emdWalletUrlSrc "http://hostB:<httpport>/em"

Run the command to relocate OMS and Repository target to Agent "B":

emctl config emrep -agent <agent on host "B">.

Note:

Because the new machine is using a different hostname from the one originally hosting the OMS, all Agents in your monitored environment must be told where to find the new OMS.

4. Locate duplicate targets from the Management Services and Repository Overview page of the Enterprise Manager console. Clicking the Duplicate Targets link, will bring you to the Duplicate Targets page. To resolve duplicate target errors, the duplicate target must be renamed on the conflicting Agent.

5. Verify that the site is fully operational.

Multiple OMS, Server Load Balancer configured, OMS restored on the same host

Multiple OMS site. All OMSs fronted by an SLB. OMS configuration backed up using the emctl exportconfig oms command. One OMS is lost.

Resolution:

1. Deinstall the OMS and Agent OracleHomes using the Oracle Universal Installer.

2. Reinstall the OMS on same host using “Install Software only, configure later” – Additional management service option. Location of the install need not be the same as the earlier install.

3. Import the OMS configuration:
emctl importconfig oms -file <exportfile>

4. Resecure the Agent that gets installed along with OMS.
emctl secure agent -emdWalletSrcUrl"http://slb:<httpport>/em"

Because the Agent that is installed with the OMS uses the same port as before, the OMS automatically blocks the Agent. You need to "resync" the Agent from the Agent homepage.

5. Verify that the site is fully operational.

Multiple OMS, Server Load Balancer configured, OMS restored on a different host

Multiple OMS site. OMSs fronted by Server Load Balancers. OMS configuration backed up using the emctl exportconfig oms command. OMS on host "A" is lost.

1. Ensure that shared loader receive directory and shared software library locations are accessible from the new OMS host (host "B")

2. Reinstall OMS on host "B" using the “Install Software only, configure later” – management service option. Location of the install need not be same as previous install.

3. Import the OMS configuration:
emctl importconfig oms -file <exportfile>

4. Resecure the Agent that gets installed along with OMS
emctl secure agent -emdWalletSrcUrl"http://slb:<httpport>/em"

5. Relocate the OMS and Repository target to reinstalled Agent:
emctl config emrep -agent <agent on hostB>

6. Locate duplicate targets from the Management Services and Repository Overview page. Relocate duplicate targets from the Agent on host "A" to the Agent on host "B". Delete the Agent on host "A".

7. Configure the SLB to include this new host in its configuration.
Comment: Is there an SLB note that covers this?

8. Verify that the site is fully operational.

Agent Backup and Recovery

The Agent is an integral software component that is deployed on each monitored host. It is responsible for monitoring all the targets running on those hosts, communicating that information to the middle-tier OMS and managing and maintaining the hosts and its targets.

Backing Up Agents

There are no special considerations for backing up Agents. As a best practice, reference Agent installs should be maintained for different platforms and kept up-to-date in terms of customizations in the emd.properties file and patches applied. Use Deployment options from the Grid Control console to install and maintain Reference Agent installs.

Recovering Agents

If an Agent is lost, it should be reinstalled by cloning from a reference install. Cloning from a reference install is often the fastest way to recover an Agent install as one does not have to track and reapply customizations and patches. Care should be taken to reinstall the Agent using the same port. Using the Enterprise Manager's Agent Resynchronization feature, a reinstalled Agent can be reconfigured using target information present in the repository. When the Agent is reinstalled using the same port, the OMS detects that it has been re-installed and blocks it temporarily to prevent the auto-discovered targets in the re-installed Agent from overwriting any customizations done previously.

Blocked Agents:

A blocked Agent is a condition where the OMS rejects all heartbeat or upload requests from the blocked Agent. Hence, a blocked Agent will not be able to upload any alerts or metric data to the OMS. However, blocked Agents continue to collect monitoring data.

The Agent can be resynchronized and unblocked from the Agent homepage by clicking on the Resynchronize Agent button. Resynchronization pushes all targets from the repository to the Agent and then unblocks the Agent.

Agent Recovery Scenarios

The following scenarios illustrate various Agent recovery situations along with the recovery steps. The Agent resynchronization feature requires that a reinstalled Agent use the same port as the previous Agent that crashed.

Agent reinstall, same port.

An Agent is monitoring multiple targets. The Agent install is lost.

1. Deinstall Agent OracleHome using the Oracle Universal Installer.

2. Install a new Agent or use the Agent clone option to reinstall the Agent though Enterprise Manager. Specify the same port as used by the crashed Agent. The location of install need not be same as previous install.
The OMS detects that Agent has been re-installed and blocks the Agent.

3. Initiate Agent Resynchronization from the Agent homepage.
All targets in the repository are pushed to the new Agent and Agent is unblocked.

4. Reconfigure User-defined Metrics if the location of User-defined Metric scripts have changed.

5. Verify that the Agent is operational and all target configurations have been restored.

Agent restore from filesystem backup

An Agent is monitoring multiple targets. File system backup for the Agent OracleHome exists. The Agent install is lost.

1. Deinstall Agent OracleHome using OUI.

2. Restore the Agent from file system backup. Start the Agent.
OMS detects that Agent has been restored from backup and blocks the Agent

3. Initiate Agent Resynchronization from the Agent homepage.
All targets in the repository are pushed to the new Agent and Agent is unblocked.

4. Verify that the Agent is functional and all target configurations have been restored using the following emctl commands:

emctl status agent

emctl upload agent

There should be no errors and no XML files in the backlog.

Recovering from a Compound OMS-Repository Failure

When both OMS and repository fail simultaneously, the recovery situation becomes more complex depending upon factors such as whether the OMS and repository are collocated, whether recovery has to be made on the same or different host, or whether there are multiple OMSs fronted by an SLB. In general, the order of recovery for this type of compound failure should be repository first followed by OMS(s) following the steps outlined in the appropriate recovery scenarios discussed earlier. The following scenarios illustrate two OMS-Repository failures and the requisite recovery steps.

Collapsed configuration, recovery on the same host, incomplete recovery of repository

Repository and the OMS are installed on same host (host "A"). No Server Load Balancer is configured. The repository database is running in noarchivelog mode. Full cold backup is available. Export of OMS configuration via emctl exportconfig oms is available. The repository, OMS and the Agent crash.

1. Recover the repository database using RMAN.

2. Since the OMS OracleHome is not available and repository resynchronization has to be initiated before starting an OMS against the restored repository, submit "resync" via the following PL/SQL block. Log into the repository as SYSMAN using SQLplus and run:
begin emd_maintenance.full_repository_resync('<resync name>'); end;

3. Deinstall the crashed OMS and Agent OracleHomes using OUI.

4. Reinstall the OMS using “Install Software only, configure later”. This is an additional management service option. Location of the install need not be same as previous install.

5. Import the OMS configuration:
emctl importconfig oms -file <exportfile>

Because the Agent that is installed with the OMS uses the same port as before, the OMS automatically blocks the Agent. At this point, you must "resync" the Agent from the Agent homepage.

6. Manually fix all pre-10.2.0.5 Agents by shutting down the Agent, deleting the agentstmp.txt, lastupld.xml, state/* and upload/* files under $AGENT_HOME/sysman/emd directory. Restart the Agent.

Note:

he Repository Synchronization function will automatically fix all 10.2.0.5 and later Agents.

7. Log into Enterprise Manager and navigate to the Management Services and Repository Overview page. Click on Repository Synchronization under Related Links. Monitor the status of resync jobs. Resubmit failed jobs, if any, after fixing the errors.

8. Verify that the site is fully operational.

Distributed configuration, Multi-OMS with SLB, recovery on different hosts, incomplete recovery of repository

Repository installed on host "X". Two OMSs are installed-- one on host "A" and one on host "B". OMSs are fronted by an SLB. Repository database running in noarchivelog mode. Full offline backup available. Host "X" and host "B" are lost.

1. Stop the OMS on host "A" using opmnctl stopall

2. Recover the database using RMAN on new host "Y"

3. Update the connect descriptor in the emoms.properties file on host "A" to point to host "Y".

4. Initiate Repository resynchronization:
emctl resync repos -full -name "<resync name>"

from the OMS Oracle home of host "A"

5. Run the command to relocate and reconfigure the repository database target:
emctl config repos -agent <agent on hostY> -oh <OH on hostY> -host <hostY> -conn_desc "<TNS connect descriptor>"

6. Export latest OMS configuration from host "A":
emctl exportconfig oms -dir <export location>

7. Ensure that shared loader receive directory and shared software library locations are accessible from the host "C".

8. Start the OMS on hostA using opmnctl startall

9. Reinstall the OMS on host "C" using the “Install Software only, configure later” – Additional management service option. Location of install need not be same as previous install.

10. Import the OMS configuration:
emctl importconfig oms -file <exportfile>

11. Update the SLB pools by replacing host "B" entries with host "C".

12. Resecure the Agent that gets installed along with OMS on host "C"
emctl secure agent -emdWalletSrcUrl "http://slb/:<httpport>/em"

13. Relocate and reconfigure the OMS and Repository target:
emctl config emrep -agent <agent on hostC> -conn_desc "<TNS connect descriptor>"

14. Locate duplicate targets from the Management Services and Repository Overview page. Relocate duplicate targets from Agent "B" to Agent "C". Delete the old Agent on host "B".

15. Manually fix all pre-10.2.0.5 Agents by shutting down the Agent, deleting the agentstmp.txt, lastupld.xml, state/* and upload/* files under $AGENT_HOME/sysman/emd directory. Restart the Agent.

16. Login to Grid Control and navigate to the Management Services and Repository Overview page. Click on Repository Synchronization under Related Links. Monitor the status of resync jobs. Resubmit failed jobs, if any , after fixing any errors mentioned.

17. Verify that the site is fully operational.

EMCTL High Availability Commands

The Enterprise Manager command-line utility (emctl) adds many new commands that allow you to perform requisite backup and recovery operations for all major components.

exportconfig oms

Exports a snapshot of the OMS configuration to the specified directory.

Usage:

emctl exportconfig oms [-sysman_pwd <sysman password>]

[-dir <backup dir>]     Specify the directory used to store the backup file

[-keep_host]            Specify to back up hostname if no SLB is defined

(Use this option only if recovery will be performed

on the machine that responds to this hostname)

importconfig oms

Imports the OMS configuration from the specified backup file.

Usage:

emctl importconfig oms [-sysman_pwd <sysman password>] [-reg_pwd <registration password>]

-file <backup file>     Required backup file to import from

[-key_only]             Specify to import emkey only

[-no_resecure]          Specify not to resecure the oms after import

(default is to resecure after import)

config emrep

Configures the OMS and repository target. The command is used to change the monitoring Agent for the target and/or the connection string used to monitor this target.

Usage:

emctl config emrep [-sysman_pwd <sysman password>]

[-agent <new agent>]    Specify a new destination Agent for emrep target

[-conn_desc [<jdbc connect descriptor>]]

Update the Connect Descriptor with value if specified,

else from value stored in the emoms.properties file.

config repos

Configures the repository database target. The command is used to change the monitoring Agent for the target and/or the monitoring properties (hostname, Oracle Home and connection string used to monitor this target).

Usage:

emctl config repos [-sysman_pwd <sysman password>]

[-agent <new agent>]    Specify new destination agent for repository target

[-host <new host>]      Specify new hostname for repository target

[-oh <new oracle home>] Specify new OracleHome for repository target

[-conn_desc [<jdbc connect descriptor>]]

Update the Connect Descriptor with the specified value,

else from the value stored in emoms.properties

resync repos

Submits a repository resynchronization operation. When the –full option is specified, all agents are instructed to upload the latest state to the repository. A list of agents can be specified using the –agentlist option to resync with a given list of agents.

Usage:

emctl resync repos (-full|-agentlist "agent names") [-name "resync name"] [-sysman_pwd "sysman password"]

abortresync repos

Aborts the currently running repository resynchronization operation. Use the –full option to stop a full repository resynchronization. Use the –agentlist option to stop resync on a list of agents.

Usage:

emctl abortresync repos (-full|-agentlist "agent names") -name "resync name" [-sysman_pwd "sysman password"]

statusresync repos

Lists the status of given repository resynchronization operation.

Usage:

emctl statusresync repos -name "resync name"

create service

Valid on Windows only. The command creates a service for the Oracle Management Services on Windows. You use this command to manage the Windows service for the OMS on a failover host in a Cold Failover Cluster setup.

Usage:

emctl create service [-user <username>] [-pwd <password>]

-name <servicename>     Name of service to be created

delete service

Valid on Windows only. Deletes the service for the Oracle Management Services on Windows.

Usage:

emctl delete service

-name <servicename>     Name of service to be deleted

resyncAgent

Resynchronizes a restored or reinstalled Agent by pushing all target configuration from the repository.

Usage:

emcli resyncAgent -agent="Agent Name"

[-keep_blocked]

10 Information Publisher

Information Publisher, Enterprise Manager's powerful reporting framework, makes information about your managed environment available to audiences across your enterprise. Strategically, reports are used to present a view of enterprise monitoring information for business intelligence purposes, but can also serve an administrative role by showing activity, resource utilization, and configuration of managed targets. IT managers can use reports to show availability of sets of managed systems. Executives can view reports on availability of applications (such as corporate email) over a period of time.

The reporting framework allows you to create and publish customized reports: Intuitive HTML-based reports can be published via the Web, stored, or e-mailed to selected recipients. Information Publisher comes with a comprehensive library of predefined reports that allow you to generate reports out-of-box without additional setup and configuration.

This chapter covers the following topics:

- About Information Publisher

- Out-of-Box Report Definitions

- Custom Reports

- Scheduling Reports

- Sharing Reports

About Information Publisher

Information Publisher provides powerful reporting and publishing capability. Information Publisher reports present an intuitive interface to critical decision-making information stored in the Management Repository while ensuring the security of this information by taking advantage of Enterprise Manager's security and access control.

Information Publisher's intuitive user-interface allows you to create and publish reports with little effort. The key benefits of using Information Publisher are:

- Provides a framework for creating content-rich, well-formatted HTML reports based on Management Repository data.

- Out-of-box reports let you start generating reports immediately without any system configuration or setup.

- Ability to schedule automatic generation of reports and store scheduled copies and/or e-mail them to intended audiences.

- Ability for Enterprise Manager administrators to share reports with the entire business community: executives, customers, and other Enterprise Manager administrators.

Information Publisher provides you with a feature-rich framework that is your central information source for your enterprise.

Out-of-Box Report Definitions

The focal point of Information Publisher is the report definition. A report definition tells the reporting framework how to generate a specific report by defining report properties such as report content, user access, and scheduling of report generation.

Information Publisher comes with a comprehensive library of predefined report definitions, allowing you to generate fully formatted HTML reports presenting critical operations and business information without any additional configuration or setup. Figure 10–1 shows an example of the Availability History (Group) report, displaying availability information for all members of a group.

Figure 10-1 Availability History (Group) Report

Description of "Figure 10-1 Availability History (Group) Report"

Generating this HTML report involved three simple steps:

Step 1: Click Availability History (Group) in the report definition list.

Step 2: Select the group for which you want to run the report.

Step 3: Click Continue to generate the fully-formed report.

Supplied report definitions are organized by functional category with each category covering key areas. The following table lists the major functional categories and areas covered by out-of-box reports.

Table 10-1 Predefined Report Definitions

Functional Category

Areas Covered

Deployment and Configuration

Client Configurations Hardware Operating System Linux Operating System Patching Application Server Configuration Oracle Application Server Software Oracle Database Configuration Oracle Database Software Oracle Home Patch Advisories Oracle Home Patch History

Enterprise Manager Setup

Management Pack Access

Monitoring

Alerts and Policy Violations

Availability History Dashboards Disabled Policies Service Alerts Service Performance and Usage Service Tests Web Application Page Performance Web Application Request Performance Web Application Transaction Performance

Security

Oracle Database Audit Oracle Database Privileges Security Policy Overview

Storage

Oracle Database Space Issues Oracle Database Space Usage

Custom Reports

Although the predefined report definitions that come with Information Publisher cover the most common reporting needs, you may want to create specialized reports. If a predefined report comes close to meeting your information requirements, but not quite, you can use Information Publisher's Create Like function to create a new report definition based on one of the existing reports definitions.

Creating Custom Reports

To create custom reports:

1. Choose whether to modify an existing report definition or start from scratch. If an existing report definition closely matches your needs, it is easy to customize it by using Create Like function.

2. Specify name, category, and sub-category. Grid Control provides default categories and sub-categories that are used for out-of-box reports. However, you can categorize custom reports in any way you like.

3. Specify any time-period and/or target parameters. The report viewer will be prompted for these parameters while viewing the report.

4. Add reporting elements. Reporting elements are pre-defined content building blocks, that allow you to add a variety of information to your report. Some examples of reporting elements are charts, tables, and images.

5. Customize the report layout. Once you have assembled the reporting elements, you can customize the layout of the report.

Report Parameters

By declaring report parameters, you allow the user to control what data is shown in the report. There are two types of parameters: target and time-period.

Example: If you are defining a report that will be used to diagnose a problem (such as a memory consumption report), the viewer will be able to see information for their target of interest.

By specifying the time-period parameter, the viewer will be able to analyze historical data for their period of interest.

Analyzing Historical Data

Information Publisher allows you to view reports for a variety of time-periods:

- Last 24 Hours/ 7 Days/ 31 Days

- Previous X Days/ Weeks/ Months/ Years (calendar units)

- This Week/ This Month/ This Year (this week so far)

- Any custom date range.

Report Elements

Report elements are the building blocks of a report definition. In general, report elements take parameters to generate viewable information. For example, the Chart from SQL element takes a SQL query to extract data from the Management Repository and a parameter specifying whether to display the data in the form of a pie, bar, or line chart. Report elements let you "assemble" a custom report definition using the Information Publisher user interface.

Information Publisher provides a variety of reporting elements. Generic reporting elements allow you to display any desired information, in the form of charts, tables or images. For example, you can include your corporate Logo, with a link to your corporate website. Monitoring elements show monitoring information, such as availability and alerts for managed targets. Service Level Reporting elements show availability, performance, usage and achieved service levels, allowing you to track compliance with Service Level Agreements, as well as share information about achieved service levels with your customers and business executives.

The following table lists the report elements that are supplied with Information Publisher.

Table 10-2 Report Elements

Report Element

Description

Generic Report Elements

Element Descriptions

Chart from SQL

Renders a line, pie or bar chart given a SQL or PL/SQL query.

Image Display

Displays a supplied image.

Separator

Displays a horizontal separator.

Styled Text

Displays text using a chosen style.

Table from SQL

Displays results of a SQL or PL/SQL query as a table.

Service Level Reporting Elements

Element Descriptions

Service Level Details

Displays Actual Service Level achieved over a time-period and violations that affected it.

Service Level Violation

Displays details on Service Level violations for a set of services over a given time-range.

Service Level Summary

Displays information on Service Levels over different time-ranges.

Services Monitoring Dashboard

Displays the Services Monitoring Dashboard showing status, performance, usage and Service-Level information for a set of Services.

Service Status Summary

Displays information on Services' Current status, Performance, Usage and Component Statuses.

Enterprise Manager Setup Element

Element Descriptions

Management Pack Access

Displays licensable targets with management pack access.

Monitoring Elements

Element Descriptions

Application Server Clusters

Displays monitoring and configuration information for Application Server Clusters.

Application Server Targets

Displays monitoring and configuration information for Application Server Targets.

Availability Timeline (Group)

Displays availability of targets over a period of time. Groups, Systems, Redundancy Groups and Clusters are supported.

Metric Details

Displays a graph of a given metric for a set of targets of the same time, over a given time-period.

Open Alerts

Displays details for outstanding alerts for a user-customizable set of targets and severities.

Oracle HTTP Traffic

Displays Oracle HTTP/HTTPS Traffic information.

Service Metric Details

Displays graphs of Performance and Usage metrics for a given service.

System Monitoring Dashboard

Displays status and alert information for a Group or System.

Web Application Page Performance

Renders page performance information for a given Web Application.

Web Application Page Performance By Category

Web Application Page Performance By Category Renders page performance information by domain/region/visitor/web server.

Web Application Request Performance

Displays details about request performance of a web application.

Web Application Transaction Performance Details

Displays transaction performance details for a given transaction.

Web Application Transaction Performance Summary

Displays summary information about transaction performance.

Web Application URL Performance

Displays time series chart showing the performance for a given URL.

Scheduling Reports

Enterprise manager allows you to view reports interactively and/or schedule generation of reports on a flexible schedule. For example, you might want to generate an "Inventory Snapshot" report of all of the servers in your environment every day at midnight.

Flexible Schedules

Grid Control provides the following scheduling options:

- One-time report generation either immediately or at any point in the future

- Periodic report generation

- Frequency: Any number of Minutes/ Hours/ Days/ Weeks/ Months/ Years

- You can generate copies indefinitely or until a specific date in the future.

Storing and Purging Report Copies

Enterprise manager allows you to store any number of scheduled copies for future reference. You can delete each stored copy manually or you can set up automated purging based on either the number of stored copies or based on retention time. For example, you can have Enterprise Manager purge all reports that are more than 90 days old.

E-mailing Reports

You can choose for scheduled reports to be e-mailed to any number of recipients. You can specify reply-to address and subject of the e-mail.

Sharing Reports

Information Publisher facilitates easy report sharing with the entire user community. Enterprise Manager administrators can share reports with other administrators and roles. However, there may be cases when you need to share reports with non-Enterprise Manager administrators, such as customers and/or business executives. To facilitate information sharing with these users, Enterprise Manager renders a separate reporting website that does not require user authentication.

Note:

To ensure that no sensitive information is compromised, only Enterprise Manager administrators with a special system privilege are allowed to publish reports to the Enterprise Manager reports website.

Information Publisher honors Enterprise Manager roles and privileges, ensuring that only Enterprise Manager administrators can create reports on the information they are allowed to see.When sharing reports, administrators have an option of allowing report viewers to see the report with the owner's privileges. For example, as a system administrator you might want to share a host's performance information with a DBA using your server, but you do not want to grant the DBA any privileges on your host target. In this case, you could create a host performance report, and allow the DBA to view it with your privileges. This way, they only see the information you want them to see, without having access to the host homepage.

11 Sizing Your Enterprise Manager Deployment

Oracle Enterprise Manager 10g Grid Control has the ability to scale for hundreds of users and thousands of systems and services on a single Enterprise Manager implementation.

This chapter describes techniques for achieving optimal performance using the Oracle Enterprise Manager application. It can also help you with capacity planning, sizing and maximizing Enterprise Manager performance in a large scale environment. By maintaining routine housekeeping and monitoring performance regularly, you insure that you will have the required data to make accurate forecasts of future sizing requirements. Receiving good baseline values for the Enterprise Manager Grid Control vital signs and setting reasonable warning and critical thresholds on baselines allows Enterprise Manager to monitor itself for you.

This chapter also provides practical approaches to backup, recovery, and disaster recovery topics while addressing different strategies when practical for each tier of Enterprise Manager.

This chapter contains the following sections:

- Oracle Enterprise Manager Grid Control Architecture Overview

- Enterprise Manager Grid Control Sizing and Performance Methodology

- Oracle Enterprise Manager Backup, Recovery, and Disaster Recovery Considerations

Oracle Enterprise Manager Grid Control Architecture Overview

The architecture for Oracle Enterprise Manager 10g Grid Control exemplifies two key concepts in application performance tuning: distribution and parallelization of processing. Each component of Grid Control can be configured to apply both these concepts.

The components of Enterprise Manager Grid Control include:

- The Management Agent - A process that is deployed on each monitored host and that is responsible for monitoring all services and components on the host. The Management Agent is also responsible for communicating that information to the middle-tier Management Service and for managing and maintaining the system and its services.

- The Management Service - A J2EE Web application that renders the user interface for the Grid Control Console, works with all Management Agents to process monitoring and jobs information, and uses the Management Repository as its data store.

- The Management Repository - The schema is an Oracle Database that contains all available information about administrators, services, and applications managed within Enterprise Manager.

Figure 11-1 Overview of Enterprise Manager Architecture Components

Description of "Figure 11-1 Overview of Enterprise Manager Architecture Components"

For more information about the Grid Control architecture, see the Oracle Enterprise Manager 10g documentation:

- Oracle Enterprise Manager Grid Control Installation and Basic Configuration

- Oracle Enterprise Manager Concepts

The Oracle Enterprise Manager 10g documentation is available at the following location on the Oracle Technology Network (OTN):

http://otn.oracle.com/documentation/oem.html

Enterprise Manager Grid Control Sizing and Performance Methodology

An accurate predictor of capacity at scale is the actual metric trend information from each individual Enterprise Manager Grid Control deployment. This information, combined with an established, rough, starting host system size and iterative tuning and maintenance, produces the most effective means of predicting capacity for your Enterprise Manager Grid Control deployment. It also assists in keeping your deployment performing at an optimal level.

Here are the steps to follow to enact the Enterprise Manager Grid Control sizing methodology:

1. If you have not already installed Enterprise Manager Grid Control 10g, choose a rough starting host configuration as listed in Table 11-1.

2. Periodically evaluate your site's vital signs (detailed later).

3. Eliminate bottlenecks using routine DBA/Enterprise Manager administration housekeeping.

4. Eliminate bottlenecks using tuning.

5. Extrapolate linearly into the future to plan for future sizing requirements.

Step one need only be done once for a given deployment. Steps two, three, and four must be done, regardless of whether you plan to grow your Enterprise Manager Grid Control site, for the life of the deployment on a regular basis. These steps are essential to an efficient Enterprise Manager Grid Control site regardless of its size or workload. You must complete steps two, three, and four before you continue on to step five. This is critical. Step five is only required if you intend to grow the deployment size in terms of monitored targets. However, evaluating these trends regularly can be helpful in evaluating any other changes to the deployment.

Step 1: Choosing a Starting Platform Grid Control Deployment

If you have not yet installed Enterprise Manager Grid Control on an initial platform, this step helps you choose a rough approximation based on experiences with real world Enterprise Manager Grid Control deployments. If you have already installed Enterprise Manager Grid Control, proceed to Step 2. Three typical deployment sizes are defined: small, medium, and large. The number and type of systems (or targets) it monitors largely defines the size of an Enterprise Manager Grid Control deployment.

Table 11-1 Management Server

Deployment Size

Hosts

CPUs/Hosts

Memory/Host (GB)

Small (100 monitored targets)

1

1 (3 GHz)

2

Medium (1,000 monitored targets)

1

2 (3 GHz)

2

Large (10,000 monitored targets)

2

2 (3 GHz) 2

2

Table 11-2 Management Repository

Deployment Size

Hosts

CPUs/Host

Memory/Host (GB)

Small

Shares host with Management Server

Shares host with Management Server

Shares host with Management Server

Medium

1

2

4

Large

2

4

6

Table 11-3 Total Management Repository Storage

Deployment Size

Minimum Tablespace Sizes*

SYSTEM**

MGMT_TABLESPACE

MGMT_ECM_DEPOT_TS

TEMP

*These are strictly minimum values and are intended as rough guidelines only. The actual size of the MGMT_TABLESPACE could vary widely from deployment to deployment due to variations in target type distribution, user customization, and several other factors. These tablespaces are defined with AUTOEXTEND set to ON by default to help mitigate space constraint issues. On raw file systems Oracle recommends using more than the minimum size to help prevent space constraint issues.
**The SYSTEM and TEMP tablespace sizes are minimums for Enterprise Manager only repositories. If Enterprise Manager is sharing the repository database with other application(s), these minimums may be too low.
Note: You cannot monitor tablespaces through the use of alerts with auto extended files in version 10g of Enterprise Manager. You can either set up TABLESPACE FULL alerts generate if you want to have greater control over the management of your tablespaces, or you can allow Oracle to grow your database and not alert you through the AUTOEXTEND feature. Therefore to exercise greater control of the TABLESPACE FULL alerts, you can turn off autoextend.

Small

600 MB

2 GB

1 GB

1 GB

Medium

600 MB

20 GB

1 GB

2 GB

Large

600 MB

200 GB

2 GB

4 GB

The previous tables show the estimated minimum hardware requirements for each deployment size. Management Servers running on more than one host, as portrayed in the large deployment above, will divide work amongst themselves.

Deploying multiple Management Servers also provides basic fail-over capabilities, with the remaining servers continuing to operate in the event of the failure of one. Use of a Server Load Balancer, or SLB, provides transparent failover for Enterprise Manager UI clients in the event of a Management Server host failure, and it also balances the request load between the available Management Servers. SLBs are host machines dedicated for load balancing purposes. SLBs can be clustered to provide fail-over capability.

Using multiple hosts for the Management Repository assumes the use of Oracle Real Application Clusters (RAC). Doing so allows the same Oracle database to be accessible on more than one host system. Beyond the storage required for the Management Server, Management Repository storage may also be required. Management Server storage is less impacted by the number of management targets. The numbers suggested in the Enterprise Manager Grid Control documentation should be sufficient in this regard.

Network Topology Considerations

A critical consideration when deploying Enterprise Manager Grid Control is network performance between tiers. Enterprise Manager Grid Control ensures tolerance of network glitches, failures, and outages between application tiers through error tolerance and recovery. The Management Agent in particular is able to handle a less performant or reliable network link to the Management Service without severe impact to the performance of Enterprise Manager as a whole. The scope of the impact, as far as a single Management Agent's data being delayed due to network issues, is not likely to be noticed at the Enterprise Manager Grid Control system wide level.

The impact of slightly higher network latencies between the Management Service and Management Repository will be substantial, however. Implementations of Enterprise Manager Grid Control have experienced significant performance issues when the network link between the Management Service and Management Repository is not of sufficient quality. The following diagram that displays the Enterprise Manager components and their connecting network link performance requirements. These are minimum requirements based on larger real world Enterprise Manager Grid Control deployments and testing.

Figure 11-2 Network Links Related to Enterprise Manager Components

You can see in Figure 11-2 that the bandwidth and latency minimum requirements of network links between Enterprise Manager Grid Control components greatly impact the performance of the Enterprise Manager application.

Step 2: Periodically Evaluate the Vital Signs of Your Site

This is the most important step of the five. Without some degree of monitoring and understanding of trends or dramatic changes in the vital signs of your Enterprise Manager Grid Control site, you are placing site performance at serious risk. Every monitored target sends data to the Management Repository for loading and aggregation through its associated Management Agent. This adds up to a considerable volume of activity that requires the same level of management and maintenance as any other enterprise application.

Enterprise Manager has "vital signs" that reflect its health. These vital signs should be monitored for trends over time as well as against established baseline thresholds. You must establish realistic baselines for the vital signs when performance is acceptable. Once baselines are established, you can use built-in Oracle Enterprise Manager Grid Control functionality to set baseline warning and critical thresholds. This allows you to be notified automatically when something significant changes on your Enterprise Manager site. The following table is a point-in-time snapshot of the Enterprise Manager Grid Control vital signs for two sites:

EM Site 1

EM Site 2

Site URL

emsite1.acme.com

emsite2.acme.com

Target Counts

Database Targets

192 (45 not up)

1218 (634 not up)

Host Targets

833 (12 not up)

1042 (236 not up)

Total Targets

2580 (306 not up)

12293 (6668 not up)

Loader Statistics

Loader Threads

6

16

Total Rows/Hour

1,692,000

2,736,000

Rows/hour/load/thread

282,000

171,000

Rows/second/load thread

475

187

Percent of Hour Run

15

44

Rollup Statistics

Rows per Second

2,267

417

Percent of Hour Run

5

19

Job Statistics

Job Dispatchers

2

4

Job Steps/second/dispatcher

32

10

Notification Statistics

Notifications per Second

8

1

Percent of Hour Run

1

13

Alert Statistics

Alerts per Hour

536

1,100

Management Service Host Statistics

Average % CPU (Host 1)

9 (emhost01)

13 (emhost01)

Average % CPU (Host 2)

6 (emhost02)

17 (emhost02)

Average % CPU (Host 3)

N/A

38 (em6003)

Average % CPU (Host 4)

N/A

12 (em6004)

Number of CPUs per host

2 X 2.8 (Xeon)

4 X 2.4 (Xeon)

Memory per Host (GB)

6

6

Management Repository Host Statistics

Average % CPU (Host 1)

12 (db01rac)

32 (em6001rac)

Average % CPU (Host 2)

Average % CPU (Host 3)

Average % CPU (Host 4)

Number of CPUs per host

Buffer Cache Size (MB)

Memory per Host (GB)

6

12

Total Management Repository Size (GB)

56

98

RAC Interconnect Traffic (MB/s)

1

4

Management Server Traffic (MB/s)

4

4

Total Management Repository I/O (MB/s)

6

27

Enterprise Manager UI Page Response/Sec

Home Page

3

6

All Host Page

3

30+

All Database Page

6

30+

Database Home Page

2

2

Host Home Page

2

2

The two Enterprise Manager sites are at the opposite ends of the scale for performance.

EM Site 1 is performing very well with high loader rows/sec/thread and high rollup rows/sec. It also has a very low percentage of hours run for the loader and the rollup. The CPU utilization on both the Management Server and Management Repository Server hosts are low. Most importantly, the UI Page Response times are excellent. To summarize, Site 1 is doing substantial work with minimal effort. This is how a well configured, tuned and maintained Oracle Enterprise Manager Grid Control site should look.

Conversely, EM Site 2 is having difficulty. The loader and rollup are working hard and not moving many rows. Worst of all are the UI page response times. There is clearly a bottleneck on Site 2, possibly more than one.

These vital signs are all available from within the Enterprise Manager interface. Most values can be found on the All Metrics page for each host, or the All Metrics page for Management Server. Keeping an eye on the trends over time for these vital signs, in addition to assigning thresholds for warning and critical alerts, allows you to maintain good performance and anticipate future resource needs. You should plan to monitor these vital signs as follows:

- Take a baseline measurement of the vital sign values seen in the previous table when the Enterprise Manager Grid Control site is running well.

- Set reasonable thresholds and notifications based on these baseline values so you can be notified automatically if they deviate substantially. This may require some iteration to fine-tune the thresholds for your site. Receiving too many notifications is not useful.

- On a daily (or weekly at a minimum) basis, watch for trends in the 7-day graphs for these values. This will not only help you spot impending trouble, but it will also allow you to plan for future resource needs.

The next step provides some guidance of what to do when the vital sign values are not within established thresholds. Also, it explains how to maintain your site's performance through routine housekeeping.

Step 3: Use DBA and Enterprise Manager Tasks To Eliminate Bottlenecks Through Housekeeping

It is critical to note that routine housekeeping helps keep your Enterprise Manager Grid Control site running well. The following are lists of housekeeping tasks and the interval on which they should be done.

Online Weekly Tasks

- Check the system error page and resolve the causes of all errors. Some may be related to product bugs, but resolve as many as you can. Look for applicable patches if you suspect a bug. Clear the error table from the Enterprise Manager interface when you are done or when you have resolved all that you can.

- Check the alerts and errors for any metric collection errors. Most of these will be due to configuration issues at the target being monitored. Resolve these errors by fixing the reported problem. The error should then clear automatically.

- Try to resolve any open alerts in the system. Also, if there are severities that are frequently oscillating between clear and warning or critical, try adjusting the threshold to stop frequent warning and critical alert conditions. Frequent alert oscillation can add significant load at the Management Server. Adjusting the threshold to a more reasonable level will help Enterprise Manager to work more efficiently for you. Adjusting the threshold for an alert may be the only way to close the alert. This is perfectly acceptable in cases where the tolerances are too tight for a metric.

- Watch for monitored targets that are always listed with a down status. Try to get them up and working again, or remove them from Oracle Enterprise Manager.

- Watch the Alert Log error metric for the Management Repository database for critical (ORA-0600, for example) errors. Resolve these as soon as possible. A search on Metalink using the error details almost always will reveal some clues to its cause and provide available patches.

- Analyze the three major tables in the Management Repository: MGMT_METRICS_RAW, MGMT_METRICS_1HOUR, and MGMT_METRICS_1DAY. If your Management Repository is in an Oracle 10g database, then these tables are automatically analyzed weekly and you can skip this task. If your Management Repository is in an Oracle version 9 database, then you will need to ensure that the following commands are run weekly:

- exec dbms_stats.gather_table_stats('SYSMAN', 'MGMT_METRICS_RAW', null, .000001, false, 'for all indexed columns', null, 'global', true, null, null, null);

- exec dbms_stats.gather_table_stats('SYSMAN', 'MGMT_METRICS_1HOUR', null, .000001, false, 'for all indexed columns', null, 'global', true, null, null, null);

- exec dbms_stats.gather_table_stats('SYSMAN', 'MGMT_METRICS_1DAY', null, .000001, false, 'for all indexed columns', null, 'global', true, null, null, null);

Offline Monthly Tasks

- Drop old partitions. Oracle Enterprise Manager automatically truncates the data and reclaims the space used by partitions older than the default retention times for each table. Enterprise Manager cannot drop partitions while the Management Service is running. Doing so may generate Enterprise Manager error messages due to SQL cursors being invalidated incorrectly. The following command must be run with all Management Servers down:

- exec EMD_MAINT_UTIL.PARTITION_MAINTENANCE;

- Rebuild and defragment indexes and reorganize tables as required. You may not actually need to rebuild any indexes or tables on a monthly basis. All you should do monthly is evaluate the Management Repository for tables and indexes that have grown significantly and been purged back down to a fraction of their allocated size. Unnecessarily building tables and indexes causes the Management Repository to work harder than necessary to reallocate needed space. The tables that require reorganization are easily identifiable. These tables will be large in allocated size with a relatively small number of rows, or actual size. In a real Management Repository, you may see one table that is approximately 800MB in size but only contains 6 rows. If the table is this badly oversized, it requires reorganization. Tables can be reorganized and rebuilt using commands similar to the following example:

- exec dbms_redefinition.start_redef_table('SYSMAN','MGMT_VIOLATIONS','temp_viol');

- CREATE TABLE temp_viol AS
SELECT * FROM mgmt_violations WHERE 1=2;

- exec dbms_redefinition.start_redef_table
('SYSMAN','MGMT_VIOLATIONS','temp_viol');

- DROP MATERIALIZED VIEW temp_viol;

These commands rebuild the table and returns its physical structure to its clean initial state. The 800 MB table is an extreme case. Smaller disparities between actual size and row count may also indicate the need for reorganization. The Management Server(s) must be down when reorganizing a table. If you reorganize the table, the indexes must also be rebuilt. This helps make index range scans more efficient again. Indexes can be reorganized using a command similar to the following example:

- ALTER INDEX SEVERITY_PRIMARY_KEY REBUILD;

There are a few tables (along with their indexes) that may require rebuilding more frequently than others based on the higher volume of inserts and deletes they typically see. These tables are:

- MGMT_VIOLATIONS

- MGMT_CURRENT_VIOLATION

- MGMT_SYSTEM_ERROR_LOG

- MGMT_SYSTEM_PERFORMANCE_LOG

- MGMT_METRIC_ERRORS

- MGMT_CURRENT_METRIC_ERRORS

- MGMT_STRING_METRIC_HISTORY

- MGMT_JOB_OUTPUT

These Management Repository tables can be prone to uneven growth patterns and can have large areas of allocated space that are unused due to infrequent, large spikes in data volume.

The best tool for determining whether reorganization is required is the DBMS_SPACE.SPACE_USAGE procedure. The procedure takes advantage of space bitmap structures in the locally managed tablespace MGMT_TABLESPACE to determine the fullness of blocks for segments. The following is an example script for this procedure:

variable unformatted_blocks number;

variable unformatted_bytes number;

variable freeBlocks_0_25 number;

variable freeBytes_0_25 number;

variable freeBlocks_25_50 number:

variable freeBytes_25_50 number;

variable freeBlocks_50_75 number;

variable freeBytes_50_75 number;

variable freeBlocks_75_100 number;

variable freeBytes_75_100 number;

variable full_blocks number;

variable full_bytes number;

begin

dbms_space.space_usage('SYSMAN','MGMT_JOB_OUTPUT','TABLE',

:unformatted_blocks,:unformatted_bytes,

:freeBlocks_0_25,:freeBytes_0_25,

:freeBlocks_25_50,:freeBytes_25_50,

:freeBlocks_50_75,:freeBytes_50_75,

:freeBlocks_75_100,:freeBytes_75_100,

:full_blocks,:full_bytes);

end;

/

print unformatted_blocks;

print freeBlocks_0_25;

print freeBlocks_25_50;

print freeBlocks_50_75;

print freeBlocks_75_100;

print full_blocks;

Beginning in version 10.2 of the Oracle RDBMS, you can take advantage of the Segment Advisor functionality to automatically examine SYSMAN segments. Segment Advisor identifies candidates for rebuild and then generates the rebuild statements.

DBMS_SPACE and Segment Advisor provide recommendations and results you can use as a guide in determing an effective course of action. However, there are times when these recommendations can and should be overridden. For example, if you know that segment free space is likely to be reused, do not reorganize a table that may have been identified for reorganization by one of the tools. It is important to keep in mind that these tools only examine the current state of space allocation for a segment. They cannot provide insight into the projected data volume of the Enterprise Manager Grid Control application at your site.

Good housekeeping will prevent many bottlenecks from occurring on your Enterprise Manager Grid Control site, but there may be times when you should investigate performance problems on your site that are not related to housekeeping. This is where the Enterprise Manager vital signs become important.

Step 4: Eliminate Bottlenecks Through Tuning

The most common causes of performance bottlenecks in the Enterprise Manager Grid Control application are listed below (in order of most to least common):

1. Housekeeping that is not being done (far and away the biggest source of performance problems)

2. Hardware or software that is incorrectly configured

3. Hardware resource exhaustion

When the vital signs are routinely outside of an established threshold, or are trending that way over time, you must address two areas. First, you must ensure that all previously listed housekeeping is up to date. Secondly, you must address resource utilization of the Enterprise Manager Grid Control application. The vital signs listed in the previous table reflect key points of resource utilization and throughput in Enterprise Manager Grid Control. The following sections cover some of the key vital signs along with possible options for dealing with vital signs that have crossed thresholds established from baseline values.

High CPU Utilization

When you are asked to evaluate a site for performance and notice high CPU utilization, there are a few common steps you should follow to determine what resources are being used and where.

1. The Management Server is typically a very minimal consumer of CPU. High CPU utilization in the Enterprise Manager Grid Control almost always manifests as a symptom at the Management Repository.

2. Use the Processes display on the Enterprise Manager Host home page to determine which processes are consuming the most CPU on any Management Service or Management Repository host that has crossed a CPU threshold.

3. Once you have established that Enterprise Manager is consuming the most CPU, use Enterprise Manager to identify what activity is the highest CPU consumer. Typically this manifests itself on a Management Repository host where most of the Management Service's work is performed. It is very rare that the Management Service itself is the source of the bottleneck. Here are a few typical spots to investigate when the Management Repository appears to be using too many resources.

1. Click on the CPU Used database resource listed on the Management Repository's Database Performance page to examine the SQL that is using the most CPU at the Management Repository.

2. Check the Database Locks on the Management Repository's Database Performance page looking for any contention issues.

High CPU utilization is probably the most common symptom of any performance bottleneck. Typically, the Management Repository is the biggest consumer of CPU, which is where you should focus. A properly configured and maintained Management Repository host system that is not otherwise hardware resource constrained should average roughly 40 percent or less total CPU utilization. A Management Server host system should average roughly 20 percent or less total CPU utilization. These relatively low average values should allow sufficient headroom for spikes in activity. Allowing for activity spikes helps keep your page performance more consistent over time. If your Enterprise Manager Grid Control site interface pages happen to be responding well (approximately 3 seconds) while there is no significant (constant) loader backlog, and it is using more CPU than recommended, you may not have to address it unless you are concerned it is part of a larger upward trend.

The recommended path for tracking down the root cause of high Management Repository CPU utilization is captured under step 3.b above. This allows you to start at the Management Repository Performance page and work your way down to the SQL that is consuming the most CPU in its processing. This approach has been used very successfully on several real world sites.

If you are running Enterprise Manager on Intel based hosts, the Enterprise Manager Grid Control Management Service and Management Repository will both benefit from Hyper-Threading (HT) being enabled on the host or hosts on which they are deployed. HT is a function of certain late models of Intel processors, which allows the execution of some amount of CPU instructions in parallel. This gives the appearance of double the number of CPUs physically available on the system. Testing has proven that HT provides approximately 1.5 times the CPU processing power as the same system without HT enabled. This can significantly improve system performance. The Management Service and Management Repository both frequently have more than one process executing simultaneously, so they can benefit greatly from HT.

Loader Vital Signs

The vital signs for the loader indicate exactly how much data is continuously coming into the system from all the Enterprise Manager Agents. The most important items here are the percent of hour runs and rows/second/thread. The (Loader) % of hour run indicates whether the loader threads configured are able to keep pace with the incoming data volume. As this value approaches 100%, it becomes apparent that the loading process is failing to keep pace with the incoming data volume. The lower this value, the more efficiently your loader is running and the less resources it requires from the Management Service host. Adding more loader threads to your Management Server can help reduce the percent of hour run for the loader.

Rows/Second/Thread is a precise measure of each loader thread's throughput per second. The higher this number, the better. Rows/Second/Thread as high as 1200 have been observed on some smaller, well configured and maintained Enterprise Manager Grid Control sites. If you have not increased the number of loader threads and this number is trending down, it may indicate a problem later. One way to overcome a decreasing rows/second/thread is to add more loader threads.

The number of Loader Threads is always set to one by default in the Management Server configuration file. Each Management Server can have a maximum of 10 loader threads. Adding loader threads to a Management Server typically increases the overall host CPU utilization by 2% to 5% on a Enterprise Manager Grid Control site with many Management Agents configured. Customers can change this value as their site requires. Most medium size and smaller configurations will never need more than one loader thread. Here is a simple guideline for adding loader threads:

Max total (across all Management Servers) loader threads = 2 X number of Management Repository host CPUs

There is a diminishing return when adding loader threads. You will not yield 100% capacity from the second, or higher, thread. There should be a positive benefit, however. As you add loader threads, you should see rows/second/thread decrease, but total rows/hour throughput should increase. If you are not seeing significant improvement in total rows/hour, and there is a constantly growing loader file backlog, it may not be worth the cost of the increase in loader threads. You should explore other tuning or housekeeping opportunities in this case.

To add more loader threads you can change the following configuration parameter:

em.loader.threadPoolSize=n

Where 'n' is a positive integer [1-10]. The default is one and any value other than [1-10] will result in the thread pool size defaulting to one. This property file is located in the {ORACLE_HOME}/sysman/config directory. Changing this parameter will require a restart of the Management Service to be reloaded with the new value.

Rollup Vital Signs

The rollup process is the aggregation mechanism for Enterprise Manager Grid Control. Once an hour, it processes all the new raw data loaded into the Management Repository table MGMT_METRICS_RAW, calculates averages and stores them in the tables MGMT_METRICS_1HOUR and MGMT_METRICS_1DAY. The two vital signs for the rollup are the rows/second and % of hour run. Due to the large volume of data rows processed by the rollup, it tends to be the largest consumer of Management Repository buffer cache space. Because of this, the rollup vital signs can be great indicators of the benefit of increasing buffer cache size.

Rollup rows/second shows exactly how many rows are being processed, or aggregated and stored, every second. This value is usually around 2,000 (+/- 500) rows per second on a site with a decent size buffer cache and reasonable speedy I/O. A downward trend over time for this value may indicate a future problem, but as long as % of hour run is under 100 your site is probably fine.

If rollup % of hour run is trending up (or is higher than your baseline), and you have not yet set the Management Repository buffer cache to its maximum, it may be advantageous to increase the buffer cache setting. Usually, if there is going to be a benefit from increasing buffer cache, you will see an overall improvement in resource utilization and throughput on the Management Repository host. The loader statistics will appear a little better. CPU utilization on the host will be reduced and I/O will decrease. The most telling improvement will be in the rollup statistics. There should be a noticeable improvement in both rollup rows/second and % of hour run. If you do not see any improvement in any of these vital signs, you can revert the buffer cache to its previous size. The old Buffer Cache Hit Ratio metric can be misleading. It has been observed in testing that Buffer Cache Hit Ratio will appear high when the buffer cache is significantly undersized and Enterprise Manager Grid Control performance is struggling because of it. There will be times when increasing buffer cache will not help improve performance for Grid Control. This is typically due to resource constraints or contention elsewhere in the application. Consider using the steps listed in the High CPU Utilization section to identify the point of contention. Grid Control also provides advice on buffer cache sizing from the database itself. This is available on the database Memory Parameters page.

One important thing to note when considering increasing buffer cache is that there may be operating system mechanisms that can help improve Enterprise Manager Grid Control performance. One example of this is the "large memory" option available on Red Hat Linux. The Linux OS Red Hat Advanced Server™ 2.1 (RHAS) has a feature called big pages. In RHAS 2.1, bigpages is a boot up parameter that can be used to pre-allocate large shared memory segments. Use of this feature, in conjunction with a large Management Repository SGA, can significantly improve overall Grid Control application performance. Starting in Red Hat Enterprise Linux™ 3, big pages functionality is replaced with a new feature called huge pages, which no longer requires a boot-up parameter.

Job, Notification, and Alert Vital Signs

Jobs, notifications, and alerts are indicators of the processing efficiency of the Management Service(s) on your Enterprise Manager Grid Control site. Any negative trends in these values are usually a symptom of contention elsewhere in the application. The best use of these values is to measure the benefit of running with more than one Management Server. There is one job dispatcher in each Management Server. Adding Management Servers will not always improve these values. In general, adding Management Servers will improve overall throughput for Grid Control when the application is not otherwise experiencing resource contention issues. Job, Notification, and Alert vital signs can help measure that improvement.

I/O Vital Signs

Monitoring the I/O throughput of the different channels in your Enterprise Manager Grid Control deployment is essential to ensuring good performance. At minimum, there are three different I/O channels on which you should have a baseline and alert thresholds defined:

- Disk I/O from the Management Repository instance to its data files

- Network I/O between the Management Server and Management Repository

- RAC interconnect (network) I/O (on RAC systems only)

You should understand the potential peak and sustained throughput I/O capabilities for each of these channels. Based on these and the baseline values you establish, you can derive reasonable thresholds for warning and critical alerts on them in Grid Control. You will then be notified automatically if you approach these thresholds on your site. Some Grid Control site administrators can be unaware or mistaken about what these I/O channels can handle on their sites. This can lead to Enterprise Manager Grid Control saturating these channels, which in turn cripples performance on the site. In such an unfortunate situation, you would see that many vital signs would be impacted negatively.

To discover whether the Management Repository is involved, you can use Grid Control to check the Database Performance page. On the Performance page for the Management Repository, click on the wait graph showing the largest amount of time spent. From this you can continue to drill down into the actual SQL code or sessions that are waiting. This should help you to understand where the bottleneck is originating.

Another area to check is unexpected I/O load from non-Enterprise Manager Grid Control sources like backups, another application, or a possible data-mining co-worker who engages in complex SQL queries, multiple Cartesian products, and so on.

Total Repository I/O trouble can be caused by two factors. The first is a lack of regular housekeeping. Some of the Grid Control segments can be very badly fragmented causing a severe I/O drain. Second, there can be some poorly tuned SQL statements consuming much of the site I/O bandwidth. These two main contributors can cause most of the Grid Control vital signs to plummet. In addition, the lax housekeeping can cause the Management Repository's allocated size to increase dramatically.

One important feature of which to take advantage is asynchronous I/O. Enabling asynchronous I/O can dramatically improve overall performance of the Grid Control application. The Sun Solaris™ and Linux operating systems have this capability, but may be disabled by default. The Microsoft Windows™ operating system uses asynchronous I/O by default. Oracle strongly recommends enabling of this operating system feature on the Management Repository hosts and on Management Service hosts as well.

The Oracle Enterprise Manager Performance Page

There may be occasions when Enterprise Manager user interface pages are slow in the absence of any other performance degradation. The typical cause for these slow downs will be an area of Enterprise Manager housekeeping that has been overlooked. The first line of monitoring for Enterprise Manger page performance is the use of Enterprise Manager Beacons. These functionalities are also useful for web applications other than Enterprise Manager.

Beacons are designed to be lightweight page performance monitoring targets. After defining a Beacon target on an Management Agent, you can then define UI performance transactions using the Beacon. These transactions are a series of UI page hits that you will manually walk through once. Thereafter, the Beacon will automatically repeat your UI transaction on a specified interval. Each time the Beacon transaction is run, Enterprise Manager will calculate its performance and store it for historical purposes. In addition, alerts can be generated when page performance degrades below thresholds you specify.

When you configure the Enterprise Manager Beacon, you begin with a single predefined transaction that monitors the home page you specify during this process. You can then add as many transactions as are appropriate. You can also set up additional Beacons from different points on your network against the same web application to measure the impact of WAN latency on application performance. This same functionality is available for all Web applications monitored by Enterprise Manager Grid Control.

After you are alerted to a UI page that is performing poorly, you can then use the second line of page performance monitoring in Enterprise Manager Grid Control. This new end-to-end (or E2E) monitoring functionality in Grid Control is designed to allow you to break down processing time of a page into its basic parts. This will allow you to pinpoint when maintenance may be required to enhance page performance. E2E monitoring in Grid Control lets you break down both the client side processing and the server side processing of a single page hit.

The next page down in the Middle Tier Performance section will break out the processing time by tier for the page. By clicking on the largest slice of the Processing Time Breakdown pie chart, which is JDBC time above, you can get the SQL details. By clicking on the SQL statement, you break out the performance of its execution over time.

The JDBC page displays the SQL calls the system is spending most of its page time executing. This SQL call could be an individual DML statement or a PL/SQL procedure call. In the case of an individual SQL statement, you should examine the segments (tables and their indexes) accessed by the statement to determine their housekeeping (rebuild and reorg) needs. The PL/SQL procedure case is slightly more involved because you must look at the procedure's source code in the Management Repository to identify the tables and associated indexes accessed by the call.

Once you have identified the segments, you can then run the necessary rebuild and reorganization statements for them with the Management Server down. This should dramatically improve page performance. There are cases where page performance will not be helped by rebuild and reorganization alone, such as when excessive numbers of open alerts, system errors, and metric errors exist. The only way to improve these calls is to address (for example, clean up or remove) the numbers of these issues. After these numbers are reduced, then the segment rebuild and reorganization should be completed to optimize performance. These scenarios are covered in Step 3: Use DBA and Enterprise Manager Tasks To Eliminate Bottlenecks Through Housekeeping. If you stay current, you should not need to analyze UI page performance as often, if at all.

Step 5: Extrapolating Linearly Into the Future for Sizing Requirements

Determining future storage requirements is an excellent example of effectively using vital sign trends. You can use two built-in Grid Control charts to forecast this: the total number of targets over time and the Management Repository size over time.

Both of the graphs are available on the All Metrics page for the Management Service. It should be obvious that there is a correlation between the two graphs. A straight line applied to both curves would reveal a fairly similar growth rate. After a target is added to Enterprise Manager Grid Control for monitoring, there is a 31-day period where Management Repository growth will be seen because most of the data that will consume Management Repository space for a target requires approximately 31 days to be fully represented in the Management Repository. A small amount of growth will continue for that target for the next year because that is the longest default data retention time at the highest level of data aggregation. This should be negligible compared with the growth over the first 31 days.

When you stop adding targets, the graphs will level off in about 31 days. When the graphs level off, you should see a correlation between the number of targets added and the amount of additional space used in the Management Repository. Tracking these values from early on in your Enterprise Manager Grid Control deployment process helps you to manage your site's storage capacity proactively. This history is an invaluable tool.

The same type of correlation can be made between CPU utilization and total targets to determine those requirements. There is a more immediate leveling off of CPU utilization as targets are added. There should be no significant increase in CPU cost over time after adding the targets beyond the relatively immediate increase. Introducing new monitoring to existing targets, whether new metrics or increased collections, would most likely lead to increased CPU utilization.

Oracle Enterprise Manager Backup, Recovery, and Disaster Recovery Considerations

Enterprise Manager incorporates a portable browser-based interface to the Grid Control console, as well as the Oracle application server technology, to serve as the middle-tier Management Service tool. The foundation of the tool remains rooted in database server technology to manage both the Management Repository and historical data. This section provides practical approaches to these high availability topics and discusses different strategies when practical for each tier of Enterprise Manager.

Best Practices for Backup

For the Oracle database, the best backup practice is to use the standard database tools and do the following:

1. Have the database in archivelog mode

2. Perform regular online backups using the Oracle Suggested Backup strategy option available through Grid Control. This strategy uses Recovery Manager (RMAN).

This strategy creates a full backup and then creates incremental backups on each subsequent run. The incremental changes are then rolled up into the baseline, creating a new full backup baseline.

Using the Oracle Suggested Backup strategy also takes advantage of the capabilities of Grid Control to execute the backups. Backup jobs are automatically scheduled through the Grid Control Job subsystem. The history of the backups is available for review and the status of the backup displays in the Job Activity section of the database target's home page.

Use of this job along with archiving and flashback technologies provides a restore point in the event of the loss of any part of the Management Repository. This backup, along with archive and online logs, allows the Management Repository to be recovered to the last completed transaction.

To enable archiving and flashback technologies, use the Recovery Settings page and enable:

1. Archive Logging
Bounce the database and restart all Management Service processes

2. Flashback Database
Bounce the database and restart all Management Service processes

3. Block Change Tracking feature to speed up backup operations.

A summary of how to configure backups using Enterprise Manager is available in the Oracle Database 2 Day DBA manual.

For additional information on database high availability best practices, review the Oracle Database High Availability Architecture and Best Practices manual.

You can set the frequency of the backup job depending on how much data is generated in the Grid Control environment and how much outage time you can tolerate if a restore is required. If the outage window is small and the Service Level Agreement can not be satisfied by restoring the database, consider additional strategies for Management Repository availability such a Real Application Cluster (RAC) or Data Guard database. Additional High Availability options for the Management Repository are documented in the Configuring Enterprise Manager for High Availability paper available from the Maximum Availability Architecture (MAA) page on the Oracle Technology Network (OTN) at http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm

Best Practices for Recovery

For the Oracle Database, the best practice for recovery is to be prepared. Because in some situations the Management Repository, Management Service, and Management Agents will not have access to Grid Control, you will need to use the command-line interface to enter the RMAN commands.

Recovering the Management Repository

If something happens to affect the Management Repository, Grid Control will not be available to provide the management interface to RMAN.

A sample syntax for database recovery using RMAN follows. For detailed information, review the information on database recovery in the Oracle Database Backup and Recovery Advanced User's Guide.

RMAN> STARTUP MOUNT;

RMAN> RESTORE DATABASE;

RMAN> RECOVER DATABASE;

RMAN> ALTER DATABASE OPEN;

When considering recovery of the Management Repository, there are two cases to consider:

- Full recovery of the Management Repository is possible
There are no special considerations for Enterprise Manager. When the database is recovered, restart the database and Management Service processes. Management Agents will then upload pending files to the Management Repository.

- Only point in time and incomplete recovery is possible
Management Agents will be unable to communicate to the Management Repository correctly until they are reset. You must perform the following steps manually:

1. Shut down the Management Agent

2. Delete the agntstmp.txt and lastupld.xml files in the $AGENT_HOME/sysman/emd directories

3. Go the /state and /upload subdirectories and clear their contents

4. Restart the Management Agent.

You must repeat these steps for each Management Agent.

In the case of incomplete recovery, Management Agents may not be able to upload data until the previous steps are completed. Additionally, there is no immediate indication in the interface that the Management Agents cannot communicate with the Management Service after this type of recovery. This information would be available from the Management Agent logs or command line Management Agent status. If incomplete recovery is required, you must perform this procedure for each Management Agent.

Recovering the Oracle Management Service

Because the Management Service is stateless, the task is to restore the binaries and configuration files in the shortest time possible. There are two alternatives in this case.

- Backup the entire software directory structure. You can restore the directory structure to the same directory path should a Management Service failure occur. At the same time, backup the Management Agent associated with this Management Service install. You will need to restore this Management Agent should a restore of the Management Service be required.

- Reinstall from the original media.

For any highly available Management Service install, it is a recommended practice that you ensure that the /recv directory is protected with a mirroring technology. The /recv directory is the directory the Management Service uses to stage files it receives from Management Agents before writing their contents to the database Management Repository.

After the Management Agent finishes transmitting its XML files to the Management Service, the Management Agent deletes its copy of the XML files. Metric data sent from the Management Agents would then be lost.

Recovering the Oracle Management Agent

The recovery of the Management Agent is similar to the Management Service recovery except that the Management Agent is not stateless. There are two strategies that can be used:

- If the host name has changed, and you are using an SLB to manage connections, you have to modify the connection pools in the SLB to drop the old host name and add the new name. If you are not using an SLB, each agent that previously pointed to the old OMS host must have its emd. properties file modified to point to the new OMS host name. You can use this procedure to handle a case where you need to bring up a new OMS on a new host because the former machine has crashed.
Assuming the host name has not changed, a disk backup and restore is sufficient.

1. Delete the agntstmp.txt and the lastupld.xml files from the /sysman/emd directory.

2. Clear the /state and /upload subdirectories of all entries before restarting the Management Agent.

3. Start the Management Agent. This step forces a rediscovery of the targets on the host.

- Reinstall the Management Agent from the original media.

As with the Management Service, it is recommended you protect the /state and /upload directories with a mirroring technology.

Best Practice for Disaster Recovery (DR)

In the event of a node failure, you can restore the database using RMAN or OS commands. To speed this process, implement Data Guard to replicate the Management Repository to a different hardware node.

Management Repository

If you are restoring the Management Repository to a new host, restore a backup of the database and modify the emoms.properties file for each Management Service manually to point to the new Management Repository location. In addition, you must update the targets.xml file for each Management Service to reflect the new Management Repository location. If there is a data loss during recovery, see Recovering the Management Repository for information.

To speed Management Repository reconnection from the Management Service in the event of a single Management Service failure, configure the Management Service with a Transparent Application Failover (TAF) aware connect string. You can configure the Management Service with a TAF connect string in the emoms.properities file that will automatically redirect communications to another node using the FAILOVER syntax. An example follows:

EM=

(description=

(failover=on)

(address_list=

(failover=on)

(address=(protocol=tcp)(port=1522)(host=EMPRIM1.us.oracle.com))

(address=(protocol=tcp)(port=1522)(host=EMPRIM2.us.oracle.com)))

(address_list=

(failover=on)

(address=(protocol=tcp)(port=1522)(host=EMSEC1.us.oracle.com))

(address=(protocol=tcp)(port=1522)(host=EMSEC2.us.oracle.com)))

(connect_data=(service_name=EMrep.us.oracle.com)))

Oracle Management Service

Preinstall the Management Service and Management Agent on the hardware that will be used for Disaster Recovery. This eliminates the step of restoring a copy of the Enterprise Manager binary files from backup and modifying the Management Service and Management Agent configuration files.

Note:

In the event of a disaster, do not restore the Management Service and Management Agent binaries from an existing backup to a new host because there are host name dependencies. Always do a fresh install.

Management Agent

In the event of a true disaster recovery, it is easier to reinstall the Management Agent and allow it to do a clean discovery of all targets running on the new host.

12 Maintaining and Troubleshooting the Management Repository

This chapter describes maintenance and troubleshooting techniques for maintaining a well-performing Management Repository.

Specifically, this chapter contains the following sections:

- Management Repository Deployment Guidelines

- Management Repository Data Retention Policies

- Changing the SYSMAN Password

- Dropping and Recreating the Management Repository

- Troubleshooting Management Repository Creation Errors

- Cross Platform Enterprise Manager Repository Migration

- Improving the Login Performance of the Console Home Page

Management Repository Deployment Guidelines

To be sure that your management data is secure, reliable, and always available, consider the following settings and configuration guidelines when you are deploying the Management Repository:

- Install a RAID-capable Logical Volume Manager (LVM) or hardware RAID on the system where the Management Repository resides. At a minimum the operating system must support disk mirroring and stripping. Configure all the Management Repository data files with some redundant configuration.

- Use Real Application Clusters to provide the highest levels of availability for the Management Repository.

- If you use Enterprise Manager to alert administrators of errors or availability issues in a production environment, be sure that the Grid Control components are configured with the same level of availability. At a minimum, consider using Oracle Data Guard to mirror the Management Repository database. Configure the Data Guard environment for no data loss.

See Also:

Oracle Database High Availability Architecture and Best Practices
Oracle Data Guard Concepts and Administration

- Oracle strongly recommends that archive logging be turned on and that a comprehensive backup strategy be in place prior to an Enterprise Manager implementation going live in a production environment. The backup strategy should include both incremental and full backups as required.

See Also:

Oracle Enterprise Manager Grid Control Installation and Basic Configuration for information about the database initialization parameters required for the Management Repository

Management Repository Data Retention Policies

When the various components of Enterprise Manager are configured and running efficiently, the Oracle Management Service gathers large amounts of raw data from the Management Agents running on your managed hosts and loads that data into the Management Repository. This data is the raw information that is later aggregated, organized, and presented to you in the Grid Control Console.

After the Oracle Management Service loads information into the Management Repository, Enterprise Manager aggregates and purges the data over time.

The following sections describe:

- The default aggregation and purging policies used to maintain data in the Management Repository.

- How you can modify the length of time the data is retained before it is aggregated and then purged from the Management Repository.

Management Repository Default Aggregation and Purging Policies

Enterprise Manager aggregates your management data by hour and by day to minimize the size of the Management Repository. Before the data is aggregated, each data point is stored in a raw data table. Raw data is rolled up, or aggregated, into a one-hour aggregated metric table. One-hour records are then rolled up into a one-day table.

After Enterprise Manager aggregates the data, the data is then considered eligible for purging. A certain period of time has to pass for data to actually be purged. This period of time is called the retention time.

The raw data, with the highest insert volume, has the shortest default retention time, which is set to 7 days. As a result, 7 days after it is aggregated into a one-hour record, a raw data point is eligible for purging.

One-hour aggregate data records are purged 31 days after they are rolled up to the one-day data table. The highest level of aggregation, one day, is kept for 365 days.

The default data retention policies are summarized in Table 12-1.

Table 12-1 Default Repository Purging Policies

Aggregate Level

Retention Time

Raw metric data

7 days

One-hour aggregated metric data

31 days

One-day aggregated metric data

365 days

If you have configured and enabled Application Performance Management, Enterprise Manager also gathers, saves, aggregates, and purges response time data. The response time data is purged using policies similar to those used for metric data. The Application Performance Management purging policies are shown in Table 12-2.

Table 12-2 Default Repository Purging Policies for Application Performance Management Data

Aggregate Level

Retention Time

Raw response time data

24 hours

One-hour aggregated response time data

7 days

One-hour distribution response time data

24 hours

One-day aggregated response time data

31 days

One-day distribution aggregated response time data

31 days

Management Repository Default Aggregation and Purging Policies for Other Management Data

Besides the metric data and Application Performance Monitoring data, other types of Enterprise Manager data accumulates over time in the Management Repository.

For example, the last availability record for a target will also remain in the Management Repository indefinitely, so the last known state of a target is preserved.

Modifying the Default Aggregation and Purging Policies

The Enterprise Manager default aggregation and purging policies were designed to provide the most available data for analysis while still providing the best performance and disk-space requirements for the Management Repository. As a result, you should not modify these policies to improve performance or increase your available disk space. Modifying these default policies can affect the performance of the Management Repository and have adverse reactions on the scalability of your Enterprise Manager installation.

However, if you plan to extract or review the raw or aggregated data using data analysis tools other than Enterprise Manager, you may want to increase the amount of raw or aggregated data available in the Management Repository. You can accomplish this by increasing the retention times for the raw or aggregated data.

To modify the default retention time for each level of management data in the Management Repository, you must insert additional rows into the MGMT_PARAMETERS table in the Management Repository database. Table 12-3 shows the parameters you must insert into the MGMT_PARAMETERS table to modify the retention time for each of the raw data and aggregate data tables.

Table names that contain "_RT_" indicate tables used for Application Performance Monitoring response time data. In the Table Name column, replace datatype with one of the three response time data types: DOMAIN, IP, or URL.

Table 12-3 Parameters for Modifying Default Data Retention Times in the Management Repository

Table Name

Parameter in MGMT_PARAMETERS Table

Default Retention Value

MGMT_METRICS_RAW

mgmt_raw_keep_window

7 days

MGMT_METRICS_1HOUR

mgmt_hour_keep_window

31 days

MGMT_METRICS_1DAY

mgmt_day_keep_window

365 days

MGMT_RT_METRICS_RAW

mgmt_rt_keep_window

24 hours

MGMT_RT_datatype_1HOUR

mgmt_rt_hour_keep_window

7 days

MGMT_RT_datatype_1DAY

mgmt_rt_day_keep_window

31 days

MGMT_RT_datatype_DIST_1HOUR

mgmt_rt_dist_hour_keep_window

24 hours

MGMT_RT_datatype_DIST_1DAY

mgmt_rt_dist_day_keep_window

31 days

Note:

If the first three tables listed in Table 8-3 are not partitioned, the Default Retention Value for each is 1, 7, and 31 days respectively, rather than the 7, 31, and 365 days listed for partitioned tables.

For example, to change the default retention time for the table MGMT_METRICS_RAW from seven days to 14 days:

1. Use SQL*Plus to connect to the Management Repository database as the Management Repository user.
The default Management Repository user is sysman.

2. Enter the following SQL to insert the parameter and change the default value:

INSERT INTO MGMT_PARAMETERS (PARAMETER_NAME, PARAMETER_VALUE)  VALUES ('mgmt_raw_keep_window','14');

Similarly, to change from the default retention time for all of the MGMT_RT_datatype_1DAY tables from 31 days to 100 days:

1. Use SQL*Plus to connect to the Management Repository database as the Management Repository user.
The default Management Repository user is sysman.

2. Enter the following SQL to insert the parameter and change the default value:

INSERT INTO MGMT_PARAMETERS (PARAMETER_NAME, PARAMETER_VALUE)  VALUES ('mgmt_rt_day_keep_window', '100');

Modifying Data Retention Policies When Targets Are Deleted

By default, when you delete a target from the Grid Control console, Enterprise Manager automatically deletes all target data from the Management Repository.

However, deleting raw and aggregated metric data for database and other data-rich targets is a resource consuming operation. Targets can have hundreds of thousands of rows of data and the act of deleting this data can degrade performance of Enterprise Manager for the duration of the deletion, especially when several targets are deleted at once.To avoid this resource-consuming operation, you can prevent Enterprise Manager from performing this task each time you delete a target. When you prevent Enterprise Manager from performing this task, the metric data for deleted targets is not purged as part of target deletion task; instead, it is purged as part of the regular purge mechanism, which is more efficient.

In addition, Oracle strongly recommends that you do not add new targets with the same name and type as the deleted targets within 24 hours of target deletion. Adding a new target with the same name and type will result in the Grid Control console showing data belonging to the deleted target for the first 24 hours.

To disable raw metric data deletion:

1. Use SQL*Plus to connect to the Management Repository as the Management Repository user.
The default Management Repository user is SYSMAN. For example:

SQL> connect sysman/oldpassword;

2. To disable metric deletion, run the following SQL command.

SQL> EXEC MGMT_ADMIN.DISABLE_METRIC_DELETION();
SQL> COMMIT;

To enable metric deletion at a later point, run the following SQL command:

1. Use SQL*Plus to connect to the Management Repository as the Management Repository user.
The default Management Repository user is SYSMAN. For example:

SQL> connect sysman/oldpassword;

2. To enable metric deletion, run the following SQL command.

SQL> EXEC MGMT_ADMIN.ENABLE_METRIC_DELETION();
SQL> COMMIT;

How to Modify the Retention Period of Job History

Enterprise Manager Grid Control has a default purge policy which removes all finished job details which are older than 30 days. This section provides details for modifying this default purge policy.

The actual purging of completed job history is implemented via a DBMS job that runs once a day in the repository database. When the job runs, it looks for finished jobs that are 'n' number of days older than the current time (value of sysdate in the repository database) and deletes these jobs. The value of 'n' is, by default, set to 30 days.

The default purge policy cannot be modified via the Enterprise Manager console, but it can be changed using SQL*Plus.

To modify this purge policy, follow these steps:

1. Log in to the repository database as the SYSMAN user, via SQL*Plus

2. Check the current values for the purge policies using the following command:
SQL> select * from mgmt_job_purge_policies;

POLICY_NAME                      TIME_FRAME
-------------------------------- ----------
SYSPURGE_POLICY                          30
REFRESHFROMMETALINKPURGEPOLICY            7
FIXINVENTORYPURGEPOLICY                   7
OPATCHPATCHUPDATE_PAPURGEPOLICY           7

The purge policy responsible for the job deletion is called SYSPURGE_POLICY. As seen above, the default value is set to 30 days.

3. To change the time period, you must drop and re-create the policy with a different time frame:
SQL> execute MGMT_JOBS.drop_purge_policy('SYSPURGE_POLICY');

PL/SQL procedure successfully completed.

SQL> execute MGMT_JOBS.register_purge_policy('SYSPURGE_POLICY', 60, null);

PL/SQL procedure successfully completed.

SQL> COMMIT;

Commit complete.

SQL> select * from mgmt_job_purge_policies;

POLICY_NAME                      TIME_FRAME
-------------------------------- ----------
SYSPURGE_POLICY                          60
....

The above commands increase the retention period to 60 days. The timeframe can also be reduced below 30 days, depending on the requirement.

You can check when the purge job will be executed next. The actual time that the job runs may vary with each Enterprise Manager installation. To determine this time in your setup follow these steps:

1. Login to the Repository database using the SYSMAN account

2. Execute the following command:
SQL> alter session set nls_date_format='mm/dd/yy hh:mi:ss pm';

SQL> select what, next_date from user_jobs where what like '%JOB_ENGINE%';

WHAT
------------------------------------------------------------------------------
NEXT_DATE
--------------------
MGMT_JOB_ENGINE.apply_purge_policies();
09/23/08 10:26:17 am

In this example, the purge policy DBMS job will run every day at 10:26:17 AM, repository time.

Changing the SYSMAN Password

The SYSMAN account is the default super user account used to set up and administer Enterprise Manager. It is also the database account that owns the objects stored in the Oracle Management Repository. From this account, you can set up additional administrator accounts and set up Enterprise Manager for use in your organization.

The SYSMAN account is created automatically in the Management Repository database during the Enterprise Manager installation. You also provide a password for the SYSMAN account during the installation.

See Also:

Oracle Enterprise Manager Grid Control Installation and Basic Configuration for information about installing Enterprise Manager

If you later need to change the SYSMAN database account password, use the following procedure:

1. Shut down all the Oracle Management Service instances that are associated with the Management Repository.

See Also:

"Controlling the Oracle Management Service"

2. Stop the agent that is monitoring the target OMS and Repository.
Failure to do this will result in the agent attempting to connect to the target with a wrong password once it is changed with SQL*Plus. This may also result in the SYSMAN account being locked which can subsequently prevent logins to the Grid Control console to change the password of the target OMS and Repository.

3. Change the password of the SYSMAN database account using the following SQL*Plus commands:

SQL>connect sysman/oldpassword;
SQL>alter user sysman identified by newpassword;

4. For each Management Service associated with the Management Repository, locate the emoms.properties configuration file.
The emoms.properties file can be found in the following directory of the Oracle Application Server Home where the Oracle Management Service is installed and deployed:

IAS_HOME/sysman/config/

5. Locate the following entries in the emoms.properties file:

oracle.sysman.eml.mntr.emdRepPwd=ece067ffc15edc4f
oracle.sysman.eml.mntr.emdRepPwdEncrypted=TRUE

6. Enter your new password in the first entry and enter FALSE in the second entry.
For example:

oracle.sysman.eml.mntr.emdRepPwd=new_password
oracle.sysman.eml.mntr.emdRepPwdEncrypted=FALSE

7. Save and exit the emoms.properties file and restart each Management Service associated with the Management Repository.

8. In the Grid Control console, click the Targets tab and then click All Targets on the sub tab.

9. Select the Management Services and Repository target and click Configure. Enterprise Manager displays the Monitoring Configurations page.

10. Enter the new password in the Repository password field and click OK.

See Also:

"Specifying New Target Monitoring Credentials"

11. After the Management Service has started, you can check the contents of the emoms.properties file to be sure the password you entered has been encrypted.
For example, the entries should appear as follows:

oracle.sysman.eml.mntr.emdRepPwd=ece067ffc15edc4f
oracle.sysman.eml.mntr.emdRepPwdEncrypted=TRUE

Overview of the MGMT_VIEW User

During repository creation, the MGMT_VIEW user is created. This view is used by Grid Control for the reporting framework to execute queries for Table from SQL and Chart from SQL report elements. The OMS is the only entity that uses the account so there is no need to know the password. However, you can still change the password if you choose, which requires that you bounce the OMS. To change the password, you can use either a PL/SQL call or an EMCTL command:

PL/SQL:

SQL> exec mgmt_view_priv.change_view_user_password('<random pwd>');

EMCTL command:

emctl config oms –change_view_user_pwd [-sysman_pwd <pwd>] [-user_pwd <pwd>] [-autogenerate]

Dropping and Recreating the Management Repository

This section provides information about dropping the Management Repository from your existing database and recreating the Management Repository after you install Enterprise Manager.

Dropping the Management Repository

To recreate the Management Repository, you first remove the Enterprise Manager schema from your Management Repository database. You accomplish this task using the -action drop argument to the RepManager script, which is described in the following procedure.

To remove the Management Repository from your database:

1. Locate the RepManager script in the following directory of the Oracle Application Server Home where you have installed and deployed the Management Service:

IAS_HOME/sysman/admin/emdrep/bin

2. At the command prompt, enter the following command:

$PROMPT> RepManager repository_host repository_port repository_SID
-sys_password password_for_sys_account -action drop

In this syntax example:

- repository_host is the machine name where the Management Repository database is located

- repository_port is the Management Repository database listener port address, usually 1521 or 1526

- repository_SID is the Management Repository database system identifier

- password_for_sys_account is the password of the SYS user for the database. For example, change_on_install.

- -action drop indicates that you want to drop the Management Repository.

Alternatively, you can use a connect descriptor to identify the database on the RepManager command line. The connect descriptor identifies the host, port, and name of the database using a standard Oracle database syntax.

For example, you can use the connect descriptor as follows to create the Management Repository:

$PROMPT> ./RepManager -connect "(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)
(HOST=host1)(PORT=1521)) (CONNECT_DATE=(SERVICE_NAME=servicename)))"
-sys_password efkl34lmn -action drop

See Also:

"Establishing a Connection and Testing the Network" in the Oracle Database Net Services Administrator's Guide for more information about connecting to a database using connect descriptors

Recreating the Management Repository

The preferred method for creating the Management Repository is to create the Management Repository during the Enterprise Manager installation procedure, which is performed using Oracle Universal Installer.

See Also:

Oracle Enterprise Manager Grid Control Installation and Basic Configuration for information about installing Enterprise Manager

However, if you need to recreate the Management Repository in an existing database, you can use the RepManager script, which is installed when you install the Management Service. Refer to the following sections for more information:

- Using the RepManager Script to Create the Management Repository

- Using a Connect Descriptor to Identify the Management Repository Database

Using the RepManager Script to Create the Management Repository

To create a Management Repository in an existing database:

1. Review the hardware and software requirements for the Management Repository as described in Oracle Enterprise Manager Grid Control Installation and Basic Configuration. and review the section "Management Repository Deployment Guidelines".

2. Locate the RepManager script in the following directory of the Oracle Management Service home directory:

ORACLE_HOME/sysman/admin/emdrep/bin

3. At the command prompt, enter the following command:

$PROMPT> ./RepManager repository_host repository_port repository_SID  -sys_password password_for_sys_account -action create

In this syntax example:

- repository_host is the machine name where the Management Repository database is located

- repository_port is the Management Repository database listener port address, usually 1521 or 1526

- repository_SID is the Management Repository database system identifier

- password_for_sys_account is the password of the SYS user for the database. For example, change_on_install.

Enterprise Manager creates the Management Repository in the database you specified in the command line.

Using a Connect Descriptor to Identify the Management Repository Database

Alternatively, you can use a connect descriptor to identify the database on the RepManager command line. The connect descriptor identifies the host, port, and name of the database using a standard Oracle database syntax.

For example, you can use the connect descriptor as follows to create the Management Repository:

$PROMPT> ./RepManager -connect "(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP) (HOST=host1)(PORT=1521)) (CONNECT_DATA=(SERVICE_NAME=servicename)))" -sys_password efkl34lmn -action create

See Also:

"Establishing a Connection and Testing the Network" in the Oracle Database Net Services Administrator's Guide for more information about connecting to a database using a connect descriptor

The ability to use a connect string allows you to provide an address list as part of the connection string. The following example shows how you can provide an address list consisting of two listeners as part of the RepManager command line. If a listener on one host becomes unavailable, the second listener can still accept incoming requests:

$PROMPT> ./RepManager -connect "(DESCRIPTION=
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=TCP)(HOST=host1)(PORT=1521)
(ADDRESS=(PROTOCOL=TCP)(HOST=host2)(PORT=1521)
(CONNECT_DATE=(SERVICE_NAME=servicename)))"
-sys_password efkl34lmn -action create

See Also:

Oracle Database High Availability Architecture and Best Practices
Oracle Enterprise Manager Grid Control Installation and Basic Configuration

Troubleshooting Management Repository Creation Errors

Oracle Universal Installer creates the Management Repository using a configuration step at the end of the installation process. If the repository configuration tool fails, note the exact error messages displayed in the configuration tools window, wait until the other configuration tools have finished, exit from Universal Installer, and then use the following sections to troubleshoot the problem.

Package Body Does Not Exist Error While Creating the Management Repository

If the creation of your Management Repository is interrupted, you may receive the following when you attempt to create or drop the Management Repository at a later time:

SQL> ERROR:
ORA-00604: error occurred at recursive SQL level 1
ORA-04068: existing state of packages has been discarded
ORA-04067: not executed, package body "SYSMAN.MGMT_USER" does not exist
ORA-06508: PL/SQL: could not find program unit being called
ORA-06512: at "SYSMAN.SETEMUSERCONTEXT", line 5
ORA-06512: at "SYSMAN.CLEAR_EMCONTEXT_ON_LOGOFF", line 4
ORA-06512: at line 4

To fix this problem, see "General Troubleshooting Techniques for Creating the Management Repository".

Server Connection Hung Error While Creating the Management Repository

If you receive an error such as the following when you try to connect to the Management Repository database, you are likely using an unsupported version of the Oracle Database:

Server Connection Hung

To remedy the problem, upgrade your database to the supported version as described in Oracle Enterprise Manager Grid Control Installation and Basic Configuration.

General Troubleshooting Techniques for Creating the Management Repository

If you encounter an error while creating the Management Repository, drop the repository by running the -drop argument to the RepManager script.

See Also:

"Dropping the Management Repository"

If the RepManager script drops the repository successfully, try creating the Management Repository again.

If you encounter errors while dropping the Management Repository, do the following:

1. Connect to the database as SYSDBA using SQL*Plus.

2. Check to see if the SYSMAN database user exists in the Management Repository database.
For example, use the following command to see if the SYSMAN user exists:

prompt> SELECT username FROM DBA_USERS WHERE username='SYSMAN';

3. If the SYSMAN user exists, drop the user by entering the following SQL*Plus command:

prompt> DROP USER SYSMAN CASCADE;

4. Check to see if the following triggers exist:

SYSMAN.EMD_USER_LOGOFF
SYSMAN.EMD_USER_LOGON

For example, use the following command to see if the EMD_USER_LOGOFF trigger exists in the database:

prompt> SELECT trigger_name FROM ALL_TRIGGERS
WHERE trigger_name='EMD_USER_LOGOFF';

5. If the triggers exist, drop them from the database using the following commands:

prompt> DROP TRIGGER SYSMAN.EMD_USER_LOGOFF;
prompt> DROP TRIGGER SYSMAN.EMD_USER_LOGON;

Cross Platform Enterprise Manager Repository Migration

There are user requirements for migrating an Enterprise Manager repository across servers - same and cross platforms.

The Enterprise Manager repository migration process is not exactly the same as database migration. In case of Enterprise Manager Repository migration you must take care of Enterprise Manager specific data, options, and pre-requisites for the repository move. You should make sure data integrity is maintained from both the Enterprise Manager and Oracle database perspective.

This brings up need for defining the process that can be followed by end users for successful and reliable migration of repository in minimum time and with maximum efficiency.

The overall strategy for migration depends on:

- The source and target database version

- The amount of data/size of repository

- Actual data to migrate [selective/full migration]

Cross platform transportable tablespace along with data pump (for metadata) is the fastest and best approach for moving large Enterprise Manager Grid Control repository from one platform to another. Other option that can be considered for migration is to use Data Pump for both data and metadata moves but this would require more time than the cross platform transportable tablespace approach for the same amount of data. The advantage to using the data pump approach is that it provides granular control over options and the overall process, as in the case of selective data being migrated and not the whole of source data. If the source and target is not on version 10g then export/import is the only way to get the data migrated cross platform.

More details on cross platform transportable tablespace, data pump, and export/import options can be found at the Oracle Technology Network (OTN) or in the Oracle Database Administrator's Guide.

Common Prerequisites

The following lists the common prerequisites for a repository migration:

- Source and target database must use the same character set and should be at same version.

- Source and target database should meet all the pre-requisites mentioned for Enterprise Manager Repository software requirements mentioned in Enterprise Manager install guide.

- If source and target database are NOT on 10g - only Export/Import can be used for cross platform migration

- If Source and target database are on 10g - either of three options Cross platform transportable tablespaces migration, Data Pump or Export/Import can be used for cross platform repository migration

- You cannot transport a tablespace to a target database in which a tablespace with the same name already exists. However, you can rename either the tablespace to be transported or the destination tablespace before the transport operation.

- To plug a transportable tablespace set into an Oracle Database on a different platform, both databases must have compatibility set to at least 10.0.

- Most of the platforms(but not all) are supported for cross-platform tablespace transport. You can query the V$TRANSPORTABLE_PLATFORM view to see the platforms that are supported, and to determine their platform IDs and their endian format (byte ordering).

- Source and Destination host should have EM agent running and configured to the instance which is to be migrated

- If target database has EM repository installed, it should be first dropped using RepManager before target database related steps are carried out.

Methodologies

The following sections discuss the methodologies of a repository migration.

Cross Platform Transportable Tablespaces

Oracle's transportable tablespace feature allows users to quickly move a user tablespace across Oracle databases. It is the most efficient way to move bulk data between databases. Prior to Oracle Database 10g, if you want to transport a tablespace, both source and target databases need to be on the same platform. Oracle Database 10g adds the cross platform support for transportable tablespaces. With the cross platform transportable tablespace, you can transport tablespaces across platforms.

Cross platform transportable tablespaces allows a database to be migrated from one platform to another (use with Data Pump or Import/Export).

Preparation for Transportable Tablespaces

Use these steps to prepare for transportable tablespaces:

1. Prepare set of user tablespaces and Check for containment violation
execute DBMS_TTS.TRANSPORT_SET_CHECK('MGMT_TABLESPACE,MGMT_ECM_DEPOT_TS', TRUE);

select * FROM transport_set_violations;

2. Shutdown OMS instances and prepare for migration
Shutdown OMS, set job queue_processes to 0 and run

@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_remove_dbms_jobs.sql

3. Make the tablespaces to be transported read only
alter tablespace MGMT_TABLESPACE read only;

alter tablespace MGMT_ECM_DEPOT_TS read only;

Extract metadata

Extract Metadata for transportable tablespaces using Data Pump Utility:

1. Create data pump directory
create directory data_pump_dir as '/scratch/gachawla/EM102/ttsdata';

2. Extract the metadata using data pump (or export )
expdp DUMPFILE=ttsem102.dmp TRANSPORT_TABLESPACES=MGMT_TABLESPACE,MGMT_ECM_DEPOT_TS TRANSPORT_FULL_CHECK=Y

3. Extract other objects ( packages, procedures, functions, temporary tables etc - Not contained in user tablespaces)
expdp SCHEMAS=SYSMAN CONTENT=METADATA_ONLY EXCLUDE=INDEX,CONSTRAINT DUMPFILE=data_pump_dir:postexp.dmp LOGFILE=data_pump_dir:postexp.log JOB_NAME=expmet

Endian check and conversion

Run Endian check and convert the datafiles if endian is different between source and destination:

1. For Endian check, run this on both source and destination database
SELECT endian_format

FROM v$transportable_platform tp, v$database d

WHERE tp.platform_name = d.platform_name;

If the source platform and the target platform are of different endianness, then an additional step must be done on either the source or target platform to convert the tablespace being transported to the target format. If they are of the same endianness, then no conversion is necessary and tablespaces can be transported as if they were on the same platform.

Example:

Source Endian
Linux IA (32-bit) - Little

Destination Endian
Solaris[tm] OE (32-bit) - Big

2. Ship datafiles, metadata dump to target and Convert datafiles using RMAN
Ship the datafiles and the metadata dump to target and On target convert all datafiles to destination endian:

CONVERT DATAFILE
'/d14/em10g/oradata/em102/mgmt.dbf',
'/d14/em10g/oradata/em102/mgmt_ecm_depot1.dbf'
FROM PLATFORM 'Linux IA (32-bit)';

Conversion via RMAN can be done either on source or target (For more details refer RMAN doc). Parallelism can be used to speed up the process if the user tablespaces contains multiple datafiles.

Import metadata and plugin tablespaces

Use the following steps to import metadata and plugin tablespaces:

1. Run RepManager to drop target repository (if target database has EM repository installed)
RepManager repository_host repository_port repository_SID -sys_password password_for_sys_account -action drop

2. Run pre import steps to create sysman user and grant privs on target database
@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_repos_user.sql

@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_pre_import.sql

3. Invoke Data Pump utility to plug the set of tablespaces into the target database.
impdp DUMPFILE=ttsem102.dmp DIRECTORY=data_pump_dir

TRANSPORT_DATAFILES=/d14/em10g/oradata/em102/mgmt.dbf,/d14/em10g/oradata/em102/mgmt_ecm_depot1.dbf

4. Import other objects (packages, procedures, functions etc)
impdp CONTENT=METADATA_ONLY EXCLUDE=INDEX,CONSTRAINT DUMPFILE=data_pump_dir:postexp.dmp LOGFILE=data_pump_dir:postexp.log

Post Plug In Steps

Follow these post plug in steps:

1. Run post plugin steps to recompile any invalids, create public synonyms, create other users, enable VPD policy, repin packages
@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_synonyms.sql

@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_post_import.sql

Check for invalid objects - compare source and destination schemas for any discrepancy in counts and invalids.

2. Bring user tablespaces back to read write mode
alter tablespace MGMT_TABLESPACE read write;

alter tablespace MGMT_ECM_DEPOT_TS read write;

3. Submit EM dbms jobs
Reset back job_queue_processes to original value and run

@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_submit_dbms_jobs.sql

4. Update OMS properties and startup OMS
Update emoms.properties to reflect the migrated repository. Update host name - oracle.sysman.eml.mntr.emdRepServer and port with the correct value and start the OMS.

5. Relocate Management Services and Repository target
If Management Services and repository target needs to be migrated to the destination host, run em_assoc. handle_relocated_target to relocate the target or recreate the target on the target host.

6. Discover/relocate Database and database Listener targets
Discover the target database and listener in EM or relocate the targets from source agent to destination agent.

Data Pump

Oracle Data Pump technology enables high-speed, parallel movement of bulk data and metadata from one database to another. Data Pump uses APIs to load and unload data instead of usual SQL commands. Data pump operations can be run via EM interface and is very useful for cross platform database migration.

The migration of the database using the Data Pump export and Data Pump import tools comprises these steps: export the data into a dump file on the source server with the expdp command; copy or move the dump file to the target server; and import the dump file into Oracle on the target server by using the impdp command; and run post import EM specific steps.

Tuning parameters that were used in original Export and Import, such as BUFFER and RECORDLENGTH, are neither required nor supported by Data Pump Export and Import

Prepare for Data Pump

Use the following steps to prepare for data pump:

1. Pre-requisite for using Data pump for EM repository
Impdp fails for EM repository because of data pump bug - Bug 4386766 - IMPDP WITH COMPRESSED INDEXES FAILS WITH ORA-14071 AND ORA-39083. This bug is fixed in 10.2. Backport is available for 10.1.0.4. This RDBMS patch has to be applied to use expdp/impdp for EM repository migration or workaround is to use exp/imp for extract and import.

2. Create data pump directory
Create directory data_pump_dir as '/scratch/gachawla/EM102/ttsdata';

3. Shutdown OMS instances and prepare for migration
Shutdown OMS, set job queue_processes to 0 and run @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_remove_dbms_jobs.sql

To improve throughput of a job, PARALLEL parameter should be used to set a degree of parallelism that takes maximum advantage of current conditions. In general, the degree of parallelism should be set to more than twice the number of CPUs on an instance.

All data pump actions are performed by multiple jobs (server processes not DBMS_JOB jobs). These jobs are controlled by a master control process which uses Advanced Queuing. At runtime an advanced queue table, named after the job name, is created and used by the master control process. The table is dropped on completion of the data pump job. The job and the advanced queue can be named using the JOB_NAME parameter.

DBMS_DATAPUMP APIs can also be used to do data pump export/import. Please refer to Data pump section in 10g administration manual for all the options.

Data Pump Export

Use these steps to run data pump export:

1. Run data pump export:

expdp FULL=y DUMPFILE=data_pump_dir:dpfull1%U.dmp, data_pump_dir:dpfull2%U.dmp PARALLEL=4 LOGFILE=data_pump_dir:dpexpfull.log JOB_NAME=dpexpfull
Verify the logs for any errors during export

Data pump direct path export sometimes fails for mgmt_metrics_raw and raises ORA 600. This is due to Bug 4221775 (4233303). This bug is fixed in 10.2. Workaround: if using expdp data pump for mgmt_metrics_raw , run expdp with ACCESS_METHOD+EXTERNAL_TABLE parameter.

expdp directory=db_export dumpfile=exp_st2.dmp logfile=exp_st2.log tables=sysman.mgmt_metrics_raw access_method=external_table

Data Pump Import

Use these steps to run data pump import:

1. Run RepManager to drop target repository (if target database has EM repository installed)
RepManager repository_host repository_port repository_SID -sys_password password_for_sys_account -action drop

2. Prepare the target database
@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_tablespaces.sql

@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_repos_user.sql

@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_pre_import.sql

3. Run data pump import
Impdp FULL=y DUMPFILE=data_pump_dir:dpfull1%U.dmp, data_pump_dir:dpfull2%U.dmp PARALLEL=4 LOGFILE=data_pump_dir:dpimpfull.log JOB_NAME=dpimpfull

Verify the logs for any issues with the import.

Post Import EM Steps

Use the following steps for post import Enterprise Manager steps:

1. Run post plugin steps to recompile any invalids, create public synonyms, create other users, enable VPD policy, repin packages
@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_synonyms.sql

@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_post_import.sql

Check for invalid objects - compare source and destination schemas for any discrepancy in counts and invalids.

2. Submit EM dbms jobs
Reset back job_queue_processes to original value and run

@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_submit_dbms_jobs.sql

3. Update OMS properties and startup OMS
Update emoms.properties to reflect the migrated repository. Update host name - oracle.sysman.eml.mntr.emdRepServer and port with the correct value and start the OMS.

4. Relocate Management Services and Repository target
If Management Services and repository target needs to be migrated to the destination host, run em_assoc. handle_relocated_target to relocate the target or recreate the target on the target host.

5. Discover/relocate Database and database Listener targets
Discover the target database and listener in EM or relocate the targets from source agent to destination agent.

Export/Import

If the source and destination database is non-10g, then export/import is the only option for cross platform database migration.

For performance improvement of export/import, set higher value for BUFFER and RECORDLENGTH . Do not export to NFS as it will slow down the process considerably. Direct path can be used to increase performance. Note - As EM uses VPD, conventional mode will only be used by Oracle on tables where policy is defined.

Also User running export should have EXEMPT ACCESS POLICY privilege to export all rows as that user is then exempt from VPD policy enforcement. SYS is always exempted from VPD or Oracle Label Security policy enforcement, regardless of the export mode, application, or utility that is used to extract data from the database.

Prepare for Export/Import

Use the following steps to prepare for Export/Import:

1. Mgmt_metrics_raw partitions check

select table_name,partitioning_type type,
partition_count count, subpartitioning_type subtype from
dba_part_tables where table_name = 'MGMT_METRICS_RAW'

If MGMT_METRICS_RAW has more than 3276 partitions please see Bug 4376351 - This is Fixed in 10.2 . Workaround is to export mgmt_metrics_raw in conventional mode.

2. Shutdown OMS instances and prepare for migration
Shutdown OMS, set job queue_processes to 0 and run @IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_remove_dbms_jobs.sql

Export

Follow these steps for export:

1. Export data
exp full=y constraints=n indexes=n compress=y file=fullem102_1.dmp log=fullem102exp_1.log

2. Export without data and with constraints
exp full=y constraints=y indexes=y rows=n ignore=y file=fullem102_2.dmp log=fullem102exp_2.log

Import

Follow these steps to import:

1. Run RepManager to drop target repository (if target database has EM repository installed)
RepManager repository_host repository_port repository_SID -sys_password password_for_sys_account -action drop

2. Pre-create the tablespaces and the users in target database
@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_tablespaces.sql

@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_repos_user.sql

@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_pre_import.sql

3. Import data
imp full=y constraints=n indexes=n file=fullem102_1.dmp log=fullem102imp_1.log

4. Import without data and with constraints
imp full=y constraints=y indexes=y rows=n ignore=y file=fullem102_2.dmp log=fullem102imp_2.log

Post Import EM Steps

Follow these steps for post import EM steps:

1. Run post plugin steps to recompile any invalids, create public synonyms, create other users, enable VPD policy, repin packages
@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_create_synonyms.sql

@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_post_import.sql

Check for invalid objects - compare source and destination schemas for any discrepancy in counts and invalids.

2. Submit EM dbms jobs
Reset back job_queue_processes to original value and run

@IAS_HOME/sysman/admin/emdrep/sql/core/latest/admin/admin_submit_dbms_jobs.sql

3. Update OMS properties and startup OMS
Update emoms.properties to reflect the migrated repository. Update host name, oracle.sysman.eml.mntr.emdRepServer and port with the correct value and start the OMS.

4. Relocate Management Services and Repository target
If Management Services and repository target needs to be migrated to the destination host, run em_assoc. handle_relocated_target to relocate the target or recreate the target on the target host.

5. Discover/relocate Database and database Listener targets
Discover the target database and listener in EM or relocate the targets from source agent to destination agent.

Post Migration Verification

These verification steps should be carried out post migration to ensure that the migration was completely successful:

- Verify any discrepancy in objects by comparing source and target databases through EM

- Verify migrated database through EM whether database is running without any issues

- Verify repository operations, dbms jobs and whether any management system errors reported

- Verify all EM functionalities are working fine after migration

- Make sure Management Services and Repository target is properly relocated by verifying through EM

Improving the Login Performance of the Console Home Page

Oracle Enterprise Manager now provides an option that will more quickly display the Console Home page even in a scenario where the Management Repository is very large. Normally, factors such as the number of alerts, errors, policies, and critical patches can contribute to delayed displayed times. Since there is no single factor nor any simple way to scale the SQL or user interface, a simple option flag has been added that removes the following page elements for all users.

When the emoms.properties flag, LargeRepository=, is set to true (when normally the default is false), the SQL for the following items is not executed and thus the items will not be displayed on the Console page.

1. Three sections from the Overview Page segment:

- All Target Alerts

- Critical

- Warning

- Errors

- All Target Policy Violations

- Critical

- Warning

- Informational

- All Target Jobs

- Problem Executions (last 7 days)

- Suspended Executions (last 7 days)

2. The page segment which includes Security Patch Violations and Critical Patch Advisories.

The Deployment Summary section would move up to fill in the vacated space.

Part II

Advanced Topics

This section of the guide discusses the application of Enterprise Manager functionality to monitor and maintain your entire ecosystem.

Part II contains the following chapters:

- Chapter 13, "Managing Compliance"

- Chapter 14, "Configuring Services"

- Chapter 15, "Extending Enterprise Manager"

13 Managing Compliance

This chapter explains how Enterprise Manager Grid Control verifies that applications in your enterprise comply with preestablished standards. This chapter contains the following sections:

- Compliance Overview

- Compliance Management

- Setting Up Compliance Evaluations

- Policy Groups

Compliance Overview

To have your enterprise run efficiently, it must adhere to standards that promote the best practices for security, configuration, and storage. Once these standards are developed, you can apply and test for these standards throughout your organization; that is, test for compliance. Compliance is the conformance to standards, or requirements, or both.

Using Enterprise Manager Grid Control, you can test the conformance of your targets for security standards, and configuration and storage requirements.

By continually testing your systems, services, and targets, you are ensuring the best possible protection and performance your system can have.

Compliance is two-fold: evaluating the compliance of your targets, and the policy group lifecycle management. The following sections describe these two concepts.

Note:

To view the compliance features:

1. Navigate to the Grid Control Home page.

2. Click the Compliance tab to access information regarding the policies, policy groups, and security statistics for your enterprise.

Compliance Management

Oracle provides two types of compliance management: policies and policy groups. Policies and policy groups define the optimal configurations of systems.

Policies and policy groups are similar in purpose, that is, they both provide rules against which managed entities are evaluated. However, there are differences:

- Rules within a policy group are managed as a set. They are viewed, maintained, evaluated, and reported, in the context of a policy group.

- Policy rules are not evaluated as a set. Policy rules are viewed, maintained, and evaluated as standalone entities.

Whether you use the out-of-box policies and policy groups defined by Oracle or customize policies to meet your particular system requirements, any deviations to your systems or applications are reported. Examples of deviations include inappropriate settings and incorrect system configurations.

This section contains the following subsections:

- Accessing Compliance Management Pages in Grid Control

- Investigating Compliance Violations

- Assessing Security

- Compliance Violations Reports

Accessing Compliance Management Pages in Grid Control

To access compliance management pages in Grid Control:

- Click the Compliance tab to view violations associated with policies and policy groups.

- Click the Policies tab for a roll-up view of all policy violations across all targets. From this tab, you can also access policy associations, the policy library, and errors.

- Click the Policy Groups tab to view the list of policy groups against which there are violations. From this tab, you can also access the library of policy groups and policy group evaluation errors.

- Click the Security At a Glance tab for a roll-up view of security statistics across the enterprise.

- Navigate to the Home page for a particular target. The links in the Policy Violations section display the number of policy violations according to severity level. Click the links to drill down to critical, warning, and informational policy violations for that target.

Investigating Compliance Violations

Here are a few suggestions for investigating compliance violations. Attend to the most critical violations or those that have the biggest impact on your enterprise.

- Study the statistics on the Enterprise Manager Grid Control Home page. In particular, look at the statistics in the All Targets Policy Violations section. The policy violations with "Critical" severity should be dealt with first.

- Study the security-related violations reported in the Security Policy Violations section. Non-compliance with these policy rules can greatly impact the security of your enterprise.

- Address targets that have the lowest compliance scores.

- For the policy violations of a particular target, examine the home page for that target. The Policy Violations section provides overview information, but it also gives you access to the Policy Trend Overview for that target.

- To deal with policies regardless of the target, navigate to the Compliance tab and then click Policies. Using this tab, you have access to all the policy violations for the enterprise, the policy associations, the policy rule library which lists all the policies, and policy evaluation errors.

- Navigate to the Policy Violations page and enter an appropriate value in the "Most Recent Violation within n days" filter.

- Suppress violations if you want to handle the violations at a later time.

- To deal with policy groups regardless of the target, click Policy Groups. Using this tab, you have access to all the evaluation results for the enterprise, the policy group library, and policy group evaluation errors.

- Navigate to the Evaluation Results page for a particular policy group. In the navigation tree, click the name of the policy group and a summary page lists all the targets along with the number of violations.

- Navigate to the Trend Overview page to see charts relating to the number of targets evaluated, the average violation count per target, number of targets by compliance score, and the average compliance score.

See Also:

"About Policies" and "About Policy Groups" in the Grid Control online help for an overview of policies and policy groups and pointers to more information about viewing and managing policies and policy groups.

Assessing Security

Security policies are available for many targets, including Host, Database Instance, Cluster Database, Listener, OC4J, Oracle HTTP Server, and Web Cache.

Security policy groups are available for Database Instance, Cluster Database, and Listener.

Because security is crucial to the stability of your enterprise, security statistics are displayed prominently in Grid Control. On the Enterprise Manager Grid Control Home page, and many target home pages, there is a separate section displaying the security statistics for the target. This allows you to pay close attention to the security health of the enterprise.

In addition, the Security At a Glance feature provides an overview of the security health of the enterprise for all the targets or specific groups. This helps you to quickly focus on security issues by showing statistics about security policy violations and noting the critical security patches that have not been applied.

Viewing Compliance Evaluation Results

To view the results of an evaluation, use any or all of the following:

- Use the Evaluation Results page accessed through the Policy Groups tab

- Study the statistics in the Policy Violations section of the target's home page

- Use the Policy Trend Overview page and the Trend Overview page available from the Evaluation Results page

- Access the Security At a Glance page

Compliance Violations Reports

The Policy Violations reports and Policy Groups reports are available through the Reports feature. These reports deal with non-suppressed violations for all targets, groups, and a single target. The reports also deal with compliance summary for a group and target. In addition, suppressed violations are reported according to all targets, groups, and a single target.

See Also:

Chapter 10, "Information Publisher"

Setting Up Compliance Evaluations

Compliance evaluation is the process of testing the policy and policy group rules against a target and recording any violations in the Oracle Management Repository.

Scheduling an Evaluation

For the evaluation to take place, you must enable the evaluation in one of two ways:

- For a policy, use the Metric Thresholds option on the Metric and Policy Settings page

- For a policy group, use the Policy Group Library page

Out-of-Box Policies and Policy Groups

Oracle provides a number of out-of-box policies (also known as policy rules) and policy groups for various targets.

When you add a target to Enterprise Manager, that target automatically uses predefined policy rules for that type of target. For example, Oracle provides security, configuration, and storage policy rules for the database instances and cluster databases. Security and configuration policy rules are provided for hosts.

Note:

Policy Groups are not automatically associated with targets.

Customizing Policies

You can customize policies by editing the existing policy rule settings. You can enable or disable a policy evaluation, change the importance for the compliance score calculation, assign a corrective action, prevent template override, override default parameter values (when possible), and exclude objects from a policy's evaluation (when possible).

See Also:

Online help for compliance scores

Defining Corrective Actions

One of the features of customizing policies is the ability to define corrective actions. Corrective Actions is a special type of job that executes automatically in response to a policy violation.

Corrective Actions utilize the Enterprise Manager Grid Control Job System and, like regular jobs, can consist of multiple steps, can be run with arbitrary host and target credentials, and reports its success or failure and its output to the Management Repository.

See Also:

Chapter 7, "Job System"

Using Templates for Monitoring

A monitoring template defines all Enterprise Manager parameters you would normally set to monitor a target.

Monitoring templates simplify the task of setting up monitoring for large numbers of targets by allowing you to specify the monitoring and policy settings once and applying them as often as needed. You can save, edit, and apply these templates across one or more targets or groups.

See Also:

"Monitoring Templates" in Chapter 1, "Monitoring"

Policy Groups

Policy groups serve as standards by which targets are measured. Policy groups report deviations and enable closed loop remediation by optionally taking action to bring systems back into compliance. Oracle provides the following policy groups:

- Secure Configuration for Oracle Database

- Secure Configuration for Oracle Real Application Cluster

- Secure Configuration for Oracle Listener

These standards represent best practices and allow you to maintain consistency across enterprise systems and configurations. The trend analysis feature allows fine grained tracking of compliance progress over time.

The following sections provide the highlights of each policy group.

Secure Configuration for Oracle Database

This policy group adheres to the security standards available for the Oracle Database. The categories deemed the most important for this policy group are:

- Post Installation
These rules ensure that a database is not compromised by having a default database server account left open that uses its default password.

- Oracle Directory and File Permissions
These rules ensure that access should be restricted, making it more difficult for an operating system user to attack the database.

- Oracle Parameters Settings
These rules ensure database initialization parameter settings are secure.

- Database Password Profile Settings
These rules ensure database profile settings are correctly defined. Oracle password management is controlled through the use of user profiles which are then assigned to database users, enabling greater control over database security.

- Database Access Settings
These rules ensure that access to and use of the database at the object level is restricted such that users are only given those privileges that are actually required to efficiently perform their jobs.

Secure Configuration for Oracle Real Application Cluster

This policy group adheres to the security standards available for the Oracle Cluster Database. The categories deemed the most important for this policy group are:

- Post Installation
These rules ensure that a database is not compromised by having a default database server account left open that uses its default password.

- Oracle Directory and File Permissions
These rules ensure that access should be restricted, making it more difficult for an operating system user to attack the database.

- Database Password Profile Settings
These rules ensure database profile settings are correctly defined. Oracle password management is controlled through the use of user profiles which are then assigned to database users, enabling greater control over database security.

- Database Access Settings
These rules ensure that access to and use of the database at the object level is restricted such that users are only given those privileges that are actually required to efficiently perform their jobs.

Secure Configuration for Oracle Listener

This policy group adheres to the security standards available for the Oracle Listener. The categories deemed the most important for this policy group are:

- Oracle Directory and File Permissions
These rules ensure that access should be restricted, making it more difficult for an operating system user to attack the database.

- Network Configuration Settings
These rules ensure that network configuration parameter settings are secure.

14 Configuring Services

This chapter describes how to configure services in Oracle Enterprise Manager 10g Grid Control Console. It contains the following sections:

- Summary of Service Management Tasks

- Setting up the System

- Creating a Service

- Configuring a Service

- Recording Web Transactions

- Monitoring Settings

- Configuring Aggregate Services

- Configuring End-User Performance Monitoring

- Managing Forms Applications

- Configuring OC4J for Request Performance Diagnostics

- Setting Up Monitoring Templates

- Configuring Service Levels

- Configuring a Service Using the Command Line Interface

- Troubleshooting Service Tests

Summary of Service Management Tasks

This table provides a summary list of all the service management features and their requirements.

Table 14-1 Summary of Service Management Tasks

Feature

Description

Requirements

Refer to

Test Performance

This feature allows you to proactively monitor services using service tests or synthetic transactions and determine their performance and availability from different user locations using beacons. For Web transactions, you can monitor the transactions at the transaction, step group and step level.

- Management Agent for enabling a beacon.

- Microsoft Internet Explorer 5.5 or later

Configuring a Service

End-User Performance Monitoring

Enterprise Manager allows you to gather end-user performance data and monitor the performance of the pages within your Web application. The End-User Performance Monitoring feature allows you to:

- Understand real end-user page response times within your application.

- Assess the user impact of performance problems.

- Analyze end user response times by by page, domain, region, visitors, and Web server.

- Oracle HTTP Server Based on Apache 2.0 or Apache HTTP Server 2.0

- Oracle Application Server Web Cache (10.1.2, 9.0.4, 9.0.3, or 9.0.2)

Configuring End-User Performance Monitoring

Interactive Transaction Tracing

Enterprise Manager provides a mechanism for interactively tracing Web transactions. This feature allows you to:

- Diagnose performance problems at the transaction level.

- Interactively trace transactions and analyze breakout of J2EE server activity times (servlet, JSP, EJB), and database times, including individual SQL statements.

- Microsoft Internet Explorer 5.5 or later for creating and playing back transactions.

- Oracle Application Server 10g (9.0.4) for playing back a transaction with trace to view J2EE server activity times.

Note: Recording a transaction is an optional feature. You can manually create a transaction by entering the required values.

Configuring Interactive Transaction Tracing

Request Performance

Enterprise Manager can gather critical request performance data about your Web application. The Request Performance feature allows you to:

- Diagnose root cause of performance problems.

- View historical tracing of J2EE middle tier activity.

- View breakouts of J2EE server processing times (servlet, JSP, EJB), and database times, including individual SQL statements.

- Correlate request performance to other Web application component metrics.

- View the full request processing call stack.

Oracle Application Server 10g (9.0.4) and above

Configuring OC4J for Request Performance Diagnostics

Root Cause Analysis

The Root Cause Analysis (RCA) feature provides you with the ability to analyze and determine possible causes of service failure.

The Topology Viewer provides a graphical representation of the service and its relationship to other services, systems and infrastructure components, with the causes identified by RCA highlighted in the display.

For the Topology Viewer

- Microsoft Internet Explorer 5.5 or higher

- Adobe SVG Viewer 3.0

Root Cause Analysis Configuration

Forms Applications

A Forms Application target in Enterprise Manager can be used to model and monitor a specific Forms application. You can:

- Record and monitor a Forms transaction.

- Measure the End-User Performance of Forms actions such as Commit, Query, Runform, Callform, Openform, and Newform.

- Microsoft Internet Explorer 5.5 or later for creating and playing back Forms transactions.

- Oracle HTTP Server or Apache HTTP Server

- Oracle Application Server Web Cache (10.1.2, or 9.0.4)

Recording and Monitoring Forms Transactions

Monitoring the End-User Performance of Forms Applications

Setting up the System

A system is the set of infrastructure components, for example hosts, databases, and application servers that work together to host your applications. Before you create a service, you must specify the system that will be used to host your service. Refer to the Enterprise Manager Online Help for details on defining systems.After you have selected the system, you must mark one or more components as key components that are critical to running your service. These key components are used to determine the availability of the service and identify possible causes of service failure for root cause analysis.

Creating a Service

Before you create a service, you must be familiar with the concepts of service management as described in the Oracle Enterprise Manager Concepts. You must also do the following:

- Install the Management Agent on the hosts on which the components of your service have been installed.

- Discover all the components for your service so that they can be listed as Enterprise Manager targets.

- Define systems on which the service is to be hosted.

To create a service, click the Targets tab and Services subtab. The Services main page is displayed. Select a service from the Add drop-down list and click Go. The following screen is displayed:

Figure 14-1 Create Service - General Page

Follow the steps in the wizard to create your service. This involves the following:

- Identifying the type of service to be created. You can define different types of services based on your requirement. Some of the services that you can define are Generic Service, Web Application, Aggregate Service, and Forms Application. A Generic Service is used to monitor a variety of different protocol based services. A Web Application is used to monitor Web transactions. Enterprise Manager provides additional monitoring and diagnostics features for Web applications. A Forms Application is used to monitor Forms transactions. Each Forms transaction can consist of one or more actions that can be monitored. You can also define other services that are specific to an application such as the OCS Service. You can combine one or more services to form an Aggregate Service.

- Specifying the name and time zone for the service.

- Selecting a system target that hosts this service and then marking the system's key components that are critical for running the service. These key components are used to determine the availability of the service and identify possible causes of service failure. For more information on defining systems and monitoring them, refer to the Service Management chapter in Oracle Enterprise Manager Concepts.

- Setting up the availability definition for the service. This can be service test-based or system-based. If you select service test, the service's availability is based on the execution of the service test by the one or more key beacons. If availability is based on system, availability is based on the status of one or more key components of the system.

- Adding one or more beacons to monitor service tests. Click Add to add one or more beacons for monitoring the service. It is recommended that you use beacons that are strategically located in your key user communities in order for them to proactively test the availability of the service from those locations. If no beacons exist, click Create to create a new beacon.

Note:

Beacons marked as key beacons will be used to determine the availability of the service. The service is available if one or more service tests can be successfully executed from at least one key beacon.
For Web applications, you can compare the performance of the service test execution from each remote beacon against the local beacon.

- Defining the metrics that will be used to measure the performance of the service. Performance metrics can be based on service tests or system components. After defining the metrics, you can specify the critical and warning thresholds. You can also specify the metric that is to be displayed in a graphical format on the Service Home page.

- Defining the metrics that will be used to measure the user demand for the service. Usage metrics can be based on one or more system components. After defining the metrics, you can specify the critical and warning thresholds. You can also specify the metric that is to be displayed in a graphical format on the Service Home page.

Note:

You can define usage metrics for system-based services only.

- After you have completed all the steps in the wizard, click Finish to create your service. Refer to the Enterprise Manager Online Help for more details on these pages.

Configuring a Service

After you have created the service, you can configure it further by selecting an option from the Monitoring Configuration page. To configure a service, select a service from the Services main page and click Configure to go to the Monitoring Configuration page. The following screen is displayed.

Figure 14-2 Monitoring Configuration Page

The following options are available:

- Availability Definition

- Performance Metrics

- Usage Metrics

- Business Metrics

- Service Tests and Beacons

- Root Cause Analysis Configuration

Apart from these options, for Web applications, the end-user and request performance monitoring features can also be configured. For more information, refer to the following sections:

- Configuring End-User Performance Monitoring

- Configuring OC4J for Request Performance Diagnostics

Availability Definition

You can modify the availability definition (service test-based or system-based) for the selected service. If availability is based on service tests, you can specify whether the service should be available when:

- All key service tests are successful (Default)

- At least one key service test is successful

Note:

A service test is considered available if it can be executed by at least one key beacon. If there are no key beacons, the service test will have an unknown status.

If availability is based on the key system components, you can specify whether the service should be available when:

- All key components are up (Default)

- At least one key component is up

You can also mark one or more components as key system components that will be used to compute the availability of the service. Key system components are used to determine the possible root cause of a service failure. For more information, refer to "Root Cause Analysis Configuration".

You can also indicate whether the service test is a key test by enabling the Key Service Test checkbox. Only key service tests are used to compute the availability of the service. You can then select the beacons that will be used to execute the key tests and determine the availability of the service.

Performance Metrics

Performance metrics are used to measure the performance of the service. If a service test has been defined for this service, then the response time measurements as a result of executing that service test can be used as a basis for the service's performance metrics. Alternatively, performance metrics from the underlying system components can also be used to determine the performance of the service. You can do the following:

- Add a performance metric for a service test. After selecting a metric, you can choose to:

- Use the metric values from one beacon. Choose this option if you want the performance of the service to be based on the performance of one specific location.

- Aggregate the metric across multiple beacons. Choose this option if you want to consider the performance from different locations. If you choose this option, you need to select the appropriate aggregation function:

Table 14-2 Beacon Aggregation Functions

Function

Description

Maximum

The maximum value of the metric from data collected across all beacons will be used. Use this function if you want to measure the worst performance across all beacons.

Minimum

The minimum value of the metric from data collected across all beacons will be used. Use this function if you want to measure the best performance across all beacons.

Average

The average value of the metric will be used. Use this function if you want to measure the 'average performance' across all beacons.

Sum

The sum of the metric values will be calculated. Use this function if you want to measure the sum of all response times across each beacon.

Note:

If you are configuring a Web transaction, you can specify the Source which can be transaction, step group or step. Based on this selection, the metric you add will be applicable at the transaction, step group, or step level.

- Add a performance metric for the underlying system components on which the service is hosted. After selecting a metric for a target, you can choose to:

- Use the metric from a specific component. Choose this option if you want the performance of the service to be based on the performance of one specific system component.

- Aggregate the metric across multiple components. Choose this option if you want to consider the performance from multiple components. If you choose this option, you need to select the appropriate aggregation function.

Table 14-3 System Aggregation Functions

Function

Description

Maximum

The maximum value of the metric across all components will be used as the value of this performance metric for the service.

Minimum

The minimum value of the metric across all components will be used as the value of this performance metric for the service.

Average

The average value of the metric across all components will be used.

Sum

The sum of values of metrics across all components will be calculated.

Note:

When a system is deleted, performance metrics associated with the system will not be collected.

- Edit a performance metric that has been defined. For service test-based performance metrics, you can modify the beacon function that should be used to calculate the metric values. For system-based performance metrics, you can modify the target type, metric, and whether the aggregation function should be used. You can also modify the Critical and Warning thresholds for the metric.

- Delete a performance metric that has been defined.

Usage Metrics

Usage metrics are used to measure the user demand for the service. Usage metrics are collected based on the usage of the underlying system components on which the service is hosted. You can do the following:

- Add a usage metric. After selecting a metric for a target, you can choose to:

- Use the metric from a specific component. Use this option if you want to monitor the usage of a specific component.

- Aggregate the metric across multiple components. Use this option if you want to statistically calculate the usage across multiple components. If you choose this option, you need select the appropriate aggregation function.

Table 14-4 Aggregation Functions - Usage Metrics

Function

Description

Maximum

The maximum value of the metric across all components will be used as the value of this usage metric for the service.

Minimum

The minimum value of the metric across all components will be used as the value of this usage metric for the service.

Average

The average value of the metric across all components will be used.

Sum

The sum of the values of metrics across all components will be calculated.

- Edit a usage metric that has been defined.

- Delete a usage metric that has been defined.

Business Metrics

Business metrics are used to measure the performance of business in an organization. These metrics are based on business indicators that can assess the business performance. You can define one or more system based metrics and specify critical and warning thresholds for these metrics. You can define business metrics for Generic Services and Aggregate Services.

Note:

This option is available only if one of the system components is a service and has business metrics associated with it.

You can do the following:

- Add a business metric. After selecting a metric for a target, you can choose to:

- Use the metric from a specific component. Use this option if you want the business metric to be based on the performance of one specific system component

- Aggregate the metric across multiple components. Use this option if you want to measure the business performance from multiple components. Select the appropriate aggregation function from the drop down list. If you choose this option, you need select the appropriate aggregation function.

Table 14-5 Aggregation Functions - Usage Metrics

Function

Description

Maximum

The maximum value of the metric across all components will be used as the value of this business metric for the service.

Minimum

The minimum value of the metric across all components will be used as the value of this business metric for the service.

Average

The average value of the metric across all components will be used.

Sum

The sum of the values of metrics across all components will be calculated.

- Edit a business metric that has been defined.

- Delete a business metric that has been defined.

You can define system based metrics only. You can configure non-system based metrics by using the Data Exchange feature which facilitates data transfer between Enterprise Manager Grid Control and other external monitoring systems. For details, refer to the Oracle Enterprise Manager Integration Guide.

Service Tests and Beacons

You can add additional service tests and specify one or more beacons that will execute these service tests. To add a service test or modify an existing service test, click the Service Test and Beacons link on the Monitoring Configuration page. The Service Tests and Beacons page appears. You can do the following:

- Add one or more service tests for your service. Select the Test Type and click Add. Some of the test types that can be defined are FTP, Web Transaction, DNS, SOAP and others. The Create Service Test page is displayed. Refer to the Enterprise Manager Online Help for details on the various types of service tests.

Note:

While defining a SOAP (Simple Object Access Protocol) service test, if the WSDL URL to be accessed is outside the company's intranet, proxy settings need to be added to the $OMS_HOME/sysman/config/emoms.properties file.
For example, to set up www-proxy.us.oracle.com as proxy, specify the values as follows:

proxyHost=www-proxy.us.oracle.com

proxyPort=80

dontProxyFor=us.oracle.com,oraclecorp.com

The proxyUser,proxyPwd,proxyRealm,and proxyPropsEncrypted properties are used to configure an authenticated proxy. After you have modified the proxy settings, you must restart the Oracle Management Service for the changes to be effective.

- After you have created the service test, you must enable it. If your service test is not enabled, it will not be executed by any of the beacons.

- Create, add, or remove a beacon. When you identify the beacon locations, select locations on your internal network or on the Internet that are important to your e-business. These are typical locations where your end users are located. For example, if your business is hosted in Canada and you have customers in the United States, use a beacon installed on a host computer in the United States to measure the availability and performance of your applications.

- After you have created the service test, you can verify it by clicking Verify Service Test.

Note:

You can define one or more service tests as key tests. These key tests are used to monitor the availability and performance of your service. Only service tests that are enabled can be designated as key tests. To set up a service test as a key test, click the Availability Definition link at the bottom of the page.

For more details on creating different types of service tests, refer to the Enterprise Manager Online Help.

Configuring the Beacons

This section lists additional beacon related configuration tasks.

- Configuring SSL Certificates for the Beacon: To configure SSL certificates for Web transaction and Port Checker service tests, follow the steps given below:

- Configuring Dedicated Beacons: Beacon functionality on an agent requires the the use of an internal Java VM. The use of a Java VM can increase the virtual memory size of the agent by several hundred megabytes. Because of memory constraints, it is preferable to create beacons only on agents that run on dedicated hosts. If you are running large numbers of tests (e.g., several hundred per minute) on a given beacon, you may also wish to install that beacon's agent on a dedicated host. To take full advantage of dedicated hardware, edit the agent's $ORACLE_HOME/sysman/config/emd.properties file. as follows:

- Set the property, ThreadPoolModel=LARGE. This allows the agent to simultaneously run many threads.

- Set the property, useAllCPUs=TRUE. This allows the agent to run on multiple CPUs simultaneously.

- Append -Xms512m -Xmx512m to the agentJavaDefines property. This increases the Java VM heap size to 512 MB.

- Configuring a Web Proxy for a Beacon: Depending on your network configuration, the beacon may need to be configured to use a Web proxy. To configure the Web proxy for a beacon, search for the beacon in the All Targets page. Select the beacon you wish to configure and click Configure. Enter the properties for the Web proxy. For example, to set up www-proxy.us.oracle.com as the beacon's Web proxy, specify the values as the following:

Proxy Host: www-proxy.us.oracle.com

Proxy Port: 80

Don't use Proxy for: .us.oracle.com,.oraclecorp.com

Note:

You cannot play Siebel service tests and Web Transaction (Browser) service tests on the same machine.

Configuring Windows Beacons for Web Transaction (Browser) Playback

To run a Web Transaction (Browser) service test, you need beacons that are running on an 10.2.0.4 or later Management Agent on Windows. The beacon drives an Internet Explorer process. This process runs as the same user as the Management Agent service.

Verifying Web Transaction (Browser) test involves the following 3 steps:

1. Navigate to the Service Tests and Beacons page and select a Web Transaction (Browser) test from the list.

2. Click Verify Test. The Verify Service Test page is displayed.

3. Select a Windows beacon and click Perform Test.

One of the common problems that you may encounter is that the Perform Test does not respond immediately.

There may be several reasons for this delay. Complicated tests may take longer to run. However, the most probable cause for delayed response is when the Internet Explorer process from the beacon is waiting for manual confirmation, which is invisible when run as a process that does not interact with desktop.

You may need to change the browser settings on the beacon machine. These settings need to be changed for the Local Service account and are account specific. Therefore, any changes to the Internet Explorer process that was opened from the Start menu on the beacon machine, will not affect the Internet Explorer process instantiated from the beacon which runs in an invisible window. To make the Internet Explorer window instantiated from the beacon visible:

1. Login as administrator to the Windows machine on which the Management Agent is running.

2. From the Start menu, click Run, type services.msc and click Enter.

3. Find the Management Agent in the list of Windows services, e.g. OracleServiceagent1.

4. Right click the Management Agent and select Properties.

5. Click the Log On tab.

6. Click the Select Allow service to interact with desktop checkbox and click OK.

7. Right click the Management Agent and select Stop, and then select Start.

To check Internet Explorer on the Management Agent machine for any dialog confirmations. For example, SSL Certificates and security warnings.

1. Use the previous procedure to make the Internet Explorer process instantiated from the beacon visible.

2. Launch Enterprise Manager (from any machine), and navigate to the Service Tests and Beacons page of the corresponding service target.

3. Select the Web Transaction (Browser) service test, and click Verify Service Test.

4. From the Verify Service Test page, select the beacon running on the Windows, and click Perform Test.

5. If it is a SSL Certificates issue, From the Windows machine on which the Management Agent is running, you will see an Internet Explorer window open and a Security Alert with a View Certificate option is displayed.

6. Select the Certificate Path tab, click the root certificate, which should have a red cross next to the name, and click the View Certificate button.

7. Click Install Certificate and proceed with the Certificate Import Wizard. (Click Next and Yes for any prompts).

Note:

Other security warnings may also pause the Internet Explorer automation process. Typically, these security warnings have a check box that allow you disable the display of all future warning messages for all Web sites. These warnings may have already been dismissed on the machine where the transaction was recorded.

8. Once this manual step has been performed, the Internet Explorer process should be in auto-pilot mode until the service test is completed. The warning message will not be displayed when you play back the service test next time.

9. Click Perform Test again to make sure the entire service test is completed automatically the second time.

To make the Internet Explorer window instantiated from the beacon invisible, you can repeat the steps 1 to 5, uncheck the Select Allow service to interact with desktop checkbox and continue with step 7.

To configure the proxy setting for Web Transaction (Browser) service tests:

1. Make the Internet Explorer process instantiated from the beacon visible.

2. Launch Enterprise Manager (from any machine), and navigate to the Service Tests and Beacons page of the corresponding service target.

3. Select the Web Transaction (Browser) service test, and click Verify Service Test.

4. From the Verify Service Test page, select the beacon running on the Windows, and click Perform Test.

5. From the Windows machine where the Management Agent is running, you should see two Internet Explorer windows open. From either of the windows, select the Tools > Internet Options.

6. Click the Connections tab and then click LAN Settings, and make all relevant changes there. These changes apply to all service tests running on this beacon.

7. Close both the Internet Explorer windows.

8. Click Perform Test again to make sure the entire service test is completed automatically the second time.

9. Make the Internet Explorer process instantiated from the beacon invisible.

Note:

At any one time, each test run launches two Internet Explorer windows. One of the windows schedules the steps during playback. The other window actually shows the site being played back.

Root Cause Analysis Configuration

You can use Root Cause Analysis (RCA) to filter a set of events to determine the cause of a higher level system, service, or application problem. RCA can help you to eliminate apparent performance problems that may otherwise appear to be root causes but which are only side effects or symptoms of the actual root cause of the problem, allowing you to more quickly identify problem areas. You can view the RCA results on the Home page or Topology page of any service that is currently down. The Topology page allows you to see a graphical representation of the service, system and component dependencies with the targets highlighted that RCA has implicated as causing the service failure.

Before running RCA, you can choose to:

- Configure the tool to run automatically whenever a service fails.

- Disable RCA by changing the default Analysis Mode to Manual.

- Define component tests for the service and thresholds for individual tests.

To configure Root Cause Analysis, follow these steps:

1. From the Service Home page, click Monitoring Configuration.

2. From the Monitoring Configuration page, click Root Cause Analysis Configuration.

3. If the current mode is set to Automatic, click Set Mode Manual to disable RCA. If you choose to perform the analysis manually, you can perform the analysis from the Service home page at anytime by choosing Perform Analysis if the service is down. If the current mode is set for Manual, click Set Mode Automatic to enable RCA when the state of the service and its components change

4. Click the link in the Component Tests column of the table for the key component you want to manage. You can then manage component tests for the service on the Component Tests page by adding, removing, or editing tests. Refer to the Enterprise Manager Online Help for details on defining component tests.

Note:

When you disable RCA and set it back to automatic mode, RCA does not store the previous history results for you, thus providing no history for later reference.

Getting the Most From Root Cause Analysis

Root Cause Analysis (RCA) can provide you with great value by filtering through large amounts of data related to your services and identifying the most significant events that have occurred that are affecting your service's availability. If you are constructing your own services to manage in Enterprise Manager it is important that the services are defined with some thought and planning in order to get the most out of RCA.

The first item to consider in getting the most from RCA is the set of dependencies that your service has on other services or system components. Be sure to identify all of the system components that your service utilizes in order to accomplish its task. If you omit a key component and the service fails, RCA will not be able to identify that component as a possible cause. Conversely, if you include components in the service definition that the service does not actually depend on, RCA may erroneously identify the component as a cause of service failures.

When building service dependencies, keep in mind that you can take advantage of the aggregate service concept that is supported by Enterprise Manager. This allows you to break your service into smaller sub-services, each with its own set of dependencies.

Your services may be easier to manage in the modular fashion, and RCA will consider not only the status of a sub-service (a service that you depend on) but also on any of the system components or service that the sub-service depends on in turn and provides you with the power to encapsulate the services a key component exposes to you in the form of a managed service that your service may then depend on.

The second item to consider in getting the most from RCA is the use of component tests. As you define the system components that your service depends on, consider that there may be aspects of these components that may result in your service failure without the component itself failing. Component tests allow RCA to test the status not only of the target itself but also the status of its key aspects.

The RCA system allows you to create component tests based on any metric that is available for the key component. Remember, this includes any user-defined metrics that you have created for the component, allowing you great flexibility in how RCA tests the aspects of that component should your service fail.

Recording Web Transactions

You can record a transaction using an intuitive playback recorder that automatically records a series of user actions and navigation paths. You can play back transactions interactively, know whether it is internal or external to the data center, and understand the in-depth break-out of response times across all tiers of the Web application for quick diagnosis.

You must install the transaction recorder in your computer to record transactions. The transaction recorder is also used for playing back and tracing transactions. The transaction recorder is downloaded from the Enterprise Manager Grid Control server the first time any of these actions is performed. The transaction recorder requires some Microsoft libraries to be installed in your computer. If these libraries are not present during installation, they are automatically downloaded and installed from the Microsoft site. Make sure that your computer has access to the Internet to download these files. After the installation has been completed, you may need to restart your computer to make the changes effective.

Monitoring Settings

For each service, you can define the frequency (which determines how often the service will be triggered against your application) and the performance thresholds. When a service exceeds its performance thresholds, an alert is generated.

To define metrics and thresholds, click Monitoring Settings for Tests link on the Service Tests and Beacons page. The Metric and Policy Settings page is displayed. Click the Monitoring Settings link. The Monitoring Settings - Thresholds page appears.

- View By Metric, Beacon - In this view, you can click Add Beacon Overrides to override the default threshold values for one or more beacons. In this case, the default thresholds will only be used for beacons without any overrides. Any new beacons added to the service will use the default thresholds. Click Add Metric to add one or more metrics.

- View By Beacon, Metric - In this view, you can click on the Default icon to toggle between Edit and View modes for a specific metric. In the Edit mode, you can modify the parameters of the metric. You can also modify the parameters of the metric for a specific beacon. In the View mode, the default parameters of the metric will be used.
Apart from these procedures, you can also define metrics at the step, and step group level for Web transactions. You can choose either of the following views:

- View By Step, Metric, Beacon: In this view, you can click Add Beacon Overrides to override the default threshold values for one or more beacons. In this case, the default thresholds will only be used for beacons without any overrides. Any new beacons added to the Web transaction will use the default thresholds. Click Add Metric to define thresholds for one or more metrics. Alerts are generated only if the value of the Data Granularity property is set to 'Transaction' for the service tests. For more information on the Web transaction properties, refer to the Create / Edit Service Test - Web Transaction help page in the Enterprise Manager Online Help.

- View By Step, Beacon, Metric: In this view, you can click on the Default icon to toggle between Edit and View modes for a specific metric. In the Edit mode, you can modify the parameters of the metric for a specific beacon. In the View mode, the default parameters of the metric will be used. Alerts are generated only if the value of the Data Granularity property is set to ' Step'.

To define the default collection frequency and collection properties, click the Collection Settings tab on the Monitoring Settings page. You can do the following:

- Specify the default collection frequency for all the beacons. To override the collection frequency for a specific beacon, click Add Beacon Overrides.

- Specify the collection properties and their corresponding values for one or more beacons.

Refer to the Enterprise Manager Online Help for more details on the defining the collection intervals and performance thresholds.

Configuring Aggregate Services

Aggregate services consist of one or more services, called subservices. A subservice is any service created in Enterprise Manager. The availability, performance, business criteria, and usage for the aggregate service depend on the availability, performance, business criteria, and usage for the individual subservices comprising the service. To create an aggregate service, navigate to the Services main page, select Aggregate Service from the Add drop-down list and click Go. The Add Aggregate Service page appears. Creating an Aggregate Service involves the following:

- Specifying the name and time zone for the service.

- Adding the services that make up this aggregate service.

- Establishing the availability definition for the aggregate service. Availability of an aggregate service depends on the availability of its constituent subservices. The availability for a subservice may depend on the successful execution of a service test or on the availability of the system components on which the subservice runs, depending how the subservice was defined.

- Defining the metrics used to measure the performance of your aggregate service. You can add performance metrics from single subservices, or based on statistical aggregations of more than one metric. Once you have selected the performance metrics, you can set the thresholds used to trigger critical and warning alerts, or remove metrics you no longer want.

- Defining the metrics used to measure the usage of your aggregate service. Usage metrics can be based on the metrics of one or more system components. You can add usage metrics from single subservices, or based on statistical aggregations of more than one metric. Once you have selected the usage metrics, you can set the thresholds used to trigger critical and warning alerts, or remove metrics you no longer want.

- Defining the metrics that are used to measure of the performance of business in the organization. These metrics are based on business indicators that can assess the business performance.You can add business metrics from single subservices, or based on statistical aggregations of more than one metric. Once you have selected the business metrics, you can set the thresholds to trigger critical and warning alerts, or remove metrics you no longer want.

After you have created an aggregate service, you can add or remove its constituent subservices, modify the availability definition and add or delete performance or usage metrics. Refer to the Enterprise Manager Online Help for details on these operations.

WARNING:

If you delete or remove a subservice from an aggregate service, the aggregate service performance, usage, and business metrics may be affected if they are based on a deleted subservice's metrics.

Configuring End-User Performance Monitoring

Enterprise Manager allows you to monitor the response time data generated by actual end-users as they access and navigate your Web site. You can gather end-user performance data and monitor the performance of the pages within your Web application. The Web servers such as OracleAS Web Cache, Oracle HTTP Server, and Apache HTTP Server collect the end-user performance data and store it in the log file. Enterprise Manager processes this data and uploads it to the Management Repository. You can then view and analyze this data on the Page Performance page.

To gather the end-user performance data, you must configure one of the following Web servers so that Website activities are logged and stored in the correct format.

- Oracle HTTP Server Based on Apache 2.0

- OracleAS Web Cache

- Apache HTTP Server 2.0 or higher

After you have configured one of these Web servers, you can enable the collection of end-user performance data. You can then view the end-user performance data on the Page Performance page in Enterprise Manager.

Before you configure the Web server, you must do the following:

- Create a Web application target that contains one of these Web servers.

- Make this Web server as a key system component for your Web application. If this Web server is a part of the Redundancy Group, make sure that the Redundancy Group is a key system component of your Web application. To enable end-user performance monitoring, you must configure the specific Web server within the Redundancy Group.

Note:

If you are using the Oracle HTTP Server Based on Apache 2.0, the Redundancy Group is referred to as the HTTP Server HA Group.

The following sections provide instructions on configuring the Web server for End-User Performance Monitoring:

- Configuring End-User Performance Monitoring Using Oracle HTTP Server Based on Apache 2.0 or Apache HTTP Server 2.0

- Configuring End-User Performance Monitoring Using Oracle Application Server Web Cache

Configuring End-User Performance Monitoring Using Oracle HTTP Server Based on Apache 2.0 or Apache HTTP Server 2.0

To enable End-User Performance Monitoring, you can use either of the following Apache server versions:

- Oracle HTTP Server Based on Apache 2.0

- Apache HTTP Server 2.0 or higher (This can be downloaded from http://www.apache.org)

Before configuring either of these Apache server versions, you must perform the following steps:

1. In the Agent Home page, select either Oracle HTTP Server or Apache HTTP Server as a target type.

2. Add the target of the corresponding type and make sure the following properties are set in the Monitoring Configuration page:

- For Oracle HTTP Server, fill in the version number (stdApache10.1.2), Log file directory and Log file name.

- For Apache HTTP Server 2.0, fill in the install home directory, Log file directory and Log file name.

Note:

If the Oracle HTTP Server is installed before the Management Agent has been installed, and is up and running during agent installation, then the target will be discovered automatically. Otherwise you need to manually create the Oracle HTTP Server target and specify the following properties: Machine name, Port number, Version of the Apache Server, Oracle home path, Log file directory (for EUM), Log file name (for EUM) where EUM refers to End-User Performance Monitoring.

3. Make sure you have created the Web application with this Web server target. For details on creating a Web application, refer to the pre-requisites in the "Configuring End-User Performance Monitoring" section.

To configure the Apache server and enable collection of end-user performance data, follow the steps given below:

1. Navigate to the Web Application Home page in the Grid Control Console and click Monitoring Configuration.

2. Click Manage Web Server Data Collection. You will see a table which lists the Web Servers including Oracle HTTP Server Based on Apache 2.0 or higher, Apache HTTP Server version 2.0 or higher, or OracleAS Web Cache.

Figure 14-3 Manage Web Server Data Collection

3. Select the Oracle HTTP Server or Apache HTTP Server from the table and click Configure. Enter the host credentials required for modifying the Apache configuration file.

4. After logging in, you will see a table containing the list of sites that are being hosted by the Apache server. These include a list of virtual hosts defined by the user in the Apache Configuration file. The up and the down arrows under the Monitoring Status column shows the corresponding site is currently being monitored. For each site, check or uncheck the Enable Monitoring checkbox to indicate whether this site is to be monitored. For the site that is to be monitored, enter the log file name in the text box to indicate the location in which the end-user performance data is to be stored. By default, the log file will be created under the logs/directory under Apache root directory. To save the log file in a different directory, enter a file name with the absolute path.

5. Make sure that the log file name and the location you specify here match the Log file name and Log file directory in the Monitoring Configuration page of the Oracle HTTP Server or Apache HTTP Server target.

6. You can also use the one button accelerator to enable all sites or disable all sites all at once.

7. To selectively disable or enable certain URLs on a specific site, select the site, click Set URLs. Click Insert Before or Insert After to create a URL rule and place it in the desired place among all URL rules. A URL rule contains a URL Pattern, URL Pattern Type, and a check box indicating if this URL is to be monitored or not. For example, a URL rule with URL Pattern "abc" and URL Pattern Type "Ends With" and Monitor unchecked means that any URL ending with "abc" will not be monitored by End-User Performance Monitoring. The user can also delete a URL rule, move a URL rule up or down to increase or decrease its priority.

8. After you have made the configuration changes, click OK to go to the Apache Restart page. Restarting the Apache server will finalize all configuration changes, and end-user performance data will be logged by the Apache server.

9. After you have configured the Apache server, you will return to the Manage Web Server Data Collection page. You can now enable the collection of end-user performance data. For more details, refer to "Starting and Stopping End-User Performance Monitoring". If you do not see data after End-User Performance Monitoring has been enabled, refer to the "Verifying and Troubleshooting End-User Performance Monitoring".

Setting up the Third Party Apache Server

To set up the Third Party Apache HTTP Server 2.0, follow these steps:

1. Install the third party Application Server.

2. Install Apache HTTP Server 2.0.

3. Install the plug-in for the Apache HTTP Server 2.0 that was provided by the Application Server.

4. Ensure that the Web application works with the Apache HTTP Server2.0 server. You can then follow the steps to configure the Apache server and enable collection of end-user performance data.

Configuring End-User Performance Monitoring Using Oracle Application Server Web Cache

Enterprise Manager uses data from Oracle Application Server Web Cache to gather statistics about the performance of pages within your Web applications. As a result, you must configure Oracle Application Server Web Cache to ensure that it logs your Web site activity and that the data is in the correct format.

When Oracle Application Server Web Cache is properly configured, Enterprise Manager can begin collecting the end-user performance data and load it into the Oracle Management Repository.

See Also:

"Configuring End-User Performance Monitoring" in the Oracle Application Server Web Cache Administrator's Guide.

The following sections describe how to configure and collect end-user performance data if you are using the OracleAS Web Cache:

- Configuring Oracle Application Server Web Cache 10.1.2

- Configuring Oracle Application Server Web Cache 9.0.4

- Configuring End-User Performance Monitoring Using Earlier Versions of Oracle Application Server Web Cache

- Configuring End-User Performance Monitoring Using Standalone Oracle Application Server Web Cache

Configuring Oracle Application Server Web Cache 10.1.2

To configure the OracleAS Web Cache for End-User Performance Monitoring, follow the instructions in the following sections:

1. Navigate to the Web Application Home page in the Grid Control Console and click Monitoring Configuration.

2. Click Manage Web Server Data Collection. Enterprise Manager displays the Manage Web Server Data Collection page.

3. Select the Web Cache target and click Configure. Enterprise Manager displays the login dialog box for the Oracle Application Server Control.

Tip:

If the login dialog box does not appear or if you see an error message in your browser window, navigate to the Web Cache Home page. Click Administer in the Related Links section. You will be prompted for the user name and password for the Application Server Control. Click Administration and scroll down and click End-User Performance Monitoring.

4. Enter the username and password for the Application Server Control user or the ias_admin account. The password for the ias_admin account is defined during the installation of Oracle Application Server.

5. After you have logged into Oracle Application Server Control, you can then configure the Oracle Application Server Web Cache using the Set Up End-User Performance Monitoring page. Check the Enable End-User Performance Monitoring checkbox and click OK to enable End-User Performance Monitoring at the Web Cache level.

6. At the site-level configuration section, select a site and check Enable Monitoring for that site.

Tip:

Disabling End-User Performance Monitoring at the Web Cache level will override site-level settings.

7. From the drop-down list, select the Access Log Format as access log:WCLF for each site you want to monitor. If this format is not in the list, click Use Required Log Format. This automatically picks up the End-User Performance Monitoring log format.

8. Click the link under the URLs to Monitor column. The URLs To Monitor page is displayed. Click Add Another Row to create a URL rule and place it in the desired place among all URL rules. A URL rule contains a URL Pattern, URL Pattern Type, and a check box indicating if this URL is to be monitored or not. For example, a URL rule with URL Pattern "abc" and URL Pattern Type "Ends With" and Monitor unchecked means that any URL ending with "abc" will not be monitored by End-User Performance Monitoring. The user can also change the priority of the URL rule by clicking Reorder. Click OK to save the changes and return to the Set Up End-User Performance Monitoring page.

9. After you have configured the Web Cache in the Set Up End-User Performance Monitoring page, click OK to save the changes. You will then return to the Web Cache Administration page in Oracle Application Server Control. Click Restart to restart the Web Cache. For more detailed information about configuring these options, click Help on the Set Up End-User Performance Monitoring page.

10. Close the Application Server Control browser window and return to the Manage Web Server Data Collection page in the Grid Control console. You can now enable the collection of end-user performance data. For more details, refer to "Starting and Stopping End-User Performance Monitoring". If you do not see data after end-user performance has been enabled, refer to "Verifying and Troubleshooting End-User Performance Monitoring".

Configuring Oracle Application Server Web Cache 9.0.4

To configure the Oracle Application Server Web Cache Manager 9.0.4, follow the instructions given in these sections:

1. Navigate to the Web Application home page in the Grid Control console and click Monitoring Configuration.

2. Click Manage Web Server Data Collection. Enterprise Manager displays the Manage Web Server Data Collection page.

3. Select the Web Cache target and click Configure. Enterprise Manager displays the login dialog box for the Web Cache Manager.

Tip:

If the login dialog box does not appear or if you receive an error message in your browser window, you may have to start the Oracle Application Server Web Cache Manager. For more information about starting and using Oracle Application Server Web Cache Manager, refer to the Oracle Application Server Web Cache Administrator's Guide.

4. Enter the username and password for the Web Cache administrator account. The first time you log in to the Oracle Application Server Web Cache administrator account, the password is administrator. The password for the ias_admin account is defined during the installation of Oracle Application Server.

5. Enable OracleAS Web Cache logging for End-User Performance Monitoring:

1. Select Logging and Diagnostics and then select End-User Performance Monitoring in the OracleAS Web Cache Manager navigator frame.
You can enable monitoring for a particular Web cache or for an entire site.

2. To enable monitoring for a particular Web cache, select the Web cache from the Cache-Specific End-User Performance Monitoring section and click Enable.
Be sure to enable the Web cache that you are using as a front-end to your Web application.

3. To enable monitoring for the entire site, select the site from the Site-Specific End-User Performance Monitoring section and click Enable.

7. Configure Oracle Application Server Web Cache to use the Web Cache Log Format (WCLF):

1. Select Logging and Diagnostics and then select Access Logs in the OracleAS Web Cache Manager navigator frame.

2. In the Cache-Specific Access Log Configuration table, click Edit Selected and enable the access log for your selected cache.

3. In the Site-Specific Access Log Configuration table, make sure that the Format style of the selected Site Name is WCLF and that it is enabled.

8. Click Apply Changes at the top of the Web Cache Manager window and restart OracleAS Web Cache by clicking Restart on the Web Cache Manager Cache Operations page.

9. Close the Web Cache Manager browser window and return to the Manage Web Server Data Collection page in the Grid Control console. You can now enable the collection of end-user performance data. For more details, refer to "Starting and Stopping End-User Performance Monitoring". If you do not see data after end-user performance has been enabled, refer to "Verifying and Troubleshooting End-User Performance Monitoring".

Configuring End-User Performance Monitoring Using Earlier Versions of Oracle Application Server Web Cache

If you are managing an earlier version of the Oracle Application Server using the Oracle Enterprise Manager 10g Grid Control Console, you can monitor your Web applications with End-User Performance Monitoring, but you cannot configure your Oracle Application Server Web Cache instance from within the Grid Control console.

Instead, you configure End-User Performance Monitoring for Oracle Application Server Web Cache 9.0.2 and 9.0.3 by running the chronos_setup.pl script on the computer that hosts your Oracle HTTP Server.

Using the chronos_setup.pl Configuration Script

Before you begin, consider the following:

- The chronos_setup.pl script is installed in the bin directory of your Management Agent home when you install the Management Agent using the instructions in Oracle Enterprise Manager Grid Control Installation and Basic Configuration.

- You must run the chronos_setup.pl script as an operating system user with the privilege to write to the document root of your Oracle HTTP Server.

- If you have trouble running the script, run it with no arguments to display the help text.

To enable End-User Performance Monitoring for Oracle Application Server Web Cache 9.0.2 or Oracle Application Server Web Cache 9.0.3, you must run the chronos_setup.pl script three times, each time with a different argument:

- Once to configure the document root for each Web server in your Web site

- Once to configure Oracle Application Server Web Cache

- Once to start collecting response time data

The following sections describe each step of enabling End-User Performance Monitoring for Oracle Application Server Web Cache 9.0.2 or Oracle Application Server Web Cache 9.0.3.

Configuring the Document Root For Each Web Server

When you run the chronos_setup.pl script with the webserver argument, the script:

- Creates a new directory inside the document root. The directory is called:

oracle_smp_chronos

- Installs two files into the oracle_smp_chronos directory:

oracle_smp_chronos.js

oracle_smp_chronos.gif

oracle_smp_eum_init.js

oracle_smp_eum_main.js

The oracle_smp_chronos.js must be installed in the document root of each Web server that serves content for your Website.

Note:

If you have more than one document root, you must run the chronos_setup.pl script on each document root.

For example, if Oracle Application Server Web Cache and your Web server are on different machines and an Oracle Management Agent is present on the Web server machine, you must run the chronos_setup.pl script with the webserver option on the Web Server host to configure the document root for the remote Web server.

If Oracle Application Server Web Cache and your Web server are installed on different machines and you have no plans to install a Management Agent or to monitor the Web server, you will need to create a directory called oracle_smp_chronos under the Web server document root directory, and using FTP, place the oracle_smp_chronos.js file in the oracle_smp_chronos directory.

To configure the document root for each Web server:

1. Change directory to the /bin directory in the Management Agent home directory.
For example:

$PROMPT> cd AGENT_HOME/bin

2. Make sure you have write access to the Web server document root directory and then run the script as follows:

$PROMPT> perl chronos_setup.pl webserver location_of_the_webserver_DocumentRoot

An example of a Document Root is as follows:

$ORACLE_HOME/Apache/Apache/htdocs

To find the location of the document root, you can perform either of these steps:

- Log in to the Oracle Application Server Release 2 (9.0.2) Enterprise Manager Web site and navigate to the Oracle HTTP Server Home Page. The document root is displayed in the General section of the HTTP Server Home Page.

- Use a text editor or a command-line search utility to search for the term DocumentRoot in the following Oracle HTTP Server configuration file:

$ORACLE_HOME/Apache/Apache/conf/httpd.conf

Configuring Oracle Application Server Web Cache for End-User Performance Monitoring

To configure Oracle Application Server Web Cache for End-User Performance Monitoring, you run the chronos_setup.pl script with the webcache argument. The script sets up Oracle Application Server Web Cache for End-User Performance Monitoring, and stops and restarts Oracle Application Server Web Cache automatically.

To configure Oracle Application Server Web Cache for End-User Performance Monitoring:

1. Make sure you have write access to the Oracle Application Server Web Cache directory.
For example, if Web Cache is installed in an Oracle Application Server home directory, you will need access to the IAS_HOME/webcache directory.

2. Change directory to the /bin directory in the Management Agent home directory.
For example:

$PROMPT> cd /private/agent_home/bin

3. Run the script as follows:

$PROMPT> perl chronos_setup.pl webcache webcache_installation_directory

Note:

After running chronos_setup.pl, if you cannot restart Oracle Application Server Web Cache, back out of the configuration process by copying the following files back to their original name and location:

- internal.xml<timestamp>

- webcache.xml<timestamp>

Starting End-User Performance Monitoring

To start End-User Performance Monitoring, you run the chronos_setup.pl script with the collection argument. The script creates a collection file for the specified target and restarts the agent.

To start End-User Performance Monitoring:

1. Log in as the user who installed the Management Agent so you have write access to the following directory:

AGENT_HOME/sysman/emd/collection

2. Change directory to the /bin directory in the Management Agent home directory.
For example:

$PROMPT> cd AGENT_HOME/bin

3. Locate the name of the Oracle Application Server Web Cache target.
You can locate the name of the target in one of three ways:

- From the Oracle Enterprise Manager 10g Grid Control Console, locate the Oracle Application Server Web Cache target on the Targets tab. The name listed in the first column of the Target table is the name you must enter as an argument to the chronos_setup.pl script. Note the use of spaces and underscores.

- Search the contents of the targets.xml configuration file, which lists all the targets managed by the Management Agent. Locate the Oracle Application Server Web Cache entry in the file and use the NAME attribute for the Web Cache target. The targets.xml file is located in the following directory of the Management Agent home:

AGENT_HOME/sysman/emd/targets.xml

- Use the emctl config agent listtargets command to list the target names and target types currently being monitored by the Management Agent.

See Also:

"Listing the Targets on a Managed Host".

4. Start the collection for the Oracle Application Server Web Cache target by running the script as follows:

$PROMPT> perl chronos_setup.pl collection webcache_targetname

Note:

If the name of the Oracle Application Server Web Cache target includes spaces, you must use quotation marks around the name

Configuring End-User Performance Monitoring Using Standalone Oracle Application Server Web Cache

Oracle Application Server Web Cache is available as a standalone download from the Oracle Technology Network (OTN). The standalone version of Oracle Application Server Web Cache allows you to improve the performance and reliability of your Web server even if you are not using Oracle Application Server.

If you are using standalone Oracle Application Server Web Cache with a third-party Web server, you can still manage Oracle Application Server Web Cache using the Oracle Enterprise Manager 10g Grid Control Console. As a result, you can also use End-User Performance Monitoring to monitor the Web applications that your users access through Oracle Application Server Web Cache.

Configuring End-User Performance Monitoring for standalone Oracle Application Server Web Cache involves the following steps, which are described in the following sections:

- Installing Standalone Oracle Application Server Web Cache

- Configuring Standalone Oracle Application Server Web Cache

- Enabling End-User Performance Monitoring for Standalone Oracle Application Server Web Cache

Installing Standalone Oracle Application Server Web Cache

To install the standalone version of Oracle Application Server Web Cache:

1. Navigate to the Oracle Technology Network (OTN):

http://otn.oracle.com/software/content.html

2. Locate and select the Oracle Application Server Web Cache download option and follow the links for your operating system.

3. Use the instructions on the OTN Web site to download Oracle Application Server Web Cache.

4. Use the instructions in the Web Cache readme file to install Oracle Application Server Web Cache in its own Oracle Home.

Configuring Standalone Oracle Application Server Web Cache

End-User Performance Monitoring uses data from Oracle Application Server Web Cache to gather statistics about the performance of pages within your Web applications. As a result, Enterprise Manager obtains End-User Performance Monitoring data only when Oracle Application Server Web Cache is configured to improve the performance and reliability of your Web server.

See Also:

Oracle Application Server Web Cache Administrator's Guide for complete instructions for configuring Oracle Application Server Web Cache

Specifically, you must perform the following Oracle Application Server Web Cache configuration tasks:

1. Change the default listening port of your HTTP Server (for example, 7777) to a new port number (for example, 7778) and restart the HTTP Server.

See Also:

"Specifying Listening Addresses and Ports" in the Enterprise Manager Online Help if you are using Oracle HTTP Server and managing the server with Enterprise Manager.
Oracle HTTP Server Administrator's Guide for information about modifying the httpd.conf file if you are not managing the server with Enterprise Manager.

2. Start Oracle Application Server Web Cache and its administration tools.

3. Configure Oracle Application Server Web Cache so it receives requests on the default port previously assigned to your Web server (for example, 7777).

4. Configure Oracle Application Server Web Cache so it so it sends cache misses to your newly defined Web server default port number (for example, 7778), which is also referred to as the origin server.

5. Create an Oracle Application Server Web Cache site and map the site to your origin server.

6. Apply the changes and restart Oracle Application Server Web Cache.

7. Test the installation to be sure Oracle Application Server Web Cache and your Web server are working properly.

Enabling End-User Performance Monitoring for Standalone Oracle Application Server Web Cache

After you have installed and configured Oracle Application Server Web Cache and tested the configuration to be sure your Web site data is being cached, you can then enable End-User Performance Monitoring.

The procedure for enabling End-User Performance Monitoring is similar to the procedures documented earlier in this chapter. Use the Oracle Application Server Control for Web Cache 10.1.2 or Oracle Application Server Web Cache Manager for Web Cache 9.0.4 to configure End-User Performance Monitoring, and use Grid Control to start End-User Performance Monitoring, as described in "Starting and Stopping End-User Performance Monitoring".

Configuring End-User Performance Monitoring for Web Page Extensions

End User Performance Monitoring feature automatically recognizes all pages with extensions htm, txt, jhtml, shtml, jsp, and asp. However, additional configuration is required if a Web Application has pages with extensions that are not recognized automatically. For example, for Web Applications that have pages with .do extension, you will have to make additional configuration so that they get recognized.

To configure end-user performance monitoring for Web page extensions that are not recognized automatically, do these:

1. Access the Web Cache or HTTP Server Home Page.

2. From the Related Links section, click Monitoring Configuration.

3. To specify single page extensions, provide the following value in the property Additional Optional Properties (for EUM)

pageext <appropriate page extension

For example, if the Web page has the extension .do, then provide the following:

pageext do

To specify multiple page extensions, provide the following value:

pageext <appropriate page extension>/<appropriate page extension>

For example, if the Web pages have the extentions .do and .html, then provide the following:

pageext do/html

Configuring End-User Performance Monitoring for Web Pages Having the Same URI

By default, Page Performance reports the performance data for the pages identified by URIs without any query parameters. For example, if the complete URL for petstore search page is /petstore/search?cat=cats, then Page Performance reports data only for /petstore/search.

This works fine if the Web Application pages can be identified by URI uniquely without any query parameters. However, it is not possible to identify the pages if a Web Application has the same URI that is used for all pages. For example, the petstore search page URL /petsore?pageid=search and the petstore cart page URL /petsore?pageid=cart.

To configure end-user performance monitoring for Web pages that have the same URI, do these:

1. Access the Web Cache or HTTP Server Home Page.

2. From the Related Links section, click Monitoring Configuration.

3. Provide the following value in the property Page Identifying Parameters (for EUM)

<query parameter name>

For example, if the URI for the petstore search page is /petsore?pageid=search, the specify the following:

pageid

The query parameters specified can be applicable to all URI paths, or specific to particular URI paths.

For example, if you want all URIs that have a query parameter called 'target' or 'event' to be reported with those query parameters, then specify the following:

target,event

For example, if you want the URIs that have '/em' as the path and have 'target' or 'event' to be reported with those query parameters, then specify the following:

/em:target,event

For example, if you want the URIs that have '/em' as the path to be reported, then specify the following:

/em/console:event

To show how the reported data will look like, here is an example. Consider that the following are the URIs for the application:

/portal/page?tab=home&event=login&id=12312312

/portal/page?tab=home&event=submit&id=553634

/portal/page?tab=admin&event=update&id=23423234

/portal/page?tab=admin&event=cancel&id=6784532

If you do not specify anything, then you will see one URI, that is, "/portal/page".

If you specify 'tab', then you will see two URIs, that is, "/portal/page?tab=home" and "/portal/page?tab=admin".

If you specify 'tab,event', then you will see four URIs (and EUM data for each), that is, the following:

"/portal/page?tab=home&event=login"

"/portal/page?tab=home&event=submit"

"/portal/page?tab=admin&event=update"

"/portal/page?tab=admin&event=cancel"

Starting and Stopping End-User Performance Monitoring

After you have configured the Web server to enable collection, you can then start collecting end-user performance data.

1. Navigate to the Web Application home page in the Grid Control console and click Monitoring Configuration.

2. Click Manage Web Server Data Collection. Enterprise Manager displays the Manage Web Server Data Collection page.

3. In the Interval (minutes) column, enter the interval at which Enterprise Manager will collect performance data.

4. Check the Collection Enabled checkbox.

5. Click Apply, review the changes and confirm by clicking Apply again. End-User Performance Monitoring collection is enabled and data will soon be uploaded to the database and shown under the Page Performance page.

To stop collecting end-user performance data:

1. Navigate to the Manage Web Server Data Collection page.

2. Clear the check box in the Collection Enabled column of the table and click Apply.

3. Click Apply again to confirm the changes.

Verifying and Troubleshooting End-User Performance Monitoring

To verify that End-User Performance Monitoring is working properly:

1. Wait a period of time to allow Enterprise Manager to begin collecting end-user performance data and to start loading the data into the Management Repository. Specifically, you should wait until the next upload of data from the Management Agent to the Management Service. The Management Service then loads the data into the Management Repository. For more information about how Enterprise Manager gathers and uploads to the repository, see Oracle Enterprise Manager Concepts.

2. Navigate to the Web Application home page, select a Web application and navigate to the Page Performance tab. Verify that there is data in the Slowest Response Times table.

3. Another way to verify the existence of end-user performance data, is to note the value of the Number of Unprocessed Samples. Samples for an hour that has not ended are referred to as Unprocessed Samples. For example, data is processed for the time period between 10 am to 11 am, 11 am to 12 pm and so on. Therefore, data from 10 am to 11 am will be considered as Unprocessed Samples if the 11 am boundary has not been crossed or if there is no incoming end-user traffic after 11 am. If this is a non-zero value, click Process Samples. End-user performance data is displayed in the Slowest Response Times table.

4. If you still do not see any data on the Page Performance page, consider the following troubleshooting tips:

1. Be sure you have completed all the steps required to configure End-User Performance Monitoring. Make sure that the Web server you are using to collect end-user performance data, is either OracleAS Web Cache or Oracle HTTP Server Based on Apache 2.0 (stdApache10.1.2), or Apache HTTP Server (2.0 or higher). You can see the Web server version in the Monitoring Configuration page.

2. To monitor Web pages from a third party Application Server, follow the instructions for installing an Apache 2.0 server with the Application Server.

3. Install End-User Performance Monitoring after installing the plug-in for the Application Server.

- When using the Apache Configuration page, log in using the same account used to install Apache.

- If the Apache server is running on a port less than 1024, the server must be started as root. Apache can be started as root with a lower privileged account by changing ownership of bin/httpd to root and setting its setuid flag. When Apache is started as root, the 'User' and 'Group' directives in httpd.conf need to be set to the user who installed the Apache server.

Note:

Only pages with a Content-Type header of text or HTML will be monitored. Pages that pass through the Apache Server with a Content-Encoding header (like gzip) will not be monitored because the JavaScript tag cannot be added to these pages.

- If your Web site uses IFrames and End-User Monitoring is not working on those pages, you will need to switch to the newer JavaScript version with IFrame support. In the <apache root>/conf/eum.conf file, follow the directions for enabling IFrame support.

4. Be sure there is enough activity on your site. If no user is visiting and using your Web application, there may be no end-user performance data to collect or to upload to the Management Repository.

5. Be sure you have waited long enough for the Management Agent on the Web server host to upload data to the repository. Check the Management Agent home page to determine the last time the Management Agent successfully uploaded data to the Management Repository.

6. Check the html source of the URLs that you wanted to monitor: make sure the tag <SCRIPT SRC="/oracle_smp_chronos/oracle_smp_chronos.js"></SCRIPT> has been appended to the HTML source of these URLs.

- If it is present, proceed to the next step.

- If it is not present, check the configuration of your OracleAS Web Cache, Oracle HTTP Server, or Apache HTTP Server. Make sure that all configurations are correct, the site has been enabled, and the Web server has been successfully restarted after saving any configuration changes.

7. Go to the OracleAS Web Cache or Apache server target home page, click Monitoring Configuration, and check if the log file in the defined Log file directory contains any recent data.

- If it does not have data, go to the next step.

- If the log file does contain data and the Web server is OracleAS Web Cache, login to Oracle Application Server Control or Web Cache Manager and make sure that the access log is in WCLF or End-User Performance Monitoring format.

8. Verify that the OracleAS Web Cache / Apache server Monitoring Configuration properties that specify the location and name of the log file are accurate.

9. Check the Web Server target Home page for any collection errors. Often, the collection error will provide information describing why performance data cannot be collected.

10. Navigate to the All Metrics page for the Web server target and check to be sure the APM Mining Performance Details metrics are being collected successfully.

Enabling End-User Performance Monitoring for Third-Party Application Servers

For enabling End-User Performance Monitoring for third-party application servers like IBM WebSphere Application Server, BEA WebLogic Managed Server, and JBoss Application Server, after you configure one of the Web servers as explained in this chapter, you have to enable the Application Server Diagnostics Pack for the Web applications hosted on these servers.

To do so, perform the following steps:

1. Click Setup on the top-right corner of the Grid Control console and navigate to the Overview of Setup page.

2. Click Management Pack Access from the panel to the left.

3. On the Management Pack Access page, select the All Targets option in the View Options section of this page.

4. Select Web Application from the Search menu, and click Go. The table lists all the Web applications monitored.

5. For the Web application for which you want to enable End-User Performance Monitoring, check Application Server Diagnostics Pack and Pack Access Agreed, and then click Apply.

6. Now return to the Web Application Home Page and click Page Performance to see the end-user performance monitoring data that has been collected.

Note:

End-User Performance Monitoring for a Web application is not supported if the J2EE container hosting that application is SSL enabled. This applies to Oracle J2EE containers, that is OC4J, and any non-Oracle J2EE containers for third-party application servers like BEA WebLogic Managed Server, IBM WebSphere Application Server, or JBoss Application Server. To activate End-User Performance Monitoring for such a Web application, disable SSL for that J2EE container.
For information about configuring SSL for Oracle Application Servers, refer to the Security Guide for your Oracle Application Server release. Documentation for all the Oracle Application releases is available from the Oracle Technology Network: http://www.oracle.com/technology/documentation/index.html.

For information about configuring SSL for third-party servers, refer to your third-party documentation.

Managing Forms Applications

A Forms Application target in Enterprise Manager can be used to model and monitor a specific Forms application. To use a Forms Application target, you must ensure that the following prerequisites are met:

- Install the Management Agent on the hosts on which the components of your Forms Application have been installed.

- Verify that all the components for your Forms Application has been discovered so that they can be listed as Enterprise Manager targets.

- Create a system that contains all the components that are required for the Forms Application that is to be monitored. The system can contain an Oracle HTTP Server, Apache HTTP Server or an OracleAS Web Cache. For more details on creating a system, refer to Setting up the System.

- After you have created a system for the Forms Application, you can create a Forms Application target using the Create Service Wizard. See Creating a Service for details. Before you create a service, you must be familiar with the concepts of service management as described in the Oracle Enterprise Manager Concepts.

After you have set up the Forms Application target, you can use it to do the following:

- Record and monitor a Forms transaction. See Recording and Monitoring Forms Transactions for details.

- Measure the End-User Performance of Forms actions such as Commit, Query, Runform, Callform, Openform, and Newform. See Monitoring the End-User Performance of Forms Applications for details.

Recording and Monitoring Forms Transactions

A Forms transaction consists of a set of user actions within a single application when using Forms. For example, an Update Employee Salary transaction may consist of several user actions like open salary form, update salary form, and save salary form. You can record multiple Forms transactions by using the intuitive playback recorder that automatically records a series of Forms actions.

Before recording a Forms transaction, you must do the following:

- Set the permissions of the .java.policy file on each Windows client. See Setting the Permissions of the .java.policy File.

- Ensure that a trusted Enterprise Manager certificate is used. See Using a Trusted Enterprise Manager Certificate.

- Add a certificate to the Enterprise Manager Agent to play back secure Forms transactions. See Adding a Forms Certificate to the Enterprise Manager Agent .

- Configure the Forms server so that Forms transactions can be recorded. See Configuring the Forms Server .

After you have performed these steps, you can install the transaction recorder to record and play back the Forms transaction. See Installing the Transaction Recorder to Record and Play Back Forms Transactions.

Setting the Permissions of the .java.policy File

You must the set the permissions of the .java.policy file on each Windows client on which the Forms transaction is being recorded. To set the permissions, follow these steps:

- Ensure that the .java.policy file is present under the user home directory. If the .java.policy file does not exist, you must create one as follows:

- Create a java.policy (without the ".") file

- Click Start and Run from your Windows desktop.

- Type cmd and click OK.

- At the DOS prompt, rename the file as follows:

move java.policy .java.policy

- After you have created the .java.policy file, set the permissions for each Forms server or Oracle Applications server as follows:

grant codeBase "URL" {

permission java.security.SecurityPermission "putProviderProperty.SunJSSE";

};

where URL needs to be replaced with the code source location of the Forms applet. By specifying the codeBase, you grant permissions to the code present in that location. For example, for an out-of-box Forms installation, you must specify the codeBase as follows:

http://formsServerHost:port/forms/java/*

where formsServerHost and port must be replaced with the host name and port number of the Forms server.

For Oracle Applications, you must specify the codeBase as follows:

http://appsHost:appsPort/OA_JAVA/oracle/apps/fnd/jar/*

where appsHost and appsPort must be replaced with the host name and port number of the Oracle Applications.

Using a Trusted Enterprise Manager Certificate

If you are using secure Enterprise Manager to record a Forms transaction running on Oracle Jinitiator or a Java plug-in, you must ensure that the Enterprise Manager certificate is trusted by Oracle Jinitiator and JPI. For Oracle Jinitiator, you must append the Enterprise Manager certificate to Jinitiator's certdb.txt file. For the Java Plug-in, you must set the certificate as trusted by JPI.

To ensure that the Enterprise Manager certificate is trusted by Jinitiator and JPI, follow these steps:

1. Export the Enterprise Manager certificate to a file.

- When you launch secure Enterprise Manager, if Enterprise Manager is using a self generated certificate, you may see a "Certificate Error". Double click on the error and click View Certificates. The Certificate window is displayed.

- Click the Details tab and then click Copy to file... to export the certificate to a file. The Certificate Export Wizard is displayed.

- Click Next in the Welcome page.

- In the Export File Format page, select Base-64 encoded X.509(.CER) and click Next.

- Click Browse to select the name and the location of the file to which the certificate is to be saved.

- Click Finish. The certificate has now been exported to a file.

2. After the certificate has been exported, you must set the certificate as trusted by Jinitiator or JPI.
For Forms applications running on Oracle Jinitiator:

- Open certdb.txt under [Jinitiator InstallRoot]\lib\security\ directory. Usually Jinitiator is installed under C:\ProgramFiles\Oracle\Jinitiator [version]).

- Use a text editor to open the file to which the certificate has been exported. Copy the contents and append it to certdb.txt.

For Forms applications running on Java plug-in:

- In the Control Panel, double click the Java program that is used to run the Forms application.

- Click the Security tab and then click Certificates.

- From the Certificate Type drop down list, select Secure Site.

- Click Import to import the file to the location in which the Enterprise Manager certificate has been saved.

- Close the certificate windows and the Java Control Panel.

3. Close the browser window. When the Forms application is accessed again, Jinitiator or JPI is restarted. This ensures that the changes to the security settings have been saved.

Adding a Forms Certificate to the Enterprise Manager Agent

To play back a secure Forms transaction, you must add a Forms certificate to the Enterprise Manager Agent by following these steps:

1. Stop the Management Agent by entering the emctl stop agent command.

2. Create an importable certificate file from the forms server certificate (Base64 encoded X.509 format) and name this file as forms.cer.

3. Copy the forms.cer to %AGENT_HOME%/jdk/jre/lib/security/ directory.

4. Run keytool with the following parameters (the keytool executable can be found under the jdk/jre/bin directory)

keytool -import -alias forms -file %AGENT_HOME%/jdk/jre/lib/security/forms.cer -keystore AGENT_HOME%/jdk/jre/lib/security/cacerts

5. You will be prompted for the cacerts password. Enter changeit as the password.

6. Start the Management Agent by entering the emctl start agent command.

For Forms6i, you need to follow these steps:

1. Stop the Management Agent by entering the emctl stop agent command.

2. Obtain forms server certificate in Base64 encoded X.509 format and append to $AGENT_HOME/sysman/config/b64InternetCertificate.txt file.

3. Start the agent by entering the emctl start agent command.

Configuring the Forms Server

Before recording a Forms transaction, you must configure the Forms server by following these steps:

1. Create a system based Forms Application target that contains Forms, OracleAS Web Cache or Oracle HTTP Server / Apache HTTP Server targets. These targets must be a part of the system of the Forms Application. They must also be key components of your Forms Application or part of a key Redundancy Group. If you are using the Oracle HTTP Server, the Redundancy Group is referred to as the HTTP Server HA Group.

2. Set up the Forms server for recording transactions:

1. Navigate to the Forms Application Home page in the Grid Control console and click Monitoring Configuration.

2. Click Enable Forms Transaction Monitoring.
The Enable Forms Transaction Monitoring page is displayed.

3. Select a Forms server from the list and click Configure.
The Configure Forms Server: Login page is displayed.

4. Enter the login credentials of the host on which Forms server is installed and click Continue.
The jar files required for Forms Transaction Monitoring (formsRecorder.jar, jsse.jar, jnet.jar, and jcert.jar) are copied into the Forms applet's archive directory (ORACLE_HOME/forms/java) and a confirmation message is displayed.

For Oracle Applications, the archive directory is located at $JAVA_TOP/oracle/apps/fnd/jar.

5. Click Yes to configure the Forms server and return to the Enable Forms Transaction Monitoring page.

After you have configured the system-based Forms Application target, you can record and play back Forms transactions to monitor the availability of the Forms application. To do so, navigate to the Monitoring Configuration page and click Availability Definition. In this page, change the Availability Definition to Service Test.

Installing the Transaction Recorder to Record and Play Back Forms Transactions

After you have configured the Forms server, you can install the transaction recorder on your computer. The transaction recorder is downloaded from the Enterprise Manager Grid Control server the first time you access the Record Forms Transaction page. The transaction recorder requires some Microsoft libraries to be installed in your computer. Make sure that your computer has access to the Internet to download these files. If these libraries are not present during installation, they are automatically downloaded and installed from the Microsoft site. After you have recorded a Forms transaction, if you need to record another one in the same browser, you must use the same JVM version for the new transaction.

You can record multiple Forms transactions on the Forms Application target and monitor these transactions periodically. Before recording a Forms transaction, ensure that all other Forms applications are closed. When you a record a Forms transaction, make sure that the following parameters are specified correctly:

- Login URL: If you selected the Login Type as Single Sign-On (SSO) or Oracle Applications Login, the Login URL must be explicitly specified.

- Connection Type: This can be:

- Socket: Ensure that the Forms server host name and port number are specified correctly.

- HTTP / HTTPS: If the Connection Type is HTTPS and a non-standard certificate is being used, you must import the certificate into the Agent Home directory.

- Forms Path: This is an optional parameter and points to the absolute path of the forms files (.fmx) on the Forms server. To find the absolute path, launch the Forms Application and view the source HTML file of the Forms launcher window. The path is stored in a variable called xmodule. Example: The path may be stored as /myvol/oracle01/apps/apps_st/appl/fnd/12.0.0/forms/US/.

Note:

This parameter is required only if the Forms transaction has been recorded on one Forms server and played back against a different Forms server with a different installation path.

For more details on recording a Forms transaction and metrics collected, refer to the Enterprise Manager Online Help.

Monitoring the End-User Performance of Forms Applications

The End-User Performance Monitoring utility allows you to measure the response time of your applications by viewing information about how quickly the responses are delivered to the end users. When you access a Forms application, the End-User Performance Monitoring utility measures the response time of Forms actions such as Commit, Query, Runform, Callform, Newform, and Openform.You can monitor the Forms actions and view reports based on the response times experienced by the user. You can also define a Watch List of the most important Forms actions to monitor and view the response metrics of these critical operations at a glance.

Note:

End-User Performance Monitoring is supported with Forms server version 6i Patch 16, 10g R2. For version 6i Patch 16, only the Commit operation can be monitored.

Before you can begin monitoring the End-User Performance of a Forms Application, you must configure the Forms and Web server to enable data collection for End-User Performance Monitoring. To configure the Forms Application for End-User Performance Monitoring, follow these steps:

- Configure the Forms server to enable End-User Performance Monitoring.

- Configure the Web server (OracleAS Web Cache or Oracle HTTP Server / Apache HTTP Server) so that it can be used for End-User Performance Monitoring.

- Enable the collection of end-user performance data.

Configuring the Forms Server for End-User Performance Monitoring

Before you can enable the collection of end-user performance data, you must first configure the Forms server. To configure the Forms server, follow these steps:

1. Navigate to the Forms Application Home page in Enterprise Manager Grid Control.

2. Click Monitoring Configuration.

3. Click Manage Web Server Data Collection.

4. On the Manage Web Server Data Collection page, select the Forms server and click Configure. The Configure Forms Server for End-User Performance Monitoring: Login page is displayed.

5. Enter the host login credentials and click Continue. The Configure Forms for End-User Performance Monitoring: Configuration Sections page is displayed.

6. Select a section and check the Enable Monitoring checkbox to enable End-User Performance Monitoring on that section. Click Enable All or Disable All to enable or disable all the sections. You can also click Add New Section to add a section without affecting existing sections. After adding the section, you can enable End-User Performance Monitoring by selecting the checkbox. You can also delete a section that you have added.

Tip:

A section is a parameter defined in the formsweb.cfg. It specifies which section of Forms configuration the user wants to run. The section usually includes the application name and other relevant parameters which are required for successful execution of the application.

7. Set the value of the End-User Performance Monitoring URL column to http://<hostname:portnumber>/oracle_smp_chronos/oracle_smp_chronos_sdk.gif. The hostname and port number are for the Web Server that is serving the Forms application.

8. After you have configured the Forms server, click OK to save the changes and return to the Manage Web Server Data Collection page.

Configuring the OracleAS Web Cache

You can use the 10.1.2 or 9.0.4 versions of OracleAS Web Cache to collect end-user performance data.

- OracleAS Web Cache 10.1.2: To configure OracleAS Web Cache 10.1.2, follow these instructions:

1. You can configure OracleAS Web Cache by using the Oracle Application Server Control. Navigate to the Forms Application home page in the Enterprise Manager Grid Control.

2. Click Monitoring Configuration.

3. Click Manage Web Server Data Collection.

4. On the Manage Web Server Data Collection page, select the Web Cache target and click Configure. The Application Server Control login dialog box is displayed.

Tip:

If the login dialog box does not appear or if you receive an error message in your browser window, navigate to the Web Cache Home page and click Administer under the Related Links. You will be prompted for the user name and password for Application Server Control. Click Administration, scroll down and click End-User Performance Monitoring.

If Application Server Control is not available, you can also use the Oracle Application Server Web Cache Manager to configure the OracleAS Web Cache for End-User Performance Monitoring. For more information about starting and using Oracle Application Server Web Cache Manager, refer to the Oracle Application Server Web Cache Administrator's Guide.

5. Enter the username and password for the Web Cache administrator account or the ias_admin account. The password for the ias_admin account is defined during the installation of Oracle Application Server.
After you have logged into Oracle Application Server Control, you can configure OracleAS Web Cache from the Set Up End-User Performance Monitoring page.

6. Select the Access Log Format as access log:WCLF for each site from the drop down list. If this format is not in the list, click Use Required Log Format.

7. You will return to the Web Cache Administration page in Oracle Application Server Control. Click Restart to restart the Web Cache. For more detailed information about configuring these options, refer to the Enterprise Manager Online Help.

8. Close the Oracle Application Server Control browser window and return to the Manage Web Server Data Collection page in the Enterprise Manager Grid Control.

- OracleAS Web Cache 9.0.4: To configure OracleAS Web Cache 9.0.4, follow these instructions:

1. You can configure OracleAS Web Cache by using the Oracle Application Server Web Cache Manager. Navigate to the Forms Application home page in the Enterprise Manager Grid Control.

2. Click Monitoring Configuration.

3. Click Manage Web Server Data Collection.

4. On the Manage Web Server Data Collection page, select the Web Cache target and click Configure. A login dialog box is displayed.

Tip:

If the login dialog box does not appear or if you receive an error message in your browser window, navigate to the Web Cache Home page and click Administer under the Related Links. You will be prompted for the user name and password for Application Server Control. Click Administration, scroll down and click End-User Performance Monitoring.

5. Enter the username and password for the Web Cache administrator account. The first time you log in to the Oracle Application Server Web Cache administrator account, the password is administrator.

6. Configure Oracle Application Server Web Cache to use the Web Cache Log Format (WCLF):

- Select Logging and Diagnostics and then select Access Logs in the OracleASWeb Cache Manager navigator frame.

- In the Cache-Specific Access Log Configuration table, click Edit Selected and enable the access log for your selected cache.

- In the Site-Specific Access Log Configuration table, make sure that the Format style of the selected Site Name is WCLF and that it is enabled.

For more details on changing the access_log format, refer to the Enterprise Manager Online Help.

7. Click Apply Changes at the top of the Oracle Application Server Web Cache Manager window and restart Oracle Application Server Web Cache by clicking Restart on the Oracle Application Server Web Cache Manager Cache Operations page.

8. Close the Oracle Application Server Web Cache Manager window and return to the Manage Web Server Data Collection page in the Grid Control console. You can now enable the collection of end-user performance data.

Configuring the Oracle HTTP Server / Apache HTTP Server

You can collect end-user performance data by using Oracle HTTP Server or Apache HTTP Server. Before you use these server, follow these steps:

1. On the Agent Home page, select the Oracle HTTP Server or Apache HTTP Server target type. If you are using a generic third party Apache server, select a Apache HTTP Server target.

2. Add the target of the corresponding type and make sure that the Log file directory and Log file name properties are set in the Monitoring Configuration page.
The Log file directory and Log file name you specify here will be used by the End-User Performance Mining Engine to upload end-user performance data.

3. Note:

If the Oracle HTTP Server is installed before the Management Agent has been installed, and is up and running during agent installation, then the target will be discovered automatically. Otherwise you need to manually create the Oracle HTTP Server target and specify the following properties: Machine name, Port number, Version of the Apache Server, Oracle home path, Log file directory (for EUM), Log file name (for EUM) where EUM refers to End-UserPerformance Monitoring.

4. Create a system target and a Forms Application target. Add the Oracle HTTP Server or Apache HTTP Server target to the system target, and make it a key component of the Forms Application target or a part of a key Redundancy Group target. If you are using Oracle HTTP Server, the Redundancy Group is referred to as HTTP Server HA Group.

5. Navigate to the Monitoring Configuration page for the Forms Application target that contains the Oracle HTTP Server or Apache HTTP Server target. Click Manage Web Server Data Collection. You will see a table which lists the Web Servers including Oracle HTTP Server, Apache HTTP Server, or OracleAS Web Cache.

6. Select the Oracle HTTP Server or Apache HTTP Server from the table and click Configure. Enter the username and password for the host on which the Oracle HTTP Server or Apache HTTP server is installed.

7. After logging in, you will see a table containing the list of sites that are being hosted by the Apache server. These include a list of virtual hosts defined by the user in the Apache Configuration file. The up and the down arrows under the Monitoring Status column shows the corresponding site is currently being monitored. For each site, check or uncheck the Enable Monitoring checkbox to indicate whether this site is to be monitored. For the site that is to be monitored, enter the log file name in the text box to indicate the location in which the end-user performance data is to be stored. By default, the log file will be created under the logs/directory under Apache root directory. To save the log file in a different directory, enter a file name with the absolute path.

8. Make sure that the log file name you specify here matches the Log file directory and Log file name in Monitoring Configuration page of the Oracle HTTP Server or Apache HTTP Server target.

9. You can also use the one button accelerator to enable all sites or disable all sites all at once.

10. After you have made the configuration changes, click OK to go to the Apache Restart page. Restarting the Apache server will finalize all configuration changes, and end-user performance data will be logged by the Apache server.

11. After you have configured the Web server, you must configure the Forms server and enable collection of the End-User Performance data from the Manage Web Server Data Collection Page. For details on configuring the Forms server, refer to the Enterprise Manager Online Help.

Starting and Stopping End-User Performance Monitoring

After you have configured the Forms and Web server to enable collection, you can then start collecting end-user performance data.

1. Navigate to the Web Application home page in the Grid Control console and click Monitoring Configuration.

2. Click Manage Web Server Data Collection. Enterprise Manager displays the Manage Web Server Data Collection page.

3. In the Interval (minutes) column, enter the interval at which Enterprise Manager will collect performance data.

4. Check the Collection Enabled checkbox.

5. Click Apply, review the changes and confirm by clicking Apply again. End-User Performance Monitoring collection is enabled and data will soon be uploaded to the database and shown under the Page Performance page.

To stop collecting end-user performance data:

1. Navigate to the Manage Web Server Data Collection page.

2. Clear the check box in the Collection Enabled column of the table and click Apply.

3. Click Apply again to confirm the changes.

Configuring OC4J for Request Performance Diagnostics

Enterprise Manager can gather critical request performance data about your Web application and display this performance data. This feature can be instrumental when you are diagnosing application server and back-end performance issues.

Before you can begin collecting request performance data, you must do the following:

- Create a Web application target and associate it with a system that contains the OC4J instances to be monitored.

- Make these OC4J instances as key system components for your Web application and enable the logging and tracing capabilities. If these OC4J instances are a part of an OC4J Cluster, make sure that this OC4J Cluster is a key system component of your Web application. To enable request performance monitoring, you must configure the specific OC4J instance within the OC4J cluster.

For more information, see the following:

- Selecting OC4J Targets for Request Performance Diagnostics

- Configuring Interactive Transaction Tracing

- Configuring OC4J Tracing for Request Performance Data

- Additional Configuration for Monitoring UIX Applications

Selecting OC4J Targets for Request Performance Diagnostics

Before you configure the OC4J target to collect request performance data, follow the steps given below to add the target to the Web application.

1. Configure the system where the OC4J targets are defined for the Web application target.

2. Navigate to the Web application Home page and click Monitoring Configuration.

3. Click System Configuration. From the list of system components displayed on this page, select one or more OC4J targets and select the checkbox in the Key Components column. The OC4J targets can now be configured and used to collect request performance data.

Configuring Interactive Transaction Tracing

When you use transactions to monitor your Web application, some of the transactions you create often involve application components such as servlets, Java Server Pages (JSPs), Enterprise Java Beans (EJBs), and database connections. Often, the best way to solve a performance problem is to trace these more complex transactions and analyze the time spent processing each application component.

Enterprise Manager provides a mechanism for tracing these transactions. Use the Service Tests and Beacons link on the Monitoring Configuration page of the Web application target to create your transactions and to trace the transactions as they are processed by the servlets, JSPs, EJBs, or database connections of your application.

However, before you can take advantage of transaction tracing, you must first enable tracing for the OC4J instance used to deploy the application. Each OC4J instance of an OC4J cluster must be configured independently. The OC4J instances of the OC4J clusters selected as key components of the Web application target are displayed on the Manage Web Server Data Collection page.

To enable tracing for an OC4J instance:

1. Navigate to the Web Application Home page and click Monitoring Configuration.

2. Click Manage OC4J Data Collection.
Enterprise Manager displays the Manage OC4J Data Collection page.

3. Select the OC4J to configure and click Enable Logging.
Enterprise Manager opens another browser window and displays the Tracing Properties page for the OC4J instance in the Application Server Control.

If you are prompted to log in to the Application Server Control Console, enter the credentials for the ias_admin administrator's account.

4. Select the following options on the Tracing Properties page:

- Enable JDBC/SQL Performance Details

- Enable Interactive Trace

You can use the default values for most of the tracing properties.

Note:

Turning on the Enable JDBC/SQL Performance Details option allows to you drilldown to actual SQL statements but this may require more resources.

5. Click Apply.
If this is the first time you are enabling OC4J tracing for this application server, Enterprise Manager displays a message stating that the transtrace application is being deployed. The Application Server Control then prompts you to restart the OC4J instance.

6. Click Yes to restart the instance and enable the tracing properties.

7. Return to the Grid Control console.
Tracing is now enabled for the selected OC4J instance.

Configuring OC4J Tracing for Request Performance Data

You must configure OC4J instances to enable tracing so that request performance data can be collected. Each OC4J instance of an OC4J cluster must be configured independently. The OC4J instances of the OC4J clusters selected as key components of the Web application target are displayed on the Manage Web Server Data Collection page. To configure the OC4 instances, follow these steps:

1. Navigate to the Web Application home page and click Monitoring Configuration.

2. Click Manage OC4J Data Collection.
Enterprise Manager displays the Manage OC4J Data Collection page.

3. For the OC4J instance that you used to deploy your application, select the check box in the Collection Enabled column.

4. In the Interval (minutes) column, enter the interval at which to collect OC4J tracing data.
The recommended interval setting is 60 minutes.

5. Select the OC4Js to configure and click Enable Logging.
Enterprise Manager opens another browser window and displays the Tracing Properties page for the OC4J instances in the Application Server Control.

If you are prompted to log in to the Application Server Control Console, enter the credentials for the ias_admin administrator's account.

6. Select the following options on the Tracing Properties page:

- Enable JDBC/SQL Performance Details

- Enable Historical Trace

You can use the default values for most of the tracing properties. However, Oracle recommends that you set the Frequency to Generate Trace File (seconds) field to 3600 seconds (equivalent to 60 minutes).

Note:

Modifying the value in the Trace File Directory field is not supported.

7. Click Apply.
If this is the first time you are enabling OC4J tracing for this application server, Enterprise Manager displays a message stating that the transtrace application is being deployed. The Application Server Control then prompts you to restart the OC4J instance.

8. Click Yes to restart the instance and enable the tracing properties.

9. Return to the Grid Control console.
Request Performance data should begin to appear on the Request Performance page as soon as data for the OC4J instance is collected and uploaded into the Management Repository.

Additional Configuration for Monitoring UIX Applications

If you used Oracle User Interface XML (UIX) to build your application, there is an additional configuration step you must perform before you can monitor the requests of your application.

See Also:

Your JDeveloper documentation for information on using UIX to develop Web applications

Before you can monitor the requests of your UIX application, you must do the following:

1. Enable tracing for the OC4J instance you used to deploy your application, as described in "Configuring OC4J Tracing for Request Performance Data".

2. Locate the following configuration file in the Application Server home directory where you deployed your UIX application:

$ORACLE_HOME/j2ee/OC4J_instance_name/config/oc4j.properties

For example, if you deployed your application in the OC4J instance called "home," locate the following configuration file:

$ORACLE_HOME/j2ee/home/config/oc4j.properties

3. Open the oc4j.properties file using your favorite text editor and add the following line to the end of the file:

oracle.dms.transtrace.dollarstrippingenabled=true

4. Save your changes and close the oc4j.properties file.

5. Restart the OC4J instance.

Setting Up Monitoring Templates

A monitoring template for a service contains definitions of one or more service tests, as well as a list of monitoring beacons. A monitoring template can be used to create service tests on any number of service targets, and specify a list of monitoring beacons.

A monitoring template must be created from a service target. Once the template is created, the user can edit the template, create copies, or delete it. Finally, the user can apply the template to other targets, which creates the service tests on the other targets and adds the monitoring beacons.

To create a Monitoring Template, follow the steps given below:

1. Click Setup to navigate to the main Setup page in Enterprise Manager.

2. Click the Monitoring Templates link in the left panel.

3. Click Create to create a monitoring template.

4. In the target selection box, enter or select a service target and click Continue.

5. In the Monitoring Template General Page, enter the name of the template that you wish to create.

6. Click Tests to add / remove or configure service tests associated with the selected service target. Make the required changes to this page and click OK to save the template to the repository.

After you have created the Monitoring Template, use the Apply option to apply this template to a service test. You can click Edit to modify the template. For more details on these operations, refer to the Online Help.

Configuring Service Tests and Beacons

You can configure the service tests and beacons associated with the template by using the options in the Tests page. A service test-based template contains the following elements:

- Variables: A variable may occur at multiple locations in the service tests. The Variables table allows you to specify default values for all the variables. These default values will be stored in the template along with the variables. You can specify values other than the default while applying the template to a target. You can perform the following operations:

- Add a variable. The variable can consist of letters, numbers and underscores only.

- Rename a variable. When you rename a variable, all variable references in the service tests will be replaced with the new name.

- Remove variables for properties within service tests. If you remove a non-password variable, all references to the variable in test properties will be replaced with the variable's default value

- Replace Text in test properties with a variable definition.

- Service Tests: You can edit the test definition and define variables for various properties. You can select the tests from the original target that are to be part of the template by clicking the Add / Remove button. You can specify whether the service test is a key test and if it should be enabled. You can also click Monitoring Settings to drill down to this page and define metrics and thresholds for the service tests.

- Beacons: Use the Add / Remove button to specify which beacons are to be included in the template. You can also specify whether each beacon is a key beacon.

Refer to the Enterprise Manager Online Help for detailed instructions on these operations.

Configuring Service Levels

A service level rule is defined as an assessment criteria used to determine service quality. It allows you to specify availability and performance criteria that your service must meet during business hours as defined in your Service Level Agreement. For example, e-mail service must be 99.99% available between 8am and 8pm, Monday through Friday.

A service level rule specifies the percentage of time a service meets the performance and availability criteria as defined in the Service Level Rule. By default, a service is expected to meet the specified criteria 85% of the time during defined business hours. You may raise or lower this percentage level according to service expectations. A service level rule is based on the following:

- Business Hours: Time range during which the service level should be calculated as specified in your Service Level Agreement.

- Availability: Allows you to specify when the service should be considered available. This will only affect the service level calculations and not the actual availability state displayed in the console. You can choose a service to be considered up when it is one or more of the following states:

- Up: By default the service is considered to be Up or available.

- Under Blackout: This option allows you to specify service blackout time (planned activity that renders the service as technically unavailable) as available service time.

- Unknown: This option allows you to specify time that a service is unmonitored because the Management Agent is unavailable be counted as available service time.

- Performance Criteria: You can optionally designate poor performance of a service as a Service Level violation. For example, if your Website is up, but it takes 10 seconds to load a single page, your service may be considered unavailable.

- Business Criteria: Business criteria are useful in determining in the health of the business processes for a particular service. You can optionally define business metrics that can affect the Service Level. A Service Level violation occurs when a critical alert is generated for a specified business metric.

Note:

The Business Criteria column is displayed only if one or more key business indicators are associated with the service. Refer to Oracle Enterprise Manager Integration Guide.

- Actual Service Level: This is calculated as percentage of time during business hours that your service meets the specified availability, performance, and business criteria.

- Expected Service Level: Denotes a minimum acceptable service level that your service must meet over any relevant evaluation period.

You can define only one service level rule for each service. The service level rule will be used to evaluate the Actual Service Level over a time period and compare it against the Expected Service Level.

Defining Service Level Rules

When you create a service, the default service rule is applied to the service. However, you must edit the service level rule for each service to accurately define the assessment criteria that is appropriate for your service. To define a service level rule:

1. Click the Targets tab and Services subtab. The Services main page is displayed.

2. Click the service name link to go to the Service Home page.

3. In the Related Links section, click Edit Service Level Rule.

4. On the Edit Service Level Rule page, specify the expected service level and the actual service level and click OK. The expected service level specifies the percentage of time a service meets the performance, usage, availability, and business criteria defined in the Service Level Rule. The actual service level defines the baseline criteria used to define service quality and includes business hours, availability, performance criteria, usage criteria, and business criteria.

Note:

Any Super Administrator, owner of the service, or Enterprise Manager administrator with OPERATOR_TARGET target privileges can define or update the Service Level Rule.

Viewing Service Level Details

You can view service level information directly from the either of the following:

- Enterprise Manager Grid Control Console -From any Service Home page, you can click on the Actual Service Level to drill down to the Service Level Details page. This page displays what Actual Service Level is achieved by the service over the last 24 hours/ 7 days / 31 days, compared to the Expected Service Level. In addition, details on service violation and time of each violation are presented in both graphical and textual formats.

- Information Publisher - Information Publisher provides an out-of-box report definition called the Services Dashboard that provides a comprehensive view of any service. From the Report Definition page, click on the Services Monitoring Dashboard report definition to generate a comprehensive view of an existing service. By default, the availability, performance, status, usage, business, and Service Level of the service are displayed. The Information Publisher also provides service-specific report elements that allow you to create your own custom report definitions. The following report elements are available:

- Service Level Details: Displays Actual Service Level achieved over a time-period and violations that affected it.

- Service Level Summary: Displays service level violations that occurred over selected time-period for a set of services.

- Services Monitoring Dashboard: Displays status, performance, usage, business, and service level information for a set of services.

- Services Status Summary: Information on one or more services' current status, performance, usage, business, and component statuses.
Refer to the Online Help for more details on the report elements.

Configuring a Service Using the Command Line Interface

Using the Command Line Interface, you can define service targets, templates and set up alerts. EM CLI is intended for use by enterprise or system administrators writing scripts (shell/batch file, perl, tcl, php, etc.) that provide workflow in the customer's business process. EM CLI can also be used by administrators interactively, and directly from an operating system console. Refer to Enterprise Manager Command Line Interface Guide for details.

Troubleshooting Service Tests

This section lists some of the common errors you may encounter while using the Forms and the Web Transaction test type. The following topics are covered here:

- Verifying and Troubleshooting Forms Transactions

- Verifying and Troubleshooting Web Transactions

Verifying and Troubleshooting Forms Transactions

The section covers the following:

- Troubleshooting Forms Transaction Playback

- TTroubleshooting Forms Transaction Recording

- Troubleshooting End-User Performance of Forms Transactions

Troubleshooting Forms Transaction Playback

This section lists some of the common errors you may encounter while playing back a Forms transaction and provides suggestions to resolve these errors.

1. Error Message: Connection to Forms Server is lost. Possible version mismatch between agentjars and formsjars.
Possible Cause: The transaction was recorded using an out-of-the-box Forms version.

Solution: Verify the version of the Forms Application that you are running by checking the version number in the About Oracle Forms Online Help. If this version is not supported, follow the steps listed under Error Message 2.

2. Error Message: Version Not Supported <forms_version>
Possible Cause: The machine on which the beacon has been installed does not contain the necessary forms jar files.

Solution: To resolve this error, follow these steps:

1. Login to the system on which the Forms server has been installed. Locate the frmall.jar (if you are using Forms 10.1 or later) or f90all.jar (if are using Forms 9.0.4 or later) under the $FORMS_HOME/forms/java directory.

2. Login to the system on which the beacon has been deployed and copy the jar file to the $ORACLE_HOME/jlib/forms/<version>/ directory. The version you specify here should be the same as the version string in the error message. Make sure that the directory is empty before you copy over the jar file.

If you are using Oracle Applications R12 and you encounter this error, follow these steps to resolve the error:

1. Login to the system in which the Oracle Application server has been deployed. Locate the following files:

$JAVA_TOP/oracle/apps/fnd/jar/fndforms.jar

$JAVA_TOP/oracle/apps/fnd/jar/fndewt.jar

2. Login to the system on which the beacon has been deployed and copy these files to the $ORACLE_HOME/jlib/forms/apps/ directory. Make sure that the directory is empty before you copy over the jar files.

Note:

You cannot monitor two deployments of Oracle Applications from the same beacon if different versions of Oracle Applications have been used.

3. Error Message: Forms URL is not pointing to the forms servlet.
Possible Cause: When the Forms transaction was recorded, the location of the forms servlet could not be determined.

Solution: Make sure that the Forms URL Parameter is pointing to the forms servlet. It should be http://<hostname>:<port>/forms/frmservlet for Forms10g or http://<hostname>:<port>/forms/f90servlet for Forms 9i. This parameter is automatically set by the Forms Transaction Recorder. But if it has not been set, you can locate the URL by following these steps:

- Launch the Forms application.

- View the source HTML file in the Forms launcher window.

- Locate the xsurl variable. The URL is stored in this variable.

4. Error Message: Could not connect to <machine name>.
Possible Cause: The machine on which the beacon has been installed cannot access the Forms Application.

Solution: Make sure the machine on which the beacon has been installed can access the Forms Application and firewalls have been properly configured. Support for playing back Forms transactions through proxy server is not available in this release.

5. Error Message: Invalid module path in the initial message.
Possible Cause: The transaction may have been incorrectly recorded or may be corrupt.

Solution: Try to record the transaction again.

6. Error Message: Cannot connect to login server.
Possible Cause: This error may occur due the following reasons:

- The Login URL that you have specified may be incorrect.

- An invalid HTTPS certificate may have been provided for the login server.

Solution:

- Verify that the Login URL is correct.

- If you are using HTTPS to connect to login server, make sure the certificate on the server is written for the login server machine itself. Make sure the SSL Certificate is imported into Agent and the CN of the certificate matches the host name of the login Server URL.

Troubleshooting Forms Transaction Recording

This section lists some troubleshooting steps that you can use when the Forms transaction cannot be recorded successfully.

1. Make sure that all your Internet Explorer instances are closed and no java runtime programs are open.

2. Start recording again with the java console open. You can view any exceptions or error messages displayed on the console.

3. You should now see the text "Forms Transaction Recorder Version: <version number>" on the console. If this text is displayed, proceed to step 5. If you do not see the text, check if the formsRecorder.jar has been copied to the Forms archive directory. You can perform this check using either of the following methods:

1. Navigate to the Forms archive directory and check if the formsRecorder.jar file is present in the directory.

2. Navigate to the Enable Forms Transaction Monitoring page, select the corresponding Forms server target and click Configure. Enter the host credentials to see if the Forms Transaction Recorder has already been configured on this Forms server. If the formsRecorder.jar is not present in the Forms archive directory, follow the steps in the Configuring the Forms Server section to configure your Forms server for transaction monitoring. After ensuring that the formsRecorder.jar is present in the archive directory of the Forms server, go back to Step 1 and try recording again.

4. If you see an exception related to the java .policy file displayed on the java console, check the file to ensure that it has the required content and is in the right location. If any errors are found, you must fix these errors and try recording again. See Setting the Permissions of the .java.policy File.

5. If the recording still fails, check if the Enterprise Manager Certificate has been imported to the secure site as described in Using a Trusted Enterprise Manager Certificate. If the certificate has not been imported, you must import it and try recording again. See Using a Trusted Enterprise Manager Certificate.

Troubleshooting End-User Performance of Forms Transactions

This section lists troubleshooting steps that you can use when the Forms transaction End-user Performance Monitoring (EUM) data is not being displayed.

1. Ensure that the Forms server is configured with EUM.
From the Manage Web Server Data Collection page, select the Forms server and click Configure. Log in using the credentials of the host where the Forms server is installed. Ensure that the correct Forms configuration section has been configured with EUM enabled and that the correct EUM URL is specified. Go to the Forms application URL (with the correct configuration section) and perform "Save" or "Query" actions to generate EUM traffic.

2. Ensure that the Web server is configured to log End-User Performance Monitoring data.
From the Manage Web Server Data Collection page, select the Web server and click Configure.

If you are using a Web Cache to log EUM data, login to the Web Cache Administration page or Web Cache Manager and check if the access_log file is set to either End-User Performance Monitoring or WCLF format. End-User Performance Monitoring data is logged into Web Cache's access_log.

If you are using HTTP Server or Apache HTTP Server, log in using the credentials of the host where the HTTP Server is installed. Then check if EUM has been enabled and note the path of the log file in the configuration page.

3. Ensure that the EUM log file is being generated.
Go to the location of the End-User Performance Monitoring log file, open the log file and search for word "sdk".

"sdk" entries indicate that there is EUM traffic and that the monitoring configuration is correct. In this situation, more time is required to collect end-user performance data. If the log file exists and "sdk" entries are found, go to step 4.

4. Check the Monitoring Configuration page of the Web Cache or HTTP Server target to ensure the parameters "Log File Directory (for EUM)" and "Log File Name (for EUM)" match that of the log file path shown on the configuration page.

5. Another way to verify the existence of end-user performance data, is to note the value of the Number of Unprocessed Samples on the Page Performance page of the Forms application. Samples for an hour that has not ended are referred to as Unprocessed Samples. For example, data is processed for the time period between 10 am to 11 am, 11 am to 12 pm and so on. Therefore, data from 10 am to 11 am will be considered Unprocessed Samples if the 11 am boundary has not been crossed or if there is no incoming end-user traffic after 11 am. If this is a non-zero value, click Process Samples. End-user performance data is displayed in the Slowest Response Times table.

Verifying and Troubleshooting Web Transactions

This section lists some of the common errors you may encounter while recording and playing back Web Transactions.

1. Scenario: Verify Service Test displays: Connection establishment timed out -- http://..../
Possible Cause: The beacon can only access that URL via a proxy server and it has not been configured.

Solution: From the All Targets page, select the beacon, click Configure and set the beacon proxy setting.

2. Scenario: Verify Service Test displays: Authorization Required -- https://...../
Possible Cause: The Basic Authentication information is not recorded automatically.

Solution: To resolve this error, follow these steps:

1. From the Service Tests and Beacons page, select the service test, click Edit.

2. Make sure you enter all the Basic Authentication information: Username, Password, and Realm.

Note:

Realm usually appears above the Username label in the Browser's authorization dialog box.

3. Scenario: Verify Service Test displays sun.security.validator.ValidatorException: No trusted certificate found -- https://....../.
Possible Cause: The beacon does not know about this SSL Certificate.

Possible Solution: From the Service Tests and Beacons page, select the service test, and click Edit. Under Advanced Properties, and set Authenticate SSL Certificates to No.

4. Scenario: Verify Service Test displays: Timeout of 300000 exceeded for https://....../ Response time = 3000000
Possible Cause: The test may be too complex to complete within the allotted time. Or, this may be an actual performance issue with the server.

Possible Solution: From the Service Tests and Beacons page, select the service test, and click Edit. If this is not a server performance issue, under Advanced Properties, increase the Timeout Value.

5. Scenario: The Verify Service Test option reports that the service as down, but the Web application is up and you can successfully play back the Web transaction.
Possible Cause: The Web application is only compatible with Internet Explorer or Mozilla-based browsers.

Possible Solution: From the Service Tests and Beacons page, select the service test, and click Edit. Under Advanced Properties, set the User Agent Header as Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) OracleEMAgentURLTiming/3.0.

Note:

For Grid Control 10.2.0.4 and beyond, this User Agent Header is set automatically during Web transaction recording.

6. Scenario: Test Performance Page does not show any step metrics.
Possible Cause: By default, only transaction-level metrics are collected.

Possible Solution: From the Service Tests and Beacons page, select the service test, click Edit, and set Data Granularity to Step.

15 Extending Enterprise Manager

Enterprise environments consist of a wide variety of components: OS platforms, hardware, software, network, and storage devices. All of these components work in concert to deliver critical information and functionality required to keep enterprise operations performing optimally and providing information to make important business decisions. While Oracle Enterprise Manager Grid Control allows you to monitor and manage a variety of components out-of-box, you may want to monitor third party components or custom applications specific to your environment. For example, with this release, you can seamlessly monitor WebLogic and WebSphere application servers. Additional plug-ins are being developed and will be announced as they become available.

In addition, you can use the same mechanism used by Oracle and partners to extend Enterprise Manager to monitor custom components via modular Management Plug-ins. Once a plug-in is defined, you use the Enterprise Manager Grid Control console to deploy the new plug-in throughout your enterprise environment.

This chapter discusses the following topics:

- Benefits of Extending Enterprise Manager

- Developing Management Plug-ins

- Management Plug-in User Interface

Benefits of Extending Enterprise Manager

Extending Enterprise Manager for monitoring additional components provides the following benefits:

- Centralize management information in a single console: When deployed, components defined by the Management Plug-in automatically appear in the Grid Control console. Being able to monitor all targets in your environment provides a consolidated view of your entire enterprise, thus allowing you to monitor and manage all components from a central point.

- Extend Enterprise Manager's monitoring and management features to non-Oracle components: Newly added components automatically inherit Enterprise Manager's powerful monitoring and management features, such as: alerts, policies, blackouts, monitoring templates, groups/systems, configuration management, and enterprise reporting.

- Comprehensive Service-Level Management: By managing all of your enterprise components with Enterprise Manager Grid Control, you can more fully utilize the Service-Level Management features offered in Enterprise Manager Grid Control.

Developing Management Plug-ins

Management Plug-ins allow you to seamlessly monitor and manage non-Oracle components from the Grid Control console by providing an easy way to specify new classes of components for Enterprise Manager to monitor. Once you have created the new Management Plug-in, you then deploy it to Management Agents within your enterprise.

The complete lifecycle of extending Enterprise Manager consists of five stages:

1. Design your Management Plug-in.

2. Develop the plug-in.

3. Validate the plug-in.

4. Package the plug-in into a portable archive file.

5. Deploy the Management Plug-in throughout your Enterprise Manager environment.

About the Oracle Management Agent

In order to understand what it takes to develop a Management Plug-in, it is important to study the architecture of the Oracle Management Agent. Oracle Management Agents are deployed on each host, and are responsible for monitoring all components on that host. Out-of-box, the Management Agent knows how to monitor default target types, such as the Oracle Database. In order to monitor a particular target type, the agent uses two XML files:

- Target Type Metadata File: specifies the metrics that should be monitored for this target type, and methods to retrieve those metrics.

- Target Type Default Collections File: specifies the default collection intervals and alert thresholds for each of the target metrics.

For each metric, the Management Agent retrieves the appropriate data using fetchlets, which are parameterized data access mechanisms that take arguments for input and return formatted data. The following table lists available fetchlets.

Table 15-1 Fetchlets

Fetchlet

Description

OS Command

Obtains metric data by executing OS commands (either individually or from scripts) that return a standard out data stream.

SQL

Executes a given SQL statement on an Oracle Database as a given user.

JDBC

Executes SQL against any JDBC-enabled database.

SNMP (v.1)

Polls an SNMP agent on a given host for corresponding instances given a list of object identifiers (OIDs). An object identifier (OID) corresponds to either a MIB variable instance or a MIB variable with multiple instances.

HTTP Data

Contacts a Web Server at a URL, and parses returned data.

URLXML

Obtains the XML content of a given URL, and extracts information based on a given pattern.

WBEM

Accesses a CIMOM and retrieves requested information using the specified CIM class.

JMX

Fetches data from a JMX-enabled server

Java Wrapper

Executes Java code to return data.

SQL Timing

Times a SQL operation.

URL Timing

Gets the contents of a given URL, timing not only the base page source but any frames or images in the page as well.

The Agent uses the information in the Target Type Default Collections file to determine, for each target, the metrics that need to be collected and the corresponding collection frequency. It then uses information in the Target Type Metadata Files to determine how to collect the data. Based on this information, the Agent uses the appropriate fetchlet to retrieve data from the monitored targets.

Designing your Management Plug-in

The design stage of the management plug-in creation process involves defining the monitoring parameters required to accurately monitor and manage your new component. This involves:

- Identifying performance and configuration metrics that should be collected.

- Determining how often each metric should be collected. Oracle recommends that the collection frequency for any metric should not be less than once every five minutes.

- Based on customer-specific operational practices, specifying default warning and/or critical thresholds on these metrics. Whenever a threshold is crossed, Enterprise Manager generates an alert, informing administrators of potential problems.

Developing Your Management Plug-in

Developing a plug-in based on the requirements identified in the design phase involves:

1. For each metric, determining the appropriate component-level API that exposes the metric.

2. Mapping a method used to retrieve a particular metric to a 'Fetchlet' provided with the Agent, writing appropriate scripts to retrieve metrics, and determining input parameters for fetchlets. For example, the OS fetchlet will require an OS script, and a URL Timing fetchlet will require a URL parameter.

3. Defining two XML files (Target Type definition file and Default Collections file) that the agent will use to monitor the new target type. This involves declaring appropriate metrics, metric collection methods, collection frequencies, and metric thresholds in the specified XML format.

The example below is an excerpt from a target-type Metadata file showing a sample metric declaration for collecting CPU Utilization. The OS Command fetchlet is used to retrieve the metric value by running an existing Perl script.

Example 15-1 Target Type Metadata File Excerpt

<Metric NAME="Load" TYPE="TABLE">

<TableDescriptor>

<ColumnDescriptor NAME="CPU Utilization" TYPE="NUMBER"/>

</TableDescriptor>

<QueryDescriptor FETCHLET_ID="OS">

<Property NAME="scriptsDir" SCOPE="SYSTEMGLOBAL"> scriptsDir     </Property>

<Property NAME="script" SCOPE="GLOBAL"> %scriptsDir% /cpu_util.pl </Property>

</QueryDescriptor>

</Metric>

Once a metric has been defined in the Target Type Metadata file, the next step is to define metric collection frequency and appropriate metric thresholds. The following example includes a sample collections file for the MyDatabase target type, where CPU Utilization is collected every 10 minutes, and a critical alert should be generated when the metric value is greater than ("GT") ninety percent.

Example 15-2 Target Type Default Collections File

<TargetCollection TYPE="MyDatabase">

<CollectionItem NAME="Load">

<Schedule>

<IntervalSchedule INTERVAL="10"/>

</Schedule>

<Condition column_name ="CPU Utilization" critical="90" operator="GT"/>

</CollectionItem>

</TargetCollection>

Validating the Plug-in

Successful integration of new target types with the Enterprise Manager framework relies on accurate XML. It is essential that the Target Type Metadata file and Default Collections file are syntactically and structurally correct. It is also critical to simulate a runtime data collection for a new target type to ensure that proper data is collected, as well as ensuring that there are no adverse performance effects. To facilitate the validation process, Enterprise Manager provides a tool call ILINT.

Packaging the Plug-in

You package all of the files associated with a Management Plug-in (metadata files, monitoring scripts, report files) into a Management Plug-in Archive (MPA) via EM CLI. The MPA is a ".jar" file that contains one or more Management Plug-ins, thus facilitating easy import/export of plug-ins.

To run the EMCLI, open a terminal window on any machine where the EMCLI client is installed and execute the add_mp_to_mpa verb. The following example demonstrates verb usage:

Example 15-3 Using the EMCLI to create a Management Plug-in Archive

emcli add_mp_to_mpa

-mpa="/my_dir/my_new_type.jar"

-mp_version="2.0"

-ttd="/my_dir/ttd/new_type.xml"

-dc="/my_dir/dc/new_type.xml"

-file="MONITORING_SCRIPT:/my_dir/script1.pl"

-file="MONITORING_SCRIPT:/my_dir/script2.pl"

-func_desc="Management Plug-in to define target type new_type"

After you have added one or more Management Plug-ins to the archive file, you are ready to deploy your plug-in to Management Agents in your enterprise.

Deploying the Management Plug-in

After a Management Plug-in Archive file is created, Enterprise Manager makes it simple to deploy the new Management Plug-in. The deploy operation is performed directly from the Grid Control console and consists of three steps:

1. Import the Management Plug-ins into the Grid Control console.

Figure 15-1 Management Plug-in Import Page

Description of "Figure 15-1 Management Plug-in Import Page"

Once imported, the plug-ins appear on the Management Plug-ins page, and can be deployed to any number of Management Agents simultaneously.

Figure 15-2 Management Plug-ins Page

Description of "Figure 15-2 Management Plug-ins Page"

2. Deploy the Management Plug-in(s) to the appropriate Agents.
Management Plug-ins can be deployed simultaneously to any number of Agents without causing any downtime to production environments. Enterprise Manager automatically copies appropriate metadata files and monitoring scripts to the selected Agents.

Figure 15-3 Deploy Management Plug-in Page

Description of "Figure 15-3 Deploy Management Plug-in Page"

Once the Management Plug-in is deployed, Management Agents can monitor targets of the new type.

3. Discover the Management Plug-ins at the Agent.
Once a Management Plug-in is deployed, the Agent knows how to monitor the new target type. At this point, all that remains is to instruct the Agent to start monitoring one or more instances of the new target type. This involves discovering targets from the Agent home page by specifying a small set of target properties, such as SID and Username for a database.

Management Plug-in User Interface

Each Enterprise Manager managed target has a default home page that provides a consolidated, at-a-glance view of its health and performance status. The home pages also provide direct access to configuration information and management/monitoring features such as Blackouts, Monitoring Configuration, and Alert History. This rich user interface is available to Management Plug-Ins out-of-box, without any custom code.

Figure 15-4 Default Target Home Page

Description of "Figure 15-4 Default Target Home Page"

Enhancing Default Target Home Pages with Real-Time Reports

Enterprise Manager provides a powerful reporting framework called Information Publisher, which allows you to:

- Create custom reports against the Management Repository via the Grid Control console or by using the Information Publisher PL/SQL API.

- Generate reports on a schedule and share them with the entire user community.

- Create out-of-box monitoring, configuration, and service-level reports.

When developing a Management Plug-in, you can enhance information available from a default target home page by developing a management report definition using the Information Publisher PL/SQL API. These report definitions can be packaged with the plug-in, and are available from the Reports tab on the default target home page.

See Also:

Chapter 10, "Information Publisher"

Figure 15-5 Default Target Home Page with Real-Time Report

Description of "Figure 15-5 Default Target Home Page with Real-Time Report"

Index

A  B  C  D  E  F  G  H  I  J  L  M  N  O  P  R  S  T  U  V  W

A

accessing

compliance management pages, 13.2.1

accounts

types of, 2.1.2

adaptive alert thresholds, purpose of, 1.2.1.1.2

ADDM, 1.1

purpose of, 1.1

administrators

administrator account, 2.1.2.2

setting up, 2.1.2

super administrator account (SYSMAN), 2.1.2.1

system and target privileges, 2.1.1

Agent Registration Password, 3.4.3

changing, 3.4.7

Agent Upload Problems

default notification rule, 4.1.2.3

AGENT_HOME/network/admin, 3.4.9.4, 3.4.9.4

Agents Unreachable

default notification rule, 4.1.2.3

aggregration and purging policies

See data retention policies

Alert Details page, picture of, 1.5

alerts, 11.2.3.1

automated responses, 1.2.4

corrective actions, 1.2.4

notification methods, 1.2.3

notifications for, 1.2.3

purpose of, 1.2.2

server-generated, 1.1

Warning Alerts page, 1.5

analyzing

end-user client data, 2.1.11

job activity, 7.3

Application Performance Management, 3.9.3

Application Server Availability and Critical/Warning States

default notification rule, 4.1.2.3

Application Server Control

starting and stopping, 8.3

starting and stopping on Windows systems, 8.2.2

archive logging

for Management Repository database, 12.1

asynchronous I/O, 11.2.4.5

automated

responses to alerts, 1.2.4

Availability History Report, picture of, 6.3, 10.2

B

Backup, 11.3.1

baseline normalized views, 1.2.1.1.2

baselines, 11.2.2

adaptive thresholds, 1.2.1.1.2

for metrics, 1.2.1.1.2

normalized views, 1.2.1.1.2

BEA WebLogic

Oracle ecosystem and, 1.2.1

Beacons, 11.2.4.6

introduction, 1.2

monitoring Web Applications over HTTPS, 3.9.3

benefits of

extending Enterprise Manager, 15.1

Information Publisher, 10.1

blackouts

command-line interface and, 1.2.5

controlling with emctl, 8.7.4

examples, 8.7.4

functionality of, 1.2.5

retroactive, 1.2.5

setting up, 2.1.5

buffer cache, 11.2.4.3

C

capacity

predicting, 11.2

Certificate dialog box

Internet Explorer, 3.9.2.1, 3.9.3

Checkpoint Firewall, Oracle ecosystem and, 1.2.1

Client System Analyzer (CSA)

setting up, 2.1.11

collecting end-user client data, 2.1.11

Command Line Interface (EMCLI)

blackouts and, 1.2.5

Management Plug-in Archive, 15.2.5

Compare Monitoring Template feature, 1.3

Compare Targets, picture of, 1.5

compliance

management pages, accessing, 13.2.1

violations, investigating, 13.2.2

compliance evaluation

scheduling, 13.3.1

setting up, 13.3

configuring

blackouts, functionality of, 1.2.5

corrective action library, 2.1.9

Management Plug-ins, 2.1.10

monitoring templates, 1.3

notification rules, 2.2.3.1

notification schedule, 2.2.3.2

preferred credentials, 2.2.2

setting up

administrators, 2.1.2

blackouts, 2.1.5

management pack access, 2.1.7

monitoring templates, 2.1.8

notification methods, 2.1.3

patching, 2.1.4

registration passwords, 2.1.6

roles, 2.1.1

target subtabs, 2.2.4

Configuring Services, 14

Availability, 14.3, 14.4.1

Beacons, 14.3

Key Beacons, 14.3

Local Beacon, 14.3

Service Test-Based, 14.3, 14.4.1

System-Based, 14.3, 14.4.1

Command Line Interface, 14.13

Create, 14.3, 14.9

End-User Performance Monitoring, 14.1, 14.8

Access Log Format, 14.8.2.1

chronos_setup.sh, 14.8.2.3.1, 14.8.2.3.3

EUM, 14.8.1

Manage Web Server Data Collection, 14.8.1, 14.8.2.2

Set URLs, 14.8.1

Standalone Web Cache, 14.8.2.4

Unprocessed Samples, 14.8.6

URL Pattern, 14.8.2.1

Web Cache Log Format, 14.8.2.2

Web Cache Manager, 14.8.2.2

Interactive Transaction Tracing, 14.1

J2EE Server Activity, 14.1

Metrics

Performance, 14.3

Usage, 14.4.3

Usage Metrics, 14.3

Monitoring Settings, 14.6

Beacon Overrides, 14.6

Collection Settings, 14.6

Data Granularity, 14.6

Frequency, 14.6

Monitoring Templates, 14.11

Beacons, 14.11.1

Service Tests, 14.11.1

Service Tests and Beacons, 14.11.1

Variables, 14.11.1

Oracle Application Server 10g (9.0.4), 14.1

Performance, 14.3

Performance Metrics, 14.4.2

Aggregation Function, 14.4.2

Recording Transactions, 14.1, 14.5, 14.9.1

Request Performance, 14.1, 14.1, 14.10

Correlate Requests, 14.1

Database Connections, 14.10.2

Enterprise Java Beans (EJBs), 14.10.2

JDeveloper, 14.10.4

Manage OC4J Data Collection, 14.10.2

OC4J Cluster, 14.10

OC4J Instances, 14.10

OC4J Tracing, 14.10.3

Oracle Application Server 10g (9.0.4), 14.1

Oracle User Interface XML (UIX), 14.10.4

Service Tests and Beacons, 14.10.2

Tracing Properties, 14.10.2

Root Cause Analysis, 14.1, 14.4.1, 14.4.6

Component Tests, 14.4.6

Topology, 14.4.6

Topology Viewer, 14.1

Service Level Rules, 14.12

Actual Service Level, 14.12

Availability, 14.12

Business Hours, 14.12

Expected Service Level, 14.12

Information Publisher, 14.12.2

Performance Criteria, 14.12

Services Dashboard, 14.12.2

Service Tests and Beacons, 14.4.5

Configuring Dedicated Beacons, 14.4.5.1

SSL Certificate, 14.4.5.1

Tests, 14.4.5

Web Proxy, 14.4.5.1

Service-Test Based Availability

Key Service Tests, 14.4.1

System, 14.2

Key Components, 14.3

System-Based Availability

Key Components, 14.4.1

Test Performance, 14.1

Thresholds

Critical, 14.3

Warning, 14.3

Time Zone, 14.3

Types

Aggregate Service, 14.7

Types of Service

Generic Service, 14.3

Types of Services

Aggregate Service, 14.3

OCS Service, 14.3

Web Application, 14.3

connect descriptor

using to identify the Management Repository database, 12.4.1, 12.4.2.2

corrective action library, setting up, 2.1.9

corrective actions

alerts and, 1.2.4

policies, 13.3.3.1

privileges required for, 1.2.4

creating

custom reports, 10.3.1

report definitions, 10.2

user-defined metric, 1.4

creating a monitoring script, 5.2.1

credentials, setting up preferred, 2.2.2

Critical URL Monitoring, as substitute for Management Agent, 1.2

CSA, 2.1.11

custom reports, 10.3

customizing

notifications, 1.2.3.1

policies, 13.3.3

D

dashboard

groups, 6.2.5

Data Guard

configuring Enterprise Manager availability, 12.1

data retention policies

for Application Performance Management data, 12.2.1

for other Management data, 12.2.2

modifying default, 12.2.3

of the Management Repository, 12.2

when targets are deleted, 12.2.4

Database Availability and Critical/Warning States

default notification rule, 4.1.2.3

Database Control

enabling security for, 3.5

starting on UNIX, 8.4.1

stopping on UNIX, 8.4.2

DBSNMP database user, 8.7.2

setting the password for, 8.7.2

Default Target Home page, picture of, 15.3.1

deleting targets

data retention policies when, 12.2.4

Deploy Management Plug-in page, picture of, 15.2.6

diagnosing

policy violations, 13.2.2

Disaster Recovery, 11.3.3

disk mirroring and stripping

Management Repository guideline, 12.1

documentation, getting from OTN, 1.5

dropping the Management Repository, 12.4

E

E2E monitoring, 11.2.4.6

em_message, 5.2.1.2

em_result, 5.2.1.2

e-mail addresses, specifying, 2.2.1

E-mail Customization, 4.1.4.1

e-mail notifications

upper limits, 4.1.2.1

e-mails, formats of, 1.2.3.1

emctl

controlling blackouts, 8.7.4

listing targets on a managed host, 8.7.3

security commands, 3.4.3

setting monitoring credentials, 8.7.2.2

starting, stopping, and checking the Management Service, 8.2

emctl config agent credentials, 8.7.2.2

emctl config agent listtargets, 8.7.3

emctl config oms

sample output, 3.2.2.1, 3.7.1, 3.7.2

emctl config oms sso, 3.2.2.1

emctl istop, 8.1.2

emctl reload, 8.7.1

emctl secure agent, 3.4.4

sample output, 3.4.4

emctl secure dbconsole, 3.5

emctl secure em, 3.4.10.2

sample output, 3.4.10.2

emctl secure lock, 3.4.6

emctl secure oms, 3.4.3, 3.4.3

sample output, 3.4.3

emctl secure setpwd, 3.4.7.2

emctl secure unlock, 3.4.6

emctl start agent, 8.1.1

emctl start blackout, 8.7.4, 8.7.4

emctl start dbconsole, 8.4.1

emctl start iasconsole, 8.3.1

emctl start oms, 8.2.1.2

emctl status agent, 8.1.1, 8.1.1

emctl status blackout, 8.7.4

emctl status oms, 8.2.1.2

emctl stop agent, 8.1.1

emctl stop blackout, 8.7.4

emctl stop dbconsole, 8.4.2

emctl stop iasconsole, 8.3.1

emctl stop oms, 8.2.1.2

emctl upload, 8.7.1

em.notification.emails_per_minute

property in emoms.properties, 4.1.1

em.notification.os_cmd_timeout

property in emoms.properties, 4.2.1.1

emoms.properties, 12.3

em.notification.emails_per_connection, 4.1.1

property in emoms.properties, 4.1.1

em.notification.emails_per_minute, 4.1.1

em.notification.os_cmd_timeout, 4.2.1.1

oracle.net.crypto_checksum_client, 3.4.9.2

oracle.net.crypto_checksum_types_client, 3.4.9.2

oracle.net.encryption_client, 3.4.9.2

oracle.net.encryption_types_client, 3.4.9.2

oracle.sysman.eml.mntr.emdRepPwd, 12.3

oracle.sysman.eml.mntr.emdRepPwdEncrypted, 12.3

oracle.sysman.emRep.dbConn.enableEncryption, 3.4.9.2

oracle.sysman.emSDK.sec.DirectoryAuthenticationType, 3.2.3

End-User Performance Monitoring

Web Server

Apache HTTP Server 2.0, 14.1, 14.1, 14.8, 14.8.1

Oracle Application Server Web Cache, 14.1, 14.1, 14.8.2

Oracle HTTP Server, 14.8, 14.8.1

Oracle HTTP Server Based on Apache 2.0, 14.1, 14.1, 14.8

OracleAS Web Cache, 14.8, 14.8

Enterprise Manager

blackouts, functionality of, 1.2.5

corrective action library, 2.1.9

Management Plug-ins, 2.1.10

monitoring templates, 1.3

setting up

administrators, 2.1.2

blackouts, 2.1.5

management pack access, 2.1.7

monitoring templates, 2.1.8

notification methods, 2.1.3

notification rules, 2.2.3.1

notification schedule, 2.2.3.2

patching, 2.1.4

preferred credentials, 2.2.2

registration passwords, 2.1.6

roles, 2.1.1

target subtabs, 2.2.4

Enterprise Manager Console, picture of, 1.5

Enterprise Manager Framework Security

about, 3.4.1

compared with Oracle HTTP Server security features, 3.4.1

configuring, 3.4

enabling for Management Repository, 3.4.9

enabling for multiple Management Services, 3.4.5

enabling for the Management Agent, 3.4.4

overview of steps required, 3.4.2

restricting HTTP access, 3.4.6

types of secure connections, 3.4.1

Extended Network, as substitute for Management Agent, 1.2

extending Enterprise Manager

benefits of, 15.1

deploying new Management Plug-in, 15.2.6

development stage, 15.2.3

lifecycle stages for plug-ins, 15.2

plug-in design, 15.2.2

plug-in packaging, 15.2.5

plug-in user interface, 15.3

validation stage, 15.2.4

F

fetchlets, descriptions of, 15.2.1

figures

Alert Details, 1.5

Availability History Report, 6.3, 10.2

Compare Targets, 1.5

Default Target Home Page, 15.3

Default Target Home Page with Real-Time Report, 15.3.1

Deploy Management Plug-in Page, 15.2.6

Enterprise Manager Console, 1.5

Enterprise Manager Grid Control Console, 1.5

Grid Control's Overview of Setup Page, 2.1

Group Administration Page, 6.2.3

Group Charts Page, 6.2.2

Group Dashboard, 6.2.5

Group Home Page, 6.2.1

Group Members Page, 6.2.4

Job Activity Page, 7.1

Management Plug-in Home Page, 15.2.6

Management Plug-in Import Page, 15.2.6

Policy Trend Overview Page, 6.2.1

Summary of Target Jobs, 7.3

Warning Alert Metric Details, 1.5

Warning Alerts Page, 1.5

G

generating HTML reports, 10.2

Grid Control

architecture overview, 11.1

components, 11.1

console home page, 1.5

Overview of Setup page, 2.1

setting up, 2.1

setting up patches, 2.1.4

sizing, 11.2, 11.2

starting, 8.6.1

starting all components of, 8.6.1

stopping, 8.6.2

stopping all components of, 8.6.2

Group Administration page, picture of, 6.2.3

Group Charts page, picture of, 6.2.2

Group Home page, picture of, 6.2.1

Group Members page, picture of, 6.2.4

groups

administrative tasks, 6.2.3

central monitoring location, 6.2.1

dashboard, 6.2.5

description and purpose, 6.1

jobs and, 7.4

management features, 6.2

member targets, 6.2.4

monitoring collective performance, 6.2.2

policy trends, 6.2.1

redundancy, 6.4

guidelines

for deploying the Management Repository, 12.1

H

Host Availability and Critical/Warning States

default notification rule, 4.1.2.3

HTTP 500 - Internal server error, 8.2.1.3

HTTP access

restricting, 3.4.6

HTTP Server Availability and Critical/Warning States

default notification rule, 4.1.2.3

HTTPS, 3.4.1

Hyper-Threading, 11.2.4.1

I

IBM WebSphere

Oracle ecosystem and, 1.2.1

ILINT tool for validation process, 15.2.4

Information Publisher, 15.3.1

Create Like function, 10.3

generating HTML reports, 10.2

overview of, 10

predefined report definitions, 10.2

predefined reports, 6.3

report definitions, 10.2

report elements, 10.3.3

reporting framework, 10

sharing reports, 10.5

viewing reports, 10.5

Internet Explorer

Certificate dialog box, 3.9.2.1, 3.9.3

security alert dialog box, 3.9.2.1

Security Information dialog box, 3.9.2.3

I/O Channels

monitoring, 11.2.4.5

istop

emctl command, 8.1.2

J

J2EE, 3.4.1

javax.net.ssl.SSLException

SSL handshake failed, 3.9.3

Job Activity page, 7.1

Job Activity page, picture of, 7.1

jobs

analyzing job activity, 7.3

defining, 7.2

definition of, 7.1

differences between runs and executions, 7.1.2

groups and, 7.4

Job Activity page, 7.1

job executions, 7.1.1

job runs, 7.1.1

library, 7.6

multitask, 7.8

notification rules for e-mail, 7.7

predefined tasks, 7.2

privileges for sharing job responsibilities, 7.5

purpose of, 7.1

using, 7.2

L

Listener Availability

default notification rule, 4.1.2.3

load balancer switches

BIG-IP, Oracle ecosystem and, 1.2.1

Loader, 11.2.4.2

loader threads, 11.2.4.2

LVM (Logical Volume Manager), 12.1

M

Management Agent, 11.1, 11.2.1.1, 11.2.2

additional Management Agent commands, 8.7

checking the status on UNIX, 8.1.1

checking the status on Windows, 8.1.3

Critical URL Monitoring as substitute, 1.2

Extended Network as substitute, 1.2

purpose of, 1.2

reinstalling, 11.3.3.3

starting and stopping on UNIX, 8.1.1

starting and stopping on Windows, 8.1.2

Management Information Base (MIB), 4.9

definition, 4.9.1

MIB variable descriptions, 4.9.2

Management Pack Access

setting up, 2.1.7

Management Plug-in Import page, picture of, 15.2.6

Management Plug-ins

deploying, 15.2.6

designing, 15.2.2

developing, 15.2, 15.2.3

extending Enterprise Manager with, 15

fetchlets, 15.2.1

Management Plug-in Archive (MPA), 15.2.5

packaging, 15.2.5

real-time reports, 15.3.1

setting up, 2.1.10

Target Type Default Collections file, 15.2.1

Target Type Metadata file, 15.2.1

user interface, 15.3

validating, 15.2.4

Management Plug-ins page, picture of, 15.2.6

Management Repository

administrators and, 2.1.2

introduction of, 1.2.1

See Oracle Management Repository

Management Server, 11.2.4.1

Management Servers

adding, 11.2.4.4

Management Service, 11.1, 11.2.1.1

starting and stopping on Windows systems, 8.2.2

managing

compliance, 13.2

groups, 6.2

policies, 13.2

master agent

Oracle Peer SNMP Master Agent service, 8.1.2

message formats, specifying, 2.2.1

MetaLink

credentials, 2.1.4

metric

baselines, 1.2.1.1.2

snapshots, 1.2.1.1.1

threshold value and alerts, 1.2.2

thresholds, 1.2.1.1

metric baselines

adaptive thresholds, 1.2.1.1.2

normalized views, 1.2.1.1.2

metrics

creating user-defined, 1.4

snapshots, 1.2.1.1.1

threshold values, 1.2.1

thresholds, 1.2.1.1

user-defined, creating, 1.4

user-defined, types of, 1.4

warning alert details page, 1.5

MGMT_ADMIN.DISABLE_METRIC_DELETION, 12.2.4

MGMT_ADMIN.ENABLE_METRIC_DELETION, 12.2.4

MGMT_METRICS_1DAY table, 12.2.3

MGMT_METRICS_1HOUR table, 12.2.3

MGMT_METRICS_RAW table, 12.2.3

MGMT_PARAMETERS table, 12.2.3

MGMT_RT_datatype_1DAY table, 12.2.3

MGMT_RT_datatype_1HOUR table, 12.2.3

MGMT_RT_datatype_DIST_1DAY table, 12.2.3

MGMT_RT_datatype_DIST_1HOUR table, 12.2.3

MGMT_RT_METRICS_RAW table, 12.2.3

MIB

See Management Information Base (MIB)

monitoring

alerts as they occur, 6.2.5

basics of, 1.2

information, accessing, 1.5

systems, 1

templates

function of, 1.3

setting up, 2.1.8

monitoring credentials

defined, 8.7.2

example of setting, 8.7.2.2

setting, 8.7.2

setting in Grid Control, 8.7.2.1

setting with emctl, 8.7.2.2

monitoring script creation, 5.2.1

monitoring template, comparing, 1.3

monitoring templates, 5.4.3

multitask jobs, 7.8

N

NetApp Filers

Oracle ecosystem and, 1.2.1

Netscape Navigator

New Site Certificate dialog box, 3.9.2.2

network/admin, 3.4.9.1, 3.4.9.2, 3.4.9.3, 3.4.9.4, 3.4.9.4

New Site Certificate dialog box

Netscape Navigator, 3.9.2.2

Notification Methods, 4.2

notification methods

based on a PL/SQL Procedure, 4.2.1.2

based on an SNMP trap, 4.2.1.3

based on operating system commands, 4.2.1.1

notification rules

definition, 4.1.2.3

out-of-box, 4.1.2.3

out-of-the-box notification rules, 4.1.2.1

subscribing to, 4.1.2.3

notification schedules, 4.1.2.1

notification system

e-mail errors, 4.10.4

notification system errors, 4.10.2

notification system, trace messages, 4.10.3

notifications

alerts, 1.2.3

assigning methods to rules, 4.6

assigning rules to methods, 4.7

customizing, 1.2.3.1

defining multiple mail servers, 4.1.1

for jobs, 7.7

long e-mail notifications, 4.1.2.1

mail server settings, 4.1.1

mail server settings in emoms.properties, 4.1.1

management information base (MIB), 4.9

methods, 1.2.3

methods to send notifications, 2.1.3

notification method, 1.2.3

notification schedules, 4.1.2.1

rules, setting up, 2.2.3.1

sample Operating System command script, 4.2.1.1

schedule, setting up, 2.2.3.2

setting up, 4.1

short email notifications, 4.1.2.1

using custom notfication methods, 4.2.1

Notifiction Rules

Custom, 4.1.2.3

O

OC4J Availability and Critical/Warning States

default notification rule, 4.1.2.3

operating system

user-defined metrics, 1.4

Operating System command

sample notification method for, 4.2.1.1

sample script, 4.2.1.1

Operating System scripts, 4.2, 4.2

while creating notification methods, 4.2.1

OPMN

See Oracle Process Management and Notification

opmnctl

using to start Web Cache, 8.2.1.3

using to stop Web Cache, 8.2.1.3

opmnctl startall, 3.2.2.1, 3.4.3, 8.2.1.1

opmnctl status, 8.2.1.3

opmnctl stopall, 3.2.2.1, 3.4.3, 8.2.1.1

opmnctl stopproc ias-component=WebCache, 8.2.1.3

ORA-12645

Parameter does not exist, 3.4.9.1

Oracle

ecosystem, 1.2.1

Technology Network (OTN), 1.5

Oracle Advanced Security, 3.4.1, 3.4.9.1

enabling for Management Repository, 3.4.9.3

enabling for the Management Agent, 3.4.9.4

Oracle Application Server Web Cache

default port number, 8.2.1.3

errors when not running, 8.2.1.3

starting and stopping with opmnctl, 8.2.1.3

Web Cache Manager, 14.8.2.2

Oracle Enterprise Manager

components, 11.2.1.1

rollup process, 11.2.4.3

Oracle Enterprise Manager 10g Grid Control

See Grid Control

Oracle Enterprise Manger

tuning, 11.2.4

Oracle Internet Directory, 3.2.2.4.1

Oracle Management Agent, 15.2.1

enabling security for, 3.4.4, 3.4.9.4

starting and stopping, 8.1

Oracle Management Repository, 11.1

data retention policies, 12.2

deployment guidelines, 12.1

dropping, 12.4

enabling Oracle Advanced Security, 3.4.9.3

enabling security for, 3.4.9

identifying with a connect descriptor, 12.4.1, 12.4.2.2

recreating, 12.4, 12.4.2

reloading data, 8.7.1

restoring, 11.3.3.1

starting the Management Repository database, 8.6.1

troubleshooting, 12.5.3

uploading data, 8.7.1

Oracle Management Service, 11.3.3.2

enabling security for, 3.4.3

enabling security for multiple Management Services, 3.4.5

modifying monitoring credentials, 8.7.2

restoring, 11.3.3.2

starting, stopping, and checking, 8.2

Oracle Process Management and Notification (OPMN)

using to start and stop the Management Service, 8.2.1.1, 8.2.2

Oracle Technology Network (OTN), 14.8.2.4.1

ORACLE_HOME/network/admin, 3.4.9.1, 3.4.9.2, 3.4.9.3

ORACLE_HOME/opmn/bin, 8.2.1.3

oracle.net.crypto_checksum_client

property in emoms.properties, 3.4.9.2

oracle.net.crypto_checksum_types_client

property in emoms.properties, 3.4.9.2

oracle.net.encryption_client

property in emoms.properties, 3.4.9.2

oracle.net.encryption_types_client

property in emoms.properties, 3.4.9.2

oracle.sysman.eml.mntr.emdRepPwd

property in emoms.properties, 12.3

oracle.sysman.eml.mntr.emdRepPwdEncrypted

property in emoms.properties, 12.3

oracle.sysman.emRep.dbConn.enableEncryption

entry in emoms.properties, 3.4.9.2

oracle.sysman.emSDK.sec.DirectoryAuthenticationType

property in emoms.properties, 3.2.3

OS scripts

See Operating System scripts

OTN (Oracle Technology Network), 14.8.2.4.1

OTN and documentation, 1.5

out-of-box

monitoring, 1.2.1

policies, 13.3.2

reports, 10.2

P

password

changing the SYSMAN password, 12.3

patch

advisories, setting up to receive, 2.1.4

patching

setup, 2.1.4

peer encapsulator service

SNMP, 8.1.2

Performance Metrics

Beacon Aggregation Function

Average, 14.4.2, 14.4.2

Maximum, 14.4.2

Minimum, 14.4.2, 14.4.2

Sum, 14.4.2, 14.4.2

System Aggregation Function

Maximum, 14.4.2

planning outage periods, blackouts, 1.2.5

PL/SQL procedures, 4.2

sample, 4.2.1.2

while creating a notification method, 4.2.1.2

while creating notfication methods, 4.2.1

PL/SQL procedures, sending notifications, 2.2.3.1

plug-ins

deploying, 15.2.6

designing, 15.2.2

developing, 15.2.3

Oracle Management Agent, 15.2.1

packaging, 15.2.5

user interface, 15.3

validating, 15.2.4

policies

assessing security policies, 13.2.3

corrective actions, defining, 13.3.3.1

customizing, 13.3.3

investigating violations, 13.2.2

managing, 13.2

out-of-box, 13.3.2

reports on violations, 13.2.5

using templates for monitoring, 13.3.3.2

violations reports, 13.2.5

policy group

Oracle database, 13.4.1

Oracle Listener, 13.4.3

RAC, 13.4.2

Policy Groups reports, 13.2.5

policy management, 13.2

Policy Trend Overview page, picture of, 6.2.1

Policy Violations reports, 13.2.5

predefined

job tasks, 7.2

preferred credentials, setting up, 2.2.2

privileges

for corrective actions, 1.2.4

for sharing job responsibilities, 7.5

ProcessManager

service used to control the Management Service on Windows systems, 8.2.2

Public Key Infrastructure (PKI), 3.4.1, 3.9.3

purging policies

See data retention policies

R

RAID-capable disk

Management Repository guideline, 12.1

real-time reports for Management Plug-ins, 15.3.1

Recovery, 11.3.1

redundancy groups, 6.4

registration passwords, setting up, 2.1.6

Repeat Notifications, 4.1.1.1

Repeat Notifications for Rules, 4.1.1.1

RepManager script, 12.4.1, 12.4.2.1

reporting framework, Information Publisher, 15.3.1

reports

creating custom reports, 10.3.1

custom, 10.3

definitions, Information Publisher, 10.2

e-mailing, 10.4.3

generating HTML report, 10.2

Information Publisher, 10

out-of-box, Information Publisher, 10.2

policy violations, 13.2.5

predefined, 6.3

predefined definitions, 10.2

predefined report definitions, 10.2

report elements, 10.3.3

scheduling, 10.4, 10.4

service-level, 15.3.1

sharing, 10.5, 10.5

storing and purging, 10.4.2

viewing, 10.5

Repository Operations Availability

default notification rule, 4.1.2.3

retroactive blackouts, 1.2.5

roles, setting up, 2.1.1

rollup process, 11.2.4.3

Root Cause Analysis

Mode

Automatic, 14.4.6

Manual, 14.4.6

root password

See also SYSMAN

when enabling security for the Database Control, 3.5

when enabling security for the Management Service, 3.4.3

S

scheduled maintenance with blackouts, 1.2.5

scheduling

reports, 10.4

reports, flexibility, 10.4.1

script registration, UDM, 5.2.2

script results, returning, 5.2.1.2

security

about Enterprise Manager security, 3.1

enabling for Database Control, 3.5

overview of steps required to enable Enterprise Manager Framework Security, 3.4.2

policies and, 13.2.3

See also Enterprise Manager Framework Security

security alert dialog box

Internet Explorer, 3.9.2.1

Security At a Glance feature, policies and, 13.2.3

security certificate alerts

responding to, 3.9.2

security features

See Enterprise Manager Framework Security

Security Information dialog box

Internet Explorer, 3.9.2.3

Server Connection Hung

error while creating the repository, 12.5.2

Server Load Balancer, 3.4.8

server-generated alerts, 1.1

Service Tests and Beacons

Tests

DNS, 14.4.5

FTP, 14.4.5

SOAP, 14.4.5

Web Transaction, 14.4.5

service-level

reports, 15.3.1

Services control panel

using to start and stop the Management Agent, 8.3.2, 8.4.3

using to start the Management Service, 8.2.2

setting

metric threshold values, 1.2.1

setting up

administrators, 2.1.2

blackouts, 2.1.5

corrective action library, 2.1.9

Grid Control, 2.1

management pack access, 2.1.7

Management Plug-ins, 2.1.10

monitoring templates, 2.1.8

notification methods, 2.1.3

notification rules, 2.2.3.1

notification schedule, 2.2.3.2

patching, 2.1.4

preferred credentials, 2.2.2

registration passwords, 2.1.6

roles, 2.1.1

target subtabs, 2.2.4

setupinfo.txt, 8.2.1.3

sharing reports, 10.5, 10.5

snapshots of metrics, 1.2.1.1.1

SNMP

Oracle Peer SNMP Master Agent service, 8.1.2

Oracle SNMP Peer Encapsulator service, 8.1.2

SNMP traps, 2.1.3, 4.2, 4.2.1, 4.2.1.3

sample, 4.2.1.3

sending notifications, 2.2.3.1

SQL

user-defined metrics, 1.4

SQL UDM, character limit, 5.3.1.3

SQL UDM, long statements, 5.3.1.3

SQLNET.CRYPTO_SEED

entry in sqlnet.ora, 3.4.9.3, 3.4.9.4

SQLNET.ENCRYPTION_SERVER

entry in sqlnet.ora, 3.4.9.3

sqlnet.ora, 3.4.9.1, 3.4.9.1

SQLNET.CRYPTO_SEED, 3.4.9.3, 3.4.9.4

SQLNET.ENCRYPTION_SERVER, 3.4.9.3

Status Codes, Corrective Actions, 4.3.2, 4.4.1

super administrator account, 2.1.2.1

support of third-party components, 1.1

SYSMAN

changing the SYSMAN password, 12.3

checking for existence of, 12.5.3

entering SYSMAN password when enabling security, 3.4.3

super administrator account, 2.1.2.1

System Dashboard, picture of, 6.2.5

system errors, notification, 4.10.2

systems

monitoring, 1

privileges for administrators, 2.1.1

T

target

definition of, 1.2

privileges for administrators, 2.1.1

subtabs, setting up, 2.2.4

target monitoring credentials

defined, 8.7.2

example of setting, 8.7.2.2

setting, 8.7.2

setting in Grid Control, 8.7.2.1

Target Type Default Collections file, 15.2.1

Target Type Metadata file, 15.2.1

targets

listing targets on a managed host, 8.7.3

templates, policies and, 13.3.3.2

third-party

components, support of, 1.1

thresholds, 11.2.2, 11.2.2

adaptive, 1.2.1.1.2

adaptive alert, 1.2.1.1.2

alerts and, 1.2.2

definition of, 1.2.1

for metrics, 1.2.1.1, 1.2.1.1

Tool Center, 2.1.10

troubleshooting

general techniques while creating the Management Repository, 12.5.3

while creating the Management Repository, 12.5

Troubleshooting Service Tests, 14.14

Forms Transactions, 14.14.1

troubleshooting, notifications, 4.10

types of accounts, 2.1.2

U

Usage Metrics

Aggregation Function

Average, 14.4.3, 14.4.4

Maximum, 14.4.3, 14.4.4

Minimum, 14.4.3, 14.4.4

Sum, 14.4.3, 14.4.4

user-defined

metrics, 1.4

metrics, creating, 1.4

User-defined metric page

Command Line, 5.2.2

Comparison Operator, 5.2.2, 5.3

Consecutive Occurrences Preceding Notification, 5.2.2, 5.3

Critical, 5.2.2, 5.3

Environment, 5.2.2

Metric Name, 5.2.2, 5.3

Metric Type, 5.2.2, 5.3

Operating System User Name and Password, 5.2.2

Response Action, 5.2.2, 5.3

Warning, 5.2.2, 5.3

User-defined metric page, Central Console, 5.2.2, 5.3

user-defined metric, example, 5.2.3

User-defined metrics, 5

user-defined metrics, 5.4.3

V

viewing

reports, 10.5

W

Warning Alert Metric Details page, picture of, 1.5

Warning Alerts page, picture of, 1.5

Web Application

Source

Step, 14.4.2

Step Group, 14.4.2

Transaction, 14.4.2

Web Applications

monitoring over HTTPS, 3.9.3

Web Cache Availability and Critical/Warning States

default notification rule, 4.1.2.3

Copyright © 2009, Oracle and/or its affiliates. All rights reserved.