5
Administering Real Application Clusters Databases with Oracle Enterprise Manager

This chapter describes how Oracle Enterprise Manager manages multiple instances. The Server Management (SRVM) component of Real Application Clusters can perform a variety of cluster database management tasks. Many of these tasks can be initiated from the Oracle Enterprise Manager console.

This section is meant to only describe Oracle Enterprise Manager administration for Real Application Clusters. Use this section as a supplement to information contained in the Oracle Enterprise Manager Administrator's Guide and other Oracle Enterprise Manager documentation.

The topics in this chapter are:

• Overview of Oracle Enterprise Manager Administration

• Starting the Console

• Displaying Objects in the Navigator Pane

• Using the Cluster Database Right-Mouse Menu

• Starting a Cluster Database

• Shutting Down a Cluster Database

• Viewing the Cluster Database Operation Results

• Viewing Cluster Database Status

• Creating a Job for a Cluster Database or Instance

• Specifying Job Details

• Registering Cluster Database Events

Overview of Oracle Enterprise Manager Administration

The Oracle Enterprise Manager Console provides a central point of control for the Oracle environment through an intuitive graphical user interface (GUI). The Oracle Enterprise Manager Console can initiate a variety of cluster database management tasks at the managed cluster nodes with the Server Management (SRVM) component.

After nodes have been discovered, you can use the Oracle Enterprise Manager Console to start, stop, and monitor services as well as to schedule jobs or register events. You also perform these tasks simultaneously on multiple nodes. You can also use the Console to manage schemas, security, and the storage features of cluster databases.

See Also: Oracle9i Real Application Clusters Installation and Configuration for configuration information. The Oracle Enterprise Manager online Help system may also answer many frequently asked questions.

Starting the Console

To use the Oracle Enterprise Manager Console, start the following components:

1. An Oracle Intelligent Agent on each of the nodes

2. Management Server

3. Console

   See Also: Oracle9i Real Application Clusters Installation and Configuration for Real Application Clusters-specific information on installing Oracle Enterprise Manager

Displaying Objects in the Navigator Pane

From the Navigator pane, you can view and manage both single- and multiple-instance databases. After nodes are discovered, the Navigator displays cluster databases, their instances, and other related services, such as a listener. The information available for cluster databases is essentially the same as for single-instance databases. Just as in single instance databases, cluster databases and all of their related elements can be administered using master/detail views and Navigator menus.

The Navigator displays all the network objects and their relationships to other objects including a direct view of objects such as user-defined groups, nodes, listeners, servers, databases, and database objects.

In the Navigator tree, cluster databases are listed under the Databases folder as siblings to single-instance databases. Just as in single instance databases, each cluster database folder contains the instances and sub-folders for Instance, Schema, Security, and Storage, as shown in Figure 5-1.

Note: If you attempt to expand a cluster database and you have not yet set preferred credentials for the database as described in Oracle9i Real Application Clusters Installation and Configuration, the Database Connect Information dialog prompts you to enter database connect information.

By selecting objects within a Cluster Database subfolder, you can access property sheets to inspect and modify properties of these objects. (Just as you can for single-instance databases.) For example, by expanding the Storage folder and then right-clicking Redo Log Groups subfolder, and then choosing Create, you can add a new redo log group. All discovered instances will be displayed, under the Cluster Database Instances folder.

When accessing cluster databases, only the Instance folder has subfolder contents that differ from those seen in single instance databases. Within the Instance folder, the instance database subfolders are split into two functional parts:

• Database-Specific File Structure

• Instance-Specific File Structure

Database-Specific File Structure

All of the database-specific functionality is seen directly under the Instance subfolder for the cluster database. The available database-specific functionality includes:

• In-doubt Transactions. This provides details on distributed database in-doubt transactions.

• Resource Consumer Groups. This lets you view, create, and modify resource consumer groups.

Instance-Specific File Structure

All instance-specific functionality appears beneath the individual instance icons within the Cluster Database Instances subfolder. The instance-specific functionality includes:

Configuration

Here you can view instance states, view and edit initialization parameters, toggle the archive mode, and view performance statistics of active resource plans.

Stored Configurations

Here you can create, edit, and store multiple startup configurations for instances. This eliminates the need to track initsid.ora parameter files.

Sessions

Here you can list the status of connected users, view the latest SQL for sessions, and terminate sessions.

Locks

Here you can view list details for currently held User type and System type locks.

Resource Plans

Here you can define and modify the resource plans for the cluster database. You can also activate a specific resource plan for an individual instance.

Resource Plan Schedule

Here you can schedule activation of a resource plan.

The contents of a typical database folder are shown in Figure 5-1.

Figure 5-1 Database Subfolders

Text description of the illustration oemadmia.gif

You can expand the Cluster Database Instances folder to display the instances belonging to each discovered database or node, as shown in Figure 5-2.

Figure 5-2 Cluster Database Instances Folder

Text description of the illustration oemadmi2.gif

Note: The instances have the same right-mouse menu as the right-mouse menu that is used in single instance databases.

Using the Cluster Database Right-Mouse Menu

Right-clicking on a cluster database displays the specialized cluster database right-mouse menu as shown in Figure 5-3.

Figure 5-3 Right Mouse Menu

Text description of the illustration oemadmi3.gif

The menu contains entries for the specialized functions listed in Table 5-1. (All others are identical to single instance Oracle Enterprise Manager.) When selected, you will see dialogs that differ from single instance databases. Note that in addition to the tools listed in Table 5-1 you will also see the single instance tools, which also work with Real Application Clusters.

Table 5-1 Cluster Database Right-Mouse Menu Functions Specific to Real Application Clusters

Option	Description
Startup	Starts the database See Also: "Starting a Cluster Database"
Shutdown	Shuts down the database See Also: "Shutting Down a Cluster Database"
Results	Displays startup and shutdown results See Also: "Viewing the Cluster Database Operation Results".
View Edit Details	Enables inspection of the state of the cluster database, including which instances are active. See Also: "Viewing Cluster Database Status"
Related Tools	Contains access to other tools which have been enabled for a cluster database.

Starting a Cluster Database

The Console enables you to start an entire cluster database or selected instances within the cluster database.

To start up a cluster database or its instances:

1. In the Navigator pane, expand Databases.

2. Right-click on a Cluster Database.

   A menu appears with options for the database.

3. Choose Startup from the menu.

   The Cluster Database Startup dialog box appears.

4. Select a startup type as shown in Table 5-2.

Table 5-2 Cluster Database Startup Types

Option	Description
No Mount	Does not mount the database upon instance startup
Mount	Mounts a database but does not open it
Open	(default) Opens the database
Force the instance(s) to start	Shuts down the currently running Oracle instances with the SHUTDOWN mode, ABORT, before restarting them. If the instances are running and FORCE is not specified, an error results. Note: You should not use the FORCE mode under normal circumstances. Use the FORCE mode only while debugging or under abnormal circumstances.
Restrict access to the database	Makes the started instances accessible only to users with the RESTRICTED SESSION system privilege. Users already connected are not affected.

5. To start up all instances, click Startup. To start up only selected instances, follow these steps:
   1. Select Instances.

      The Select Instances to Start dialog box appears.

   2. Select the instances to start up in the Available list, then click Add.

   3. Click OK to close the Select Instances to Start dialog box.

   4. Click Startup from the Cluster Database Startup dialog box.

   The Cluster Database Startup Results dialog box displays the progress of the startup operation, as described in "Viewing the Cluster Database Operation Results".

   If the instances were started successfully, the Cluster Database Started message box appears with a successful message.

   Click OK in the Cluster Database Started message to acknowledge the message, then click Close in the Cluster Database Startup Results.

   If the startup fails, the Cluster Database Started message box appears with a failure message. Click View Details to view more information in the Cluster Database Startup Results dialog box about why the startup failed, then click Close.

   Shutting Down a Cluster Database

   The Console enables you to shut down an entire cluster database or selected instances within a cluster database. Once all instances are shut down, the database is also considered shut down.

   Note: : A cluster database can occasionally be completely down, but some of its services, such as the database listener, might remain running.

   To shut down a cluster database or its instances:

   1. In the Navigator pane, expand Databases.

   2. Right-click on a cluster database.

      A menu appears with options for the database.

   3. Choose Shutdown from the menu.

      The Cluster Database Shutdown dialog box appears.

   4. Select a shutdown type, as shown in Table 5-3.

Table 5-3 Shutdown Types

Option	Description
Normal	Waits for the currently connected users to disconnect from the database, prohibits further connects, and closes and dismounts the database before shutting down the instance. Instance recovery is not required on next startup.
Immediate	(default) Does not wait for current calls to complete, prohibits further connects, and closes and dismounts the database.The instance is immediately shut down. Connected users are not required to disconnect and instance recovery is not required on next startup.
Abort	Proceeds with the fastest possible shutdown. Connected users are not required to disconnect. The database is not closed or dismounted, but the instances are shut down. Instance recovery is required on next startup. Note: You must use this option if a background process terminates abnormally.
Transactional	Waits for transactions to complete before shutting down
Shutdown Database Only	(default) Shuts down the database only. The services required for an instance, such as the listener, remain up and available.
Shutdown Database And Other Services	Shuts down the database and associated services, such as the listener

5. To shut down all instances, click Shutdown.

   To shut down only selected instances, follow these additional steps:

   1. Select Instances.

      The Select Instances to Stop dialog box appears.

   2. Select the instances to stop in the Available list, then click Add.

   3. Click OK to close the Select Instances to Stop dialog box.

   4. Click Shutdown from the Cluster Database Shutdown dialog box.

   The Cluster Database Shutdown Progress dialog box displays the progress of the shutdown operation.

   See Also: "Viewing the Cluster Database Operation Results"

   If the instances were shut down successfully, the Cluster Database Stopped message box appears with a successful message.

   Click OK in the Cluster Database Stopped message to acknowledge the message, then click Close in the Cluster Database Shutdown Results.

   If the shutdown fails, the Cluster Database Stopped message box appears with a failure message. Click View Details to view more information in the Cluster Database Shutdown Progress dialog box about why the shutdown failed, then click Close.

   Viewing the Cluster Database Operation Results

   The Cluster Database Startup/Shutdown Results dialog displays information about the progress of the instance startup or shutdown operation you selected:

   The operation results are presented in two views:

   • Status Details Tab

   • Output Tab

   The Cluster Database Startup/Shutdown Results dialog box automatically displays during a startup or shutdown operation. You can also initiate it with the following steps:

   1. In the Navigator pane, expand Databases.

   2. Right-click on a cluster database.

      A menu appears with options for the database.

   3. Choose Results from the menu.

   Status Details Tab

   While a startup or shutdown operation is running against a cluster, the Status Details tab progress display is shown and updated dynamically as the operation progresses.

   A successful startup operation for a three-node cluster looks like the following in the Status Details tab, as shown in Figure 5-4.

   Figure 5-4 Status Details Tab

   
   
   Text description of the illustration startdet.gif

   A successful shutdown operation for a three-node cluster looks like the Status Details tab shown in Figure 5-5.

   Figure 5-5 Successful Shutdown Results

   
   
   Text description of the illustration stopdeta.gif

   Table 5-5 shows the possible states that each component might experience.

   Table 5-4 Possible Component States

   State	Description
   Up (green flag)	The component is running.
   Down (red flag)	The component is not running.
   In Progress (timer)	Oracle Enterprise Manager cannot determine the state of the component. This state occurs typically when the component startup or shutdown operation has not completed.
   Component does not exist on this node (blank background)	The component was not configured on the node. Not all components (listener, instance) are required on every node.

   Output Tab

   The Output tab displays the commands executed by the node and any associated error messages in textual format.

   A successful shutdown looks like the following in the Output tab, as shown in Figure 5-6.

   Figure 5-6 Successful Shutdown Results

   
   
   Text description of the illustration emstartd.gif

   Oracle Enterprise Manager displays a Results tab that is similar to the Shutdown tab in Figure 5-6.

   Viewing Cluster Database Status

   The Edit Cluster Database dialog box displays status information about the database, such as instances available in the cluster and the status of cluster components.

   Note: Because this dialog box requires a connection to a database, this dialog box will not appear if the cluster database is down.

   To view status information about a database:

   1. In the Navigator pane, expand Databases > database_name.

   2. Right-click on a cluster database under the Databases folder in the Navigator pane.

      A menu appears with options for the database.

   3. Choose Edit from the menu.

      The Edit Cluster Database dialog box appears.

   The operation results are presented in two views:

   • General Tab

   • Status Details Tab

   General Tab

   The General tab displays information about the currently running instances by querying V$ACTIVE_INSTANCES table, as shown in Figure 5-7. Oracle Enterprise Manager makes a connection to the cluster database. Therefore, this tab will not appear if the database is down.

   Figure 5-7 General Tab

   
   
   Text description of the illustration status.gif

   Table 5-5 describes the fields in the Output Tab Form.

   Table 5-5 Fields in Output Tab Form

   Column Name	Description
   Instance Number	Identifies the instance number
   Instance Name	The name specified for the instance and the node it is running on. This name has the following format: node:instance_name.
   Secondary	Indicates if the node is an secondary instance in a primary/secondary instance configuration

   Status Details Tab

   Note: This tab is not available for a Windows NT or Windows 2000 cluster database because SRVCTL on Windows NT and Windows 2000 does not generate status details.

   The Status Details Tab displays an overall view of the state of the cluster and related components, as shown in Figure 5-8. This tab displays the status of the various components, such as listeners and instances, for all nodes.

   Figure 5-8 Status Details Tab

   
   
   Text description of the illustration statusde.gif

   Table 5-6 shows the possible states that each component might experience.

   Table 5-6 Component States

   State	Description
   Up (green flag)	The component is running.
   Down (red flag)	The component is not running.
   In Progress (timer)	Oracle Enterprise Manager cannot determine the state of the component. This state occurs typically when the component startup or shutdown operation has not completed.
   Component does not exist on this node (blank background)	The component was not configured on the node. Not all components (listener, instance) are required to exist on every node.

   Creating a Job for a Cluster Database or Instance

   The job scheduling system provides a highly reliable and flexible mechanism for you to schedule and automate repetitive jobs on both the cluster database and instances.

   The Console contains a full-featured scheduling tool that enables you to develop a customized schedule. This provides you with actual "lights out" management capability so that you can focus on other tasks. A rich selection of jobs is provided for cluster databases.

   You can create a job with a cluster database or instance as the destination. To create a new job, follow these steps:

   1. Choose Job > Create Job.

   2. Complete the tabs of the Create Job property sheet.

   3. When you are satisfied with your job settings, click Submit to submit the job to Oracle Intelligent Agent. The job appears in the Active Jobs window.

   4. Click Save to save the job. The job appears in the Job Library window. You can modify or submit a saved job at a later time.

      Note: There is usually a slight delay between submitting the job and notification by Oracle Intelligent Agent.

   Specifying Job Details

   From the Create Job property sheet, you can specify the details of a new job. The Create Job property sheet contains the tabs listed in Table 5-7.

   Table 5-7 Tabs Contained in the Create Job Property Sheet

   Tab	Description
   General	Specify the job name, description, destination type, and destination.
   Tasks	Choose the task(s) for the job.
   Parameters	Set the run-time parameters for the tasks. The parameters that appear on this tab depend on which task(s) you chose on the Task list box.
   Schedule	Schedule the time and frequency for Oracle Enterprise Manager to run the job.
   Permissions	Specify the administrator to perform the job.

   The following tabs contain options specific to cluster databases:

   • General Tab

   • Tasks Tab

   • Parameters Tab

     See Also: Oracle Enterprise Manager Administrator's Guide for general job scheduling information

   General Tab

   As shown in Figure 5-9, from the General tab you can specify:

   • Job Name

   • Destination Type (either Cluster Database or Instance)

   • Description

   • Destinations

   Figure 5-9 General Tab

   
   
   Text description of the illustration job1.gif

   The General tab contains the options in Table 5-8.

   Table 5-8 General Tab Options

   Option	Description
   Job Name	Enter the name of the new job.
   Description	Enter a description of the job.
   Destination Type	Select a destination type from the drop-down list box. You can select from the following options: Cluster Database, instance, database, listener, or node.
   Available Destinations	The destinations are determined by your selection of the Destination Type. The destinations include Cluster Databases, instances, databases, listeners, and nodes. Click the destinations of the job in the Available Destinations list, then click Add to move the destination to the Selected Destinations list. To remove a destination from a job, click the destination in the Selected Destinations list, then click Remove.
   Fixit Job	Select this check box to use this job as the fixit job for an event occurrence. The job cannot be scheduled.

   Tasks Tab

   From the Tasks tab, choose the task(s) for the job to perform. The list of tasks that appear is different depending on whether you select a cluster database or instance as your Destination Type from the General tab.

   Move the tasks between the Available Tasks and Selected Tasks lists with the Add and Remove buttons.

   Tasks for Cluster Database Destinations

   If your Destination Type is a cluster database, you can select from these tasks that are specific to Real Application Clusters:

   • Shutdown Cluster Database

   • Startup Cluster Database

     See Also: Oracle Enterprise Manager Administrator's Guide for a description of these tasks and the parameters to set

   Parameters Tab

   From the Parameters tab, you can specify parameter settings for the job tasks you selected on the Tasks tab. The parameters that display vary according to the job task. Parameters specific to cluster database startup and shutdown tasks are described below.

   See Also: Oracle Enterprise Manager Administrator's Guide for a description of parameters to set for instance tasks

   Cluster Database Startup Task

   When you select the Startup Cluster Database task on the Tasks tab, the following display appears.

   Complete the parameters on the tab and click Submit to run the cluster database startup task.

   The Parameters tab for Startup contains the options listed in Table 5-9.

   Table 5-9 Parameters Tab for Startup

   Parameters	Description
   Startup	Select the startup modes for the job from the drop-down list box.
   Connect As	Specify the role. Only Normal is allowed for Oracle7. For Oracle8i and subsequent releases, SYSOPER and SYSDBA roles allow you the maximum database administration privileges. You require SYSDBA or SYSOPER privileges to run job tasks such as shutdown or startup on the database. See Also: Oracle9i Database Administrator's Guide for more information about SYSOPER and SYSDBA roles If you attempt to connect as SYSDBA and do not have SYSDBA privileges, an error message states that an invalid user name or password was entered.
   Override Preferred Credentials	You can use the preferred credentials that have been set up for the database, or you can enter another database user name and password.

   Parameters for the Cluster Database Shutdown Task

   When you select Shutdown Cluster Database on the Tasks tab, the following display in Figure 5-10 appears.

   Figure 5-10 Shutdown Cluster Database Parameters

   
   
   Text description of the illustration job4.gif

   Complete the parameter entries on the tab and click Submit to run the cluster database shutdown task.

   The Parameters tab for Shutdown contains the options listed in Table 5-10.

   Table 5-10 Parameters Tab for Shutdown

   Options	Description
   Mode	Click Immediate (default) or the Abort.
   Connect As	Click SYSDBA (default) or SYSOPER.
   Override Preferred Credentials	You can use the preferred credentials that have been set up for the database, or you can use another database user name and password.

   Registering Cluster Database Events

   Oracle Enterprise Manager includes an Event Management feature. This feature enables you to monitor tests on cluster database instances. All of the tests that are available for single instance databases can also be run on cluster databases. In addition to these standard event tests, there are specialized event tests available only for cluster database instances. These tests include: global cache converts, pings, timeouts, consistent read requests, freelist waits, and so forth. Data from the Oracle Data Gatherer is used to create these Event Management reports.

   To access the Event Management feature, from the Event menu, select Create Event. The Create Event window opens. There, you can select targets, set parameters, and set notification preferences, using the following sub-pages:

   • The General Page is used to select the cluster database and cluster database instance target types.

   • The Test Page is used select the type of specialized cluster databases tests that you want to schedule.

   • The Parameters Page, is used to select the parameter for your test event.

   • The Schedule Page is used for scheduling the execution of a test event.

   • The Access Page is used for setting notification options. By default, notifications are sent directly to the SYSADMIN console. Depending on your hardware configuration, you can also select notification through a telephone pager and/or e-mail.

   Typically, you will set parameters and notifications so that the system will inform you when a given parameter limit has been reached. The Event Management Tests submenu is shown in Figure 5-11.

   Figure 5-11 Event Management Tests Menu

   
   
   Text description of the illustration eventman.gif