4 Administering Database Instances and Cluster Databases

Web-based Oracle Enterprise Manager Database Control and Grid Control interfaces let you mange Oracle Real Application Clusters (Oracle RAC) databases. The Enterprise Manager console is a central point of control for the Oracle environment. Use the Database Control console to initiate cluster database management tasks. Use the Grid Control console to administer multiple Oracle RAC databases and cluster nodes.

This chapter describes how to administer your Oracle Real Application Clusters (Oracle RAC) environment. It explains the startup and shutdown tasks for database components and how to administer parameters and parameter files in Oracle RAC. This chapter includes the following sections:

Overview of Oracle RAC Database Management

Oracle RAC is a technology that links one or more individual computers so that they function as one system. Oracle RAC enables each computer that is a member of a cluster, or node, to share access to the Oracle database. If one cluster node fails or is taken offline, then the other cluster nodes continue operating and the entire Oracle RAC database remains available. This means that two or more inexpensive computers appear to applications as if they were a much more powerful, and more expensive, computer.

To increase the performance of a two-node Oracle RAC database, you can add cluster nodes. Each additional node can help speed up application processing, support more users or processes, or both. In addition, you can also add cluster nodes to increase the availability and reliability of a two-node Oracle RAC database. The more nodes that your Oracle RAC environment has, the less the effect that the loss of any individual node has on the database.


If you are using Oracle Database Standard Edition, your cluster must adhere to the license restrictions. Refer to Oracle Database Licensing Information for specific details on these restrictions

See Also:

Chapter 9, "Adding Nodes and Instances in Oracle Real Application Clusters Environments" for more information about adding nodes to your Oracle RAC cluster

An Oracle RAC database requires three components: cluster nodes, shared storage, and Oracle Clusterware. Although you can choose how many nodes your cluster should have and what type of shared storage to use, this guide describes one specific configuration for a two-node cluster. This two-node configuration uses Automatic Storage Management (ASM) for storage management and Recovery Manager (RMAN) for the backup and recovery strategy.

Most administration tasks are the same for Oracle single-instance and Oracle RAC databases. This guide provides additional instructions for some of the database administration tasks specific to Oracle RAC, as well as some recommendations for managing Oracle RAC databases.

Administering Oracle RAC with Enterprise Manager

The Web-based Oracle Enterprise Manager Database Control console and the Oracle Enterprise Manager Grid Control console let you manage Oracle RAC databases. Enterprise Manager is a central point of control for the Oracle environment that you access by way of a graphical user interface (GUI). You can use Enterprise Manager to create and modify services, and to start and stop the cluster database instances and the cluster database. Use Enterprise Manager Database Control for cluster database management tasks. Use Enterprise Manager Grid Control to administer your entire Oracle RAC environment, not just the Oracle RAC database.

Also note the following points about monitoring Oracle RAC environments:

  • Performance monitoring features, such as Automatic Workload Repository and Statspack, are Oracle RAC-aware.


    Instead of using Statspack, Oracle recommends that you use the more sophisticated management and monitoring features of the Oracle Database 10g Diagnostic and Tuning packs which include the Automatic Workload Repository (AWR).
  • You can use global dynamic performance views, or GV$ views, to view statistics across instances. These views are based on the single-instance V$ views.

When you log in to Enterprise Manager using a client browser, the Cluster Database Home page appears. The Cluster Database Home page is similar to a single-instance Database Home page. However, on the Cluster Database Home page, Enterprise Manager displays the system state and availability of the entire Oracle RAC database. This includes a summary about alert messages and job activity, as well as links to all the database and Automatic Storage Management (ASM) instances.

See Also:

Oracle Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide for more information about monitoring Oracle RAC performance

Overview of Oracle Real Application Clusters: Oracle By Example Series

Oracle By Example (OBE) has a series of tutorials created for Oracle Database 2 Day DBA. Included in this series is an OBE tutorial that introduces you to the management of an Oracle RAC database using Enterprise Manager. To view this OBE tutorial, go to the following URL


Starting and Stopping Oracle RAC Databases and Database Instances

Typically, you start up and shut down the cluster database from the Enterprise Manager Cluster Database Home page. By using this page for cluster database startup and shutdown operations, you ensure that all the instances that belong to the Oracle RAC database are in a consistent state. This enables you to more easily manage an Oracle RAC database.

You can also start and stop individual instances in an Oracle RAC database. However, starting and stopping one instance in an Oracle RAC database does not stop or start other instances. To completely stop an Oracle RAC database, you must shut down all of its instances.

To start and stop an entire Oracle RAC database, assuming you are using a server parameter file (SPFILE):

  1. Point your Web browser to the following URL and log in to Enterprise Manager:


    For example, http://docrac1.mycompany.com:1158/em.

  2. On the Cluster Database Home page, in the General section, click Startup if the database is down, or Shutdown if the database is started.

  3. On the Startup/Shutdown: Specify Credentials page, enter the cluster database host credentials for the database nodes. The host credentials are the user name and password for a user who is a member of the OSDBA or OSOPER operating system group.

  4. On the Startup/Shutdown: Select Operation page, click Startup All to start all the instances, or click Shutdown All to stop all the instances.

  5. On the Startup/Shutdown: Confirmation page, click Yes.

To start and stop individual instances, go to the Oracle RAC Database Startup and Shutdown page and select the instance that you want to start or stop. Then start or stop the instance as needed.


You can start up and shut down individual instances from each instance's home page. However, it is easier to perform instance startup and shutdown operations directly from the Database Startup and Shutdown page.

You can also start up and shut down instances with SQL*Plus or Server Control (SRVCTL).

See Also:

Oracle Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide for more information about using command-line interfaces to start and stop Oracle RAC database instances

Viewing and Modifying Oracle RAC Initialization Parameters

Managing initialization parameters for an Oracle RAC database is essentially the same as managing them for a single-instance Oracle database. Note the following differences for parameters in Oracle RAC databases:

  • Parameters that are cluster-specific have the value Cluster Database in the Category column.

  • Parameters that are the same on each instance in the Oracle RAC database are identified in the Instance column with an asterisk (*).

  • Parameters that are set to different values on each instance of an Oracle RAC database are listed by instance number.

The administration of initialization parameters in Oracle RAC environments is slightly different from single-instance database parameter administration. For example, if you change a parameter setting that is marked by an asterisk, which indicates that the parameter is a clusterwide database initialization parameter, then you change that parameter's setting for all the instances in your Oracle RAC database. If you change an initialization parameter prefixed with an instance name, or an instance-specific initialization parameter, then the change affects only that instance; the change does not affect the parameter's settings on other database instances.

See Also:

Oracle Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide for more information about initialization parameters in Oracle RAC environments

Configuring Initialization Parameters for an Oracle RAC Database

By default, Oracle Database sets most parameters to a default value and this value is the same across all instances. However, many initialization parameters can also have different values on different instances as described in Oracle Database Reference. Other parameters must either be unique or identical across instances, as described in the following sections:

Parameters that Must Have Identical Settings on All Instances

Certain initialization parameters that are critical at database creation or that affect certain database operations must have the same value for every instance in an Oracle RAC database. Specify these parameter values in the SPFILE, or within the individual PFILEs for each instance. The following list contains the parameters that must be identical on every instance:


















The setting for DML_LOCKS must be identical on every instance only if set to zero.

Parameters that Must Have Unique Settings on All Instances

Oracle RAC uses the INSTANCE_NUMBER parameter to distinguish among instances at startup. Oracle RAC uses the number value of the THREAD parameter to assign redo log groups to specific instances. To simplify administration, use the same number for both the THREAD and INSTANCE_NUMBER parameters for each instance.

If you use the ROLLBACK_SEGMENTS parameter to specify the names of the rollback segments to be used for storing the undo of each instance, then Oracle recommends you use the instance SID as part of each unique rollback segment name. If the parameter UNDO_MANAGEMENT is set to AUTO, automatic undo management mode is used by the Oracle RAC database, and the setting of ROLLBACK_SEGMENTS is ignored. When using automatic undo management, Oracle Database generates unique names for the undo segments used by each instance.

If you use automatic undo management in your Oracle RAC database, then set the UNDO_TABLESPACE parameter to a different undo tablespace for each instance.

Parameters that Should Have Identical Settings on All Instances

Oracle recommends that you set the values for the following parameters to the same value on all instances. Although you can have different settings for these parameters on different instances, setting each parameter to the same value on all instances simplifies administration:


    Different values for instances in your Oracle RAC database are likely to increase overhead because of additional automatic synchronization performed by the database processing.

    When using Oracle Streams with your Oracle RAC database, the value should be greater than zero.


    This parameter determines a databasewide limit on the number of users defined in the database and it is useful to have the same value on all instances of your database so you can see the current value no matter which instance you are using. Setting different values may cause additional warning messages to be generated during instance startup or cause commands related to database user management to fail on some instances.


    If you do not use the same value for all your instances, then you unnecessarily complicate media recovery. The recovering instance expects the required archive log file names to have the format defined by its own value of LOG_ARCHIVE_FORMAT, regardless of which instance created the archive log files.

    Databases that support Oracle Data Guard, either to send or receive archive log files, must use the same value of LOG_ARCHIVE_FORMAT for all instances.


    If this parameter does not identify the same file to all instances, then each instance may act differently and unpredictably in failover, load-balancing, or standard operations. Additionally, a change you make to the SPFILE with an ALTER SYSTEM SET or ALTER SYSTEM RESET command is saved only in the SPFILE used by the instance where you run the command. Your change will not be reflected in instances using different SPFILEs.

    If the SPFILE values are different in instances for which the values were set by the server, then you should restart the instances that are not using the default SPFILE.


    By setting different values for UNDO_RETENTION in each instance, you are likely to reduce scalability and encounter unpredictable actions following a failover. Therefore, you should carefully consider whether or not you will accrue any benefits before you assign different values for this parameter to the instances in your Oracle RAC database.

Editing Initialization Parameter Settings for an Oracle RAC Database

To view or modify the initialization parameters with Enterprise Manager, click on the Administration tab on the Cluster Database home page. On the Administration page, click Initialization Parameters under Database Configuration. On the Initialization Parameters page, you can click either Current or SPFile to modify the parameter settings.

Modifying Initialization Parameter for Oracle RAC Using the Current Tab

Click the Current tab of the Initialization Parameters page in Enterprise Manager to manage the initialization parameter settings for your cluster database. The initialization parameters file contains a list of configuration parameters for that instance and database. You can set these parameters to particular values to initialize many of the memory and process settings of an Oracle instance.

You can filter the Initialization Parameters page to show only those parameters that meet the criteria of the filter you enter in the Filter by name field. Optionally, you can select Show All to display on one page all parameters currently used by the running instance(s).

The Instance column shows the instances for which the parameter has the value listed in the table. An asterisk (*) indicates that the parameter has the same value for all remaining instances of the cluster database. For example, if open_cursors = 200 for docrac1 and docrac2, and open_cursors = 275 for docrac3, then the Instance column for open_cursors = 200 displays an asterisk, while displaying "docrac3" for open_cursors = 275. This shorthand saves space for cluster databases with many instances.

To modify the parameter values, select a parameter from the Select column and do one of the following:

  • Click Add to add the selected parameter to a different instance. Enter a new instance name and value in the newly created row in the table.

  • Click Reset to reset the value of the selected parameter. Note that you may only reset only parameters that do not have an asterisk in the Instance column. The value of the selected column is reset to the value of the remaining instances (that is, the row with the asterisk).

After you make changes to one or more of the parameters, click Apply to accept and invoke the changes.

Modifying Initialization Parameter for Oracle RAC Using the SPFile Tab

From the Initialization Parameters page for cluster databases, select the SPFile tab to display and change the parameters for the current server parameter file. A server parameter file is a type of repository for initialization parameters that is maintained on the server where the Oracle database server runs. It is a server-side initialization parameter file. Initialization parameters stored in a server parameter file are persistent, in that any changes made to the parameters while an instance is running can persist across instance shutdown and startup. Click Apply changes in SPFile mode... to apply the changes to the running instance.

Select Running to show the parameters for the current running instance of the initialization parameters file. For more information about each parameter, click the parameter name in the Name column.

Similar to the Current tab, you can Add or Reset parameters. Note that resetting parameters in the SPFile tab is different than resetting them in the Current tab: Reset deletes the selected parameter entry from the SPFILE and applies to both asterisk and non-asterisk parameters. If you reset a parameter with an asterisk in the Instance column, the entry will be deleted from both the SPFILE and the table. Only parameters for non-asterisk instances will remain. If you reset the remaining entry for a parameter, it will be deleted from both the SPFILE and the table, but will be replaced by a dummy parameter with an empty value field and an asterisk in the Instance column; this enables you to specify a new value, add new entries, and so on.

Resetting a parameter with only one instance resets the value of that parameter.

Example: Modifying the OPEN_CURSORS Parameter

Suppose that the open_cursors parameter has two entries in the SPFILE:

*.open_cursors = 200 
docrac1.open_cursors = 250

If you click Reset for *.open_cursors, then Enterprise Manager deletes that entry from both the SPFILE and the displayed list of parameters, leaving only docrac1.open_cursors = 250 displayed. If you click Reset for docrac1.open_cursors, Enterprise Manager also deletes this parameter entry from both the SPFILE and the displayed list of parameters, but then a new entry, *.open_cursors = <EMPTY> is added to the displayed list of parameters in place of the reset parameter.

See Also:

Oracle Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide for more information about using a server parameter file in an Oracle Real Application Clusters environment

Modifying the SERVICE_NAMES Parameter for Oracle RAC

The SERVICE_NAMES initialization parameter specifies one or more names by which clients can connect to the instance. The instance registers its service names with the Listener. When a client requests a service, the Listener determines which instances offer the requested service and routes the client to the appropriate instance.

In an Oracle RAC database, you should not modify this parameter directly. Instead, define services for your database and database instances using the Clustered Managed Database Services page in Enterprise Manager. If you need to change a service, you can use either Enterprise Manager or SRVCTL.

See Also:

Chapter 7, "Managing Database Workload Using Services" for more information about using services with Oracle RAC

Configuring the Server Parameter File for Oracle Real Application Clusters

When you create the database, Oracle creates an SPFILE in the file location that you specify. This location can be an ASM disk group, cluster file system file, or a shared raw device. In the environment described by this guide, the SPFILE is created on an ASM disk group.

All instances in the cluster database use the same SPFILE at startup. Oracle RAC uses a traditional PFILE only if an SPFILE does not exist or if you specify PFILE in your STARTUP command. Oracle recommends that you use SPFILE to simplify administration, maintain parameter setting consistency, and to guarantee parameter setting persistence across database shutdown and startup events. In addition, you can configure RMAN to back up your SPFILE.

Administering Storage in Oracle RAC

Most administration tasks for managing storage are the same for Oracle single-instance and Oracle RAC databases. This section provides additional instructions for using Enterprise Manager to manage some of the storage structures of an Oracle RAC database.

This section describes the following topics:

Administering Automatic Undo Management in Oracle RAC

Oracle automatically manages undo segments within a specific undo tablespace that is assigned to an instance. Only the instance assigned to the undo tablespace can modify the contents of that tablespace. However, each instance can read the undo data blocks created by any instance. Also, when performing transaction recovery, any instance can update any undo tablespace, as long as that undo tablespace is not currently being used by another instance for undo generation or transaction recovery. You assign undo tablespaces in your Oracle RAC database by specifying a different value for the UNDO_TABLESPACE parameter for each instance in your SPFILE or individual PFILEs. You cannot simultaneously use automatic undo management and manual undo management in an Oracle RAC database. In other words, all instances of an Oracle RAC database must operate in the same undo mode.

See Also:

Oracle Database 2 Day DBA for more information about managing the undo data for your database

Administering Automatic Storage Management in Oracle RAC

ASM automatically optimizes storage to maximize performance by managing the storage configuration across the disks that ASM manages. ASM does this by evenly distributing the storage load across all the available storage within your cluster database environment. ASM partitions your total disk space requirements into uniformly sized units across all the disks in a disk group. ASM can also automatically mirror data to prevent data loss. Because of these features, ASM also significantly reduces your administrative overhead.

As in single-instance Oracle databases, using ASM in Oracle RAC does not require I/O tuning. The following topics describe ASM and ASM administration:

About Automatic Storage Management Components in Oracle RAC

When you create your database, Oracle Database creates one ASM instance on each node in your Oracle RAC environment if one does not already exist. Each ASM instance has either an SPFILE or PFILE type parameter file. For the environment described in this guide, the ASM instances use PFILEs.

Modifying Disk Group Configurations for ASM in Oracle RAC

When you create a disk group for a cluster, or add new disks to an existing clustered disk group, you must prepare only the underlying physical storage on shared disks. The shared disk requirement is the only substantial difference between using ASM in an Oracle RAC database compared to using it in a single-instance Oracle database. ASM automatically rebalances the storage load after you add or delete a disk or disk group.

In a cluster, each ASM instance manages the metadata updates to the disk groups for the node on which it is running. In addition, each ASM instance coordinates disk group metadata with other nodes in the cluster. As in single-instance Oracle databases, you can use Enterprise Manager, DBCA, SQL*Plus, and SRVCTL to administer disk groups for ASM in an Oracle RAC environment.

See Also:

Oracle Database Administrator's Guide for information on how to use SQL*Plus to administer ASM instances

Standalone ASM Disk Group Management

When you create a database using DBCA and you select the ASM storage option, DBCA creates the ASM instances for you if they do not already exist. However, you can also use the standalone ASM disk group management feature to create and manage an ASM instance and its associated disk groups independently of creating a new database.

Administering ASM Instances and Disk Groups with Enterprise Manager

You can perform administrative operations on ASM disk groups with Enterprise Manager such as adding and deleting them. You can also monitor ASM disk group performance as well as control disk group availability at the instance level. For example, some of the Oracle RAC-specific features for ASM that are provided by Enterprise Manager are:

  • When you add a disk group, the disk group definition includes a check box to indicate whether or not the disk group is automatically mounted to all the cluster database instances.

  • The default Disk Group Performance page displays instance-level performance details when you click a performance characteristic such as Write Response Time or I/O Throughput.

  • When you mount and dismount ASM disk groups, you can use a check box to indicate which instances should mount or dismount a particular ASM Disk Group.

Administering Redo Logs in Oracle RAC

Managing redo log files in Oracle RAC environments is similar to managing redo log files in single-instance Oracle Database environments. This section provides an overview of some of the additional concepts and procedures for configuring redo log files in Oracle RAC environments.

About Redo Log Groups and Redo Threads in Oracle RAC Databases

Redo logs contain a record of changes that have been made to datafiles. In a single-instance Oracle database, redo logs are stored in two or more redo log file groups. Each of these groups contains a redo log file and possibly one or more mirrored copies of that file. In an Oracle RAC database, each instance requires its own set of redo log groups, which is known as a redo thread. Mirrored copies of the redo log files provide your system with extra protection against data loss that is due to hardware failures or data corruption. If a redo log file is unreadable, then the Oracle Database attempts to access its mirrored copy. You should place the redo log file mirrors on different disk devices from the primary redo log files.

Each instance's redo thread must contain at least two redo log groups. Oracle recommends that each of your instances has a redo thread that contains the same number of redo log groups and, as with single-instance Oracle databases, each group should contain the same number of members. For example, in an Oracle RAC database with two instances, each instance could have a redo thread that contains five redo log groups. This is a total of 10 redo log groups for the database. Each of these redo log groups could contain two members: a redo log and its mirrored copy. If you create your Oracle RAC database with DBCA, then your Oracle RAC database automatically implements a configuration that meets the Oracle recommendations.

In an Oracle RAC database, each instance writes and archives the redo log groups in its redo thread in the same manner that single-instance Oracle databases do. However, in recovery mode, the instance performing the recovery is able to read and process all the redo threads for the database, regardless of which instance generated the redo thread. This enables a running instance to recover the work completed by one or more failed instances. This also enables users to continue their work without waiting for the failed instance to be restarted. For example, assume that you have an Oracle RAC database with two instances, instance A and instance B. If instance A is down, then instance B can read the redo log files for both instance A and B to ensure a successful recovery.

In an Oracle RAC database, all the redo log files reside on shared storage. In addition, each instance must have access to the redo log files of all the other instances in the cluster. If your Oracle RAC database uses ASM, then ASM manages the shared storage for the redo log files and the access to those files.

Using Enterprise Manager to View and Create Online Redo Log Files

To access the redo log file groups with Enterprise Manager:

  1. Go to the Cluster Database Home Page, and click the Administration tab.

  2. On the Cluster Database Administration page, under the Storage column, select Redo Log Groups.

On the Redo Log Groups page, you can create additional redo log groups and add members to the redo log group. The Thread column identifies the instance, or redo thread, to which a redo log file belongs.

See Also:

Exploring Your Cluster Database: Oracle By Example Series

Oracle By Example (OBE) has a series of tutorials for Oracle RAC databases. This OBE steps you through the basic administrative tasks described in this chapter and includes annotated screen shots.

To view the Exploring Your Cluster Database OBE, go to the Web site