Skip Headers

Oracle® Real Application Clusters Administrator's Guide
10g Release 1 (10.1)

Part Number B10765-01
Go to Documentation Home
Go to Book List
Book List
Go to Table of Contents
Go to Index
Go to Master Index
Master Index
Go to Feedback page

Go to previous page
Go to next page
View PDF

3 Administering Storage

This chapter describes storage topics such as Automated Storage Management (ASM) and the Oracle Cluster Registry (OCR). The topics in this chapter are:

Overview of Storage in Oracle Real Application Clusters

Storage for RAC databases must be shared. In other words, datafiles must reside on either a cluster file system or on shared raw devices. Additionally, for each instance you must create at least two redo log files that reside on shared storage devices. You must also create one shared device for each instance to store its own tablespace for automatic undo management. If needed, you can use a local file system to store, for example, client-side parameter files (PFILEs).

Unless otherwise noted, Oracle storage features such as Automatic Storage Management (ASM), Oracle Managed Files, automatic segment-space management, and so on, function the same in RAC databases as they do in Oracle single-instance databases. Refer to the Oracle Database Administrator's Guide for additional information about these storage features.

If your platform does not support a cluster file system or if you do not want to use a cluster file system to store datafiles for RAC, then you must create additional raw devices as described inOracle Database 2 Day DBA and Oracle Real Application Clusters Installation and Configuration Guide. Oracle recommends that you use ASM for datafile storage as described in the next section.

Automatic Storage Management

ASM automatically optimizes storage to maximize performance by rebalancing the storage configuration across the disks that ASM manages. ASM spreads the storage load across all available storage within your cluster database environment for optimal performance. ASM partitions your total disk space into uniformly sized units across all disks in a disk group. ASM can also automatically mirror data to prevent data loss.

To use ASM in RAC, you only need to select ASM as your storage option when you create your database with the Database Configuration Assistant (DBCA). When your database is operating, storage management only requires you to administer disk groups as described in this section. As in single-instance Oracle databases, using ASM in RAC does not require I/O tuning.

Automatic Storage Management Components in RAC

When you use ASM, there is an ASM instance on each node of a RAC database. Each ASM instance has either an SPFILE or PFILE type parameter file. You may need to back up the parameter files and the TNS entries if you are using a non-default Oracle Net Listener.

Modifying Disk Group Configurations for ASM in RAC

When you add disk groups in RAC, you only need to prepare the underlying operating system. ASM automatically rebalances the storage load after you add a disk group or when you delete a disk group.

When you add disk groups, you can also set a redundancy level and identify failure groups which are sets of disks that share a common resource. ASM uses failure groups to store redundant copies of your data to ensure that the loss of one failure group does not result in data loss. The default redundancy level is normal, which mirrors to two failure groups. You can also specify high redundancy for three-way mirroring. You can only use failure groups if you enable mirroring.

In RAC, each ASM instance manages all disk and failure groups for each node. In addition, each ASM instance coordinates disk and failure group information with other nodes in a RAC cluster database. As in single-instance Oracle databases, you can use the DBCA, SQL*Plus, and the Server Control Utility (SRVCTL) to administer disk groups for ASM in RAC as described in the following sections.

Administering ASM Instances and ASM Disk Groups with Enterprise Manager in RAC

You can add or administer Automatic Storage Management (ASM) disk groups with Enterprise Manager. You can also monitor ASM disk group performance and control disk group availability at the instance level. For example, the additional ASM-related Enterprise Manager tasks that you can perform in RAC are:

  • When you add a disk group, the disk group definition includes a checkbox to indicate whether the disk group is automatically mounted to all of the cluster database instances.

  • When you examine the default Disk Group Performance page, you can see an instance-level performance details by clicking a performance characteristic such as Write Response Time or I/O Throughput.

  • When you mount and dismount ASM disk groups, you can use a checkbox to indicate which instances mount or dismount the ASM Disk Group.

See Also:

Oracle Database Administrator's Guide and Oracle Database 2 Day DBA for detailed information about using Enterprise Manager

Administering ASM Instances with SRVCTL in RAC

You can use the Server Control Utility (SRVCTL) to add, remove, enable, and disable an ASM instance as described in the following procedures:

Use the following syntax to add configuration information about an existing ASM instance:

srvctl add asm -n <node_name> -i <asm_instance_name> -o <oracle_home>


For all of the SRVCTL commands in this section for which the -i option is not required, if you do not specify an instance name, then the command applies to all the ASM instances on the node.

Use the following syntax to remove an ASM instance:

srvctl remove asm -n <node_name> [-i <asm_instance_name>]

Use the following syntax to enable an ASM instance:

srvctl enable asm -n <node_name> [-i <asm_instance_name>]

Use the following syntax to disable an ASM instance:

srvctl disable asm -n <node_name> [-i <asm_instance_name>]

You can also use SRVCTL to start, stop, and obtain the status of an ASM instance as in the following examples.

Use the following syntax to start an ASM instance:

srvctl start asm -n <node_name> [-i <asm_instance_name>] [-o <start_options]

Use the following syntax to stop an ASM instance:

srvctl stop asm -n <node_name> [-i <asm_instance_name>] [-o <stop_options]

Use the following syntax to configure an ASM instance:

srvctl config asm -n <node_name>

Use the following syntax to obtain the status of an ASM instance:

srvctl status asm -n <node_name>

Datafile Access in Real Application Clusters

All RAC instances must be able to access all datafiles. If a datafile needs to be recovered when the database is opened, then the first RAC instance to start performs the recovery and verifies access to the file. As other instances start up, they also verify their access to the datafiles. Similarly, when you add a tablespace or datafile or bring a tablespace or datafile online, all instances verify access to the file or files.

If you add a datafile onto a disk that other instances cannot access, then verification fails. Verification also fails if instances access different copies of the same datafile. If verification fails for any instance, then diagnose and fix the problem. Then execute the ALTER SYSTEM CHECK DATAFILES statement on each instance in your RAC database to verify datafile access.

Redo Log File Storage in Real Application Clusters

Each instance has its own online redo log groups which are called an instance's thread of online redo. Create these online redo log groups and establish group members as described in the Oracle Database Administrator's Guide.

Each instance must have at least two groups of online redo log files in its own thread. When the current group fills, an instance begins writing to the next log file group. If your database is in ARCHIVELOG mode, then each instance must save filled log files into its own archive log thread and update the control file with the status of its thread.


MAXLOGHISTORY is useful for sites with demanding availability requirements. This option can help you administer recovery, especially when there are many instances and many log files.

Automatic Undo Management in Real Application Clusters

Oracle automatically manages undo segments within a specific undo tablespace that is assigned to an instance. Only the instance assigned to the undo tablespace can modify the contents of that tablespace. However, all instances can always read all undo blocks for consistent read purposes. Also, any instance can update any undo tablespace during transaction recovery, as long as that undo tablespace is not currently used by another instance for undo generation or transaction recovery. You assign undo tablespaces in your RAC database by specifying a different value for the UNDO_TABLESPACE parameter for each instance in your SPFILE or individual PFILEs. You cannot simultaneously use automatic undo management and manual undo management in a RAC database. In other words, all instances of a RAC database must operate in the same undo mode.

See Also:

Oracle Database Administrator's Guide for detailed information about creating and managing undo tablespaces

Switching Undo Tablespaces for Instances in Real Application Clusters

You can dynamically switch undo tablespace assignments. Redirect the assignment for an instance by executing the ALTER SYSTEM SET UNDO_TABLESPACE statement. You might do this during planned maintenance to redirect an instance to use an idle undo tablespace. For example, assume that you have specified instances db1 and db2 to access undo tablespaces undotbs01 and undotbs02 respectively, and that you have an idle undo tablespace undotbs03. To redirect instance db1 to use undo tablespace undotbs03, execute the following SQL statement from instance db1:


In this example, the previously used undo tablespace undotbs01 remains assigned to instance db1 until the instance's last active transaction commits. User transactions proceed while Oracle performs the switching operation as follows:

  • The switching process does not wait for all user transactions to commit.

  • Oracle places the previous undo tablespace in a pending-offline state if there are active transactions in that tablespace. This means that the pending-offline tablespace may be unavailable for other instances until all transactions against that tablespace are committed.

Refer to the Oracle Database Administrator's Guide for more information about the ALTER SYSTEM SET UNDO_TABLESPACE statement and information about setting the undo retention period and dropping undo tablespaces.

Managing the Oracle Cluster Registry in Real Application Clusters

This section describes managing the Oracle Cluster Registry (OCR). The OCR contains cluster and database configuration information for RAC and Cluster Ready Services (CRS) such as the cluster node list, and cluster database instance to node mapping, and CRS application resource profiles.

There are two methods of copying OCR content and using the content for recovery. The first method uses automatically generated physical OCR file copies and the second method uses manually created logical OCR export files as described in the following sections:

Managing Oracle Cluster Registry Backup Files in Real Application Clusters

One CRS instance in a cluster automatically creates OCR backups every four hours. At any one time, the last three backup copies are always retained. The CRS instance also creates and retains an OCR backup for each full day and at the end of each week. You cannot customize the backup frequencies or the number of files that Oracle retains. However, because of the importance of OCR information, Oracle recommends that you use the ocrconfig tool to make copies of the automatically generated backup files. at least once daily. The ocrconfig tool commands are described later in this section.

Use the ocrconfig tool with the showbackup option to identify the backup files. Then copy each file to a location that is redundant to the location of the original OCR backup file. This ensures that there are at least two copies of each OCR backup file. Oracle also recommends that the OCR location reside on RAID arrays. If possible, use a backup location that is shared by all nodes in the cluster. The default target location of each OCR backup file is as follows where <cluster name> is the name you assigned to the cluster when you installed CRS:

<CRS Home>/cdata/<cluster name>

You can change this location using the following syntax:

ocrconfig -backuploc <directory name>

Restoring the Oracle Cluster Registry from Automatically Generated OCR Backups

Before attempting to resolve any configuration-related problems, retry the application that failed. If the problem continues, then use one of the following platform-specific procedures to restore the OCR configuration.

Restoring the Oracle Cluster Registry on UNIX-Based Systems

  1. Stop the following OCR clients: CSSD, CRSD, EVMD, and the EVM logger daemons.

  2. Shut down all the nodes in the cluster and restart from one node in single-user mode.

  3. Identify the recent backups using the ocrconfig -showbackup command.

  4. Execute the restore by applying an OCR backup file identified in Step 3 with the ocrconfig -restore <file name> command.

  5. Restart the CRS, CSS, and EVM daemons and resume operations in cluster mode.

Restoring the Oracle Cluster Registry on Windows-Based Systems

  1. Shut down all but one node in your RAC database.

  2. Disable the following OCR clients and stop them using the Service Control Panel: OracleClusterVolumeService, OracleCSService, OracleCRService, and the OracleEVMService.

  3. Identify the recent backups using the ocrconfig -showbackup command.

  4. Execute the restore by applying an OCR backup file identified in Step 3 with the ocrconfig -restore <file name> command.

  5. Start all the services that were stopped in step 2. Restart all the instances and resume operations in cluster mode.

Managing the Oracle Cluster Registry with OCR Exports

In addition to using the automatically created OCR backup files, you should also export the OCR contents before and after making significant configuration changes. Do this by using the ocrconfig -export command. This enables you to restore the OCR if your configuration changes cause errors. For example, if you have unresolvable configuration problems, or if you are unable to restart your clusterware, restore your configuration using one of the following platform-specific procedures:

Importing Oracle Cluster Registry Content on UNIX-Based Systems

  1. Shut down all the nodes in the cluster and restart one of them in single-user mode.

  2. Import an OCR export file using the ocrconfig -import command from any node.

  3. Start all the nodes in the cluster in multi-user mode.

To avoid stopping the entire cluster, use this alternate method:

  1. Update inittab entries on all the nodes and remove the CRS-specific entries. Retain a copy of the original inittab file.

  2. Run the /etc/init.d/ stop command on all the nodes.

  3. Import the OCR export file using the ocrconfig -import command from any node.

  4. Restore the original inittab file on all nodes.

  5. Run the /etc/init.d/ start command on all nodes.

  6. Run the /etc/init q command.

Importing Oracle Cluster Registry Content on Windows-Based Systems

  1. Stop the following OCR clients using the Service Control Panel: OracleClusterVolumeService, OracleCMService, OracleEVMService, OracleCSService, and the OracleCRService.

  2. Import an OCR export file using the ocrconfig -import command from one node.

  3. Re-start all of the affected services on all nodes.

The ocrconfig Tool Command Syntax and Options

Execute OCR commands with administrative privileges on UNIX-based platforms or as a user with Administrator privileges on Windows-based systems. The OCR command syntax is as follows where <options> is a verb in Table 3-1:

ocrconfig -<option>

Table 3-1 The ocrconfig Command Options

Option Purpose
-export To export the contents of the OCR into a target file.
-import To import OCR contents from a previously exported OCR file.
-restore To restore the OCR from an automatically created OCR backup file.
-backuploc To change OCR backup file location. For this entry, use a full path that is accessible by all nodes.
-showbackup To display the location, timestamp, and the node name of origin for the last three automatically created backup files.
-upgrade To upgrade the OCR to a later version.
-downgrade To downgrade the OCR to an earlier version.
-help To display help for the ocrconfig commands.

For example, to export the OCR contents to a binary file, use the ocrconfig command with the following syntax where file_name is the file to which you want to export the OCR contents as follows:

ocrconfig -export file_name

Do not attempt to edit the export file. If OCR clients such as the clusterware are running, then Oracle reports an error.

Implementing the Oracle Hardware Assisted Resilient Data Initiative for the OCR

The Oracle Hardware Assisted Resilient Data (HARD) initiative prevents data corruptions before they occur. With HARD enabled, the OCR writes HARD-compatible blocks. To determine whether the device used by the OCR supports HARD and enable it, review the Oracle HARD white paper at:

Upgrading and Downgrading the OCR Configuration in Real Application Clusters

When you install CRS, Oracle automatically executes the ocrconfig -upgrade command. To downgrade, follow the downgrade instructions for each component and also downgrade the OCR using the ocrconfig -downgrade command.

HARD-Compatible OCR Blocks in Oracle9i

In Oracle9i, the OCR did not write HARD-compatible blocks. If the device used by OCR is enabled for HARD, then use the method described in the HARD white paper to disable HARD for the OCR device before downgrading your OCR. If you do not disable HARD, then the downgrade operation fails.

Importing and Exporting Cluster Database Configuration Information with SRVCONFIG

Use SRVCONFIG to import and export SRVM cluster database configuration information to and from the OCR. For example, the following syntax exports the configuration information to a text file:

srvconfig -exp file_name.txt

The following syntax imports the configuration information from a text file into the OCR:

srvconfig -imp file_name.txt