This chapter explains instance recovery and how to use Recovery Manager (RMAN) to back up and restore Oracle Real Application Clusters (Oracle RAC) databases. This chapter also describes Oracle RAC instance recovery, parallel backup, recovery with SQL*Plus, and using the Fast Recovery Area in Oracle RAC. The topics in this chapter include:
For restore and recovery in Oracle RAC environments, you do not have to configure the instance that performs the recovery to also be the sole instance that restores all of the data files. In Oracle RAC, data files are accessible from every node in the cluster, so any node can restore archived redo log files.
Oracle Clusterware Administration and Deployment Guide for information about backing up and restoring the Oracle Clusterware components such as the Oracle Cluster Registry (OCR) and the voting disk
RMAN Backup Scenario for Noncluster File System Backups
In a noncluster file system environment, each node can back up only to a locally mounted noncluster file system directory. For example,
node1 cannot access the archived redo log files on
node3 unless you configure the network file system for remote access. If you configure a network file system file for backups, then each node backs up its archived redo logs to a local directory.
RMAN Restore Scenarios for Oracle RAC
This section describes the following common RMAN restore scenarios:
The restore and recovery procedures in a cluster file system scheme do not differ substantially from Oracle noncluster scenarios.
Restoring Backups from a Cluster File System
The scheme that this section describes assumes that you are using the "Oracle Automatic Storage Management and Cluster File System Archiving Scheme". In this scheme, assume that
node3 performed the backups to a cluster file system. If
node3 is available for the restore and recovery operation, and if all of the archived logs have been backed up or are on disk, then run the following commands to perform complete recovery:
RESTORE DATABASE; RECOVER DATABASE;
node3 performed the backups but is unavailable, then configure a media management device for one of the remaining nodes and make the backup media from
node3 available to this node.
If you configured RMAN as described in "Configuring Channels to Use Automatic Load Balancing", then, to load balance the channels across nodes, note that channels cannot be load balanced before at least one instance has successfully opened the database. This means that the channels will not be load balanced across the nodes during a full database restore. To achieve load balancing of channels for
RECOVER commands, you can temporarily reallocate channels by running commands similar to the following:
ALLOCATE CHANNEL DEVICE TYPE sbt C1 CONNECT '@racinst_1'
ALLOCATE CHANNEL DEVICE TYPE sbt C2 CONNECT '@racinst_2'
Restoring Backups from a Noncluster File System
The scheme that this section describes assumes that you are using the "Noncluster File System Local Archiving Scheme". In this scheme, each node archives locally to a different directory. For example,
node1 archives to
node2 archives to
node3 archives to
/arc_dest_3. You must configure a network file system file so that the recovery node can read the archiving directories on the remaining nodes.
If all nodes are available and if all archived redo logs have been backed up, then you can perform a complete restore and recovery by mounting the database and running the following commands from any node:
Because the network file system configuration enables each node read access to the redo log files on other nodes, then the recovery node can read and apply the archived redo logs located on the local and remote disks. No manual transfer of archived redo logs is required.
Using RMAN or Oracle Enterprise Manager to Restore the Server Parameter File (SPFILE)
RMAN can restore the server parameter file either to the default location or to a location that you specify.
You can also use Oracle Enterprise Manager to restore the SPFILE. From the Backup/Recovery section of the Maintenance tab, click Perform Recovery. The Perform Recovery link is context-sensitive and navigates you to the SPFILE restore only when the database is closed.
Instance Recovery in Oracle RAC
Learn about instance recovery in Oracle RAC.
Instance failure occurs when software or hardware problems disable an instance. After instance failure, Oracle Database automatically uses the online redo logs to perform recovery as described in this section.
Single Node Failure in Oracle RAC
Applications that were running on a node before it failed continue running by using failure recognition and recovery. This provides consistent and uninterrupted service if hardware or software fails. When one instance performs recovery for another instance, the surviving instance reads online redo logs generated by the failed instance and uses that information to ensure that committed transactions are recorded in the database. Thus, data from committed transactions is not lost. The instance performing recovery rolls back transactions that were active at the time of the failure and releases resources used by those transactions.
All online redo logs must be accessible for instance recovery. Therefore, Oracle recommends that you mirror your online redo logs.
Multiple-Node Failures in Oracle RAC
When multiple node failures occur, if one instance survives, then Oracle RAC performs instance recovery for any other instances that fail. If all instances of an Oracle RAC database fail, then Oracle Database automatically recovers the instances the next time one instance opens the database. The instance performing recovery can mount the database in either cluster database or exclusive mode from any node of an Oracle RAC database. This recovery procedure is the same for Oracle Database running in shared mode as it is for Oracle Database running in exclusive mode, except that one instance performs instance recovery for all of the failed instances.
Using RMAN to Create Backups in Oracle RAC
Oracle Database provides RMAN for backing up and restoring the database.
RMAN enables you to back up, restore, and recover data files, control files, SPFILEs, and archived redo logs. RMAN is included with the Oracle Database server and it is installed by default. You can run RMAN from the command line or you can use it from the Backup Manager in Oracle Enterprise Manager. In addition, RMAN is the recommended backup and recovery tool if you are using Oracle Automatic Storage Management (Oracle ASM). The procedures for using RMAN in Oracle RAC environments do not differ substantially from those for Oracle noncluster environments.
Channel Connections to Cluster Instances with RMAN
Channel connections to the instances are determined using the connect string defined by channel configurations. For example, in the following configuration, three channels are allocated using
dbauser/pwd@service_name. If you configure the SQL Net service name with load balancing turned on, then the channels are allocated at a node as decided by the load balancing algorithm.
CONFIGURE DEVICE TYPE sbt PARALLELISM 3;
CONFIGURE DEFAULT DEVICE TYPE TO sbt;
CONFIGURE CHANNEL DEVICE TYPE SBT CONNECT 'dbauser/pwd@service_name'
However, if the service name used in the connect string is not for load balancing, then you can control at which instance the channels are allocated using separate connect strings for each channel configuration, as follows:
CONFIGURE DEVICE TYPE sbt PARALLELISM 3;
CONFIGURE CHANNEL 1.. CONNECT 'dbauser/pwd@mydb_1';
CONFIGURE CHANNEL 2.. CONNECT 'dbauser/pwd@mydb_2';
CONFIGURE CHANNEL 3.. CONNECT 'dbauser/pwd@mydb_3';
In the previous example, it is assumed that
mydb_3 are SQL*Net service names that connect to pre-defined nodes in your Oracle RAC environment. Alternatively, you can also use manually allocated channels to backup your database files. For example, the following command backs up the SPFILE, control file, data files and archived redo logs:
ALLOCATE CHANNEL CH1 CONNECT 'dbauser/pwd@mydb_1';
ALLOCATE CHANNEL CH2 CONNECT 'dbauser/pwd@mydb_2';
ALLOCATE CHANNEL CH3 CONNECT 'dbauser/pwd@mydb_3';
BACKUP DATABASE PLUS ARCHIVED LOG;
During a backup operation, if at least one channel allocated has access to the archived log, then RMAN automatically schedules the backup of the specific log on that channel. Because the control file, SPFILE, and data files are accessible by any channel, the backup operation of these files is distributed across the allocated channels.
For a local archiving scheme, there must be at least one channel allocated to all of the nodes that write to their local archived logs. For a cluster file system archiving scheme, if every node writes the archived logs in the same cluster file system, then the backup operation of the archived logs is distributed across the allocated channels.
node1 has the database mounted while the instances on
node3 have the database open, then the backup fails.
Oracle Database Backup and Recovery Reference for more information about the
CONNECT clause of the
CONFIGURE CHANNEL statement
Node Affinity Awareness of Fast Connections
In some cluster database configurations, some nodes of the cluster have faster access to certain data files than to other data files. RMAN automatically detects this situation, which is known as node affinity awareness. When deciding which channel to use to back up a particular data file, RMAN gives preference to the nodes with faster access to the data files that you want to back up. For example, if you have a three-node cluster, and if
node1 has faster read/write access to data files 7, 8, and 9 than the other nodes, then
node1 has greater node affinity to those files than
Deleting Archived Redo Logs after a Successful Backup
If you have configured the automatic channels as defined in section "Channel Connections to Cluster Instances with RMAN", then you can use the following example to delete the archived logs that you backed up
n times. The device type can be
DELETE ARCHIVELOG ALL BACKED UP n TIMES TO DEVICE TYPE device_type;
During a delete operation, if at least one channel allocated has access to the archived log, then RMAN automatically schedules the deletion of the specific log on that channel. For a local archiving scheme, there must be at least one channel allocated that can delete an archived log. For a cluster file system archiving scheme, if every node writes to the archived logs on the same cluster file system, then the archived log can be deleted by any allocated channel.
If you have not configured automatic channels, then you can manually allocate the maintenance channels as follows and delete the archived logs.
ALLOCATE CHANNEL FOR MAINTENANCE DEVICE TYPE DISK CONNECT 'SYS/oracle@node1';
ALLOCATE CHANNEL FOR MAINTENANCE DEVICE TYPE DISK CONNECT 'SYS/oracle@node2';
ALLOCATE CHANNEL FOR MAINTENANCE DEVICE TYPE DISK CONNECT 'SYS/oracle@node3';
DELETE ARCHIVELOG ALL BACKED UP n TIMES TO DEVICE TYPE device_type;
Autolocation for Backup and Restore Commands
RMAN automatically performs autolocation of all files that it must back up or restore. If you use the noncluster file system local archiving scheme, then a node can only read the archived redo logs that were generated by an instance on that node. RMAN never attempts to back up archived redo logs on a channel it cannot read.
During a restore operation, RMAN automatically performs the autolocation of backups. A channel connected to a specific node only attempts to restore files that were backed up to the node. For example, assume that log sequence 1001 is backed up to the drive attached to
node1, while log 1002 is backed up to the drive attached to
node2. If you then allocate channels that connect to each node, then the channel connected to
node1 can restore log 1001 (but not 1002), and the channel connected to
node2 can restore log 1002 (but not 1001).
Media Recovery in Oracle RAC
Media recovery must be user-initiated through a client application, whereas instance recovery is automatically performed by the database. In these situations, use RMAN to restore backups of the data files and then recover the database. The procedures for RMAN media recovery in Oracle RAC environments do not differ substantially from the media recovery procedures for noncluster environments.
The node that performs the recovery must be able to restore all of the required data files. That node must also be able to either read all of the required archived redo logs on disk or be able to restore them from backups.
When recovering a database with encrypted tablespaces (for example after a
SHUTDOWN ABORT or a catastrophic error that brings down the database instance), you must open the Oracle Wallet after database mount and before you open the database, so the recovery process can decrypt data blocks and redo.
Parallel Recovery in Oracle RAC
Oracle Database automatically selects the optimum degree of parallelism for instance, crash, and media recovery.
Oracle Database applies archived redo logs using an optimal number of parallel processes based on the availability of CPUs. You can use parallel instance recovery and parallel media recovery in Oracle RAC databases as described under the following topics:
Parallel Recovery with RMAN
RECOVER commands, Oracle Database automatically makes parallel the following three stages of recovery:
Restoring Data Files
When restoring data files, the number of channels you allocate in the RMAN recover script effectively sets the parallelism that RMAN uses. For example, if you allocate five channels, you can have up to five parallel streams restoring data files.
Applying Incremental Backups
Similarly, when you are applying incremental backups, the number of channels you allocate determines the potential parallelism.
Applying Archived Redo Logs
Disabling Parallel Recovery
Disabling Instance and Crash Recovery Parallelism
Using a Fast Recovery Area in Oracle RAC
To use a fast recovery area in Oracle RAC, you must place it on an Oracle ASM disk group, on a Cluster File System, or on a shared directory that is configured through a network file system file for each Oracle RAC instance.
In other words, the fast recovery area must be shared among all of the instances of an Oracle RAC database. In addition, set the parameter
DB_RECOVERY_FILE_DEST to the same value on all instances.
Oracle Enterprise Manager enables you to set up a fast recovery area. To use this feature:
From the Cluster Database home page, click the Maintenance tab.
Under the Backup/Recovery options list, click Configure Recovery Settings.
Specify your requirements in the Fast Recovery Area section of the page.
Click Help on this page for more information.