Oracle ASM Mirroring and Disk Group Redundancy
This section contains the following topics:
Mirroring, Redundancy, and Failure Group Options
If you specify mirroring for a file, then Oracle ASM automatically stores redundant copies of the file extents in separate failure groups. Failure groups apply to normal, high, flex, and extended redundancy disk groups. You can define the failure groups for each disk group when you create or alter the disk group.
There are multiple types of disk groups based on the Oracle ASM redundancy level. Table 4-2 lists the types with their supported and default mirroring levels. The default mirroring levels indicate the mirroring level with which each file is created unless a different mirroring level is designated.
Table 4-2 Mirroring options for Oracle ASM disk group types
Disk Group Type | Supported Mirroring Levels | Default Mirroring Level |
---|---|---|
EXTERNAL redundancy |
Unprotected (none) |
Unprotected |
NORMAL redundancy |
Two-way, three-way, unprotected (none) |
Two-way |
HIGH redundancy |
Three-way |
Three-way |
FLEX redundancy |
Two-way, three-way, unprotected (none) |
Two-way (newly-created) |
EXTENDED redundancy |
Two-way, three-way, unprotected (none) |
Two-way |
For normal and high disk group types, the redundancy level controls how many disk failures are tolerated without dismounting the disk group or losing data. Each file is allocated based on its own redundancy, but the default comes from the disk group.
For the flex group type, the number of failures tolerated before dismount demands on the number of failure groups. For five or more failure groups, two disk failures are tolerated. For three or four failure groups, one disk failure is tolerated.
For the extended disk group type, each site is similar to a flex disk group. If the site has five failure groups or more, two disk failures with in a site can be tolerated before the site becomes compromised. If the site has three or four failure groups, the site can tolerate one disk failure before the site is compromised. When two sites are compromised, the disk group dismounts. An extended disk group requires a minimum of three failure groups for each data site.
For flex and extended disk groups, mirroring describes the availability of the files within a disk group, not the disk group itself. For example: If a file is unprotected in a flex disk group that has five failure groups, then after one failure the disk group is still mounted, but the file becomes unavailable.
The redundancy levels are:
-
EXTERNAL redundancy
Oracle ASM does not provide mirroring redundancy and relies on the storage system to provide RAID functionality. Any write error causes a forced dismount of the disk group. All disks must be located to successfully mount the disk group.
-
NORMAL redundancy
Oracle ASM provides two-way mirroring by default, which means that all files are mirrored so that there are two copies of every extent. A loss of one Oracle ASM disk is tolerated. You can optionally choose three-way or unprotected mirroring.
A file specified with
HIGH
redundancy (three-way mirroring) in aNORMAL
redundancy disk group provides additional protection from a bad disk sector in one disk, plus the failure of another disk. However, this scenario does not protect against the failure of two disks. -
HIGH redundancy
Oracle ASM provides three-way (triple) mirroring by default. A loss of two Oracle ASM disks in different failure groups is tolerated.
-
FLEX redundancy
Oracle ASM provides two-way mirroring by default for newly-created flex disk groups. For migrated flex disk groups, the default values are obtained from the template values in the normal or high redundancy disk groups before migration. For migration from normal redundancy, if the template defaults were not changed, then the flex defaults are two-way mirroring. For migration from high redundancy, if the template defaults were not changed, then the flex defaults are three-way mirroring.
-
EXTENDED redundancy
Oracle ASM provides two-way mirroring by default. The redundancy setting describes redundancy within a data site. For example: If there is a two-way mirrored file in a two-data-site extended disk group, then there are four copies of the file, two in each data site.
Oracle ASM file groups in a flex or extended disk group can have different redundancy levels.
If there are not enough online failure groups to satisfy the file mirroring (redundancy attribute value) specified in the disk group file type template, Oracle ASM allocates as many mirrors copies as possible and subsequently allocates the remaining mirrors when sufficient online failure groups are available.
Failure groups enable the mirroring of metadata and user data. System reliability can diminish if your environment has an insufficient number of failure groups.
This section contains these topics:
See Also:
-
Managing Oracle ASM Flex Disk Groups for information about Oracle ASM flex disk groups, extended disk groups, and file groups
-
Managing Disk Group Templates for information about specifying Oracle ASM disk group templates
Oracle ASM Failure Groups
Failure groups are used to store mirror copies of data. When Oracle ASM allocates an extent for a normal redundancy file, Oracle ASM allocates a primary copy and a secondary copy. Oracle ASM chooses the disk on which to store the secondary copy so that it is in a different failure group than the primary copy. Each copy is on a disk in a different failure group so that the simultaneous failure of all disks in a failure group does not result in data loss.
A failure group is a subset of the disks in a disk group, which could fail at the same time because they share hardware. The failure of common hardware must be tolerated. Four drives that are in a single removable tray of a large JBOD (Just a Bunch of Disks) array should be in the same failure group because the tray could be removed making all four drives fail at the same time. Drives in the same cabinet could be in multiple failure groups if the cabinet has redundant power and cooling so that it is not necessary to protect against failure of the entire cabinet. However, Oracle ASM mirroring is not intended to protect against a fire in the computer room that destroys the entire cabinet.
There are always failure groups even if they are not explicitly created. If you do not specify a failure group for a disk, then Oracle automatically creates a new failure group containing just that disk, except for disk groups containing disks on Oracle Exadata cells.
A normal redundancy disk group must contain at least two failure groups. A high redundancy disk group must contain at least three failure groups. However, Oracle recommends using more failure groups. A small number of failure groups, or failure groups of uneven capacity, can create allocation problems that prevent full use of all of the available storage.
Oracle recommends a minimum of three failure groups for normal redundancy disk groups and five failure groups for high redundancy disk groups to maintain the necessary number of copies of the Partner Status Table (PST) and to ensure robustness with respect to storage hardware failures.
In the event of a system failure, three failure groups in a normal redundancy disk group allow a comparison among three PSTs to accurately determine the most up to date and correct version of the PST, which could not be done with a comparison between only two PSTs. Similarly with a high redundancy disk group, if two failure groups are offline, then Oracle ASM would be able to make a comparison among the three remaining PSTs.
If configuring an extra failure group presents a problem with storage capacity management, then a quorum failure group can be used as the extra failure group to store a copy of the PST. A quorum failure group does not require the same capacity as the other failure groups.
Failure groups can be specified as regular or quorum failure groups. For information about quorum failure groups, see "Storing Oracle Cluster Registry and Voting Files in Oracle ASM Disk Groups".
See Also:
Oracle Exadata documentation for information about Oracle Exadata failure groups
How Oracle ASM Manages Disk Failures
Depending on the redundancy level of a disk group and how you define failure groups, the failure of one or more disks could result in either of the following:
-
The disks are first taken offline and then automatically dropped. In this case, the disk group remains mounted and serviceable. In addition, because of mirroring, all of the disk group data remains accessible. After the disk drop operation, Oracle ASM performs a rebalance to restore full redundancy for the data on the failed disks.
-
The entire disk group is automatically dismounted, which means loss of data accessibility.
Guidelines for Using Failure Groups
The following are guidelines for using failure groups:
-
Each disk in a disk group can belong to only one failure group.
-
Failure groups should all be of the same size. Failure groups of different sizes may lead to reduced availability.
-
Oracle ASM requires at least two failure groups to create a normal redundancy disk group and at least three failure groups to create a high redundancy disk group.
Failure Group Frequently Asked Questions
This section discusses frequently asked questions about failure group under the following topics:
How Many Failure Groups Should I Create?
Choosing the number of failure groups to create depends on the types of failures that must be tolerated without data loss. For small numbers of disks, such as fewer than 20, it is usually best to use the default failure group creation that puts every disk in its own failure group.
Using the default failure group creation for small numbers of disks is also applicable for large numbers of disks where your main concern is disk failure. For example, a disk group might be configured from several small modular disk arrays. If the system must continue operating when an entire modular array fails, then a failure group should consist of all of the disks in one module. If one module fails, then all of the data on that module is relocated to other modules to restore redundancy. Disks should be placed in the same failure group if they depend on a common piece of hardware whose failure must be tolerated with no loss of availability.
How are Multiple Failure Groups Recovered after Simultaneous Failures?
A simultaneous failure can occur if there is a failure of a piece of hardware used by multiple failure groups. This type of failure usually forces a dismount of the disk group if all disks are unavailable.
When Should External, Normal, or High Redundancy Be Used?
Oracle ASM mirroring runs on the database server and Oracle recommends to off load this processing to the storage hardware RAID controller by using external redundancy. You can use normal redundancy in the following scenarios:
-
Storage system does not have RAID controller
-
Mirroring across storage arrays
-
Extended cluster configurations
In general, Oracle ASM mirroring is the Oracle alternative to third party logical volume managers. Oracle ASM mirroring eliminates the deployment of additional layers of software complexity in your Oracle Database environment.
Oracle ASM Recovery from Read and Write I/O Errors
Read errors can be the result of a loss of access to the entire disk or media corruptions on an otherwise a healthy disk. Oracle ASM tries to recover from read errors on corrupted sectors on a disk. When a read error by the database or Oracle ASM triggers the Oracle ASM instance to attempt bad block remapping, Oracle ASM reads a good copy of the extent and copies it to the disk that had the read error.
-
If the write to the same location succeeds, then the underlying allocation unit (sector) is deemed healthy. This might be because the underlying disk did its own bad block reallocation.
-
If the write fails, Oracle ASM attempts to write the extent to a new allocation unit on the same disk. If this write succeeds, the original allocation unit is marked as unusable. If the write fails, the disk is taken offline.
One unique benefit on Oracle ASM based mirroring is that the database instance is aware of the mirroring. For many types of logical corruptions such as a bad checksum or incorrect System Change Number (SCN), the database instance proceeds through the mirror side looking for valid content and proceeds without errors. If the process in the database that encountered the read can obtain the appropriate locks to ensure data consistency, it writes the correct data to all mirror sides.
When encountering a write error, a database instance sends the Oracle ASM instance a disk offline message.
-
If database can successfully complete a write to at least one extent copy and receive acknowledgment of the offline disk from Oracle ASM, the write is considered successful.
-
If the write to all mirror side fails, database takes the appropriate actions in response to a write error such as taking the tablespace offline.
When the Oracle ASM instance receives a write error message from a database instance or when an Oracle ASM instance encounters a write error itself, the Oracle ASM instance attempts to take the disk offline. Oracle ASM consults the Partner Status Table (PST) to see whether any of the disk's partners are offline. If too many partners are offline, Oracle ASM forces the dismounting of the disk group. Otherwise, Oracle ASM takes the disk offline.
The ASMCMD remap
command was introduced to address situations where a range of bad sectors exists on a disk and must be corrected before Oracle ASM or database I/O. For information about the remap
command, see "remap".
Oracle ASM Fast Mirror Resync
Restoring the redundancy of an Oracle ASM disk group after a transient disk path failure can be time consuming. This is especially true if the recovery process requires rebuilding an entire Oracle ASM disk group. Oracle ASM fast mirror resync significantly reduces the time to resynchronize a failed disk in such situations. When you replace the failed disk, Oracle ASM can quickly resynchronize the Oracle ASM disk extents.
Note:
To use this feature, the disk group compatibility attributes must be set to 11.1
or higher.
Any problems that make a failure group temporarily unavailable are considered transient failures that can be recovered by the Oracle ASM fast mirror resync feature. For example, transient failures can be caused by disk path malfunctions, such as cable failures, host bus adapter failures, controller failures, or disk power supply interruptions.
Oracle ASM fast resync keeps track of pending changes to extents on an offline disk during an outage. The extents are resynced when the disk is brought back online.
By default, Oracle ASM drops a disk in 12 hours after it is taken offline. You can set the DISK_REPAIR_TIME
disk group attribute to delay the drop operation by specifying a time interval to repair the disk and bring it back online. The time can be specified in units of minutes (m
or M
) or hours (h
or H
). If you omit the unit, then the default unit is hours. The DISK_REPAIR_TIME
disk group attribute can only be set with the ALTER
DISKGROUP
SQL statement and is only applicable to normal and high redundancy disk groups.
If the attribute is not set explicitly, then the default value (12h
) applies to disks that have been set to OFFLINE
mode without an explicit DROP
AFTER
clause. Disks taken offline due to I/O errors do not have a DROP
AFTER
clause.
The default DISK_REPAIR_TIME
attribute value is an estimate that should be adequate for most environments. However, ensure that the attribute value is set to the amount of time that you think is necessary in your environment to fix any transient disk error, and during which you are able to tolerate reduced data redundancy.
The elapsed time (since the disk was set to OFFLINE
mode) is incremented only when the disk group containing the offline disks is mounted. The REPAIR_TIMER
column of V$ASM_DISK
shows the amount of time left (in seconds) before an offline disk is dropped. After the specified time has elapsed, Oracle ASM drops the disk. You can override this attribute with the ALTER
DISKGROUP
OFFLINE
DISK
statement and the DROP
AFTER
clause.
Note:
If a disk is offlined by Oracle ASM because of an I/O (write) error or is explicitly offlined using the ALTER
DISKGROUP
... OFFLINE
statement without the DROP
AFTER
clause, then the value specified for the DISK_REPAIR_TIME
attribute for the disk group is used.
Altering the DISK_REPAIR_TIME
attribute has no effect on offline disks. The new value is used for any disks that go offline after the attribute is updated. You can confirm this behavior by viewing the Oracle ASM alert log.
If an offline disk is taken offline for a second time, then the elapsed time is reset and restarted. If another time is specified with the DROP
AFTER
clause for this disk, the first value is overridden and the new value applies. A disk that is in OFFLINE
mode cannot be dropped with an ALTER
DISKGROUP
DROP
DISK
statement; an error is returned if attempted. If for some reason the disk must be dropped (such as the disk cannot be repaired) before the repair time has expired, a disk can be dropped immediately by issuing a second OFFLINE
statement with a DROP
AFTER
clause specifying 0h
or 0m
.
You can use ALTER
DISKGROUP
to set the DISK_REPAIR_TIME
attribute to a specified hour or minute value, such as 6 hours or 360 minutes. For example:
ALTER DISKGROUP data SET ATTRIBUTE 'disk_repair_time' = '6h' ALTER DISKGROUP data SET ATTRIBUTE 'disk_repair_time' = '360m'
After you repair the disk, run the SQL statement ALTER
DISKGROUP
ONLINE
DISK
. This statement brings a repaired disk group back online to enable writes so that no new writes are missed. This statement also starts a procedure to copy of all of the extents that are marked as stale on their redundant copies.
If a disk goes offline when the Oracle ASM instance is in rolling upgrade mode, the disk remains offline until the rolling upgrade has ended and the timer for dropping the disk is stopped until the Oracle ASM cluster is out of rolling upgrade mode. Examples of taking disks offline and bringing them online follow.
The following example takes disk DATA_001
offline and drops it after fifteen minutes.
ALTER DISKGROUP data OFFLINE DISK DATA_001 DROP AFTER 15m;
The next example takes the disk DATA_001
offline and drops it after the time period designated by DISK_REPAIR_TIME
elapses:
ALTER DISKGROUP data OFFLINE DISK DATA_001;
This example takes all of the disks in failure group FG2
offline and drops them after the time period designated by DISK_REPAIR_TIME
elapses. If you used a DROP
AFTER
clause, then the disks would be dropped after the specified time:
ALTER DISKGROUP data OFFLINE DISKS IN FAILGROUP FG2;
The next example brings all of the disks in failure group FG2
online:
ALTER DISKGROUP data ONLINE DISKS IN FAILGROUP FG2;
This example brings only disk DATA_001
online:
ALTER DISKGROUP data ONLINE DISK DATA_001;
This example brings all of the disks in disk group DATA
online:
ALTER DISKGROUP data ONLINE ALL;
Querying the V$ASM_OPERATION
view while you run ALTER
DISKGROUP
ONLINE
statements displays the name and state of the current operation that you are performing. For example, the following SQL query shows values in the PASS
column during an online operation.
SQL> SELECT GROUP_NUMBER, PASS, STATE FROM V$ASM_OPERATION; GROUP_NUMBER PASS STAT ------------ --------- ---- 1 RESYNC RUN 1 REBALANCE WAIT 1 COMPACT WAIT
An offline operation does not generate a display in a V$ASM_OPERATION
view query.
The resync checkpoint maintains the position in the resync process in the event of a failure. If the resync process or an instance fails, then the next resync operation continues from the position where the previous process left off, rather than starting from the beginning or the resync process.
You can set the FAILGROUP_REPAIR_TIME
and CONTENT.TYPE
disk group attributes. The FAILGROUP_REPAIR_TIME
disk group attribute specifies a default repair time for the failure groups in the disk group. The CONTENT.TYPE
disk group attribute specifies the type of data expected to be stored in a disk group. You can set these attributes with ASMCA, ASMCMD mkdg
, or SQL CREATE
and ALTER
DISKGROUP
statements.
The ASMCMD lsop
command shows the resync time estimate. There are separate rows in the V$ASM_OPERATION
table for different phases of rebalance: disk resync, rebalance, and data compaction.
The ASMCMD online
command has a power
option to specify the power for the online operation. The SQL ALTER
DISKGROUP
REPLACE
DISK
statement also has the power option.
The ASMCMD chdg
command provides the replace
option in addition to the add
and drop
tags. The ASMCMD mkdg
command has an additional time parameter (-t
) to specify the time to offline a failure group.
See Also:
-
Disk Group Compatibility for information about disk group compatibility
-
Managing Disk Group Attributes for information about disk group attributes
-
Oracle Grid Infrastructure Installation and Upgrade Guide and Upgrading and Patching Oracle ASM for information about upgrading Oracle ASM
-
Oracle Database SQL Language Reference for information about
ALTER
DISKGROUP
-
Oracle Database SQL Language Reference for information about
CREATE
DISKGROUP
Preferred Read Failure Groups
When you configure Oracle ASM failure groups, it might be more efficient for a node to read from an extent that is closest to the node, even if that extent is a secondary extent. In other words, you can configure Oracle ASM to read from a secondary extent if that extent is closer to the node instead of Oracle ASM reading from the primary copy which might be farther from the node. Using the preferred read failure groups feature is most useful in extended clusters.
To use this feature, Oracle recommends that you configure at least one mirrored extent copy from a disk that is local to a node in an extended cluster. However, a failure group that is preferred for one instance might be remote to another instance in the same Oracle RAC database. The parameter setting for preferred read failure groups is instance specific.
Note:
In an Oracle extended cluster, which contains nodes that span multiple physically separated sites, the PREFERRED_READ.ENABLED
disk group attribute controls whether preferred read functionality is enabled for a disk group. If preferred read functionality is enabled, then this functionality enables an instance to determine and read from disks at the same site as itself, which can improve performance. Whether or not PREFERRED_READ.ENABLED
has been enabled, preferred read can be set at the failure group level on an Oracle ASM instance or a client instance in a cluster with the ASM_PREFERRED_READ_FAILURE_GROUPS
initialization parameter, which is available for backward compatibility.
Configuring and Administering Preferred Read Failure Groups
To configure this feature, set the ASM_PREFERRED_READ_FAILURE_GROUPS
initialization parameter to specify a list of failure group names as preferred read disks.
Set the parameter where diskgroup
is the name of the disk group and failuregroup
is the name of the failure group, separating these variables with a period. Oracle ASM ignores the name of a failure group that you use in this parameter setting if the failure group does not exist in the named disk group. You can append multiple values using commas as a separator as follows:
ASM_PREFERRED_READ_FAILURE_GROUPS
=
diskgroup
.failuregroup
,...
In an extended cluster, the failure groups that you specify with settings for the ASM_PREFERRED_READ_FAILURE_GROUPS
parameter should only contain disks that are local to the instance. For normal redundancy disk groups, there should be only one failure group on each site of the extended cluster.
If there are multiple mirrored copies and you have set a value for the ASM_PREFERRED_READ_FAILURE_GROUPS
parameter, then Oracle ASM first reads the copy that resides on a preferred read disk. If that read fails, then Oracle ASM attempts to read from the next mirrored copy that might not be on a preferred read disk.
Having multiple failure groups on one site can cause the loss of access to the disk group by the other sites if the site containing multiple failure groups fails. In addition, by having multiple failure groups on a site, an extent might not be mirrored to another site. This can diminish the read performance of the failure group on the other site.
For example, for a normal redundancy disk group, if a site contains two failure groups of a disk group, then Oracle ASM might put both mirror copies of an extent on the same site. In this configuration, Oracle ASM cannot protect against data loss from a site failure.
You should configure at most two failure groups on a site for a high redundancy disk group. If there are three sites in an extended cluster, for the same reason previously mentioned, then you should only create one failure group.
For a two-site extended cluster, a normal redundancy disk group only has two failure groups. In this case, you can only specify one failure group as a preferred read failure group for each instance.
You can use views to identify preferred read failure groups, such as the V$ASM_DISK
view that shows whether a disk is a preferred read disk by the value in the PREFERRED_READ
column. You can also use V$ASM_DISK
to verify whether local disks in an extended cluster are preferred read disks. Use the Oracle ASM disk I/O statistics to verify that read operations are using the preferred read disks that you configured.
If a disk group is not optimally configured for an extended cluster, then Oracle ASM records warning messages in the alert logs. To identify specific performance issues with Oracle ASM preferred read failure groups, use the V$ASM_DISK_IOSTAT
view. This view displays disk I/O statistics for each Oracle ASM client. You can also query the V$ASM_DISK_IOSTAT
view on a database instance. However, this query only shows the I/O statistics for the database instance. In general, optimal preferred read extended cluster configurations balance performance with disk group availability.
Both the Oracle ASM clients and Oracle ASM require Oracle Database 11g Release 1 (11.1) or higher to use preferred read failure groups.
Note:
If you do not specify failure groups for a disk group, each disk in the disk group belongs to its own failure group. Oracle does not recommend that you configure multiple preferred read failure groups in a disk group for an Oracle ASM instance. For any given instance, if you specify multiple failure groups in the same disk group as preferred read, a warning message is written to the alert log.
See Also:
-
PREFERRED_READ.ENABLED for information about the
PREFERRED_READ.ENABLED
disk group attribute -
ASM_PREFERRED_READ_FAILURE_GROUPS for more information about the
ASM_PREFERRED_READ_FAILURE_GROUPS
initialization parameter -
Oracle Real Application Clusters Administration and Deployment Guide for information about configuring preferred read disks in extended clusters
-
Oracle Database Reference for details about the
V$ASM
* dynamic performance views