C H A P T E R  4

Using the Remote Mirror Software

This chapter includes examples of how to use the Remote Mirror software command sndradm. The topics in this chapter include:

TABLE 4-1 lists the names used in the examples in this chapter:


TABLE 4-1 Example Names and Devices Used in This Chapter

Primary host name

rmshost1

Primary volume

/dev/rdsk/c0t117d0s3

Primary bitmap

/dev/vx/rdsk/bmap/bm1

Secondary host name

rmshost2

Secondary volume

/dev/rdsk/c0t117d0s5

Secondary bitmap

/dev/vx/rdsk/bmap/bm2

Set name (assigned by the software)

rmshost2:/dev/rdsk/c0t117d0s5


Depending on the example, either site can be the primary or secondary host of the remote copy operation. As shown in TABLE 5-4, you must perform all synchronization operations from the primary host session.

To monitor the operation of the Remote Mirror software, use the /usr/sbin/dsstat command described in Appendix A.


Getting Started



Note - Use the same disk management method (software volume management or raw disk) with the Remote Mirror software and the Point-in-Time Copy software on the primary site and the secondary site volumes. The Sun StorageTek Availability Suite software replicates data at the block level, and the block counts for a given size are different for a disk slice and a volume. Therefore, a Remote Mirror replication or reverse sync or a Point-in-Time full independent copy or reverse copy, might fail if the target size is smaller than the source size.



The following sections describe the initial steps for using the Remote Mirror software:

Enabling the Volume Sets

The first step to using the Remote Mirror software is to enable the software on volume sets. Make sure you perform this step on both the primary and secondary hosts. A common user error is enabling volume sets on only one host.



caution icon

Caution - When creating volume sets, do not create secondary or bitmap volumes using partitions that include cylinder 0. Data loss might occur. See VTOC Information.



When enabling Remote Mirror volume sets, it is assumed that the primary volume and the secondary volume contain data that is different. Therefore, when enabling the Remote Mirror set, the sndradm -e command is used, causing the enable to set all the bits in the bitmap, indicating that the data on the volumes are different.

If, when enabling Remote Mirror volume sets, it is known that the primary volume and the secondary volume contain data that is 100% identical, when enabling the Remote Mirror set, the sndradm -E command is used, causing the enable to clear all the bits in the bitmap, indicating that the data on the volumes are the same.

In the following example, the primary and secondary volume are assumed to be different and the set will be enabled to replicate in asynchronous mode.


procedure icon  To Enable the Volume Sets

1. Log in to the primary host rmshost1 as the superuser.

2. Enable the volume sets:


rmshost1# sndradm -e rmshost1 /dev/rdsk/c0t117d0s3 /dev/vx/rdsk/bmap/bm1 \rmshost2 /dev/rdsk/c0t117d0s5 /dev/vx/rdsk/bmap/bm2 ip async

3. Log in to the secondary host rmshost2 as the superuser.

4. Enable the volume sets:


rmshost2# sndradm -e rmshost1 /dev/rdsk/c0t117d0s3 /dev/vx/rdsk/bmap/bm1 \rmshost2 /dev/rdsk/c0t117d0s5 /dev/vx/rdsk/bmap/bm2 ip async

The following events occur:

Establishing Volume Copies for the First Time

The next step is to perform a full forward synchronization to copy the primary volume contents to the secondary volume. The Remote Mirror software performs this initial primary-volume-to-secondary-volume copy while also forwarding any new primary volume updates to the secondary volume.

The volumes can be made identical using methods other than a full synchronization. When network latencies justify it, you can perform the initial synchronization of a volume set by backing up a source or primary volume on magnetic tape on one site, then restoring the volume from the tape on the other site. During the period between when the backup is finished and the restore is started, place the source or primary volume in logging mode. Make sure that the backup copy is a physical copy (for example, by using the dd(1M) command) and not a logical copy (for example, one made using the tar(1M) or cpio(1M) commands). The copies must have identical blocks, not just identical files. In this case, use the sndradm -E command instead of sndradm -e to enable the volume sets.


procedure icon  To Synchronize the Volumes During Update

1. Log in to the primary host rmshost1 as superuser.

2. Unmount the secondary volume. You can keep the primary volume mounted.

3. Synchronize the volumes:


rmshost1# sndradm -m rmshost2:/dev/rdsk/c0t117d0s5

4. Check the synchronization progress:


rmshost1# dsstat -m sndr

After the synchronization is complete, the Remote Mirror software continues to replicate any primary volume changes to the secondary volume. Keep the secondary volume unmounted during replication or until you are ready to permit your application to write to it. When you are ready to permit write operations to the secondary volume, place the volume sets into logging mode and mount the volume. The software continues tracking changes through the bitmap until you are ready to update or resynchronize the volumes.

Updating the Secondary Volume

This section describes the commands to use when you are ready to update the secondary volume to resynchronize volumes.

Optionally, you can also use the Sun StorageTek Availability Suite Point-in-Time Copy software to ensure data consistency at the primary and secondary volumes. Using this software helps guarantee a known good copy of your data in case of a network link failure during the synchronization. See the Sun StorageTek Availability Suite 4.0 Point-in-Time Copy Software Administration Guide for details about the iiadm command.

During resynchronization, write ordering is not preserved and the secondary volumes are consequently inconsistent. Creating PIT copies on the secondary volume prior to initiating a resynchronization is recommended practice to guarantee there is a consistent data set. See Autosynchronization for the options to use to configure the auto synchronization daemon to perform these PITs automatically prior to resynchronization.



Note - You must place the related Remote Mirror volume set in logging mode (only if a Remote Mirror volume is a target of Point-in-Time Copy update/copy) for the point-in-time copy software to successfully perform an enable, copy, update, or reset operation on a Remote Mirror volume. If the volume set is not in logging mode, the point-in-time copy operation fails and the Remote Mirror software reports that the operation is denied




procedure icon  To Resynchronize Primary and Secondary Volumes

1. Log in to the primary host rmshost1 as the superuser.

2. Quiesce any applications writing to the primary volume.



Note - You are not necessarily required to quiesce your applications but doing so helps guarantee a consistent copy of your data. Quiescing applications also helps guarantee a consistent copy for the Point-in-Time Copy software. If you choose not to quiesce your applications and not to use the Point-in-Time Copy software, the Remote Mirror software still forwards any data updates to the secondary volume during replication.



3. (Optional) Take a Point-in-Time Copy snapshot of the primary volume. Ensure that the primary volume is in logging mode (sndradm -l) and then use the iiadm command.

4. (Optional) Take a Point-in-Time Copy snapshot of the secondary volume. Ensure that the secondary volume is in logging mode and then use the iiadm command.

5. Copy only the changed data from the primary volume to the secondary volume:


rmshost1# sndradm -u rmshost2:/dev/rdsk/c0t117d0s5

6. Check the synchronization progress:


rmshost1# dsstat -m sndr

When the update synchronization is completed, the secondary volume is a block-for-block copy of the primary volume and the bitmaps are cleared to 0. See If a Network Link Fails.


If a Network Link Fails

The Remote Mirror software uses a periodic signal to monitor the health of primary and secondary systems. If the software cannot sense a health monitor signal, it assumes an interruption in the Remote Mirror software service.

The Remote Mirror software then places all volume sets at the primary site into logging mode. During logging mode, the software updates only the primary volume bitmap. (The software assumes that the secondary volume is not mounted and not being written to.) See Logging and Stopping Replication and Starting Logging.



Note - The secondary site does not automatically switch to logging mode if a break in the network occurs or if the primary site is down. This is to protect the data at the secondary site from unwanted writes. An administrator must cause the software to actually fail over to the secondary site by issuing the sndradm -l command on the secondary.



You can introduce interruptions intentionally to exercise remote failure strategies, for example, during the disaster recovery rehearsals described in Rehearsing Disaster Recovery.

When Not To Resynchronize Volumes

Resynchronization is discouraged if the interruption might be the warning of a larger, rolling disaster. Maintain the secondary site in a dated but consistent state rather than risk a disastrous interruption that leaves the secondary site inconsistent and difficult to recover from. The autosynchronization option is disabled by default for this reason. See Choosing Automatic or Manual Resynchronization.

Autosynchronization

The autosynchronization feature is designed to synchronize the primary and secondary volumes after a network link failure has been restored. When autosynchronization is enabled, the operation occurs only when replication occurs. For example, when you enable autosynchronization for a set, the software attempts to synchronize the primary and secondary volumes only when replication is performed. When you put the set into logging mode, the software does not synchronize the primary and secondary volumes. However, putting the set into logging mode does not disable autosynchronization. When a new synchronization request is issued, for example using the sndradm -u command, the autosynchronization feature becomes active again.

To enable or disable autosynchronization, use the sndradm -a command described in Enabling or Disabling Autosynchronization. See also Choosing Automatic or Manual Resynchronization and Autosynchronization.

Resynchronizing Volumes Manually



caution icon

Caution - While resynchronization is occurring, the secondary volume data is temporarily inconsistent and cannot be relied on for recovery. Consistency is restored when the resynchronization finishes executing. To ensure data integrity, use the Point-in-Time Copy software regularly to create a snapshot of data at both sites.



Typically, interruptions in the Remote Mirror software services are infrequent.

If the secondary volume state is unknown because of system or disk failure, make full volume copies to reestablish matching Remote Mirror software volume sets. In this case, use the sndradm -m command to fully update the secondary volume set.

Perform the procedures in To Synchronize the Volumes During Update.


Rehearsing Disaster Recovery

The Remote Mirror software enables you to perform disaster rehearsals, encouraging verification of your disaster plans. Perform rehearsals regularly and refine them whenever a significant change is made to the primary or secondary host environments.

When rehearsing disasters or in an actual disk disaster or failure, keep the failed volumes under control of the Remote Mirror software. Do not disable the software. The Remote Mirror software marks the device as failed when the software is unable to read from or write to the device. In the case of a primary failure, for example, the Remote Mirror software continues to provide read and write services to the host application using the secondary volume at the remote site.


procedure icon  To Rehearse a Primary Volume or Site Failure

1. Simulate a primary volume or site disaster by using one of the following methods:

2. When the data is destaged, mount the secondary volume in read-write mode so your application can write to it.

3. Configure your application to read and write to the secondary volume.

The secondary bitmap volume tracks the volume changes.

4. Fix the "failure" at the primary volume by using one of the following methods:



Note - If the autosynchronization feature is enabled, the Remote Mirror software resynchronizes the primary volume from the secondary volume when the link is re-established. If you also install and configure the Point-in-Time Copy software, this software takes a snapshot copy of your secondary volume data before performing the reverse update synchronization. Consider whether this approach is appropriate for your disaster recovery plan.



You can now choose to resynchronize your volumes:

5. Perform an update by choosing one of the following methods:


procedure icon  To Rehearse a Secondary Volume or Site Failure

1. Simulate a secondary volume or site disaster by using one of the following methods:

2. Fix the failure at the primary volume by using one of the following methods:



Note - If the autosynchronization feature is enabled, the Remote Mirror software resynchronizes the secondary volume from the primary volume when the link is re-established. If you also install and configure the Point-in-Time Copy software, this software takes a snapshot copy of your secondary volume data before performing the reverse update synchronization. Consider whether this approach is appropriate for your disaster recovery plan.



You can now choose to resynchronize your volumes.

3. Perform the update by choosing one of the following methods:


Handling Primary Volume Failures



Note - Keep the failed volumes under control of the Remote Mirror software. Do not disable the software. The Remote Mirror software marks the device as failed when the software is unable to read to or write from the device. The Remote Mirror software continues to provide read and write services to the host application using the secondary volume at the remote site.



The Remote Mirror software provides continuous data access during primary volume failures. The Remote Mirror software high-availability features are a superset of RAID-1 and RAID-5 storage protection that can be optionally configured for the primary volumes. The Remote Mirror software's remote volume access features start only after the disk protection schemes on the primary system are unable to provide data access to the local devices.

In the linear and striped (RAID 0) cases, failure of a single disk storing the primary volume triggers the Remote Mirror software to transparently redirect disk reads and writes to the remote storage system.

If the primary logical volume is locally mirrored (RAID 1) across two physical disks on the same system, a single disk failure results in its local mirror disk handling all requests for cache staging on a read miss and cache destaging. The Remote Mirror software relies on the remote site secondary devices only if both local mirrors fail.

If the primary volume is RAID-5 protected, its contents are striped across several physical disks. The local system considers the primary volume inaccessible and yields to the Remote Mirror software's remote volume access only when two or more of the disks in the RAID 5 stripe fail.

Recovering From a Primary Site Disaster

The Remote Mirror software minimizes the effects of a disaster at the primary site by enabling you to keep the secondary storage images updated. Although the secondary Remote Mirror software cache contains the latest writes issued on the primary before the disaster, that data might not have been destaged to the secondary disks yet. After detecting an interruption in the Remote Mirror software service, the Remote Mirror software automatically destages the secondary Remote Mirror software cache to its corresponding secondary volumes.

After all the secondary volumes have been updated with the latest Remote Mirror software cache images, the secondary volumes can be accessed by the secondary hosts. The dsstat command shows information confirming that destaging is complete. Run application-level recovery procedures to ensure a well-known state at the secondary site. The workload can then be switched to the secondary hosts for continued business operation.

Until the extent of the primary failure is understood, keep the Remote Mirror software enabled at the secondary site to track disk areas that are being modified.

Restoring a Primary Site From the Secondary Site

If the primary host becomes inoperative and primary data on the primary disks is lost, update logs at the secondary systems have little value. You must flush the cache and perform a full reverse synchronization on the repaired or replaced primary host. In other words, volume-to-volume copies from the secondary to the primary are required for all Remote Mirror software-managed volumes. This reverse synchronization process ensures that only the latest data is deposited on the primary disks. See Rehearsing Disaster Recovery.


Disabling Remote Replication



caution icon

Caution - Disable remote replication onlywhen the primary and secondary volumes will no longer be associated.



Disabling the Remote Mirror software breaks the connection between primary and secondary volumes, discards any bitmap information, and removes the host and volume information from the Sun StorageTek configuration. After disabling the Remote Mirror software, enable and full synchronization (full volume copy) operations are necessary to re-establish the Remote Mirror software relationship and to ensure that the contents of each volume match. See Enabling and Disabling Volume Sets and Establishing Volume Copies for the First Time.


Swapping the Remote Mirror Hosts



caution icon

Caution - Before performing this procedure, ensure that I/O operations are not occurring to the volumes at the primary and secondary hosts. Data corruption will result if I/O operations continue.



In case of disaster recovery or link failure situations, you can also swap the Remote Mirror host roles to provide access to your critical data. That is, the primary host can become the secondary host and the secondary host can become the primary host. This alternate scheme enables you to recover the old primary host and, if you choose, switch back to the original roles.

The basic steps to swap host roles are as follows:

1. Quiesce the application accessing the primary volume. Unmount the volume if necessary.

2. Disable the Remote Mirror software on the primary site (Site-A). This step also discards the primary bitmap. A full copy is required when the set is enabled.

3. Disable the Remote Mirror software on the secondary site (Site-B).

4. Enable the Remote Mirror software on the new primary site (Site-B) with Site-B specified as the primary site.

5. Enable the Remote Mirror software on the new secondary site (Site-A) with Site-A specified as the secondary site.

6. At the new primary site (Site-B), synchronize the volumes from the primary to the secondary.

7. Perform any modifications or recovery procedures required by your application. For example, if you are using a database application, you might have to copy data and control files to the new secondary host after the synchronization.

8. Restart the application on the new primary site (Site-B). Mount the volumes if necessary.



Note - It may be useful to use a volume set file for ease of administration. See Setting Up a Volume Set File.



The rdc.cf Volume Set File

Following is an example of an rdc.cf volume set file. See also Setting Up a Volume Set File.


rmshost1 /dev/rdsk/c0t117d0s3 /dev/vx/rdsk/bmap/bm1 \rmshost2 /dev/rdsk/c0t117d0s5 /dev/vx/rdsk/bmap/bm2 ip sync

Your actual volume set file can be any name. The rdc.cf file name is used as an example here.


TABLE 4-2 Example Names and Devices Used in This Procedure

Primary host name (Site-A)

rmshost1

Primary volume

/dev/rdsk/c0t117d0s3

Primary bitmap

/dev/vx/rdsk/bmap/bm1

Secondary host (Site-B)

rmshost2

Secondary volume

/dev/rdsk/c0t117d0s5

Secondary bitmap

/dev/vx/rdsk/bmap/bm2

Transmission protocol

ip

Replication mode

sync

Set name (assigned by the software)

rmshost2:/dev/rdsk/c0t117d0s5



procedure icon To Disable the Software at Site-A



caution icon

Caution - Before performing this procedure, ensure that I/O operations are not occurring to the volumes at the primary and secondary hosts. Data corruption will result if I/O operations continue to occur. Before performing this procedure, quiesce the application writing to the Remote Mirror volumes and then unmount those volumes.



The following example assumes that the /rdc.cf volume set file is already created and the volumes specified in it are enabled.

1. At Site-A, disable the Remote Mirror software and discard the Remote Mirror scoreboard bitmap:


rmshost1# sndradm -dn -f /rdc.cf

2. Edit the rdc.cf file to swap the Site-A primary host information and Site-B secondary host information.

For example, in the entry shown in The rdc.cf Volume Set File, change rmshost1 to rmshost2 and rmshost2 to rmshost1.


rmshost2 /dev/rdsk/c0t117d0s3 /dev/vx/rdsk/bmap/bm1 \rmshost1 /dev/rdsk/c0t117d0s5 /dev/vx/rdsk/bmap/bm2 ip sync

3. If possible, unmount the Remote Mirror volumes:


rmshost1# umount mount-point


procedure icon  To Change the Site-B Secondary Host to the Primary Host

1. At Site-B, disable the Remote Mirror software and discard the Remote Mirror scoreboard bitmap:


rmshost2# sndradm -dn -f /rdc.cf

2. Edit the rdc.cf file to swap the Site-A primary host information and Site-B secondary host information.

For example, in the entry shown in The rdc.cf Volume Set File, change rmshost1 to rmshost2 and rmshost2 to rmshost1.

3. Enable the Remote Mirror software at both hosts:


rmshost1# sndradm -En -f /rdc.cf
rmshost2# sndradm -En -f /rdc.cf

Use the -E enable option to ensure that the bitmap contents are clear (0), indicating that a synchronization is not required.

4. If desired, at Site-A, perform a full synchronization from Site-B to Site-A.


rmshost1# sndradm -mn -f /rdc.cf

5. Perform any modifications or recovery procedures required by your application.