C H A P T E R  3

Point-in-Time Copy Software

This chapter discusses Point-in-Time Copy software troubleshooting issues.

The following topics are included:


Common User Errors

This section describes some common errors that you may encounter when using Point-in-Time Copy software.

Safeguarding the VTOC

For information on how to safeguard the volume table of contents (VTOC), refer to Safeguarding the Solaris VTOC.

Accessibility Issues

The most common class of user errors when using the Point-in-Time Copy software is accessibility issues in the specification of the Master, Shadow, Bitmap, and optional overflow volumes, which are configured using the iiadm utility. The best means to resolve these types of errors is to use standard Solaris utilities to resolve these issues; specifically, format(1M), prtvtoc(1M), and dd(1M).

A typical iiadm enable command using Solaris RAW devices is as follows:


iiadm -e ind /dev/rdsk/c0t1d0s0 /dev/rdsk/c0t2d0s0 \/dev/rdsk/c0t3d0s0

A failure of this command may be due to incorrect device specifications, incorrect partition sizing, or a failure to access the devices from this Solaris node. Resolving issues with the following ten commands should be the first step towards resolving accessibility problems.


# format /dev/rdsk/c0t1d0s0
# format /dev/rdsk/c0t2d0s0
# format /dev/rdsk/c0t3d0s0
# prtvtoc /dev/rdsk/c0t1d0s0
# prtvtoc /dev/rdsk/c0t2d0s0
# prtvtoc /dev/rdsk/c0t3d0s0
# dd if=/dev/rdsk/c0t1d0s0 of=/dev/null count=1
# dd if=/dev/rdsk/c0t2d0s0 of=/dev/null count=1
# dd if=/dev/rdsk/c0t3d0s0 of=/dev/null count=1
# dsbitmap -p /dev/rdsk/c0t1d0s0

Functionality Issues

The next class of user errors when using the Point-in-Time Copy software is perceived functionality issues. The functionality of the Point-in-Time Copy software is to instantly copy all the data from the Master volume to the Shadow volume in a fraction of a second. Note the following two commands:


# iiadm -e ind /dev/rdsk/c0t1d0s0 /dev/rdsk/c0t2d0s0 \/dev/rdsk/c0t3d0s0

 


# dd if=/dev/rdsk/c0t1d0s0 of=/dev/rdsk/c0t2d0s0

These commands are essentially equivalent for the initial configuration of a Point-in-Time Copy set, except that the second command can take hours or days to complete, instead of a fraction of second. In addition, neither the master nor the shadow volume can be used during the entire duration of the second command, while they both can be used in the first. Therefore if functionality issues are not what they seem, use the second command with very small volumes to assure the functionality that is desired works in the configuration needed.

Data Integrity Issues

After an enable, copy, or update command, the contents of the master volume are instantly available on the shadow volume. In all cases, there is no need to wait for a background copy to complete (iiadm -w <shadow-volume>) except when next using Export (iiadm -E) or disable (iiadm -d). What this means is that the "backup window" is no longer than the time it takes to quiesce the volume, take a Point-in-Time Copy, and resume the volume. At this point, the master volume can be used instantly as it was just before the Point-in-Time Copy was taken, and the shadow volume can be used instantly for whatever purpose it was created.

During an enable, copy, or update command, it is highly recommended that the master volume be quiesced and all cached data blocks flushed to disk so that a Point-in-Time Copy that is taken does not capture an I/O operation in progress. Capturing an I/O operation in progress may result in shadow volume data that appears inconsistent, meaning that utilities like fsck(1M), database recovery tools, or similar software may have to make indeterminate decisions about the validity of an incomplete I/O operation. The means by which the Point-in-Time Copy software instantly makes a copy of the master volume to the shadow volume provides the same I/O consistency issues as a Solaris node "panicking" while I/O is in progress.


Configuration

This section discusses configuration issues.

Set Status

For a brief status of Point-in-Time sets on the system, use iiadm -e dep:


# iiadm -e dep /dev/rdsk/c7t0d0s6 /dev/rdsk/c7t1d0s6 \/dev/rdsk/c7t2d0s6

For a detailed status, use iiadm -i:


# iiadm -i
/dev/rdsk/c7t0d0s6: (master volume)
/dev/rdsk/c7t1d0s6: (shadow volume)
/dev/rdsk/c7t2d0s6: (bitmap volume)
Dependent copy
Latest modified time: Thu Nov  3 13:18:44 2005
Volume size: 21470084458
Shadow chunks total: 33547006 Shadow chunks used: 0
Percent of bitmap set: 2
        (bitmap dirty)

Volume Configuration

This section discusses volume configuration.

Raw Partition

The following command creates a Point-in-Time Copy set consisting of raw partitions, where the master is /dev/rdsk/c7t0d0s6, the shadow is /dev/rdsk/c7t1d0s6, and the bitmap is /dev/rdsk/c7t2d0s6.


# iiadm -e ind /dev/rdsk/c7t0d0s6 /dev/rdsk/c7t1d0s6 \/dev/rdsk/c7t2d0s6

Since this is an independent (ind) set, the shadow volume must be the same size or greater then the master volume. The bitmap volume must be sized according to the following command:


# dsbitmap -p /dev/rdsk/c7t0d0s6

Solaris Volume Manager

The following command creates a Point-in-Time Copy set consisting of Solaris Volume Manager volumes, where the master is /dev/md/rdsk/d1, the shadow is /dev/md/rdsk/d2, and the bitmap is /dev/md/rdsk/d3.


# iiadm -e dep /dev/md/rdsk/d1 /dev/md/rdsk/d2 /dev/md/rdsk/d3

Since this is a dependent (dep) set, the shadow volume can be either the same size or greater than the master volume, or, if smaller, the set becomes a compact dependent shadow set.

The bitmap volume must be sized according to the following command, making sure to select the correct sizing for either "Full size dependent shadow" or "Compact dependent shadow".


# dsbitmap -p /dev/md/rdsk/d1

Veritas Volume Manager

The following command creates a Point-in-Time Copy set consisting of VxVM volumes, where the master is /dev/vx/rdsk/ii-dg/d21, the shadow is /dev/vx/rdsk/ii-dg/d22, and the bitmap is /dev/vx/rdsk/ii-dg/d23.


# iiadm -e dep /dev/vx/rdsk/ii-dg/d21 /dev/vx/rdsk/ii-dg/d22 /dev/vx/rdsk/ii-dg/d23


Export/Import/Join

This section provides basic information on using Export/Import/Join within a Sun Cluster. For more detailed information, refer to the "Best Practice for using Export, Import, Join in a Sun Cluster OE 3.1" (10/3).

Resource Group

Make sure that the device groups are in a Sun Cluster SUNW.HAStoragePlus resource within a Sun Cluster resource group. See the best practice guide for detailed steps.

Exporting

When using Export/Import Join within a Sun Cluster, the shadow volume must be on a different global device or volume manager controlled device group than its associated master and bitmap volumes. Doing so will allow the shadow volume's device group to be switched between various nodes in a Sun Cluster and to be used as an exportable shadow volume.



Note - Sun Cluster DID devices are not supported as a Master, Shadow, or Bitmap volume due to disk data fencing functionality when Sun Cluster failure events are active. Sun Cluster Global devices, which are symmetrical in name to the DID devices, are supported.



To create such a set, with the master and bitmap in one device group (oracle, in this example), and the shadow in another (backup, in this example), use the -ne flags to iiadm:


# iiadm -ne ind /dev/md/oracle/rdsk/d1 /dev/md/backup/rdsk/d1 \
/dev/md/oracle/rdsk/d2

Make sure that the Point-in-Time Copy set is fully independent before exporting the shadow volume. This is confirmed by wait (iiadm -w) returning on the shadow volume:


# iiadm -w /dev/md/backup/rdsk/d1

Importing

If you would like to import the shadow on another Sun Cluster node, the second bitmap used for the import operation must be in the same global device or volume manager controlled device group as the exported shadow.



Note - The exportable shadow MUST be imported with the -C local tag, so that the system can differentiate between the highly available Point-in-Time Copy set and the locally accessible exportable shadow, which have the exact same name.




# iiadm -C local -I /dev/md/backup/rdsk/d1 /dev/md/backup/rdsk/d2

Export/Join With no Import

It is possible to perform an Export/Join sequence with no Import step. To perform a join operation, there is still the requirement for a secondary bitmap volume, but since this secondary bitmap volume was NOT used during a recent Import operation, it contains stale data or uninitialized data. Prior to performing a join operation, you must copy (using the Solaris dd utility) the current bitmap volume over the contents of the second bitmap volume, so that the second bitmap volume's data is in a known state. The failure to perform this manual initialization step may result in the join operation failing, or when state data is used, it may cause an inconsistency between what is actually on the shadow volume and the current state as recorded in the bitmap.


Server Performace Diagnosis

This section discusses server performance diagnosis issues.

Variables and Configuration Files

If the Sun StorageTek Availability Suite software is used with a file system, tuning the number of SV threads might produce better performance.

When a file system flushes its cache, it generates many parallel write operations. The SV's default setting of 32 threads could produce a bottleneck. You can increase the number of SV threads. The maximum number of threads allowed is 1024.



Note - Each thread consumes 32k of memory.



The sv_threads value is in the /usr/kernel/drv/sv.conf file. Because the file is read when a module loads, changes to the sv_threads value do not take effect until you reboot the system.

dsstat Utility

The dsstat(1M) utility is useful for observing real-time read and write performance through volumes under Point-in-Time control. The dsstat tool is similar in usage to iostat, taking as arguments an interval length and iteration count.


# dsstat -m ii 1 2                                              
name              t  s    pct role    kps   tps  svt
dsk/avsuite/vol0  I  -   0.00  mst      0     0    0
dsk/avsuite/vol1               shd      0     0    0
dsk/avsuite/vol2               bmp      0     0    0
                               ovr <<not attached>>
dsk/avsuite/vol0  I  -   0.00  mst      0     0    0
dsk/avsuite/vol1               shd      0     0    0
dsk/avsuite/vol2               bmp      0     0    0
                               ovr <<not attached>>

iostat Utility

The iostat(1M) utility may also be used to monitor Point-in-Time Copy performance. Besides being able to access performance data on the underlying storage volumes under Point-in-Time Copy control, Point-in-Time Copy sets also appear in iostat under the names iib[n], iim[n], and iis[n] for a bitmap, master, and shadow volume, respectively (where n is the set id).


# iostat -xncz
     cpu
 us sy wt id
  4  1  0 95
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.3   0   0 iib0
    0.0    0.0    0.2    0.0  0.0  0.0    0.0   15.3   0   0 iim0
    0.0    0.0    0.0    0.2  0.0  0.0    0.0   12.6   0   0 iis0
    0.1    0.1    1.2    0.5  0.0  0.0    0.0   20.3   0   0 c0t0d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.3   0   0 gsdbc
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.3   0   0 sdbc0
    0.0    0.0    1.0    0.0  0.0  0.0    0.0    0.5   0   0 c4t50020F23000009DAd0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.3   0   0 c4t50020F23000036EAd0
    0.1    0.0    0.1    0.0  0.0  0.0    0.0    0.5   0   0 c4t50020F230000024Ed0
    0.1    0.0    0.1    0.0  0.0  0.0    0.0    0.4   0   0 c4t50020F23000001C7d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.1   0   0 c4t50020F23000003DAd0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.1   0   0 c4t50020F23000003EBd0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.1   0   0 c4t50020F2300000A24d0

ptree Command

The ptree(1) command shows the parent proc esses for a given process id. It is particularly useful for troubleshooting hung processes, or processes that were invoked through Sun Cluster scripts.

For example, if a ps command shows that the svboot process is running:


# ps -ef | grep svboot
root  9829  9824   0 09:56:21 ?           0:00 /usr/sbin/svboot -C avsuite -s

run ptree on the process id for svboot to see its parents:


359   /usr/cluster/lib/sc/clexecd
  360   /usr/cluster/lib/sc/clexecd
    12812 sh -c /usr/cluster/lib/sc/run_reserve -C SUNWvxvm -C SUNWvxvm -s avsu
      12813 /usr/bin/ksh /usr/cluster/lib/sc/run_reserve -C SUNWvxvm -C SUNWvxv
        12815 /usr/bin/ksh /usr/opt/SUNWesm/cluster/sbin/reconfig stop avsuite
          12818 /usr/bin/ksh /usr/cluster/lib/dscfg/stop/10sv stop avsuite
            12826 /usr/sbin/svboot -C avsuite -s

Configuration - Files

The file /var/adm/ds.log contains a record of Availability Suite configuration and control activity, including which Point-in-Time Copy sets have had the following operations by the iiadm and iiboot utilities:


InfoDoc Summary

Below is a tabular summary of the SunSolve InfoDocs written to address common customer issues for Point-in-Time Copy software. If you believe you are experiencing one of these issues, contact your Sun Service Representative for a swift resolution.


TABLE 3-1 InfoDocs Addressing Point-in-Time Copy Software Issues

InfoDoc ID

Issue

71559

Cannot remove SVM, Veritas volumes, or DR LUNs under Availability Suite Software

77167

Booting either host causes entire sync in Remote Mirror or Point-in-Time Copy

77901

Cannot Enable Point-in-Time Copy devices within Sun Cluster

78723

II: Volume in use