C H A P T E R  2

Configuring the Sun StorageTek Availability Suite Software


Note - This guide assumes that you have already installed the volume manager software and the Sun Cluster software on each node in your cluster.





caution icon

Caution - Do not install the Sun StorageTek Availability Suite software on a system running the initial release of the Sun Cluster 3.0 software.



The Sun StorageTek Availability Suite 4.0 Software Installation and Configuration Guide, listed in Related Documentation, describes how to install the Sun StorageTek Availability Suite software in a non-clustered environment. The installation steps to install this software in a Sun Cluster environment are generally the same as described in the installation guides. This chapter describes the differences when you install the software in a Sun Cluster environment.

The topics in this chapter include:


Setting the Location of the AVS Cluster-Specific Configuration Database

When installed in a Sun Cluster operating environment, AVS software requires a configuration database for information specific to Sun Cluster. This Sun Cluster configuration database is in addition to the local configuration database, which is still required (see the Sun StorageTek Availability Suite 4.0 Software Installation and Configuration Guide for more details).

A single Sun Cluster configuration location must be shared by all cluster nodes, and must be explicitly set on each node. Running dscfgadm with no arguments will prompt you to set the cluster configuration location, if it has not yet been set on that node. The location may also be changed later by running dscfgadm -s on all nodes of the cluster.

The cluster configuration database must be located on a raw slice of a did device. For help in finding an unused did device, the scdidadm -L command shows the local and shared disks by device ID. Refer to TABLE 2-1 for the requirements for this configuration location.



Note - Ensure that the slice does not contain disk label private areas (for example, slice 2 on a Solaris OS-formatted volume). The disk label region is contained in the first sectors of cylinder 0 of a disk. See VTOC Information.



 

.


TABLE 2-1 Configuration Location Requirements and Considerations

Item

Requirement or Consideration

Location

A raw device on a did service. For example: /dev/did/rdsk/d0s7.

Availability

  • The raw device must be accessible by all nodes of the cluster.
  • The location must be writable by the superuser user.
  • The location is available or persistent at system startup and reboots.
  • The slice used for the configuration database cannot be used by any other application (for example a file system or a database).

Disk space

The configuration location requires 5.5 Mbytes of disk space. If you specify a file for the configuration location during the installation, the file of the appropriate size is automatically created.

 

Note: If you specify a volume or a slice for the configuration location, only 5.5 Mbytes of the space is used, the remainder is unused.

Mirroring

Consider configuring RAID (such as mirrored partitions) for the location and ensure that you mirror the location to another disk in the array. The location cannot be stored on the same disk as the replicated volumes.



Backing Up Configuration Information

In addition to the local configuration information (see the Sun StorageTek Availability Suite 4.0 Software Installation and Configuration Guide), the cluster-specific configuration information should also be periodically backed up. You may wish to set up a cron(1M) job to periodically back up the Sun Cluster configuration database automatically. In addition, back up the configuration information whenever you change your configuration, for example, adding and deleting volumes.


procedure icon  To Back up Configuration Information

1. On each node of the Sun Cluster, back up the local configuration database (see the Sun StorageTek Availability Suite 4.0 Software Installation and Configuration Guide).


# cp /etc/dscfg_local /var/backups/dscfg_local

2. On one node of the Sun Cluster, perform the following two steps:

a. Back up the /etc/dscfg_cluster reference file, containing the path to the AVS cluster database.


# cp /etc/dscfg_cluster /var/backups/dscfg_cluster

b. Back up the contents of the cluster-specific configuration database to a file, using the dd command. The size of the database is 5.5 MB.


# dd if=/dev/did/rdsk/d3s4 of=/var/backups/dscfg_cluster_data bs=512k count=11

3. To determine the device you have specified for the cluster-specific configuration database, run a cat command on the /etc/dscfg_cluster file or run
dscfgadm -i.

a.


# cat /etc/dscfg_cluster 
/dev/did/rdsk/d3s4

b. or


# dscfgadm -i
SERVICE         STATE           ENABLED
nws_scm         online          true
nws_sv          online          true
nws_ii          online          true
nws_rdc         online          true
nws_rdcsyncd    online          true
 
Availability Suite Configuration:
Local configuration database: valid
cluster configuration database: valid
cluster configuration location: /dev/did/rdsk/d3s4


Editing the Bitmap Parameter Files

Bitmap volumes are used by the Remote Mirror and Point-in-Time Copy software to track differences between volumes and provide information for volume updates. The Sun StorageTek Availability Suite software documentation listed in Related Documentation describes the bitmap size and other requirements.

In a Sun Cluster environment, bitmap volumes must be part of the same disk device group or cluster resource group as the corresponding primary host or secondary hosts data volume.

The Remote Mirror and Point-in-Time Copy software include two configuration files that determine how bitmap volumes are written to and saved:



Note - The Sun StorageTek Availability Suite Remote Mirror and Point-in-Time Copy software do not support bitmap files. The software uses regular raw devices to store bitmaps. These raw devices must be located on a disk separate from the disk that contains your data.



Setting the Bitmap Operation Mode

A bitmap maintained on disk can persist across a system crash, depending on the setting of rdc_bitmap_mode in /usr/kernel/drv/rdc.conf. The default setting is 0. Set the bitmap mode to 1, as in the following example:

single-step bulletOpen the rdc.conf file and locate the following section. Edit the value for the bitmap mode, save the file, and close it.


# rdc_bitmap_mode
# - Sets the mode of the RDC bitmap operation, acceptable values are:
#   0 - autodetect bitmap mode depending on the state of SDBC (default).
#   1 - force bitmap writes for every write operation, so an update resync
#       can be performed after a crash or reboot.
#   2 - only write the bitmap on shutdown, so a full resync is
#       required after a crash, but an update resync is required after
#       a reboot.
#
rdc_bitmap_mode=1;

The /usr/kernel/drv/ii.conf File

The /usr/kernel/drv/ii.conf file contains one setting that sets the Point-in-Time Copy bitmap save mode:

A bitmap maintained on disk can persist across a system crash when this field is set to 1.


procedure icon  To Edit the ii.conf File

1. Open the /usr/kernel/drv/ii.conf file using a text editor such as vi(1).

2. In a Sun Cluster environment, set the bitmap mode to 1.

For example:


# bitmap volume storage strategy:
# 0 indicates kernel memory loaded from bitmap volume when shadow is resumed
#   and saved to bitmap volume when shadow is suspended.
# 1 indicates permanent SDBC storage, bitmap volume is updated directly as
#   bits are changed.
# 2 indicates that if FWC is present strategy 1 is used, otherwise strategy 0.
ii_bitmap=1;

3. Save and exit the file.

4. Disable and re-enable the data services as follows:


# dscfgadm -d
# dscfgadm -e

 


Supported Configurations for the Remote Mirror Software

Adding Host Names

This step ensures that the host names in the /etc/hosts file are read and known by machines running the Availability Suite software.


procedure icon  To Edit the /etc/hosts File

Place the names and IP addresses of all machines you plan to use with the Remote Mirror software in the /etc/hosts file. Make sure you include the logical host names and IP addresses of the logical hosts you plan to use with the Remote Mirror software in the /etc/hosts file.

single-step bullet Add the names and IP addresses of all machines you plan to use with the remote mirror software to the /etc/hosts file.

Edit this file on each machine where you are installing and running the remote mirror software.

Using Autosynchronization

Consider the following when using autosynchronization with Sun Cluster:

 

Rules for the Remote Mirror Software

For example, you cannot have a primary volume with a disk device group name of sndrdg and a primary bitmap volume with a disk device group name of sndrdg2 in the same remote mirror volume set.

To improve the failover affinity between a configured resource group name and the Remote Mirror device group configured within, it is suggested that the resource group name you specify consist of the disk device group name appended with -stor-rg. For example, if the group name is sndrdg, then the resource group name would be sndrdg-stor-rg.

Typically, the remote mirror primary host is part of one cluster configuration, while the replicating secondary host might or might not be part of a different cluster.

Three configurations for the Remote Mirror software are supported:

Remote Mirror Primary Host On a Cluster Node

In this configuration, the remote mirror primary host is the logical host you created in the remote mirror resource group for the remote mirror disk group using the scrgadm command; for example, see To configure Sun Cluster for HAStorage or HAStoragePlus.

If you have configured the remote mirror autosynchronization feature on the primary host, the Remote Mirror software starts an update resynchronization from the primary host for all affected remote mirror volume sets following a switchover or failover event, if the autosynchronization feature is enabled for those volume sets. This operation is performed after the resource group and network switchover operation is complete. See the sndradm man page and the Sun StorageTek Availability Suite 4.0 Remote Mirror Software Administration Guide for a description of the sndradm -a command to set the autosynchronization feature.

Remote Mirror Secondary Host On a Cluster Node

In this configuration, the remote mirror secondary host is the logical host you created in the remote mirror resource group for the remote mirror disk group using the scrgadm command; for example, see To configure Sun Cluster for HAStorage or HAStoragePlus.

Operations such as update resynchronizations occur and are issued from the primary host machine. Following a switchover or failover event, the Remote Mirror software attempts to start an update resynchronization for all affected remote mirror volume sets, if the autosynchronization feature is enabled for those volume sets. However, the remote mirror secondary host in a remote mirror volume set cannot initiate an update resynchronization.

This operation is performed after the resource group and network switchover operation is complete. In this case, the remote mirror secondary host switchover appears to be a short network outage to the remote mirror primary host.

If you have configured the remote mirror autosynchronization feature on the primary host, the sndrsyncd synchronization daemon attempts to resynchronize the volume sets if the system reboots or link failures occur. See the sndradm man page and the Sun StorageTek Availability Suite 4.0 Remote Mirror Software Administration Guide for a description of the sndradm -a command to set the autosynchronization feature.

If this feature is disabled (its default setting) and volume sets are logging but not replicating, perform the updates manually using the sndradm command.

Remote Mirror Primary and Secondary Hosts On a Cluster Node

Remote mirror replication within the cluster is not supported. That is, remote mirror replication when the primary and secondary hosts reside in the same cluster and the primary, secondary, and bitmap volumes in a volume set reside in the same disk device group.

If the remote mirror primary and secondary hosts are configured in different clusters, for operating considerations see Remote Mirror Primary Host On a Cluster Node and Remote Mirror Secondary Host On a Cluster Node.


Supported Configurations for the Point-in-Time Copy Software

Following are some rules for Point-in-Time Copy software:

For example, you cannot have a master volume with a disk device group name of
ii-group and a shadow volume with a disk device group name of ii-group2 in the same volume set.


Disk Device Groups and the Sun StorageTek Availability Suite Software

The Solstice DiskSuitetrademark and VERITAS Volume Manager (VxVM) can arrange disk devices into a group to be mastered by a cluster node. You can then configure these disk device groups to fail over to another cluster node, as described in Configuring the Sun Cluster Environment.

The Solstice DiskSuite and VxVM device paths contain the disk device group. When operating in a Sun Cluster environment, the Sun StorageTek Availability Suite commands sndradm and iiadm automatically detect and use the disk device group as configured in Configuring the Sun Cluster Environment.

You can also use the sndradm and iiadm commands to select specified disk device groups or to operate on a volume set as a local node-only configuration entry. See Using the Sun StorageTek Availability Suite iiadm and sndradm Commands


Handling Raw Devices in a Sun Cluster OE

The Sun StorageTek Availability Suite supports the use of raw devices in a Sun Cluster OE (Operating Environment), although the volume configuration procedures to handle raw device are very different from devices under the control of VERITAS Volume Manager (VxVM) or Solaris Volume Manager (SVM).

In a Sun Cluster OE, the Availability Suite supports raw devices only through the use of Sun Cluster Global Devices (as in /dev/global/rdsk/d8s0). Availability Suite can not use Sun Cluster DID devices (such as/dev/did/rdsk/d8s0), because Sun Cluster's disk fencing and data pathing software causes connectivity problems when directly using DID devices.

For example:


iiadm -e ind /dev/global/rdsk/d8s0 /dev/global/rdsk/d8s1 /dev/global/rdsk/d8s2

When the Point-in-Time Copy software is configured on a Global Device, any node in a Sun Cluster can concurrently access the data on the Point-in-Time Copy software master or shadow volume. One node in the Sun Cluster has direct I/O access; all other nodes have cluster-interconnect I/O access to the Point-in-Time Copy software sets that are configured.

Depending on Point-in-Time Copy software set usage patterns, when a specific node is dominating the I/O workload of either the master or shadow volume, the Global Device can be moved to the node of high application usage as follows:


scswitch -z -D dsk/d<n> -h Sun-Cluster-Nodename

Doing so will improve performance and decrease the amount of cluster interconnect traffic.

Global devices are very much like SVM and VxVM volumes, and can be switched between nodes. Although Availability Suite is supported on Global Devices, because of the default DID device behavior of Sun Cluster, each global device is in its own "named" Sun Cluster device group. For example, /dev/global/rdsk/d8 is in device group "dsk/d8". This default Sun Cluster behavior (all constituent volumes in a Point-in-Time Copy must be in the same Sun Cluster device group) would force you to put the Point-in-Time Copy software master, shadow and bitmap volume into the same Global Device, yielding very poor I/O performance.

By default in a Sun Cluster OE, all DID devices are mapped to global devices of the same name. For instance the following DID devices:


/dev/did/rdsk/d6, dsk/d7 & dsk/d8

are mapped to the following global devices:


/dev/global/rdsk/d6, dsk/d7 & dsk/d8

If you tried to put a Point-in-Time Copy software master, shadow and bitmap on separate global devices, this enable operation would fail as follows because of the constituent volume rule stated above.


procedure icon  To Create a Named Global Device Group

1. Bring the devices to the node.


# scswitch -z -D dsk/d6,dsk/d7,dsk/d8 -h Sun-Cluster-Nodename

2. Try to enable a Point-in-Time Copy set spanning the device groups.

In this example, the groups are dsk/d6, dsk/d7, and dsk/d8.


# iiadm -e dep /dev/global/rdsk/d6s0 /dev/global/rdsk/d7s0 /dev/global/rdsk/d8s0
iiadm: Volumes are not in same disk group:

To resolve this issue, you need to reconfigure these different global devices into a "named" global device group.

3. Sun Cluster "switch" the device groups into maintenance mode.


# scswitch -m -D dsk/d6
# scswitch -m -D dsk/d7
# scswitch -m -D dsk/d8

4. Sun Cluster unconfigure the old names from the configuration.


# scconf -r -D name=dsk/d6
# scconf -r -D name=dsk/d7
# scconf -r -D name=dsk/d8

5. Define a new named device group configuration name containing the DID devices.

In this example, the named group is "AVSuite."


# scconf -a -D type=rawdisk, name=AVSuite, nodelist=Sun-Cluster-Node1, Sun-Cluster-Node2, .., Sun-Cluster-NodeN, preferenced=false, failback=disabled, numsecondaries=, globaldev=d6, globaldev=d7, globaldev=d8

6. Bring the named global device group (AVSuite) to the current Sun Cluster node.

In this example, the named group is "AVSuite."


# scswitch -z -D AVSuite -h Sun-Cluster-Node1

7. Try again to enable a Point-in-Time Copy set spanning the named groups.

In this example, the named groups are dsk/d6, dsk/d7, and dsk/d8.


# iiadm -e dep /dev/global/rdsk/d6s0 /dev/global/rdsk/d7s0 /dev/global/rdsk/d8s0

This time, the command is successful.

8. Verify that the cluster tag was set successfully.

Note that after running iiadm -i /dev/global/rdsk/d7s0, the cluster tag is AVSuite, and not dsk/d7.


# iiadm -i /dev/global/rdsk/d7s0
 
/dev/global/rdsk/d6s0: (master volume)
/dev/global/rdsk/d7s0: (shadow volume)
/dev/global/rdsk/d8s0: (bitmap volume)
Cluster tag: AVSuite
Dependent copy
Volume size: 212127744
Shadow chunks total: 3314496 Shadow chunks used: 0
Percent of bitmap set: 0
        (bitmap clean)

9. The newly named global device can now be switched to other Sun Cluster nodes, using its new device group name.


# scswitch -z -D AVSuite -h Sun-Cluster-Node2


procedure icon  To Remove a Named Global Device Group

After you have created "named" global devices, there may be a need to remove them at some future time. Following are the steps to restore the Global Device configuration back to its initial state.

1. Disable the use of the "named" Global Devices.


# iiadm -d /dev/global/rdsk/d7s0

2. Sun Cluster "switch" the device groups into maintenance mode.

In this example, the named group is "AVSuite."


# scswitch -m -D AVSuite

3. Sun Cluster unconfigure the named global device configuration.


# scconf -r -D name=AVSuite

4. Define a (default) dsk/dn device group configuration name for all previously configured DID devices.


# scconf -a -D type=rawdisk, name="dsk/d6", nodelist=Sun-Cluster-Node1,Sun-Cluster-Node2,..,Sun-Cluster-NodeN, preferenced=false,failback=disabled, numsecondaries=, globaldev=d6
# scconf -a -D type=rawdisk, name="dsk/d7", nodelist=Sun-Cluster-Node1,Sun-Cluster-Node2,..,Sun-Cluster-NodeN, preferenced=false, failback=disabled, numsecondaries=, globaldev=d7
# scconf -a -D type=rawdisk, name="dsk/d8", nodelist=Sun-Cluster-Node1,Sun-Cluster-Node2,..,Sun-Cluster-NodeN, preferenced=false, failback=disabled, numsecondaries=, globaldev=d8

5. Bring the named global device groups to the current Sun Cluster node.


# scswitch -z -D dsk/d6,dsk/d7,dsk/d8 -h Sun-Cluster-Node1

6. Verify that the original restriction is now back.


# iiadm -e dep /dev/global/rdsk/d6s0 /dev/global/rdsk/d7s0 /dev/global/rdsk/d8s0
iiadm: Volumes are not in same disk group:


Configuring the Sun Cluster Environment

The procedures in this section describe how to configure the Sun Cluster software for use with the Remote Mirror and Point-in-Time Copy software. The Sun Cluster 3.2 Data Installation and Configuration Guide contains more information about configuring and administering Sun Cluster data services. See the scrgadm(1M) and scswitch(1M) man pages for more information.

The general configuration steps are:

1. Log on to any node in the cluster.

2. Configure a disk device group using your volume manager.

3. Register the SUNW.HAStorage or SUNW.HAStoragePlus resource type.

4. Create a resource group.

5. Add SUNW.HAStorage or SUNW.HAStoragePlus to the disk device group.

6. (Remote Mirror step only) Add a logical failover host to the resource group.

7. Enable the resource group and bring it online.

 


procedure icon  To configure Sun Cluster for HAStorage or HAStoragePlus

1. Log on as the root user on any node in the cluster.

2. Configure a disk device group using your volume manager software.

See the documentation that came with your volume manager software. Also you might check the currently configured groups before configuring a new disk device group. For example, use the metaset(1M), vxdg, or vxprint commands, depending on your volume manager software.

3. Register SUNW.HAStorage or SUNW.HAStoragePlus as a resource type.


# scrgadm -a -t SUNW.HAStorage

# scrgadm -a -t SUNW.HAStoragePlus

 

Note - For information on HAStorage and HAStoragePlus resource types, please refer to the SunCluster documentation. The SunCluster 3.0 5/02 Supplement (Part No. 816-3380-10) contains detailed information.



4. Create a resource group for the devicegroup.


# scrgadm -a -g devicegroup-stor-rg -h node1,node2

devicegroup

is the required disk device group name.

-h node1,node2

specifies the cluster nodes that can master this resource group. If you do not specify these nodes, it defaults to all the nodes in the cluster.


 

5. For a SUNW.HAStorage resource, use the following command to add the resource to the resource group.


# scrgadm -a -j devicegroup-stor -g devicegroup-stor-rg \-t SUNW.HAStorage \-x ServicePaths=devicegroup -x AffinityOn=True

devicegroup

Disk device group name.

-x ServicePaths=

Specifies the extension property that the Sun StorageTek Availability Suite software relies on. In this case, use the disk device devicegroup.

-x AffinityOn=True

Specifies that the SUNW.HAStorage resource needs to perform an affinity switchover for the global devices and cluster file systems defined in -x ServicePaths.

It also enforces co-location of resource groups and disk device groups on the same node, thus enhancing the performance of disk-intensive data services.

If the device group is switched to another node while the SUNW.HAstorage resource is online, AffinityOn has no effect and the resource group does not migrate along with the device group. On the other hand, if the resource group is switched to another node, AffinityOn being set to True causes the device group to follow the resource group to the new node.


 

6. For a a SUNW.HAStoragePlus resource, use the following command to add the resource to the resource group.


# scrgadm -a -j devicegroup-stor -g devicegroup-stor-rg \-t SUNW.HAStoragePlus \-x GlobalDevicePaths=devicegroup -x AffinityOn=True

-x GlobalDevicePaths=

specifies the extension property that the Sun StorageTek Availability Suite software relies on. In this case, use the disk device devicegroup.

-x AffinityOn=True

specifies that the SUNW.HAStoragePlus resource needs to perform an affinity switchover for the global devices and cluster file systems defined in
-x GlobalDevicePaths.

It also enforces co-location of resource groups and disk device groups on the same node, thus enhancing the performance of disk-intensive data services.

If the device group is switched to another node while the SUNW.HAstoragePlus resource is online, AffinityOn has no effect and the resource group does not migrate along with the device group. On the other hand, if the resource group is switched to another node, AffinityOn being set to True causes the device group to follow the resource group to the new node.


 

7. Add a logical hostname resource to the resource group.



Note - Perform this step for the remote mirror volumes only. This step is not needed for Point-in-Time Copy volumes.




# scrgadm -a -L [-j lhost-stor] -g devicegroup-stor-rg \-l lhost1, lhost2,..., lhostN -n nafo0@node,nafo0@node

-j lhost-stor

Optional resource lhost-stor. If you do not specify this option and resource, the name defaults to the first logical hostname specified in the -l option.

-l lhost1,lhost2,...lhostN

Specifies a comma-separated list of UNIX hostnames (logical hostnames) by which clients communicate with the Sun StorageTek Availability Suite software in the resource group.

-n nafo0@node,nafo0@node

Specifies the comma-separated list of Network Adapter Failover (NAFO) groups on each node.

 

node can be a node name or ID. You can display the node ID using the scconf -p command.


 

8. Enable the resources in the resource group, manage the resource group, and bring the resource group online.


# scswitch -Z -g devicegroup-stor-rg

9. Verify that the resource is online.

a. Run the following command on any cluster node.


# scstat -g

b. Look for the resource group state field to determine if the resource group is online on the nodes specified in the node list.

10. For the HAStoragePlus resource, verify that the resource group can be failed between nodes.


# scswitch -z -g devicegroup-stor-rg -h fail-to node 

The above command fails the resource group to the specified node.


# scswitch -S -h fail-from node

The above command fails ALL resources from the specified node.

Configuring the HAStoragePlus Resource Types with Volume Sets

This example shows how to configure a resource group on a locally mounted Sun Cluster global device partition.

You can configure the HAStoragePlus resource to fail over resource groups as well as individual volume sets to another node in the cluster. When configuring a resource type with volume sets, consider the following:

where:


-j iidg-rs

is the resource name.

-g iidg

is the resource group name.

-x GlobalDevicePaths=

specifies the extension property GlobalDevicePath and raw device volume names for the point-in-time copy volume set.