Creating Disk Sets in a Cluster

This section describes how to create disk sets for a cluster configuration. You might not need to create disk sets under the following circumstances:

• If you used SunPlex Installer to install Solstice DiskSuite, one to three disk sets might already exist. See Using SunPlex Installer to Install Sun Cluster Software for information about the metasets that were created by SunPlex Installer.

• To create a Solaris Volume Manager for Sun Cluster disk set for use by Oracle Real Application Clusters, do not use these procedures. Instead, perform the procedures in “Creating a Multi-Owner Disk Set in Solaris Volume Manager for Sun Cluster for the Oracle Real Application Clusters Database” in Sun Cluster Data Service for Oracle Real Application Clusters Guide for Solaris OS.

The following procedures are in this section:

• How to Create a Disk Set

• How to Add Drives to a Disk Set

• How to Repartition Drives in a Disk Set

• How to Create an md.tab File

• How to Activate Metadevices or Volumes

How to Create a Disk Set

Perform this procedure to create disk sets.

1. Determine whether the cluster will have more than three disk sets after you create the new disk sets.

   • If the cluster will have no more than three disk sets, skip to Step 2.

   • If the cluster will have four or more disk sets, perform the following steps to prepare the cluster.

     You must perform this task whether you are installing disk sets for the first time or whether you are adding more disk sets to a fully configured cluster.

   1. On any node of the cluster, check the value of the md_nsets variable in the /kernel/drv/md.conf file.

   2. If the total number of disk sets in the cluster will be greater than the existing value of md_nsets minus one, increase the value of md_nsets to the desired value.

      The maximum permissible number of disk sets is one less than the configured value of md_nsets. The maximum possible value of md_nsets is 32, therefore the maximum permissible number of disk sets that you can create is 31.

   3. Ensure that the /kernel/drv/md.conf file is identical on each node of the cluster.

      ---

      Caution –

      Failure to follow this guideline can result in serious Solstice DiskSuite or Solaris Volume Manager errors and possible loss of data.

      ---

   4. If you made changes to the md.conf file on any node, perform the following steps to make those changes active.

      1. From one node, shut down the cluster.

         # scshutdown -g0 -y

      2. Reboot each node of the cluster.

         ok> boot

   5. On each node in the cluster, run the devfsadm(1M) command.

      You can run this command on all nodes in the cluster at the same time.

   6. From one node of the cluster, run the scgdevs(1M) command to update the global-devices namespace.

   7. On each node, verify that the scgdevs command has completed processing before you attempt to create any disk sets.

      The scgdevs command calls itself remotely on all nodes, even when the command is run from just one node. To determine whether the scgdevs command has completed processing, run the following command on each node of the cluster.

      % ps -ef | grep scgdevs

2. Ensure that the disk set you intend to create meets one of the following requirements.

   • If the disk set is configured with exactly two disk strings, the disk set must connect to exactly two nodes and use exactly two mediator hosts. These mediator hosts must be the same two hosts used for the disk set. See Configuring Dual-String Mediators for details on how to configure dual-string mediators.

   • If the disk set is configured with more than two disk strings, ensure that for any two disk strings S1 and S2, the sum of the number of drives on those strings exceeds the number of drives on the third string S3. Stated as a formula, the requirement is that count(S1) + count(S2) > count(S3).

3. Ensure that the local state database replicas exist.

   For instructions, see How to Create State Database Replicas.

4. Become superuser on the cluster node that will master the disk set.

5. Create the disk set.

   The following command creates the disk set and registers the disk set as a Sun Cluster disk device group.

   # metaset -s setname -a -h node1 node2

   -s setname

   Specifies the disk set name

   -a

   Adds (creates) the disk set

   -h node1

   Specifies the name of the primary node to master the disk set

   node2

   Specifies the name of the secondary node to master the disk set

   ---

   Note –

   When you run the metaset command to configure a Solstice DiskSuite or Solaris Volume Manager device group on a cluster, the command designates one secondary node by default. You can change the desired number of secondary nodes in the device group by using the scsetup(1M) utility after the device group is created. Refer to “Administering Disk Device Groups” in Sun Cluster System Administration Guide for Solaris OS for more information about how to change the numsecondaries property.

   ---

6. Verify the status of the new disk set.

   # metaset -s setname

7. Add drives to the disk set.

   Go to Adding Drives to a Disk Set.

Example—Creating a Disk Set

The following command creates two disk sets, dg-schost-1 and dg-schost-2, with the nodes phys-schost-1 and phys-schost-2 specified as the potential primaries.

# metaset -s dg-schost-1 -a -h phys-schost-1 phys-schost-2
# metaset -s dg-schost-2 -a -h phys-schost-1 phys-schost-2

Adding Drives to a Disk Set

When you add a drive to a disk set, the volume management software repartitions the drive as follows so that the state database for the disk set can be placed on the drive.

• A small portion of each drive is reserved in slice 7 for use by Solstice DiskSuite or Solaris Volume Manager software. The remainder of the space on each drive is placed into slice 0.

• Drives are repartitioned when they are added to the disk set only if slice 7 is not configured correctly.

• Any existing data on the drives is lost by the repartitioning.

• If slice 7 starts at cylinder 0, and the drive partition is large enough to contain a state database replica, the drive is not repartitioned.

How to Add Drives to a Disk Set

1. Become superuser on the node.

2. Ensure that the disk set has been created.

   For instructions, see How to Create a Disk Set.

3. List the DID mappings.

   # scdidadm -L

   • Choose drives that are shared by the cluster nodes that will master or potentially master the disk set.

   • Use the full device-ID path names when you add drives to a disk set.

   The first column of output is the DID instance number, the second column is the full physical path name, and the third column is the full device-ID path name (pseudo path). A shared drive has more than one entry for the same DID instance number.

   In the following example, the entries for DID instance number 2 indicate a drive that is shared by phys-schost-1 and phys-schost-2, and the full device-ID path name is /dev/did/rdsk/d2.

   1       phys-schost-1:/dev/rdsk/c0t0d0 /dev/did/rdsk/d1 2       phys-schost-1:/dev/rdsk/c1t1d0 /dev/did/rdsk/d2 2       phys-schost-2:/dev/rdsk/c1t1d0 /dev/did/rdsk/d2 3       phys-schost-1:/dev/rdsk/c1t2d0 /dev/did/rdsk/d3 3       phys-schost-2:/dev/rdsk/c1t2d0 /dev/did/rdsk/d3 …

4. Take ownership of the disk set.

   # metaset -s setname -t

   -s setname

   Specifies the disk set name

   -t

   Takes ownership of the disk set

5. Add the drives to the disk set.

   Use the full device-ID path name.

   # metaset -s setname -a drivename

   -a

   Adds the drive to the disk set

   drivename

   Full device-ID path name of the shared drive

   ---

   Note –

   Do not use the lower-level device name (cNtXdY) when you add a drive to a disk set. Because the lower-level device name is a local name and not unique throughout the cluster, using this name might prevent the metaset from being able to switch over.

   ---

6. Verify the status of the disk set and drives.

   # metaset -s setname

7. (Optional) To repartition drives for use in metadevices or volumes, go to How to Repartition Drives in a Disk Set.

8. Go to How to Create an md.tab File to define metadevices or volumes by using an md.tab file.

Example—Adding Drives to a Disk Set

The metaset command adds the drives /dev/did/rdsk/d1 and /dev/did/rdsk/d2 to the disk set dg-schost-1.

# metaset -s dg-schost-1 -a /dev/did/rdsk/d1 /dev/did/rdsk/d2

How to Repartition Drives in a Disk Set

The metaset(1M) command repartitions drives in a disk set so that a small portion of each drive is reserved in slice 7 for use by Solstice DiskSuite or Solaris Volume Manager software. The remainder of the space on each drive is placed into slice 0. To make more effective use of the drive, use this procedure to modify the disk layout. If you allocate space to slices 1 through 6, you can use these slices when you set up Solstice DiskSuite metadevices or Solaris Volume Manager volumes.

1. Become superuser on the cluster node.

2. Use the format command to change the disk partitioning for each drive in the disk set.

   When you repartition a drive, you must meet the following conditions to prevent the metaset(1M) command from repartitioning the drive.

   • Create slice 7 starting at cylinder 0, large enough to hold a state database replica. See your Solstice DiskSuite or Solaris Volume Manager administration guide to determine the size of a state database replica for your version of the volume-manager software.

   • Set the Flag field in slice 7 to wu (read-write, unmountable). Do not set it to read-only.

   • Do not allow slice 7 to overlap any other slice on the drive.

   See the format(1M) man page for details.

3. Define metadevices or volumes by using an md.tab file.

   Go to How to Create an md.tab File.

How to Create an md.tab File

Create an /etc/lvm/md.tab file on each node in the cluster. Use the md.tab file to define Solstice DiskSuite metadevices or Solaris Volume Manager volumes for the disk sets that you created.

If you are using local metadevices or volumes, ensure that local metadevices or volumes names are distinct from the device-ID names used to form disk sets. For example, if the device-ID name /dev/did/dsk/d3 is used in a disk set, do not use the name /dev/md/dsk/d3 for a local metadevice or volume. This requirement does not apply to shared metadevices or volumes, which use the naming convention /dev/md/setname/{r}dsk/d#.

To avoid possible confusion between local metadevices or volumes in a cluster environment, use a naming scheme that makes each local metadevice or volume name unique throughout the cluster. For example, for node 1 choose names from d100-d199. And for node 2 use d200-d299.

1. Become superuser on the cluster node.

2. List the DID mappings for reference when you create your md.tab file.

   Use the full device-ID path names in the md.tab file in place of the lower-level device names (cNtXdY).

   # scdidadm -L

   In the following example, the first column of output is the DID instance number, the second column is the full physical path name, and the third column is the full device-ID path name (pseudo path).

   1       phys-schost-1:/dev/rdsk/c0t0d0 /dev/did/rdsk/d1 2       phys-schost-1:/dev/rdsk/c1t1d0 /dev/did/rdsk/d2 2       phys-schost-2:/dev/rdsk/c1t1d0 /dev/did/rdsk/d2 3       phys-schost-1:/dev/rdsk/c1t2d0 /dev/did/rdsk/d3 3       phys-schost-2:/dev/rdsk/c1t2d0 /dev/did/rdsk/d3 …

3. Create an /etc/lvm/md.tab file and edit it by hand with your preferred text editor.

   See your Solstice DiskSuite or Solaris Volume Manager documentation and the md.tab(4) man page for details on how to create an md.tab file.

   ---

   Note –

   If you have existing data on the drives that will be used for the submirrors, you must back up the data before metadevice or volume setup. Then restore the data onto the mirror.

   ---

4. Activate the metadevices or volumes that are defined in the md.tab files.

   Go to How to Activate Metadevices or Volumes.

Example—Sample md.tab File

The following sample md.tab file defines the disk set that is named dg-schost-1. The ordering of lines in the md.tab file is not important.

dg-schost-1/d0 -m dg-schost-1/d10 dg-schost-1/d20
    dg-schost-1/d10 1 1 /dev/did/rdsk/d1s0
    dg-schost-1/d20 1 1 /dev/did/rdsk/d2s0

The following example uses Solstice DiskSuite terminology. For Solaris Volume Manager, a trans metadevice is instead called a transactional volume and a metadevice is instead called a volume. Otherwise, the following process is valid for both volume managers.

The sample md.tab file is constructed as follows.

1. The first line defines the device d0 as a mirror of metadevices d10 and d20. The -m signifies that this device is a mirror device.

   dg-schost-1/d0 -m dg-schost-1/d0 dg-schost-1/d20

2. The second line defines metadevice d10, the first submirror of d0, as a one-way stripe.

   dg-schost-1/d10 1 1 /dev/did/rdsk/d1s0

3. The third line defines metadevice d20, the second submirror of d0, as a one-way stripe.

   dg-schost-1/d20 1 1 /dev/did/rdsk/d2s0

How to Activate Metadevices or Volumes

Perform this procedure to activate Solstice DiskSuite metadevices or Solaris Volume Manager volumes that are defined in md.tab files.

1. Become superuser on the cluster node.

2. Ensure that md.tab files are located in the /etc/lvm directory.

3. Ensure that you have ownership of the disk set on the node where the command will be executed.

4. Take ownership of the disk set.

   # metaset -s setname -t

   -s setname

   Specifies the disk set name

   -t

   Takes ownership of the disk set

5. Activate the disk set's metadevices or volumes, which are defined in the md.tab file.

   # metainit -s setname -a

   -a

   Activates all metadevices in the md.tab file

6. For each master and log device, attach the second submirror (submirror2).

   When the metadevices or volumes in the md.tab file are activated, only the first submirror (submirror1) of the master and log devices is attached, so submirror2 must be attached by hand.

   # metattach mirror submirror2

7. Repeat Step 3 through Step 6 for each disk set in the cluster.

   If necessary, run the metainit(1M) command from another node that has connectivity to the drives. This step is required for cluster-pair topologies, where the drives are not accessible by all nodes.

8. Check the status of the metadevices or volumes.

   # metastat -s setname

   See the metastat(1M) man page for more information.

9. If your cluster contains disk sets that are configured with exactly two disk enclosures and two nodes, add dual-string mediators.

   Go to Configuring Dual-String Mediators.

10. Go to How to Create Cluster File Systems to create a cluster file system.

Example—Activating Metadevices or Volumes in the md.tab File

In the following example, all metadevices that are defined in the md.tab file for disk set dg-schost-1 are activated. Then the second submirrors of master device dg-schost-1/d1 and log device dg-schost-1/d4 are activated.

# metainit -s dg-schost-1 -a
# metattach dg-schost-1/d1 dg-schost-1/d3
# metattach dg-schost-1/d4 dg-schost-1/d6