Creating Disk Sets in a Cluster

This section describes how to create disk sets for a cluster configuration. When you create a Solaris Volume Manager disk set in a Sun Cluster environment, the disk set is automatically registered with the Sun Cluster software as a device group of type svm. To create or delete an svm device group, you must use Solaris Volume Manager commands and utilities to create or delete the underlying disk set of the device group.

The following table lists the tasks that you perform to create disk sets. Complete the procedures in the order that is indicated.

How to Create a Disk Set

Perform this procedure to create disk sets.

1. SPARC: (Solaris 9) Determine whether, after you create the new disk sets, the global cluster will have more than three disk sets.

   • If the cluster will have no more than three disk sets, skip to Step 9.

   • If the cluster will have four or more disk sets, proceed to Step 2 to prepare the cluster. You must perform this task whether you are installing disk sets for the first time or whether you are adding more disk sets to a fully configured cluster.

   • If the cluster runs on the Solaris 10 OS, Solaris Volume Manager automatically makes the necessary configuration changes. Skip to Step 9.

2. On any node of the cluster, check the value of the md_nsets variable in the /kernel/drv/md.conf file.

3. If the total number of disk sets in the cluster will be greater than the existing value of md_nsets minus one, increase the value of md_nsets to the desired value.

   The maximum permissible number of disk sets is one less than the configured value of md_nsets. The maximum possible value of md_nsets is 32, therefore the maximum permissible number of disk sets that you can create is 31.

4. Ensure that the /kernel/drv/md.conf file is identical on each node of the cluster.

   ---

   Caution –

   Failure to follow this guideline can result in serious Solaris Volume Manager errors and possible loss of data.

   ---

5. If you made changes to the md.conf file on any node, perform the following steps to make those changes active.

   1. On one node, become superuser.

   2. From one node, shut down the cluster.

      phys-schost# cluster shutdown -g0 -y

   3. Reboot each node of the cluster.

      • On SPARC based systems, do the following:

        ok boot

      • On x86 based systems, do the following:

        When the GRUB menu is displayed, select the appropriate Solaris entry and press Enter. The GRUB menu appears similar to the following:

        GNU GRUB version 0.95 (631K lower / 2095488K upper memory) +-------------------------------------------------------------------------+ | Solaris 10 /sol_10_x86 | | Solaris failsafe | | | +-------------------------------------------------------------------------+ Use the ^ and v keys to select which entry is highlighted. Press enter to boot the selected OS, 'e' to edit the commands before booting, or 'c' for a command-line.

        For more information about GRUB based booting, see Chapter 11, GRUB Based Booting (Tasks), in System Administration Guide: Basic Administration.

6. On each node in the cluster, run the devfsadm(1M) command.

   You can run this command on all nodes in the cluster at the same time.

7. From one node of the cluster, update the global-devices namespace.

   phys-schost# cldevice populate

   See the cldevice(1CL) man page for more information.

8. On each node, verify that the command has completed processing before you attempt to create any disk sets.

   The command executes remotely on all nodes, even though the command is run from just one node. To determine whether the command has completed processing, run the following command on each node of the cluster.

   phys-schost# ps -ef | grep scgdevs

9. Ensure that the disk set that you intend to create meets one of the following requirements.

   • If the disk set is configured with exactly two disk strings, the disk set must connect to exactly two nodes and use exactly two mediator hosts. These mediator hosts must be the same two hosts used for the disk set. See Configuring Dual-String Mediators for details on how to configure dual-string mediators.

   • If the disk set is configured with more than two disk strings, ensure that for any two disk strings S1 and S2, the sum of the number of drives on those strings exceeds the number of drives on the third string S3. Stated as a formula, the requirement is that count(S1) + count(S2) > count(S3).

10. Ensure that the local state database replicas exist.

    For instructions, see How to Create State Database Replicas.

11. Become superuser on the cluster node that will master the disk set.

12. Create the disk set.

    The following command creates the disk set and registers the disk set as a Sun Cluster device group.

    phys-schost# metaset -s setname -a -h node1 node2

    -s setname

    Specifies the disk set name.

    -a

    Adds (creates) the disk set.

    -h node1

    Specifies the name of the primary node to master the disk set.

    node2

    Specifies the name of the secondary node to master the disk set

    ---

    Note –

    When you run the metaset command to configure a Solaris Volume Manager device group on a cluster, the command designates one secondary node by default. You can change the desired number of secondary nodes in the device group by using the clsetup utility after the device group is created. Refer to Administering Device Groups in Sun Cluster System Administration Guide for Solaris OS for more information about how to change the numsecondaries property.

    ---

13. If you are configuring a replicated Solstice DiskSuite or Solaris Volume Manager device group, set the replication property for the device group.

    phys-schost# cldevicegroup sync device-group-name

    For more information about data replication, see Chapter 4, Data Replication Approaches, in Sun Cluster System Administration Guide for Solaris OS.

14. Verify the status of the new disk set.

    phys-schost# metaset -s setname

15. As needed, set device group properties.

    phys-schost# cldevicegroup set -p name=value devicegroup

    -p

    Specifies a device-group property.

    name

    Specifies the name of a property.

    value

    Specifies the value or setting of the property.

    devicegroup

    Specifies the name of the device group. The device-group name is the same as the disk-set name.

    See the cldevicegroup(1CL) for information about device-group properties.

Example 4–6 Creating a Disk Set

The following command creates two disk sets, dg-schost-1 and dg-schost-2, with the nodes phys-schost-1 and phys-schost-2 specified as the potential primaries.

phys-schost# metaset -s dg-schost-1 -a -h phys-schost-1 phys-schost-2
phys-schost# metaset -s dg-schost-2 -a -h phys-schost-1 phys-schost-2

Next Steps

Add drives to the disk set. Go to Adding Drives to a Disk Set.

Adding Drives to a Disk Set

When you add a drive to a disk set, the volume management software repartitions the drive as follows so that the state database for the disk set can be placed on the drive.

• A small portion of each drive is reserved for use by Solaris Volume Manager software. In volume table of contents (VTOC) labeled devices, slice 7 is used. In Extensible Firmware Interface (EFI) labeled devices, slice 6 is used. The remainder of the space on each drive is placed into slice 0.

• Drives are repartitioned when they are added to the disk set only if the target slice is not configured correctly.

• Any existing data on the drives is lost by the repartitioning.

• If the target slice starts at cylinder 0, and the drive partition is large enough to contain a state database replica, the drive is not repartitioned.

How to Add Drives to a Disk Set

Before You Begin

Ensure that the disk set has been created. For instructions, see How to Create a Disk Set.

1. Become superuser.

2. List the DID mappings.

   phys-schost# cldevice show | grep Device

   • Choose drives that are shared by the cluster nodes that will master or potentially master the disk set.

   • Use the full DID device name, which has the form /dev/did/rdsk/dN,when you add a drive to a disk set.

   In the following example, the entries for DID device /dev/did/rdsk/d3 indicate that the drive is shared by phys-schost-1 and phys-schost-2.

   === DID Device Instances === DID Device Name: /dev/did/rdsk/d1 Full Device Path: phys-schost-1:/dev/rdsk/c0t0d0 DID Device Name: /dev/did/rdsk/d2 Full Device Path: phys-schost-1:/dev/rdsk/c0t6d0 DID Device Name: /dev/did/rdsk/d3 Full Device Path: phys-schost-1:/dev/rdsk/c1t1d0 Full Device Path: phys-schost-2:/dev/rdsk/c1t1d0 …

3. Become owner of the disk set.

   phys-schost# cldevicegroup switch -n node devicegroup

   -n node

   Specifies the node to take ownership of the device group.

   devicegroup

   Specifies the device group name, which is the same as the disk set name.

4. Add the drives to the disk set.

   Use the full DID path name.

   phys-schost# metaset -s setname -a /dev/did/rdsk/dN

   -s setname

   Specifies the disk set name, which is the same as the device group name.

   -a

   Adds the drive to the disk set.

   ---

   Note –

   Do not use the lower-level device name (cNtXdY) when you add a drive to a disk set. Because the lower-level device name is a local name and not unique throughout the cluster, using this name might prevent the metaset from being able to switch over.

   ---

5. Verify the status of the disk set and drives.

   phys-schost# metaset -s setname

Example 4–7 Adding Drives to a Disk Set

The metaset command adds the drives /dev/did/rdsk/d1 and /dev/did/rdsk/d2 to the disk set dg-schost-1.

phys-schost# metaset -s dg-schost-1 -a /dev/did/rdsk/d1 /dev/did/rdsk/d2

Next Steps

To repartition drives for use in volumes, go to How to Repartition Drives in a Disk Set.

Otherwise, go to How to Create an md.tab File to define metadevices or volumes by using an md.tab file.

How to Repartition Drives in a Disk Set

The metaset(1M) command repartitions drives in a disk set so that a small portion of each drive is reserved for use by Solaris Volume Manager software. In volume table of contents (VTOC) labeled devices, slice 7 is used. In Extensible Firmware Interface (EFI) labeled devices, slice 6 is used. The remainder of the space on each drive is placed into slice 0. To make more effective use of the drive, use this procedure to modify the disk layout. If you allocate space to VTOC slices 1 through 6 or EFI slices 1 through 5, you can use these slices when you set up Solaris Volume Manager volumes.

1. Become superuser.

2. Use the format command to change the disk partitioning for each drive in the disk set.

   When you repartition a drive, you must meet the following conditions to prevent the metaset(1M) command from repartitioning the drive.

   • Create slice 7 for VTOC or slice 6 for EFI starting at cylinder 0, large enough to hold a state database replica. See your Solaris Volume Manager administration guide to determine the size of a state database replica for your version of the volume-manager software.

   • Set the Flag field in the target slice to wu (read-write, unmountable). Do not set it to read-only.

   • Do not allow the target slice to overlap any other slice on the drive.

   See the format(1M) man page for details.

Next Steps

Define volumes by using an md.tab file. Go to How to Create an md.tab File.

How to Create an md.tab File

Create an /etc/lvm/md.tab file on each node in the cluster. Use the md.tab file to define Solaris Volume Manager volumes for the disk sets that you created.

If you are using local volumes, ensure that local volume names are distinct from the device-ID names that are used to form disk sets. For example, if the device-ID name /dev/did/dsk/d3 is used in a disk set, do not use the name /dev/md/dsk/d3 for a local volume. This requirement does not apply to shared volumes, which use the naming convention /dev/md/setname/{r}dsk/d#.

1. Become superuser.

2. List the DID mappings for reference when you create your md.tab file.

   Use the full DID device names in the md.tab file in place of the lower-level device names (cN tXdY). The DID device name takes the form /dev/did/rdsk/dN.

   phys-schost# cldevice show | grep Device

   === DID Device Instances === DID Device Name: /dev/did/rdsk/d1 Full Device Path: phys-schost-1:/dev/rdsk/c0t0d0 DID Device Name: /dev/did/rdsk/d2 Full Device Path: phys-schost-1:/dev/rdsk/c0t6d0 DID Device Name: /dev/did/rdsk/d3 Full Device Path: phys-schost-1:/dev/rdsk/c1t1d0 Full Device Path: phys-schost-2:/dev/rdsk/c1t1d0 …

3. Create an /etc/lvm/md.tab file and edit it with your preferred text editor.

   ---

   Note –

   If you have existing data on the drives that will be used for the submirrors, you must back up the data before volume setup. Then restore the data onto the mirror.

   ---

   To avoid possible confusion between local volumes on different nodes in a cluster environment, use a naming scheme that makes each local volume name unique throughout the cluster. For example, for node 1 choose names from d100 to d199. And for node 2 use d200 to d299.

   See your Solaris Volume Manager documentation and the md.tab(4) man page for details about how to create an md.tab file.

Example 4–8 Sample md.tab File

The following sample md.tab file defines the disk set that is named dg-schost-1. The ordering of lines in the md.tab file is not important.

dg-schost-1/d0 -m dg-schost-1/d10 dg-schost-1/d20
    dg-schost-1/d10 1 1 /dev/did/rdsk/d1s0
    dg-schost-1/d20 1 1 /dev/did/rdsk/d2s0

The sample md.tab file is constructed as follows.

1. The first line defines the device d0 as a mirror of volumes d10 and d20. The -m signifies that this device is a mirror device.

   dg-schost-1/d0 -m dg-schost-1/d0 dg-schost-1/d20

2. The second line defines volume d10, the first submirror of d0, as a one-way stripe.

   dg-schost-1/d10 1 1 /dev/did/rdsk/d1s0

3. The third line defines volume d20, the second submirror of d0, as a one-way stripe.

   dg-schost-1/d20 1 1 /dev/did/rdsk/d2s0

Next Steps

Activate the volumes that are defined in the md.tab files. Go to How to Activate Volumes.

How to Activate Volumes

Perform this procedure to activate Solaris Volume Manager volumes that are defined in md.tab files.

1. Become superuser.

2. Ensure that md.tab files are located in the /etc/lvm directory.

3. Ensure that you have ownership of the disk set on the node where the command will be executed.

4. Take ownership of the disk set.

   phys-schost# cldevicegroup switch -n node devicegroup

   -n node

   Specifies the node that takes ownership.

   devicegroup

   Specifies the disk set name.

5. Activate the disk set's volumes, which are defined in the md.tab file.

   phys-schost# metainit -s setname -a

   -s setname

   Specifies the disk set name.

   -a

   Activates all volumes in the md.tab file.

6. Repeat Step 3 through Step 5 for each disk set in the cluster.

   If necessary, run the metainit(1M) command from another node that has connectivity to the drives. This step is required for cluster-pair topologies, where the drives are not accessible by all nodes.

7. Check the status of the volumes.

   phys-schost# metastat -s setname

   See the metastat(1M) man page for more information.

8. (Optional) Capture the disk partitioning information for future reference.

   phys-schost# prtvtoc /dev/rdsk/cNtXdYsZ > filename

   Store the file in a location outside the cluster. If you make any disk configuration changes, run this command again to capture the changed configuration. If a disk fails and needs replacement, you can use this information to restore the disk partition configuration. For more information, see the prtvtoc(1M) man page.

9. (Optional) Make a backup of your cluster configuration.

   An archived backup of your cluster configuration facilitates easier recovery of the your cluster configuration. For more information, see How to Back Up the Cluster Configuration in Sun Cluster System Administration Guide for Solaris OS.

Example 4–9 Activating Volumes in the md.tab File

In the following example, all volumes that are defined in the md.tab file for disk set dg-schost-1 are activated.

phys-schost# metainit -s dg-schost-1 -a

Next Steps

If your cluster contains disk sets that are configured with exactly two disk enclosures and two nodes, add dual-string mediators. Go to Configuring Dual-String Mediators.

Otherwise, go to How to Create Cluster File Systems to create a cluster file system.