Creating Disk Sets in a Cluster

This section describes how to create disk sets for a cluster configuration. When you create a Solaris Volume Manager disk set in an Oracle Solaris Cluster environment, the disk set is automatically registered with the Oracle Solaris Cluster software as a device group of type svm. To create or delete an svm device group, you must use Solaris Volume Manager commands and utilities to create or delete the underlying disk set of the device group.

The following table lists the tasks that you perform to create disk sets. Complete the procedures in the order that is indicated.

Table 4-2 Task Map: Installing and Configuring Solaris Volume Manager Software

How to Create a Disk Set

Perform this procedure to create disk sets.

1. On each node in the cluster, run the devfsadm(1M) command.

   You can run this command on all nodes in the cluster at the same time.

2. From one node of the cluster, update the global-devices namespace.

   phys-schost# cldevice populate

   See the cldevice(1CL) man page for more information.

3. On each node, verify that the command has completed processing before you attempt to create any disk sets.

   The command executes remotely on all nodes, even though the command is run from just one node. To determine whether the command has completed processing, run the following command on each node of the cluster.

   phys-schost# ps -ef | grep scgdevs

4. Ensure that the disk set that you intend to create meets one of the following requirements.

   • If the disk set is configured with exactly two disk strings, the disk set must connect to exactly two nodes and use two or three mediator hosts. These mediator hosts must include the two hosts attached to the enclosures containing the disk set. See Configuring Dual-String Mediators for details on how to configure dual-string mediators.

   • If the disk set is configured with more than two disk strings, ensure that for any two disk strings S1 and S2, the sum of the number of drives on those strings exceeds the number of drives on the third string S3. Stated as a formula, the requirement is that count(S1) + count(S2) > count(S3).

5. Ensure that the local state database replicas exist.

   For instructions, see How to Create State Database Replicas.

6. Become superuser on the cluster node that will master the disk set.
7. Create the disk set.

   The following command creates the disk set and registers the disk set as an Oracle Solaris Cluster device group.

   phys-schost# metaset -s setname -a -h node1 node2

   -s setname

   Specifies the disk set name.

   -a

   Adds (creates) the disk set.

   -h node1

   Specifies the name of the primary node to master the disk set.

   node2

   Specifies the name of the secondary node to master the disk set

   ---

   Note - When you run the metaset command to configure a Solaris Volume Manager device group on a cluster, the command designates one secondary node by default. You can change the desired number of secondary nodes in the device group by using the clsetup utility after the device group is created. Refer to Administering Device Groups in Oracle Solaris Cluster System Administration Guide for more information about how to change the numsecondaries property.

   ---

8. If you are configuring a replicated Solaris Volume Manager device group, set the replication property for the device group.

   phys-schost# cldevicegroup sync device-group-name

   For more information about data replication, see Chapter 4, Data Replication Approaches, in Oracle Solaris Cluster System Administration Guide.

9. Verify the status of the new disk set.

   phys-schost# metaset -s setname

10. As needed, set device group properties.

    phys-schost# cldevicegroup set -p name=value devicegroup

    -p

    Specifies a device-group property.

    name

    Specifies the name of a property.

    value

    Specifies the value or setting of the property.

    devicegroup

    Specifies the name of the device group. The device-group name is the same as the disk-set name.

    See the cldevicegroup(1CL) for information about device-group properties.

Example 4-6 Creating a Disk Set

The following command creates two disk sets, dg-schost-1 and dg-schost-2, with the nodes phys-schost-1 and phys-schost-2 specified as the potential primaries.

phys-schost# metaset -s dg-schost-1 -a -h phys-schost-1 phys-schost-2
phys-schost# metaset -s dg-schost-2 -a -h phys-schost-1 phys-schost-2

Next Steps

Add drives to the disk set. Go to Adding Drives to a Disk Set.

Adding Drives to a Disk Set

When you add a drive to a disk set, the volume management software repartitions the drive as follows so that the state database for the disk set can be placed on the drive.

• A small portion of each drive is reserved for use by Solaris Volume Manager software. In volume table of contents (VTOC) labeled devices, slice 7 is used. In Extensible Firmware Interface (EFI) labeled devices, slice 6 is used. The remainder of the space on each drive is placed into slice 0.

• Drives are repartitioned when they are added to the disk set only if the target slice is not configured correctly.

• Any existing data on the drives is lost by the repartitioning.

• If the target slice starts at cylinder 0, and the drive partition is large enough to contain a state database replica, the drive is not repartitioned.

How to Add Drives to a Disk Set

Before You Begin

Ensure that the disk set has been created. For instructions, see How to Create a Disk Set.

1. Become superuser.
2. List the DID mappings.

   phys-schost# cldevice show | grep Device

   • Choose drives that are shared by the cluster nodes that will master or potentially master the disk set.

   • Use the full DID device name, which has the form /dev/did/rdsk/dN,when you add a drive to a disk set.

   In the following example, the entries for DID device /dev/did/rdsk/d3 indicate that the drive is shared by phys-schost-1 and phys-schost-2.

   === DID Device Instances ===                   
   DID Device Name:                                /dev/did/rdsk/d1
     Full Device Path:                               phys-schost-1:/dev/rdsk/c0t0d0
   DID Device Name:                                /dev/did/rdsk/d2
     Full Device Path:                               phys-schost-1:/dev/rdsk/c0t6d0
   DID Device Name:                                /dev/did/rdsk/d3
     Full Device Path:                               phys-schost-1:/dev/rdsk/c1t1d0
     Full Device Path:                               phys-schost-2:/dev/rdsk/c1t1d0
   …

3. Become owner of the disk set.

   phys-schost# cldevicegroup switch -n node devicegroup

   -n node

   Specifies the node to take ownership of the device group.

   devicegroup

   Specifies the device group name, which is the same as the disk set name.

4. Add the drives to the disk set.

   Use the full DID path name.

   phys-schost# metaset -s setname -a /dev/did/rdsk/dN

   -s setname

   Specifies the disk set name, which is the same as the device group name.

   -a

   Adds the drive to the disk set.

   ---

   Note - Do not use the lower-level device name (cNtXdY) when you add a drive to a disk set. Because the lower-level device name is a local name and not unique throughout the cluster, using this name might prevent the metaset from being able to switch over.

   ---

5. Verify the status of the disk set and drives.

   phys-schost# metaset -s setname

Example 4-7 Adding Drives to a Disk Set

The metaset command adds the drives /dev/did/rdsk/d1 and /dev/did/rdsk/d2 to the disk set dg-schost-1.

phys-schost# metaset -s dg-schost-1 -a /dev/did/rdsk/d1 /dev/did/rdsk/d2

Next Steps

To repartition drives for use in volumes, go to How to Repartition Drives in a Disk Set.

Otherwise, go to How to Create an md.tab File to define metadevices or volumes by using an md.tab file.

How to Repartition Drives in a Disk Set

The metaset(1M) command repartitions drives in a disk set so that a small portion of each drive is reserved for use by Solaris Volume Manager software. In volume table of contents (VTOC) labeled devices, slice 7 is used. In Extensible Firmware Interface (EFI) labeled devices, slice 6 is used. The remainder of the space on each drive is placed into slice 0. To make more effective use of the drive, use this procedure to modify the disk layout. If you allocate space to VTOC slices 1 through 6 or EFI slices 1 through 5, you can use these slices when you set up Solaris Volume Manager volumes.

1. Become superuser.
2. Use the format command to change the disk partitioning for each drive in the disk set.

   When you repartition a drive, you must meet the following conditions to prevent the metaset(1M) command from repartitioning the drive.

   • Create slice 7 for VTOC or slice 6 for EFI starting at cylinder 0, large enough to hold a state database replica. See your Solaris Volume Manager administration guide to determine the size of a state database replica for your version of the volume-manager software.

   • Set the Flag field in the target slice to wu (read-write, unmountable). Do not set it to read-only.

   • Do not allow the target slice to overlap any other slice on the drive.

   See the format(1M) man page for details.

Next Steps

Define volumes by using an md.tab file. Go to How to Create an md.tab File.

How to Create an md.tab File

Create an /etc/lvm/md.tab file on each node in the cluster. Use the md.tab file to define Solaris Volume Manager volumes for the disk sets that you created.

1. Become superuser.
2. List the DID mappings for reference when you create your md.tab file.

   Use the full DID device names in the md.tab file in place of the lower-level device names (cN tXdY). The DID device name takes the form /dev/did/rdsk/dN.

   phys-schost# cldevice show | grep Device

   === DID Device Instances ===                   
   DID Device Name:                                /dev/did/rdsk/d1
     Full Device Path:                               phys-schost-1:/dev/rdsk/c0t0d0
   DID Device Name:                                /dev/did/rdsk/d2
     Full Device Path:                               phys-schost-1:/dev/rdsk/c0t6d0
   DID Device Name:                                /dev/did/rdsk/d3
     Full Device Path:                               phys-schost-1:/dev/rdsk/c1t1d0
     Full Device Path:                               phys-schost-2:/dev/rdsk/c1t1d0
   …

3. Create an /etc/lvm/md.tab file and edit it with your preferred text editor.

   ---

   Note - If you have existing data on the drives that will be used for the submirrors, you must back up the data before volume setup. Then restore the data onto the mirror.

   ---

   To avoid possible confusion between local volumes on different nodes in a cluster environment, use a naming scheme that makes each local volume name unique throughout the cluster. For example, for node 1 choose names from d100 to d199. And for node 2 use d200 to d299.

   See your Solaris Volume Manager documentation and the md.tab(4) man page for details about how to create an md.tab file.

Example 4-8 Sample md.tab File

The following sample md.tab file defines the disk set that is named dg-schost-1. The ordering of lines in the md.tab file is not important.

dg-schost-1/d0 -m dg-schost-1/d10 dg-schost-1/d20
    dg-schost-1/d10 1 1 /dev/did/rdsk/d1s0
    dg-schost-1/d20 1 1 /dev/did/rdsk/d2s0

The sample md.tab file is constructed as follows.

1. The first line defines the device d0 as a mirror of volumes d10 and d20. The -m signifies that this device is a mirror device.

   dg-schost-1/d0 -m dg-schost-1/d0 dg-schost-1/d20

2. The second line defines volume d10, the first submirror of d0, as a one-way stripe.

   dg-schost-1/d10 1 1 /dev/did/rdsk/d1s0

3. The third line defines volume d20, the second submirror of d0, as a one-way stripe.

   dg-schost-1/d20 1 1 /dev/did/rdsk/d2s0

Next Steps

Activate the volumes that are defined in the md.tab files. Go to How to Activate Volumes.

How to Activate Volumes

Perform this procedure to activate Solaris Volume Manager volumes that are defined in md.tab files.

1. Become superuser.
2. Ensure that md.tab files are located in the /etc/lvm directory.
3. Ensure that you have ownership of the disk set on the node where the command will be executed.
4. Take ownership of the disk set.

   phys-schost# cldevicegroup switch -n node devicegroup

   -n node

   Specifies the node that takes ownership.

   devicegroup

   Specifies the disk set name.

5. Activate the disk set's volumes, which are defined in the md.tab file.

   phys-schost# metainit -s setname -a

   -s setname

   Specifies the disk set name.

   -a

   Activates all volumes in the md.tab file.

6. Repeat Step 3 through Step 5 for each disk set in the cluster.

   If necessary, run the metainit(1M) command from another node that has connectivity to the drives. This step is required for cluster-pair topologies, where the drives are not accessible by all nodes.

7. Check the status of the volumes.

   phys-schost# metastat -s setname

   See the metastat(1M) man page for more information.

8. (Optional) Capture the disk partitioning information for future reference.

   phys-schost# prtvtoc /dev/rdsk/cNtXdYsZ > filename

   Store the file in a location outside the cluster. If you make any disk configuration changes, run this command again to capture the changed configuration. If a disk fails and needs replacement, you can use this information to restore the disk partition configuration. For more information, see the prtvtoc(1M) man page.

9. (Optional) Make a backup of your cluster configuration.

   An archived backup of your cluster configuration facilitates easier recovery of the your cluster configuration. For more information, see How to Back Up the Cluster Configuration in Oracle Solaris Cluster System Administration Guide.

Example 4-9 Activating Volumes in the md.tab File

In the following example, all volumes that are defined in the md.tab file for disk set dg-schost-1 are activated.

phys-schost# metainit -s dg-schost-1 -a

Next Steps

If your cluster contains disk sets that are configured with exactly two disk enclosures and two nodes, add dual-string mediators. Go to Configuring Dual-String Mediators.

Otherwise, go to How to Create Cluster File Systems to create a cluster file system.