Chapter 4 Configuring Solaris Volume Manager Software

Configure your local and multihost disks for Solaris Volume Manager software by using the procedures in this chapter, along with the planning information in Planning Volume Management. See your Solaris Volume Manager documentation for additional details.

The Enhanced Storage module of Solaris Management Console is not compatible with Sun Cluster software. Use the command-line interface or Sun Cluster utilities to configure Solaris Volume Manager software.

The following sections are in this chapter:

• Configuring Solaris Volume Manager Software

• Creating Disk Sets in a Cluster

• Configuring Dual-String Mediators

Configuring Solaris Volume Manager Software

The following table lists the tasks that you perform to configure Solaris Volume Manager software for Sun Cluster configurations. Complete the procedures in the order that is indicated.

SPARC: How to Set the Number of Volume Names and Disk Sets

This procedure is only required for the Solaris 9 OS. If the cluster runs on the Solaris 10 OS, proceed to How to Create State Database Replicas.

With the Solaris 10 release, Solaris Volume Manager has been enhanced to configure volumes dynamically. You no longer need to edit the nmd and the md_nsets parameters in the /kernel/drv/md.conf file. New volumes are dynamically created, as needed.

This procedure describes how to determine the number of Solaris Volume Manager volume names and disk sets that you need for your configuration. This procedure also describes how to modify the /kernel/drv/md.conf file to specify these numbers.

The default number of volume names per disk set is 128, but many configurations need more than the default. Increase this number before you implement a configuration, to save administration time later.

At the same time, keep the value of the nmdfield and the md_nsets field as low as possible. Memory structures exist for all possible devices as determined by nmdand md_nsets, even if you have not created those devices. For optimal performance, keep the value of nmd and md_nsets only slightly higher than the number of volumes that you plan to use.

Before You Begin

Have available the completed Device Group Configurations Worksheet.

1. Calculate the total number of disk sets that you expect to need in the cluster, then add one more disk set for private disk management.

   The cluster can have a maximum of 32 disk sets, 31 disk sets for general use plus one disk set for private disk management. The default number of disk sets is 4. You supply this value for the md_nsets field in Step 3.

2. Calculate the largest volume name that you expect to need for any disk set in the cluster.

   Each disk set can have a maximum of 8192 volume names. You supply this value for the nmd field in Step 3.

   1. Determine the quantity of volume names that you expect to need for each disk set.

      If you use local volumes, ensure that each local volume name on which a global-devices file system, /global/.devices/node@ nodeid, is mounted is unique throughout the cluster and does not use the same name as any device-ID name in the cluster.

      ---

      Tip –

      Choose a range of numbers to use exclusively for device-ID names and a range for each node to use exclusively for its local volume names. For example, device-ID names might use the range from d1 to d100. Local volumes on node 1 might use names in the range from d100 to d199. And local volumes on node 2 might use d200 to d299.

      ---

   2. Calculate the highest of the volume names that you expect to use in any disk set.

      The quantity of volume names to set is based on the volume name value rather than on the actual quantity . For example, if your volume names range from d950 to d1000, Solaris Volume Manager software requires that you set the value at 1000 names, not 50.

3. On each node, become superuser and edit the /kernel/drv/md.conf file.

   ---

   Caution –

   All cluster nodes (or cluster pairs in the cluster-pair topology) must have identical /kernel/drv/md.conf files, regardless of the number of disk sets served by each node. Failure to follow this guideline can result in serious Solaris Volume Manager errors and possible loss of data.

   ---

   1. Set the md_nsets field to the value that you determined in Step 1.

   2. Set the nmd field to the value that you determined in Step 2.

4. On each node, perform a reconfiguration reboot.

   phys-schost# touch /reconfigure phys-schost# shutdown -g0 -y -i6

   Changes to the /kernel/drv/md.conf file become operative after you perform a reconfiguration reboot.

Next Steps

Create local state database replicas. Go to How to Create State Database Replicas.

How to Create State Database Replicas

Perform this procedure on each node in the global cluster.

1. Become superuser.

2. Create state database replicas on one or more local devices for each cluster node.

   Use the physical name (cNtXdY sZ), not the device-ID name (dN), to specify the slices to use.

   phys-schost# metadb -af slice-1 slice-2 slice-3

   ---

   Tip –

   To provide protection of state data, which is necessary to run Solaris Volume Manager software, create at least three replicas for each node. Also, you can place replicas on more than one device to provide protection if one of the devices fails.

   ---

   See the metadb(1M) man page and your Solaris Volume Manager documentation for details.

3. Verify the replicas.

   phys-schost# metadb

   The metadb command displays the list of replicas.

Example 4–1 Creating State Database Replicas

The following example shows three state database replicas. Each replica is created on a different device.

phys-schost# metadb -af c0t0d0s7 c0t1d0s7 c1t0d0s7
phys-schost# metadb
flags            first blk      block count
    a       u       16          8192         /dev/dsk/c0t0d0s7
    a       u       16          8192         /dev/dsk/c0t1d0s7
    a       u       16          8192         /dev/dsk/c1t0d0s7

Next Steps

To mirror file systems on the root disk, go to Mirroring the Root Disk.

Otherwise, go to Creating Disk Sets in a Cluster to create Solaris Volume Manager disk sets.

Mirroring the Root Disk

Mirroring the root disk prevents the cluster node itself from shutting down because of a system disk failure. Four types of file systems can reside on the root disk. Each file-system type is mirrored by using a different method.

Use the following procedures to mirror each type of file system.

• How to Mirror the Root (/) File System

• How to Mirror the Global Devices Namespace

• How to Mirror File Systems Other Than Root (/) That Cannot Be Unmounted

• How to Mirror File Systems That Can Be Unmounted

For local disk mirroring, do not use /dev/global as the path when you specify the disk name. If you specify this path for anything other than cluster file systems, the system cannot boot.

How to Mirror the Root (/) File System

Use this procedure to mirror the root (/) file system.

If the global-devices namespace is on a lofi-created file, this procedure includes the mirroring of the global-devices namespace.

1. Become superuser.

2. Place the root slice in a single-slice (one-way) concatenation.

   Specify the physical disk name of the root-disk slice (cNtXdYsZ).

   phys-schost# metainit -f submirror1 1 1 root-disk-slice

3. Create a second concatenation.

   phys-schost# metainit submirror2 1 1 submirror-disk-slice

4. Create a one-way mirror with one submirror.

   phys-schost# metainit mirror -m submirror1

   ---

   Note –

   If the device is a local device to be used to mount a global-devices file system, /global/.devices/node@nodeid, the volume name for the mirror must be unique throughout the cluster.

   ---

5. Set up the system files for the root (/) directory.

   phys-schost# metaroot mirror

   This command edits the /etc/vfstab and /etc/system files so the system can be booted with the root (/) file system on a metadevice or volume. For more information, see the metaroot(1M) man page.

6. Flush all file systems.

   phys-schost# lockfs -fa

   This command flushes all transactions out of the log and writes the transactions to the master file system on all mounted UFS file systems. For more information, see the lockfs(1M) man page.

7. Move any resource groups or device groups from the node.

   phys-schost# clnode evacuate from-node

   from-node

   Specifies the name of the node from which to evacuate resource or device groups.

8. Reboot the node.

   This command remounts the newly mirrored root (/) file system.

   phys-schost# shutdown -g0 -y -i6

9. Attach the second submirror to the mirror.

   phys-schost# metattach mirror submirror2

   See the metattach(1M) man page for more information.

10. If the disk that is used to mirror the root disk is physically connected to more than one node (multihosted), disable fencing for that disk.

    Disabling fencing for the device prevents unintentional fencing of a node from its boot device if the boot device is connected to multiple nodes.

    phys-schost# cldevice set -p default_fencing=nofencing submirror-disk

    -p

    Specifies a device property.

    default_fencing=nofencing

    Disables fencing for the specified device.

    For more information about the default_fencing property, see the cldevice(1CL) man page.

11. Record the alternate boot path for possible future use.

    If the primary boot device fails, you can then boot from this alternate boot device. See Special Considerations for Mirroring root (/) in Solaris Volume Manager Administration Guide or Creating a RAID-1 Volume in Solaris Volume Manager Administration Guide for more information about alternate boot devices.

    phys-schost# ls -l /dev/rdsk/root-disk-slice

12. Repeat Step 1 through Step 11 on each remaining node of the cluster.

    Ensure that each volume name for a mirror on which a global-devices file system, /global/.devices/node@nodeid, is to be mounted is unique throughout the cluster.

Example 4–2 Mirroring the Root (/) File System

The following example shows the creation of mirror d0 on the node phys-schost-1, which consists of submirror d10 on partition c0t0d0s0 and submirror d20 on partition c2t2d0s0. Device c2t2d0 is a multihost disk, so fencing is disabled. The example also displays the alternate boot path for recording.

phys-schost# metainit -f d10 1 1 c0t0d0s0
d11: Concat/Stripe is setup
phys-schost# metainit d20 1 1 c2t2d0s0
d12: Concat/Stripe is setup
phys-schost# metainit d0 -m d10
d10: Mirror is setup
phys-schost# metaroot d0
phys-schost# lockfs -fa
phys-schost# clnode evacuate phys-schost-1
phys-schost# shutdown -g0 -y -i6
phys-schost# metattach d0 d20
d0: Submirror d20 is attachedphys-schost# cldevice set -p default_fencing=nofencing c2t2d0
phys-schost# ls -l /dev/rdsk/c2t2d0s0
lrwxrwxrwx  1 root     root          57 Apr 25 20:11 /dev/rdsk/c2t2d0s0 
–> ../../devices/node@1/pci@1f,0/pci@1/scsi@3,1/disk@2,0:a,raw

Next Steps

To mirror the global devices namespace, /global/.devices/node@nodeid, go to How to Mirror the Global Devices Namespace.

To mirror file systems than cannot be unmounted, go to How to Mirror File Systems Other Than Root (/) That Cannot Be Unmounted.

To mirror user-defined file systems, go to How to Mirror File Systems That Can Be Unmounted.

Otherwise, go to Creating Disk Sets in a Cluster to create a disk set.

Troubleshooting

Some of the steps in this mirroring procedure might cause an error message similar to metainit: dg-schost-1: d1s0: not a metadevice. Such an error message is harmless and can be ignored.

How to Mirror the Global Devices Namespace

Use this procedure to mirror the global devices namespace, /global/.devices/node@nodeid/.

Do not use this procedure if the global-devices namespace is on a lofi-based file. Instead, go to How to Mirror the Root (/) File System.

1. Become superuser.

2. Place the global devices namespace slice in a single-slice (one-way) concatenation.

   Use the physical disk name of the disk slice (cNtXdY sZ).

   phys-schost# metainit -f submirror1 1 1 diskslice

3. Create a second concatenation.

   phys-schost# metainit submirror2 1 1 submirror-diskslice

4. Create a one-way mirror with one submirror.

   phys-schost# metainit mirror -m submirror1

   ---

   Note –

   The volume name for a mirror on which a global-devices file system, /global/.devices/node@nodeid, is to be mounted must be unique throughout the cluster.

   ---

5. Attach the second submirror to the mirror.

   This attachment starts a synchronization of the submirrors.

   phys-schost# metattach mirror submirror2

6. Edit the /etc/vfstab file entry for the /global/.devices/node@nodeid file system.

   Replace the names in the device to mount and device to fsck columns with the mirror name.

   phys-schost# vi /etc/vfstab #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # /dev/md/dsk/mirror /dev/md/rdsk/mirror /global/.devices/node@nodeid ufs 2 no global

7. Repeat Step 1 through Step 6 on each remaining node of the cluster.

8. Wait for the synchronization of the mirrors, started in Step 5, to be completed.

   Use the metastat(1M) command to view mirror status and to verify that mirror synchronization is complete.

   phys-schost# metastat mirror

9. If the disk that is used to mirror the global devices namespace is physically connected to more than one node (multihosted), disable fencing for that disk.

   Disabling fencing for the device prevents unintentional fencing of a node from its boot device if the boot device is connected to multiple nodes.

   phys-schost# cldevice set -p default_fencing=nofencing submirror-disk

   -p

   Specifies a device property.

   default_fencing=nofencing

   Disables fencing for the specified device.

   For more information about the default_fencing property, see the cldevice(1CL) man page.

Example 4–3 Mirroring the Global Devices Namespace

The following example shows creation of mirror d101, which consists of submirror d111 on partition c0t0d0s3 and submirror d121 on partition c2t2d0s3. The /etc/vfstab file entry for /global/.devices/node@1 is updated to use the mirror name d101. Device c2t2d0 is a multihost disk, so fencing is disabled.

phys-schost# metainit -f d111 1 1 c0t0d0s3
d111: Concat/Stripe is setup
phys-schost# metainit d121 1 1 c2t2d0s3
d121: Concat/Stripe is setup
phys-schost# metainit d101 -m d111
d101: Mirror is setup
phys-schost# metattach d101 d121
d101: Submirror d121 is attached
phys-schost# vi /etc/vfstab
#device        device        mount    FS     fsck    mount    mount
#to mount      to fsck       point    type   pass    at boot  options
#
/dev/md/dsk/d101 /dev/md/rdsk/d101 /global/.devices/node@1 ufs 2 no global
phys-schost# metastat d101
d101: Mirror
      Submirror 0: d111
         State: Okay
      Submirror 1: d121
         State: Resyncing
      Resync in progress: 15 % done
…
phys-schost# cldevice show phys-schost-3:/dev/rdsk/c2t2d0 
=== DID Device Instances ===                   

DID Device Name:                                /dev/did/rdsk/d2
  Full Device Path:                               phys-schost-1:/dev/rdsk/c2t2d0
  Full Device Path:                               phys-schost-3:/dev/rdsk/c2t2d0
…

phys-schost# cldevicegroup show | grep dsk/d2
Device Group Name:                              dsk/d2
…
  Node List:                                      phys-schost-1, phys-schost-3
…
  localonly:                                      false
phys-schost# cldevicegroup remove-node -n phys-schost-3 dsk/d2
phys-schost# cldevice set -p default_fencing=nofencing c2t2d0

Next Steps

To mirror file systems other than root (/) that cannot be unmounted, go to How to Mirror File Systems Other Than Root (/) That Cannot Be Unmounted.

To mirror user-defined file systems, go to How to Mirror File Systems That Can Be Unmounted

Otherwise, go to Creating Disk Sets in a Cluster to create a disk set.

Troubleshooting

Some of the steps in this mirroring procedure might cause an error message similar to metainit: dg-schost-1: d1s0: not a metadevice. Such an error message is harmless and can be ignored.

How to Mirror File Systems Other Than Root (/) That Cannot Be Unmounted

Use this procedure to mirror file systems other than root (/) that cannot be unmounted during normal system usage, such as /usr, /opt, or swap.

1. Become superuser.

2. Place the slice on which an unmountable file system resides in a single-slice (one-way) concatenation.

   Specify the physical disk name of the disk slice (cNtX dYsZ).

   phys-schost# metainit -f submirror1 1 1 diskslice

3. Create a second concatenation.

   phys-schost# metainit submirror2 1 1 submirror-diskslice

4. Create a one-way mirror with one submirror.

   phys-schost# metainit mirror -m submirror1

   ---

   Note –

   The volume name for this mirror does not need to be unique throughout the cluster.

   ---

5. Repeat Step 1 through Step 4 for each remaining unmountable file system that you want to mirror.

6. On each node, edit the /etc/vfstab file entry for each unmountable file system you mirrored.

   Replace the names in the device to mount and device to fsck columns with the mirror name.

   phys-schost# vi /etc/vfstab #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # /dev/md/dsk/mirror /dev/md/rdsk/mirror /filesystem ufs 2 no global

7. Move any resource groups or device groups from the node.

   phys-schost# clnode evacuate from-node

   from-node

   Specifies the name of the node from which to move resource or device groups.

8. Reboot the node.

   phys-schost# shutdown -g0 -y -i6

9. Attach the second submirror to each mirror.

   This attachment starts a synchronization of the submirrors.

   phys-schost# metattach mirror submirror2

10. Wait for the synchronization of the mirrors, started in Step 9, to complete.

    Use the metastat(1M) command to view mirror status and to verify that mirror synchronization is complete.

    phys-schost# metastat mirror

11. If the disk that is used to mirror the unmountable file system is physically connected to more than one node (multihosted), disable fencing for that disk.

    Disabling fencing for the device prevents unintentional fencing of a node from its boot device if the boot device is connected to multiple nodes.

    phys-schost# cldevice set -p default_fencing=nofencing submirror-disk

    -p

    Specifies a device property.

    default_fencing=nofencing

    Disables fencing for the specified device.

    For more information about the default_fencing property, see the cldevice(1CL) man page.

Example 4–4 Mirroring File Systems That Cannot Be Unmounted

The following example shows the creation of mirror d1 on the node phys-schost-1 to mirror /usr, which resides on c0t0d0s1. Mirror d1 consists of submirror d11 on partition c0t0d0s1 and submirror d21 on partition c2t2d0s1. The /etc/vfstab file entry for /usr is updated to use the mirror name d1. Device c2t2d0 is a multihost disk, so fencing is disabled.

phys-schost# metainit -f d11 1 1 c0t0d0s1
d11: Concat/Stripe is setup
phys-schost# metainit d21 1 1 c2t2d0s1
d21: Concat/Stripe is setup
phys-schost# metainit d1 -m d11
d1: Mirror is setup
phys-schost# vi /etc/vfstab
#device        device        mount    FS     fsck    mount    mount
#to mount      to fsck       point    type   pass    at boot  options
#
/dev/md/dsk/d1 /dev/md/rdsk/d1 /usr ufs  2       no global
…
phys-schost# clnode evacuate phys-schost-1
phys-schost# shutdown -g0 -y -i6
phys-schost# metattach d1 d21
d1: Submirror d21 is attached
phys-schost# metastat d1
d1: Mirror
      Submirror 0: d11
         State: Okay
      Submirror 1: d21
         State: Resyncing
      Resync in progress: 15 % done
…
phys-schost# cldevice show phys-schost-3:/dev/rdsk/c2t2d0
…
DID Device Name:                                /dev/did/rdsk/d2
phys-schost# cldevicegroup show dsk/d2
Device Group Name:                              dsk/d2
…
  Node List:                                      phys-schost-1, phys-schost-3
…
  localonly:                                      false
phys-schost# cldevicegroup remove-node -n phys-schost-3 dsk/d2
phys-schost# cldevice set -p default_fencing=nofencing c2t2d0

Next Steps

To mirror user-defined file systems, go to How to Mirror File Systems That Can Be Unmounted.

Otherwise, go to Creating Disk Sets in a Cluster to create a disk set.

Troubleshooting

Some of the steps in this mirroring procedure might cause an error message similar to metainit: dg-schost-1: d1s0: not a metadevice. Such an error message is harmless and can be ignored.

How to Mirror File Systems That Can Be Unmounted

Use this procedure to mirror user-defined file systems that can be unmounted. In this procedure, the nodes do not need to be rebooted.

1. Become superuser.

2. Unmount the file system to mirror.

   Ensure that no processes are running on the file system.

   phys-schost# umount /mount-point

   See the umount(1M) man page and Chapter 18, Mounting and Unmounting File Systems (Tasks), in System Administration Guide: Devices and File Systems for more information.

3. Place in a single-slice (one-way) concatenation the slice that contains a user-defined file system that can be unmounted.

   Specify the physical disk name of the disk slice (cNtX dYsZ).

   phys-schost# metainit -f submirror1 1 1 diskslice

4. Create a second concatenation.

   phys-schost# metainit submirror2 1 1 submirror-diskslice

5. Create a one-way mirror with one submirror.

   phys-schost# metainit mirror -m submirror1

   ---

   Note –

   The volume name for this mirror does not need to be unique throughout the cluster.

   ---

6. Repeat Step 1 through Step 5 for each mountable file system to be mirrored.

7. On each node, edit the /etc/vfstab file entry for each file system you mirrored.

   Replace the names in the device to mount and device to fsck columns with the mirror name.

   phys-schost# vi /etc/vfstab #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # /dev/md/dsk/mirror /dev/md/rdsk/mirror /filesystem ufs 2 no global

8. Attach the second submirror to the mirror.

   This attachment starts a synchronization of the submirrors.

   phys-schost# metattach mirror submirror2

9. Wait for the synchronization of the mirrors, started in Step 8, to be completed.

   Use the metastat(1M) command to view mirror status.

   phys-schost# metastat mirror

10. If the disk that is used to mirror the user-defined file system is physically connected to more than one node (multihosted), disable fencing for that disk.

    Disabling fencing for the device prevents unintentional fencing of a node from its boot device if the boot device is connected to multiple nodes.

    phys-schost# cldevice set -p default_fencing=nofencing submirror-disk

    -p

    Specifies a device property.

    default_fencing=nofencing

    Disables fencing for the specified device.

    For more information about the default_fencing property, see the cldevice(1CL) man page.

11. Mount the mirrored file system.

    phys-schost# mount /mount-point

    See the mount(1M) man page and Chapter 18, Mounting and Unmounting File Systems (Tasks), in System Administration Guide: Devices and File Systems for more information.

Example 4–5 Mirroring File Systems That Can Be Unmounted

The following example shows creation of mirror d4 to mirror /export, which resides on c0t0d0s4. Mirror d4 consists of submirror d14 on partition c0t0d0s4 and submirror d24 on partition c2t2d0s4. The /etc/vfstab file entry for /export is updated to use the mirror name d4. Device c2t2d0 is a multihost disk, so fencing is disabled.

phys-schost# umount /export
phys-schost# metainit -f d14 1 1 c0t0d0s4
d14: Concat/Stripe is setup
phys-schost# metainit d24 1 1 c2t2d0s4
d24: Concat/Stripe is setup
phys-schost# metainit d4 -m d14
d4: Mirror is setup
phys-schost# vi /etc/vfstab
#device        device        mount    FS     fsck    mount    mount
#to mount      to fsck       point    type   pass    at boot  options
#
# /dev/md/dsk/d4 /dev/md/rdsk/d4 /export ufs 2 no    global
phys-schost# metattach d4 d24
d4: Submirror d24 is attached
phys-schost# metastat d4
d4: Mirror
       Submirror 0: d14
          State: Okay
       Submirror 1: d24
          State: Resyncing
       Resync in progress: 15 % done
…
phys-schost# cldevice show phys-schost-3:/dev/rdsk/c2t2d0
…
DID Device Name:                                /dev/did/rdsk/d2
phys-schost# cldevicegroup show dsk/d2
Device Group Name:                              dsk/d2
…
  Node List:                                      phys-schost-1, phys-schost-2
…
  localonly:                                      false
phys-schost# cldevicegroup remove-node -n phys-schost-3 dsk/d2
phys-schost# cldevice set -p default_fencing=nofencing c2t2d0 
phys-schost# mount /export

Next Steps

To create a disk set, go to Creating Disk Sets in a Cluster. Alternatively, if you will create a multi-owner disk set for use by Oracle Real Application Clusters, go to How to Create a Multi-Owner Disk Set in Solaris Volume Manager for Sun Cluster for the Oracle RAC Database in Sun Cluster Data Service for Oracle RAC Guide for Solaris OS.

If you have sufficient disk sets for your needs, go to one of the following:

• If your cluster contains disk sets that are configured with exactly two disk enclosures and two nodes, you must add dual-string mediators. Go to Configuring Dual-String Mediators.

• If your cluster configuration does not require dual-string mediators, go to How to Create Cluster File Systems.

Troubleshooting

Some of the steps in this mirroring procedure might cause an error message that is similar to metainit: dg-schost-1: d1s0: not a metadevice. Such an error message is harmless and can be ignored.

Creating Disk Sets in a Cluster

This section describes how to create disk sets for a cluster configuration. When you create a Solaris Volume Manager disk set in a Sun Cluster environment, the disk set is automatically registered with the Sun Cluster software as a device group of type svm. To create or delete an svm device group, you must use Solaris Volume Manager commands and utilities to create or delete the underlying disk set of the device group.

The following table lists the tasks that you perform to create disk sets. Complete the procedures in the order that is indicated.

How to Create a Disk Set

Perform this procedure to create disk sets.

1. SPARC: (Solaris 9) Determine whether, after you create the new disk sets, the global cluster will have more than three disk sets.

   • If the cluster will have no more than three disk sets, skip to Step 9.

   • If the cluster will have four or more disk sets, proceed to Step 2 to prepare the cluster. You must perform this task whether you are installing disk sets for the first time or whether you are adding more disk sets to a fully configured cluster.

   • If the cluster runs on the Solaris 10 OS, Solaris Volume Manager automatically makes the necessary configuration changes. Skip to Step 9.

2. On any node of the cluster, check the value of the md_nsets variable in the /kernel/drv/md.conf file.

3. If the total number of disk sets in the cluster will be greater than the existing value of md_nsets minus one, increase the value of md_nsets to the desired value.

   The maximum permissible number of disk sets is one less than the configured value of md_nsets. The maximum possible value of md_nsets is 32, therefore the maximum permissible number of disk sets that you can create is 31.

4. Ensure that the /kernel/drv/md.conf file is identical on each node of the cluster.

   ---

   Caution –

   Failure to follow this guideline can result in serious Solaris Volume Manager errors and possible loss of data.

   ---

5. If you made changes to the md.conf file on any node, perform the following steps to make those changes active.

   1. On one node, become superuser.

   2. From one node, shut down the cluster.

      phys-schost# cluster shutdown -g0 -y

   3. Reboot each node of the cluster.

      • On SPARC based systems, do the following:

        ok boot

      • On x86 based systems, do the following:

        When the GRUB menu is displayed, select the appropriate Solaris entry and press Enter. The GRUB menu appears similar to the following:

        GNU GRUB version 0.95 (631K lower / 2095488K upper memory) +-------------------------------------------------------------------------+ | Solaris 10 /sol_10_x86 | | Solaris failsafe | | | +-------------------------------------------------------------------------+ Use the ^ and v keys to select which entry is highlighted. Press enter to boot the selected OS, 'e' to edit the commands before booting, or 'c' for a command-line.

        For more information about GRUB based booting, see Chapter 11, GRUB Based Booting (Tasks), in System Administration Guide: Basic Administration.

6. On each node in the cluster, run the devfsadm(1M) command.

   You can run this command on all nodes in the cluster at the same time.

7. From one node of the cluster, update the global-devices namespace.

   phys-schost# cldevice populate

   See the cldevice(1CL) man page for more information.

8. On each node, verify that the command has completed processing before you attempt to create any disk sets.

   The command executes remotely on all nodes, even though the command is run from just one node. To determine whether the command has completed processing, run the following command on each node of the cluster.

   phys-schost# ps -ef | grep scgdevs

9. Ensure that the disk set that you intend to create meets one of the following requirements.

   • If the disk set is configured with exactly two disk strings, the disk set must connect to exactly two nodes and use exactly two mediator hosts. These mediator hosts must be the same two hosts used for the disk set. See Configuring Dual-String Mediators for details on how to configure dual-string mediators.

   • If the disk set is configured with more than two disk strings, ensure that for any two disk strings S1 and S2, the sum of the number of drives on those strings exceeds the number of drives on the third string S3. Stated as a formula, the requirement is that count(S1) + count(S2) > count(S3).

10. Ensure that the local state database replicas exist.

    For instructions, see How to Create State Database Replicas.

11. Become superuser on the cluster node that will master the disk set.

12. Create the disk set.

    The following command creates the disk set and registers the disk set as a Sun Cluster device group.

    phys-schost# metaset -s setname -a -h node1 node2

    -s setname

    Specifies the disk set name.

    -a

    Adds (creates) the disk set.

    -h node1

    Specifies the name of the primary node to master the disk set.

    node2

    Specifies the name of the secondary node to master the disk set

    ---

    Note –

    When you run the metaset command to configure a Solaris Volume Manager device group on a cluster, the command designates one secondary node by default. You can change the desired number of secondary nodes in the device group by using the clsetup utility after the device group is created. Refer to Administering Device Groups in Sun Cluster System Administration Guide for Solaris OS for more information about how to change the numsecondaries property.

    ---

13. If you are configuring a replicated Solstice DiskSuite or Solaris Volume Manager device group, set the replication property for the device group.

    phys-schost# cldevicegroup sync device-group-name

    For more information about data replication, see Chapter 4, Data Replication Approaches, in Sun Cluster System Administration Guide for Solaris OS.

14. Verify the status of the new disk set.

    phys-schost# metaset -s setname

15. As needed, set device group properties.

    phys-schost# cldevicegroup set -p name=value devicegroup

    -p

    Specifies a device-group property.

    name

    Specifies the name of a property.

    value

    Specifies the value or setting of the property.

    devicegroup

    Specifies the name of the device group. The device-group name is the same as the disk-set name.

    See the cldevicegroup(1CL) for information about device-group properties.

Example 4–6 Creating a Disk Set

The following command creates two disk sets, dg-schost-1 and dg-schost-2, with the nodes phys-schost-1 and phys-schost-2 specified as the potential primaries.

phys-schost# metaset -s dg-schost-1 -a -h phys-schost-1 phys-schost-2
phys-schost# metaset -s dg-schost-2 -a -h phys-schost-1 phys-schost-2

Next Steps

Add drives to the disk set. Go to Adding Drives to a Disk Set.

Adding Drives to a Disk Set

When you add a drive to a disk set, the volume management software repartitions the drive as follows so that the state database for the disk set can be placed on the drive.

• A small portion of each drive is reserved for use by Solaris Volume Manager software. In volume table of contents (VTOC) labeled devices, slice 7 is used. In Extensible Firmware Interface (EFI) labeled devices, slice 6 is used. The remainder of the space on each drive is placed into slice 0.

• Drives are repartitioned when they are added to the disk set only if the target slice is not configured correctly.

• Any existing data on the drives is lost by the repartitioning.

• If the target slice starts at cylinder 0, and the drive partition is large enough to contain a state database replica, the drive is not repartitioned.

How to Add Drives to a Disk Set

Before You Begin

Ensure that the disk set has been created. For instructions, see How to Create a Disk Set.

1. Become superuser.

2. List the DID mappings.

   phys-schost# cldevice show | grep Device

   • Choose drives that are shared by the cluster nodes that will master or potentially master the disk set.

   • Use the full DID device name, which has the form /dev/did/rdsk/dN,when you add a drive to a disk set.

   In the following example, the entries for DID device /dev/did/rdsk/d3 indicate that the drive is shared by phys-schost-1 and phys-schost-2.

   === DID Device Instances === DID Device Name: /dev/did/rdsk/d1 Full Device Path: phys-schost-1:/dev/rdsk/c0t0d0 DID Device Name: /dev/did/rdsk/d2 Full Device Path: phys-schost-1:/dev/rdsk/c0t6d0 DID Device Name: /dev/did/rdsk/d3 Full Device Path: phys-schost-1:/dev/rdsk/c1t1d0 Full Device Path: phys-schost-2:/dev/rdsk/c1t1d0 …

3. Become owner of the disk set.

   phys-schost# cldevicegroup switch -n node devicegroup

   -n node

   Specifies the node to take ownership of the device group.

   devicegroup

   Specifies the device group name, which is the same as the disk set name.

4. Add the drives to the disk set.

   Use the full DID path name.

   phys-schost# metaset -s setname -a /dev/did/rdsk/dN

   -s setname

   Specifies the disk set name, which is the same as the device group name.

   -a

   Adds the drive to the disk set.

   ---

   Note –

   Do not use the lower-level device name (cNtXdY) when you add a drive to a disk set. Because the lower-level device name is a local name and not unique throughout the cluster, using this name might prevent the metaset from being able to switch over.

   ---

5. Verify the status of the disk set and drives.

   phys-schost# metaset -s setname

Example 4–7 Adding Drives to a Disk Set

The metaset command adds the drives /dev/did/rdsk/d1 and /dev/did/rdsk/d2 to the disk set dg-schost-1.

phys-schost# metaset -s dg-schost-1 -a /dev/did/rdsk/d1 /dev/did/rdsk/d2

Next Steps

To repartition drives for use in volumes, go to How to Repartition Drives in a Disk Set.

Otherwise, go to How to Create an md.tab File to define metadevices or volumes by using an md.tab file.

How to Repartition Drives in a Disk Set

The metaset(1M) command repartitions drives in a disk set so that a small portion of each drive is reserved for use by Solaris Volume Manager software. In volume table of contents (VTOC) labeled devices, slice 7 is used. In Extensible Firmware Interface (EFI) labeled devices, slice 6 is used. The remainder of the space on each drive is placed into slice 0. To make more effective use of the drive, use this procedure to modify the disk layout. If you allocate space to VTOC slices 1 through 6 or EFI slices 1 through 5, you can use these slices when you set up Solaris Volume Manager volumes.

1. Become superuser.

2. Use the format command to change the disk partitioning for each drive in the disk set.

   When you repartition a drive, you must meet the following conditions to prevent the metaset(1M) command from repartitioning the drive.

   • Create slice 7 for VTOC or slice 6 for EFI starting at cylinder 0, large enough to hold a state database replica. See your Solaris Volume Manager administration guide to determine the size of a state database replica for your version of the volume-manager software.

   • Set the Flag field in the target slice to wu (read-write, unmountable). Do not set it to read-only.

   • Do not allow the target slice to overlap any other slice on the drive.

   See the format(1M) man page for details.

Next Steps

Define volumes by using an md.tab file. Go to How to Create an md.tab File.

How to Create an md.tab File

Create an /etc/lvm/md.tab file on each node in the cluster. Use the md.tab file to define Solaris Volume Manager volumes for the disk sets that you created.

If you are using local volumes, ensure that local volume names are distinct from the device-ID names that are used to form disk sets. For example, if the device-ID name /dev/did/dsk/d3 is used in a disk set, do not use the name /dev/md/dsk/d3 for a local volume. This requirement does not apply to shared volumes, which use the naming convention /dev/md/setname/{r}dsk/d#.

1. Become superuser.

2. List the DID mappings for reference when you create your md.tab file.

   Use the full DID device names in the md.tab file in place of the lower-level device names (cN tXdY). The DID device name takes the form /dev/did/rdsk/dN.

   phys-schost# cldevice show | grep Device

   === DID Device Instances === DID Device Name: /dev/did/rdsk/d1 Full Device Path: phys-schost-1:/dev/rdsk/c0t0d0 DID Device Name: /dev/did/rdsk/d2 Full Device Path: phys-schost-1:/dev/rdsk/c0t6d0 DID Device Name: /dev/did/rdsk/d3 Full Device Path: phys-schost-1:/dev/rdsk/c1t1d0 Full Device Path: phys-schost-2:/dev/rdsk/c1t1d0 …

3. Create an /etc/lvm/md.tab file and edit it with your preferred text editor.

   ---

   Note –

   If you have existing data on the drives that will be used for the submirrors, you must back up the data before volume setup. Then restore the data onto the mirror.

   ---

   To avoid possible confusion between local volumes on different nodes in a cluster environment, use a naming scheme that makes each local volume name unique throughout the cluster. For example, for node 1 choose names from d100 to d199. And for node 2 use d200 to d299.

   See your Solaris Volume Manager documentation and the md.tab(4) man page for details about how to create an md.tab file.

Example 4–8 Sample md.tab File

The following sample md.tab file defines the disk set that is named dg-schost-1. The ordering of lines in the md.tab file is not important.

dg-schost-1/d0 -m dg-schost-1/d10 dg-schost-1/d20
    dg-schost-1/d10 1 1 /dev/did/rdsk/d1s0
    dg-schost-1/d20 1 1 /dev/did/rdsk/d2s0

The sample md.tab file is constructed as follows.

1. The first line defines the device d0 as a mirror of volumes d10 and d20. The -m signifies that this device is a mirror device.

   dg-schost-1/d0 -m dg-schost-1/d0 dg-schost-1/d20

2. The second line defines volume d10, the first submirror of d0, as a one-way stripe.

   dg-schost-1/d10 1 1 /dev/did/rdsk/d1s0

3. The third line defines volume d20, the second submirror of d0, as a one-way stripe.

   dg-schost-1/d20 1 1 /dev/did/rdsk/d2s0

Next Steps

Activate the volumes that are defined in the md.tab files. Go to How to Activate Volumes.

How to Activate Volumes

Perform this procedure to activate Solaris Volume Manager volumes that are defined in md.tab files.

1. Become superuser.

2. Ensure that md.tab files are located in the /etc/lvm directory.

3. Ensure that you have ownership of the disk set on the node where the command will be executed.

4. Take ownership of the disk set.

   phys-schost# cldevicegroup switch -n node devicegroup

   -n node

   Specifies the node that takes ownership.

   devicegroup

   Specifies the disk set name.

5. Activate the disk set's volumes, which are defined in the md.tab file.

   phys-schost# metainit -s setname -a

   -s setname

   Specifies the disk set name.

   -a

   Activates all volumes in the md.tab file.

6. Repeat Step 3 through Step 5 for each disk set in the cluster.

   If necessary, run the metainit(1M) command from another node that has connectivity to the drives. This step is required for cluster-pair topologies, where the drives are not accessible by all nodes.

7. Check the status of the volumes.

   phys-schost# metastat -s setname

   See the metastat(1M) man page for more information.

8. (Optional) Capture the disk partitioning information for future reference.

   phys-schost# prtvtoc /dev/rdsk/cNtXdYsZ > filename

   Store the file in a location outside the cluster. If you make any disk configuration changes, run this command again to capture the changed configuration. If a disk fails and needs replacement, you can use this information to restore the disk partition configuration. For more information, see the prtvtoc(1M) man page.

9. (Optional) Make a backup of your cluster configuration.

   An archived backup of your cluster configuration facilitates easier recovery of the your cluster configuration. For more information, see How to Back Up the Cluster Configuration in Sun Cluster System Administration Guide for Solaris OS.

Example 4–9 Activating Volumes in the md.tab File

In the following example, all volumes that are defined in the md.tab file for disk set dg-schost-1 are activated.

phys-schost# metainit -s dg-schost-1 -a

Next Steps

If your cluster contains disk sets that are configured with exactly two disk enclosures and two nodes, add dual-string mediators. Go to Configuring Dual-String Mediators.

Otherwise, go to How to Create Cluster File Systems to create a cluster file system.

Configuring Dual-String Mediators

This section provides information and procedures to configure dual-string mediator hosts. Dual-string mediators are required for all Solaris Volume Manager disk sets that are configured with exactly two disk strings and two cluster nodes. The use of mediators enables the Sun Cluster software to ensure that the most current data is presented in the instance of a single-string failure in a dual-string configuration.

A dual-string mediator, or mediator host, is a cluster node that stores mediator data. Mediator data provides information about the location of other mediators and contains a commit count that is identical to the commit count that is stored in the database replicas. This commit count is used to confirm that the mediator data is in sync with the data in the database replicas.

A disk string consists of a disk enclosure, its physical drives, cables from the enclosure to the node or nodes, and the interface adapter cards.

The following table lists the tasks that you perform to configure dual-string mediator hosts. complete the procedures in the order that is indicated.

Requirements for Dual-String Mediators

The following rules apply to dual-string configurations that use mediators.

• Disk sets must be configured with exactly two mediator hosts. Those two mediator hosts must be the same two cluster nodes that are used for the disk set.

• A disk set cannot have more than two mediator hosts.

• Mediators cannot be configured for disk sets that do not meet the two-string and two-host criteria.

These rules do not require that the entire cluster must have exactly two nodes. Rather, only those disk sets that have two disk strings must be connected to exactly two nodes. An N+1 cluster and many other topologies are permitted under these rules.

How to Add Mediator Hosts

Perform this procedure if your configuration requires dual-string mediators.

1. Become superuser on the node that currently masters the disk set to which you intend to add mediator hosts.

2. Add each node with connectivity to the disk set as a mediator host for that disk set.

   phys-schost# metaset -s setname -a -m mediator-host-list

   -s setname

   Specifies the disk set name.

   -a

   Adds to the disk set.

   -m mediator-host-list

   Specifies the name of the node to add as a mediator host for the disk set.

   See the mediator(7D) man page for details about mediator-specific options to the metaset command.

Example 4–10 Adding Mediator Hosts

The following example adds the nodes phys-schost-1 and phys-schost-2 as mediator hosts for the disk set dg-schost-1. Both commands are run from the node phys-schost-1.

phys-schost# metaset -s dg-schost-1 -a -m phys-schost-1
phys-schost# metaset -s dg-schost-1 -a -m phys-schost-2

Next Steps

Check the status of mediator data. Go to How to Check the Status of Mediator Data.

How to Check the Status of Mediator Data

Before You Begin

Ensure that you have added mediator hosts as described in How to Add Mediator Hosts.

1. Display the status of the mediator data.

   phys-schost# medstat -s setname

   -s setname

   Specifies the disk set name.

   See the medstat(1M) man page for more information.

2. If Bad is the value in the Status field of the medstat output, repair the affected mediator host.

   Go to How to Fix Bad Mediator Data.

Next Steps

Go to How to Create Cluster File Systems to create a cluster file system.

How to Fix Bad Mediator Data

Perform this procedure to repair bad mediator data.

1. Identify all mediator hosts with bad mediator data as described in the procedure How to Check the Status of Mediator Data.

2. Become superuser on the node that owns the affected disk set.

3. Remove all mediator hosts with bad mediator data from all affected disk sets.

   phys-schost# metaset -s setname -d -m mediator-host-list

   -s setname

   Specifies the disk set name.

   -d

   Deletes from the disk set.

   -m mediator-host-list

   Specifies the name of the node to remove as a mediator host for the disk set.

4. Restore each mediator host that you removed in Step 3.

   phys-schost# metaset -s setname -a -m mediator-host-list

   -a

   Adds to the disk set.

   -m mediator-host-list

   Specifies the name of the node to add as a mediator host for the disk set.

   See the mediator(7D) man page for details about mediator-specific options to the metaset command.

Next Steps

Determine from the following list the next task to perform that applies to your cluster configuration. If you need to perform more than one task from this list, go to the first of those tasks in this list.

• To create cluster file systems, go to How to Create Cluster File Systems.

• To create non-global zones on a node, go to How to Create a Non-Global Zone on a Global-Cluster Node.

• SPARC: To configure Sun Management Center to monitor the cluster, go to SPARC: Installing the Sun Cluster Module for Sun Management Center.

• Install third-party applications, register resource types, set up resource groups, and configure data services. See the documentation that is supplied with the application software and the Sun Cluster Data Services Planning and Administration Guide for Solaris OS.