Sun Cluster 3.0 U1 Installation Guide

Installing and Configuring Solstice DiskSuite Software

Before you begin, have available the following information.

Mappings of your storage disk drives.
The following completed configuration planning worksheets from the Sun Cluster 3.0 Release Notes. See "Planning Volume Management" for planning guidelines.
- "Local File System Layout Worksheet"
- "Disk Device Group Configurations Worksheet"
- "Volume Manager Configurations Worksheet"
- "Metadevices Worksheet (Solstice DiskSuite)"

The following table lists the tasks to perform to install and configure Solstice DiskSuite software for Sun Cluster configurations.

Note -

If you used SunPlex Manager to install Solstice DiskSuite software, the procedures "How to Install Solstice DiskSuite Software" through "How to Create Metadevice State Database Replicas" are already completed.

Table A-1 Task Map: Installing and Configuring Solstice DiskSuite Software


Task	For Instructions, Go To ...
Plan the layout of your Solstice DiskSuite configuration.	"Planning Volume Management" "Solstice DiskSuite Configuration Example"
Install Solstice DiskSuite software.	"How to Install Solstice DiskSuite Software"
Calculate the number of metadevice names and disksets needed for your configuration, and modify the `/kernel/drv/md.conf` file.	"How to Set the Number of Metadevice Names and Disksets"
Create metadevice state database replicas on the local disks.	"How to Create Metadevice State Database Replicas"
(Optional) Mirror file systems on the root disk.	"Mirroring the Root Disk"
Create disksets by using the `metaset` command.	"How to Create a Diskset"
Add disk drives to the disksets.	"How to Add Drives to a Diskset"
Repartition drives in a diskset to allocate space to slices 1 through 6.	"How to Repartition Drives in a Diskset"
List device ID pseudo-driver mappings and define metadevices in the `/etc/lvm/md.tab` files.	"How to Create an `md.tab` File"
Initialize the `md.tab` files.	"How to Activate Metadevices"
For dual-string configurations, configure mediator hosts, check the status of mediator data, and, if necessary, fix bad mediator data.	"How to Add Mediator Hosts" "How to Check the Status of Mediator Data" "How to Fix Bad Mediator Data"
Configure the cluster.	"Configuring the Cluster"

How to Install Solstice DiskSuite Software

Note -

If you used SunPlex Manager to install Solstice DiskSuite software, do not perform this procedure. Instead, proceed to "Mirroring the Root Disk".

Perform this task on each node in the cluster.

Become superuser on the cluster node.

If you install from the CD-ROM, insert the Solaris 8 Software 2 of 2 CD-ROM into the CD-ROM drive on the node.

This step assumes that the Volume Management daemon vold(1M) is running and configured to manage CD-ROM devices.

Install the Solstice DiskSuite software packages.

Note -
If you have Solstice DiskSuite software patches to install, do not reboot after you install the Solstice DiskSuite software.

Install software packages in the order shown in the following example.
# cd /cdrom/sol_8_sparc_2/Solaris_8/EA/products/DiskSuite_4.2.1/sparc/Packages # pkgadd -d . SUNWmdr SUNWmdu [SUNWmdx] optional-pkgs
The SUNWmdr and SUNWmdu packages are required for all Solstice DiskSuite installations. The SUNWmdx package is also required for the 64-bit Solstice DiskSuite installation.

See your Solstice DiskSuite installation documentation for information about optional software packages.

If you installed from a CD-ROM, eject the CD-ROM.

Install any Solstice DiskSuite patches.

See the Sun Cluster 3.0 U1 Release Notes for the location of patches and installation instructions.

Repeat Step 1 through Step 5 on the other nodes of the cluster.

From one node of the cluster, manually populate the global device namespace for Solstice DiskSuite.
# scgdevs

Set the number of metadevice names and disksets expected in the cluster.

Go to "How to Set the Number of Metadevice Names and Disksets".

How to Set the Number of Metadevice Names and Disksets

Note -

If you used SunPlex Manager to install Solstice DiskSuite software, do not perform this procedure. Instead, proceed to "Mirroring the Root Disk".

This procedure describes how to calculate the quantity of metadevice names needed for your configuration, and how to modify the /kernel/drv/md.conf file.

Tip -

The default number of metadevice names per diskset is 128, but many configurations need more than the default. Increase this number before you implement a configuration, to save administration time later.

Calculate the largest metadevice name you need for any diskset in the cluster.

Each diskset can have a maximum of 8192 metadevice names. You will supply this calculated value for the nmd field.
1. Calculate the quantity of metadevice names you need for each diskset.
  
  If you use local metadevices, ensure that each local metadevice name is unique throughout the cluster and does not use the same name as any device ID (DID) in the cluster.
  
  Tip -
  Choose a range of numbers to use exclusively for DID names and a range for each node to use exclusively for its local metadevice names. For example, DIDs would use names in the range from d1 to d1000, local metadevices on node 1 would use names in the range from d1100 to d1199, local metadevices on node 2 would use d1200 to d1299, and so on.
2. Determine the largest of the metadevice names to be used in any diskset.
  
  The quantity of metadevice names to set is based on the metadevice name value rather than on the actual quantity. For example, if your metadevice names range from d950 to d1000, Solstice DiskSuite software requires 1000 names, not 50.

Calculate the total expected number of disksets in the cluster, then add one for private disk management.

The cluster can have a maximum of 32 disksets. The default number of disksets is 4. You will supply this calculated value for the md_nsets field.

On each node, edit the /kernel/drv/md.conf file.

Caution -
All cluster nodes (or cluster pairs in the cluster-pair topology) must have identical /kernel/drv/md.conf files, regardless of the number of disksets served by each node. Failure to follow this guideline can result in serious Solstice DiskSuite errors and possible loss of data.
1. If the calculated quantity of metadevice names exceeds 128, set the nmd field to the quantity calculated in Step 1.
2. Set the md_nsets field to the quantity calculated in Step 2.

On each node, perform a reconfiguration reboot.
# touch /reconfigure # shutdown -g0 -y -i6
Changes to the /kernel/drv/md.conf file become operative after you perform a reconfiguration reboot.

Create local replicas.

Go to "How to Create Metadevice State Database Replicas".

How to Create Metadevice State Database Replicas

Note -

If you used SunPlex Manager to install Solstice DiskSuite software, do not perform this procedure. Instead, proceed to "Mirroring the Root Disk".

Perform this procedure on each node in the cluster.

Become superuser on the cluster node.

Create replicas on one or more local disks for each cluster node by using the metadb command.

See the metadb(1M) man page and your Solstice DiskSuite documentation for details.

Tip -
To provide protection of metadevice state data, which is necessary to run Solstice DiskSuite software, create at least three replicas for each node. Also, you can place replicas on more than one disk to provide protection if one of the disks fails.

Verify the replicas.
# metadb

Do you intend to mirror file systems on the root disk?
- If yes, go to "Mirroring the Root Disk".
- If no, go to "How to Create a Diskset" to create Solstice DiskSuite disksets.

Example--Creating Metadevice State Database Replicas

The following example shows three metadevice state database replicas, each created on a different disk.

# metadb -af c0t0d0s7 c0t1d0s7 c1t0d0s7
# metadb
flags            first blk      block count
    a       u       16            1034         /dev/dsk/c0t0d0s7
    a       u       1050          1034         /dev/dsk/c0t1d0s7
    a       u       2084          1034         /dev/dsk/c1t0d0s7

Mirroring the Root Disk

Mirroring the root disk prevents the cluster node itself from shutting down because of a system disk failure. Four types of file systems can reside on the root disk. Each file system type is mirrored by using a different method.

Use the following procedures to mirror each type of file system.

Note -

Some of the steps in these mirroring procedures can cause an error message similar to the following, which is harmless and can be ignored.

metainit: dg-schost-1: d1s0: not a metadevice

Caution -

For local disk mirroring, do not use /dev/global as the path when you specify the disk name. If you specify this path for anything other than cluster file systems, the system cannot boot.

How to Mirror the Root (/) File System

Use this procedure to mirror the root (/) file system.

Become superuser on a node of the cluster.

Use the metainit(1M) command to put the root slice in a single-slice (one-way) concatenation.
# metainit -f submirror1 1 1 root-disk-slice

Create a second concatenation.

# metainit -f submirror2 1 1 submirror-disk-slice

Create a one-way mirror with one submirror.
# metainit mirror -m submirror1
Note -
The metadevice name for the mirror must be unique throughout the cluster.

Run the metaroot(1M) command.

This command edits the /etc/vfstab and /etc/system files so the system can be booted with the root (/) file system on a metadevice.
# metaroot mirror

Run the lockfs(1M) command.

This command flushes all transactions out of the log and writes the transactions to the master file system on all mounted UFS file systems.
# lockfs -fa

Evacuate any resource groups or device groups from the node.
# scswitch -S -h node
-S

Evacuates all resource groups and device groups

-h node

Specifies the name of the node from which to evacuate resource or device groups

Reboot the node.
# shutdown -g0 -y -i6

Use the metattach(1M) command to attach the second submirror to the mirror.
# metattach mirror submirror2

If the disk used to mirror the root disk is physically connected to more than one node (multiported), enable the localonly property of the raw disk device group for the disk used to mirror the root disk.

You must enable the localonly property to prevent unintentional fencing of a node from its boot device if the boot device is connected to multiple nodes.
1. If necessary, use the scdidadm -L command to display the full device ID (DID) pseudo-driver name of the raw disk device group.
  
  In the following example, the raw disk device group name dsk/d2 is part of the third column of output, which is the full DID pseudo-driver name.
  # scdidadm -L ... 1 phys-schost-3:/dev/rdsk/c1t1d0 /dev/did/rdsk/d2 # scconf -c -D name=dsk/d2,localonly=true
  For more information about the localonly property, see the scconf_dg_rawdisk(1M) man page.
2. Use the scconf(1M) command to enable the localonly property.
  # scconf -c -D name=rawdisk-groupname,localonly=true
  -D name=rawdisk-groupname
  
  Specifies the name of the raw disk device group

Record the alternate boot path for possible future use.
# ls -l /dev/rdsk/root-disk-slice

Repeat Step 1 through Step 11 on each remaining node of the cluster.

Ensure that each metadevice name for a mirror is unique throughout the cluster.

Do you intend to mirror the global namespace, /global/.devices/node@nodeid?
- If yes, go to "How to Mirror the Global Namespace".
- If no, go to Step 14.

Do you intend to mirror file systems than cannot be unmounted?
- If yes, go to "How to Mirror File Systems That Cannot Be Unmounted".
- If no, go to Step 15.

Do you intend to mirror user-defined file systems?
- If yes, go to "How to Mirror User-Defined File Systems".
- If no, go to "How to Create a Diskset" to create a diskset.

Example--Mirroring the Root (/) File System

The following example shows creation of mirror d0 on the node phys-schost-1, which consists of submirror d10 on partition c0t0d0s0 and submirror d20 on partition c2t2d0s0. Disk c2t2d0 is a multiported disk, so the localonly property is enabled.

(Create the mirror)
# metainit -f d10 1 1 c0t0d0s0
d11: Concat/Stripe is setup
# metainit -f d20 1 1 c2t2d0s0
d12: Concat/Stripe is setup
# metainit d0 -m d10
d10: Mirror is setup
# metaroot d0
# lockfs -fa
 
(Reboot the node)
# scswitch -S -h phys-schost-1
# shutdown -g0 -y -i6
 
(Attach the second submirror)
# metattach d0 d20
d0: Submirror d20 is attached
 
(Enable the localonly property of the mirrored disk's
raw disk device group)
# scconf -c -D name=dsk/d2,localonly=true
 
(Record the alternate boot path)
# ls -l /dev/rdsk/c2t2d0s0
lrwxrwxrwx  1 root     root          57 Apr 25 20:11 /dev/rdsk/c2t2d0s0 -> 
../../devices/node@1/pci@1f,0/pci@1/scsi@3,1/disk@2,0:a,raw

How to Mirror the Global Namespace

Use this procedure to mirror the global namespace, /global/.devices/node@nodeid.

Become superuser on a node of the cluster.

Put the global namespace slice in a single slice (one-way) concatenation.
# metainit -f submirror1 1 1 diskslice

Create a second concatenation.

# metainit -f submirror2 1 1 submirror-diskslice

Create a one-way mirror with one submirror.
# metainit mirror -m submirror1
Note -
The metadevice name for the mirror must be unique throughout the cluster.

Attach the second submirror to the mirror.

This attachment starts a sync of the submirrors.
# metattach mirror submirror2

Edit the /etc/vfstab file entry for the /global/.devices/node@nodeid file system.

Replace the names in the device to mount and device to fsck columns with the mirror name.

# vi /etc/vfstab
#device        device          mount           FS      fsck    mount   mount
#to mount      to fsck         point           type    pass    at boot options
#
/dev/md/dsk/mirror /dev/md/rdsk/mirror /global/.devices/node@nodeid ufs 2 no global

Repeat Step 1 through Step 6 on each remaining node of the cluster.

Ensure that each metadevice name for a mirror is unique throughout the cluster.

Wait for the sync of the mirrors, started in Step 5, to complete.

Use the metastat(1M) command to view mirror status.
# metastat mirror

If the disk used to mirror the global namespace is physically connected to more than one node (multiported), enable the localonly property of the raw disk device group for the disk used to mirror the global namespace.

You must enable the localonly property to prevent unintentional fencing of a node from its boot device if the boot device is connected to multiple nodes.
1. If necessary, use the scdidadm -L command to display the full device ID (DID) pseudo-driver name of the raw disk device group.
  
  In the following example, the raw disk device group name dsk/d2 is part of the third column of output, which is the full DID pseudo-driver name.
  # scdidadm -L ... 1 phys-schost-3:/dev/rdsk/c1t1d0 /dev/did/rdsk/d2 # scconf -c -D name=dsk/d2,localonly=true
  For more information about the localonly property, see the scconf_dg_rawdisk(1M) man page.
2. Use the scconf(1M) command to enable the localonly property.
  # scconf -c -D name=rawdisk-groupname,localonly=true
  -D name=rawdisk-groupname
  
  Specifies the name of the raw disk device group

Do you intend to mirror file systems than cannot be unmounted?
- If yes, go to "How to Mirror File Systems That Cannot Be Unmounted".
- If no, go to Step 11.

Do you intend to mirror user-defined file systems?
- If yes, go to "How to Mirror User-Defined File Systems".
- If no, go to "How to Create a Diskset" to create a diskset.

Example--Mirroring the Global Namespace

The following example shows creation of mirror d101, which consists of submirror d111 on partition c0t0d0s3 and submirror d121 on partition c2t2d0s3. The /etc/vfstab file entry for /global/.devices/node@1 is updated to use the mirror name d101. Disk c2t2d0 is a multiported disk, so the localonly property is enabled.

(Create the mirror)
# metainit -f d111 1 1 c0t0d0s3
d111: Concat/Stripe is setup
# metainit -f d121 1 1 c2t2d0s3
d121: Concat/Stripe is setup
# metainit d101 -m d111
d101: Mirror is setup
# metattach d101 d121
d101: Submirror d121 is attached
 
(Edit the /etc/vfstab file)
# vi /etc/vfstab
#device        device          mount           FS      fsck    mount   mount
#to mount      to fsck         point           type    pass    at boot options
#
/dev/md/dsk/d101 /dev/md/rdsk/d101 /global/.devices/node@1 ufs 2 no global
 
(View the sync status)
# metastat d101
d101: Mirror
      Submirror 0: d111
         State: Okay
      Submirror 1: d121
         State: Resyncing
      Resync in progress: 15 % done
...
 
(Identify the DID name of the mirrored disk's raw
disk device group)
# scdidadm -L
...
1         phys-schost-3:/dev/rdsk/c2t2d0     /dev/did/rdsk/d2
 
(Enable the localonly property of the mirrored disk's
raw disk device group)
# scconf -c -D name=dsk/d2,localonly=true

How to Mirror File Systems That Cannot Be Unmounted

Use this procedure to mirror file systems that cannot be unmounted during normal system usage, such as /usr, /opt, or swap.

Become superuser on a node of the cluster.

Put the slice on which an unmountable file system resides in a single slice (one-way) concatenation.
# metainit -f submirror1 1 1 diskslice

Create a second concatenation.

# metainit -f submirror2 1 1 submirror-diskslice

Create a one-way mirror with one submirror.
# metainit mirror -m submirror1
Note -
The metadevice name for the mirror does not need to be unique throughout the cluster.

Repeat Step 1 through Step 4 for each unmountable file system to be mirrored.

On each node, edit the /etc/vfstab file entry for each unmountable file system you mirrored.

Replace the names in the device to mount and device to fsck columns with the mirror name.

# vi /etc/vfstab
#device        device          mount           FS      fsck    mount   mount
#to mount      to fsck         point           type    pass    at boot options
#
/dev/md/dsk/mirror /dev/md/rdsk/mirror /filesystem  ufs     2       no      global

Evacuate any resource groups or device groups from the node.
# scswitch -S -h node
-S

Evacuates all resource groups and device groups

-h node

Specifies the name of the node from which to evacuate resource or device groups

Reboot the node.
# shutdown -g0 -y -i6

Attach the second submirror to each mirror.

This attachment starts a sync of the submirrors.
# metattach mirror submirror2

Wait for the sync of the mirrors, started in Step 9, to complete.

Use the metastat(1M) command to view mirror status.
# metastat mirror

If the disk used to mirror the unmountable file system is physically connected to more than one node (multiported), enable the localonly property of the raw disk device group for the disk used to mirror the unmountable file system.

You must enable the localonly property to prevent unintentional fencing of a node from its boot device if the boot device is connected to multiple nodes.
1. If necessary, use the scdidadm -L command to display the full device ID (DID) pseudo-driver name of the raw disk device group.
  
  In the following example, the raw disk device group name dsk/d2 is part of the third column of output, which is the full DID pseudo-driver name.
  # scdidadm -L ... 1 phys-schost-3:/dev/rdsk/c1t1d0 /dev/did/rdsk/d2 # scconf -c -D name=dsk/d2,localonly=true
  For more information about the localonly property, see the scconf_dg_rawdisk(1M) man page.
2. Use the scconf(1M) command to enable the localonly property.
  # scconf -c -D name=rawdisk-groupname,localonly=true
  -D name=rawdisk-groupname
  
  Specifies the name of the raw disk device group

Do you intend to mirror user-defined file systems?
- If yes, go to "How to Mirror User-Defined File Systems".
- If no, go to "How to Create a Diskset" to create a diskset.

Example--Mirroring Unmountable File Systems

The following example shows the creation of mirror d1 on the node phys-schost-1 to mirror /usr, which resides on c0t0d0s1. Mirror d1 consists of submirror d11 on partition c0t0d0s1 and submirror d21 on partition c2t2d0s1. The /etc/vfstab file entry for /usr is updated to use the mirror name d1. Disk c2t2d0 is a multiported disk, so the localonly property is enabled.

(Create the mirror)
# metainit -f d11 1 1 c0t0d0s1
d11: Concat/Stripe is setup
# metainit -f d21 1 1 c2t2d0s1
d21: Concat/Stripe is setup
# metainit d1 -m d11
d1: Mirror is setup
 
(Edit the /etc/vfstab file)
# vi /etc/vfstab
#device        device          mount           FS      fsck    mount   mount
#to mount      to fsck         point           type    pass    at boot options
#
/dev/md/dsk/d1 /dev/md/rdsk/d1 /usr            ufs     2       no      global
 
(Reboot the node)
# scswitch -S -h phys-schost-1
# shutdown -g0 -y -i6
 
(Attach the second submirror)
# metattach d1 d21
d1: Submirror d21 is attached
 
(View the sync status)
# metastat d1
d1: Mirror
      Submirror 0: d11
         State: Okay
      Submirror 1: d21
         State: Resyncing
      Resync in progress: 15 % done
...
 
(Identify the DID name of the mirrored disk's raw
disk device group)
# scdidadm -L
...
1         phys-schost-3:/dev/rdsk/c2t2d0     /dev/did/rdsk/d2
 
(Enable the localonly property of the mirrored disk's
raw disk device group)
# scconf -c -D name=dsk/d2,localonly=true

How to Mirror User-Defined File Systems

Use this procedure to mirror user-defined file systems. In this procedure, nodes do not need to be rebooted.

Become superuser on a node of the cluster.

Put the slice on which a user-defined file system resides in a single slice (one-way) concatenation.
# metainit -f submirror1 1 1 diskslice

Create a second concatenation.

# metainit -f submirror2 1 1 submirror-diskslice

Create a one-way mirror with one submirror.
# metainit mirror -m submirror1
Note -
The metadevice name for the mirror does not need to be unique throughout the cluster.

Repeat Step 1 through Step 4 for each user-defined file system to be mirrored.

On each node, edit the /etc/vfstab file entry for each user-defined file system you mirrored.

Replace the names in the device to mount and device to fsck columns with the mirror name.

# vi /etc/vfstab
#device        device          mount           FS      fsck    mount   mount
#to mount      to fsck         point           type    pass    at boot options
#
/dev/md/dsk/mirror /dev/md/rdsk/mirror /filesystem  ufs     2       no      global

Attach the second submirror to the mirror.

This attachment starts a sync of the submirrors.
# metattach mirror submirror2

Wait for the sync of the mirrors, started in Step 7, to complete.

Use the metastat(1M) command to view mirror status.
# metastat mirror

If the disk used to mirror the user-defined file system is physically connected to more than one node (multiported), enable the localonly property of the raw disk device group for the disk used to mirror the user-defined file system.

You must enable the localonly property to prevent unintentional fencing of a node from its boot device if the boot device is connected to multiple nodes.
1. If necessary, use the scdidadm -L command to display the full device ID (DID) pseudo-driver name of the raw disk device group.
  
  In the following example, the raw disk device group name dsk/d4 is part of the third column of output, which is the full DID pseudo-driver name.
  # scdidadm -L ... 1 phys-schost-3:/dev/rdsk/c1t1d0 /dev/did/rdsk/d2 # scconf -c -D name=dsk/d2,localonly=true
  For more information about the localonly property, see the scconf_dg_rawdisk(1M) man page.
2. Use the scconf(1M) command to enable the localonly property.
  # scconf -c -D name=rawdisk-groupname,localonly=true
  -D name=rawdisk-groupname
  
  Specifies the name of the raw disk device group

Create a diskset.

Go to "How to Create a Diskset".

Example--Mirroring User-Defined File Systems

The following example shows creation of mirror d4 to mirror /home, which resides on c0t0d0s4. Mirror d4 consists of submirror d14 on partition c0t0d0s4 and submirror d24 on partition c2t2d0s4. The /etc/vfstab file entry for /home is updated to use the mirror name d4. Disk c2t2d0 is a multiported disk, so the localonly property is enabled.

(Create the mirror)
# metainit -f d14 1 1 c0t0d0s4
d14: Concat/Stripe is setup
# metainit -f d24 1 1 c2t2d0s4
d24: Concat/Stripe is setup
# metainit d4 -m d14
d4: Mirror is setup
 
(Edit the /etc/vfstab file)
# vi /etc/vfstab
#device        device          mount           FS      fsck    mount   mount
#to mount      to fsck         point           type    pass    at boot options
#
/dev/md/dsk/d4 /dev/md/rdsk/d4 /home           ufs     2       no      global
 
(Attach the second submirror)
# metattach d4 d24
d4: Submirror d24 is attached
 
(View the sync status)
# metastat d4
d4: Mirror
      Submirror 0: d14
         State: Okay
      Submirror 1: d24
         State: Resyncing
      Resync in progress: 15 % done
...
 
(Identify the DID name of the mirrored disk's raw
disk device group)
# scdidadm -L
...
1         phys-schost-3:/dev/rdsk/c2t2d0     /dev/did/rdsk/d2
 
(Enable the localonly property of the mirrored disk's
raw disk device group)
# scconf -c -D name=dsk/d2,localonly=true

How to Create a Diskset

Perform this procedure for each diskset you create.

Note -

If you used SunPlex Manager to install Solstice DiskSuite, one to three disksets might already exist. See "Using SunPlex Manager to Install Sun Cluster Software" for information about the metasets created by SunPlex Manager.

Ensure that the diskset you intend to create meets one of the following requirements.
- If configured with exactly two disk strings, the diskset must connect to exactly two nodes and use exactly two mediator hosts, which must be the same two hosts used for the diskset. See "Mediators Overview" for details on how to set up mediators.
- If configured with more than two disk strings, ensure that for any two disk strings S1 and S2, the sum of the number of disks on those strings exceeds the number of disks on the third string S3. Stated as a formula, the requirement is that count(S1) + count(S2) > count(S3).

Ensure that root is a member of group 14.

# vi /etc/group
...
sysadmin::14:root
...

Ensure that the local metadevice state database replicas exist.

For instructions, see "How to Create Metadevice State Database Replicas".

Become superuser on the cluster node that will master the diskset.

Create the diskset.

This command also registers the diskset as a Sun Cluster disk device group.
# metaset -s setname -a -h node1 node2
-s setname

Specifies the diskset name

-a

Adds (creates) the diskset

-h node1

Specifies the name of the primary node to master the diskset

node2

Specifies the name of the secondary node to master the diskset

Verify the status of the new diskset.
# metaset -s setname

Add drives to the diskset.

Go to "Adding Drives to a Diskset".

Example--Creating a Diskset

The following command creates two disksets, dg-schost-1 and dg-schost-2, with the nodes phys-schost-1 and phys-schost-2 assigned as the potential primaries.

# metaset -s dg-schost-1 -a -h phys-schost-1 phys-schost-2
# metaset -s dg-schost-2 -a -h phys-schost-1 phys-schost-2

Adding Drives to a Diskset

When you add a disk drive to a diskset, Solstice DiskSuite repartitions it as follows so that the metadevice state database for the diskset can be placed on the drive.

A small portion of each drive is reserved in slice 7 for use by Solstice DiskSuite software. The remainder of the space on each drive is placed into slice 0.
Drives are repartitioned when they are added to the diskset only if slice 7 is not set up correctly.
Any existing data on the disks is lost by the repartitioning.
If slice 7 starts at cylinder 0, and the disk is large enough to contain a state database replica, the disk is not repartitioned.

How to Add Drives to a Diskset

Become superuser on the node.

Ensure that the diskset has been created.

For instructions, see "How to Create a Diskset".

List the device ID (DID) mappings.

# scdidadm -L

Choose drives that are shared by the cluster nodes that will master or potentially master the diskset.
Use the full DID pseudo-driver names when you add drives to a diskset.

The first column of output is the DID instance number, the second column is the full path (physical path), and the third column is the full DID pseudo-driver name (pseudo path). A shared drive has more than one entry for the same DID instance number.

In the following example, the entries for DID instance number 2 indicate a drive that is shared by phys-schost-1 and phys-schost-2, and the full DID name is /dev/did/rdsk/d2.

1       phys-schost-1:/dev/rdsk/c0t0d0 /dev/did/rdsk/d1
2       phys-schost-1:/dev/rdsk/c1t1d0 /dev/did/rdsk/d2
2       phys-schost-2:/dev/rdsk/c1t1d0 /dev/did/rdsk/d2
3       phys-schost-1:/dev/rdsk/c1t2d0 /dev/did/rdsk/d3
3       phys-schost-2:/dev/rdsk/c1t2d0 /dev/did/rdsk/d3
...

Take ownership of the diskset.
# metaset -s setname -t
-s setname

Specifies the diskset name

-t

Takes ownership of the diskset

Add the drives to the diskset.

Use the full DID pseudo-driver name.
# metaset -s setname -a DIDname
-a

Adds the disk drive to the diskset

DIDname

Device ID (DID) name of the shared disk

Note -
Do not use the lower-level device name (cNtXdY) when you add a drive to a diskset. Because the lower-level device name is a local name and not unique throughout the cluster, using this name might prevent the metaset from being able to switch over.

Verify the status of the diskset and drives.
# metaset -s setname

Do you intend to repartition drives for use in metadevices?
- If yes, go to "How to Repartition Drives in a Diskset".
- If no, go to "How to Create an md.tab File" to define metadevices by using an md.tab file.

Example--Adding Drives to a Diskset

The metaset command adds the disk drives /dev/did/dsk/d1 and /dev/did/dsk/d2 to the diskset dg-schost-1.

# metaset -s dg-schost-1 -a /dev/did/dsk/d1 /dev/did/dsk/d2

How to Repartition Drives in a Diskset

The metaset(1M) command repartitions drives in a diskset so that a small portion of each drive is reserved in slice 7 for use by Solstice DiskSuite software. The remainder of the space on each drive is placed into slice 0. To make more effective use of the disk, use this procedure to modify the disk layout. If you allocate space to slices 1 through 6, you can use these slices when you set up metadevices.

Become superuser on the cluster node.

Use the format(1M) command to change the disk partitioning for each drive in the diskset.

When you repartition a drive, you must meet the following conditions to prevent the metaset(1M) command from repartitioning the disk.
- Create a partition 7 starting at cylinder 0 that is large enough to hold a state database replica (approximately 2 Mbytes).
- Set the Flag field in slice 7 to V_UNMT (unmountable) and do not set it to read-only.
- Do not allow slice 7 to overlap any other slice on the disk.
See the format(1M) man page for details.

Define metadevices by using an md.tab file.

Go to "How to Create an md.tab File".

How to Create an `md.tab` File

Create an /etc/lvm/md.tab file on each node in the cluster. Use the md.tab file to define metadevices for the disksets you created.

Note -

If you are using local metadevices, ensure that local metadevices names are distinct from the device ID (DID) names used to form disksets. For example, if the DID name /dev/did/dsk/d3 is used in a diskset, do not use the name /dev/md/dsk/d3 for a local metadevice. This requirement does not apply to shared metadevices, which use the naming convention /dev/md/setname/{r}dsk/d#.

Tip -

To avoid possible confusion between local metadevices in a cluster environment, use a naming scheme that makes each local metadevice name unique throughout the cluster. For example, for node 1 choose names from d100-d199, for node 2 use d200-d299, and so on.

Become superuser on the cluster node.

List the DID mappings for reference when you create your md.tab file.

Use the full DID pseudo-driver names in the md.tab file in place of the lower-level device names (cNtXdY).

# scdidadm -L

In the following example, the first column of output is the DID instance number, the second column is the full path (physical path), and the third column is the full DID pseudo-driver name (pseudo path).

1       phys-schost-1:/dev/rdsk/c0t0d0 /dev/did/rdsk/d1
2       phys-schost-1:/dev/rdsk/c1t1d0 /dev/did/rdsk/d2
2       phys-schost-2:/dev/rdsk/c1t1d0 /dev/did/rdsk/d2
3       phys-schost-1:/dev/rdsk/c1t2d0 /dev/did/rdsk/d3
3       phys-schost-2:/dev/rdsk/c1t2d0 /dev/did/rdsk/d3
...

Create an /etc/lvm/md.tab file and edit it by hand with your preferred text editor.

See your Solstice DiskSuite documentation and the md.tab(4) man page for details on how to create an md.tab file.

Note -
If you have existing data on the disks that will be used for the submirrors, you must back up the data before metadevice setup and restore it onto the mirror.

Activate the metadevices defined in the md.tab files.

Go to "How to Activate Metadevices".

Example--Sample `md.tab` File

The following sample md.tab file defines the metadevices for the diskset named dg-schost-1. The ordering of lines in the md.tab file is not important.

dg-schost-1/d0 -t dg-schost-1/d1 dg-schost-1/d4
    dg-schost-1/d1 -m dg-schost-1/d2
        dg-schost-1/d2 1 1 /dev/did/rdsk/d1s4
        dg-schost-1/d3 1 1 /dev/did/rdsk/d55s4
    dg-schost-1/d4 -m dg-schost-1/d5
        dg-schost-1/d5 1 1 /dev/did/rdsk/d3s5
        dg-schost-1/d6 1 1 /dev/did/rdsk/d57s5

The sample md.tab file is constructed as follows.

The first line defines the trans metadevice d0 to consist of a master (UFS) metadevice d1 and a log device d4. The -t signifies this is a trans metadevice. The master and log devices are specified by their position after the -t flag.
dg-schost-1/d0 -t dg-schost-1/d1 dg-schost-1/d4
The second line defines the master device as a mirror of the metadevices. The -m in this definition signifies a mirror device, and one of the submirrors, d2, is associated with the mirror device, d1.
dg-schost-1/d1 -m dg-schost-1/d2
The fifth line similarly defines the log device, d4, as a mirror of metadevices.
dg-schost-1/d4 -m dg-schost-1/d5
The third line defines the first submirror of the master device, d2, as a one-way stripe.
dg-schost-1/d2 1 1 /dev/did/rdsk/d1s4
The fourth line defines the second submirror of the master device, d3.
dg-schost-1/d3 1 1 /dev/did/rdsk/d55s4
Finally, the log device submirrors, d5 and d6, are defined. In this example, simple metadevices for each submirror are created.
dg-schost-1/d5 1 1 /dev/did/rdsk/d3s5 dg-schost-1/d6 1 1 /dev/did/rdsk/d57s5

How to Activate Metadevices

Perform this procedure to activate metadevices defined in the md.tab files.

Become superuser on the cluster node.

Ensure that md.tab files are located in the /etc/lvm directory.

Ensure that you have ownership of the diskset on the node where the command will be executed.

Take ownership of the diskset.
# metaset -s setname -t
-s setname

Specifies the diskset name

-t

Takes ownership of the diskset

Activate the diskset's metadevices, which are defined in the md.tab file.
# metainit -s setname -a
-a

Activates all metadevices in the md.tab file

For each master and log device, attach the second submirror (submirror2).

When the metadevices in the md.tab file are activated, only the first submirror (submirror1) of the master and log devices is attached, so submirror2 must be attached by hand.
# metattach mirror submirror2

Repeat Step 3 through Step 6 for each diskset in the cluster.

If necessary, run the metainit(1M) command from another node that has connectivity to the disks. This step is required for cluster-pair topologies, where the disks are not accessible by all nodes.

Check the status of the metadevices.
# metastat -s setname
See the metastat(1M) man page for more information.

Does your cluster contain disksets configured with exactly two disk enclosures and two nodes?
- If yes, those disksets require mediators. Go to "Mediators Overview" to add mediator hosts.
- If no, go to "How to Add Cluster File Systems" to create a cluster file system.

Example--Activating Metadevices in the `md.tab` File

In the following example, all metadevices defined in the md.tab file for diskset dg-schost-1 are activated. Then the second submirrors of master device dg-schost-1/d1 and log device dg-schost-1/d4 are activated.

# metainit -s dg-schost-1 -a
# metattach dg-schost-1/d1 dg-schost-1/d3
# metattach dg-schost-1/d4 dg-schost-1/d6

Mediators Overview

A mediator, or mediator host, is a cluster node that stores mediator data. Mediator data provides information on the location of other mediators and contains a commit count that is identical to the commit count stored in the database replicas. This commit count is used to confirm that the mediator data is in sync with the data in the database replicas.

Mediators are required for all Solstice DiskSuite disksets configured with exactly two disk strings and two cluster nodes. A disk string consists of a disk enclosure, its physical disks, cables from the enclosure to the node(s), and the interface adapter cards. The use of mediators enables the Sun Cluster software to ensure that the most current data is presented in the instance of a single-string failure in a dual-string configuration. The following rules apply to dual-string configurations that use mediators.

Disksets must be configured with exactly two mediator hosts, and those two mediator hosts must be the same two cluster nodes used for the diskset.
A diskset cannot have more than two mediator hosts.
Mediators cannot be configured for disksets that do not meet the two-string and two-host criteria.

These rules do not require that the entire cluster have exactly two nodes. Rather, they only require that those disksets that have two disk strings must be connected to exactly two nodes. An N+1 cluster and many other topologies are permitted under these rules.

How to Add Mediator Hosts

Perform this procedure if your configuration requires mediators.

Become superuser on the node that currently masters the diskset you intend to add mediator hosts to.

Run the metaset(1M) command to add each node with connectivity to the diskset as a mediator host for that diskset.
# metaset -s setname -a -m mediator-host-list
-s setname

Specifies the diskset name

-a

Adds to the diskset

-m mediator-host-list

Specifies the name of the node to add as a mediator host for the diskset

See the mediator(7) man page for details about mediator-specific options to the metaset command.

Check the status of mediator data.

Go to "How to Check the Status of Mediator Data".

Example--Adding Mediator Hosts

The following example adds the nodes phys-schost-1 and phys-schost-2 as mediator hosts for the diskset dg-schost-1. Both commands are run from the node phys-schost-1.

# metaset -s dg-schost-1 -a -m phys-schost-1
# metaset -s dg-schost-1 -a -m phys-schost-2

How to Check the Status of Mediator Data

Add mediator hosts as described in "How to Add Mediator Hosts".

Run the medstat command.
# medstat -s setname
-s setname

Specifies the diskset name

See the medstat(1M) man page for more information.

Is Bad the value in the Status field?
- If yes, go to "How to Fix Bad Mediator Data" to repair the affected mediator host.
- If no, go to "How to Add Cluster File Systems" to create a cluster file system.

How to Fix Bad Mediator Data

Perform this procedure to repair bad mediator data.

Identify the mediator host(s) with bad mediator data as described in the procedure "How to Check the Status of Mediator Data".

Become superuser on the node that owns the affected diskset.

Remove the mediator host(s) with bad mediator data from all affected disksets.
# metaset -s setname -d -m mediator-host-list
-s setname

Specifies the diskset name

-d

Deletes from the diskset

-m mediator-host-list

Specifies the name of the node to remove as a mediator host for the diskset

Restore the mediator host.
# metaset -s setname -a -m mediator-host-list
-a

Adds to the diskset

-m mediator-host-list

Specifies the name of the node to add as a mediator host for the diskset

See the mediator(7) man page for details about mediator-specific options to the metaset command.

Create a cluster file system.

Go to "How to Add Cluster File Systems".

Installing and Configuring Solstice DiskSuite Software

How to Install Solstice DiskSuite Software

How to Set the Number of Metadevice Names and Disksets

How to Create Metadevice State Database Replicas

Example--Creating Metadevice State Database Replicas

Mirroring the Root Disk

How to Mirror the Root (/) File System

Example--Mirroring the Root (/) File System

How to Mirror the Global Namespace

Example--Mirroring the Global Namespace

How to Mirror File Systems That Cannot Be Unmounted

Example--Mirroring Unmountable File Systems

How to Mirror User-Defined File Systems

Example--Mirroring User-Defined File Systems

How to Create a Diskset

Example--Creating a Diskset

Adding Drives to a Diskset

How to Add Drives to a Diskset

Example--Adding Drives to a Diskset

How to Repartition Drives in a Diskset

How to Create an md.tab File

Example--Sample md.tab File

How to Activate Metadevices

Example--Activating Metadevices in the md.tab File

Mediators Overview

How to Add Mediator Hosts

Example--Adding Mediator Hosts

How to Check the Status of Mediator Data

How to Fix Bad Mediator Data

How to Create an `md.tab` File

Example--Sample `md.tab` File

Example--Activating Metadevices in the `md.tab` File