Install and configure your local and multihost disks for VERITAS Volume Manager (VxVM) by using the procedures in this chapter, along with the planning information in Planning Volume Management. See your VxVM documentation for additional details.
The following information and procedures are in this chapter:
SPARC: How to Install VERITAS Volume Manager Software and Encapsulate the Root Disk
SPARC: How to Assign a New Minor Number to a Disk Device Group
This section provides information and procedures to install and configure VxVM software on a Sun Cluster configuration.
The following table lists the tasks to perform to install and configure VxVM software for Sun Cluster configurations.
Table 4–1 SPARC: Task Map: Installing and Configuring VxVM Software
Task |
Instructions |
---|---|
1. Plan the layout of your VxVM configuration. | |
2. Determine how you will create the root disk group on each node. | |
3. For VxVM 3.5 and earlier, install VxVM software and create the root disk group. For VxVM 4.0, the creation of a root disk group is optional. |
|
| |
| |
4. Create disk groups and volumes. | |
5. If necessary, resolve any minor-number conflicts between disk device groups by assigning a new minor number. |
SPARC: How to Assign a New Minor Number to a Disk Device Group |
6. Verify the disk groups and volumes. | |
7. Configure the cluster. |
For VxVM 4.0, the creation of a root disk group is optional. If you do not intend to create a root disk group, proceed to SPARC: How to Install VERITAS Volume Manager Software Only.
For VxVM 3.5 and earlier, each cluster node requires the creation of a root disk group after VxVM is installed. This disk group is used by VxVM to store configuration information, and has the following restrictions.
Access to a node's root disk group must be restricted to only that node.
Remote nodes must never access data stored in another node's root disk group.
Do not use the scconf(1M) command to register the root disk group as a disk device group.
Whenever possible, configure the root disk group for each node on a nonshared disk.
Sun Cluster software supports the following methods to configure the root disk group.
Encapsulate the node's root disk – This method enables the root disk to be mirrored, which provides a boot alternative if the root disk is corrupted or damaged. To encapsulate the root disk you need two free disk slices as well as free cylinders, preferably at the beginning or the end of the disk.
Use local nonroot disks – This method provides an alternative to encapsulating the root disk. If a node's root disk is encapsulated, certain tasks you might later perform, such as upgrade the Solaris OS or perform disaster recovery procedures, could be more complicated than if the root disk is not encapsulated. To avoid this potential added complexity, you can instead initialize or encapsulate local nonroot disks for use as root disk groups.
A root disk group that is created on local nonroot disks is local to that node, neither globally accessible nor highly available. As with the root disk, to encapsulate a nonroot disk you need two free disk slices as well as free cylinders at the beginning or the end of the disk.
See your VxVM installation documentation for more information.
Install VxVM by using one of the following installation methods, depending on how you intend to create the root disk group.
If you intend to encapsulate the root disk, go to SPARC: How to Install VERITAS Volume Manager Software and Encapsulate the Root Disk.
If you intend to create the root disk group on local nonroot disks, go to SPARC: How to Install VERITAS Volume Manager Software Only.
This procedure uses the scvxinstall(1M) command to install VxVM software and encapsulate the root disk in one operation.
If you intend to create the root disk group on local, nonroot disks, go instead to SPARC: How to Install VERITAS Volume Manager Software Only.
Perform this procedure on each node that you intend to install with VxVM. You can install VERITAS Volume Manager (VxVM) on all nodes of the cluster, or on only those nodes that are physically connected to the storage devices that VxVM will manage.
If you later need to unencapsulate the root disk, follow the procedures in SPARC: How to Unencapsulate the Root Disk.
Ensure that the cluster meets the following prerequisites.
All nodes in the cluster are running in cluster mode.
The root disk of the node you install has two free (unassigned) partitions.
Have available the following information.
Mappings of your storage disk drives.
The following completed configuration planning worksheets. See Planning Volume Management for planning guidelines.
Become superuser on a node that you intend to install with VxVM.
Insert the VxVM CD-ROM into the CD-ROM drive on the node.
Start scvxinstall in interactive mode.
Press Ctrl-C at any time to abort the scvxinstall command.
# scvxinstall |
See the scvxinstall(1M) man page for more information.
When prompted whether to encapsulate root, type yes.
Do you want Volume Manager to encapsulate root [no]? y |
When prompted, provide the location of the VxVM CD-ROM.
If the appropriate VxVM CD-ROM is found, the location is displayed as part of the prompt within brackets. Press Enter to accept this default location.
Where is the volume manager cdrom [default]? |
If the VxVM CD-ROM is not found, the prompt is displayed without a default location. Type the location of the CD-ROM or CD-ROM image.
Where is the volume manager cdrom? |
When prompted, type your VxVM license key.
Please enter license key: license |
The scvxinstall command automatically performs the following tasks:
Installs the required VxVM software, licensing, and man-page packages, but does not install the GUI packages
Creates a root disk group by encapsulating the root disk
Updates the /global/.devices entry in the /etc/vfstab file
See the scvxinstall(1M) man page for further details.
Two automatic reboots occur during installation. After all installation tasks are completed, scvxinstall automatically reboots the node the second time unless you press Ctrl-C when prompted. If you press Ctrl-C to abort the second reboot, you must reboot the node later to complete VxVM installation.
If you intend to enable the VxVM cluster feature, supply the cluster feature license key.
See your VxVM documentation for information about how to add a license.
(Optional) Install the VxVM GUI.
See your VxVM documentation for information about installing the VxVM GUI.
Eject the CD-ROM.
Install any VxVM patches.
See “Patches and Required Firmware Levels” in Sun Cluster 3.1 Release Notes for the location of patches and installation instructions.
(Optional) If you prefer not to have VxVM man pages reside on the cluster node, remove the man-page package.
# pkgrm VRTSvmman |
Repeat Step 3 through Step 13 to install VxVM on additional nodes.
If you intend to enable the VxVM cluster feature, you must install VxVM on all nodes of the cluster.
If you will not install VxVM on one or more nodes, modify the /etc/name_to_major file on each non-VxVM node.
On a node installed with VxVM, determine the vxio major number setting.
# grep vxio /etc/name_to_major |
Become superuser on a node that you do not intend to install with VxVM.
Edit the /etc/name_to_major file and add an entry to set the vxio major number to NNN, the number derived in Step a.
# vi /etc/name_to_major vxio NNN |
Initialize the vxio entry.
# drvconfig -b -i vxio -m NNN |
Repeat Step b through Step d on all other nodes that you do not intend to install with VxVM.
When you finish, each node of the cluster should have the same vxio entry in its /etc/name_to_major file.
(Optional) To mirror the encapsulated root disk, go to SPARC: How to Mirror the Encapsulated Root Disk.
After you install VxVM and encapsulate the root disk, perform this procedure on each node on which you mirror the encapsulated root disk.
Mirror the encapsulated root disk.
Follow the procedures in your VxVM documentation. For maximum availability and simplified administration, use a local disk for the mirror. See Guidelines for Mirroring the Root Disk for additional guidelines.
Do not use a quorum device to mirror a root disk. Using a quorum device to mirror a root disk might prevent the node from booting from the root-disk mirror under certain circumstances.
Display the DID mappings.
# scdidadm -L |
From the DID mappings, locate the disk that is used to mirror the root disk.
Extract the raw-disk device-group name from the device-ID name of the root-disk mirror.
The name of the raw-disk device group follows the convention dsk/dN, where N is a number. In the following output, the portion of a scdidadm output line from which you extract the raw-disk device-group name is highlighted in bold.
N node:/dev/rdsk/cNtXdY /dev/did/rdsk/dN |
View the node list of the raw-disk device group.
Output looks similar to the following.
# scconf -pvv | grep dsk/dN Device group name: dsk/dN … (dsk/dN) Device group node list: phys-schost-1, phys-schost-3 … |
If the node list contains more than one node name, remove from the node list all nodes except the node whose root disk you mirrored.
Only the node whose root disk you mirrored should remain in the node list for the raw-disk device group.
# scconf -r -D name=dsk/dN,nodelist=node |
Specifies the cluster-unique name of the raw-disk device group
Specifies the name of the node or nodes to remove from the node list
Enable the localonly property of the raw-disk device group.
When the localonly property is enabled, the raw-disk device group is used exclusively by the node in its node list. This usage prevents unintentional fencing of the node from its boot device if the boot device is connected to multiple nodes.
# scconf -c -D name=dsk/dN,localonly=true |
For more information about the localonly property, see the scconf_dg_rawdisk(1M) man page.
Repeat this procedure for each node in the cluster whose encapsulated root disk you want to mirror.
Create disk groups.
The following example shows a mirror created of the root disk for the node phys-schost-1. The mirror is created on the disk c1t1d0, whose raw-disk device-group name is dsk/d2. Disk c1t1d0 is a multihost disk, so the node phys-schost-3 is removed from the disk's node list and the localonly property is enabled.
(Display the DID mappings) # scdidadm -L … 2 phys-schost-1:/dev/rdsk/c1t1d0 /dev/did/rdsk/d2 2 phys-schost-3:/dev/rdsk/c1t1d0 /dev/did/rdsk/d2 … (Display the node list of the mirror disk's raw-disk device group) # scconf -pvv | grep dsk/d2 Device group name: dsk/d2 … (dsk/d2) Device group node list: phys-schost-1, phys-schost-3 … (Remove phys-schost-3 from the node list) # scconf -r -D name=dsk/d2,nodelist=phys-schost-3 (Enable the localonly property) # scconf -c -D name=dsk/d2,localonly=true |
This procedure uses the scvxinstall command to install VERITAS Volume Manager (VxVM) software only.
To create the root disk group by encapsulating the root disk, do not use this procedure. Instead, go to SPARC: How to Install VERITAS Volume Manager Software and Encapsulate the Root Disk to install VxVM software and encapsulate the root disk in one operation.
Perform this procedure on each node that you want to install with VxVM. You can install VxVM on all nodes of the cluster, or on only those nodes that are physically connected to the storage devices that VxVM will manage.
Ensure that all nodes in the cluster are running in cluster mode.
Have available any VERITAS Volume Manager (VxVM) license keys you need to install.
Become superuser on a cluster node that you intend to install with VxVM.
Insert the VxVM CD-ROM into the CD-ROM drive on the node.
Start scvxinstall in noninteractive installation mode.
# scvxinstall -i -L {license | none} |
Installs VxVM but does not encapsulate the root disk
Installs the specified license. The none argument specifies that no additional license key is being added.
The scvxinstall command automatically performs the following tasks:
Installs the required VxVM software, licensing, and man-page packages, but does not install the GUI packages
Installs any specified license key
See the scvxinstall(1M) man page for information.
(Optional) Install the VxVM GUI.
See your VxVM documentation for information about installing the VxVM GUI.
Eject the CD-ROM.
Install any VxVM patches.
See “Patches and Required Firmware Levels” in Sun Cluster 3.1 Release Notes for the location of patches and installation instructions.
(Optional) If you prefer not to have VxVM man pages reside on the cluster node, remove the man-page package.
# pkgrm VRTSvmman |
Repeat Step 3 through Step 9 to install VxVM on any additional nodes.
If you intend to enable the VxVM cluster feature, you must install VxVM on all nodes of the cluster.
If you do not install one or more nodes with VxVM, modify the /etc/name_to_major file on each non-VxVM node.
On a node that is installed with VxVM, determine the vxio major number setting.
# grep vxio /etc/name_to_major |
Become superuser on a node that you do not intend to install with VxVM.
Edit the /etc/name_to_major file and add an entry to set the vxio major number to NNN, the number derived in Step a.
# vi /etc/name_to_major vxio NNN |
Initialize the vxio entry.
# drvconfig -b -i vxio -m NNN |
Repeat Step a through Step c on all other nodes that you do not intend to install with VxVM.
When you finish, each node of the cluster should have the same vxio entry in its /etc/name_to_major file.
(Optional) To create a root disk group, go to SPARC: How to Create a Root Disk Group on a Nonroot Disk.
Reboot each node.
# shutdown -g0 -y -i6 |
Create disk groups.
Use this procedure to create a root disk group by encapsulating or initializing local disks other than the root disk.
Become superuser on the node.
(Optional) If the disks are to be encapsulated, ensure that each disk has at least two slices with 0 cylinders.
If necessary, use the format(1M) command to assign 0 cylinders to each VxVM slice.
Start the vxinstall utility.
# vxinstall |
When prompted, make the following choices or entries.
If you intend to enable the VxVM cluster feature, supply the cluster feature license key.
Choose Custom Installation.
Do not encapsulate the boot disk.
Choose any disks to add to the root disk group.
Do not accept automatic reboot.
If the root disk group that you created contains one or more disks that connect to more than one node, enable the localonly property.
Use the following command to enable the localonly property of the raw-disk device group for each shared disk in the root disk group.
# scconf -c -D name=dsk/dN,localonly=true |
When the localonly property is enabled, the raw-disk device group is used exclusively by the node in its node list. This usage prevents unintentional fencing of the node from the disk that is used by the root disk group if that disk is connected to multiple nodes.
For more information about the localonly property, see the scconf_dg_rawdisk(1M) man page.
Move any resource groups or device groups from the node.
# scswitch -S -h from-node |
Moves all resource groups and device groups
Specifies the name of the node from which to move resource or device groups
Reboot the node.
# shutdown -g0 -y -i6 |
Use the vxdiskadm command to add multiple disks to the root disk group.
The root disk group becomes tolerant of a disk failure when it contains multiple disks. See VxVM documentation for procedures.
Create disk groups.
Use this procedure to create your VxVM disk groups and volumes.
After a disk group is registered with the cluster as a disk device group, you should never import or deport a VxVM disk group by using VxVM commands. The Sun Cluster software can handle all cases where disk groups need to be imported or deported. See “Administering Disk Device Groups” in Sun Cluster System Administration Guide for Solaris OS for procedures on how to manage Sun Cluster disk device groups.
Perform this procedure from a node that is physically connected to the disks that make the disk group you add.
Have available the following information.
Mappings of your storage disk drives. See the appropriate manual in the Sun Cluster Hardware Administration Collection to perform an initial installation of your storage device.
The following completed configuration planning worksheets.
See Planning Volume Management for planning guidelines.
Become superuser on the node that will have ownership of the disk group.
Create a VxVM disk group and volume.
If you are installing Oracle Real Application Clusters, create shared VxVM disk groups by using the cluster feature of VxVM as described in the VERITAS Volume Manager Administrator's Reference Guide. Otherwise, create VxVM disk groups by using the standard procedures that are documented in the VxVM documentation.
You can use Dirty Region Logging (DRL) to decrease volume recovery time if a node failure occurs. However, DRL might decrease I/O throughput.
If the VxVM cluster feature is not enabled, register the disk group as a Sun Cluster disk device group.
If the VxVM cluster feature is enabled, do not register a shared disk group as a Sun Cluster disk device group. Instead, go to SPARC: How to Verify the Disk Group Configuration.
Start the scsetup(1M) utility.
# scsetup |
Choose Device groups and volumes
.
Choose Register a VxVM disk group
.
Follow the instructions to specify the VxVM disk device group that you want to register as a Sun Cluster disk device group.
If you encounter the following error message when you attempt to register the disk device group, reminor the disk device group.
scconf: Failed to add device group - in use |
To reminor the disk device group, use the procedure SPARC: How to Assign a New Minor Number to a Disk Device Group. This procedure enables you to assign a new minor number that does not conflict with a minor number used by existing disk device groups.
When finished, quit the scsetup utility.
Verify that the disk device group is registered.
Look for the disk device information for the new disk that is displayed by the following command.
# scstat -D |
If you experience a stack overflow when the disk device group is brought online, the default value of the thread stack size might be insufficient. Add the following entry to the /etc/system file on each node, where size is a number greater than 8000, the default setting:
set cl_comm:rm_thread_stacksize=0xsize |
If you change any configuration information for a VxVM disk group or volume, you must register the configuration changes by using the scsetup utility. Configuration changes you must register include adding or removing volumes and changing the group, owner, or permissions of existing volumes. See “Administering Disk Device Groups” in Sun Cluster System Administration Guide for Solaris OS for procedures to register configuration changes to a disk device group.
If disk device group registration fails because of a minor-number conflict with another disk group, you must assign the new disk group a new, unused minor number. Perform this procedure to reminor a disk group.
Become superuser on a node of the cluster.
Determine the minor numbers in use.
# ls -l /global/.devices/node@1/dev/vx/dsk/* |
Choose any other multiple of 1000 that is not in use to become the base minor number for the new disk group.
Assign the new base minor number to the disk group.
# vxdg reminor diskgroup base-minor-number |
Go to SPARC: How to Create and Register a Disk Group of SPARC: How to Create and Register a Disk Group to register the disk group as a Sun Cluster disk device group.
This example uses the minor numbers 16000-16002 and 4000-4001. The vxdg reminor command reminors the new disk device group to use the base minor number 5000.
# ls -l /global/.devices/node@1/dev/vx/dsk/* /global/.devices/node@1/dev/vx/dsk/dg1 brw------- 1 root root 56,16000 Oct 7 11:32 dg1v1 brw------- 1 root root 56,16001 Oct 7 11:32 dg1v2 brw------- 1 root root 56,16002 Oct 7 11:32 dg1v3 /global/.devices/node@1/dev/vx/dsk/dg2 brw------- 1 root root 56,4000 Oct 7 11:32 dg2v1 brw------- 1 root root 56,4001 Oct 7 11:32 dg2v2 # vxdg reminor dg3 5000 |
Perform this procedure on each node of the cluster.
Verify that only the local disks are included in the root disk group, and disk groups are imported on the current primary node only.
# vxdisk list |
Verify that all volumes have been started.
# vxprint |
Verify that all disk groups have been registered as Sun Cluster disk device groups and are online.
# scstat -D |
Configure the cluster.
Go to Configuring the Cluster.
Perform this procedure to unencapsulate the root disk.
Ensure that only Solaris root file systems are present on the root disk.
The Solaris root file systems are root (/), swap, the global devices namespace, /usr, /var, /opt, and /home. If any other file systems reside on the root disk, back them up and remove them from the root disk.
Become superuser on the node that you intend to unencapsulate.
Move all resource groups and device groups from the node.
# scswitch -S -h from-node |
Moves all resource groups and device groups
Specifies the name of the node from which to move resource or device groups
Determine the node-ID number of the node.
# clinfo -nN |
Unmount the global-devices file system for this node, where N is the node ID number that is returned in Step 4.
# umount /global/.devices/node@N |
View the /etc/vfstab file and determine which VxVM volume corresponds to the global-devices file system.
# vi /etc/vfstab #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # #NOTE: volume rootdiskxNvol (/global/.devices/node@N) encapsulated #partition cNtXdYsZ |
Remove the VxVM volume that corresponds to the global-devices file system from the root disk group.
For VxVM 3.5 and earlier, use the following command:
# vxedit -rf rm rootdiskxNvol |
For VxVM 4.0, use the following command:
# vxedit -g rootdiskgroup |
Do not store data other than device entries for global devices in the global-devices file system. All data in the global-devices file system is destroyed when you remove the VxVM volume. Only data that is related to global devices entries is restored after the root disk is unencapsulated.
Unencapsulate the root disk.
Do not accept the shutdown request from the command.
# /etc/vx/bin/vxunroot |
See your VxVM documentation for details.
Use the format(1M) command to add a 512-Mbyte partition to the root disk to use for the global-devices file system.
Use the same slice that was allocated to the global-devices file system before the root disk was encapsulated, as specified in the /etc/vfstab file.
Set up a file system on the partition that you created in Step 9.
# newfs /dev/rdsk/cNtXdYsZ |
Determine the DID name of the root disk.
# scdidadm -l cNtXdY 1 phys-schost-1:/dev/rdsk/cNtXdY /dev/did/rdsk/dN |
In the /etc/vfstab file, replace the path names in the global-devices file system entry with the DID path that you identified in Step 11.
The original entry would look similar to the following.
# vi /etc/vfstab /dev/vx/dsk/rootdiskxNvol /dev/vx/rdsk/rootdiskxNvol /global/.devices/node@N ufs 2 no global |
The revised entry that uses the DID path would look similar to the following.
/dev/did/dsk/dNsX /dev/did/rdsk/dNsX /global/.devices/node@N ufs 2 no global |
Mount the global-devices file system.
# mount /global/.devices/node@N |
From one node of the cluster, repopulate the global-devices file system with device nodes for any raw-disk devices and Solstice DiskSuite or Solaris Volume Manager devices.
# scgdevs |
VxVM devices are recreated during the next reboot.
Reboot the node.
# reboot |
Repeat this procedure on each node of the cluster to unencapsulate the root disk on those nodes.