Using VxVM in the Sun Cluster Environment
VERITAS Volume Manager (VxVM) and the VxVM cluster feature are variations of the same volume manager. The VxVM cluster feature is only used in Oracle Parallel Server (OPS) configurations. This section describes using disks under the control of the volume manager to administer:
-
Volume manager disks
-
Disk groups
-
Subdisks
-
Plexes
-
Volumes
Refer to the appropriate section for a complete discussion of administering these objects.
Objects Administration Overview (VxVM)
Objects under the control of a volume manager are created and administered using either command-line utilities or the Visual Administrator graphical user interface.
Read the information in this chapter before using the VxVM documentation to administer objects under the control of a volume manager in a Sun Cluster configuration. The procedures presented here are one method for performing the following tasks. Use the method that works best for your particular configuration.
These objects generally have the following relationship:
-
Disks are placed under volume manger control and are grouped into disk groups.
-
One or more subdisks (each representing a specific portion of a disk) are combined to form plexes, or mirrors.
-
A volume is composed of one or more plexes.
The default disk group is rootdg (the root disk group). You can create additional disk groups as necessary. The primary administration tasks that you perform on disk groups involve adding and removing disks.
Before using a disk that you have placed in a disk group, you must set up disks and subdisks (under volume manager control) to build plexes, or mirrors, using the physical disk's slices. A plex can be a concatenation or stripe.
With VxVM, applications access volumes (created on volume manager disks) rather than slices.
The following sections describe the VxVM command-line programs to use when performing a task. Optionally, you can use the graphical user interface for all the tasks unless directed otherwise.
Note -
On nodes running Sun Cluster HA data services, never manually run the vxdg import or deport options on a disk group that is under the control of Sun Cluster, unless the logical host for that disk group is in maintenance mode. Before manually importing or deporting a disk group, you must either stop Sun Cluster on all nodes that can master that disk group (by running scadmin stopnode on all such nodes), or use the haswitch -m command to switch any corresponding logical host into maintenance mode. When you are ready to return control of the disk group to Sun Cluster, the safest course is to deport the disk group before running scadmin startnode or before using haswitch(1M) to place the logical host back under the control of Sun Cluster.
Administering Disks
Before a disk can be used by VxVM, it must be identified, or initialized, as a disk that is under control of a volume manager. A fully initialized disk can be added to a disk group, used to replace a previously failed disk, or used to create a new disk group.
How to Initialize and Configure a Disk (VxVM)
-
Ensure that no data is on the disk.
This is important because existing data is destroyed if the disk is initialized.
-
Insert the disk device and install it in the disk enclosure by following the instructions in the accompanying hardware documentation.
-
Initialize the disk and add it to a disk group.
This is commonly done by using either the vxdiskadm menus or the graphical user interface. Alternately, you can use the command line utilities vxdisksetup and vxdg addisk to initialize the disk and place it in a disk group.
Taking a Disk Offline
Occasionally, you may need to take a physical disk offline. If the disk is corrupted, you need to disable it and remove it. You also must disable a disk before moving the physical disk device to another location to be connected to another system.
To take a physical disk offline, first remove the disk from its disk group. Then place the disk offline by using the vxdisk(1M) command.
Removing a Disk
You can remove a disk to move it to another system, or you may remove the disk because the disk is failing or has failed. Alternatively, if the volumes are no longer needed, they can be removed.
To remove a disk from the disk group, use the vxdg(1M) command. To remove the disk from volume manager control by removing the private and pubic partitions, use the vxdiskunsetup(1M) command. Refer to the vxdg(1M) and vxdiskunsetup(1M) man pages for complete information on these commands.
Administering Disk Groups
For VxVM, it is most convenient to create and populate disk groups from the active node that is the default master of the particular disk group. In an N+1 configuration, each of these default master nodes shares multihost disk connectivity with only one other node in the cluster, the hot-standby node. By using these nodes to populate the disk groups, you avoid the risk of generating improperly configured groups.
Creating a Disk Group (VxVM)
You can use either the vxdiskadm menus or the graphical user interface to create a new disk group. Alternately, you can use the command-line utility vxdg init.
Once the disk groups have been created and populated, each one should be deported by using the vxdg deport command. Then, each group should be imported onto the hot-standby node by using the -t option. The -t option is important, as it prevents the import from persisting across the next boot. All VxVM plexes and volumes should be created, and volumes started, before continuing.
Moving a Disk to a Different Disk Group (VxVM)
Use the following procedure to move a disk to a different disk group.
How to Move a Disk to a Different Disk Group (VxVM)
To move a disk between disk groups, remove the disk from one disk group and add it to the other.
This example moves the physical disk c1t0d1 from disk group acct to disk group log_node1 by using command-line utilities.
-
Use the vxprint(1M) command to determine if the disk is in use.
# vxprint -g acct TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0 dg acct acct - - - - - - dm c1t0d0 c1t0d0s2 - 2050272 - - - - dm c1t0d1 c1t0d1s2 - 2050272 - - - - dm c2t0d0 c2t0d0s2 - 2050272 - - - - dm c2t0d1 c2t0d1s2 - 2050272 - - - - v newvol gen ENABLED 204800 - ACTIVE - - pl newvol-01 newvol ENABLED 205632 - ACTIVE - - sd c1t0d1-01 newvol-01 ENABLED 205632 0 - - - pl newvol-02 newvol ENABLED 205632 - ACTIVE - - sd c2t0d1-01 newvol-02 ENABLED 205632 0 - - - v vol01 gen ENABLED 1024000 - ACTIVE - - pl vol01-01 vol01 ENABLED 1024128 - ACTIVE - - sd c1t0d0-01 vol01-01 ENABLED 1024128 0 - - - pl vol01-02 vol01 ENABLED 1024128 - ACTIVE - - sd c2t0d0-01 vol01-02 ENABLED 1024128 0 - - -
-
Use the vxedit(1M) command to remove the volume to free up the c1t0d1 disk.
You must run the vxedit command from the node mastering the shared disk group.
# vxedit -g acct -fr rm newvol
The -f option forces an operation. The -r option makes the operation recursive.
-
Remove the c1t0d1 disk from the acct disk group.
You must run the vxdg command from the node mastering the shared disk group.
# vxdg -g acct rmdisk c1t0d1
-
Add the c1t0d1 disk to the log_node1 disk group.
# vxdg -g log_node1 adddisk c1t0d1
Caution - This procedure does not save the configuration or data on the disk.
This is the acct disk group after c1t0d1 is removed.
# vxprint -g acct TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0 dg acct acct - - - - - - dm c1t0d0 c1t0d0s2 - 2050272 - - - - dm c2t0d0 c2t0d0s2 - 2050272 - - - - dm c2t0d1 c2t0d1s2 - 2050272 - - - - v vol01 gen ENABLED 1024000 - ACTIVE - - pl vol01-01 vol01 ENABLED 1024128 - ACTIVE - - sd c1t0d0-01 vol01-01 ENABLED 1024128 0 - - - pl vol01-02 vol01 ENABLED 1024128 - ACTIVE - - sd c2t0d0-01 vol01-02 ENABLED 1024128 0 - - -
This is the log_node1 disk group after c1t0d1 is added.
# vxprint -g log_node1 TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0 dg log_node1 log_node1 - - - - - - dm c1t0d1 c1t0d1s2 - 2050272 - - - - dm c1t3d0 c1t3d0s2 - 2050272 - - - - dm c2t3d0 c2t3d0s2 - 2050272 - - - - #
To change permissions or ownership of volumes, you must use the vxedit command.
Caution - Do not use chmod or chgrp. The permissions and ownership set by chmod or chgrp are automatically reset to root during a reboot.
Here is an example of the permissions and ownership of the volumes vol01 and vol02 in the /dev/vx/rdsk directory before a change.
# ls -l crw------- 1 root root nnn,nnnnn date time vol01 crw------- 1 root root nnn,nnnnn date time vol02 ...
This an example for changing the permissions and ownership for vol01.
# vxedit -g group_name set mode=755 user=oracle vol01
After the edit, note how the permissions and ownership have changed.
# ls -l crwxr-xr-x 1 oracle root nnn,nnnnn date time vol01 crw------- 1 root root nnn,nnnnn date time vol02 ...
Administering VxVM Objects
Volumes, or virtual disks, can contain file systems or applications such as databases. A volume can consist of up to 32 plexes, each of which contains one or more subdisks. In order for a volume to be usable, it must have at least one associated plex with at least one associated subdisk. Note that all subdisks within a volume must belong to the same disk group.
Creating Volumes and Adding Mirrors to Volumes
Use the graphical user interface or the command-line utility vxassist(1M) to create volumes in each disk group, and to create an associated mirror for each volume.
The actual size of a VxVM device is slightly less than the full disk drive size. VxVM reserve a small amount of space for private use, called the private region.
Note -
The use of the same volume name is allowed if the volumes belong to different disk groups.
Adding Dirty Region Logging
Dirty Region Logging (DRL) is an optional property of a volume, used to provide a speedy recovery of mirrored volumes after a system failure. DRL keeps track of the regions that have changed due to I/O writes to a mirrored volume and uses this information to recover only the portions of the volume that need to be recovered.
Creating a Log File for an Existing Volume
Log subdisks are used to store the dirty region log of a volume that has DRL enabled. A volume with DRL has at least one log subdisk; multiple log subdisks can be used to mirror the dirty region log. Each log subdisk is associated with one of the volume's plexes. Only one log subdisk can exist per plex. If the plex contains only a log subdisk and no data subdisks, that plex can be referred to as a log plex. The log subdisk can also be associated with a regular plex containing data subdisks, in which case the log subdisk risks becoming unavailable in the event that the plex must be detached due to the failure of one of its data subdisks.
Use the graphical user interface or the command-line utility vxassist(1M) to create a log for an existing volume.
Using Hot-Relocation
Hot-relocation is the ability of a system to automatically react to I/O failures on redundant (mirrored or RAID5) volume manager objects, and to restore redundancy and access to those objects. Hot-relocation is supported only on configurations using VxVM. VxVM detects I/O failures on volume manager objects and relocates the affected subdisks to disks designated as spare disks or free space within the disk group. VxVM then reconstructs the objects that existed before the failure and makes them redundant and accessible again.
When a partial disk failure occurs (that is, a failure affecting only some subdisks on a disk), redundant data on the failed portion of the disk is relocated, and the existing volumes consisting of the unaffected portions of the disk remain accessible.
Note -
Hot-relocation is performed only for redundant (mirrored or RAID5) subdisks on a failed disk. Non-redundant subdisks on a failed disk are not relocated, but you are notified of their failure.
A spare disk must be initialized and placed in a disk group as a spare before it can be used for replacement purposes. If no disks have been designated as spares when a failure occurs, VxVM automatically uses any available free space in the disk group in which the failure occurs. If there is not enough spare disk space, a combination of spare space and free space is used. You can designate one or more disks as hot-relocation spares within each disk group. Disks can be designated as spares with the vxedit(1M) command.
Using VxFS File Systems
You can configure and specify either UFS or VxFS file systems associated with a logical host's disk groups on volumes of type fsgen. When a cluster node masters a logical host, the logical host's file systems associated with the disk groups are mounted on the mastering node's specified mount points.
During a logical host reconfiguration sequence, it is necessary to check file systems with the fsck(1M) command. Though this process is performed in non-interactive parallel mode on UFS file systems, it can affect the overall time of the reconfiguration sequence. The logging feature of UFS, SDS, and VxFS file systems greatly reduce the time that fsck(1M) takes prior to mounting file systems.
When the switchover of a data service is required along with volume recovery, the recovery takes longer than allowed in the reconfiguration steps. This causes step time-outs and the node aborts.
Consequently, when setting up mirrored volumes, always add a DRL log to decrease volume recovery time in the event of a system crash. When mirrored volumes are used in the cluster environment, DRL must be assigned for volumes greater than 500 Mbytes.
Use VxFS if large file systems (greater than 500 Mbytes) are used for HA data services. Under most circumstances, VxFS is not bundled with Sun Cluster and must be purchased separately from VERITAS.
Note -
Although it is possible to configure logical hosts with very small mirrored file systems, you should use Dirty Region Logging (DRL) or VxFS file systems because of the possibility of time-outs as the size of the file system increases.
Growing a File System
To grow a striped or RAID5 volume containing a file system, you must have the free space on the same number of disks that are currently in the stripe or RAID5 volume. For example, if you have four 1GB disks striped together (giving you a 4GB file system), and you wish to add 1GB of space (to yield a 5GB filesystem), you must have four new disks, each with at least .25GB of free space. In other words, you can not add one disk to a 4-disk stripe.
The VxVM graphical user interface will choose the disks on which to grow your file system. To select the specific disks on which to grow the file system, use the command line interface instead.
UFS file systems cannot be shrunk. The only way to "shrink" a file system is to recreate the volume, run newfs on the volume, and then restore the data from backup.
Administering Local Mirrors
Local disks can be mirrored. If a single mirror fails, use the instructions in your volume manager documentation to replace the failed mirror and resynchronize the replacement disk with the good disk.
- © 2010, Oracle Corporation and/or its affiliates