test

Sun Cluster 2.2 Cluster Volume Manager Guide

Chapter 2 Cluster Volume Manager

This chapter provides an overview of Cluster Volume Manager (CVM), utilities and daemons, error messages, troubleshooting, and terminology used with CVM.

The following topics are covered in this document:

This document assumes that the system administrator has a general understanding of the Sun StorEdge Volume Manager (SSVM). Refer to the SSVM documentation set for more information.

2.1 Cluster Volume Manager Overview

Cluster Volume Manager (CVM) is an extension of SSVM that enables multiple hosts to simultaneously access and manage a given set of disks under volume manager control. A cluster is a set of hosts sharing a set of disks; each host is referred to as a node in the cluster. The nodes are connected across a network. If one node fails, the other node(s) can still access the disks. CVM presents the same logical view of the disk configurations (including changes) on all nodes.

Note -

CVM currently supports up to four nodes per cluster.

2.1.1 Shared Volume Manager Objects

When used with CVM, volume manager objects are capable of being shared by all of the nodes in a given cluster.

CVM allows for two types of disk groups:

With CVM, most disk groups are shared. However, the root disk group (rootdg) is always a private disk group.

The volume manager disks within a shared disk group are shared by all of the nodes in a cluster, so multiple nodes in a cluster can access a given volume manager disk simultaneously. Likewise, all volumes in a shared disk group can be shared by all nodes in a cluster. A volume manager disk or volume is considered shared when it is accessed by more than one node at the same time.

Note -

Only raw device access is performed through CVM. CVM does not support shared volumes with file systems. CVM does not currently support RAID5 volumes in cluster-shareable disk groups. RAID5 volumes can, however, be used in private disk groups attached to specific nodes of a cluster.

2.1.2 How Cluster Volume Management Works

CVM works together with an externally-provided cluster manager, which is a daemon that informs CVM of changes in cluster membership. Each node starts up independently and has its own copies of the operating environment, CVM, and the cluster manager. When a node joins a cluster, it gains access to shared disks. When a node leaves a cluster, it no longer has access to those shared disks. The system administrator joins a node to a cluster by starting the cluster manager on that node.

Figure 2-1 illustrates a simple cluster arrangement. All of the nodes are connected by a network. The nodes are then connected to a cluster-shareable disk group. To the cluster manager, all nodes are the same. However, CVM requires that one node act as the master node; the other nodes are slave nodes. The master node is responsible for coordinating certain volume manager activities. CVM software determines which node performs the master function (any node is capable of being a master node); this role only changes if the master node leaves the cluster. If the master leaves the cluster, one of the slave nodes becomes the new master. In this example, Node 1 is the master node and Node 2, Node 3, and Node 4 are the slave nodes.

Figure 2-1 Example of a Four-Node Cluster

Graphic

The system administrator designates a disk group as cluster-shareable using the vxdg utility. Refer to "2.2.3 The vxdg Command" for further information. Once a disk group is imported as cluster-shareable for one node, the disk headers are marked with the cluster ID. When other nodes join the cluster, they will recognize the disk group as being cluster-shareable and import it. The system administrator can import or deport a shared disk group at any time; the operation takes places in a distributed fashion on all nodes.

Each physical disk is marked with a unique disk ID. When the cluster starts up on the master, it imports all the shared disk groups (except for any that have the noautoimport attribute set). When a slave tries to join, the master sends it a list of the disk IDs it has imported and the slave checks to see if it can access all of them. If the slave cannot access one of the imported disks on the list, it abandons its attempt to join the cluster. If it can access all of the disks on the list, it imports the same set of shared disk groups as the master and joins the cluster. When a node leaves the cluster, it deports all its imported shared disk groups, but they remain imported on the surviving nodes. This is true if either type of node, master or slave, leaves the cluster.

Any reconfiguration to a shared disk group is performed with the cooperation of all nodes. Configuration changes to the disk group happen simultaneously on all nodes and the changes are identical. These changes are atomic in nature, so they either occur simultaneously on all nodes or do not occur at all.

All members of the cluster have simultaneous read and write access to any cluster-shareable disk group. Access by the active nodes of the cluster is not affected by a failure in any other node. The data contained in a cluster-shareable disk group is available as long as at least one node is active in the cluster. Regardless of which node accesses the cluster-shareable disk group, the configuration of the disk group looks the same. Applications running on each node can access the data on the volume manager disks simultaneously.

Note -

CVM does not protect against simultaneous writes to shared volumes by more than one node. It is assumed that any consistency control is done at the application level (using a distributed lock manager, for example).

2.1.2.1 Configuration and Initialization

Before any nodes can join a new cluster for the first time, the system administrator must supply certain configuration information. This information is supplied during cluster manager setup and is normally stored in some type of cluster manager configuration database. The precise content and format of this information is dependent on the characteristics of the cluster manager.

The type of information required by CVM is as follows:

2.1.2.2 Cluster Reconfiguration

Any time there is a change in the state of the cluster (in the form of a node leaving or joining), a cluster reconfiguration occurs. The cluster manager for each node monitors other nodes in the cluster and calls the cluster reconfiguration utility, vxclust, when there is a change in cluster membership. vxclust coordinates cluster reconfigurations and provides communication between CVM and the cluster manager. The cluster manager and vxclust work together to ensure that each step in the cluster reconfiguration is completed in the correct order.

During a cluster reconfiguration, I/O to shared disks is suspended. It is resumed when the reconfiguration completes. Applications can therefore appear to be frozen for a short time.

If other operations (such as volume manager operations or recoveries) are in progress, the cluster reconfiguration can be delayed until those operations have completed. Volume reconfigurations (described later) do not take place at the same time as cluster reconfigurations. Depending on the circumstances, an operation can be held up and restarted later. In most cases, cluster reconfiguration takes precedence. However, if the volume reconfiguration is in the commit stage, it will complete first.

For further information on cluster reconfiguration, refer to "2.2.1 The vxclust Command".

2.1.2.3 Volume Reconfiguration

Volume reconfiguration is the process of creating, changing, and removing the volume manager objects in the configuration (such as disk groups, volumes, mirrors, and so forth.). In a cluster, this process is performed with the cooperation of all nodes. Volume reconfiguration is distributed to all nodes; identical configuration changes occur on all nodes simultaneously.

The vxconfigd daemons play an active role in volume reconfiguration. For the reconfiguration to succeed, vxconfigd must be running on all nodes.

The reconfiguration is initiated and coordinated by the initiating node, which is the node on which the system administrator is running the utility that is requesting changes to volume manager objects.

The utility on the initiating node contacts its local vxconfigd daemon, which performs some local checking to make sure that a requested change is reasonable. For instance, it will fail an attempt to create a new disk group when one with the same name already exists. vxconfigd on the initiating node then sends messages with the details of the changes to the vxconfigd daemons on all other nodes in the cluster. The vxconfigds on each of the non-initiating nodes then perform their own checking. For example, a non-initiating node checks that it does not have a private disk group with the same name as the one being created; if the operation involves a new disk, each node checks that it can access that disk. When all of the vxconfigds on all the nodes agree that the proposed change is reasonable, each vxconfigd notifies its kernel and the kernels then cooperate to either commit or abort the transaction. Before the transaction can commit, all of the kernels ensure that no I/O is underway. The master is responsible for initiating a reconfiguration and coordinating the transaction commit.

If vxconfigd on any node goes away during a reconfiguration process, all nodes will be notified and the operation will fail. If any node leaves the cluster, the operation will fail unless the master has already committed the operation. If the master leaves the cluster, the new master (which was a slave previously) either completes or fails the operation. This depends on whether or not it received notification of successful completion from the previous master. This notification is done in such a way that if the new master did not receive it, neither did any other slave.

If a node attempts to join the cluster while a volume reconfiguration is being performed, the results depend on how far the reconfiguration has progressed. If the kernel is not yet involved, the volume reconfiguration is suspended and restarted when the join is complete. If the kernel is involved, the join waits until the reconfiguration is complete.

When an error occurs (such as when a check on a slave fails or a node leaves the cluster), the error is returned to the utility and a message is issued to the console on the initiating node to identify the node on which the error occurred.

2.1.2.4 Node Shutdown

The system administrator can shut down the cluster on a given node by invoking the cluster manager shutdown procedure on that node. This terminates cluster components after cluster applications have been stopped. CVM supports clean node shutdown, which is the ability of a node to leave the cluster gracefully when all access to shared volumes has ceased. The host is still operational, but cluster applications cannot be run on it.

CVM maintains global state information for each volume. This enables CVM to accurately determine which volumes need recovery when a node crashes. When a node leaves the cluster due to a crash or by some other means that is not clean, CVM determines which volumes can have writes that have not completed and the master resynchronizes those volumes. If Dirty Region Logging (DRL) is active for any of those volumes, it will be used.

Clean node shutdown should be used after, or in conjunction with, a procedure to halt all cluster applications. Depending on the characteristics of the clustered application and its shutdown procedure, it can be a long time before the shutdown is successful (minutes to hours). For instance, many applications have the concept of draining, where they accept no new work, but complete any work in progress before exiting. This process can take a long time if, for instance, a long-running transaction is active.

When the CVM shutdown procedure is invoked, it checks all volumes in all shared disk groups on the node that is being shut down and then either proceeds with the shutdown or fails:

2.1.2.5 Node Abort

If a node does not leave cleanly, it is either because the host crashed or because some cluster component decided to make the node leave on an emergency basis. The ensuing cluster reconfiguration calls the CVM abort function. This function makes an attempt to halt all access to shared volumes at once, though the operation waits until I/O that is at the disk completes. I/O operations that have not yet been started are failed, and the shared volumes are removed. Applications that were accessing the shared volumes, therefore, fail with errors.

After a node abort or crash, the shared volumes must be recovered (either by a surviving node or by a subsequent cluster restart) because it is very likely that there are unsynchronized mirrors.

2.1.2.6 Cluster Shutdown

When all the nodes in the cluster leave, the determination of whether or not the shared volumes should be recovered has to be made at the next cluster startup. If all nodes left cleanly, there is no need for recovery. If the last node left cleanly and resynchronization resulting from the non-clean leave of earlier nodes was complete, there is no need for recovery. However, recovery must be performed if the last node did not leave cleanly or if resynchronization from previous leaves was not complete.

2.1.3 Disks and CVM

The nodes in a cluster must always agree on the status of a disk. In particular, if one node cannot write to a given disk, all nodes must stop accessing that disk before the results of the write operation are returned to the caller. Therefore, if a node cannot contact a disk, it should contact another node to check on the disk's status. If the disk has failed, no node will be able to access it and the nodes can agree to detach the disk. If the disk has not failed, but rather the access paths from some of the nodes have failed, the nodes cannot agree on the status of the disk. Some policy must exist to resolve this type of discrepancy. The current policy is to detach the disk, even though it has not failed. While this resolves the status issue, it leaves mirrored disks with fewer mirrors and non-mirrored volumes with no access to data.

Note -

This policy is the current implementation option. In a future release of CVM, the system administrator will be able to select another policy. For example, an alternative policy would be for any node that cannot access the disk to leave the cluster; this maintains integrity of data access at the cost of keeping some nodes outside the cluster.

2.1.4 Dirty Region Logging and CVM

Dirty Region Logging (DRL) is an optional property of a volume that provides speedy recovery of mirrored volumes after a system failure. Dirty Region Logging is supported in cluster-shareable disk groups. This section provides a brief overview of DRL and outlines differences between SSVM DRL and the CVM implementation of DRL. For more information on DRL, refer to Chapter 1 in the applicable Sun StorEdge Volume Manager 2.6 System Administrator's Guide.

DRL keeps track of the regions that have changed due to I/O writes to a mirrored volume and uses this information to recover only the portions of the volume that need to be recovered. DRL logically divides a volume into a set of consecutive regions and maintains a dirty region log that contains a status bit representing each region of the volume. Log subdisks store the dirty region log of a volume that has DRL enabled. A volume with DRL has at least one log subdisk, that is associated with one of the volume's plexes.

Before writing any data to the volume, the regions being written are marked dirty in the log. If a write causes a log region to become dirty when it was previously clean, the log is synchronously written to disk before the write operation can occur. On system restart, the volume manager recovers only those regions of the volume which are marked as dirty in the dirty region log.

The CVM implementation of DRL differs slightly from the SSVM implementation. The following sections outline some of the differences and discuss some aspects of the CVM implementation.

2.1.4.1 Log Format and Size

As with SSVM, the CVM dirty region log exists on a log subdisk in a mirrored volume.

A SSVM dirty region log has a recovery map and a single active map. A CVM dirty region log, however, has one recovery map and multiple active maps (one for each node in the cluster). Unlike SSVM, CVM places its recovery map at the beginning of the log.

The CVM dirty region log size is typically larger than a SSVM dirty region log, as it must accommodate active maps for all nodes in the cluster plus a recovery map. The size of each map within the dirty region log is one or more whole blocks. vxassist automatically takes care of allocating a sufficiently large dirty region log.

The log size depends on the volume size and the number of nodes. The log must be large enough to accommodate all maps (one map per node plus a recovery map). Each map should be one block long for each two gigabytes of volume size. For a two-gigabyte volume in a two-node cluster, a log size of five blocks (one block per map) should be sufficient; this is the minimum log size. A four-gigabyte volume in a four-node cluster requires a log size of ten blocks, and so on.

2.1.4.2 Compatibility

Except for the addition of a CVM specific magic number, CVM DRL headers are the same as their SSVM counterparts.

It is possible to import a SSVM disk group (and its volumes) as a CVM shared disk group and vice versa. However, the dirty region logs of the imported disk group can be considered invalid and a full recovery can result.

If a CVM shared disk group is imported by a SSVM system, SSVM will consider the logs of the CVM volumes to be invalid and will conduct a full volume recovery. After this recovery completes, the volume manager will use the CVM Dirty Region Logging.

CVM is capable of performing a DRL recovery on a non-shared SSVM volume. However, if a SSVM volume is moved to a CVM system and imported as shared, the dirty region log will probably be too small to accommodate all the nodes in the cluster. CVM will therefore mark the log invalid and do a full recovery anyway. Similarly, moving a DRL volume from a two-node cluster to a four-node cluster will result in too small a log size, which CVM will handle with a full volume recovery. In both cases, the system administrator is responsible for allocating a new log of sufficient size.

2.1.4.3 Recovery With DRL

When one or more nodes in a cluster crash, DRL needs to be able to handle the recovery of all volumes in use by those nodes when the crash(es) occurred. On initial cluster startup, all active maps are incorporated into the recovery map; this is done during the volume start operation.

Nodes that crash (that is, leave the cluster as dirty) are not allowed to rejoin the cluster until their DRL active maps have been incorporated into the recovery maps on all affected volumes. The recovery utilities compare a crashed node's active maps with the recovery map and make any necessary updates before the node can rejoin the cluster and resume I/O to the volume (which overwrites the active map). During this time, other nodes can continue to perform I/O.

The CVM kernel tracks which nodes have crashed. If multiple node recoveries are underway in a cluster at a given time, their respective recoveries and recovery map updates can compete with each other. The CVM kernel therefore tracks changes in the DRL recovery state and prevents I/O operation collisions.

The master performs volatile tracking of DRL recovery map updates for each volume and prevents multiple utilities from changing the recovery map simultaneously.

2.2 Cluster Volume Manager Utilities and Daemons

The following utilities and/or daemons have been created or modified for use with CVM:

2.2.1 The vxclust Command

Note -

vxclust is not a portable utility. Although vxclust performs the same or similar operations in relation to the SSVM, it needs to be modified or rewritten to operate with each particular cluster manager.

vxclust is the volume manager cluster reconfiguration utility. vxclust coordinates changes in cluster membership and provides communication between the cluster manager and CVM. Whenever there is a cluster reconfiguration, vxclust is called by the cluster manager. vxclust notifies the volume manager kernel and the vxconfigd daemon whenever cluster reconfiguration takes place.

Every time there is a cluster reconfiguration, every node currently in the cluster runs the vxclust utility at each of several well-orchestrated steps. Cluster manager facilities ensure that the same step is executed on all nodes at the same time. A given step only starts when the previous one has completed on all nodes. At each step in the reconfiguration, vxclust determines what CVM should do next. After informing CVM of its next action, vxclust waits for the outcome (success, failure, or retry) and communicates that to the cluster manager.

If a node does not respond to a vxclust request within a specific timeout period, that node aborts. vxclust then decides whether to restart the reconfiguration or give up, depending on the circumstances. If the cause of the reconfiguration is a local, uncorrectable error, vxclust stops the action. If a node cannot complete an operation because another node has left, the surviving node times out. In this case, vxclust requests a reconfiguration with the expectation that another node will leave. If no other node leaves, vxclust will cause the local node to leave.

If a reconfiguration step fails, vxclust returns an error to the cluster manager. The cluster manager can decide to abort the node, causing its immediate departure from the cluster. Any I/O in progress to the shared disk fails and access to the shared disks is stopped.

vxclust decides what actions to take when it is informed of changes in the cluster. If a new master node is required (due to failure of the previous master), vxclust determines which node becomes the new master.

2.2.2 The vxconfigd Command

The volume manager configuration daemon, vxconfigd, maintains configurations of volume manager objects. vxconfigd receives cluster-related instructions from the vxclust utility. A separate copy of vxconfigd resides on each node; these copies communicate with each other through networking facilities. For each node in a cluster, Volume manager utilities communicate with the vxconfigd running on that particular node; utilities do not attempt to connect with vxconfigd daemons on other nodes. During startup of the cluster, vxclust tells vxconfigd to begin cluster operation and tells it whether it is a master or slave node.

When a node is initialized for cluster operation, vxclust tells the kernel and vxconfigd that the node is about to join the cluster and provides vxconfigd with the following information (from the cluster manager configuration database):

2.2.2.1 vxconfigd Recovery

The volume manager vxconfigd daemon can be stopped and/or restarted at any time. While vxconfigd is stopped, volume reconfigurations cannot take place and other nodes cannot join the cluster until vxconfigd is restarted. In the cluster, the vxconfigd daemons on the slaves are always connected to the vxconfigd daemon on the master. It is therefore not advisable to stop the vxconfigd daemon on any clustered node.

If vxconfigd is stopped for some reason, different actions are taken depending on which node has a stopped daemon:

2.2.3 The vxdg Command

The vxdg utility manages volume manager disk groups. You can use vxdg to specify that a disk group is cluster-sharable. The -s option to vxdg is provided to initialize or import a disk group as "shared."

If the cluster software has been run to set up the cluster, create a shared disk group with the following command:


vxdg -s init diskgroup [medianame=]accessname

where diskgroup is the disk group name; medianame is the administrative name chosen for the disk; and accessname is the disk access name (or device name).

Disk groups can be imported as shared using vxdg -s import. If the disk groups were set up before the cluster software was run, you can import the disk groups into the cluster arrangement with the command:


vxdg -s import diskgroup

where diskgroup is the disk group name or ID. On subsequent cluster restarts, the disk group will automatically be imported as shared. Note that it can be necessary to deport the disk group (using vxdg deport diskgroup) before invoking this command.

A disk group can be converted from shared to private by deporting it through vxdg deport and then importing it with vxdg import diskgroup.

Note -

The system cannot tell if a disk is shared. To protect data integrity when dealing with disks that can be accessed by multiple systems, an system administrator must be careful to use the correct designation when adding a disk to a disk group. If an administrator attempts to add a disk that is not physically shared to a shared disk group, the volume manager allows this on the node where the disk is accessible if that node is the only one in the cluster. However, other nodes will not be able to join the cluster. Furthermore, if an administrator attempts to add the same disk to different disk groups on two nodes at the same time, the results are undefined. All configurations should therefore be handled on one node only.

vxdg has a force option (-f) that can be used to force-import a disk group or force-add a disk to a disk group.

Note -

The force option (-f) should be used with caution and should be used only if the system administrator is fully aware of the possible consequences.

When a cluster is restarted, CVM can refuse to auto-import a disk group for one of the following reasons:

vxdg can also be used to list shared disk groups. The following command displays one line of information for each disk group:

vxdg list

The output from this command is as follows: 


NAME         STATE             ID
rootdg       enabled           774215886.1025.teal
group2       enabled,shared    774575420.1170.teal
group1       enabled,shared    774222028.1090.teal

Shared disk groups are designated with the flag shared.

The following command displays one line of information for each shared disk group:


vxdg -s list

The output for this command is as follows: 


NAME         STATE            ID
group2       enabled,shared   774575420.1170.teal
group1       enabled,shared   774222028.1090.teal

The following command shows information about one specific disk group, including whether it is shared or not:


vxdg list diskgroup

where diskgroup is the disk group name.

The output for vxdg list group1 on the master (for the disk group group1) is as follows: 


Group:     group1
dgid:      774222028.1090.teal
import-id: 32768.1749
flags:     shared
copies:    nconfig=default nlog=default
config:    seqno=0.1976 permlen=1456 free=1448 templen=6 loglen=220
config disk c1t0d0s2 copy 1 len=1456 state=clean online
config disk c1t1d0s2 copy 1 len=1456 state=clean online
log disk c1t0d0s2 copy 1 len=220
log disk c1t1d0s2 copy 1 len=220

Note that the flags: field is set to shared. The output for the same command is slightly different on a slave.

2.2.4 The vxdisk Command

The vxdisk utility manages volume manager disks. You can use vxdisk to determine whether a disk is part of a cluster-shareable disk group, as follows: 


vxdisk list accessname

where accessname is the disk access name (or device name).

The output from this command (for the device c1t0d0s2) is as follows: 


Device:    c1t0d0s2
devicetag: c1t0d0
type:      sliced
clusterid:    cvm
disk:      name=disk01 id=774215890.1035.teal
group:     name=group1 id=774222028.1090.teal
flags:     online ready private autoconfig shared imported
pubpaths:  block=/dev/dsk/c1t0d0s4 char=/dev/rdsk/c1t0d0s4
privpaths: block=/dev/dsk/c1t0d0s3 char=/dev/rdsk/c1t0d0s3
version:   2.1
iosize:    min=512 (bytes) max=248 (blocks)
public:    slice=4 offset=0 len=2050272
private:   slice=3 offset=1 len=2015
update:    time=778564769 seqno=0.1614
headers:   0 248
configs:   count=1 len=1456
logs:      count=1 len=220
Defined regions:
 config   priv 000017-000247[000231]: copy=01 offset=000000 enabled
 config   priv 000249-001473[001225]: copy=01 offset=000231 enabled
 log      priv 001474-001693[000220]: copy=01 offset=000000 enabled

Note that the clusterid: field is set to cvm (the name of the cluster) and the flags: field includes an entry for shared. When a node is not joined, the flags: field contains the autoimport flag instead of imported.

2.2.5 The vxrecover Command

The vxrecover utility recovers plexes and volumes after disk replacement.

When a node leaves the cluster, it can leave some mirrors in an inconsistent state. The vxrecover utility performs recovery on all volumes in this state. The -c option causes vxrecover to perform recovery for all volumes in cluster-shareable disk groups. vxclust automatically calls vxrecover -c, when necessary.

Note -

While vxrecover is active, there can be some degradation in system performance.

2.2.6 The vxdctl Command

The vxdctl utility manages some aspects of the volume configuration daemon, vxconfigd. The -c option can be used to request cluster information. You can use vxdctl as follows to determine whether vxconfigd is enabled and/or running: 


vxdctl -c mode

Depending on the circumstances, the output is similar to the following: 


mode: enabled: cluster active - MASTER

mode: enabled: cluster active - SLAVE

mode: enabled: cluster inactive

mode: enabled: cluster active - role not set
Note -

If vxconfigd is disabled, no cluster information is displayed.

2.2.7 The vxstat Command

vxstat returns statistics for specified objects. In the CVM environment, vxstat gathers statistics from all of the nodes in the cluster. The statistics give the total usage, by all nodes, for the requested objects. If a local object is specified, its local usage is returned.

vxstat enables the caller to optionally specify a subset of nodes: 


vxstat -g diskgroup -n node[,node...]

where node is an integer. If a comma-separated list of nodes is supplied, vxstat displays the sum of the statistics for the nodes in the list.

In the following example, vxstat is instructed to obtain statistics for Node 2, volume vol1: 


vxstat -g rootdg -n 2 vol1

This can produce output similar to the following: 


OPERATIONS           BLOCKS         AVG TIME(ms)
TYP  NAME          READ     WRITE      READ     WRITE    READ  WRITE
vol  vol1          2421         0    600000         0    99.0    0.0

vxstat can also obtain and display statistics for the entire cluster, as follows: 


vxstat -b

The statistics for all nodes are added together. For example, if Node 1 did 100 I/Os and Node 2 did 200 I/Os, vxstat -b would return 300.

2.3 Error Messages

This section presents error messages that can occur with CVM. Each message is accompanied by an explanation and a suggested user action.

Note -

Some of these messages can appear on the console; others are returned by vxclust. 


error in cluster processing

This can be due to an operation inconsistent with the current state of the cluster (such as an attempt to import or deport a shared disk group from the slave). It can also be caused by an unexpected sequence of commands from vxclust.

Make sure that the operation can be performed in the current environment.


cannot find disk on slave node

A slave node cannot find a shared disk. This is accompanied by the syslog message: 


vxvm:vxconfigd cannot find disk disk

Make sure that the same set of shared disks is online on both nodes.

Examine the disks on both the master and the slave with the command vxdisk list and make sure that the same set of disks with the shared flag is visible on both nodes. If not, check connections to the disks.


disk in use by another cluster

An attempt was made to import a disk group whose disks are stamped with the ID of another cluster.

If the disk group is not imported by another cluster, retry the import using the -C (clear import) flag. 


vxclust not there

An error during an attempt to join the cluster caused vxclust to fail. This can be caused by the failure of another node during a join or by the failure of vxclust.

Retry the join. An error message on the other node can clarify the problem.


unable to add portal for cluster

vxconfigd was not able to create a portal for communication with the vxconfigd on the other node. This can happen in a degraded system that is experiencing shortages of system resources (such as memory or file descriptors).

If the system does not appear to be degraded, stop and restart vxconfigd and try again.


vol recovery in progress

A node that crashed attempted to rejoin the cluster before its DRL map was merged into the recovery map.

Retry the join again later (when the merge operation has completed). 


cannot assign minor #

The slave attempted to join, but an existing volume on the slave has the same minor number as a shared volume on the master.

This message should be accompanied by the following console message: 


WARNING vxvm:vxconfigd minor number ### disk group group in use

Before retrying the join, use vxdg reminor diskgroup ### (see the vxdg(1M) manual page) to choose a new minor number range either for the disk group on the master or for the conflicting disk group on the slave. If there are open volumes in the disk group, the reminor operation will not take effect until the disk group is deported and updated either explicitly or through system restart.


master sent no data

During the slave join protocol, a message without data was received. This message is only likely to be seen in the case of a programming error.

Contact Customer Support for more information. 


join in progress

An attempt was made to import or deport a shared disk group during a cluster reconfiguration.

Retry later. 


join not allowed now

A slave attempted to join the cluster when the master was not ready. The slave will retry automatically. If the retry succeeds, the following message should appear: 


vxclust: slave join complete

No action is necessary if the join eventually completes. Otherwise, investigate the cluster monitor on the master. 


Disk reserved by other host

An attempt to put a disk online whose controller has been reserved by another host will fail with this error.

No action is necessary. The cluster manager will free the disk and the volume manager will put the disk online when the node joins the cluster.


vxvm:vxconfigd: group group exists

The slave tried to join the cluster, but there is already a shared disk group in the cluster with the same name as one of its private disk groups.

Use the vxdg newname operation to rename either the shared disk group on the master or the private disk group on the slave.


WARNING: vxvm:vxio: Plex plex detached from volume volume
NOTICE: vol_kmsg_send_wait_callback: got error 22
NOTICE: commit: NOTE: Reason found for abort: code=6

These messages can appear during a plex detach operation on a slave.

These messages provide information and require no user action.


WARNING: vxvm:vxio: read error on Plex plex of shared volume volume 
offset 10 \ length 1
WARNING: vxvm:vxio: Plex plex detached from volume volume 
NOTICE: commit: NOTE: Reason found for abort: code=2
NOTICE: ktcvm_check: sent to slave node: node=1 mid=196

These messages can appear during a plex detach operation on the master.

These messages provide information and require no user action.

2.4 Recovery

The basic recovery actions required for CVM are very similar to the recovery action required for SSVM under identical circumstances. However, when running CVM in clustered mode, additional steps are required and certain restrictions apply. The most significant difference is that recovery of shared disk groups must be initiated from the master node.

There are multiple ways to monitor the status of volume manager objects. The most frequently used mechanisms include vxprint command, VxVa (GUI) and vxnotify command. Most error conditions generate console messages or send messages to the messages file. CVM automatically recovers from some of events (for example, failure of one of the cluster nodes), through the Sun Cluster framework, other events need system administrator intervention. Recovery is done by using one or more of the following utilities: vxva, vxdiskadm, vxreattach, vxrecover, and/or vxvol. Some of these utilities internally call other utilities like vxdg, vxdisk, vxplex, and so on. To understand what is needed to recover from a particular situation you must have a solid understanding of volume manager utilities. For volume recovery and/or disk replacement procedures, refer to the Sun StorEdge Volume Manager 2.6 User's Guide and the Sun StorEdge Volume Manager 2.6 System Administrator's Guide.

The following section describes the process of recovering from some of the most commonly encountered situations.

2.4.1 Disk/Controller/Storage Device Failure

Failure of a disk, controller, or other storage device may make one or more devices inaccessible from one or more nodes. If a device was being accessed at the time of failure, that device is detached from the disk group. The data layout of a mirrored device should be such that no single failure can make both and/or all mirrors unavailable.

The first step of recovery is to make the failed device(s) accessible again, which includes:

  1. Rectify the fault condition (hardware and/or software) and make sure the devices are accessible again.

  2. Run vxdctl enable on all nodes of clusters.

  3. Run vxreattach on the master node.

  4. Run vxreattach on the other nodes that have non-shared disk groups.

  5. Verify (by running vxprint) that the devices have been reattached. (Under certain circumstances, vxreattach may not reattach a disk removed and/or replaced disks. These disks must be manually reattached using vxdg/vxdiskadm/vxva.

  6. Run vxrecover -sb on the master node.

  7. Run vxrecover -g <dg> -sb on another node with a non-shared disk group.

2.4.2 Recovery Examples

The following examples show some typical recovery situations. You start the recovery process by checking the operating mode of the cluster nodes; recovery must be performed on the master node (for a non-shared disk group, recovery must be performed on the node where the disk group was imported).


Root@capri:/# vxdctl -c mode
mode: enabled: cluster active - SLAVE
Root@palermo:/# vxdctl -c mode
mode: enabled: cluster active - MASTER

To check the available disk groups on both nodes, you can use vxdg list:


Root@capri:/# vxdg list
NAME   STATE          ID
rootdg enabled        885258939.1025.capri
test   enabled,shared 885331519.1233.palermo
Root@palermo:/# vxdg list
NAME   STATE          ID
rootdg enabled        885258917.1025.palermo
test   enabled,shared 885331519.1233.palermo

In this case there is one non-shared (rootdg) and one shared (test) disk group. The disk group ID of rootdg differs between the two hosts, even though the name is the same. Notice the state of the volume manager objects. For each object KSTATE and STATE, you can consider the object state and decide whether or not recovery is warranted.


vxprint -g test
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg test         test         -        -        -        -        -       -

dm disk1        c4t0d6s2     -        8379057  -        -        -       -
dm disk2        c5t0d6s2     -        8379057  -        -        -       -

v  test1        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test1-01     test1        ENABLED  132867   -        ACTIVE   -       -
sd c4t0d6s2-01  test1-01     ENABLED  132867   0        -        -       -
pl test1-02     test1        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-01  test1-02     ENABLED  132867   0        -        -       -

v  test2        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test2-01     test2        ENABLED  132867   -        ACTIVE   -       -
sd c4t0d6s2-02  test2-01     ENABLED  132867   0        -        -       -
pl test2-02     test2        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-02  test2-02     ENABLED  132867   0        -        -       -

If device c4t0d6s2 or all devices under controller c4 become unavailable, the device is detached from the disk group. The following example shows how the vxprint output looks after fault injection.


vxprint -g test
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg test         test         -        -        -        -        -       -

dm disk1        -            -        -        -        NODEVICE -       -
dm disk2        c5t0d6s2     -        8379057  -        -        -       -

v  test1        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test1-01     test1        DISABLED 132867   -        NODEVICE -       -
sd c4t0d6s2-01  test1-01     DISABLED 132867   0        NODEVICE -       -
pl test1-02     test1        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-01  test1-02     ENABLED  132867   0        -        -       -

v  test2        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test2-01     test2        DISABLED 132867   -        NODEVICE -       -
sd c4t0d6s2-02  test2-01     DISABLED 132867   0        NODEVICE -       -
pl test2-02     test2        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-02  test2-02     ENABLED  132867   0        -        -       -

Notice that the state of the DM entry, disk1, and the subdisk and plex using this disk is NODEVICE. In this case, the device should be reattached when it becomes accessible again. The vxdisk list output shows the state of the disk. If the device state has changed, you should run vxdctl enable before you run vxdisk list. 


vxdctl enable vxdisk list | grep c[45]t0d6s2
c4t0d6s2     sliced    -            -            error  shared
c5t0d6s2     sliced    disk2        test         online shared
-            -         disk1        test        failed was:c4t0d6s2

Notice that c5t0d6s2 is online, but c4t0d6s2 is in an error state. If the device was accessible to some nodes but not others, the vxdisk list output might differ between nodes (nodes that can still access the device will show it online). Now we can rectify the fault condition (in this case palermo lost connectivity to one of the SSAs; connection was restored later). At this point, running vxreattach is enough to reattach the devices.

Next, you can run vxdctl enable and verify that the devices are now accessible (device state is online). The following examples show the use of the vxdisk and vxdg commands. 


vxdctl enable
vxdisk -a online
vxdisk list | grep c[45]t0d6s2
c4t0d6s2     sliced    -            -            error  shared
c5t0d6s2     sliced    disk2        test         online shared
-            -         disk1        test         failed was:c4t0d6s2

The preceding listing shows that c4t0d6s2, which is no longer associated with any disk group, was associated with disk group test as DM disk1. You can reattach it with the command vxdg -g test -k adddisk disk1=c4t0d6s2, after you verify that disk1 is still disassociated, and c4t0d6s2 is the right disk (that is, it has not been swapped).


vxprint -d -g test -F "%name %nodarec %diskid"
disk1 on 882484157.1163.palermo
disk2 off 884294145.1517.palermo

The preceding listing shows the DM name to disk ID association. Since the nodarec attribute of disk1 is on, it is still disassociated. Disk 882484157.1163.palermo used to be associated with it. If you did not physically replace or move the disk, this disk ID should correspond to c4t0d6s2. If it was replaced by a new initialized disk, you may not find a matching disk ID. To verify the disk ID, you can run the command vxdisk -s list.


vxdisk -s list c4t0d6s2 c5t0d6s2
Disk:   c4t0d6s2
type:   sliced
flags:  online ready private autoconfig shared autoimport
diskid: 882484157.1163.palermo
dgname: test
dgid:   885331519.1233.palermo
clusterid: italia
Disk:   c5t0d6s2
type:   sliced
flags:  online ready private autoconfig shared autoimport imported
diskid: 884294145.1517.palermo
dgname: test
dgid:   885331519.1233.palermo
clusterid: italia

The preceding listing shows that disk c4t0d6s2 ID is 882484157.1163.palermo. Verifying the association this way is rather tedious. Fortunately, vxreattach (with the -c option) can show you the disk group and DM with which a disk should be reattached:


vxreattach -c  c4t0d6s2
test disk1

You can now associate the disk using the command vxdg -g test -k adddisk disk1=c4t0d6s2. Under most circumstances, running vxreattach takes care of all the preceding steps (running vxdctl enable and reattaching devices with respective disk groups). However, if a disk was removed administratively using the vxdiskadm command, or was physically replaced, it must be replaced using the vxdg command (option 5) or the VxVM GUI.


vxreattach -bv
! vxdg -g 885331519.1233.palermo -k adddisk disk1=c4t0d6s2

You can verify state change of the DM and plex using the vxprint command.


vxprint -g test
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg test         test         -        -        -        -        -       -

dm disk1        c4t0d6s2     -        8379057  -        -        -       -
dm disk2        c5t0d6s2     -        8379057  -        -        -       -

v  test1        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test1-01     test1        DISABLED 132867   -        IOFAIL   -       -
sd c4t0d6s2-01  test1-01     ENABLED  132867   0        -        -       -
pl test1-02     test1        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-01  test1-02     ENABLED  132867   0        -        -       -

v  test2        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test2-01     test2        DISABLED 132867   -        RECOVER  -       -
sd c4t0d6s2-02  test2-01     ENABLED  132867   0        -        -       -
pl test2-02     test2        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-02  test2-02     ENABLED  132867   0        -        -       -

You can recover the volume and plex now by specifying the -rb option of vxreattach to start vxrecover.


vxrecover -g test -vb
job 026404 dg test volume test1: reattach plex test1-01
ps -ef | grep plex
    root 26404 26403  1 13:58:04 ?        0:01 /usr/lib/vxvm/type/fsgen/vxplex -U fsgen -g 885331519.1233.palermo -- att test1
    root 26406   916  0 13:58:10 console  0:00 grep plex

Running in the background, vxrecover started vxplex to attach the plex to the volume (note the STALE state and ATT in tutil0). 


vxprint -g test               
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg test         test         -        -        -        -        -       -
dm disk1        c4t0d6s2     -        8379057  -        -        -       -
dm disk2        c5t0d6s2     -        8379057  -        -        -       -
v  test1        fsgen        ENABLED  131072   -        ACTIVE   ATT1    -
pl test1-01     test1        ENABLED  132867   -        STALE    ATT     -
sd c4t0d6s2-01  test1-01     ENABLED  132867   0        -        -       -
pl test1-02     test1        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-01  test1-02     ENABLED  132867   0        -        -       -
v  test2        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test2-01     test2        DISABLED 132867   -        RECOVER  -       -
sd c4t0d6s2-02  test2-01     ENABLED  132867   0        -        -       -
pl test2-02     test2        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-02  test2-02     ENABLED  132867   0        -        -       -
Root@palermo:/ # job 026404 done status=0
job 026408 dg test volume test2: reattach plex test2-01
job 026408 done status=0

After the volumes have been recovered, check the state of the devices again (KSTATE should be ENABLED and STATE should be ACTIVE):


vxprint -g test               
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg test         test         -        -        -        -        -       -
dm disk1        c4t0d6s2     -        8379057  -        -        -       -
dm disk2        c5t0d6s2     -        8379057  -        -        -       -
v  test1        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test1-01     test1        ENABLED  132867   -        ACTIVE   -       -
sd c4t0d6s2-01  test1-01     ENABLED  132867   0        -        -       -
pl test1-02     test1        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-01  test1-02     ENABLED  132867   0        -        -       -
v  test2        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test2-01     test2        ENABLED  132867   -        ACTIVE   -       -
sd c4t0d6s2-02  test2-01     ENABLED  132867   0        -        -       -
pl test2-02     test2        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-02  test2-02     ENABLED  132867   0        -        -       -

Now the recovery is complete. If the fault had occurred on the slave node rather than the master node, the behavior might vary slightly. Following fault injection the vxprint output is similar to the following listing:


vxprint -g test
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg test         test         -        -        -        -        -       -

dm disk1        c4t0d6s2     -        8379057  -        -        -       -
dm disk2        c5t0d6s2     -        8379057  -        -        -       -

v  test1        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test1-01     test1        DETACHED 132867   -        IOFAIL   -       -
sd c4t0d6s2-01  test1-01     ENABLED  132867   0        -        -       -
pl test1-02     test1        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-01  test1-02     ENABLED  132867   0        -        -       -

v  test2        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test2-01     test2        ENABLED  132867   -        ACTIVE   -       -
sd c4t0d6s2-02  test2-01     ENABLED  132867   0        -        -       -
pl test2-02     test2        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-02  test2-02     ENABLED  132867   0        -        -       -

Since the devices are not detached, running vxrecover on the master after the slave can access the disk again will be enough. However, if the disk is removed administratively, it must be added using vxdg/vxdiskadm/vxva (vxreattach does not work) and then recovered by using vxrecover.


vxdg -k rmdisk disk1
vxprint -g test
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg test         test         -        -        -        -        -       -
dm disk1        -            -        -        -        REMOVED  -       -
dm disk2        c5t0d6s2     -        8379057  -        -        -       -
v  test1        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test1-01     test1        DISABLED 132867   -        REMOVED  -       -
sd c4t0d6s2-01  test1-01     DISABLED 132867   0        REMOVED  -       -
pl test1-02     test1        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-01  test1-02     ENABLED  132867   0        -        -       -
v  test2        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test2-01     test2        DISABLED 132867   -        REMOVED  -       -
sd c4t0d6s2-02  test2-01     DISABLED 132867   0        REMOVED  -       -
pl test2-02     test2        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-02  test2-02     ENABLED  132867   0        -        -       -

Note that vxreattach reports any disk that it can reattach. However, you can reattach the disk manually as follows:


vxdg -g test -k adddisk disk1=c4t0d6s2
vxprint -g test
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg test         test         -        -        -        -        -       -

dm disk1        c4t0d6s2     -        8379057  -        -        -       -
dm disk2        c5t0d6s2     -        8379057  -        -        -       -

v  test1        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test1-01     test1        DISABLED 132867   -        RECOVER  -       -
sd c4t0d6s2-01  test1-01     ENABLED  132867   0        -        -       -
pl test1-02     test1        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-01  test1-02     ENABLED  132867   0        -        -       -
v  test2        fsgen        ENABLED  131072   -        ACTIVE   -       -
pl test2-01     test2        DISABLED 132867   -        RECOVER  -       -
sd c4t0d6s2-02  test2-01     ENABLED  132867   0        -        -       -
pl test2-02     test2        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-02  test2-02     ENABLED  132867   0        -        -       -
vxrecover -v -g test
job 026416 dg test volume test1: reattach plex test1-01
waiting...
job 026416 done status=0
job 026417 dg test volume test2: reattach plex test2-01
waiting...
job 026417 done status=0

The following example shows how you can reattach and recover:


# ps -ef | grep vx
    root 21935     1  1 20:10:31 ?        5:36 vxconfigd
    root 29295     1  0 14:29:11 ?        0:00 /usr/sbin/vxrecover -c -v -s
    root 29349     1  0 14:29:13 ?        0:00 /usr/sbin/vxrecover -c -v -s
    root 29399 29295  0 14:29:14 ?        0:00 /usr/lib/vxvm/type/fsgen/vxvol -U fsgen -g 885331519.1233.palermo -- resync tes
    root 29507 29399  0 14:29:16 ?        0:00 /usr/lib/vxvm/type/fsgen/vxvol -U fsgen -g 885331519.1233.palermo -- resync tes
    root 29508 29349  0 14:29:17 ?        0:00 /usr/lib/vxvm/type/fsgen/vxvol -U fsgen -g 885331519.1233.palermo -- resync tes
    root 29509 29508  0 14:29:17 ?        0:00 /usr/lib/vxvm/type/fsgen/vxvol -U fsgen -g 885331519.1233.palermo -- resync tes
    root 29511   916  0 14:29:21 console  0:00 grep vx
vxprint -g test
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg test         test         -        -        -        -        -       -
dm disk1        c4t0d6s2     -        8379057  -        -        -       -
dm disk2        c5t0d6s2     -        8379057  -        -        -       -
v  test1        fsgen        ENABLED  131072   -        SYNC     -       -
pl test1-01     test1        ENABLED  132867   -        ACTIVE   -       -
sd c4t0d6s2-01  test1-01     ENABLED  132867   0        -        -       -
pl test1-02     test1        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-01  test1-02     ENABLED  132867   0        -        -       -

v  test2        fsgen        ENABLED  131072   -        SYNC     -       -
pl test2-01     test2        ENABLED  132867   -        ACTIVE   -       -
sd c4t0d6s2-02  test2-01     ENABLED  132867   0        -        -       -
pl test2-02     test2        ENABLED  132867   -        ACTIVE   -       -
sd c5t0d6s2-02  test2-02     ENABLED  132867   0        -        -       -

Notice that the state of the volumes is now SYNC. Their state will be ACTIVE after vxplex completes.