2.2 Cluster Volume Manager Utilities and Daemons

The following utilities and/or daemons have been created or modified for use with CVM:

• vxclust

• vxconfigd

• vxdg

• vxdisk

• vxrecover

• vxdctl

• vxstat

  The following sections contain information about how each of these utilities is used in a cluster environment. For further details on any of these utilities, refer to their manual pages.

  ---

  Note -

  Most volume manager commands require superuser privileges.

  ---

2.2.1 The vxclust Command

vxclust is not a portable utility. Although vxclust performs the same or similar operations in relation to the SSVM, it needs to be modified or rewritten to operate with each particular cluster manager.

vxclust is the volume manager cluster reconfiguration utility. vxclust coordinates changes in cluster membership and provides communication between the cluster manager and CVM. Whenever there is a cluster reconfiguration, vxclust is called by the cluster manager. vxclust notifies the volume manager kernel and the vxconfigd daemon whenever cluster reconfiguration takes place.

Every time there is a cluster reconfiguration, every node currently in the cluster runs the vxclust utility at each of several well-orchestrated steps. Cluster manager facilities ensure that the same step is executed on all nodes at the same time. A given step only starts when the previous one has completed on all nodes. At each step in the reconfiguration, vxclust determines what CVM should do next. After informing CVM of its next action, vxclust waits for the outcome (success, failure, or retry) and communicates that to the cluster manager.

If a node does not respond to a vxclust request within a specific timeout period, that node aborts. vxclust then decides whether to restart the reconfiguration or give up, depending on the circumstances. If the cause of the reconfiguration is a local, uncorrectable error, vxclust stops the action. If a node cannot complete an operation because another node has left, the surviving node times out. In this case, vxclust requests a reconfiguration with the expectation that another node will leave. If no other node leaves, vxclust will cause the local node to leave.

If a reconfiguration step fails, vxclust returns an error to the cluster manager. The cluster manager can decide to abort the node, causing its immediate departure from the cluster. Any I/O in progress to the shared disk fails and access to the shared disks is stopped.

vxclust decides what actions to take when it is informed of changes in the cluster. If a new master node is required (due to failure of the previous master), vxclust determines which node becomes the new master.

2.2.2 The vxconfigd Command

The volume manager configuration daemon, vxconfigd, maintains configurations of volume manager objects. vxconfigd receives cluster-related instructions from the vxclust utility. A separate copy of vxconfigd resides on each node; these copies communicate with each other through networking facilities. For each node in a cluster, Volume manager utilities communicate with the vxconfigd running on that particular node; utilities do not attempt to connect with vxconfigd daemons on other nodes. During startup of the cluster, vxclust tells vxconfigd to begin cluster operation and tells it whether it is a master or slave node.

When a node is initialized for cluster operation, vxclust tells the kernel and vxconfigd that the node is about to join the cluster and provides vxconfigd with the following information (from the cluster manager configuration database):

• Cluster ID

• Node IDs

• Master node ID

• Node's role

• Network address of the vxconfigd on each node

  On the master node, vxconfigd sets up the shared configuration (that is, imports the shared disk groups) and informs vxclust when it is ready for slaves to join.

  On slave nodes, vxclust tells the kernel and vxconfigd when the slave node can join the cluster. When the slave node joins the cluster, vxconfigd and the volume manager kernel communicate with their counterparts on the master in order to set up the shared configuration.

  When a node leaves the cluster, vxclust notifies the kernel and vxconfigd on all the other nodes. The master node then performs any necessary cleanup. If the master node leaves the cluster, vxclust chooses a new master node and notifies the kernel and vxconfigd on all nodes of the choice.

  vxconfigd also participates in volume reconfiguration. Refer to "2.1.2.3 Volume Reconfiguration", for information on vxconfigd's role in volume reconfiguration.

2.2.2.1 vxconfigd Recovery

The volume manager vxconfigd daemon can be stopped and/or restarted at any time. While vxconfigd is stopped, volume reconfigurations cannot take place and other nodes cannot join the cluster until vxconfigd is restarted. In the cluster, the vxconfigd daemons on the slaves are always connected to the vxconfigd daemon on the master. It is therefore not advisable to stop the vxconfigd daemon on any clustered node.

If vxconfigd is stopped for some reason, different actions are taken depending on which node has a stopped daemon:

• If vxconfigd is stopped on the slave(s), the master takes no action. When vxconfigd is restarted on the slave, the slave's vxconfigd attempts to reconnect to the master's and re-acquire the information about the shared configuration. (The view of the kernel of the shared configuration is unaffected, and so is access to the shared disks.) Until the slave vxconfigd has successfully rejoined to the master, it has very little information about the shared configuration and any attempts to display or modify the shared configuration can fail. In particular, if the shared disk groups are listed (using vxdg list), they will be marked as disabled; when the rejoin has completed successfully, they will be marked as enabled.

• If vxconfigd is stopped on the master, vxconfigd on the slave(s) attempts to rejoin to the master periodically. This will not succeed until vxconfigd is restarted on the master. In this case, the slave vxconfigd information about the shared configuration has not been lost, so configuration displays are accurate.

• If vxconfigd is stopped on both the master and the slave(s), the slave will not display accurate configuration information until vxconfigd has been restarted on both nodes and they have reconnected again.

  When vxclust notices that vxconfigd is stopped on a node, vxclust restarts vxconfigd.

  ---

  Note -

  With SSVM, the -r reset option to vxconfigd restarts vxconfigd and creates all states from scratch. This option is not available while a node is in the cluster because it causes the loss of cluster information; if this option is used under these circumstances, vxconfigd will not start.

  ---

2.2.3 The vxdg Command

The vxdg utility manages volume manager disk groups. You can use vxdg to specify that a disk group is cluster-sharable. The -s option to vxdg is provided to initialize or import a disk group as "shared."

If the cluster software has been run to set up the cluster, create a shared disk group with the following command:

vxdg -s init diskgroup [medianame=]accessname

where diskgroup is the disk group name; medianame is the administrative name chosen for the disk; and accessname is the disk access name (or device name).

Disk groups can be imported as shared using vxdg -s import. If the disk groups were set up before the cluster software was run, you can import the disk groups into the cluster arrangement with the command:

vxdg -s import diskgroup

where diskgroup is the disk group name or ID. On subsequent cluster restarts, the disk group will automatically be imported as shared. Note that it can be necessary to deport the disk group (using vxdg deport diskgroup) before invoking this command.

A disk group can be converted from shared to private by deporting it through vxdg deport and then importing it with vxdg import diskgroup.

The system cannot tell if a disk is shared. To protect data integrity when dealing with disks that can be accessed by multiple systems, an system administrator must be careful to use the correct designation when adding a disk to a disk group. If an administrator attempts to add a disk that is not physically shared to a shared disk group, the volume manager allows this on the node where the disk is accessible if that node is the only one in the cluster. However, other nodes will not be able to join the cluster. Furthermore, if an administrator attempts to add the same disk to different disk groups on two nodes at the same time, the results are undefined. All configurations should therefore be handled on one node only.

vxdg has a force option (-f) that can be used to force-import a disk group or force-add a disk to a disk group.

The force option (-f) should be used with caution and should be used only if the system administrator is fully aware of the possible consequences.

When a cluster is restarted, CVM can refuse to auto-import a disk group for one of the following reasons:

• A disk in that disk group is no longer accessible because of hardware errors on the disk. In this case, you can import the disk group again with the force option as follows:

  vxdg -s -f import diskgroup

• Some of the nodes to which disks in the disk group are attached are not currently in the cluster, so the disk group cannot access all of its disks. In this case, a forced import is unsafe and should not be attempted (because it can result in inconsistent mirrors).

  If CVM will not add a disk to an existing disk group (because that disk is not attached to the same node(s) as the other disks in the disk group), you can force-add the disk as follows:

  vxdg -f adddisk -g diskgroup [medianame=]accessname

vxdg can also be used to list shared disk groups. The following command displays one line of information for each disk group:

vxdg list

The output from this command is as follows:

NAME         STATE             ID
rootdg       enabled           774215886.1025.teal
group2       enabled,shared    774575420.1170.teal
group1       enabled,shared    774222028.1090.teal 

Shared disk groups are designated with the flag shared.

The following command displays one line of information for each shared disk group:

vxdg -s list

The output for this command is as follows:

NAME         STATE            ID
group2       enabled,shared   774575420.1170.teal
group1       enabled,shared   774222028.1090.teal 

The following command shows information about one specific disk group, including whether it is shared or not:

vxdg list diskgroup

where diskgroup is the disk group name.

The output for vxdg list group1 on the master (for the disk group group1) is as follows:

Group:     group1
dgid:      774222028.1090.teal
import-id: 32768.1749
flags:     shared
copies:    nconfig=default nlog=default
config:    seqno=0.1976 permlen=1456 free=1448 templen=6 loglen=220
config disk c1t0d0s2 copy 1 len=1456 state=clean online
config disk c1t1d0s2 copy 1 len=1456 state=clean online
log disk c1t0d0s2 copy 1 len=220
log disk c1t1d0s2 copy 1 len=220 

Note that the flags: field is set to shared. The output for the same command is slightly different on a slave.

2.2.4 The vxdisk Command

The vxdisk utility manages volume manager disks. You can use vxdisk to determine whether a disk is part of a cluster-shareable disk group, as follows:

vxdisk list accessname

where accessname is the disk access name (or device name).

The output from this command (for the device c1t0d0s2) is as follows:

Device:    c1t0d0s2
devicetag: c1t0d0
type:      sliced
clusterid:    cvm
disk:      name=disk01 id=774215890.1035.teal
group:     name=group1 id=774222028.1090.teal
flags:     online ready private autoconfig shared imported
pubpaths:  block=/dev/dsk/c1t0d0s4 char=/dev/rdsk/c1t0d0s4
privpaths: block=/dev/dsk/c1t0d0s3 char=/dev/rdsk/c1t0d0s3
version:   2.1
iosize:    min=512 (bytes) max=248 (blocks)
public:    slice=4 offset=0 len=2050272
private:   slice=3 offset=1 len=2015
update:    time=778564769 seqno=0.1614
headers:   0 248
configs:   count=1 len=1456
logs:      count=1 len=220
Defined regions:
 config   priv 000017-000247[000231]: copy=01 offset=000000 enabled
 config   priv 000249-001473[001225]: copy=01 offset=000231 enabled
 log      priv 001474-001693[000220]: copy=01 offset=000000 enabled 

Note that the clusterid: field is set to cvm (the name of the cluster) and the flags: field includes an entry for shared. When a node is not joined, the flags: field contains the autoimport flag instead of imported.

2.2.5 The vxrecover Command

The vxrecover utility recovers plexes and volumes after disk replacement.

When a node leaves the cluster, it can leave some mirrors in an inconsistent state. The vxrecover utility performs recovery on all volumes in this state. The -c option causes vxrecover to perform recovery for all volumes in cluster-shareable disk groups. vxclust automatically calls vxrecover -c, when necessary.

While vxrecover is active, there can be some degradation in system performance.

2.2.6 The vxdctl Command

The vxdctl utility manages some aspects of the volume configuration daemon, vxconfigd. The -c option can be used to request cluster information. You can use vxdctl as follows to determine whether vxconfigd is enabled and/or running:

vxdctl -c mode

Depending on the circumstances, the output is similar to the following:

mode: enabled: cluster active - MASTER

mode: enabled: cluster active - SLAVE

mode: enabled: cluster inactive

mode: enabled: cluster active - role not set 

If vxconfigd is disabled, no cluster information is displayed.

2.2.7 The vxstat Command

vxstat returns statistics for specified objects. In the CVM environment, vxstat gathers statistics from all of the nodes in the cluster. The statistics give the total usage, by all nodes, for the requested objects. If a local object is specified, its local usage is returned.

vxstat enables the caller to optionally specify a subset of nodes:

vxstat -g diskgroup -n node[,node...]

where node is an integer. If a comma-separated list of nodes is supplied, vxstat displays the sum of the statistics for the nodes in the list.

In the following example, vxstat is instructed to obtain statistics for Node 2, volume vol1:

vxstat -g rootdg -n 2 vol1

This can produce output similar to the following:

OPERATIONS           BLOCKS         AVG TIME(ms)
TYP  NAME          READ     WRITE      READ     WRITE    READ  WRITE
vol  vol1          2421         0    600000         0    99.0    0.0 

vxstat can also obtain and display statistics for the entire cluster, as follows:

vxstat -b

The statistics for all nodes are added together. For example, if Node 1 did 100 I/Os and Node 2 did 200 I/Os, vxstat -b would return 300.