Sun Cluster 2.2 System Administration Guide

Chapter 14 Administering Sun StorEdge A5000s

This chapter provides instructions for administering Sun StorEdge A5000 disks.

Use the service manual for your Sun StorEdge A5000 disks and your volume management software documentation when you are replacing or repairing disk hardware in the Sun Cluster configuration.

Recovering From Power Loss

When power is lost to one Sun StorEdge A5000, I/O operations generate errors that are detected by your volume management software. Errors are not reported until I/O transactions are made to the disk.

You should monitor the configuration for these events using the commands described in Chapter 2, Sun Cluster Administration Tools.

How to Recover From Power Loss (Solstice DiskSuite)

These are the high-level steps to recover from power loss to a disk enclosure in a Solstice DiskSuite environment:

Identifying the errored replicas
Returning the errored replicas to service
Identifying the errored devices
Returning the errored devices to service
Resyncing the disks

These are the detailed steps to recover from power loss to a disk enclosure in a Solstice DiskSuite environment.

When power is restored, use the metadb(1M) command to identify the errored replicas:
# metadb -s diskset

Return replicas to service.

After the loss of power, all metadevice state database replicas on the affected disk enclosure chassis enter an errored state. Because metadevice state database replica recovery is not automatic, it is safest to perform the recovery immediately after the disk enclosure returns to service. Otherwise, a new failure can cause a majority of replicas to be out of service and cause a kernel panic. This is the expected behavior of Solstice DiskSuite when too few replicas are available.

While these errored replicas will be reclaimed at the next takeover (haswitch(1M) or reboot(1M)), you might want to return them to service manually by first deleting and then adding them back.

Note -
Make sure that you add back the same number of replicas that were deleted on each slice. You can delete multiple replicas with a single metadb(1M) command. If you need multiple copies of replicas on one slice, you must add them in one invocation of the metadb(1M) command using the -c flag.

Use the metastat(1M) command to identify the errored metadevices.
# metastat -s diskset

Return errored metadevices to service using the metareplace(1M) command, and resync the disks.
# metareplace -s diskset -e mirror component
The -e option transitions the component (slice) to the available state and performs a resync.

Components that have been replaced by a hot spare should be the last devices replaced using the metareplace(1M) command. If the hot spare is replaced first, it could replace another errored submirror as soon as it becomes available.

You can perform a resync on only one component of a submirror (metadevice) at a time. If all components of a submirror were affected by the power outage, each component must be replaced separately. It takes approximately 10 minutes to resync a 1.05GB disk.

If both disksets in a symmetric configuration were affected by the power outage, you can resync each diskset's affected submirrors concurrently. Log into each host separately to recover that host's diskset by running metareplace(1M) on each.

Note -
Depending on the number of submirrors and the number of components in these submirrors, the resync actions can require a considerable amount of time. A single submirror made up of 30 1.05GB drives might take about five hours to complete. A more manageable configuration made up of five component submirrors might take only 50 minutes to complete.

How to Recover From Power Loss (VxVM)

Power failures can detach disk drives and cause plexes to become detached, and thus, unavailable. The volume remains active, however, because the remaining plexes in a mirrored volume are still available. It is possible to reattach the disk drives and recover from this condition without halting nodes in the cluster.

These are the high-level steps to recover from power loss to a disk enclosure in an VxVM configuration:

Determining the errored plex(es) by using the vxprint and vxdisk commands
Fixing the problem that caused the power loss
Using the drvconfig and disks commands to create the /devices and /dev entries
Scanning the current disk configuration
Reattaching disks that had transient errors
Verifying there are no more errors
(Optional) For shared disk groups, running the vxdg command for each disk that was powered off
Starting volume recovery

These are the detailed steps to recover from power loss to a disk enclosure in an VxVM configuration.

Use the vxprint command to view the errored plexes.

Optionally, specify a disk group with the -g diskgroup option.

Use the vxdisk command to identify the errored disks.

# vxdisk list
DEVICE       TYPE      DISK         GROUP        STATUS
..
-            -         c1t5d0       toi          failed was:c1t5d0s2
...

Fix the condition that resulted in the problem so that power is restored to all failed disks.

Be sure that the disks are spun up before proceeding.

Enter the following commands on all nodes in the cluster.

In some cases, the drive(s) must be rediscovered by the node(s).
# drvconfig # disks

Enter the following commands on all nodes in the cluster.

The volume manager must scan the current disk configuration again.
# vxdctl enable # vxdisk -a online

Enter the following command first on the master node, then on the remaining nodes in the cluster.

This will reattach disks that had transitory failures.
# vxreattach

Verify the output of the vxdisk command to see if there are any more errors.
# vxdisk list

If media was replaced, from the master node enter the following command for each disk that has been disconnected.

The physical disk and the volume manager access name for that disk must be reconnected.

# vxdg -g diskgroup -k adddisk medianame=accessname

The values for medianame and accessname appear at the end of the vxdisk list command output.

For example:

# vxdg -g toi -k adddisk c1t5d0=c1t5d0s2
# vxdg -g toi -k adddisk c1t5d1=c1t5d1s2
# vxdg -g toi -k adddisk c1t5d2=c1t5d2s2
# vxdg -g toi -k adddisk c1t5d3=c1t5d3s2
# vxdg -g toi -k adddisk c1t5d4=c1t5d4s2

You can also use the vxdiskadm command, or the graphical user interface, to reattach the disks.

From the node, start volume recovery.

If you have shared disk groups, use the -svc options to the vxrecover command.
# vxrecover -bv [-g diskgroup]

(Optional) Use the vxprint -g command to view the changes.

Administering Sun StorEdge A5000s

This section describes procedures for administering Sun StorEdge A5000 components. Use the procedures described in your server hardware manual to identify the failed component.

Repairing a Lost Sun StorEdge A5000 Connection

When a connection from a disk enclosure to one of the cluster nodes fails, the failure is probably due to a bad SCSI-2 cable or an SBus card.

In any event, the node on which the failure occurred will begin generating errors when the failure is discovered. Later accesses to the disk enclosure will generate additional errors. The node will exhibit the same behavior as though power had been lost to the disk enclosure. I/O operations from the other nodes in the cluster are unaffected by this type of failure.

To diagnose the failure, use the procedures for testing the card module in the service manual for your Sun Cluster node to determine which component failed. You should free up one node and the disk enclosure that appears to be down, for hardware debugging.

How to Repair a Lost Sun StorEdge A5000 Connection

Prepare the Sun Cluster system for component replacement.

Depending on the cause of the connection loss, prepare the Sun Cluster node with one of the following procedures.
- If the failed component is an SBus FC-100 host adapter, see Chapter 7, Administering Server Components, to prepare the Sun Cluster node for power down.
- If the problem is a bad FC-100 fiber optic cable, the volume management software will have detected the problem and prepared the system for cable replacement.

Replace the failed component.

If the FC-100 fiber optic cable or SBus FC-100 host adapter fails, refer to the Sun StorEdge A5000 Installation and Service Manual for detailed instructions on replacing them.

Recover from volume management software errors.

Use the procedures described in "Recovering From Power Loss".

This completes the procedure for repairing a lost connection.

Administering Sun StorEdge A5000 Disks

This section describes how to add and replace Sun StorEdge A5000 disks in a Sun Cluster configuration.

Adding or Replacing Sun StorEdge A5000 Disks

When adding or replacing Sun StorEdge A5000 disks, be sure to refer to the documentation that came with your system for more information.

Note -

When replacing a failed A5000 disk under VxVM control, you cannot simply pull out the disk and replace it with a new one. This is because each disk has a unique World Wide Name (WWN). See "Replacing a SPARCstorage Array Controller and Changing the World Wide Name", for more information about the WWN.

How to Add a Sun StorEdge A5000 Disk (Solstice DiskSuite)

Use the luxadm command to insert the new disk.

Physically install the new disk or disks when prompted. Repeat for each node that is physically connected to the array.
# luxadm insert enclosure.slot

Insert the new disk drive and enter Return.

Use format to create a disk label and repartition, if needed.

Use scdidadm(1M) to discover the new disk and create a DID instance for it.

This should be run from node1 only. See the scdidadm(1M) man page for details.
# scdidadm -r -H node2,node3...

This completes the disk addition procedure.

How to Add a Sun StorEdge A5000 Disk (VxVM)

Use the luxadm command to prepare the loop for a new device.

Physically install the new disk or disks when prompted.
# luxadm insert

Notify VxVM of the new disk.
# vxdctl enable

Use the vxdiskadm command to bring the new disk(s) into VxVM control.

Enter 1 (Add or initialize one or more disks).

This completes the disk addition procedure.

How to Replace a Sun StorEdge A5000 Disk (Solstice DiskSuite)

Identify all metadevices or applications using the failing disk.

If the metadevices are mirrored or RAID5, the disk can be replaced without stopping the metadevices. Otherwise all I/O to the disk must be stopped using the appropriate commands. For example, use the umount(1M) command to unmount a file system on a stripe or concatenation.

Preserve the disk label, if necessary.

For example:
# prvtoc /dev/rdsk/c1t3d0s2 > /tmp/c1t3d0.vtoc

(optional) Use metareplace to replace the disk slices if the disk has not been hot-spared.

For example:
# metareplace d1 c1t3d0s2 c1t2d0s2 d1: device c1t3d0s2 is replaced with c1t2d0s2

Use luxadm -F to remove the disk.

The -F option is required because Solstice DiskSuite does not offline disks. Repeat the command for all hosts, if the disk is multihosted. For example:

# luxadm remove -F /dev/rdsk/c1t3d0s2
WARNING!!! Please ensure that no filesystems are mounted on these device(s).  All data on these devices should have been backed up.The list of devices which will be removed is:  

1: Box Name "macs1" rear slot 1

Please enter `q' to Quit or <Return> to Continue: stopping:  Drive in "macs1" rear  slot 1....Done

offlining: Drive in "macs1" rear  slot 1....Done

Hit <Return> after removing the device(s).

Note -

The FPM icon for the disk drive to be removed should be blinking. The amber LED under the disk drive should also be blinking.

Remove the disk drive and enter Return.

The output should look similar to the following:

Hit <Return> after removing the device(s).  

Drive in Box Name "macs1" rear slot 1    

Removing Logical Nodes:  

Removing c1t3d0s0 Removing c1t3d0s1 Removing c1t3d0s2 Removing c1t3d0s3 Removing c1t3d0s4 Removing c1t3d0s5 Removing c1t3d0s6 Removing c1t3d0s7 Removing c2t3d0s0 Removing c2t3d0s1 Removing c2t3d0s2 Removing c2t3d0s3 Removing c2t3d0s4 Removing c2t3d0s5 Removing c2t3d0s6 Removing c2t3d0s7
#

Repeat Step 4 for all nodes, if the disk array is in a multi-host configuration.

Use the luxadm insert command to insert the new disk.

Repeat for all nodes. The output should be similar to the following:

# luxadm insert macs1,r1
The list of devices which will be inserted is:  

1: Box Name "macs1" rear slot 1

Please enter `q' to Quit or <Return> to Continue: Hit <Return> after inserting the device(s).

Insert the disk drive and enter Return.

The output should be similar to the following:

Hit <Return> after inserting the device(s).  Drive in Box Name "macs1" rear slot 1  Logical Nodes under /dev/dsk and /dev/rdsk : c1t3d0s0 c1t3d0s1 c1t3d0s2 c1t3d0s3 c1t3d0s4 c1t3d0s5 c1t3d0s6 c1t3d0s7 c2t3d0s0 c2t3d0s1 c2t3d0s2 c2t3d0s3 c2t3d0s4 c2t3d0s5 c2t3d0s6 c2t3d0s7#

Note -

The FPM icon for the disk drive you replaced should be lit. In addition, the green LED under the disk drive should be blinking.

Use scdidadm(1M) to update the DID pseudo device information.

On all nodes connected to the disk, execute the following command to update new Disk ID information.
# scdidadm -R DID_instance
where DID_instance is the instance number of the disk that was replaced. Refer to the scdidadm(1M) man page for more information.

Reboot all nodes connected to the new disk.

To avoid down time, use the haswitch(1M) command to switch ownership of all logical hosts that can be mastered by the node to be rebooted. For example,
# haswitch phys-hahost2 hahost1 hahost2

Label the disk, if necessary.

For example:

# cat /tmp/c1t3d0.vtoc | fmthard -s - /dev/rdsk/c1t3d0s2
fmthard:  New volume table of contents now in place.

Replace the metadb, if necessary.

For example:
# metadb -d c1t3d0s0; metadb -a c1t3d0s0

Enable the new disk slices with metareplace -e.

For example:
# metareplace -e d0 c1t3d0s0 d0: device c1t3d0s0 is enabled
This completes the disk replacement procedure.

How to Replace a Sun StorEdge A5000 Disk (VxVM)

Identify all volumes or applications using the failing disk.

If the volumes are mirrored or RAID5, the disk can be replaced without stopping the volume. Otherwise all I/O to the disk must be stopped using the appropriate commands. For example, use the umount(1M) command to unmount a file system on a stripe or concatenation.

Use the vxdiskadm command to replace and offline a disk device.

For VxVM, perform these commands on the machine mastering the logical host owning the disk group.

Enter 4 (Remove a disk for replacement), then enter 11 (Disable [offline] a disk device).

You can use the graphical user interface instead, if you prefer it.

Use the luxadm command to remove the device and device nodes.

This command is interactive and will prompt you to physically remove the disk. Perform the command on every node connected to the array. For example:
# luxadm remove_device -F /dev/rdsk/c2t20d0s2

Physically replace the disk, then use the luxadm command to insert the new disk.

This creates the new device and device nodes. Perform the command on every node connected to the array. For example:
# luxadm insert_device ratbert,r4

Notify the volume manager of the new disk.
# vxdctl enable

Use the vxdiskadm command to bring the new disk under VxVM control.

Enter 5 (Replace a failed or removed disk).

(Optional) The volume can now be restored if necessary.

This completes the disk replacement procedure.