test

Sun Cluster 2.2 System Administration Guide

Chapter 11 Administering SPARCstorage Arrays

This chapter provides instructions for administering SPARCstorage Array Model 100 Series, SPARCstorage Array Model 200 Series with differential SCSI, and SPARCstorage Array Model 200 Series with RSM disk trays.

This chapter includes the following procedures:

Use the service manual for your SPARCstorage Array and your volume manager documentation when replacing or repairing SPARCstorage Array hardware in the Sun Cluster configuration.

11.1 Recovering From Power Loss

When power is lost to one SPARCstorage Array, I/O operations generate errors that are detected by the volume management software. Errors are not reported until I/O transactions are made to the disk. Hot spare activity can be initiated if affected devices are set up for hot sparing.

You should monitor the configuration for these events. See Chapter 2, Sun Cluster Administration Tools, for more information on monitoring the configuration.

11.1.1 How to Recover From Power Loss (Solstice DiskSuite)

These are the high-level steps to recover from power loss to a SPARCstorage Array in a Solstice DiskSuite configuration:

These are the detailed steps to recover from power loss to a SPARCstorage Array in a Solstice DiskSuite configuration.

  1. When power is restored, run the metadb(1M) command to identify the errored replicas.

    # metadb -s diskset

  2. Return replicas to service.

    After the loss of power, all metadevice state database replicas on the affected SPARCstorage Array chassis enter an errored state. Because metadevice state database replica recovery is not automatic, it is safest to perform the recovery immediately after the SPARCstorage Array returns to service. Otherwise, a new failure can cause a majority of replicas to be out of service and cause a kernel panic. This is the expected behavior of Solstice DiskSuite when too few replicas are available.

    While these errored replicas will be reclaimed at the next takeover (haswitch(1M) or reboot(1M)), it is best to return them to service manually by first deleting them, then adding them back.

    Note -

    Make sure that you add back the same number of replicas that were deleted on each slice. You can delete multiple replicas with a single metadb(1M) command. If you need multiple copies of replicas on one slice, you must add them in one invocation of metadb(1M) using the -c flag.

  3. Run the metastat(1M) command to identify the errored metadevices.

    # metastat -s diskset

  4. Return errored metadevices to service by using the metareplace(1M) command, which will cause a resync of the disks. 

    # metareplace -s diskset -e mirror component

    The -e option transitions the component (slice) to the Available state and performs a resync.

    Components that have been replaced by a hot spare should be replaced last by using the metareplace(1M) command. If the hot spare is replaced first, it could replace another errored submirror as soon as it becomes available.

    You can perform a resync on only one component of a submirror (metadevice) at a time. If all components of a submirror were affected by the power outage, each component must be replaced separately. It takes approximately 10 minutes to resync a 1.05GB disk.

    If more than one diskset was affected by the power outage, you can resync each diskset's affected submirrors concurrently. Log into each host separately to recover that host's diskset by running the metareplace(1M) command on each.

    Note -

    Depending on the number of submirrors and the number of components in these submirrors, the resync actions can require a considerable amount of time. A single submirror made up of 30 1.05GB drives might take about five hours to complete. A more manageable configuration made up of five component submirrors might take only 50 minutes to complete.

11.1.2 How to Recover From Power Loss (SSVM or CVM)

Power failures can detach disk drives and cause plexes to become detached, and thus, unavailable. In a mirror, however, the volume remains active, because the remaining plex(es) in the volume are still available. It is possible to reattach the disk drives and recover from this condition without halting nodes in the cluster.

These are the high-level steps to recover from power loss to a SPARCstorage Array in an SSVM configuration:

These are the detailed steps to recover from power loss to a SPARCstorage Array in an SSVM configuration.

  1. Run the vxprint command to view the errored plexes.

    Optionally, specify a disk group with the -g diskgroup option.

  2. Run the vxdisk command to identify the errored disks.

    # vxdisk list
 DEVICE       TYPE      DISK         GROUP        STATUS
 ..
 -            -         c1t5d0       toi          failed was:c1t5d0s2
 ...

  3. Fix the condition that resulted in the problem so that power is restored to all failed disks.

    Be sure that the disks are spun up before proceeding.

  4. Enter the following commands on all nodes in the cluster.

    In some cases, the drive(s) must be rediscovered by the node(s).

    # drvconfig
# disks

  5. Enter the following commands on all nodes in the cluster.

    SSVM or CVM must scan the current disk configuration again.

    # vxdctl enable
# vxdisk -a online

  6. Enter the following command on all nodes in the cluster.

    Note -

    For CVM, enter the command first on the master node, then on the slave nodes.

    This will reattach disks that had transitory failures.

    # vxreattach

  7. Verify the output of the vxdisk command to see if there are any more errors.

    # vxdisk list

    If there are still errors, rerun the vxreattach command as described in Step 6.

  8. (CVM only) If you have shared disk groups, and if media was replaced from the master node, repeat the following command for each disk that has been disconnected.

    The physical disk and the volume manager access name for that disk must be reconnected. 

    # vxdg -g disk-group-name -k adddisk medianame=accessname

    The values for medianame and accessname appear at the end of the vxdisk list command output.

    For example:

    # vxdg -g toi -k adddisk c1t5d0=c1t5d0s2
# vxdg -g toi -k adddisk c1t5d1=c1t5d1s2
# vxdg -g toi -k adddisk c1t5d2=c1t5d2s2
# vxdg -g toi -k adddisk c1t5d3=c1t5d3s2
# vxdg -g toi -k adddisk c1t5d4=c1t5d4s2

    You can also use the vxdiskadm command, or the graphical user interface, to reconnect the disks.

  9. From the node, or from the master node for shared disk groups, start volume recovery.

    # vxrecover -bv [-g diskgroup]

  10. (Optional) Run the vxprint -g command to view the changes.

11.2 Repairing a Lost SPARCstorage Array Connection

When a connection from a SPARCstorage Array to one of the hosts fails, the failure is probably due to a fiber optic cable, an SBus FC/S card, or an FC/OM module.

The host on which the failure occurred will begin generating errors when the failure is discovered. Later accesses to the SPARCstorage Array will generate additional errors. The host behaves as though power had been lost to the SPARCstorage Array.

I/O operations from the other nodes in the cluster are unaffected by this type of failure.

To diagnose the failure, inspect the SPARCstorage Array's display. The display will show whether the A or B connection has been lost. Use the procedures for testing the FC/S card and FC/OM modules in the service manual for your Sun Cluster node to determine which component failed. For hardware debugging, free up one Sun Cluster node and the SPARCstorage Array that appears to be down.

11.2.1 How to Repair a Lost SPARCstorage Array Connection

  1. Prepare the Sun Cluster system for component replacement.

    Depending on the cause of the connection loss, prepare the Sun Cluster system with one of the following procedures.

  2. Replace the failed component.

    If the fiber optic cable, SBus FC/S card, or an FC/OM module fails, refer to the service manual for your Sun Cluster node for detailed instructions on replacing them.

  3. Recover from volume management software errors.

    Use the procedures described in "11.1 Recovering From Power Loss".

11.3 Adding a SPARCstorage Array

You can add SPARCstorage Arrays to a Sun Cluster configuration at any time.

You must review the disk group configuration in your cluster before adding a SPARCstorage Array. To determine the impact of the SPARCstorage Array on your disk group configuration, refer to the configuration planning information in the Sun Cluster 2.2 Software Installation Guide.

11.3.1 How to Add a SPARCstorage Array

  1. Shut down the cluster node that is to receive the new SPARCstorage Array.

    Use the procedure "4.2.1 How to Stop Sun Cluster on a Cluster Node" to shut down the node.

  2. Install the Fibre Channel SBus card (FC/S) in the node.

    Use the instructions in the hardware service manual for your Sun Cluster node to install the FC/S card.

    Note -

    Install the FC/S card in the first available empty SBus slot, following all other cards in the node. This will ensure the controller numbering will be preserved if the Solaris operating environment is reinstalled. Refer to "1.4 Instance Names and Numbering", for more information.

  3. Connect the cables to the SPARCstorage Array and FC/S card.

    Use the instructions in the hardware service manual for your Sun Cluster node.

  4. Perform a reconfiguration reboot of the node. 

    ok boot -r

  5. Run the haswitch(1M) command to switch ownership of all logical hosts that can be mastered to the rebooted node.

    phys-hahost1# haswitch phys-hahost2 hahost1 hahost2

  6. Repeat "" through Step 4 on the other nodes that are connected to this SPARCstorage Array.

  7. Switch ownership of the logical hosts back to the appropriate default master, if necessary.

    phys-hahost1# haswitch phys-hahost2 hahost2

  8. Add the disks in the SPARCstorage Arrays to the selected disk group(s).

    Use the instructions in your volume manager documentation to add the disks to the selected disk group(s). Also, refer the Sun Cluster 2.2 Software Installation Guide for information on Solstice DiskSuite, SSVM, and CVM.

  9. (Solstice DiskSuite configurations only) After adding the disks to the diskset by using the metaset(1M) command, run the scadmin(1M) command to reserve and enable failfast on the specified disks.

    phys-hahost1# scadmin reserve cNtXdYsZ

11.4 Administering SPARCstorage Array Trays

This section describes procedures for administering SPARCstorage Array trays. Use the procedures described in your node hardware manual to identify the tray associated with the failed component.

To guard against data loss and a failure that might require you to replace the entire SPARCstorage Array chassis, set up all mirrors so that a single chassis contains only one submirror.

Note -

There are several different SPARCstorage Array models supported by Sun Cluster. The procedures in this section are only applicable to the SPARCstorage Array 100 series.

11.4.1 How to Take a SPARCstorage Array Tray Out of Service (Solstice DiskSuite)

Before removing a SPARCstorage Array tray, you must halt all I/O and spin down all drives in the tray. The drives automatically spin up if I/O requests are made, so it is necessary to stop all I/O before the drives are spun down.

These are the high-level steps to take a SPARCstorage Array tray out of service in a Solstice DiskSuite configuration:

If the entire SPARCstorage Array is being serviced, you must perform these steps on each tray.

These are the detailed steps to take a SPARCstorage Array tray out of service in a Solstice DiskSuite configuration.

  1. Switch ownership of the affected logical hosts to other nodes by using the haswitch(1M) command.

    phys-hahost1# haswitch phys-hahost1 hahost1 hahost2

    The SPARCstorage Array tray to be removed might contain disks included in more than one logical host. If this is the case, switch ownership of all logical hosts with disks using this tray to another node in the cluster. The luxadm(1M) command will be used later to spin down the disks. In this example, the haswitch(1M) command switched the logical hosts to phys-hahost1, enabling phys-hahost2 to perform the administrative functions.

  2. Use the metastat(1M) command on all affected logical hosts to identify all submirrors containing slices on the tray to be removed. 

    phys-hahost1# metastat -s disksetname

  3. Stop I/O to the submirrors whose components (slices) are on the affected tray.

    Use the metaoffline(1M) command for this step. This takes the submirror offline. You can use the metadetach(1M) command to stop the I/O, but the resync cost is greater.

    When the submirrors on a tray are taken offline, the corresponding mirrors provide only one-way mirroring (that is, there will be no data redundancy). (A three-way mirror does not have this problem.) When the mirror is brought back online, an automatic resync occurs.

    With all affected submirrors offline, I/O to the tray is stopped.

  4. Use the metadb(1M) command to identify any replicas on the tray.

    Save the metadb(1M) output to use when you replace the tray.

  5. Use the metahs(1M) command to identify any available hot spare devices and associated submirrors.

    Save the metahs(1M) output to use when you replace the tray.

  6. If NVRAM is enabled, flush the NVRAM data on the appropriate controller, tray, or disk(s). 

    phys-hahost1# luxadm sync_cache pathname

    A confirmation appears, indicating that NVRAM data has been flushed. See "11.7.3 Flushing and Purging NVRAM", for details on flushing NVRAM data.

  7. Spin down the tray using the luxadm stop command.

    When the tray lock light is out, remove the tray and perform the required service. 

    phys-hahost1# luxadm stop c1

11.4.2 How to Take a SPARCstorage Array Tray Out of Service (SSVM or CVM)

Before removing a SPARCstorage Array tray, you must halt all I/O and spin down all drives in the tray. The drives automatically spin up if I/O requests are made, so it is necessary to stop all I/O before the drives are spun down.

These are the high-level steps to take a SPARCstorage Array tray out of service in an SSVM configuration:

If the entire SPARCstorage Array is being serviced, you must perform these steps on each tray.

These are the detailed steps to take a SPARCstorage Array tray out of service in an SSVM configuration.

  1. Switch ownership of the affected logical hosts to other nodes by using the haswitch(1M) command.

    phys-hahost1# haswitch phys-hahost1 hahost1 hahost2

    The SPARCstorage Array tray to be removed might contain disks included in more than one logical host. If this is the case, switch ownership of all logical hosts with disks using this tray to another node in the cluster. The luxadm(1M) command will be used later to spin down the disks. In this example, the haswitch(1M) command switched the logical hosts to phys-hahost1, enabling phys-hahost1 to perform the administrative functions.

  2. Identify all the volumes and corresponding plexes on the disks in the tray which is being taken out of service.

    1. From the physical device address cNtNdN, obtain the controller number and the target number.

      For example, if the device address is c3t2d0, the controller number is 3 and the target is 2.

    2. Identify SSVM or CVM devices on the affected tray from a vxdisk list output.

      If the target is 0 or 1, identify all devices with physical addresses beginning with cNt0 and cNt1. If the target is 2 or 3, identify all devices with physical addresses beginning with cNt2 and cNt3. If the target is 4 or 5, identify all devices with physical addresses beginning with cNt4 and cNt5. Here is an example of how vxdisk can be used to obtain the information.

      # vxdisk -g diskgroup -q list | egrep c3t2\|c3t3 | nawk '{print $3}'

    3. Identify all plexes on the above devices by using the appropriate version (csh, ksh, or Bourne shell) of the following command.

      PLLIST=`vxprint -ptq -g diskgroup -e '(aslist.sd_dm_name in
("c3t2d0","c3t3d0","c3t3d1")) && (pl_kstate=ENABLED)' | nawk '{print $2}'`

      For csh, the syntax is set PLLIST .... For ksh, the syntax is export PLLIST= .... The Bourne shell requires the command export PLLIST after the variable is set.

  3. After you have set the variable, stop I/O to the volumes whose components (subdisks) are on the tray.

    Make sure all volumes associated with that tray are detached (mirrored or RAID5 configurations) or stopped (simple plexes). Issue the following command to detach a mirrored plex.

    # vxplex det ${PLLIST}

    An alternate command for detaching each plex in a tray is:

    # vxplex -g diskgroup -v volume det plex

    To stop I/O to simple plexes, unmount any file systems or stop database access.

    Note -

    Mirrored volumes will still be active because the other half of the mirror is still available.

  4. If NVRAM is enabled, flush the NVRAM data on the appropriate controller, tray, or disk(s). Otherwise, skip to Step 5. 

    # luxadm sync_cache pathname

    A confirmation appears, indicating that NVRAM data has been flushed. See "11.7.3 Flushing and Purging NVRAM", for details on flushing NVRAM data.

  5. To remove the tray, use the luxadm stop command to spin it down.

    When the tray lock light is out, remove the tray and perform the required service. 

    # luxadm stop c1

11.4.3 How to Return a SPARCstorage Array Tray to Service (Solstice DiskSuite)

These are the high-level steps to return a SPARCstorage Array tray back to service in a Solstice DiskSuite configuration.

If the entire SPARCstorage Array has been serviced, you must perform these steps on each tray.

These are the detailed steps to return a SPARCstorage Array tray back to service in a Solstice DiskSuite configuration.

  1. If the SPARCstorage Array was removed, spin up the drives in the SPARCstorage Array tray. Otherwise, skip to Step 3.

    When you have completed work on a SPARCstorage Array tray, replace the tray in the chassis. The disks will spin up automatically. However, if the disks fail to spin up, run the luxadm(1M) start command to manually spin up the entire tray. There is a short delay (several seconds) between invocation of the command and spin-up of drives in the SPARCstorage Array. In this example, c1 is the controller ID:

    phys-hahost1# luxadm start c1

  2. Add all metadevice state database replicas that were deleted from disks on this tray.

    Use the information saved from Step 4 in the procedure "11.4.1 How to Take a SPARCstorage Array Tray Out of Service (Solstice DiskSuite)" to restore the metadevice state database replicas. 

    phys-hahost1# metadb -s hahost1 -a deleted-replicas

    To add multiple replicas to the same slice, use the -c option.

  3. After the disks spin up, place online all the submirrors that were taken offline.

    Use the metaonline(1M) command appropriate for the disks in this tray. 

    phys-hahost1# metaonline -s hahost1 d15 d35
phys-hahost1# metaonline -s hahost1 d24 d54
...

    When the metaonline(1M) command is run, an optimized resync operation automatically brings the submirrors up-to-date. The optimized resync copies only those regions of the disk that were modified while the submirror was offline. This is typically a very small fraction of the submirror capacity.

    Run metaonline(1M) as many times as necessary to bring back online all of the submirrors.

    Note -

    If you used the metadetach(1M) command to detach the submirror rather than metaoffline(1M), you must synchronize the entire submirror using the metattach(1M) command. This typically takes about 10 minutes per Gigabyte of data.

  4. Add back all hot spares that were deleted when the SPARCstorage Array was taken out of service.

    Use the metahs(1M) command as appropriate for your hot spare configuration. Use the information saved from Step 5 in the procedure "11.4.1 How to Take a SPARCstorage Array Tray Out of Service (Solstice DiskSuite)" to replace your hot spares. 

    phys-hahost1# metahs -s hahost1 -a hotsparepool cNtXdYsZ

  5. Switch each logical host back to its default master, if necessary. 

    phys-hahost1# haswitch phys-hahost2 hahost2

11.4.4 How to Return a SPARCstorage Array Tray to Service (SSVM or CVM)

These are the high-level steps to return a SPARCstorage Array tray back to service in a SSVM or CVM configuration:

If the entire SPARCstorage Array has been serviced, you must perform these steps on each tray.

These are the detailed steps to return a SPARCstorage Array tray back to service in an SSVM configuration.

  1. If the SPARCstorage Array was removed, spin up the drives in the SPARCstorage Array tray. Otherwise, skip to Step 2.

    When you have completed work on a SPARCstorage Array tray, replace the tray in the chassis. The disks will spin up automatically. However, if the disks fail to spin up, run the luxadm(1M) start command to manually spin up the entire tray. There is a short delay (several seconds) between invocation of the command and spin-up of drives in the SPARCstorage Array. In this example, c1 is the controller ID.

    phys-hahost1# luxadm start c1

  2. After the disks spin up, monitor the volume management recovery.

    Previously affected volumes that are on the tray should begin to come back online, and the rebuilding of data should start automatically within a few minutes. If necessary, use the vxreattach and vxrecover commands to reattach disks and recover from error. Refer to the respective man pages for more information.

    Note -

    DRL subdisks that were detached must be manually reattached.

  3. Switch each logical host back to its default master, if necessary. 

    phys-hahost1# haswitch phys-hahost2 hahost2

11.5 Replacing a SPARCstorage Array Controller and Changing the World Wide Name

The SPARCstorage Array controller has a unique identifier known as the World Wide Name (WWN) that identifies the controller to Solaris. Therefore, when SPARCstorage Array failures make it necessary to replace the controller or the entire chassis containing the controller, special procedures apply.

The WWN is like the host ID stored in the host IDPROM of a SPARC machine. The last four digits of the SPARCstorage Array WWN are displayed on the LCD panel of the chassis. The WWN is part of the /devices path associated with the SPARCstorage Array and its component drives.

If you must replace the SPARCstorage Array controller or the entire chassis, the Sun Cluster nodes will discover the new WWN when they are rebooted. To avoid confusion of the upper layers of Sun Cluster software by the new WWN, change the WWN of the new controller to be the WWN of the old controller. (This is similar to swapping the IDPROM when replacing a system board in a SPARC machine.)

Consider the following situations when deciding which WWN replacement procedure to use:

11.5.1 How to Change a SPARCstorage Array World Wide Name Using a Maintenance System

This procedure describes how to change a SPARCstorage Array controller and replace its WWN with the WWN of the failed controller. This procedure enables you to replace a SPARCstorage Array controller without taking down any nodes in the cluster.

This procedure makes use of a "maintenance system," which can be any Sun Microsystems architecture capable of supporting a SPARCstorage Array. The presence of a maintenance system enables you to complete this procedure without taking down any nodes in the cluster.

This system should be loaded with the same version of the Solaris Operating Environment (2.6 or 7) as the cluster nodes, and should have all applicable patches. It also should have available a CD-ROM drive, a Fibre Channel SBus Card (FC/S), and a Fibre Channel Optical Module (FC/OM). The system should have the proper FCODE and hardware revisions. Alternately, you can boot the maintenance system over the net.

Note -

If a "maintenance system" is not available, use one of the cluster nodes for this purpose by following the steps in this procedure.

These are the high-level steps to change a SPARCstorage Array World Wide Name (WWN) using a maintenance system:

These are the detailed steps to change a SPARCstorage Array World Wide Name by using a maintenance system.

  1. If the failed SPARCstorage Array controller is the quorum controller, select a new quorum controller by using the scconf(1M) command.

    Refer to the scconf(1M) man page for more information.

  2. Determine the WWN of the broken SPARCstorage Array.

    • If the SPARCstorage Array is powered down, use the following instructions to obtain the WWN.

      The WWN is composed of 12 hexadecimal digits. The digits are shown as part of the device path component. They are the last 12 digits following the characters pln@a0, excluding the comma. Use the ls(1) command on a cluster node connected to the SSA to identify the current WWN.

    # ls -l /dev/rdsk/cNt0d0s0
...SUNW,pln@a0000000,7412bf ...

    In this example, the WWN for the SPARCstorage Array being replaced is 0000007412bf. The variable N in the device name represents the controller number for the broken SPARCstorage Array. The string "t0d0s0" is just an example. Use a device name that you know exists on the SPARCstorage Array, or use /dev/rdsk/cN* to match all devices.

    • If the SPARCstorage Array is up and running, you can obtain the WWN by using the luxadm(1M) command.

      When you run luxadm(1M) with the display option and specify a controller, all the information about the SPARCstorage Array is displayed. The serial number reported by luxadm(1M) is the WWN.

    # /usr/sbin/luxadm display cN

  3. Detach the optical cable from the faulty SPARCstorage Array Controller.

  4. Replace the faulty controller.

    Use the instructions in your SPARCstorage Array service manual to perform this step.

    If the SPARCstorage Array has not failed entirely or is being swapped for a reason other than controller failure, prepare for the swap by performing the steps described in "11.4 Administering SPARCstorage Array Trays", for each tray in the SPARCstorage Array.

    If the SPARCstorage Array controller has failed entirely, your volume manager has already prepared for the swap.

  5. Attach the optical cable from the maintenance system to the new controller.

  6. Enter the OpenBoot PROM on the maintenance system and boot it with "mini-unix."

    Do this from the distribution CD (or its network equivalent) to put the maintenance system into single-user mode and to obtain an in-memory version of the device structure that contains the new SPARCstorage Array WWN.

    <#0> ok boot cdrom -s

 or

 <#0> ok boot netqe1 -s

    Use "mini-unix" to avoid making any permanent device information changes.

  7. Run the luxadm download command to set the WWN. 

    # /usr/sbin/luxadm -s -w WWN download cN

    WWN is the 12-digit WWN of the replaced controller and N is the controller number from cNtXdX in the device name. You should have obtained the WWN in Step 2.

    Note -

    The leading zeros must be entered as part of the WWN to make a total of 12 digits.

    Caution - Caution -

    Do not interrupt the download process. Wait for the shell prompt after completion of the luxadm(1M)command.

  8. After the prompt is redisplayed, reset the SSA.

    The new address should appear in the window on the SPARCstorage Array.

  9. Shut down the maintenance system.

  10. Reattach the SPARCstorage Array controller to the cluster nodes.

  11. Verify the SPARCstorage Array firmware level from the cluster node.

    Use the luxadm(1M) command to determine the current version of the firmware. Specify the controller number (N in the example) to the luxadm(1M) command. 

    # /usr/sbin/luxadm display cN
    Note -

    If the Solaris system detects an old version of firmware on your system, it displays a message on the console and in /var/adm/messages similar to the following: 


    NOTICE: pln0: Old SSA firmware has been detected (Ver:3.11) : 
Expected (Ver:3.12) - Please upgrade

  12. (Optional) To upgrade the controller's firmware, follow these steps.

    1. Download the proper firmware. Refer to the README file in the firmware patch for details.

      # /usr/sbin/ssaadm download -f path/ssafirmware cN

      where path is the path to the directory where the firmware is stored and N is the controller number. For example: 

      # /usr/sbin/ssaadm download -f /usr/lib/firmware/ssa/ssafirmware cN

    2. Reset the SPARCstorage Array by pressing the SYS OK button on the unit.

      There will be a short delay while the unit reboots.

    3. Verify the firmware level again (using Step 11). If the firmware level or WWN is still incorrect, repeat Step 12 using a different controller.

  13. Begin volume manager recovery.

    Refer to "11.4 Administering SPARCstorage Array Trays". Wait until the SPARCstorage Array is online for all nodes, and all nodes can see all the disks.

11.5.2 How to Change a SPARCstorage Array World Wide Name

Caution - Caution -

This procedure will not work if the root disk is encapsulated by SSVM or CVM, or if the boot disk of one of the nodes is on this SPARCstorage Array. For those situations, use the procedure "11.5.1 How to Change a SPARCstorage Array World Wide Name Using a Maintenance System".

Note -

If a quorum controller fails, you must select a new quorum controller before shutting down a node.

These are the high-level steps to change a SPARCstorage Array World Wide Name:

These are the detailed steps to change a SPARCstorage Array World Wide Name.

  1. If the failed SPARCstorage Array controller is the quorum controller, select a new quorum controller by using the scconf(1M) command.

    Refer to the scconf(1M) man page for more information.

  2. On the cluster node that is connected to the SSA being repaired, stop the Sun Cluster software and halt the system.

    Use the scadmin(1M) command to switch ownership of all logical hosts to the other nodes in the cluster, and to stop Sun Cluster. Then run the halt(1M) command to stop the machine.

    In this example, phys-hahost2 is the node from which the repair procedure is performed.

    phys-hahost2# scadmin stopnode
...
phys-hahost2# halt

  3. Determine the WWN of the broken SPARCstorage Array.

    • If the SPARCstorage Array is powered down, use the following instructions to obtain the WWN.

      The WWN is composed of 12 hexadecimal digits. The digits are shown as part of the device path component containing the characters pln@a0. They are the last 12 digits following the characters pln@a0, excluding the comma. Use the ls(1) command on a cluster node connected to the SSA to identify the current WWN.

    phys-hahost1# ls -l /dev/rdsk/cNt0d0s0
...SUNW,pln@a0000000,7412bf ...

    In this example, the WWN for the SPARCstorage Array being replaced is 0000007412bf. The variable N in the device name represents the controller number for the broken SPARCstorage Array. The string t0d0s0 is just an example. Use a device name that you know exists on the SPARCstorage Array, or use /dev/rdsk/cN* to match all devices.

    • If the SPARCstorage Array is up and running, you can obtain the WWN by using the luxadm(1M) command.

      When you run luxadm(1M) with the display option and specify a controller, all the information about the SPARCstorage Array is displayed. The serial number reported by luxadm(1M) is the WWN.

    phys-hahost1# /usr/sbin/luxadm display cN

  4. Replace the controller or SPARCstorage Array.

    Use the instructions in your SPARCstorage Array service manual to perform this step.

    • If the SPARCstorage Array has not failed completely or is being swapped for a reason other than controller failure, prepare for the swap by performing the steps described in "11.4 Administering SPARCstorage Array Trays", for each tray in the SPARCstorage Array.

    • If the SPARCstorage Array controller has failed completely, your volume manager has already prepared for the swap.

  5. Enter the OpenBoot PROM on the halted node and boot it with "mini-unix."

    Do this from the distribution CD (or net equivalent) to put the host into single-user mode and to obtain an in-memory version of the device structure that contains the new SPARCstorage Array WWN. 

    <#0> ok boot cdrom -s

 or

 <#0> ok boot netqe1 -s

    Use "mini-unix" to avoid making any permanent device information changes to the cluster node.

  6. Determine the controller number for the new SPARCstorage Array.

    Use the ls(1) command and the four digits displayed on the LCD display of the new SPARCstorage Array to identify the controller number.

    In this example, the four digits shown on the LCD display are 143b. Note that the device name c*t0d0s0 uses pattern matching for the controller number but specifies a slice that is known to exist. This reduces the number of lines generated in the output. 

    # ls -l /dev/rdsk/c*t0d0s0 | grep -i 143b
lrwxrwxrwx   1 root     root          98 Mar 14 13:38
 /dev/rdsk/c3t0d0s0 ->
 ../../devices/iommu@f,e0000000/sbus@f,e0001000/SUNW,soc@3,0/SUNW
 ,pln@a0000000,74143b/ssd@0,0:a,raw

    In this example, 3 (from /dev/rdsk/c3...) is the controller number of the new SPARCstorage Array under "mini-unix".

    Note -

    The hex digits in the LCD display are in mixed case--letters A, C, E, and F are in upper case, and letters b and d are in lower case. The example uses grep -i to ignore case in the comparison.

  7. Run the luxadm download command to set the WWN.

    Use the controller number determined in Step 6. For example, the following command would change the WWN from the current value to the value determined in Step 3, 0000007412bf. The SPARCstorage Array controller is Controller 3.

    phys-hahost2# /usr/sbin/luxadm download -w 0000007412bf c3
    Note -

    The leading zeros must be entered as part of the WWN to make a total of 12 digits.

    Caution - Caution -

    Do not interrupt the download process. Wait for the shell prompt after completion of the luxadm(1M) command.

  8. Reset the SPARCstorage Array by pressing the SYS OK button on the unit.

    There will be a short delay while the unit reboots and begins communicating with the Sun Cluster nodes.

  9. Abort "mini-unix" and boot the host normally.

    Send a break to the console, and boot the machine.

  10. Verify the SPARCstorage Array firmware level from the cluster node.

    Use the luxadm(1M) command to determine the current version of the firmware. Specify the controller number (N in the example) to the luxadm(1M) command. 

    phys-hahost2# /usr/sbin/luxadm display cN
    Note -

    If the Solaris system detects an old version of firmware on your system, it displays a message on the console and in /var/adm/messages similar to the following: 


    NOTICE: pln0: Old SSA firmware has been detected (Ver:3.11) : 
Expected (Ver:3.12) - Please upgrade

  11. (Optional) To upgrade the controller's firmware, follow these steps.

    1. Download the proper firmware. Refer to the README file in the firmware patch for details.

      # /usr/sbin/ssaadm download -f path/ssafirmware cN

      where path is the path to the directory where the firmware is stored and N is the controller number. For example:

      # /usr/sbin/ssaadm download -f /usr/lib/firmware/ssa/ssafirmware cN

    2. Reset the SPARCstorage Array using the SYS OK button on the unit.

      There will be a short delay while the unit reboots.

    3. Re-verify the firmware level (see Step 10). If either the firmware level or WWN are still incorrect, then repeat Step 11 using a different controller.

  12. Start the node.

    phys-hahost2# scadmin startnode

  13. Switch back the logical hosts to the default master, if necessary.

  14. Complete the replacement by restoring the volume manager components onto the repaired SPARCstorage Array.

    This procedure is described in "11.4 Administering SPARCstorage Array Trays".

  15. Reboot the other nodes in the cluster, if necessary.

    You might need to reboot the other cluster nodes, if they are unable to recognize all disks in the SPARCstorage Array following the replacement. If this is the case, use the scadmin stopnode command to stop Sun Cluster activity, then reboot. After the reboot, if necessary, switch the logical hosts back to their default masters. See the scadmin(1M) man page for more information.

11.6 Administering SPARCstorage Array Disks

As part of standard Sun Cluster administration, you should monitor the status of the configuration. See Chapter 2, Sun Cluster Administration Tools," for information about monitoring methods. During the monitoring process you might discover problems with multihost disks. The following sections provide instructions for correcting these problems.

Sun Cluster supports these SSA disk types:

Depending on which type you have and the electrical and mechanical characteristics of this disk enclosure, adding a disk might require you to prepare all disks connected to a particular controller, all disks in a particular array tray, or only the disk being added. For example, in the SPARCstorage Array 200 series with the differential SCSI tray, you must prepare the array controller and the disk enclosure. In the SPARCstorage Array 200 series with RSM (214 RSM), you need to prepare only the new disk. In the SPARCstorage Array 110, you must prepare a single tray.

If you have a SPARCstorage Array 100 series array, follow the steps as documented. If you have a SPARCstorage Array 200 series array with differential SCSI tray, you must bring down all disks attached to the array controller that will connect to the new disk. This means you repeat all of the tray-specific steps for all disk enclosures attached to the array controller that will connect to the new disk. If you have a SPARCstorage Array 214 RSM, you need not perform any of the tray-specific steps, since individual disk drives can be installed without affecting other disks.

Refer to the hardware service manual for your multihost disk expansion unit for a description of your disk enclosure.

11.6.1 Adding a SPARCstorage Array Disk

Depending upon the disk enclosure, adding SPARCstorage Array (SSA) multihost disks might involve taking off line all volume manager objects in the affected disk tray or disk enclosure. Additionally, the disk tray or disk enclosure might contain disks from more than one disk group, requiring that a single node own all of the affected disk groups.

11.6.2 How to Add a SPARCstorage Array Disk (Solstice DiskSuite)

These are the high-level steps to add a multihost disk in a Solstice DiskSuite configuration:

These are the detailed steps to add a new multihost disk to a Solstice DiskSuite configuration.

  1. Switch ownership of the logical host that will include the new disk to other nodes in the cluster.

    Switch over any logical hosts with disks in the tray you are removing.

    phys-hahost1# haswitch phys-hahost1 hahost1 hahost2

  2. Determine the controller number of the tray to which the disk will be added.

    SPARCstorage Arrays are assigned World Wide Names (WWN). The WWN on the front of the SPARCstorage Array also appears as part of the /devices entry, which is linked by pointer to the /dev entry containing the controller number. For example: 

    phys-hahost1# ls -l /dev/rdsk | grep -i WWN | tail -1

    If the WWN on the front of the SPARCstorage Array is 36cc, the following output will display, and the controller number would be c2:

    phys-hahost1# ls -l /dev/rdsk | grep -i 36cc | tail -1
lrwxrwxrwx  1 root   root       94 Jun 25 22:39 c2t5d2s7 ->
 ../../devices/io-
 unit@f,e1200000/sbi@0,0/SUNW,soc@3,0/SUNW,pln@a0000800,201836cc/
 ssd@5,2:h,raw

  3. Use the luxadm(1M) command with the display option to view the empty slots. 

    phys-hahost1# luxadm display c2

                      SPARCstorage Array Configuration
 ...
                           DEVICE STATUS
       TRAY 1                 TRAY 2                 TRAY 3
 slot
 1     Drive: 0,0             Drive: 2,0             Drive: 4,0
 2     Drive: 0,1             Drive: 2,1             Drive: 4,1
 3     NO SELECT              NO SELECT              NO SELECT
 4     NO SELECT              NO SELECT              NO SELECT
 5     NO SELECT              NO SELECT              NO SELECT
 6     Drive: 1,0             Drive: 3,0             Drive: 5,0
 7     Drive: 1,1             NO SELECT              NO SELECT
 8     NO SELECT              NO SELECT              NO SELECT
 9     NO SELECT              NO SELECT              NO SELECT
 10    NO SELECT              NO SELECT              NO SELECT
 ...

    The empty slots are shown with a NO SELECT status. The output shown here is from a SPARCstorage Array 110; your display will be slightly different if you are using a different series SPARCstorage Array.

    Determine the tray to which you will add the new disk. If you can add the disk without affecting other drives, such as in the SPARCstorage Array 214 RSM, skip to Step 11.

    In the remainder of the procedure, Tray 2 is used as an example. The slot selected for the new disk is Tray 2 Slot 7. The new disk will be known as c2t3d1.

  4. Locate all hot spares affected by the installation.

    To determine the status and location of all hot spares, run the metahs(1M) command with the -i option on each of the logical hosts. 

    phys-hahost1# metahs -s hahost1 -i
...
 phys-hahost1# metahs -s hahost2 -i
...
    Note -

    Save a list of the hot spares. The list is used later in this maintenance procedure. Be sure to note the hot spare devices and their hot spare pools.

  5. Use the metahs(1M) command with the -d option to delete all affected hot spares.

    Refer to the man page for details on the metahs(1M) command.

    phys-hahost1# metahs -s hahost1 -d hot-spare-pool components
 phys-hahost1# metahs -s hahost2 -d hot-spare-pool components

  6. Locate all metadevice state database replicas that are on affected disks.

    Run the metadb(1M) command on each of the logical hosts to locate all metadevice state databases. Direct the output into temporary files.

    phys-hahost1# metadb -s hahost1 > /usr/tmp/mddb1
phys-hahost1# metadb -s hahost2 > /usr/tmp/mddb2

    The output of metadb(1M) shows the location of metadevice state database replicas in this disk enclosure. Save this information for the step in which you restore the replicas.

  7. Delete the metadevice state database replicas that are on affected disks.

    Keep a record of the number and locale of the replicas that you delete. The replicas must be restored in a later step. 

    phys-hahost1# metadb -s hahost1 -d replicas
phys-hahost1# metadb -s hahost2 -d replicas

  8. Run the metastat(1M) command to determine all the metadevice components on affected disks.

    Direct the output from metastat(1M) to a temporary file so that you can use the information later when deleting and re-adding the metadevices.

    phys-hahost1# metastat -s hahost1 > /usr/tmp/replicalog1
phys-hahost1# metastat -s hahost2 > /usr/tmp/replicalog2

  9. Take offline all submirrors containing affected disks.

    Use the temporary files to create a script to take offline all affected submirrors in the disk expansion unit. If only a few submirrors exist, run the metaoffline(1M) command to take each offline. The following is a sample script. 

    #!/bin/sh
 # metaoffline -s <diskset> <mirror> <submirror>

 metaoffline -s hahost1 d15 d35
 metaoffline -s hahost2 d15 d35
 ...

  10. Spin down the affected disks.

    Spin down the SPARCstorage Array disks in the tray using the luxadm(1M) command.

    phys-hahost1# luxadm stop -t 2 c2

  11. Add the new disk.

    Use the instructions in your multihost disk expansion unit service manual to perform the hardware procedure of adding the disk. After the addition:

    • If your disk enclosure is a SPARCstorage Array 214 RSM, skip to Step 16. (This type of disk can be added without affecting other drives.)

    • For all other SPARCstorage Array types, proceed with Step 12.

  12. Make sure all disks in the tray spin up.

    The disks in the SPARCstorage Array tray should spin up automatically, but if the tray fails to spin up within two minutes, force the action using the following command:

    phys-hahost1# luxadm start -t 2 c2

  13. Bring the submirrors back online.

    Modify the script that you created in Step 9 to bring the submirrors back online. 

    #!/bin/sh
 # metaonline -s <diskset> <mirror> <submirror>

 metaonline -s hahost1 d15 d35
 metaonline -s hahost2 d15 d35
 ...

  14. Restore the hot spares that were deleted in Step 5. 

    phys-hahost1# metahs -s hahost1 -a hot-spare-pool components
phys-hahost1# metahs -s hahost2 -a hot-spare-pool components

  15. Restore the original count of metadevice state database replicas to the devices in the tray.

    The replicas were removed in Step 7. 

    phys-hahost1# metadb -s hahost1 -a replicas
 
phys-hahost1# metadb -s hahost2 -a replicas

  16. Run the drvconfig(1M) and disks(1M) commands to create the new entries in /devices, /dev/dsk, and /dev/rdsk for all new disks. 

    phys-hahost1# drvconfig
phys-hahost1# disks

  17. Switch ownership of the logical host to which this disk will be added to the other node that is connected to the SPARCstorage Array.

    This assumes a topology in which each disk is connected to two nodes.

    phys-hahost1# haswitch phys-hahost2 hahost2

  18. Run the drvconfig(1M) and disks(1M) commands on the cluster node that now owns the diskset to which this disk will be added.

    phys-hahost2# drvconfig
phys-hahost2# disks

  19. Run the scdidadm(1M) command to initialize the new disk for use by the DID pseudo driver.

    You must run scdidadm(1M) on Node 0 in the cluster. Refer to the Sun Cluster 2.2 Software Installation Guide for details on the DID pseudo driver. 

    phys-hahost2# scdidadm -r

  20. Add the disk to a diskset.

    The command syntax is as follows, where diskset is the name of the diskset containing the failed disk, and drive is the DID name of the disk in the form dN (for new installations of Sun Cluster), or cNtYdZ (for installations that upgraded from HA 1.3):

    # metaset -s diskset -a drive
    Caution - Caution -

    The metaset(1M) command might repartition this disk automatically. See the Solstice DiskSuite documentation for more information.

  21. Use the scadmin(1M) command to reserve and enable failfast on the specified disk that has just been added to the diskset.

    phys-hahost2# scadmin reserve cNtXdYsZ

  22. Perform the usual administration actions on the new disk.

    You can now perform the usual administration steps that bring a new drive into service. These include partitioning the disk, adding it to the configuration as a hot spare, or configuring it as a metadevice. See the Solstice DiskSuite documentation for more information on these tasks.

  23. If necessary, switch logical hosts back to their default masters.

11.6.3 How to Add a SPARCstorage Array Disk (SSVM or CVM)

These are the high-level steps to add a multihost disk to an SSVM or CVM configuration:

These are the detailed steps to add a new multihost disk to an SSVM configuration.

  1. Switch ownership of the logical host that will include the new disk to another node in the cluster.

    Switch over any logical hosts with disks in the tray you are removing.

    phys-hahost1# haswitch phys-hahost1 hahost1 hahost2
    Note -

    In a mirrored configuration, you may not need to switch logical hosts as long as the node is not shut down.

  2. Determine the controller number of the tray to which the disk will be added.

    SPARCstorage Arrays are assigned World Wide Names (WWN). The WWN on the front of the SPARCstorage Array also appears as part of the /devices entry, which is linked by pointer to the /dev entry containing the controller number. For example: 

    phys-hahost1# ls -l /dev/rdsk | grep -i WWN | tail -1

    If the WWN on the front of the SPARCstorage Array is 36cc, the following output will display and the controller number would be c2:

    phys-hahost1# ls -l /dev/rdsk | grep -i 36cc | tail -1
lrwxrwxrwx  1 root   root       94 Jun 25 22:39 c2t5d2s7 ->
 ../../devices/io-
 unit@f,e1200000/sbi@0,0/SUNW,soc@3,0/SUNW,pln@a0000800,201836cc/
 ssd@5,2:h,raw
 phys-hahost1#

  3. Use the luxadm(1M) command with the display option to view the empty slots.

    If you can add the disk without affecting other drives, skip to Step 11. 

    phys-hahost1# luxadm display c2

                      SPARCstorage Array Configuration
 ...
                           DEVICE STATUS
       TRAY 1                 TRAY 2                 TRAY 3
 slot
 1     Drive: 0,0             Drive: 2,0             Drive: 4,0
 2     Drive: 0,1             Drive: 2,1             Drive: 4,1
 3     NO SELECT              NO SELECT              NO SELECT
 4     NO SELECT              NO SELECT              NO SELECT
 5     NO SELECT              NO SELECT              NO SELECT
 6     Drive: 1,0             Drive: 3,0             Drive: 5,0
 7     Drive: 1,1             NO SELECT              NO SELECT
 8     NO SELECT              NO SELECT              NO SELECT
 9     NO SELECT              NO SELECT              NO SELECT
 10    NO SELECT              NO SELECT              NO SELECT
 ...

    The empty slots are shown with a NO SELECT status. The output shown here is from a SPARCstorage Array 110; your display will be slightly different if you are using a different series SPARCstorage Array.

    Determine the tray to which you will add the new disk.

    In the remainder of the procedure, Tray 2 is used as an example. The slot selected for the new disk is Tray 2 Slot 7. The new disk will be known as c2t3d1.

  4. Identify all the volumes and corresponding plexes on the disks in the tray which will contain the new disk.

    1. From the physical device address cNtNdN, obtain the controller number and the target number.

      In this example, the controller number is 2 and the target is 3.

    2. Identify devices from a vxdisk list output.

      Here is an example of how vxdisk can be used to obtain the information.

      # vxdisk -g diskgroup -q list | nawk '/^c2/ {print $3}'

      Record the volume media name for the disks from the output of the command.

    3. Identify all plexes on the above devices by using the appropriate version (csh, ksh, or Bourne shell) of the following command.

      PLLIST=`vxprint -ptq -g diskgroup -e '(aslist.sd_dm_name in ("c2t3d0")) &&
 (pl_kstate=ENABLED)' | nawk '{print $2}'`

      For csh, the syntax is set PLLIST .... For ksh, the syntax is export PLLIST= .... The Bourne shell requires the command export PLLIST after the variable is set.

  5. After you have set the variable, stop I/O to the volumes whose components (subdisks) are on the tray.

    Make sure all volumes associated with that tray are detached (mirrored or RAID5 configurations) or stopped (simple plexes). Issue the following command to detach a mirrored plex.

    # vxplex -g diskgroup det ${PLLIST}

    An alternate command for detaching each plex in a tray is:

    # vxplex -g diskgroup -v volume det plex

    To stop I/O to simple plexes, unmount any file systems or stop database access.

    Note -

    Mirrored volumes will still be active because the other half of the mirror is still available.

  6. Add the new disk.

    Use the instructions in your multihost disk expansion unit service manual to perform the hardware procedure of adding the disk.

  7. Make sure all disks in the tray spin up.

    The disks in the SPARCstorage Array tray should spin up automatically, but if the tray fails to spin up within two minutes, force the action with the following command:

    phys-hahost1# luxadm start -t 2 c2

  8. Run the drvconfig(1M) and disks(1M) commands to create the new entries in /devices, /dev/dsk, and /dev/rdsk for all new disks. 

    phys-hahost1# drvconfig
phys-hahost1# disks

  9. Force the SSVM vxconfigd driver to scan for new disks.

    phys-hahost1# vxdctl enable

  10. Bring the new disk under VM control by using the vxdiskadd command.

  11. Perform the usual administration actions on the new disk.

    You can now perform the usual administration steps that bring a new drive into service. These include partitioning the disk, adding it to the configuration as a hot spare, or configuring it as a plex.

    This completes the procedure of adding a multihost disk to an existing SPARCstorage Array.

11.6.4 Replacing a SPARCstorage Array Disk

This section describes replacing a SPARCstorage Array (SSA) multihost disk without interrupting Sun Cluster services (online replacement) when the volume manager is reporting problems such as:

11.6.5 How to Replace a SPARCstorage Array Disk (Solstice DiskSuite)

These are the high-level steps to replace a multihost disk in a Solstice DiskSuite configuration. Some of the steps in this procedure apply only to configurations using SPARCstorage Array 100 series or SPARCstorage Array 200 series with the differential SCSI tray.

These are the detailed steps to replace a failed multihost disk in a Solstice DiskSuite configuration.

  1. Switch ownership of the affected logical hosts to other nodes by using the haswitch(1M) command.

    phys-hahost1# haswitch phys-hahost1 hahost1 hahost2

    The SPARCstorage Array tray containing the failed disk might contain disks included in more than one logical host. If this is the case, switch ownership of all logical hosts with disks using this tray to another node in the cluster.

  2. Identify the disk to be replaced by examining metastat(1M) and /var/adm/messages output.

    When metastat(1M) reports that a device is in maintenance state or some of the components have been replaced by hot spares, you must locate and replace the device. A sample metastat(1M) output follows. In this example, device c3t3d4s0 is in maintenance state. 

    phys-hahost1# metastat -s hahost1
...
  d50:Submirror of hahost1/d40
       State: Needs Maintenance
       Stripe 0:
           Device       Start Block      Dbase      State          Hot Spare
           c3t3d4s0     0                No         Okay           c3t5d4s0
 ...

    Check /var/adm/messages to see what kind of problem has been detected.

    ...
Jun 1 16:15:26 host1 unix: WARNING: /io-
unit@f,e1200000/sbi@0.0/SUNW,pln@a0000000,741022/ssd@3,4(ssd49):  
Jun 1 16:15:26 host1 unix: Error for command `write(I))' Err
Jun 1 16:15:27 host1 unix: or Level: Fatal
Jun 1 16:15:27 host1 unix: Requested Block 144004, Error Block: 715559
Jun 1 16:15:27 host1 unix: Sense Key: Media Error
Jun 1 16:15:27 host1 unix: Vendor `CONNER':
Jun 1 16:15:27 host1 unix: ASC=0x10(ID CRC or ECC error),ASCQ=0x0,FRU=0x15
...

  3. Determine the location of the problem disk by running the luxadm(1M) command.

    The luxadm(1M) command lists the trays and the drives associated with them. The output differs for each SPARCstorage Array series. This example shows output from a SPARCstorage Array 100 series array. The damaged drive is highlighted below. 

    phys-hahost1# luxadm display c3
         SPARCstorage Array Configuration
 Controller path:
 /devices/iommu@f,e0000000/sbus@f,e0001000/SUNW,soc@0,0/SUNW,pln@
 a0000000,779a16:ctlr
          DEVICE STATUS
          TRAY1          TRAY2          TRAY3
 Slot
 1        Drive:0,0      Drive:2,0      Drive:4,0
 2        Drive:0,1      Drive:2,1      Drive:4,1
 3        Drive:0,2      Drive:2,2      Drive:4,2
 4        Drive:0,3      Drive:2,3      Drive:4,3
 5        Drive:0,4      Drive:2,4      Drive:4,4
 6        Drive:1,0      Drive:3,0      Drive:5,0
 7        Drive:1,1      Drive:3,1      Drive:5,1
 8        Drive:1,2      Drive:3,2      Drive:5,2
 9        Drive:1,3      Drive:3,3      Drive:5,3
 10       Drive:1,4      Drive:3,4      Drive:5,4

          CONTROLLER STATUS
 Vendor:    SUN
 Product ID:  SSA110
 Product Rev: 1.0
 Firmware Rev: 3.9
 Serial Num: 000000741022
 Accumulate performance Statistics: Enabled

  4. Detach all submirrors with components on the disk being replaced.

    If you are detaching a submirror that has a failed component, you must force the detach using the metadetach -f command. The following example command detaches submirror d50 from metamirror d40. 

    phys-hahost1# metadetach -s hahost1 -f d40 d50

  5. Use the metaclear(1M) command to clear the submirrors detached in Step 4.

    phys-hahost1# metaclear -s hahost1 -f d50

  6. Before deleting replicas and hot spares, make a record of the location (slice), number of replicas, and hot spare information (names of the devices and list of devices that contain hot spare pools) so that the actions can be reversed following the disk replacement.

  7. Delete all hot spares that have Available status and are in the same tray as the problem disk.

    This includes all hot spares, regardless of their logical host assignment. In the following example, the metahs(1M) command reports hot spares on hahost1, but shows that none are present on hahost2" 

    phys-hahost1# metahs -s hahost1 -i
 hahost1:hsp000 2 hot spares
         c1t4d0s0                Available       2026080 blocks
         c3t2d5s0                Available       2026080 blocks
 phys-hahost1# metahs -s hahost1 -d hsp000 c3t2d4s0
 hahost1:hsp000:
         Hotspare is deleted
 phys-hahost1# metahs -s hahost2 -i
 phys-hahost1#
 hahost1:hsp000 1 hot spare
 			c3t2d5s0                Available       2026080 blocks

  8. Use the metaset(1M) command to remove the failed disk from the diskset.

    The syntax for the command is shown below. In this example, diskset is the name of the diskset containing the failed disk, and drive is the DID name of the disk in the form dN (for new installations of Sun Cluster), or cNtYdZ (for installations that upgraded from HA 1.3).

    # metaset -s diskset -d drive

    This operation can take up to fifteen minutes or more, depending on the size of your configuration and the number of disks.

  9. Delete any metadevice state database replicas that are on disks in the tray to be serviced.

    The metadb(1M) command with the -s option reports replicas in a specified diskset. 

    phys-hahost1# metadb -s hahost1
phys-hahost1# metadb -s hahost2
phys-hahost1# metadb -s hahost1 -d replicas-in-tray
phys-hahost1# metadb -s hahost2 -d replicas-in-tray

  10. Locate the submirrors using components that reside in the affected tray.

    One method is to use the metastat(1M) command to create temporary files that contain the names of all metadevices. For example: 

    phys-hahost1# metastat -s hahost1 > /usr/tmp/hahost1.stat
phys-hahost1# metastat -s hahost2 > /usr/tmp/hahost2.stat

    Search the temporary files for the components in question (c3t3dn and c3t2dn in this example). The information in the temporary files will look like this:

    ...
 hahost1/d35: Submirror of hahost1/d15
    State: Okay
    Hot Spare pool: hahost1/hsp100
    Size: 2026080 blocks
    Stripe 0:
       Device      Start Block     Dbase     State      Hot Spare
       c3t3d3s0    0               No        Okay      
 hahost1/d54: Submirror of hahost1/d24
    State: Okay
    Hot Spare pool: hahost1/hsp106
    Size: 21168 blocks
    Stripe 0:
       Device      Start Block     Dbase     State      Hot Spare
       c3t3d3s6    0               No        Okay      
 ...

  11. Take offline all other submirrors that have components in the affected tray.

    Using the output from the temporary files in Step 10, run the metaoffline(1M) command on all submirrors in the affected tray. 

    phys-hahost1# metaoffline -s hahost1 d15 d35
phys-hahost1# metaoffline -s hahost1 d24 d54
...

    Run metaoffline(1M) as many times as necessary to take all the submirrors off line. This forces Solstice DiskSuite to stop using the submirror components.

  12. If enabled, flush the NVRAM on the controller, tray, individual disk or disks.

    phys-hahost1# luxadm sync_cache pathname

    A confirmation appears, indicating that NVRAM has been flushed. See "11.7.3 Flushing and Purging NVRAM", for details on flushing NVRAM data.

  13. Spin down all disks in the affected SPARCstorage Array tray(s).

    Use the luxadm stop command to spin down the disks. Refer to the luxadm(1M) man page for details. 

    phys-hahost1# luxadm stop -t 2 c3
    Caution - Caution -

    Do not run any Solstice DiskSuite commands while a SPARCstorage Array tray is spun down because the commands might have the side effect of spinning up some or all of the drives in the tray.

  14. Replace the disk.

    Refer to the hardware service manuals for your SPARCstorage Array for details on this procedure.

  15. Update the DID driver's database with the new device ID.

    Use the -l flag to scdidadm(1M) to identify the DID name for the lower-level device name of the drive to be replaced. Then update the DID drive database using the -R flag to scdidadm(1M). Refer to the Sun Cluster 2.2 Software Installation Guide for details on the DID pseudo driver.

    phys-hahost1# scdidadm -o name -l /dev/rdsk/c3t3d4
6	phys-hahost1:/dev/rdsk/c3t3d4	/dev/did/rdsk/d6
 phys-hahost1# scdidadm -R d6

  16. Make sure all disks in the affected multihost disk expansion unit spin up.

    The disks in the multihost disk expansion unit should spin up automatically. If the tray fails to spin up within two minutes, force the action by using the following command:

    phys-hahost1# luxadm start -t 2 c3

  17. Add the new disk back into the diskset by using the metaset(1M) command.

    This step automatically adds back the number of replicas that were deleted from the failed disk. The command syntax is as follows, where diskset is the name of the diskset containing the failed disk, and drive is the DID name of the disk in the form dN (for new installations of Sun Cluster), or cNtYdZ (for installations that upgraded from HA 1.3):

    # metaset -s diskset -a drive

  18. (Optional) If you deleted replicas that belonged to other disksets from disks that were in the same tray as the errored disk, use the metadb(1M) command to add back the replicas.

    phys-hahost1# metadb -s hahost2 -a deleted-replicas

    To add multiple replicas to the same slice, use the -c option.

  19. Use the scadmin(1M) command to reserve and enable failfast on the specified disk that has just been added to the diskset.

    phys-hahost2# scadmin reserve c3t3d4

  20. Use the format(1M) or fmthard(1M) command to repartition the new disk.

    Make sure that you partition the new disk exactly as the disk that was replaced. (Saving the disk format information was recommended in Chapter 1, Preparing for Sun Cluster Administration.)

  21. Use the metainit(1M) command to reinitialize disks that were cleared in Step 5. 

    phys-hahost1# metainit -s hahost1 d50

  22. Bring online all submirrors that were taken off line in Step 11.

    phys-hahost1# metaonline -s hahost1 d15 d35
phys-hahost1# metaonline -s hahost1 d24 d54
...

    Run the metaonline(1M) command as many times as necessary to bring online all the submirrors.

    When the submirrors are brought back online, Solstice DiskSuite automatically performs resyncs on all the submirrors, bringing all data up-to-date.

    Note -

    Running the metastat(1M) command at this time would show that all metadevices with components residing in the affected tray are resyncing.

  23. Attach submirrors that were detached in Step 4.

    Use the metattach(1M) command to perform this step. See the metattach(1M) man page for details. 

    phys-hahost1# metattach -s hahost1 d40 d50

  24. Replace any hot spares in use in the submirrors attached in Step 23.

    If a submirror had a hot spare replacement in use before you detached the submirror, this hot spare replacement will be in effect after the submirror is reattached. This step returns the hot spare to Available status. 

    phys-hahost1# metareplace -s hahost1 -e d40 c3t3d4s0

  25. Restore all hot spares that were deleted in Step 7.

    Use the metahs(1M) command to add back the hot spares. See the metahs(1M) man page for details. 

    phys-hahost1# metahs -s hahost1 -a hsp000 c3t2d5s0

  26. If necessary, switch logical hosts back to their default masters.

    phys-hahost1# haswitch phys-hahost2 hahost2

  27. Verify that the replacement corrected the problem.

    phys-hahost1# metastat -s hahost1

11.6.6 How to Replace a SPARCstorage Array Disk (SSVM or CVM)

In an SSVM or CVM configuration, it is possible to replace a SPARCstorage Array disk without halting the system, as long as the configuration is mirrored.

Note -

If you need to replace a disk in a bootable SPARCstorage Array, do not remove the SSA trays containing the boot disk of the hosts. Instead, shut down the host whose boot disk is present on that tray. Let the cluster software reconfigure the surviving nodes for failover to take effect before servicing the faulty disk. Refer to the SPARCstorage Array User's Guide for more information.

These are the high-level steps to replace a multihost disk in an SSVM environment using SPARCstorage Array 100 series disks.

These are the detailed steps to replace a multihost disk in an SSVM environment using SPARCstorage Array 100 series disks.

  1. If the replaced disk is a quorum device, use the scconf -q command to change the quorum device to a different disk.

  2. Identify all the volumes and corresponding plexes on the disks in the tray which contains the faulty disk.

    1. From the physical device address cNtNdN, obtain the controller number and the target number.

      For example, if the device address is c3t2d0, the controller number is 3 and the target is 2.

    2. Identify devices from a vxdisk list output.

      If the target is 0 or 1, identify all devices with physical addresses beginning with cNt0 and cNt1, where N is the controller number. If the target is 2 or 3, identify all devices with physical addresses beginning with cNt2 and cNt3. If the target is 4 or 5, identify all devices with physical addresses beginning with cNt4 and cNt5. Here is an example of how vxdisk can be used to obtain the information.

      # vxdisk -g diskgroup -q list | egrep c3t2\|c3t3 | nawk '{print $3}'

    3. Record the volume media name for the faulty disk from the output of the command.

      You will need this name in Step 10.

    4. Identify all plexes on the above devices by using the appropriate version (csh, ksh, or Bourne shell) of the following command.

      PLLIST=`vxprint -ptq -g diskgroup -e '(aslist.sd_dm_name in
 ("c3t2d0","c3t3d0","c3t3d1")) && (pl_kstate=ENABLED)' | nawk '{print $2}'`

      For csh, the syntax is set PLLIST .... For ksh, the syntax is export PLLIST= .... The Bourne shell requires the command export PLLIST after the variable is set.

  3. After you have set the variable, stop I/O to the volumes whose components (subdisks) are on the tray.

    Make sure all volumes associated with that tray are detached (mirrored or RAID5 configurations) or stopped (simple plexes). Issue the following command to detach a mirrored plex.

    # vxplex det ${PLLIST}

    An alternate command for detaching each plex in a tray is:

    # vxplex -g diskgroup -v volume det plex

    To stop I/O to simple plexes, unmount any file systems or stop database access.

    Note -

    Mirrored volumes will still be active because the other half of the mirror is still available.

  4. Remove the disk from the disk group.

    # vxdg -g diskgroup rmdisk diskname

  5. Spin down the disks in the tray.

    # luxadm stop -t tray controller

  6. Replace the faulty disk.

  7. Spin up the drives.

    # luxadm start -t tray controller

  8. Initialize the replacement disk.

    # vxdisksetup -i devicename

  9. Scan the current disk configuration again.

    Enter the following commands on all nodes in the cluster.

    # vxdctl enable
# vxdisk -a online

  10. Add the new disk to the disk group.

    The device-media-name is the volume media name recorded in Step 2.

    # vxdg -g diskgroup -k adddisk device-media-name=device-name

  11. Resynchronize the volumes.

    # vxrecover -g diskgroup -b -o

11.7 Administering SPARCstorage Array NVRAM

NVRAM supports the fast write capability for SPARCstorage Arrays. Without NVRAM, synchronous write requests from a program must be committed to disk, and an acknowledgment must be received by the program, before another request can be submitted. The NVRAM caches write requests in non-volatile memory and periodically flushes the data to disk. Once the data is in the NVRAM, an acknowledgment is returned to the program just as if the data had been written to disk. This enhances performance of write-intensive applications using SPARCstorage Arrays.

The procedures described here use the command-line interface. However, in Solstice DiskSuite configurations, you also can use the metatool graphical user interface to administer NVRAM for a disk, tray, or controller. For more information on Solstice DiskSuite, see the Solstice DiskSuite documentation.

Caution - Caution -

Use this functionality with care. It provides a powerful way to manage the SPARCstorage Array. Back up your data before using these procedures.

11.7.1 Enabling and Disabling NVRAM

Fast writes can be configured:

When fast write is enabled, it can be saved--across power cycles--as part of the SPARCstorage Array's configuration.

If the NVRAM battery is low, missing, or has failed, then fast write is disabled on the controller.

Before enabling fast write, you must stop all I/O to the controller or disk. In particular, ensure that diskset ownership has been released because an implicit I/O stream exists while ownership of a diskset is maintained. The following procedure explains how to stop all I/O.

Use the luxadm(1M) command to enable and disable NVRAM. Refer to the luxadm(1M) man page for complete information on this command.

Note -

For CVM, NVRAM should be disabled.

11.7.2 How to Enable and Disable NVRAM

These are the high-level steps to enable or disable NVRAM:

These are the detailed steps to enable or disable NVRAM.

  1. Identify the controller, tray, or individual disk whose NVRAM is to be enabled or disabled.

    You can use the luxadm(1M) command to display information for a specified controller, tray, or individual disk. For example, the following display identifies all of the disks on Controller c2. 

    phys-hahost1# luxadm display c2
                     SPARCstorage Array Configuration

 Controller path:
 /devices/iommu@f,e0000000/sbus@f,e0001000/SUNW,soc@0,0/SUNW,pln@a0000000,779a16:ctlr
                           DEVICE STATUS
       TRAY 1                 TRAY 2                 TRAY 3
 slot
 1     Drive: 0,0             Drive: 2,0             Drive: 4,0
 2     Drive: 0,1             Drive: 2,1             Drive: 4,1
 3     NO SELECT              NO SELECT              NO SELECT
 4     NO SELECT              NO SELECT              NO SELECT
 5     NO SELECT              NO SELECT              NO SELECT
 6     Drive: 1,0             Drive: 3,0             Drive: 5,0
 7     Drive: 1,1             NO SELECT              NO SELECT
 8     NO SELECT              NO SELECT              NO SELECT
 9     NO SELECT              NO SELECT              NO SELECT
 10    NO SELECT              NO SELECT              NO SELECT
                          CONTROLLER STATUS
 ...

  2. Stop all I/O to the affected device.

    Solstice DiskSuite:

    SSVM or CVM:

  3. Enable or disable fast write on the controller or individual disk.

    Use one of the three options to the luxadm(1M) command, depending on whether you are enabling fast write for all writes, enabling fast write only for synchronous writes, or disabling fast write.

    • -e enables fast write for all writes

    • -c enables fast write for only synchronous writes

    • -d disables fast write

    The following example saves the NVRAM configuration across power cycles and enables fast write for all writes. See the luxadm(1M) man page for details on these options.

    phys-hahost# luxadm fast_write -s -e pathname

    A confirmation appears, indicating that fast write has been enabled.

  4. Perform the steps needed to bring the component into normal operation under Sun Cluster.

    Solstice DiskSuite:

    SSVM or CVM:

11.7.3 Flushing and Purging NVRAM

The luxadm sync_cache command flushes any outstanding writes from NVRAM to the disk drive. If you get an error while flushing data, you must purge the data using the luxadm purge command. Purging data "throws away" any outstanding writes in NVRAM.

Caution - Caution -

Purging fast write data should be performed with caution, and only when a drive has failed, as it could result in the loss of data.

If the NVRAM battery is low, missing, or has failed, then NVRAM is non-functional and data is lost.

11.7.4 How to Flush and Purge NVRAM

These are the high-level steps to flush or purge all outstanding writes for the selected controller (and all disks) or individual writes from the NVRAM to disk:

These are the detailed steps to flush or purge NVRAM data.

  1. Identify the controller or the individual disk to flush or purge.

    You can use the luxadm(1M) command to display information for a specified controller, tray, or individual disk. For example, the following display identifies all of the disks on Controller c2. 

    phys-hahost1# luxadm display c2
                     SPARCstorage Array Configuration

 Controller path:
 /devices/iommu@f,e0000000/sbus@f,e0001000/SUNW,soc@0,0/SUNW,pln@a0000000,779a16:ctlr
                           DEVICE STATUS
       TRAY 1                 TRAY 2                 TRAY 3
 slot
 1     Drive: 0,0             Drive: 2,0             Drive: 4,0
 2     Drive: 0,1             Drive: 2,1             Drive: 4,1
 3     NO SELECT              NO SELECT              NO SELECT
 4     NO SELECT              NO SELECT              NO SELECT
 5     NO SELECT              NO SELECT              NO SELECT
 6     Drive: 1,0             Drive: 3,0             Drive: 5,0
 7     Drive: 1,1             NO SELECT              NO SELECT
 8     NO SELECT              NO SELECT              NO SELECT
 9     NO SELECT              NO SELECT              NO SELECT
 10    NO SELECT              NO SELECT              NO SELECT
                          CONTROLLER STATUS
 Vendor:        SUN    
 Product ID:    SSA110         
 Product Rev:   1.0
 Firmware Rev:  3.9
 Serial Num:    000000779A16
 Accumulate Performance Statistics: Enabled
 phys-hahost1#

  2. Stop all I/O to the affected device.

    Solstice DiskSuite:

    SSVM or CVM:

  3. Flush or purge the NVRAM on a controller, tray, or individual disk.

    If you can access drives in the SPARCstorage Array, flush the NVRAM. Only purge the NVRAM if you can no longer access the SPARCstorage Array or disk.

    phys-hahost1# luxadm sync_cache pathname
 or
 phys-hahost1# luxadm purge pathname

    A confirmation appears, indicating that NVRAM has been flushed or purged.

  4. Perform the steps needed to bring the component into normal operation under Sun Cluster.

    Solstice DiskSuite:

    SSVM or CVM: