Sun Cluster System Administration Guide for Solaris OS

Administering Quorum Overview

You can use the scconf(1M) command to perform all quorum administrative procedures. In addition, you can accomplish some procedures by using the scsetup(1M) interactive utility or the SunPlex Manager GUI. Whenever possible, quorum procedures are described in this chapter using scsetup. The SunPlex Manager online help describes how to perform quorum procedures using the GUI.

If any quorum-related scconf command is interrupted or fails, the quorum configuration information can become inconsistent in the cluster configuration database. If this occurs, either rerun the command or run scconf with the reset option to reset the quorum configuration.


Note –

The scsetup command is an interactive interface to the scconf command. When scsetup runs, the command generates scconf commands. These generated commands are shown in the examples at the end of the procedures.


There are two commands that can be used to view the quorum configuration: scstat -q option and scconf -p. Most of the verification steps in this chapter use scconf, but you can substitute scstat -q if you find its output more useful.

Table 5–1 Task List: Administering Quorum

Task 

For Instructions, Go To... 

Add a quorum device to a cluster 

    - Use scsetup(1M)

Adding a Quorum Device

Remove a quorum device from a cluster 

    - Use scsetup (to generate scconf)

How to Remove a Quorum Device

Remove the last quorum device from a cluster 

    - Use scsetup (to generate scconf)

How to Remove the Last Quorum Device From a Cluster

Replace a quorum device in a cluster 

    - Use the add and remove procedures 

How to Replace a Quorum Device

Modify a Quorum Device List    - Use the add and remove procedures 

How to Modify a Quorum Device Node List

Put a quorum device into maintenance state 

(While in maintenance state, the quorum device does not participate in voting to establish the quorum.) 

    - Use scsetup (to generate scconf)

How to Put a Quorum Device Into Maintenance State

Reset the quorum configuration to its default state 

    - Use scsetup (to generate scconf)

How to Bring a Quorum Device Out of Maintenance State

List the quorum devices and vote counts 

    - Use scconf(1M)

How to List the Quorum Configuration

Dynamic Reconfiguration With Quorum Devices

There are a few issues you must consider when completing dynamic reconfiguration (DR) operations on quorum devices in a cluster.

To remove a quorum device, you must complete the following steps, in the order indicated.

Table 5–2 Task Map: Dynamic Reconfiguration with Quorum Devices

Task 

For Instructions, Go To... 

1. Enable a new quorum device to replace the one being removed. 

Adding a Quorum Device

2. Disable the quorum device to be removed. 

How to Remove a Quorum Device

3. Perform the DR remove operation on the device being removed. 

Sun Enterprise 10000 DR Configuration Guide and the Sun Enterprise 10000 Dynamic Reconfiguration Reference Manual (from the Solaris 8 on Sun Hardware and Solaris 9 on Sun Hardware collections.)

Adding a Quorum Device

This section provides procedures to add a quorum device. For information on determining the number of quorum vote counts necessary for your cluster, recommended quorum configurations, and failure fencing, see Quorum and Quorum Devices in Sun Cluster Concepts Guide for Solaris OS.

Sun Cluster supports two types of quorum devices: SCSI and Network Appliance (NetApp) NAS. Procedures for adding these devices are provided in the following sections:

You can also accomplish these procedures by using the SunPlex Manager GUI. See the SunPlex Manager online help for more information.

See the scsetup(1M) and scconf(1M) man pages for information on the commands used in the following procedures.

ProcedureHow to Add a SCSI Quorum Device

To complete this procedure, identify a disk drive by its device ID (DID), which is shared by the nodes. Use the scdidadm command to see the list of DID names. Refer to the scdidadm(1M) man page for additional information.

Steps
  1. Become superuser on any node of the cluster.

  2. Start the scsetup utility.


    # scsetup
    

    The scsetup Main Menu is displayed.

  3. Type the number that corresponds to the option for Quorum.

    The Quorum Menu is displayed.

  4. Type the number that corresponds to the option for adding a quorum device, then type yes when the scsetup utility asks you to confirm that you want to add a quorum device.

    The scsetup utility asks what type of quorum device you want to add.

  5. Type the number that corresponds to the option for a SCSI quorum device.

    The scsetup utility asks which global device you want to use .

  6. Type the global device you are using.

    The scsetup utility asks you to confirm that the new quorum device should be added the global device you specified.

  7. Type yes to continue adding the new quorum device.

    If the new quorum device is added successfully, the scsetup utility displays a message to that affect.

  8. Verify that the quorum device has been added.


    # scstat -q
    
  9. Repeat Step 3 through Step 8 for each group of nodes that shares a storage enclosure.


Example 5–1 Adding a SCSI Quorum Device

The following example shows the scconf command generated by scsetup when it adds a SCSI quorum device and a verification step.


Become superuser on any cluster node.

[Execute the scsetup utility:]
# scsetup
Select Quorum>Add a quorum device.
Select SCSI device.
Type the global device your are using.
d20
Type yes to continue adding the new quorum device.

yes
[Verify that the scconf command completed successfully:]
 
scconf -a -q globaldev=d20
 
    Command completed successfully.
Quit the scsetup Quorum Menu and Main Menu.
[Verify the quorum device is added:]
# scstat -q
 
-- Quorum Summary --
 
  Quorum votes possible: 4
  Quorum votes needed:   3
  Quorum votes present:  4
 
-- Quorum Votes by Node --
 
                    Node Name           Present Possible Status
                    ---------           ------- -------- ------
  Node votes:       phys-schost-1       1        1       Online
  Node votes:       phys-schost-2       1        1       Online
 
-- Quorum Votes by Device --
 
                 Device Name         Present Possible Status
                 -----------         ------- -------- ------
  Device votes:  /dev/did/rdsk/d20s1  1        1       Online
  Device votes:  /dev/did/rdsk/d20s2  1        1       Online

ProcedureHow to Add a Network Appliance Network-Attached Storage (NAS) Quorum Device

When you use a Network Appliance (NetApp) network-attached storage (NAS) device as a quorum device, the following are required:

See the following Network Appliance NAS documentation for information about creating and setting up a Network Appliance NAS device and LUN. You can access the following documents at http://now.netapp.com.

See the following Sun Cluster documentation for information on installing a NetApp NAS storage device in a Sun Cluster environment: Sun Cluster 3.1 With Network-Attached Storage Devices Manual for Solaris OS.

Steps
  1. Make sure that all Sun Cluster nodes are online and can communicate with the NetApp clustered filer.

  2. Become superuser on any node of the cluster.

  3. Start the scsetup utility.


    # scsetup
    

    The scsetup Main Menu is displayed.

  4. Type the number that corresponds to the option for Quorum.

    The Quorum Menu is displayed.

  5. Type the number that corresponds to the option for adding a quorum device, then type yes when the scsetup utility asks you to confirm that you want to add a quorum device.

    The scsetup utility asks what type of quorum device you want to add.

  6. Type the number that corresponds to the option for a netapp_nas quorum device, then type yes when the scsetup utility asks you to confirm that you want to add a netapp_nas quorum device.

    The scsetup utility asks you provide the name of the new quorum device.

  7. Type the name of the quorum device you are adding.

    The quorum device name can be any name you choose. The name is only used to process future administrative commands.

    The scsetup utility asks you to provide the name of the filer for the new quorum device.

  8. Type the name of the filer of the new quorum device.

    This name is the network accessible name or address of the filer.

    The scsetup utility asks you to provide the LUN ID for the filer.

  9. Type the ID of the quorum device LUN on the filer.

    The scsetup utility asks you to confirm that the new quorum device should be added on the filer.

  10. Type yes to continue adding the new quorum device.

    If the new quorum device is added successfully, the scsetup utility displays a message to that affect.

  11. Verify that the quorum device has been added.


    # scstat -q
    
  12. Repeat Step 4 through Step 11 for each group of nodes that shares a storage enclosure.


Example 5–2 Adding a NetApp NAS Quorum Device

The following example shows the scconf command generated by scsetup when it adds a NetApp NAS quorum device and a verification step.


Make sure that all Sun Cluster nodes are online and can communicate with the NetApp clustered filer.

Become superuser on any cluster node.
[Execute the scsetup utility:]
# scsetup
Select Quorum>Add a quorum device.
Select Netapp_nas quorum device.
Type the name of the quorum device you are adding.

qd1
Type the name of the filer of the new quorum device.

nas1.sun.com
Type the ID of the quorum device LUN on the filer.
0
Type yes to continue adding the new quorum device.

yes
[Verify that the scconf command completed successfully:]
 
scconf -a -q name=qd1,type-=netapp_nas,filer=nas1.sun.com,lun_id=0
 
    Command completed successfully.
Quit the scsetup Quorum Menu and Main Menu.
[Verify the quorum device is added:]
# scstat -q
 
-- Quorum Summary --
 
  Quorum votes possible: 5
  Quorum votes needed:   3
  Quorum votes present:  5
 
-- Quorum Votes by Node --
 
                    Node Name           Present Possible Status
                    ---------           ------- -------- ------
  Node votes:       phys-schost-1       1        1       Online
  Node votes:       phys-schost-2       1        1       Online
 
-- Quorum Votes by Device --
 
                   Device Name         Present Possible Status
                   -----------         ------- -------- ------
 Device votes:     qd1                  1        1       Online
 Device votes:     /dev/did/rdsk/d3s2   1        1       Online
 Device votes:     /dev/did/rdsk/d4s2   1        1       Online

ProcedureHow to Remove a Quorum Device

You can also accomplish this procedure by using the SunPlex Manager GUI. See the SunPlex Manager online help for more information.

When a quorum device is removed, it no longer participates in the voting to establish quorum. Note that all two-node clusters require that at least one quorum device be configured. If this is the last quorum device on a cluster, scconf(1M) will fail to remove the device from the configuration.


Note –

If the device you intend to remove is the last quorum device in the cluster, see the procedure How to Remove the Last Quorum Device From a Cluster.


Steps
  1. Become superuser on any node in the cluster.

  2. Determine the quorum device to be removed.


    # scconf -pv | grep Quorum
    
  3. Execute the scsetup(1M) utility.


    # scsetup
    

    The Main Menu is displayed.

  4. Type the number that corresponds to the option for Quorum.

  5. Type the number that corresponds with the option to remove a quorum device.

    Answer the questions displayed during the removal process.

  6. Quit scsetup.

  7. Verify that the quorum device is removed.


    # scstat -q
    

Example 5–3 Removing a Quorum Device

This example shows how to remove a quorum device from a cluster with two or more quorum devices configured.


Become superuser on any node and place the node to be removed in maintenance state.

[Determine the quorum device to be removed:]
# scconf -pv | grep Quorum
[Execute the scsetup utility:]
# scsetup
Select Quorum>Remove a quorum device.
Answer the questions when prompted.
[Verify that the scconf command completed successfully:]
 
scconf -r -q globaldev=d4
 
    Command completed successfully.
Quit the scsetup Quorum Menu and Main Menu.
[Verify the quorum device is removed:]
# scstat -q
 
-- Quorum Summary --
 
  Quorum votes possible:      3
  Quorum votes needed:        2
  Quorum votes present:       3
 
-- Quorum Votes by Node --
 
                    Node Name           Present Possible Status
                    ---------           ------- -------- ------
  Node votes:       phys-schost-1       1        1       Online
  Node votes:       phys-schost-2       1        1       Online
 
-- Quorum Votes by Device --
 
                 Device Name         Present Possible Status
                 -----------         ------- -------- ------
  Device votes:  /dev/did/rdsk/d3s2  1        1       Online

ProcedureHow to Remove the Last Quorum Device From a Cluster

If the device you intend to remove is not the last quorum device in the cluster, use the previous procedure, How to Remove a Quorum Device.


Note –

All two-node clusters require at least one configured quorum device. If this is the last quorum device on a two-node cluster, the cluster must be placed into install mode before scconf(1M) allows you to remove the device from the configuration. This should only be done if a node is being removed from the cluster.


Steps
  1. Become superuser on any node in the cluster and place the node to be removed in maintenance state.

    See How to Put a Node Into Maintenance State.

  2. Place the cluster in install mode.


    # scconf -c -q installmode
    
  3. Remove the quorum device using the scconf command.

    The scsetup(1M) cluster-administration menu options are not available while the cluster is in install mode.


    # scconf -r -q name=device
    
  4. Verify that the quorum device has been removed.


    # scstat -q
    

Example 5–4 Removing the Last Quorum Device

This example shows how to remove the last remaining quorum device in a cluster configuration.


[Become superuser on any node.]
[Place the cluster in install mode:]
# scconf -c -q installmode
[Remove the quorum device:]
# scconf -r -q name=d3
[Verify the quorum device has been removed:]
# scstat -q
 
-- Quorum Summary --
 
  Quorum votes possible:      2
  Quorum votes needed:        2
  Quorum votes present:       2
 
-- Quorum Votes by Node --
 
                    Node Name           Present Possible Status
                    ---------           ------- -------- ------
  Node votes:       phys-schost-1       1        1       Online
  Node votes:       phys-schost-2       1        1       Online
 
-- Quorum Votes by Device --
 
                    Device Name         Present Possible Status
                    -----------         ------- -------- ------

ProcedureHow to Replace a Quorum Device

Use this procedure to replace an existing quorum device with another quorum device. You can replace a quorum device with a similar device type, such as replacing a NAS device with another NAS device, or you can replace the device with a dissimilar device, such as replacing a NAS device with a shared disk.

Steps
  1. Configure a new quorum device.

    You need to first add a new quorum device to the configuration to take the place of the old device. See Adding a Quorum Device to add a new quorum device to the cluster.

  2. Remove the device that you are replacing as a quorum device.

    See How to Remove a Quorum Device to remove the old quorum device from the configuration.

  3. If the quorum device is a failed disk, replace the disk.

    Refer to the hardware procedures for your disk enclosure in the Sun Cluster 3.0-3.1 Hardware Administration Manual for Solaris OS.

ProcedureHow to Modify a Quorum Device Node List

You can use the scsetup(1M) utility to add a node to or remove a node from the node list of an existing quorum device. To modify a quorum device's node list, you must remove the quorum device, modify the physical connections of nodes to the quorum device you removed, then add the quorum device to the cluster configuration again. When a quorum device is added, scconf(1M) automatically configures the node-to-disk paths for all nodes attached to the disk.

Steps
  1. Become superuser on any node of the cluster.

  2. Determine the name of the quorum device you are modifying.


    # scconf -p | grep Quorum
    
  3. Enter the scsetup utility.


    # scsetup
    

    The Main Menu is displayed.

  4. Type the number that corresponds to the Quorum option.

    The Quorum Menu is displayed.

  5. Type the number that corresponds to the option to remove a quorum device.

    Follow the instructions. You will be asked the name of the disk to be removed.

  6. Add or delete the physical node connections to the quorum device.

  7. Type the number that corresponds to the option to add a quorum device.

    Follow the instructions. You will be asked the name of the disk to be used as the quorum device.

  8. Verify that the quorum device has been added.


    # scstat -q
    

Example 5–5 Modifying a Quorum Device Node List

The following example shows how to use the scsetup utility to add nodes to or delete nodes from a quorum device node list. In this example, the quorum device name is d2, and the final result of the procedures adds another node to the quorum device node list.


[Become superuser on any node in the cluster.
]
[Determine the quorum device name:]
# scconf -p | grep Quorum
Quorum devices:                                    d2
Quorum device name:                                d2
  Quorum device votes:                             1
  Quorum device enabled:                           yes
  Quorum device name:                              /dev/did/rdsk/d2s2
  Quorum device hosts (enabled):                   phys-schost-1 phys-schost-2
  Quorum device hosts (disabled): 
[Execute the utility:]
# scsetup
Type the number that corresponds with the quorum option.
.
Type the number that corresponds with the option to remove a quorum device.
.
Answer the questions when prompted.
   You Will Need:         
Example:
   quorum device name     d2
[Verify that the scconf command completed successfully:]
 
scconf -r -q globaldev=d2
Command completed successfully.

Type the number that corresponds with the Quorum option.
.
Type the number that corresponds with the option to add a quorum device.
.Answer the questions when prompted.
   You Will Need:         Example:

   quorum device name     d2

[Verify that the scconf command completed successfully:] 

scconf -a -q globaldev=d2
Command completed successfully.

Quit the scsetup utility.
[Verify that the correct nodes have paths to the quorum device. 
In this example, note that phys-schost-3 has been added to the 
enabled hosts list.]
# scconf -p | grep Quorum
Quorum devices:                                    d2
Quorum device name:                                d2
  Quorum device votes:                             2
  Quorum device enabled:                           yes
  Quorum device name:                              /dev/did/rdsk/d2s2
  Quorum device hosts (enabled):                   phys-schost-1 phys-schost-2
                                                   phys-schost-3
  Quorum device hosts (disabled): 
[Verify that the modified quorum device is online.]
# scstat -q
-- Quorum Votes by Device --
                    Device Name         Present Possible Status
                    -----------         ------- -------- ------
  Device votes:     /dev/did/rdsk/d2s2  1        1       Online
[Verify the quorum device is removed:]
# scstat -q
 
-- Quorum Summary --
 
  Quorum votes possible: 4
  Quorum votes needed:   3
  Quorum votes present:  4
 
-- Quorum Votes by Node --
 
                    Node Name           Present Possible Status
                    ---------           ------- -------- ------
  Node votes:       phys-schost-1       1        1       Online
  Node votes:       phys-schost-2       1        1       Online
 
-- Quorum Votes by Device --
 
                 Device Name         Present Possible Status
                 -----------         ------- -------- ------
  Device votes:  /dev/did/rdsk/d3s2  1        1       Online
  Device votes:  /dev/did/rdsk/d4s2  1        1       Online

ProcedureHow to Put a Quorum Device Into Maintenance State

Use the scconf(1M) command to put a quorum device into maintenance state. The scsetup(1M) utility does not currently have this capability. You can also accomplish this procedure by using the SunPlex Manager GUI. See the SunPlex Manager online help for more information.

Put a quorum device into maintenance state when taking the quorum device out of service for an extended period of time. This way, the quorum device's quorum vote count is set to zero and does not contribute to the quorum count while the device is being serviced. While in maintenance state, the quorum device's configuration information is preserved.


Note –

All two-node clusters require at least one configured quorum device. If this is the last quorum device on a two-node cluster, scconf will fail to put the device into maintenance state.


To put a cluster node into maintenance state, see How to Put a Node Into Maintenance State.

Steps
  1. Become superuser on any node of the cluster.

  2. Put the quorum device into the maintenance state.


    # scconf -c -q name=device,maintstate
    
    -c

    Specifies the change form of the scconf command.

    -q

    Manages the quorum options.

    name=device

    Specifies the DID name of the disk device to change, for example, d4.

    maintstate

    Puts the shared quorum device into maintenance state.

  3. Verify that the quorum device is now in maintenance state.

    The output for the device you placed in maintenance state should read zero for the Quorum Device Votes.


    # scconf -p | grep -i quorum
    

Example 5–6 Putting a Quorum Device Into Maintenance State

The following example shows how to put a quorum device into maintenance state and how to verify the results.


# scconf -c -q name=d20,maintstate
# scconf -p | grep -i quorum
  Node quorum vote count:                          1
  Node quorum vote count:                          1
Quorum devices:                                    d20
Quorum device name:                                d20
  Quorum device votes:                             0
  Quorum device enabled:                           no
  Quorum device name:                              /dev/did/rdsk/d20s2
  Quorum device hosts (enabled):                   phys-schost-2 phys-schost-3
  Quorum device hosts (disabled): 

See Also

To re-enable the quorum device, see How to Bring a Quorum Device Out of Maintenance State.

To put a node into maintenance state, see How to Put a Node Into Maintenance State.

ProcedureHow to Bring a Quorum Device Out of Maintenance State

Run this procedure each time a quorum device is in maintenance state and you want to bring the quorum device out of maintenance state and reset the quorum vote count to the default.


Caution – Caution –

If you do not specify either the globaldev or node options, the quorum count is reset for the entire cluster.


When you configure a quorum device, Sun Cluster software assigns the quorum device a vote count of N-1 where N is the number of connected votes to the quorum device. For example, a quorum device that is connected to two nodes with nonzero vote counts has a quorum count of one (two minus one).

Steps
  1. Become superuser on any node of the cluster.

  2. Reset the quorum count.


    # scconf -c -q name=device,reset
    
    -c

    Specifies the change form of the scconf command.

    -q

    Manages the quorum options.

    name=device

    Specifies the DID name of the quorum device to reset, for example, d4.

    reset

    The change flag that resets quorum.

  3. If you are resetting the quorum count because a node was in maintenance state, reboot the node.

  4. Verify the quorum vote count.


    # scconf -p | grep -i quorum
    

Example 5–7 Resetting the Quorum Vote Count (Quorum Device)

The following example resets the quorum count for a quorum device back to the default and verifies the result.


# scconf -c -q name=d20,reset
# scconf -p | grep -i quorum
  Node quorum vote count:                          1
  Node quorum vote count:                          1
Quorum devices:                                    d20
Quorum device name:                                d20
  Quorum device votes:                             1
  Quorum device enabled:                           yes
  Quorum device name:                              /dev/did/rdsk/d20s2
  Quorum device hosts (enabled):                   phys-schost-2 phys-schost-3
  Quorum device hosts (disabled): 

ProcedureHow to List the Quorum Configuration

You can also accomplish this procedure by using the SunPlex Manager GUI. See the SunPlex Manager online help for more information.

You do not need to be superuser to list the quorum configuration.


Note –

When you increase or decrease the number of node attachments to a quorum device, the quorum vote count is not automatically recalculated. You can reestablish the correct quorum vote if you remove all quorum devices and then add them back into the configuration. For a two-node cluster, temporarily add a new quorum device before you remove and add back the original quorum device. Then remove the temporary quorum device.


Step

    Use scconf(1M) to list the quorum configuration.


    # scconf -p | grep -i quorum
    

Example 5–8 Listing the Quorum Configuration


# scconf -p | grep “Quorum | vote”

 Node quorum vote count:                           1
 Node quorum vote count:                           1
Quorum devices:                                    d20
Quorum device name:                                d20
  Quorum device votes:                             1
  Quorum device enabled:                           yes
  Quorum device name:                              /dev/did/rdsk/d20s2
  Quorum device hosts (enabled):                   phys-schost-2 phys-schost-3
  Quorum device hosts (disabled):

ProcedureHow to Repair a Quorum Device

Use this procedure to replace a malfunctioning quorum device.

Steps
  1. Remove the disk device that you are replacing as a quorum device.


    Note –

    If the device you intend to remove is the last quorum device, you might want to first add another disk as a new quorum device. This will ensure that there will be a valid quorum device in case there is a failure during the replacement procedure. See Adding a Quorum Device to add a new quorum device.


    See How to Remove a Quorum Device to remove a disk device as a quorum device .

  2. Replace the disk device.

    To replace the disk device, see the hardware procedures for the disk enclosure in the Sun Cluster 3.0-3.1 Hardware Administration Manual for Solaris OS.

  3. Add the replaced disk as a new quorum device.

    See Adding a Quorum Device to add a disk as a new quorum device.


    Note –

    If you added an additional quorum device in Step 1, it is now safe to remove it. See How to Remove a Quorum Device to remove the quorum device.