Sun Cluster System Administration Guide for Solaris OS

Adding a Quorum Device

This section provides procedures to add a quorum device. For information about determining the number of quorum vote counts necessary for your cluster, recommended quorum configurations, and failure fencing, see Quorum and Quorum Devices in Sun Cluster Concepts Guide for Solaris OS.


Caution – Caution –

Do not add a disk that is currently configured as a quorum device to a Solaris ZFS storage pool. When a configured quorum device is added to a Solaris ZFS storage pool, the disk is relabeled as an EFI disk and quorum configuration information is lost and the disk no longer provides a quorum vote to the cluster. Once a disk is in a storage pool, that disk can then be configured as a quorum device. You can also unconfigure the disk, add it to the storage pool, and then reconfigure the disk as a quorum device.


The Sun Cluster software supports the following types of quorum devices:

Procedures for adding these devices are provided in the following sections:


Note –

You cannot configure replicated disks as quorum devices. If you try to add a replicated disk as a quorum device, you receive the following error message and the command exits with an error code.


Disk-name is a replicated device. Replicated devices cannot be
configured as quorum devices.

A shared-disk quorum device is any attached storage device that is supported by Sun Cluster software. The shared disk is connected to two or more nodes of your cluster. If you turn fencing on, a dual-ported disk can be configured as a quorum device that uses SCSI-2 or SCSI-3 (the default is SCSI-2). If fencing is turned on and your shared device is connected to more than two nodes, you can configure your shared disk as a quorum device that uses the SCSI-3 protocol (the default protocol for more than two nodes). You can use the SCSI override flag to make the Sun Cluster software use the SCSI-3 protocol for dual-ported shared disks.

If you turn fencing off for a shared disk, you can then configure the disk as a quorum device that uses the software quorum protocol. This would be true regardless of whether the disk supports SCSI-2 or SCSI-3 protocols. Software quorum is a protocol developed by Sun Microsystems that emulates a form of SCSI Persistent Group Reservations (PGR).


Caution – Caution –

If you are using disks that do not support SCSI (such as SATA), you should turn SCSI fencing off.


For quorum devices, you can use a disk that contains user data or is a member of a device group. View the protocol that is used by the quorum subsystem with a shared disk by looking at the access-mode value for the shared disk in the output from the cluster show command.

You can also accomplish these procedures by using the Sun Cluster Manager GUI. See the Sun Cluster Manager online help for more information.

See the clsetup(1CL) and clquorum(1CL) man pages for information about the commands that are used in the following procedures.

ProcedureHow to Add a Shared Disk Quorum Device

Sun Cluster software supports shared-disk (both SCSI and SATA) devices as quorum devices. A SATA device does not support a SCSI reservation, and you must disable the SCSI reservation fencing flag and use the software quorum protocol to configure these disks as quorum devices.

To complete this procedure, identify a disk drive by its device ID (DID), which is shared by the nodes. Use the cldevice show command to see the list of DID names. Refer to the cldevice(1CL) man page for additional information.

Use this procedure to configure SCSI or SATA devices.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any node of the cluster.

  2. Start the clsetup utility.


    # clsetup
    

    The clsetup Main Menu is displayed.

  3. Type the number that corresponds to the option for Quorum.

    The Quorum Menu is displayed.

  4. Type the number that corresponds to the option for adding a quorum device, then type yes when the clsetup utility asks you to confirm the quorum device that you are adding.

    The clsetup utility asks what type of quorum device you want to add.

  5. Type the number that corresponds to the option for a shared-disk quorum device.

    The clsetup utility asks which global device you want to use.

  6. Type the global device you are using.

    The clsetup utility asks you to confirm that the new quorum device should be added to the global device you specified.

  7. Type yes to continue adding the new quorum device.

    If the new quorum device is added successfully, the clsetup utility displays a message to that effect.

  8. Verify that the quorum device has been added.


    # clquorum list -v
    

Example 6–1 Adding a Shared Disk Quorum Device

The following example shows the clquorum command generated by clsetup when it adds a shared-disk quorum device and a verification step.


Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any cluster node.

[Start the clsetup utility:]
# clsetup
[Select Quorum>Add a quorum device]
[Answer the questions when prompted.]
[You will need the following information.]
  [Information:                       Example:]
  [Directly attached shared disk      shared_disk]
  [Global device		                  d20]

[Verify that the clquorum command was completed successfully:]
 clquorum add d20
 
    Command completed successfully.
[Quit the clsetup Quorum Menu and Main Menu.]
[Verify that the quorum device is added:]
# clquorum list -v
 
Quorum         Type
-------         ----
d20              shared_disk
scphyshost-1    node
scphyshost-2    node

ProcedureHow to Add a Sun NAS Quorum Device

To complete this procedure, identify a disk drive by its device ID (DID), which is shared by the nodes. Use the cldevice show command to see the list of DID names. Refer to the cldevice(1CL) man page for additional information.


Note –

Sun Cluster supports a Sun NAS quorum device only on a two-node cluster configuration.


The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Use the Sun NAS GUI to set up an iSCSI device on the Sun NAS filer.

    1. Create a file volume that is approximately 50 Mbytes in size.


      File Volume Operations -> Create File Volume
    2. For each node, create an iSCSI access list.


      iSCSI Configuration -> Configure Access List
      1. Use the name of the cluster as the iSCSI access list name.

      2. Add the initiator node name of each cluster node to the access list. CHAP and IQN are not needed.

    3. Configure the iSCSI LUN


      iSCSI Configuration -> Configure iSCSI LUN

      You can use the name of the backing file volume as the name of the LUN. Add the access list for each node to the LUN.

  2. On each of the cluster nodes, discover the iSCSI LUN and set the iSCSI access list to static configuration.


    # iscsiadm modify discovery -s enable
    
    	# iscsiadm list discovery
    	Discovery:
    	        Static: enabled
    	        Send Targets: disabled
    	        iSNS: disabled
    
    	#  iscsiadm add static-config iqn.LUNName,IPAddress_of_NASDevice
    # devfsadm -i iscsi
    # cldevice refresh
    
  3. From one cluster node, configure the DIDs for the iSCSI LUN.


    # /usr/cluster/bin/scgdevs
  4. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any node of the cluster.

  5. Start the clsetup utility.


    # clsetup
    

    The clsetup Main Menu is displayed.

  6. Type the number that corresponds to the option for Quorum.

    The Quorum Menu is displayed.

  7. Type the number that corresponds to the option for adding a quorum device, then type yes when the clsetup utility asks you to confirm the quorum device that you are adding.

    The clsetup utility asks what type of quorum device you want to add.

  8. Type the number that corresponds to the option for a shared-disk quorum device.

    The clsetup utility asks which global device you want to use.

  9. Type the global device you are using.

    The clsetup utility asks you to confirm that the new quorum device should be added to the global device you specified.

  10. Type yes to continue adding the new quorum device.

    If the new quorum device is added successfully, the clsetup utility displays a message to that effect.

  11. Verify that the quorum device has been added.


    # clquorum list -v
    

Example 6–2 Adding a Sun NAS Quorum Device

The following example shows the clquorum command generated by clsetup when it adds a Sun NAS quorum device and a verification step.


Add an iSCSI device on the Sun NAS filer.
Use the Sun NAS GUI to create a file volume that is approximately 50mb in size.
File Volume Operations -> Create File Volume
For each node, create an iSCSI access list.
iSCSI Configuration -> Configure Access List
Add the initiator node name of each cluster node to the access list.
*** Need GUI or command syntax for this step. ***
Configure the iSCSI LUN
iSCSI Configuration -> Configure iSCSI LUN
On each of the cluster nodes, discover the iSCSI LUN and set the iSCSI access list to static configuration.
iscsiadm modify discovery -s enable
iscsiadm list discovery
Discovery:
   Static: disable
   Send Targets: enables
   iSNS: disabled
iscsiadm add status-config
iqn.1986-03.com.sun0-1:000e0c66efe8.4604DE16.thinquorum,10.11.160.20
devsadm -i iscsi
From one cluster node, configure the DID devices for the iSCSI LUN.
/usr/cluster/bin/scgdevs
/usr/cluster/bin/scgdevs
Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any cluster node.

[Start the clsetup utility:]
# clsetup
[Select Quorum>Add a quorum device]
[Answer the questions when prompted.]
[You will need the following information.]
  [Information:                           Example:]
  [Directly attached shared disk device   shared_disk]
  [Global device		                      d20

[Verify that the clquorum command was completed successfully:]
 clquorum add d20
 
    Command completed successfully.
[Quit the clsetup Quorum Menu and Main Menu.]
[Verify that the quorum device is added:]
# clquorum list -v
 
Quorum         Type
-------         ----
d20             shared_disk
scphyshost-1    node
scphyshost-2    node

ProcedureHow to Add a Network Appliance Network-Attached Storage (NAS) Quorum Device

When you use a Network Appliance (NetApp) network-attached storage (NAS) device as a quorum device, the following are required:

See the following Sun Cluster documentation for information about installing a NetApp NAS storage device in a Sun Cluster environment: Sun Cluster 3.1 - 3.2 With Network-Attached Storage Devices Manual for Solaris OS.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Make sure that all Sun Cluster nodes are online and can communicate with the NetApp clustered filer.

  2. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any node of the cluster.

  3. Start the clsetup utility.


    # clsetup
    

    The clsetup Main Menu is displayed.

  4. Type the number that corresponds to the option for Quorum.

    The Quorum Menu is displayed.

  5. Type the number that corresponds to the option for adding a quorum device. Then type yes to confirm that you are adding a quorum device.

    The clsetup utility asks what type of quorum device you want to add.

  6. Type the number that corresponds to the option for a netapp_nas quorum device. Then type yes to confirm that you are adding a netapp_nas quorum device.

    The clsetup utility asks you provide the name of the new quorum device.

  7. Type the name of the quorum device you are adding.

    The quorum device name can be any name you choose. The name is only used to process future administrative commands.

    The clsetup utility asks you to provide the name of the filer for the new quorum device.

  8. Type the name of the filer of the new quorum device.

    This name is the network-accessible name or address of the filer.

    The clsetup utility asks you to provide the LUN ID for the filer.

  9. Type the ID of the quorum device LUN on the filer.

    The clsetup utility asks if to the new quorum device should be added on the filer.

  10. Type yes to continue adding the new quorum device.

    If the new quorum device is added successfully, the clsetup utility displays a message to that effect.

  11. Verify that the quorum device has been added.


    # clquorum list -v
    

Example 6–3 Adding a NetApp NAS Quorum Device

The following example shows the clquorum command generated by clsetup when it adds a NetApp NAS quorum device. The example also shows a verification step.


Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any cluster node.

[Start the clsetup utility:]
# clsetup
[Select Quorum>Add a quorum device]
[Answer the questions when prompted.]
[You will need the following information.]
  [Information:                Example:]
  [Quorum Device               Netapp_nas quorum device]
  [Name:        		         qd1]
  [Filer:                      nas1.sun.com]
  [LUN ID:                     0]

[Verify that the clquorum command was completed successfully:]
 clquorum add -t netapp_nas -p filer=nas1.sun.com,-p lun_id=0 qd1
 
    Command completed successfully.
[Quit the clsetup Quorum Menu and Main Menu.]
[Verify that the quorum device is added:]
# clquorum list -v
 
Quorum         Type
-------         ----
qd1             netapp_nas
scphyshost-1    node
scphyshost-2    node

ProcedureHow to Add a Quorum Server Quorum Device

Before You Begin

Before you can add a Sun Cluster Quorum Server as a quorum device, the Sun Cluster Quorum Server software must be installed on the host machine and the quorum server must be started and running. For information about installing the quorum server, see the How to Install and Configure Quorum Server Software in Sun Cluster Software Installation Guide for Solaris OS.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any node of the cluster.

  2. Make sure that all Sun Cluster nodes are online and can communicate with the Sun Cluster Quorum Server.

    1. Ensure that network switches that are directly connected to cluster nodes meet one of the following criteria:

      • The switch supports Rapid Spanning Tree Protocol (RSTP).

      • Fast port mode is enabled on the switch.

      One of these features is required to ensure immediate communication between cluster nodes and the quorum server. If this communication is significantly delayed by the switch, the cluster interprets this prevention of communication as loss of the quorum device.

    2. If the public network uses variable-length subnetting, also called Classless Inter-Domain Routing (CIDR), modify the following files on each node.

      If you use classful subnets, as defined in RFC 791, you do not need to perform these steps.

      1. Add to the /etc/inet/netmasks file an entry for each public subnet that the cluster uses.

        The following is an example entry which contains a public-network IP address and netmask:


        10.11.30.0	255.255.255.0
      2. Append netmask + broadcast + to the hostname entry in each /etc/hostname.adapter file.


        nodename netmask + broadcast +
        
    3. On each node in the cluster, add the quorum server hostname to the /etc/inet/hosts file or the /etc/inet/ipnodes file.

      Add a hostname-to-address mapping to the file, such as the following.


      ipaddress qshost1
      
      ipaddress

      The IP address of the computer where the quorum server is running.

      qshost1

      The hostname of the computer where the quorum server is running.

    4. If you use a naming service, add the quorum server host's name-to-address mapping to the name-service database.

  3. Start the clsetup utility.


    # clsetup
    

    The clsetup Main Menu is displayed.

  4. Type the number that corresponds to the option for Quorum.

    The Quorum Menu is displayed.

  5. Type the number that corresponds to the option for adding a quorum device. Then type yes to confirm that you are adding a quorum device.

    The clsetup utility asks what type of quorum device you want to add.

  6. Type the number that corresponds to the option for a quorum-server quorum device. Then type yes to confirm that you are adding a quorum-server quorum device.

    The clsetup utility asks you to provide the name of the new quorum device.

  7. Type the name of the quorum device you are adding.

    The quorum device name can be any name you choose. The name is only used to process future administrative commands.

    The clsetup utility asks you to provide the name of the filer for the new quorum device.

  8. Type the name of the host of the quorum server.

    This name specifies the IP address of the machine where the quorum server runs or the hostname of the machine on the network.

    Depending on the IPv4 or IPv6 configuration of the host, the IP address of the machine must be specified in the /etc/hosts file, the /etc/inet/ipnodes file, or both.


    Note –

    The machine you specify must be reachable by all cluster nodes and must run the quorum server.


    The clsetup utility asks you to provide the port number of the quorum server.

  9. Type the port number that is used by the quorum server to communicate with the cluster nodes.

    The clsetup utility asks you to confirm that the new quorum device should be added.

  10. Type yes to continue adding the new quorum device.

    If the new quorum device is added successfully, the clsetup utility displays a message to that effect.

  11. Verify that the quorum device has been added.


    # clquorum list -v
    

Example 6–4 Adding a Quorum Server Quorum Device

The following example shows the clquorum command that is generated by clsetup when it adds a quorum server quorum device. The example also shows a verification step.


Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on any cluster node.

[Start the clsetup utility:]
# clsetup
[Select Quorum > Add a quorum device]
[Answer the questions when prompted.]
[You will need the following information.]
  [Information:                Example:]
  [Quorum Device               quorum_server quorum device]
  [Name:        		           qd1]
  [Host Machine Name:          10.11.124.84]
  [Port Number:                9001]

[Verify that the clquorum command was completed successfully:]
 clquorum add -t quorum_server -p qshost=10.11.124.84,-p port=9001 qd1
 
    Command completed successfully.
[Quit the clsetup Quorum Menu and Main Menu.]
[Verify that the quorum device is added:]
# clquorum list -v
 
Quorum         Type
-------         ----
qd1             quorum_server
scphyshost-1    node
scphyshost-2    node

# clquorum status 

=== Cluster Quorum ===
-- Quorum Votes Summary --
 
        Needed     Present    Possible
        ------     -------    -------- 
        3          5           5
   
-- Quorum Votes by Node --
 
  Node Name        Present    Possible   Status
  ---------        -------    --------  ------
  phys-schost-1    1          1          Online
  phys-schost-2    1          1          Online
 
-- Quorum Votes by Device --
 
  Device Name       Present   Possible   Status
  -----------       -------   --------   ------
  qd1               1         1          Online
  d3s2              1         1          Online
  d4s2              1         1          Online