Sun Cluster System Administration Guide for Solaris OS

Administering the Cluster Overview

Table 7–1 Task List: Administering the Cluster

Task 

For Instructions, Go To 

Change the name of the cluster 

How to Change the Cluster Name

List node IDs and their corresponding node names 

How to Map Node ID to Node Name

Permit or deny new nodes to add themselves to the cluster 

How to Work With New Cluster Node Authentication

Change the time for a cluster using the Network Time Protocol (NTP) 

How to Reset the Time of Day in a Cluster

Bring down a node to the OpenBoot PROM ok prompt on a SPARC based system or to a Boot Subsystem on an x86 based system

SPARC: How to Enter the OpenBoot PROM (OBP) on a Node

Change the private hostname 

How to Change the Private Hostname

Put a cluster node in maintenance state 

How to Put a Node Into Maintenance State

Bring a cluster node out of maintenance state 

How to Bring a Node Out of Maintenance State

Add a node to a cluster 

How to Add a Node to the Authorized Node List

Remove a node from a cluster 

How to Remove a Node From the Cluster Software Configuration

ProcedureHow to Change the Cluster Name

If necessary, you can change the cluster name after initial installation.

Steps
  1. Become superuser on any node in the cluster.

  2. Type the scsetup command.


    # scsetup
    

    The Main Menu is displayed.

  3. To change the cluster name, type 8 (Other cluster properties).

    The Other Cluster Properties menu is displayed.

  4. Make your selection from the menu and follow the onscreen instructions.


Example 7–1 Changing the Cluster Name

The following example shows the scconf(1M) command generated from the scsetup(1M) utility to change to the new cluster name, dromedary.


# scconf -c -C cluster=dromedary

ProcedureHow to Map Node ID to Node Name

During Sun Cluster installation, each node is automatically assigned a unique node ID number. The node ID number is assigned to a node in the order in which it joins the cluster for the first time; once assigned, the number cannot be changed. The node ID number is often used in error messages to identify which cluster node the message concerns. Use this procedure to determine the mapping between node IDs and node names.

You do not need to be superuser to list configuration information.

Step

    Use the scconf(1M) command to list the cluster configuration information.


    % scconf -pv | grep "Node ID"
    

Example 7–2 Mapping the Node ID to the Node Name

The following example shows the node ID assignments


% scconf -pv | grep “Node ID”
(phys-schost-1) Node ID:                    1
(phys-schost-2) Node ID:                    2
(phys-schost-3) Node ID:                    3

ProcedureHow to Work With New Cluster Node Authentication

Sun Cluster enables you to determine if new nodes can add themselves to the cluster and with what type of authentication. You can permit any new node to join the cluster over the public network, deny new nodes from joining the cluster, or indicate a specific node that can join the cluster. New nodes can be authenticated by using either standard UNIX or Diffie-Hellman (DES) authentication. If you select DES authentication, you must also configure all necessary encryption keys before a node can join. See the keyserv(1M) and publickey(4) man pages for more information.

Steps
  1. Become superuser on any node in the cluster.

  2. Enter the scsetup(1M) utility.


    # scsetup
    

    The Main Menu is displayed.

  3. To work with cluster authentication, type 7 (New nodes).

    The New Nodes menu is displayed.

  4. Make your selection from the menu and follow the onscreen instructions.


Example 7–3 Preventing New Machines From Being Added to the Cluster

The following example shows the scconf command generated from the scsetup utility that would prevent new machines from being added to the cluster.


# scconf -a -T node=.


Example 7–4 Permitting All New Machines to Be Added to the Cluster

The following example shows the scconf command generated from the scsetup utility that would enable all new machines to be added to the cluster.


# scconf -r -T all


Example 7–5 Specifying a New Machine to Be Added to the Cluster

The following example shows the scconf command generated from the scsetup utility to enable a single new machine to be added to the cluster.


# scconf -a -T node=phys-schost-4


Example 7–6 Setting the Authentication to Standard UNIX

The following example shows the scconf command generated from the scsetup utility to reset to standard UNIX authentication for new nodes joining the cluster.


# scconf -c -T authtype=unix


Example 7–7 Setting the Authentication to DES

The following example shows the scconf command generated from the scsetup utility to use DES authentication for new nodes joining the cluster.


# scconf -c -T authtype=des

When using DES authentication, you need to also configure all necessary encryption keys before a node can join the cluster. See the keyserv(1M) and publickey(4) man pages for more information.


ProcedureHow to Reset the Time of Day in a Cluster

Sun Cluster uses the Network Time Protocol (NTP) to maintain time synchronization between cluster nodes. Adjustments in the cluster occur automatically as needed when nodes synchronize their time. See the Sun Cluster Concepts Guide for Solaris OS and the Network Time Protocol User's Guide for more information.


Caution – Caution –

When using NTP, do not attempt to adjust the cluster time while the cluster is up and running. This includes using the date(1), rdate(1M), xntpd(1M), or svcadm(1M) commands interactively or within cron(1M) scripts.


Steps
  1. Become superuser on any node in the cluster.

  2. Shut down the cluster.


    # scshutdown -g0 -y
    
  3. Verify that the node is showing the ok prompt or the Select (b)oot or (i)nterpreter prompt on the Current Boot Parameters screen.

  4. Boot the node in non-cluster mode by using the boot(1M) or the b command with the -x option.

    • SPARC:


      ok boot -x
      
    • x86:


                            <<< Current Boot Parameters >>>
      Boot path: /pci@0,0/pci8086,2545@3/pci8086,1460@1d/pci8086,341a@7,1/
      sd@0,0:a
      Boot args:
      
      Type    b [file-name] [boot-flags] <ENTER>  to boot with options
      or      i <ENTER>                           to enter boot interpreter
      or      <ENTER>                             to boot with defaults
      
                        <<< timeout in 5 seconds >>>
      Select (b)oot or (i)nterpreter: b -x
      
  5. On a single node, set the time of day by running the date command.


    # date HHMM.SS
    
  6. On the other machines, synchronize the time to that node by running the rdate(1M) command.


    # rdate hostname
    
  7. Boot each node to restart the cluster.


    # reboot
    
  8. Verify that the change took place on all cluster nodes.

    On each node, run the date command.


    # date
    

ProcedureSPARC: How to Enter the OpenBoot PROM (OBP) on a Node

Use this procedure if you need to configure or change OpenBoot PROM settings.

Steps
  1. Connect to the terminal concentrator port.


    # telnet tc_name tc_port_number
    
    tc_name

    Specifies the name of the terminal concentrator.

    tc_port_number

    Specifies the port number on the terminal concentrator. Port numbers are configuration dependent. Typically, ports 2 and 3 (5002 and 5003) are used for the first cluster installed at a site.

  2. Shut down the cluster node gracefully by using the scswitch(1M) command to evacuate any resource or disk device groups and then shutdown.


    # scswitch -S -h node[,...]
    # shutdown -g0 -y -i0
    

    Caution – Caution –

    Do not use send brk on a cluster console to shut down a cluster node.


  3. Execute the OBP commands.

ProcedureHow to Change the Private Hostname

Use this procedure to change the private hostname of a cluster node after installation has been completed.

Default private hostnames are assigned during initial cluster installation. The default private hostname takes the form clusternode< nodeid>-priv, for example: clusternode3-priv . You should only change a private hostname if the name is already in use in the domain.


Caution – Caution –

Do not attempt to assign IP addresses to new private hostnames. The clustering software assigns them.


Steps
  1. Disable, on all nodes in the cluster, any Data Service resources or other applications that might cache private hostnames.


    # scswitch -n -j resource1, resource2
    

    Include the following in the applications you disable.

    • HA-DNS and HA-NFS services, if configured.

    • Any application which has been custom configured to use the private hostname.

    • Any application which is being used by clients over the private interconnect.

    See the scswitch(1M) man page and the Sun Cluster Data Services Planning and Administration Guide for Solaris OS for information about using the scswitch command.

  2. Bring down the Network Time Protocol (NTP) daemon on each node of the cluster.

    • If you are using Solaris 8 or Solaris 9, use the xntpd command to bring down the Network Time Protocol (NTP) daemon. See the xntpd(1M) man page for more information about the NTP daemon.


      # /etc/init.d/xntpd.cluster stop
      
    • If you are using Solaris 10, use the svcadm command to bring down the Network Time Protocol (NTP) daemon. See the svcadm(1M) man page for more information about the NTP daemon.


      # svcadm disable ntp
      
  3. Run the scsetup(1M) utility to change the private hostname of the appropriate node.

    It is only necessary to do this from one of the nodes in the cluster.


    Note –

    When selecting a new private hostname, be sure the name is unique to the cluster node.


  4. Select 5, Private Hostnames, from the Main Menu.

  5. Select 1, Change a Private Hostname, from the Private Hostnames Menu.

    Answer the questions when prompted. You will be asked the name of the node whose private hostname is being changed (clusternode< nodeid>-priv), and the new private hostname.

  6. Flush the name service cache.

    Do this on each node in the cluster. This prevents the cluster applications and data services from trying to access the old private hostname.


    # nscd -i hosts
    
  7. Edit the ntp.conf.cluster file on each node to change the private hostname to the new one.

    Use whatever editing tool you prefer.

    If this is done at install time, also remember to remove names for nodes which are configured; the default template comes pre-configured with sixteen nodes. Typically, the ntp.conf.cluster file will be identical on each cluster node.

  8. Verify that you can successfully ping the new private hostname from all cluster nodes.

  9. Restart the NTP daemon.

    Do this on each node of the cluster.

    • If you are using Solaris 8 or Solaris 9, use the xntpd command to restart the NTP daemon.


      # /etc/init.d/xntpd.cluster start
      
    • If you are using Solaris 10, use the svcadm command to restart the NTP daemon.


      # svcadm enable ntp
      
  10. Enable all Data Service resources and other applications that were disabled in Step 1.


    # scswitch -e -j resource1, resource2
    

    See the scswitch man page and the Sun Cluster Data Services Planning and Administration Guide for Solaris OS for information about using the scswitch command.


Example 7–8 Changing the Private Hostname

The following example changes the private hostname from clusternode2-priv to clusternode4-priv, on node phys-schost-2 .


[Disable all applications and data services as necessary.
]
phys-schost-1# /etc/init.d/xntpd stop
phys-schost-1# scconf -p | grep node
 ...
 Cluster nodes:                  phys-schost-1 phys-schost-2 phys-
 schost-3
 Cluster node name:                                 phys-schost-1
  Node private hostname:                           clusternode1-priv
 Cluster node name:                                 phys-schost-2
  Node private hostname:                           clusternode2-priv
 Cluster node name:                                 phys-schost-3
  Node private hostname:                           clusternode3-priv
 ...
phys-schost-1# scsetup
phys-schost-1# nscd -i hosts
phys-schost-1# vi /etc/inet/ntp.conf
 ...
 peer clusternode1-priv
 peer clusternode4-priv
 peer clusternode3-priv
phys-schost-1# ping clusternode4-priv
phys-schost-1# /etc/init.d/xntpd start
[Enable all applications and data services disabled at the beginning of the procedure.
]

ProcedureHow to Put a Node Into Maintenance State

Put a cluster node into maintenance state when taking the node out of service for an extended period of time. This way, the node does not contribute to the quorum count while it is being serviced. To put a cluster node into maintenance state, the node must be brought down using scswitch(1M) and shutdown(1M).


Note –

Use the Solaris shutdown command to shut down a single node. The scshutdown command should be used only when shutting down an entire cluster.


When a cluster node is brought down and put into maintenance state, all quorum devices that are configured with ports to the node have their quorum vote counts decremented by one. The node and quorum device vote counts are incremented by one when the node is taken out of maintenance mode and brought back online.

You need to use the scconf(1M) command to put a cluster node into maintenance state. The scsetup(1M) utility does not include the functionality for putting a quorum device into maintenance state.

Steps
  1. Become superuser on the node to be put into maintenance state.

  2. Evacuate any resource groups and disk device groups from the node.


    # scswitch -S -h node[,...]
    
    -S

    Evacuates all device services and resource groups from the specified node.

    -h node[,...]

    Specifies the node from which you are switching resource groups and devices groups.

  3. Shut down the node that you evacuated.


    # shutdown -g0 -y -i0
    
  4. Become superuser on another node in the cluster and put the node that you shut down in Step 3 into maintenance state.


    # scconf -c -q node=node,maintstate
    
    -c

    Specifies the change form of the scconf command.

    -q

    Manages the quorum options.

    node=node

    Specifies the node name or node ID of the node to change.

    maintstate

    Puts the node into maintenance state.

  5. Verify that the cluster node is now in maintenance state.


    # scstat -q
    

    The node you put into maintenance state should have a Status of offline and 0 (zero) for Present and Possible quorum votes.


Example 7–9 Putting a Cluster Node Into Maintenance State

The following example moves a cluster node into maintenance state and verifies the results. The scstat -q output shows the Node votes for phys-schost-1 to be 0 (zero) and the status to be Offline. The Quorum Summary should also show reduced vote counts. Depending on your configuration, the Quorum Votes by Device output might indicate that some quorum disk devices are offline as well.


[On the node to be put into maintenance state:
]
phys-schost-1# scswitch -S -h phys-schost-1
phys-schost-1# shutdown -g0 -y -i0

[On another node in the cluster:]
phys-schost-2# scconf -c -q node=phys-schost-1,maintstate
phys-schost-2# scstat -q

-- Quorum Summary --
  Quorum votes possible:      3
  Quorum votes needed:        2
  Quorum votes present:       3

-- Quorum Votes by Node --
                    Node Name           Present Possible Status
                    ---------           ------- -------- ------
  Node votes:       phys-schost-1       0        0       Offline
  Node votes:       phys-schost-2       1        1       Online
  Node votes:       phys-schost-3       1        1       Online

-- Quorum Votes by Device --
                    Device Name         Present Possible Status
                    -----------         ------- -------- ------
  Device votes:     /dev/did/rdsk/d3s2  0        0       Offline
  Device votes:     /dev/did/rdsk/d17s2 0        0       Offline
  Device votes:     /dev/did/rdsk/d31s2 1        1       Online

See Also

To bring a node back online, see How to Bring a Node Out of Maintenance State.

ProcedureHow to Bring a Node Out of Maintenance State

Use the following procedure to bring a node back online and reset the quorum vote count to the default. For cluster nodes, the default quorum count is one. For quorum devices, the default quorum count is N-1, where N is the number of nodes with non-zero vote counts that have ports to the quorum device.

When a node has been put into maintenance state, the node's quorum vote count is decremented by one. All quorum devices that are configured with ports to the node will also have their quorum vote counts decremented. When the quorum vote count is reset and a node is brought back out of maintenance state, both the node's quorum vote count and the quorum device vote count are incremented by one.

Run this procedure any time a node has been put into maintenance state and you are bringing it out of maintenance state.


Caution – Caution –

If you do not specify either the globaldev or node options, the quorum count is reset for the entire cluster.


Steps
  1. Become superuser on any node of the cluster, other than the one in maintenance state.

  2. Are you bringing a node out of maintenance state in a two-node cluster?

  3. If using quorum, reset the cluster quorum count from a node other than the one in maintenance state.

    You must reset the quorum count from a node other than the node in maintenance state before rebooting the node, or it might hang waiting for quorum.


    # scconf -c -q node=node,reset
    
    -c

    Specifies the change form of the scconf command.

    -q

    Manages the quorum options.

    node=node

    Specifies the name of the node to be reset, for example, phys-schost-1.

    reset

    The change flag that resets quorum.

  4. Boot the node that you want to bring out of maintenance state.

  5. Verify the quorum vote count.


    # scstat -q
    

    The node you brought out of maintenance state should have a status of online and show the appropriate vote count for Present and Possible quorum votes.


Example 7–10 Bringing a Cluster Node Out of Maintenance State and Resetting the Quorum Vote Count

The following example resets the quorum count for a cluster node and its quorum devices to their defaults and verifies the result. The scstat -q output shows the Node votes for phys-schost-1 to be 1 and the status to be online. The Quorum Summary should also show an increase in vote counts.


phys-schost-2# scconf -c -q node=phys-schost-1,reset

On phys-schost-1:


phys-schost-1# scstat -q

-- Quorum Summary --

  Quorum votes possible:      6
  Quorum votes needed:        4
  Quorum votes present:       6

-- Quorum Votes by Node --

                    Node Name           Present Possible Status
                    ---------           ------- -------- ------
  Node votes:       phys-schost-1       1        1       Online

  Node votes:       phys-schost-2       1        1       Online
  Node votes:       phys-schost-3       1        1       Online

-- Quorum Votes by Device --

                    Device Name         Present Possible Status
                    -----------         ------- -------- ------
  Device votes:     /dev/did/rdsk/d3s2  1        1       Online
  Device votes:     /dev/did/rdsk/d17s2 1        1       Online
  Device votes:     /dev/did/rdsk/d31s2 1        1       Online