Sun Cluster System Administration Guide for Solaris OS

Chapter 8 Adding and Removing a Node

This chapter provides instructions on how to add a node to a cluster and how to remove a node:

For information about cluster maintenance tasks, see Chapter 9, Administering the Cluster.

Adding a Node to a Cluster

This section describes how to add a node to a global cluster or a zone cluster. You can create a new zone-cluster node on a node of the global cluster that hosts the zone cluster, as long as that global-cluster node does not already host a node of that particular zone cluster. You cannot convert an existing non-voting node on a global cluster into a zone-cluster node.

In this chapter, phys-schost# reflects a global-cluster prompt. The clzonecluster interactive shell prompt is clzc:schost>.

The following table lists the tasks to perform to add a node to an existing cluster. Perform the tasks in the order shown.

Table 8–1 Task Map: Adding a Node to an Existing Global or Zone Cluster

Task 

Instructions 

Install the host adapter on the node and verify that the existing cluster interconnects can support the new node 

Sun Cluster 3.1 - 3.2 Hardware Administration Manual for Solaris OS

Add shared storage 

Sun Cluster 3.1 - 3.2 Hardware Administration Manual for Solaris OS

Prepare the cluster for additional nodes 

How to Prepare the Cluster for Additional Global-Cluster Nodes in Sun Cluster Software Installation Guide for Solaris OS

Add the node to the authorized node list by using clsetup

How to Add a Node to the Authorized Node List

Install and configure the software on the new cluster node 

Chapter 2, Installing Software on Global-Cluster Nodes, in Sun Cluster Software Installation Guide for Solaris OS

If the cluster is configured in a Sun Cluster Geographic Edition partnership, configure the new node as an active participant in the configuration 

How to Add a New Node to a Cluster in a Partnership in Sun Cluster Geographic Edition System Administration Guide

ProcedureHow to Add a Node to the Authorized Node List

Before adding a Solaris host or a virtual machine to an existing global cluster or a zone cluster, ensure that the node has all of the necessary hardware correctly installed and configured, including an operational physical connection to the private cluster interconnect.

For hardware installation information, refer to the Sun Cluster 3.1 - 3.2 Hardware Administration Manual for Solaris OS or the hardware documentation that shipped with your server.

This procedure enables a machine to install itself into a cluster by adding its node name to the list of authorized nodes for that cluster.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. On a current global-cluster member, become superuser on the current cluster member. Perform these steps from a node of a global cluster.

  2. Ensure that you have correctly completed all prerequisite hardware installation and configuration tasks that are listed in the task map for Table 8–1.

  3. Start the clsetup utility.


    phys-schost# clsetup
    

    The Main Menu is displayed.


    Note –

    To add a node to a zone cluster, use the clzonecluster utility. See Step 9 for instructions to manually add a zone to a zone cluster.


  4. Type the number that corresponds to the option for displaying the New Nodes Menu and press the Return key.

  5. Type the number that corresponds to the option to modify the authorized list and press the Return key. Specify the name of a machine that can add itself.

    Follow the prompts to add the node's name to the cluster. You are asked for the name of the node to be added.

  6. Verify that the task has been performed successfully.

    The clsetup utility prints a “Command completed successfully” message if it completes the task without error.

  7. To prevent any new machines from being added to the cluster, type the number that corresponds to option to instruct the cluster to ignore requests to add new machines. Press the Return key.

    Follow the clsetup prompts. This option tells the cluster to ignore all requests over the public network from any new machine that is trying to add itself to the cluster.

  8. Quit the clsetup utility.

  9. To manually add a node to a zone cluster, you must specify the Solaris host and the virtual node name. You must also specify a network resource to be used for public network communication on each node. In the following example, the zone name is sczone, and bge0 is the public network adapter on both machines.


    clzc:sczone>add node
    clzc:sczone:node>set physical-host=phys-cluster-1
    clzc:sczone:node>set hostname=hostname1
    clzc:sczone:node>add net
    clzc:sczone:node:net>set address=hostname1
    clzc:sczone:node:net>set physical=bge0
    clzc:sczone:node:net>end
    clzc:sczone:node>end
    clzc:sczone>add node
    clzc:sczone:node>set physical-host=phys-cluster-2
    clzc:sczone:node>set hostname=hostname2
    clzc:sczone:node>add net
    clzc:sczone:node:net>set address=hostname2
    clzc:sczone:node:net>set physical=bge0
    clzc:sczone:node:net>end
    clzc:sczone:node>end
    

    For detailed instructions on configuring the node, see Configuring a Zone Cluster in Sun Cluster Software Installation Guide for Solaris OS.

  10. Install and configure the software on the new cluster node.

    Use either scinstall or JumpStartTM software to complete the installation and configuration of the new node, as described in the Sun Cluster Software Installation Guide for Solaris OS.


Example 8–1 Adding a Global-Cluster Node to the Authorized Node List

The following example shows how to add a node named phys-schost-3 to the authorized node list in an existing cluster.


[Become superuser and execute the clsetup utility.]
phys-schost# clsetup
[Select New nodes>Specify the name of a machine which may add itself.]
[Answer the questions when prompted.]
[Verify that the scconf command completed successfully.]
 
claccess allow -h phys-schost-3
 
    Command completed successfully.
[Select Prevent any new machines from being added to the cluster.]
[Quit the clsetup New Nodes Menu and Main Menu.]
[Install the cluster software.]

See Also

clsetup(1CL)

For a complete list of tasks for adding a cluster node, see Table 8–1, “Task Map: Adding a Cluster Node.”

To add a node to an existing resource group, see the Sun Cluster Data Services Planning and Administration Guide for Solaris OS.

Creating a Non-Voting Node (Zone) in a Global Cluster

This section provides the following information and procedures to create a non-voting node, simply referred to as a zone, on a global-cluster node.

ProcedureHow to Create a Non-Voting Node in a Global Cluster

  1. Become superuser on the global-cluster node where you are creating the non-voting node.

    You must be working in the global zone.

  2. For the Solaris 10 OS, verify on each node that multiuser services for the Service Management Facility (SMF) are online.

    If services are not yet online for a node, wait until the state changes to online before you proceed to the next step.


    phys-schost# svcs multi-user-server node
    STATE          STIME    FMRI
    online         17:52:55 svc:/milestone/multi-user-server:default
  3. Configure, install, and boot the new zone.


    Note –

    You must set the autoboot property to true to support resource-group functionality in the non-voting node on the global cluster.


    Follow procedures in the Solaris documentation:

    1. Perform procedures in Chapter 18, Planning and Configuring Non-Global Zones (Tasks), in System Administration Guide: Solaris Containers-Resource Management and Solaris Zones.

    2. Perform procedures in Installing and Booting Zones in System Administration Guide: Solaris Containers-Resource Management and Solaris Zones.

    3. Perform procedures in How to Boot a Zone in System Administration Guide: Solaris Containers-Resource Management and Solaris Zones.

  4. Verify that the zone is in the ready state.


    phys-schost# zoneadm list -v
    ID  NAME     STATUS       PATH
     0  global   running      /
     1  my-zone  ready        /zone-path
    
  5. For a whole-root zone with the ip-type property set to exclusive: If the zone might host a logical-hostname resource, configure a file system resource that mounts the method directory from the global zone.


    phys-schost# zonecfg -z sczone
    zonecfg:sczone> add fs
    zonecfg:sczone:fs> set dir=/usr/cluster/lib/rgm
    zonecfg:sczone:fs> set special=/usr/cluster/lib/rgm
    zonecfg:sczone:fs> set type=lofs
    zonecfg:sczone:fs> end
    zonecfg:sczone> exit
    
  6. (Optional) For a shared-IP zone, assign a private IP address and a private hostname to the zone.

    The following command chooses and assigns an available IP address from the cluster's private IP-address range. The command also assigns the specified private hostname, or host alias, to the zone and maps it to the assigned private IP address.


    phys-schost# clnode set -p zprivatehostname=hostalias node:zone
    
    -p

    Specifies a property.

    zprivatehostname=hostalias

    Specifies the zone private hostname, or host alias.

    node

    The name of the node.

    zone

    The name of the global-cluster non-voting node.

  7. Perform the initial internal zone configuration.

    Follow the procedures in Performing the Initial Internal Zone Configuration in System Administration Guide: Solaris Containers-Resource Management and Solaris Zones. Choose either of the following methods:

    • Log in to the zone.

    • Use an /etc/sysidcfg file.

  8. In the non-voting node, modify the nsswitch.conf file.

    These changes enable the zone to resolve searches for cluster-specific hostnames and IP addresses.

    1. Log in to the zone.


      phys-schost# zlogin -c zonename
      
    2. Open the /etc/nsswitch.conf file for editing.


      sczone# vi /etc/nsswitch.conf
      
    3. Add the cluster switch to the beginning of the lookups for the hosts and netmasks entries, followed by the files switch.

      The modified entries should appear similar to the following:


      …
      hosts:      cluster files nis [NOTFOUND=return]
      …
      netmasks:   cluster files nis [NOTFOUND=return]
      …
    4. For all other entries, ensure that the files switch is the first switch that is listed in the entry.

    5. Exit the zone.

  9. If you created an exclusive-IP zone, configure IPMP groups in each /etc/hostname.interface file that is on the zone.

    You must configure an IPMP group for each public-network adapter that is used for data-service traffic in the zone. This information is not inherited from the global zone. See Public Networks in Sun Cluster Software Installation Guide for Solaris OS for more information about configuring IPMP groups in a cluster.

  10. Set up name-to-address mappings for all logical hostname resources that are used by the zone.

    1. Add name-to-address mappings to the /etc/inet/hosts file on the zone.

      This information is not inherited from the global zone.

    2. If you use a name server, add the name-to-address mappings.

Removing a Node From a Cluster

This section provides instructions on how to remove a node on a global cluster or a zone cluster. You can also remove a specific zone cluster from a global cluster. The following table lists the tasks to perform to remove a node from an existing cluster. Perform the tasks in the order shown.


Caution – Caution –

If you remove a node using only this procedure for a RAC configuration, the removal might cause the node to panic during a reboot. For instructions on how to remove a node from a RAC configuration, see How to Remove Sun Cluster Support for Oracle RAC From Selected Nodes in Sun Cluster Data Service for Oracle RAC Guide for Solaris OS. After you complete that process, follow the appropriate steps below.


Table 8–2 Task Map: Removing a Node

Task 

Instructions 

Move all resource groups and device groups off the node to be removed 

clnode evacuate node

Verify that the node can be removed by checking the allowed hosts 

If the node cannot be removed, give the node access to the cluster configuration 

claccess show node

claccess allow -h node-to-remove

Remove the node from all device groups 

How to Remove a Node From a Device Group (Solaris Volume Manager)

 

Remove all quorum devices connected to the node being removed 

This step is optional if you are removing a node from a two-node cluster.

How to Remove a Quorum Device

Note that although you must remove the quorum device before you remove the storage device in the next step, you can add the quorum device back immediately afterward. 

How to Remove the Last Quorum Device From a Cluster

Put the node being removed into noncluster mode 

How to Put a Node Into Maintenance State

Remove a node from a zone cluster 

How to Remove a Node From a Zone Cluster

Remove a node from the cluster software configuration 

How to Remove a Node From the Cluster Software Configuration

(Optional) Uninstall Sun Cluster software from a cluster node 

How to Uninstall Sun Cluster Software From a Cluster Node

Remove an entire zone cluster 

How to Remove a Zone Cluster

ProcedureHow to Remove a Node From a Zone Cluster

You can remove a node from a zone cluster by halting the node, uninstalling it, and removing the node from the configuration. If you decide later to add the node back into the zone cluster, follow the instructions in Table 8–1. Most of these steps are performed from the global-cluster node.

  1. Become superuser on a node of the global cluster.

  2. Shut down the zone-cluster node you want to remove by specifying the node and its zone cluster.


    phys-schost# clzonecluster halt -n node zoneclustername
    

    You can also use the clnode evacuate and shutdown commands within a zone cluster.

  3. Uninstall the zone-cluster node.


    phys-schost# clzonecluster uninstall -n node zoneclustername
    
  4. Remove the zone-cluster node from the configuration.

    Use the following commands:


    phys-schost# clzonecluster configure zoneclustername
    

    clzc:sczone> remove node physical-host=zoneclusternodename
    
  5. Verify that the node was removed from the zone cluster.


    phys-schost# clzonecluster status
    

ProcedureHow to Remove a Node From the Cluster Software Configuration

Perform this procedure to remove a node from the global cluster.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Ensure that you have removed the node from all resource groups, device groups, and quorum device configurations and put it into maintenance state before you continue with this procedure.

  2. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on the node that you want to remove. Perform all steps in this procedure from a node of the global cluster.

  3. Boot the global-cluster node that you want to remove into noncluster mode. For a zone-cluster node, follow the instructions in How to Remove a Node From a Zone Cluster before you perform this step.

    • On SPARC based systems, run the following command.


      ok boot -x
      
    • On x86 based systems, run the following commands.


      shutdown -g -y -i0
      
      Press any key to continue
    1. In the GRUB menu, use the arrow keys to select the appropriate Solaris entry and type e to edit its commands.

      The GRUB menu appears similar to the following:


      GNU GRUB version 0.95 (631K lower / 2095488K upper memory)
      +-------------------------------------------------------------------------+
      | Solaris 10 /sol_10_x86                                                  |
      | Solaris failsafe                                                        |
      |                                                                         |
      +-------------------------------------------------------------------------+
      Use the ^ and v keys to select which entry is highlighted.
      Press enter to boot the selected OS, 'e' to edit the
      commands before booting, or 'c' for a command-line.

      For more information about GRUB based booting, see Booting an x86 Based System by Using GRUB (Task Map) in System Administration Guide: Basic Administration.

    2. In the boot parameters screen, use the arrow keys to select the kernel entry and type e to edit the entry.

      The GRUB boot parameters screen appears similar to the following:


      GNU GRUB version 0.95 (615K lower / 2095552K upper memory)
      +----------------------------------------------------------------------+
      | root (hd0,0,a)                                                       |
      | kernel /platform/i86pc/multiboot                                     |
      | module /platform/i86pc/boot_archive                                  |
      +----------------------------------------------------------------------+
      Use the ^ and v keys to select which entry is highlighted.
      Press 'b' to boot, 'e' to edit the selected command in the
      boot sequence, 'c' for a command-line, 'o' to open a new line
      after ('O' for before) the selected line, 'd' to remove the
      selected line, or escape to go back to the main menu.
    3. Add -x to the command to specify system boot into noncluster mode.


      [ Minimal BASH-like line editing is supported. For the first word, TAB
      lists possible command completions. Anywhere else TAB lists the possible
      completions of a device/filename. ESC at any time exits. ]
      
      grub edit> kernel /platform/i86pc/multiboot -x
    4. Press the Enter key to accept the change and return to the boot parameters screen.

      The screen displays the edited command.


      GNU GRUB version 0.95 (615K lower / 2095552K upper memory)
      +----------------------------------------------------------------------+
      | root (hd0,0,a)                                                       |
      | kernel /platform/i86pc/multiboot -x                                  |
      | module /platform/i86pc/boot_archive                                  |
      +----------------------------------------------------------------------+
      Use the ^ and v keys to select which entry is highlighted.
      Press 'b' to boot, 'e' to edit the selected command in the
      boot sequence, 'c' for a command-line, 'o' to open a new line
      after ('O' for before) the selected line, 'd' to remove the
      selected line, or escape to go back to the main menu.-
    5. Type b to boot the node into noncluster mode.

      This change to the kernel boot parameter command does not persist over the system boot. The next time you reboot the node, it will boot into cluster mode. To boot into noncluster mode instead, perform these steps again to add the -x option to the kernel boot parameter command.


      Note –

      If the node to be removed is not available or can no longer be booted, run the following command on any active cluster node: clnode clear -F <node-to-be-removed>. Verify the node removal by running clnode status <nodename>.


  4. From the node you want to remove, delete the node from the cluster.


    phys-schost# clnode remove -F
    

    If the clnode remove command fails and a stale node reference exists, run clnode clear -F nodename on an active node.


    Note –

    If you are removing the last node in the cluster, the node must be in noncluster mode with no active nodes left in the cluster.


  5. From another cluster node, verify the node removal.


    phys-schost# clnode status nodename
    
  6. Complete the node removal.


Example 8–2 Removing a Node From the Cluster Software Configuration

This example shows how to remove a node (phys-schost-2) from a cluster. The clnode remove command is run in noncluster mode from the node you want to remove from the cluster (phys-schost-2).


[Remove the node from the cluster:]
phys-schost-2# clnode remove
phys-schost-1# clnode clear -F phys-schost-2
[Verify node removal:]
phys-schost-1# clnode status
-- Cluster Nodes --
                    Node name           Status
                    ---------           ------
  Cluster node:     phys-schost-1       Online

See Also

To uninstall Sun Cluster software from the removed node, see How to Uninstall Sun Cluster Software From a Cluster Node.

For hardware procedures, see the Sun Cluster 3.1 - 3.2 Hardware Administration Manual for Solaris OS.

For a comprehensive list of tasks for removing a cluster node, seeTable 8–2.

To add a node to an existing cluster, see How to Add a Node to the Authorized Node List.

ProcedureHow to Remove a Non-Voting Node (Zone) From a Global Cluster

  1. Become superuser on the global-cluster node where you created the non-voting node.

  2. Delete the non-voting node from the system.

    Follow the procedures in Deleting a Non-Global Zone From the System in System Administration Guide: Solaris Containers-Resource Management and Solaris Zones.

ProcedureHow to Remove Connectivity Between an Array and a Single Node, in a Cluster With Greater Than Two-Node Connectivity

Use this procedure to detach a storage array from a single cluster node, in a cluster that has three-node or four-node connectivity.

The phys-schost# prompt reflects a global-cluster prompt. Perform this procedure on a global cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix B, Sun Cluster Object-Oriented Commands.

  1. Back up all database tables, data services, and volumes that are associated with the storage array that you are removing.

  2. Determine the resource groups and device groups that are running on the node to be disconnected.


    phys-schost# clresourcegroup status
    phys-schost# cldevicegroup status
    
  3. If necessary, move all resource groups and device groups off the node to be disconnected.


    Caution (SPARC only) – Caution (SPARC only) –

    If your cluster is running Oracle RAC software, shut down the Oracle RAC database instance that is running on the node before you move the groups off the node. For instructions, see the Oracle Database Administration Guide.



    phys-schost# clnode evacuate node
    

    The clnode evacuate command switches over all device groups from the specified node to the next-preferred node. The command also switches all resource groups from voting or non-voting nodes on the specified node to the next-preferred voting or non-voting node.

  4. Put the device groups into maintenance state.

    For the procedure on acquiescing I/O activity to Veritas shared disk groups, see your VxVM documentation.

    For the procedure on putting a device group in maintenance state, see How to Put a Node Into Maintenance State.

  5. Remove the node from the device groups.

    • If you use VxVM or a raw disk, use the cldevicegroup(1CL) command to remove the device groups.

    • If you use Solstice DiskSuite, use the metaset command to remove the device groups.

  6. For each resource group that contains an HAStoragePlus resource, remove the node from the resource group's node list.


    phys-schost# clresourcegroup remove-node -z zone -n node + | resourcegroup
    
    node

    The name of the node.

    zone

    The name of the non-voting node that can master the resource group. Specify zone only if you specified a non-voting node when you created the resource group.

    See the Sun Cluster Data Services Planning and Administration Guide for Solaris OS for more information about changing a resource group's node list.


    Note –

    Resource type, resource group, and resource property names are case sensitive when clresourcegroup is executed.


  7. If the storage array that you are removing is the last storage array that is connected to the node, disconnect the fiber-optic cable between the node and the hub or switch that is connected to this storage array (otherwise, skip this step).

  8. If you are removing the host adapter from the node that you are disconnecting, and power off the node. If you are removing the host adapter from the node that you are disconnecting, skip to Step 11.

  9. Remove the host adapter from the node.

    For the procedure on removing host adapters, see the documentation for the node.

  10. Without booting the node, power on the node.

  11. If Oracle RAC software has been installed, remove the Oracle RAC software package from the node that you are disconnecting.


    phys-schost# pkgrm SUNWscucm 
    

    Caution (SPARC only) – Caution (SPARC only) –

    If you do not remove the Oracle RAC software from the node that you disconnected, the node panics when the node is reintroduced to the cluster and potentially causes a loss of data availability.


  12. Boot the node in cluster mode.

    • On SPARC based systems, run the following command.


      ok boot
      
    • On x86 based systems, run the following commands.

      When the GRUB menu is displayed, select the appropriate Solaris entry and press Enter. The GRUB menu appears similar to the following:


      GNU GRUB version 0.95 (631K lower / 2095488K upper memory)
      +-------------------------------------------------------------------------+
      | Solaris 10 /sol_10_x86                                                  |
      | Solaris failsafe                                                        |
      |                                                                         |
      +-------------------------------------------------------------------------+
      Use the ^ and v keys to select which entry is highlighted.
      Press enter to boot the selected OS, 'e' to edit the
      commands before booting, or 'c' for a command-line.
  13. On the node, update the device namespace by updating the /devices and /dev entries.


    phys-schost# devfsadm -C 
     cldevice refresh
    
  14. Bring the device groups back online.

    For procedures about bringing a Veritas shared disk group online, see your Veritas Volume Manager documentation.

    For information about bringing a device group online, see How to Bring a Node Out of Maintenance State.

ProcedureHow to Correct Error Messages

To correct any error messages that occurred while attempting to perform any of the cluster node removal procedures, perform the following procedure.

  1. Attempt to rejoin the node to the global cluster. Perform this procedure only on a global cluster.


    phys-schost# boot
    
  2. Did the node successfully rejoin the cluster?

    • If no, proceed to Step b.

    • If yes, perform the following steps to remove the node from device groups.

    1. If the node successfully rejoins the cluster, remove the node from the remaining device group or groups.

      Follow procedures in How to Remove a Node From All Device Groups.

    2. After you remove the node from all device groups, return to How to Uninstall Sun Cluster Software From a Cluster Node and repeat the procedure.

  3. If the node could not rejoin the cluster, rename the node's /etc/cluster/ccr file to any other name you choose, for example, ccr.old.


    # mv /etc/cluster/ccr /etc/cluster/ccr.old
    
  4. Return to How to Uninstall Sun Cluster Software From a Cluster Node and repeat the procedure.