test

Sun Cluster System Administration Guide for Solaris OS

Removing a Cluster Node

This section provides instructions on how to remove a cluster node. The following table lists the tasks to perform to remove a node from an existing cluster. To complete the procedure correctly, the tasks must be performed in the order shown.

Caution – Caution –

Do not use this procedure if your cluster is running an OPS configuration. At this time, removing a node in an OPS configuration might cause nodes to panic at reboot.

Table 8–3 Task Map: Removing a Cluster Node

Task 

Instructions 

Move all resource groups and device groups off of the node to be removed. 

   - Use clnode(1CL) evacuate

# clnode evacuate node

Remove the node from all resource groups. 

   - Use clresourcegroup(1CL) remove-node

Sun Cluster Data Services Planning and Administration Guide for Solaris OS

Remove node from all device groups 

   - Use cldevicegroup(1CL) remove node, metaset(1M), and clsetup(1CL)

How to Remove a Node From a Device Group (Solaris Volume Manager)

SPARC: How to Remove a Node From a Device Group (VERITAS Volume Manager)

How to Remove a Node From a Raw-Disk Device Group

Caution: If the number of desired secondaries is configured as 2 or more, it must be decreased to 1.

Remove all fully connected quorum devices. 

   - Use clsetup.

Caution: Do not remove the quorum device if you are removing a node from a two-node cluster.

How to Remove a Quorum Device

Note that although you must remove the quorum device before you remove the storage device in the next step, you can add the quorum device back immediately afterward. 

Remove all fully connected storage devices from the node.  

  - Use clresourcegroupclresourcegroup(1CL) clresourcegroup(1CL) devfsadm(1M), cldevice(1CL) refresh.

Caution: Do not remove the quorum device if you are removing a node from a two-node cluster. How to Remove Connectivity Between an Array and a Single Node, in a Cluster With Greater Than Two-Node Connectivity

Add back the quorum devices (to only the nodes that are intended to remain in the cluster). 

  - Use clsetup

Adding a Quorum Device

Put the node being removed into maintenance state. 

   - Use clnode(1CL) evacuate, shutdown, and clquorum(1CL) disable

How to Put a Node Into Maintenance State

Remove all logical transport connections (transport cables and adapters) from the node being removed. 

   - Use clsetup.

How to Remove Cluster Transport Cables, Transport Adapters, and Transport Switches

Remove all quorum devices connected to the node being removed. 

   - Use cluster set, clquorum remove.

How to Remove the Last Quorum Device From a Cluster

Remove node from the cluster software configuration. 

   - Use clnode remove.

How to Remove a Node From the Cluster Software Configuration

(Optional) Uninstall Sun Cluster software from a cluster node 

How to Uninstall Sun Cluster Software From a Cluster Node

ProcedureHow to Remove a Node From the Cluster Software Configuration

Perform this procedure to remove a node from the cluster.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix A, Sun Cluster Object-Oriented Commands.

  1. Ensure that you have correctly completed all prerequisite tasks listed in Removing a Cluster Node.

    Note –

    Ensure that you have removed the node from all resource groups, device groups, and quorum device configurations and put it into maintenance state before you continue with this procedure.

  2. Become superuser or assume a role that provides solaris.cluster.modify RBAC authorization on the node you want to remove.

  3. Boot the node you want to remove into noncluster mode.

    • On SPARC based systems, perform the following command:


      ok boot -x

    • On x86 based system, perform the following commands:


      phys-schost# shutdown -g -y -i0

Press any key to continue

    1. In the GRUB menu, use the arrow keys to select the appropriate Solaris entry and type e to edit its commands.

      The GRUB menu appears similar to the following:


      GNU GRUB version 0.95 (631K lower / 2095488K upper memory)
+-------------------------------------------------------------------------+
| Solaris 10 /sol_10_x86                                                  |
| Solaris failsafe                                                        |
|                                                                         |
+-------------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, or 'c' for a command-line.

      For more information about GRUB based booting, see Chapter 11, GRUB Based Booting (Tasks), in System Administration Guide: Basic Administration.

    2. In the boot parameters screen, use the arrow keys to select the kernel entry and type e to edit the entry.

      The GRUB boot parameters screen appears similar to the following: 


      GNU GRUB version 0.95 (615K lower / 2095552K upper memory)
+----------------------------------------------------------------------+
| root (hd0,0,a)                                                       |
| kernel /platform/i86pc/multiboot                                     |
| module /platform/i86pc/boot_archive                                  |
+----------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted.
Press 'b' to boot, 'e' to edit the selected command in the
boot sequence, 'c' for a command-line, 'o' to open a new line
after ('O' for before) the selected line, 'd' to remove the
selected line, or escape to go back to the main menu.

    3. Add -x to the command to specify that the system boot into noncluster mode.


      [ Minimal BASH-like line editing is supported. For the first word, TAB
lists possible command completions. Anywhere else TAB lists the possible
completions of a device/filename. ESC at any time exits. ]

grub edit> kernel /platform/i86pc/multiboot -x

    4. Press the Enter key to accept the change and return to the boot parameters screen.

      The screen displays the edited command. 


      GNU GRUB version 0.95 (615K lower / 2095552K upper memory)
+----------------------------------------------------------------------+
| root (hd0,0,a)                                                       |
| kernel /platform/i86pc/multiboot -x                                  |
| module /platform/i86pc/boot_archive                                  |
+----------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted.
Press 'b' to boot, 'e' to edit the selected command in the
boot sequence, 'c' for a command-line, 'o' to open a new line
after ('O' for before) the selected line, 'd' to remove the
selected line, or escape to go back to the main menu.-

    5. Type b to boot the node into noncluster mode.

      Note –

      This change to the kernel boot parameter command does not persist over the system boot. The next time you reboot the node, it will boot into cluster mode. To boot into noncluster mode instead, perform these steps again to add the -x option to the kernel boot parameter command.

  4. Remove the node from the cluster.


    # clnode remove nodename

  5. From another cluster node, verify the node removal.


    # clnode status nodename

  6. If you intend to uninstall the Sun Cluster software from the removed node, proceed to How to Uninstall Sun Cluster Software From a Cluster Node. If you do not intend to uninstall the Sun Cluster software from the remove node, you can physically remove the node from the cluster by removing the hardware connections as described in Sun Cluster 3.1 - 3.2 Hardware Administration Manual for Solaris OS.

Example 8–12 Removing a Node From the Cluster Software Configuration

This example shows how to remove a node (phys-schost-2) from a cluster. The clnode remove command is run from the node you want to remove from the cluster ( phys-schost-1).


[Remove the node from the cluster:]
phys-schost-1# clnode remove phys-schost-2
[Verify node removal:]
phys-schost-1# clnode status phys-schost-2
-- Cluster Nodes --
                    Node name           Status
                    ---------           ------
  Cluster node:     phys-schost-1       Online
See Also

To uninstall Sun Cluster software from the removed node, see How to Uninstall Sun Cluster Software From a Cluster Node.

For hardware procedures, see the Sun Cluster 3.1 - 3.2 Hardware Administration Manual for Solaris OS.

For an overall list of tasks for removing a cluster node, see Table 8–3.

To add a node to an existing cluster, see How to Add a Node to the Authorized Node List.

ProcedureHow to Remove Connectivity Between an Array and a Single Node, in a Cluster With Greater Than Two-Node Connectivity

Use this procedure to detach a storage array from a single cluster node, in a cluster that has three-node or four-node connectivity.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix A, Sun Cluster Object-Oriented Commands.

  1. Back up all database tables, data services, and volumes that are associated with the storage array that you are removing.

  2. Determine the resource groups and device groups that are running on the node to be disconnected.


    # clresourcegroup status
# cldevicegroup status

  3. If necessary, move all resource groups and device groups off the node to be disconnected.

    Caution (SPARC only) – Caution (SPARC only) –

    If your cluster is running Oracle Parallel Server/Oracle RAC software, shut down the Oracle Parallel Server/Oracle RAC database instance that is running on the node before you move the groups off the node. For instructions see the Oracle Database Administration Guide.


    # clnode evacuate node

    The clnode evacuate command switches over all device groups from the specified node to the next preferred node. The command also switches all resource groups from global or non-global zones on the specified node to the next-preferred global or non-global zones on other nodes.

  4. Put the device groups into maintenance state.

    For the procedure on acquiescing I/O activity to Veritas shared disk groups, see your VxVM documentation.

    For the procedure on putting a device group in maintenance state, see the How to Put a Node Into Maintenance State.

  5. Remove the node from the device groups.

    • If you use VxVM or raw disk, use the cldevicegroup(1CL)command to remove the device groups.

    • If you use Solstice DiskSuite, use the metaset command to remove the device groups.

  6. For each resource group that contains an HAStoragePlus resource, remove the node from the resource group's node list. 


    # clresourcegroup remove-node -z zone -n node + | resourcegroup
    node

    The name of the node.

    zone

    The name of the non-global zone on node that can master the resource group. Specify zone only if you specified a non-global zone when you created the resource group.

    See the Sun Cluster Data Services Planning and Administration Guide for Solaris OS for more information about changing a resource group's nodelist.

    Note –

    Resource type, resource group, and resource property names are case sensitive when clresourcegroup is executed.

  7. If the storage array you are removing is the last storage array that is connected to the node, disconnect the fiber-optic cable between the node and the hub or switch that is connected to this storage array (otherwise, skip this step).

  8. If you are removing the host adapter from the node that you are disconnecting, shut down and power off the node. If you are removing the host adapter from the node that you are disconnecting, skip to Step 11.

  9. Remove the host adapter from the node.

    For the procedure on removing host adapters, see the documentation that shipped with your node.

  10. Without booting the node, power on the node.

  11. SPARC: If Oracle Parallel Server/Oracle RAC software has been installed, remove the Oracle Parallel Server/Oracle RAC software package from the node that you are disconnecting.


    # pkgrm SUNWscucm
    Caution (SPARC only) – Caution (SPARC only) –

    If you do not remove the Oracle Parallel Server/Oracle RAC software from the node that you disconnected, the node panics when the node is reintroduced to the cluster and potentially causes a loss of data availability.

  12. Boot the node in cluster mode.

    • On SPARC based systems, perform the following command:


      ok boot

    • On x86 based systems, do the following:

      When the GRUB menu is displayed, select the appropriate Solaris entry and press Enter. The GRUB menu appears similar to the following:


      GNU GRUB version 0.95 (631K lower / 2095488K upper memory)
+-------------------------------------------------------------------------+
| Solaris 10 /sol_10_x86                                                  |
| Solaris failsafe                                                        |
|                                                                         |
+-------------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, or 'c' for a command-line.

  13. On the node, update the device namespace by updating the /devices and /dev entries.


    # devfsadm -C 
# cldevice refresh

  14. Bring the device groups back online.

    For procedures about bringing a VERITAS shared disk group online, see your VERITAS Volume Manager documentation.

    For information on bringing a device group online, see How to Bring a Node Out of Maintenance State.

ProcedureHow to Uninstall Sun Cluster Software From a Cluster Node

Perform this procedure to uninstall Sun Cluster software from a cluster node before you disconnect it from a fully established cluster configuration. You can use this procedure to uninstall software from the last remaining node of a cluster.

Note –

To uninstall Sun Cluster software from a node that has not yet joined the cluster or is still in installation mode, do not perform this procedure. Instead, go to “How to Uninstall Sun Cluster Software to Correct Installation Problems” in the Sun Cluster Software Installation Guide for Solaris OS.

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the long and short forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix A, Sun Cluster Object-Oriented Commands.

  1. Ensure that you have correctly completed all prerequisite tasks in the task map for removing a cluster node.

    See Table 8–3.

    Note –

    Ensure that you have removed the node from the cluster configuration using clnode remove before you continue with this procedure.

  2. Become superuser on an active cluster member other than the node you are uninstalling.

  3. From the active cluster member, add the node that you intend to uninstall to the cluster's node authentication list.


    # claccess allow -h hostname
    -h

    Specifies the name of the node to be added to the node's authentication list.

    Alternately, you can use the clsetup(1CL) utility. See How to Add a Node to the Authorized Node List for procedures.

  4. Become superuser on the node to uninstall.

  5. Reboot the node into noncluster mode.

    • SPARC: Perform the following command:


      # shutdown -g0 -y -i0ok boot -x

    • x86: Perform the following:


      # shutdown -g0 -y -i0
...
                      <<< Current Boot Parameters >>>
Boot path: /pci@0,0/pci8086,2545@3/pci8086,1460@1d/pci8086,341a@7,1/
sd@0,0:a
Boot args:

Type    b [file-name] [boot-flags] <ENTER>  to boot with options
or      i <ENTER>                           to enter boot interpreter
or      <ENTER>                             to boot with defaults

                  <<< timeout in 5 seconds >>>
Select (b)oot or (i)nterpreter: b -x

  6. In the /etc/vfstab file, remove all globally mounted file system entries except the /global/.devices global mounts.

  7. If you intend to reinstall Sun Cluster software on this node, remove the Sun Cluster entry from the Sun Java Enterprise System (Java ES) product registry.

    If the Java ES product registry contains a record that Sun Cluster software was installed, the Java ES installer shows the Sun Cluster component grayed out and does not permit reinstallation.

    1. Start the Java ES uninstaller.

      Run the following command, where ver is the version of the Java ES distribution from which you installed Sun Cluster software.


      # /var/sadm/prod/SUNWentsysver/uninstall

    2. Follow the prompts to select Sun Cluster to uninstall.

      For more information about using the uninstall command, see Chapter 8, Uninstalling, in Sun Java Enterprise System 5 Installation Guide for UNIX in Sun Java Enterprise System 5 Installation Guide for UNIX.

  8. If you do not intend to reinstall the Sun Cluster software on this cluster, disconnect the transport cables and the transport switch, if any, from the other cluster devices.

    1. If the uninstalled node is connected to a storage device that uses a parallel SCSI interface, install a SCSI terminator to the open SCSI connector of the storage device after you disconnect the transport cables.

      If the uninstalled node is connected to a storage device that uses Fibre Channel interfaces, no termination is necessary.

    2. Follow the documentation that shipped with your host adapter and server for disconnection procedures.

ProcedureHow to Correct Error Messages

To correct any error messages that occurred while attempting to perform any of the cluster node removal procedures , perform the following procedure.

  1. Attempt to rejoin the node to the cluster.


    # boot

  2. Did the node successfully rejoin the cluster?

    • If no, proceed to Step 3.

    • If yes, perform the following steps to remove the node from device groups.

    1. If the node successfully rejoins the cluster, remove the node from the remaining device group or groups.

      Follow procedures in How to Remove a Node From All Device Groups.

    2. After you remove the node from all device groups, return to How to Uninstall Sun Cluster Software From a Cluster Node and repeat the procedure.

  3. If the node could not rejoin the cluster, rename the node's /etc/cluster/ccr file to any other name you choose, for example, ccr.old.


    # mv /etc/cluster/ccr /etc/cluster/ccr.old

  4. Return to How to Uninstall Sun Cluster Software From a Cluster Node and repeat the procedure.

Troubleshooting a Node Uninstallation

This section describes error messages you might receive when you run the scinstall -r command and the corrective actions to take.

Unremoved Cluster File-System Entries

The following error messages indicate that the node you removed still has cluster file systems referenced in its vfstab file.


Verifying that no unexpected global mounts remain in /etc/vfstab ... failed
scinstall:  global-mount1 is still configured as a global mount.
scinstall:  global-mount1 is still configured as a global mount.
scinstall:  /global/dg1 is still configured as a global mount.
 
scinstall:  It is not safe to uninstall with these outstanding errors.
scinstall:  Refer to the documentation for complete uninstall instructions.
scinstall:  Uninstall failed.

To correct this error, return to How to Uninstall Sun Cluster Software From a Cluster Node and repeat the procedure. Ensure that you successfully complete Step 6 in the procedure before you rerun the scinstall -r command.

Unremoved Listing in Device Groups

The following error messages indicate that the node you removed is still listed with a device group.


Verifying that no device services still reference this node ... failed
scinstall:  This node is still configured to host device service "
service".
scinstall:  This node is still configured to host device service "
service2".
scinstall:  This node is still configured to host device service "
service3".
scinstall:  This node is still configured to host device service "
dg1".
 
scinstall:  It is not safe to uninstall with these outstanding errors.          
scinstall:  Refer to the documentation for complete uninstall instructions.
scinstall:  Uninstall failed.