Chapter 4 Maintaining Cluster Interconnect Hardware

This chapter describes the procedures to maintain cluster interconnect hardware. The procedures in this chapter apply to all interconnects that Sun Cluster software supports:

• Ethernet

• SPARC: Peripheral Component Interconnect-Scalable Coherent Interface (PCI-SCI)

• InfiniBand

This chapter contains the following procedures:

• How to Add an Interconnect Component

• How to Replace an Interconnect Component

• How to Remove an Interconnect Component

• How to Upgrade Transport Adapter Firmware

For more information, see the following documentation:

• For conceptual information about cluster interconnects, see Cluster Interconnect in Sun Cluster Concepts Guide for Solaris OS.

• For information about administering cluster interconnects, see Administering the Cluster Interconnects in Sun Cluster System Administration Guide for Solaris OS.

Maintaining Interconnect Hardware in a Running Cluster

The following table lists procedures about maintaining cluster interconnect hardware.

Table 4–1 Task Map: Maintaining Cluster Interconnect Hardware

Task	Instructions
Add an interconnect component.	How to Add an Interconnect Component
Replace an interconnect component.	How to Replace an Interconnect Component
Remove an interconnect component.	How to Remove an Interconnect Component
Upgrade transport adapter firmware	How to Upgrade Transport Adapter Firmware

Interconnect components include the following components:

• Transport adapter

• Transport cable

• Transport junction (switch)

How to Add an Interconnect Component

This procedure defines interconnect component as any one of the following components:

• Transport adapter

• Transport cable

• Transport junction (switch)

This section contains the procedure for adding interconnect components to nodes in a running cluster.

Before You Begin

This procedure relies on the following prerequisites and assumptions:

• Your cluster is operational and all nodes are powered on.

• If virtual local area networks (VLANs) are configured, more than one cluster might be impacted by removing a transport junction. Ensure that all clusters are prepared for the removal of a transport junction. Also, record the configuration information of the transport junction you plan to replace and configure the new transport junction accordingly.

  For more information about how to configure VLANs, see Configuring VLANs as Private Interconnect Networks.

1. Determine if you need to shut down and power off the node that is to be connected to the interconnect component you are adding.

   • If you are adding a transport junction, you do not need to shut down and power off the node. Proceed to Step 2.

   • If you are adding a transport cable, you do not need to shut down and power off the node. Proceed to Step 2.

   • If your node has Dynamic Reconfiguration (DR) enabled and you are replacing a transport adapter, you do not need to shut down and power off the node. Proceed to Step 2.

   • If your node does not have DR enabled and you are adding a transport adapter, shut down and power off the node with the transport adapter you are adding.

     For the full procedure about shutting down a node, see Chapter 3, Shutting Down and Booting a Cluster, in Sun Cluster System Administration Guide for Solaris OS.

2. Install the interconnect component.

   • If you are using an Ethernet or InfiniBand interconnect, see How to Install Ethernet or InfiniBand Transport Cables and Transport Junctions for cabling diagrams and considerations.

   • If you are using a PCI-SCI interconnect, see SPARC: How to Install PCI-SCI Transport Cables and Transport Junctions for cabling diagrams and considerations.

   • For the procedure about installing transport adapters or setting transport adapter DIP switches, see the documentation that shipped with your host adapter and node hardware.

   • If your interconnect uses jumbo frames, review the requirements in Requirements When Using Jumbo Frames and see the Sun GigaSwift documentation for instructions.

3. If you shut down the node in Step 1, perform a reconfiguration boot to update the new Solaris device files and links. Otherwise, skip this step.

See Also

• To increase Oracle Real Application Clusters performance, set the max-vc-number parameter. See SPARC: Installing PCI-SCI Cluster Interconnect Hardware for more information.

• To reconfigure Sun Cluster software with the new interconnect component, see Chapter 7, Administering Cluster Interconnects and Public Networks, in Sun Cluster System Administration Guide for Solaris OS.

How to Replace an Interconnect Component

This procedure defines interconnect component as any one of the following components:

• Transport adapter

• Transport cable

• Transport junction (switch)

---

Caution –

You must maintain at least one cluster interconnect between the nodes of a cluster. The cluster does not function without a working cluster interconnect. You can check the status of the interconnect with one of the following commands:

• The Sun Cluster 3.2 command clinterconnect status.

• The Sun Cluster 3.1 command scstat -W.

For more details about checking the status of the cluster interconnect, see How to Check the Status of the Cluster Interconnect in Sun Cluster System Administration Guide for Solaris OS.

---

You might perform this procedure in the following scenarios:

• You need to replace a failed transport adapter.

• You need to replace a failed transport cable.

• You need to replace a failed transport junction.

For conceptual information about transport adapters, transport cables, and transport junction, see Cluster Interconnect in Sun Cluster Concepts Guide for Solaris OS.

Before You Begin

This procedure relies on the following prerequisites and assumptions.

• Your cluster has another functional interconnect path to maintain cluster communications while you perform this procedure.

• Your cluster is operational and all nodes are powered on.

• Identify the interconnect component that you want to replace. Remove that interconnect component from the cluster configuration by using the procedure in How to Remove Cluster Transport Cables, Transport Adapters, and Transport Switches in Sun Cluster System Administration Guide for Solaris OS.

  ---

  Note –

  PCI-SCI interconnect requires that you remove the interconnect component from the cluster configuration because the SCI driver needs to be updated. If the SCI driver is not updated, the nodes cannot use that interconnect to communicate. Although this practice is optional for all other interconnect technology, remove the interconnect component from the cluster configuration to prevent error messages from appearing on your console.

  ---

• If virtual local area networks (VLANs) are configured, more than one cluster might be impacted by removing a transport junction. Ensure that all clusters are prepared for the removal of a transport junction. Also, record the configuration information of the transport junction you plan to replace and configure the new transport junction accordingly.

  For more information about how to configure VLANs, see Configuring VLANs as Private Interconnect Networks.

1. Determine if you need to shut down and power off the node that is connected to the interconnect component you are replacing.

   • If you are replacing a transport junction, you do not need to shut down and power off the node. Proceed to Step 2.

   • If you are replacing a transport cable, you do not need to shut down and power off the node. Proceed to Step 2.

   • If your node has DR enabled and you are replacing a transport adapter, you do not need to shut down and power off the node. Proceed to Step 2.

   • If your node does not have DR enabled and you are replacing a transport adapter, shut down and power off the node with the transport adapter you are replacing.

     For the full procedure about how to shut down a node, see Chapter 3, Shutting Down and Booting a Cluster, in Sun Cluster System Administration Guide for Solaris OS.

2. Disconnect the failed interconnect component from other cluster devices.

   For the procedure about how to disconnect cables from transport adapters, see the documentation that shipped with your host adapter and node.

3. Connect the new interconnect component to other cluster devices.

   • If you are replacing an Ethernet or InfiniBand interconnect, see How to Install Ethernet or InfiniBand Transport Cables and Transport Junctions for cabling diagrams and considerations.

   • If you are replacing a PCI-SCI interconnect, see SPARC: How to Install PCI-SCI Transport Cables and Transport Junctions for cabling diagrams and considerations.

   • If your interconnect uses jumbo frames, review the requirements in Requirements When Using Jumbo Frames and see the Sun GigaSwift documentation for instructions. Refer to ce Sun Ethernet Driver Considerations for details of how to edit the ce.conf file according to the GigaSwift documentation's instructions.

4. If you shut down the node in Step 1, perform a reconfiguration boot to update the new Solaris device files and links. Otherwise, skip this step.

See Also

To reconfigure Sun Cluster software with the new interconnect component, see How to Add Cluster Transport Cables, Transport Adapters, or Transport Switches in Sun Cluster System Administration Guide for Solaris OS.

How to Remove an Interconnect Component

This procedure defines interconnect component as any one of the following components:

• Transport adapter

• Transport cable

• Transport junction (switch)

---

Caution –

You must maintain at least one cluster interconnect between the nodes of a cluster. The cluster does not function without a working cluster interconnect. You can check the status of the interconnect with one of the following commands:

• The Sun Cluster 3.2 command clinterconnect status.

• The Sun Cluster 3.1 command scstat -W.

For more details about checking the status of the cluster interconnect, see How to Check the Status of the Cluster Interconnect in Sun Cluster System Administration Guide for Solaris OS.

---

You might perform this procedure in the following scenarios:

• You need to remove an unused transport adapter.

• You need to remove an unused transport cable.

• You need to remove an unused transport junction.

• You want to migrate from a two–node cluster that uses switches to a point-to-point configuration.

For conceptual information about transport adapters, transport cables, and transport junctions, see Cluster Interconnect in Sun Cluster Concepts Guide for Solaris OS.

Before You Begin

This procedure assumes that your cluster is operational and all nodes are powered on.

Before you perform this procedure, perform the following tasks:

• If you are migrating from a two–node cluster that uses switches to a point-to-point configuration, install a crossover cable before you remove a switch.

• Identify the interconnect component that you want to remove. Remove that interconnect component from the cluster configuration by using the procedure in How to Remove Cluster Transport Cables, Transport Adapters, and Transport Switches in Sun Cluster System Administration Guide for Solaris OS.

  ---

  Note –

  PCI-SCI interconnect requires that you remove the interconnect component from the cluster configuration because the SCI driver needs to be updated. If the SCI driver is not updated, the nodes cannot use that interconnect to communicate. Although this practice is optional for all other interconnect technology, remove the interconnect component from the cluster configuration to prevent error messages from appearing on your console.

  ---

• If you plan to use virtual local area networks (VLANs) in your cluster interconnect, configure the transport junction. For more information about how configure VLANs, see Configuring VLANs as Private Interconnect Networks.

1. Determine if you need to shut down and power off the node that is connected to the interconnect component you are removing.

   • If you are removing a transport junction you, do not need to shut down and power off the node. Proceed to Step 2.

   • If you are removing a transport cable you, do not need to shut down and power off the node. Proceed to Step 2.

   • If your node has DR enabled and you are removing a transport adapter, you do not need to shut down and power off the node. Proceed to Step 2.

   • If your node does not have DR enabled and you are removing a transport adapter, shut down and power off the node with the transport adapter you are removing.

     For the full procedure about shutting down a node, see Chapter 3, Shutting Down and Booting a Cluster, in Sun Cluster System Administration Guide for Solaris OS.

2. Disconnect the interconnect component from other cluster devices.

   For the procedure about how to disconnect cables from transport adapters, see the documentation that shipped with your host adapter and node.

3. Remove the interconnect component.

   For the procedure about how to remove interconnect component, see the documentation that shipped with your host adapter, nodes, or switch.

4. If you shut down the node in Step 1, perform a reconfiguration boot to update the new Solaris device files and links. Otherwise, skip this step.

See Also

To reconfigure Sun Cluster software with the new interconnect component, see How to Add Cluster Transport Cables, Transport Adapters, or Transport Switches in Sun Cluster System Administration Guide for Solaris OS.

How to Upgrade Transport Adapter Firmware

You might perform this procedure in the following scenarios:

• You want to use firmware bug fixes.

• You want to use new firmware features.

Use this procedure to update transport adapter firmware.

Before You Begin

This procedure provides the long forms of the Sun Cluster commands. Most commands also have short forms. Except for the forms of the command names, the commands are identical. For a list of the commands and their short forms, see Appendix A, Sun Cluster Geographic Edition Object-Oriented Commands.

To perform this procedure, become superuser or assume a role that provides solaris.cluster.read and solaris.cluster.modify role-based access control (RBAC) authorization.

1. Determine the resource groups and the device groups that are online on the node. This node is the node on which you are upgrading transport adapter firmware.

   • If you are using Sun Cluster 3.2, use the following commands:

     # clresourcegroup status -n nodename # cldevicegroup status -n nodename

   • If you are using Sun Cluster 3.1, use the following command:

     # scstat

   Note the device groups, the resource groups, and the node list for the resource groups. You will need this information to restore the cluster to its original configuration in Step 4.

2. Migrate the resource groups and device groups off the node on which you plan to upgrade the firmware.

   • If you are using Sun Cluster 3.2, use the following command:

     # clnode evacuate fromnode

   • If you are using Sun Cluster 3.1, use the following command:

     # scswitch -S -h fromnode

3. Perform the firmware upgrade.

   This process might require you to boot into noncluster mode. If it does, boot the node into cluster mode before proceeding. For the procedure about how to upgrade your transport adapter firmware, see the patch documentation.

4. If you moved device groups off their original node in Step 2, restore the device groups that you identified in Step 1 to their original node.

   Perform the following step for each device group you want to return to the original node.

   • If you are using Sun Cluster 3.2, use the following command:

     # cldevicegroup switch -n nodename devicegroup1[ devicegroup2 …]

     -n nodename

     The node to which you are restoring device groups.

     devicegroup1[ devicegroup2 …]

     The device group or groups that you are restoring to the node.

   • If you are using Sun Cluster 3.1, use the following command:

     # scswitch -z -D devicegroup -h nodename

   In these commands, devicegroup is one or more device groups that are returned to the node.

5. If you moved resource groups off their original node in Step 2 restore the resource groups that you identified in Step 1 to their original node.

   • If you are using Sun Cluster 3.2, use the following command:

     Perform the following step for each resource group you want to return to the original node.

     # clresourcegroup switch -n nodename resourcegroup1[ resourcegroup2 …]

     nodename

     For failover resource groups, the node to which the groups are returned. For scalable resource groups, the node list to which the groups are returned.

     resourcegroup1[ resourcegroup2 …]

     The resource group or groups that you are returning to the node or nodes.

     resourcegroup

     The resource group that is returned to the node or nodes.

   • If you are using Sun Cluster 3.1, use the following command:

     # scswitch -z -g resourcegroup -h nodename