Chapter 3 Installing Cluster Interconnect Hardware and Configuring VLANs

This chapter describes the procedures to install cluster interconnect hardware. Where appropriate, this chapter includes separate procedures for the interconnects that Sun Cluster software supports:

• Ethernet

• SPARC: PCI-SCI

• InfiniBand

This chapter contains the following information:

• Installing Ethernet or InfiniBand Cluster Interconnect Hardware

• SPARC: Installing PCI-SCI Cluster Interconnect Hardware

• SPARC: Installing Sun Fire Link Cluster Interconnect Hardware

• Configuring VLANs as Private Interconnect Networks

Use the following information to learn more about cluster interconnects:

• For conceptual information about cluster interconnects, see Cluster Interconnect in Sun Cluster Concepts Guide for Solaris OS.

• For information about how to administer cluster interconnects, see Chapter 7, Administering Cluster Interconnects and Public Networks, in Sun Cluster System Administration Guide for Solaris OS.

Interconnect Requirements and Restrictions

This section contains requirements on interconnect operation when using certain special features.

Cluster Interconnect and Routing

Heartbeat packets that are sent over the cluster interconnect are not IP based. As a result, these packets cannot be routed. If you install a router between two cluster nodes that are connected through cluster interconnects, heartbeat packets cannot find their destination. Your cluster consequently fails to work correctly.

To ensure that your cluster works correctly, you must set up the cluster interconnect in the same layer 2 (data link) network and in the same broadcast domain. The cluster interconnect must be located in the same layer 2 network and broadcast domain even if the cluster nodes are located in different, remote data centers. Cluster nodes that are arranged remotely are described in more detail in Chapter 7, Campus Clustering With Sun Cluster Software.

Cluster Interconnect Speed Requirements

An interconnect path is one network step in the cluster private network: from a node to a node, from a node to a switch, or from the switch to another node. Each path in your cluster interconnect must use the same networking technology, whether Ethernet or peripheral component interconnect-scalable coherent interface (PCI-SCI).

All interconnect paths must also operate at the same speed. This means, for example, that if you are using Ethernet components that are capable of operating at different speeds, and if your cluster configuration does not allow these components to automatically negotiate a common network speed, you must configure them to operate at the same speed.

Ethernet Switch Configuration When in the Cluster Interconnect

When configuring Ethernet switches for your cluster private interconnect, disable the spanning tree algorithm on ports that are used for the interconnect

Requirements When Using Jumbo Frames

If you use Scalable Data Services and jumbo frames on your public network, ensure that the Maximum Transfer Unit (MTU) of the private network is the same size or larger than the MTU of your public network.

---

Note –

Scalable services cannot forward public network packets that are larger than the MTU size of the private network. The scalable services application instances will not receive those packets.

---

Consider the following information when configuring jumbo frames:

• The maximum MTU size for an InfiniBand interface is typically less than the maximum MTU size for an Ethernet interface.

• If you use switches in your private network, ensure they are configured to the MTU sizes of the private network interfaces.

For information about how to configure jumbo frames, see the documentation that shipped with your network interface card. See your Solaris OS documentation or contact your Sun sales representative for other Solaris restrictions.

Requirements and Restrictions When Using InfiniBand in the Cluster Interconnect

The following requirements and guidelines apply to Sun Cluster Geographic Edition configurations that use InfiniBand adapters:

• A two-node cluster must use InfiniBand switches. You cannot directly connect the InfiniBand adapters to each other.

• Sun InfiniBand switches support up to nine nodes in a cluster.

• Jumbo frames are not supported on a cluster that uses InfiniBand adapters.

• If only one InfiniBand adapter is installed on a cluster node, each of its two ports must be connected to a different InfiniBand switch.

• If two InfiniBand adapters are installed in a cluster node, leave the second port on each adapter unused. For example, connect port 1 on HCA 1 to switch 1 and connect port 1 on HCA 2 to switch 2.

• VLANs are not supported on a cluster that uses InfiniBand switches.

Restriction on SCI Card Placement

Do not place a Scalable Coherent Interface (SCI) card in the 33 MHz PCI slot (slot 1) of the hot swap PCI+ (hsPCI+) I/O assembly. This placement can cause a system panic.

Installing Ethernet or InfiniBand Cluster Interconnect Hardware

The following table lists procedures for installing Ethernet or InfiniBand cluster interconnect hardware. Perform the procedures in the order that they are listed. This section contains the procedure for installing cluster hardware during an initial installation of a cluster, before you install Sun Cluster software.

Table 3–1 Installing Ethernet Cluster Interconnect Hardware

Task	For Instructions
Install the transport adapters.	The documentation that shipped with your nodes and host adapters
Install the transport cables.	How to Install Ethernet or InfiniBand Transport Cables and Transport Junctions
If your cluster contains more than two nodes, install a transport junction (switch).	How to Install Ethernet or InfiniBand Transport Cables and Transport Junctions

How to Install Ethernet or InfiniBand Transport Cables and Transport Junctions

Use this procedure to install Ethernet or InfiniBand transport cables and transport junctions (switches).

1. If not already installed, install transport adapters in your cluster nodes.

   See the documentation that shipped with your host adapters and node hardware.

2. If necessary, install transport junctions and optionally configure the transport junctions' IP addresses.

   ---

   Note –

   (InfiniBand Only) If you install one InfiniBand adapter on a cluster node, two InfiniBand switches are required. Each of the two ports must be connected to a different InfiniBand switch.

   If two InfiniBand adapters are connected to a cluster node, connect only one port on each adapter to the InfiniBand switch. The second port of the adapter must remain disconnected. Do not connect ports of the two InfiniBand adapters to the same InfiniBand switch.

   ---

3. Install the transport cables.

   • (Ethernet Only) As the following figure shows, a cluster with only two nodes can use a point-to-point connection, requiring no transport junctions.

     Figure 3–1 (Ethernet Only) Typical Two-Node Cluster Interconnect

     
     

     (Ethernet Only) For a point-to-point connection, you can use either UTP or fibre. With fibre, use a standard patch cable. A crossover cable is unnecessary. With UTP, see your network interface card documentation to determine whether you need a crossover cable.

     ---

     Note –

     (Ethernet Only) You can optionally use transport junctions in a two-node cluster. If you use a transport junction in a two-node cluster, you can more easily add additional nodes later. To ensure redundancy and availability, always use two transport junctions.

     ---

   • As the following figure shows, a cluster with more than two nodes requires transport junctions. These transport junctions are Ethernet or InfiniBand switches (customer-supplied).

     Figure 3–2 Typical Four-Node Cluster Interconnect

     
     

See Also

To install and configure the Sun Cluster software with the new interconnect, see Chapter 2, Installing Software on the Cluster, in Sun Cluster Software Installation Guide for Solaris OS.

(Ethernet Only) To configure jumbo frames on the interconnect, review the requirements in Requirements When Using Jumbo Frames and see the Sun GigaSwift documentation for instructions.

SPARC: Installing PCI-SCI Cluster Interconnect Hardware

Table 3–2 lists procedures about how to install Peripheral Component Interconnect-Scalable Coherent Interface (PCI-SCI) cluster interconnect hardware. Perform the procedures in the order that the procedures are listed. This section contains a procedure about how to install cluster hardware during an initial installation of a cluster before you install Sun Cluster software.

Table 3–2 SPARC: Task Map: Installing PCI-SCI Cluster Interconnect Hardware

Task	For Instructions, Go To
Install the transport adapters.	The documentation that shipped with your nodes and host adapters
Install the PCI-SCI transport cables.	SPARC: How to Install PCI-SCI Transport Cables and Transport Junctions
If you have a three-node or four-node cluster, install a PCI-SCI transport junction (switch).	SPARC: How to Install PCI-SCI Transport Cables and Transport Junctions

SPARC: How to Install PCI-SCI Transport Cables and Transport Junctions

Use this procedure to install PCI-SCI transport cables and transport junctions (switches).

When you perform this procedure, the following error messages might be displayed on your console.

• If you are using Solaris 8:

  Nov 13 20:11:43 e04a ip: ip_rput_dlpi(scid0): DL_ERROR_ACK for DL_ENABMULTI_REQ(29), errno 7, unix 0 Nov 13 20:11:43 e04a ip: ip: joining multicasts failed (7) on scid0 - will use link layer broadcasts for multicast

  These error messages are displayed because the associated driver does not support the multicast feature. These error messages are displayed when the ip module probes the driver. Sun Cluster Geographic Edition software does not use the multicast feature on the private interconnect. You can safely ignore these error messages.

• If you are using Solaris 9

  Dec 4 17:40:14 e03a in.routed[132]: write(rt_sock) RTM_ADD 172.17.0.128/25 -->172.17.0.129 metric=0 flags=0: File exists Dec 4 17:40:19 e03a in.routed[132]: interface scid0 to 172.17.0.129 broken: in=0 ierr=0 out=0 oerr=4

  These error messages are responses to the way Solaris 9 handles SCI dlpi interfaces. Solaris 9 uses the in.routed routing protocol as the default routing protocol. You can safely ignore these error messages. The in.routed routing protocol is the source of these error messages.

• This issue is fixed in Solaris 10 and no error messages should be displayed in this situation.

1. If not already installed, install PCI-SCI transport adapters in your cluster nodes.

   For the procedure about how to install PCI-SCI transport adapters and set their DIP switches, see the documentation that shipped with your PCI-SCI host adapters and node hardware.

   ---

   Note –

   Sbus-SCI host adapters are not supported by Sun Cluster software. If you are upgrading from a Sun Cluster 2.2 cluster, remove any Sbus-SCI host adapters from the cluster nodes. If you do not remove these adapters, you might see panic error messages during the SCI self test.

   ---

2. Install the PCI-SCI transport cables and optionally, transport junctions, depending on how many nodes are in your cluster.

   • Configuration With Point-to-Point Connections:

     A two-node cluster can use a point-to-point connection, requiring no transport junction.

     1. Connect the ends of the cables that are marked SCI Out to the I connectors on the adapters.

     2. Connect the ends of the cables that are marked SCI In to the O connectors of the adapters as shown in the previous diagram.

     See the following diagrams for cabling details.

   • Configuration With Transport Junction:

     A three-node or four-node cluster requires SCI transport junctions.

     1. Set the Unit selectors on the fronts of the SCI transport junctions to F. Do not use the X-Ports on the SCI transport junctions.

     2. Connect the ends of the cables that are marked SCI Out to the I connectors on the adapters and the Out connectors on the transport junctions.

     3. Connect the ends of the cables that are marked SCI In to the O connectors of the adapters and In connectors on the transport junctions, as shown in the previous diagram.

     See the following diagrams for cabling details. For the procedure about how to install and cable, see the SCI switch documentation that shipped with your hardware switches.

   Figure 3–3 Configuration With Point-to-Point Connections: Two Interconnects

   
   

   Figure 3–4 Configuration With Point-to-Point Connections: Four Interconnects

   
   

   Figure 3–5 Configuration With Transport Junction: Two Interconnects

   
   

   Figure 3–6 Configuration With Transport Junction: Four Interconnects

   
   

Troubleshooting

If you have problems with your PCI-SCI interconnect, perform the following tasks:

• Verify that the LED on the PCI-SCI transport adapter is blinking green rapidly. For detailed LED interpretations and actions, see the documentation that shipped with your host adapter.

• Verify that the PCI-SCI transport adapter card's DIP switch settings are correct. For more information, see the documentation that shipped with your PCI-SCI host adapter.

• Verify that the PCI-SCI cables are correctly connected. The PCI-SCI cable connects to the connector that is marked SCI In on the PCI-SCI adapter cards. If you are using transport junctions, the PCI-SCI cable also connects to the Out ports on the SCI transport junctions.

• Verify that the PCI-SCI cables are correctly connected. The PCI-SCI cable connects to the connector that is marked SCI Out on the PCI-SCI adapter cards. If you are using transport junctions, the PCI-SCI cable also connects to the In ports on the SCI transport junctions.

• Verify that the PCI-SCI switch unit selectors are set to F.

See Also

• To increase Oracle Real Application Clusters performance, set the max-vc-number parameter in the /etc/system file for each node. Choose the value that corresponds to the number of interconnects in your configuration:

  • (2 PCI-SCI interconnects) max-vc-number = 32768

  • (3 PCI-SCI interconnects) max-vc-number = 49152

  • (4 PCI-SCI interconnects) max-vc-number = 65536

• To install the Sun Cluster software and configure the Sun Cluster software with the new interconnect, see Chapter 2, Installing Software on the Cluster, in Sun Cluster Software Installation Guide for Solaris OS.

SPARC: Installing Sun Fire Link Cluster Interconnect Hardware

Table 3–3 lists procedures about how to install Sun Fire Link cluster interconnect hardware. Perform the procedures in the order that the procedures are listed.

Table 3–3 SPARC: Task Map: Installing Sun Fire Link Cluster Interconnect Hardware

Task	For Instructions
Install the transport adapters (Sun Fire Link optical module).	Sun Fire Link Hardware Installation Guide
Install the Sun Fire Link transport cables (Sun Fire Link cables)	Sun Fire Link Hardware Installation Guide
If you have a three-node or four-node cluster, install a Sun Fire Link transport junction (Sun Fire Link switch).	Sun Fire Link Hardware Installation Guide
Perform the Sun Fire Link software installation.	Sun Fire Link Software Installation Guide
Create and activate a dual-controller Sun Fire Link fabric.	Sun Fire Link Fabric Administrator's Guide

Configuring VLANs as Private Interconnect Networks

Sun Cluster software supports the use of private interconnect networks over switch-based virtual local area networks (VLANs). In a switch-based VLAN environment, Sun Cluster software enables multiple clusters and nonclustered systems to share an Ethernet transport junction (switch) in two different configurations.

---

Note –

Even if clusters share the same switch, create a separate VLAN for each cluster.

By default, Sun Cluster uses the same set of IP addresses on the private interconnect. Creating a separate VLAN for each cluster ensures that IP traffic from one cluster does not interfere with IP traffic from another cluster. Unless you have customized the default IP address for the private interconnect, as described in How to Change the Private Network Address or Address Range of an Existing Cluster in Sun Cluster System Administration Guide for Solaris OS, create a separate VLAN for each cluster.

---

The implementation of switch-based VLAN environments is vendor-specific. Because each switch manufacturer implements VLAN differently, the following guidelines address Sun Cluster software requirements with regard to configuring VLANs with cluster interconnects.

• You must understand your capacity needs before you set up a VLAN configuration. You must know the minimum bandwidth necessary for your interconnect and application traffic.

  For the best results, set the Quality of Service (QOS) level for each VLAN to accommodate basic cluster traffic and the desired application traffic. Ensure that the bandwidth that is allocated to each VLAN extends from node to node.

  To determine the basic cluster traffic requirements, use the following equation. In this equation, n equals the number of nodes in the configuration, and s equals the number of switches per VLAN.

  n (s-1) x 10Mb

• Interconnect traffic must be placed in the highest-priority queue.

• All ports must be equally serviced, similar to a round robin or first-in, first-out model.

• You must verify that you have correctly configured your VLANs to prevent path timeouts.

The first VLAN configuration enables nodes from multiple clusters to send interconnect traffic across one pair of Ethernet transport junctions. Sun Cluster software requires a minimum of one transport junction, and each transport junction must be part of a VLAN that is located on a different switch. The following figure is an example of the first VLAN configuration in a two-node cluster. VLAN configurations are not limited to two-node clusters.

Figure 3–7 First VLAN Configuration

The second VLAN configuration uses the same transport junctions for the interconnect traffic of multiple clusters. However, the second VLAN configuration has two pairs of transport junctions that are connected by links. This configuration enables VLANs to be supported in a campus cluster configuration with the same restrictions as other campus cluster configurations. The following figure illustrates the second VLAN configuration.

Figure 3–8 Second VLAN Configuration