This chapter provides the software procedures for administering the Sun Cluster interconnects and public networks.
Administering the cluster interconnects and public networks consists of both hardware and software procedures. Typically, you configure the cluster interconnects and public networks, including Internet Protocol (IP) Network Multipathing groups, when you initially install and configure the cluster. If you later need to alter a cluster interconnect network configuration, you can use the software procedures in this chapter. For information about configuring IP Network Multipathing groups in a cluster, see the section Administering the Public Network.
This is a list of the procedures in this chapter.
How to Add Cluster Transport Cables, Transport Adapters, or Transport Junctions
How to Remove Cluster Transport Cables, Transport Adapters, and Transport Junctions
How to Administer IP Network Multipathing Groups in a Cluster
For a high-level description of the related procedures in this chapter, see Table 6–1 and Table 6–3.
Refer to the Sun Cluster Concepts Guide for Solaris OS document for background and overview information on the cluster interconnects and public networks.
This section provides the procedures for reconfiguring cluster interconnects, such as cluster transport adapters and cluster transport cables. These procedures require that you install Sun Cluster software.
Most of the time, you can use the scsetup(1M) utility to administer the cluster transport for the cluster interconnects. See the scsetup man page for more information.
For cluster software installation procedures, see the Sun Cluster Software Installation Guide for Solaris OS. For procedures about servicing cluster hardware components, see the Sun Cluster Hardware Administration Manual for Solaris OS .
You can usually choose to use the default port name, where appropriate, during cluster interconnect procedures. The default port name is the same as the internal node ID number of the node that hosts the adapter end of the cable. However, you cannot use the default port name for certain adapter types, such as SCI.
Task |
For Instructions, Go To... |
---|---|
Administer the cluster transport - Use scsetup(1M) | |
Check the status of the cluster interconnect - Use scstat | |
Add a cluster transport cable, transport adapter, or transport junction - Use scstat(1M) |
How to Add Cluster Transport Cables, Transport Adapters, or Transport Junctions |
Remove a cluster transport cable, transport adapter, or transport junction - Use scsetup |
How to Remove Cluster Transport Cables, Transport Adapters, and Transport Junctions |
Enable a cluster transport cable - Use scsetup | |
Disable a cluster transport cable - Use scsetup |
There are a few issues you must consider when completing dynamic reconfiguration (DR) operations on cluster interconnects.
All of the requirements, procedures, and restrictions that are documented for the Solaris DR feature also apply to Sun Cluster DR support (except for the operating environment quiescence operation). Therefore, review the documentation for the Solaris DR feature before using the DR feature with Sun Cluster software. You should review in particular the issues that affect non-network IO devices during a DR detach operation.
Sun Cluster rejects DR remove-board operations performed on active private interconnect interfaces.
If the DR remove-board operation pertains to an active private interconnect interface, Sun Cluster rejects the operation and identifies the interface that would be affected by the operation.
Sun Cluster requires that each cluster node has at least one functioning path to every other cluster node. Do not disable a private interconnect interface that supports the last path to any cluster node.
Complete the following procedures in the order indicated when performing DR operations on public network interfaces.
Table 6–2 Task Map: Dynamic Reconfiguration with Public Network Interfaces
Task |
For Instructions, Go To... |
---|---|
1. Disable and remove the interface from the active interconnect |
How to Remove Cluster Transport Cables, Transport Adapters, and Transport Junctions |
2. Perform the DR operation on the public network interface. |
Sun Enterprise 10000 DR Configuration Guide and the Sun Enterprise 10000 Dynamic Reconfiguration Reference Manual (from the Solaris 8 on Sun Hardware and Solaris 9 on Sun Hardware collections) |
You can also accomplish this procedure by using the SunPlex Manager GUI. See the SunPlex Manager online help for more information.
You do not need to be logged in as superuser to perform this procedure.
Check the status of the cluster interconnect.
# scstat -W |
Refer to the following for common status messages.
Status Message |
Description and Possible Action |
---|---|
Path online |
The path is currently functioning correctly. No action is necessary. |
Path waiting |
The path is currently being initialized. No action is necessary. |
Path faulted |
The path is not functioning. This can be a transient state when paths are going between the waiting and online state. If the message persists when scstat -W is rerun, take corrective action. |
The following example shows the status of a functioning cluster interconnect.
# scstat -W -- Cluster Transport Paths -- Endpoint Endpoint Status -------- -------- ------ Transport path: phys-schost-1:qfe1 phys-schost-2:qfe1 Path online Transport path: phys-schost-1:qfe0 phys-schost-2:qfe0 Path online Transport path: phys-schost-1:qfe1 phys-schost-3:qfe1 Path online Transport path: phys-schost-1:qfe0 phys-schost-3:qfe0 Path online Transport path: phys-schost-2:qfe1 phys-schost-3:qfe1 Path online Transport path: phys-schost-2:qfe0 phys-schost-3:qfe0 Path online |
You can also accomplish this procedure by using the SunPlex Manager GUI. See the SunPlex Manager online help for more information.
Ensure that the physical cluster transport cables are installed.
For the procedure on installing a cluster transport cable, see the Sun Cluster Hardware Administration Manual for Solaris OS .
Become superuser on any node in the cluster.
Enter the scsetup utility.
# scsetup |
The Main Menu is displayed.
Access the Cluster Interconnect Menu by typing 3 (Cluster interconnect).
If your configuration uses SCI adapters, do not accept the default when you are prompted for the adapter connections (the port name) during the “Add” portion of this procedure. Instead, provide the port name (0, 1, 2, or 3) found on the Dolphin switch, to which the node is physically cabled.
Add the transport cable by typing 1 (Add a transport cable).
Follow the instructions and enter the requested information.
Add the transport adapter by typing 2 (Add a transport adapter to a node).
Follow the instructions and enter the requested information.
Add the transport junction by typing 3 (Add a transport junction).
Follow the instructions and enter the requested information.
Verify that the cluster transport cable, transport adapter, or transport junction is added.
# scconf -p | grep cable # scconf -p | grep adapter # scconf -p | grep junction |
The following example shows how to add a transport cable, transport adapter, or transport junction to a node using the scsetup command.
[Ensure the physical cable is installed.] # scsetup Select Cluster interconnect. Select either Add a transport cable, Add a transport adapter to a node, or Add a transport junction. Answer the questions when prompted. You Will Need: Example: node names phys-schost-1 adapter names qfe2 junction names hub2 transport type dlpi [Verify that the scconf command completed successfully:] Command completed successfully. Quit the scsetup Cluster Interconnect Menu and Main Menu. [Verify that the cable, adapter, and junction are added:] # scconf -p | grep cable Transport cable: phys-schost-2:qfe0@1 ethernet-1@2 Enabled Transport cable: phys-schost-3:qfe0@1 ethernet-1@3 Enabled Transport cable: phys-schost-1:qfe0@0 ethernet-1@1 Enabled # scconf -p | grep adapter Node transport adapters: qfe2 hme1 qfe0 Node transport adapter: qfe0 Node transport adapters: qfe0 qfe2 hme1 Node transport adapter: qfe0 Node transport adapters: qfe0 qfe2 hme1 Node transport adapter: qfe0 # scconf -p | grep junction Cluster transport junctions: hub0 hub1 hub2 Cluster transport junction: hub0 Cluster transport junction: hub1 Cluster transport junction: hub2 |
You can also accomplish this procedure by using the SunPlex Manager GUI. See the SunPlex Manager online help for more information.
Use the following procedure to remove cluster transport cables, transport adapters, and transport junctions from a node configuration. When a cable is disabled, the two endpoints of the cable remain configured. An adapter cannot be removed if it is still in use as an endpoint on a transport cable.
Each cluster node needs at least one functioning transport path to every other node in the cluster. No two nodes should be isolated from one another. Always verify the status of a node's cluster interconnect before disabling a cable. Only disable a cable connection after you have verified that it is redundant; that is, that another connection is available. Disabling a node's last remaining working cable takes the node out of cluster membership.
Become superuser on any node in the cluster.
Check the status of the remaining cluster transport path.
# scstat -W |
If you receive an error such as “path faulted” while attempting to remove one node of a two-node cluster, investigate the problem before continuing with this procedure. Such a problem could indicate that a node path is unavailable. Removing the remaining good path takes the node out of cluster membership and could result in a cluster reconfiguration.
Enter the scsetup utility.
# scsetup |
The Main Menu is displayed.
Access the Cluster Interconnect Menu by typing 3 (Cluster interconnect).
Remove the cable by typing 4 (Remove a transport cable).
Follow the instructions and enter the requested information. You will need to know the applicable node names, adapter names, and junction names.
If you are removing a physical cable, disconnect the cable between the port and the destination device.
Remove the adapter by typing 5 (Remove a transport adapter from a node).
Follow the instructions and enter the requested information. You will need to know the applicable node names, adapter names, and junction names.
If you are removing a physical adapter from a node, see the Sun Cluster Hardware Administration Manual for Solaris OS for hardware service procedures.
Remove the junction by typing 6 (Remove a transport junction).
Follow the instructions and enter the requested information. You will need to know the applicable node names, adapter names, and junction names.
A junction cannot be removed if any of the ports are still in use as endpoints on any transport cables.
Verify that the cable or the adapter has been removed.
# scconf -p | grep cable # scconf -p | grep adapter # scconf -p | grep junction |
The transport cable or adapter removed from the given node should not appear in the output from this command.
The following example shows how to remove a transport cable, transport adapter, or transport junction using the scsetup command.
[Become superuser on any node in the cluster.] [Enter the utility:] # scsetup Type 3 (Cluster interconnect). Select either Add a transport cable, Add a transport adapter to a node, or Add a transport junction. Answer the questions when prompted. You Will Need: Example: node names phys-schost-1 adapter names qfe1 junction names hub1 [Verify that the scconf command completed successfully:] “Command completed successfully.” Quit the scsetup Cluster Interconnect Menu and Main Menu. [Verify that the cable, adapter, or junction is removed:] # scconf -p | grep cable Transport cable: phys-schost-2:qfe0@1 ethernet-1@2 Enabled Transport cable: phys-schost-3:qfe0@1 ethernet-1@3 Enabled Transport cable: phys-schost-1:qfe0@0 ethernet-1@1 Enabled # scconf -p | grep adapter Node transport adapters: qfe2 hme1 qfe0 Node transport adapter: qfe0 Node transport adapters: qfe0 qfe2 hme1 Node transport adapter: qfe0 Node transport adapters: qfe0 qfe2 hme1 Node transport adapter: qfe0 # scconf -p | grep junction Cluster transport junctions: hub0 hub2 Cluster transport junction: hub0 Cluster transport junction: hub2 |
You can also accomplish this procedure by using the SunPlex Manager GUI. See the SunPlex Manager online help for more information.
This option is used to enable an already existing cluster transport cable.
Become superuser on any node in the cluster.
Enter the scsetup(1M) utility.
# scsetup |
The Main Menu is displayed.
Access the Cluster Interconnect Menu by typing 2 (Cluster interconnect).
Enable the transport cable by typing 7 (Enable a transport cable).
Follow the instructions when prompted. You need to enter both the node and the adapter names of one of the endpoints of the cable you are trying to identify.
Verify that the cable is enabled.
# scconf -p | grep cable |
This example shows how to enable a cluster transport cable on adapter qfe-1 located on the node phys-schost-2.
[Become superuser on any node.] [Enter the scsetup utility:] # scsetup Select Cluster interconnect>Enable a transport cable. Answer the questions when prompted. You will need the following information. You Will Need: Example: node names phys-schost-2 adapter names qfe1 junction names hub1 [Verify that the scconf command completed successfully:] scconf -c -m endpoint=phys-schost-2:qfe1,state=enabled Command completed successfully. Quit the scsetup Cluster Interconnect Menu and Main Menu. [Verify that the cable is enabled:] # scconf -p | grep cable Transport cable: phys-schost-2:qfe1@0 ethernet-1@2 Enabled Transport cable: phys-schost-3:qfe0@1 ethernet-1@3 Enabled Transport cable: phys-schost-1:qfe0@0 ethernet-1@1 Enabled |
You can also accomplish this procedure by using the SunPlex Manager GUI. See the SunPlex Manager online help for more information.
You might need to disable a cluster transport cable to temporarily shut down a cluster interconnect path. This is useful when troubleshooting a cluster interconnect problem or when replacing cluster interconnect hardware.
When a cable is disabled, the two endpoints of the cable remain configured. An adapter cannot be removed if it is still in use as an endpoint in a transport cable.
Each cluster node needs at least one functioning transport path to every other node in the cluster. No two nodes should be isolated from one another. Always verify the status of a node's cluster interconnect before disabling a cable. Only disable a cable connection after you have verified that it is redundant; that is, that another connection is available. Disabling a node's last remaining working cable takes the node out of cluster membership.
Become superuser on any node in the cluster.
Check the status of the cluster interconnect before disabling a cable.
# scstat -W |
If you receive an error such as “path faulted” while attempting to remove one node of a two-node cluster, investigate the problem before continuing with this procedure. Such a problem could indicate that a node path is unavailable. Removing the remaining good path takes the node out of cluster membership and could result in a cluster reconfiguration.
Enter the scsetup(1M) utility.
# scsetup |
The Main Menu is displayed.
Access the Cluster Interconnect Menu by typing 3 (Cluster interconnect).
Disable the cable by typing 8 (Disable a transport cable).
Follow the instructions and enter the requested information. All of the components on this cluster interconnect will be disabled. You need to enter both the node and the adapter names of one of the endpoints of the cable you are trying to identify.
Verify that the cable is disabled.
# scconf -p | grep cable |
This example shows how to disable a cluster transport cable on adapter qfe-1 located on the node phys-schost-2.
[Become superuser on any node.] [Enter the scsetup utility:] # scsetup Select Cluster interconnect>Disable a transport cable. Answer the questions when prompted. You will need the following information. You Will Need: Example: node names phys-schost-2 adapter names qfe1 junction names hub1 [Verify that the scconf command completed successfully:] scconf -c -m endpoint=phys-schost-2:qfe1,state=disabled Command completed successfully. Quit the scsetup Cluster Interconnect Menu and Main Menu. [Verify that the cable is disabled:] # scconf -p | grep cable Transport cable: phys-schost-2:qfe1@0 ethernet-1@2 Disabled Transport cable: phys-schost-3:qfe0@1 ethernet-1@3 Enabled Transport cable: phys-schost-1:qfe0@0 ethernet-1@1 Enabled |
Sun Cluster 3.1 4/04 supports the Solaris implementation of Internet Protocol (IP) Network Multipathing for public networks. Basic IP Network Multipathing administration is the same for both cluster and non-cluster environments. Multipathing administration is covered in the appropriate Solaris documentation. However, review the guidelines that follow before administering IP Network Multipathing in a Sun Cluster environment.
Before performing IP Network Multipathing procedures on a cluster, consider the guidelines below.
Each public network adapter must belong to a multipathing group.
The local-mac-address? variable must have a value of true for Ethernet adapters.
For multipathing groups that contain two or more adapters, you must configure a test IP address for each adapter in the group. If a multipathing group contains only one adapter, you do not need to configure a test IP address.
Test IP addresses for all adapters in the same multipathing group must belong to a single IP subnet.
Test IP addresses must not be used by normal applications because they are not highly available.
There are no restrictions on multipathing group naming. However, when configuring a resource group, the netiflist naming convention is any multipathing name followed by either the nodeID number or the node name. For example, given a multipathing group named sc_ipmp0, the netiflist naming could be either sc_ipmp0@1 or sc_ipmp0@phys-schost-1, where the adapter is on the node phys-schost-1 which has the nodeID of 1.
Avoid unconfiguring (unplumbing) or bringing down an adapter of an IP Network Multipathing group without first switching over the IP addresses from the adapter to be removed to an alternate adapter in the group, using the if_mpadm(1M) command.
Avoid rewiring adapters to different subnets without first removing them from their respective multipathing groups.
Logical adapter operations can be done on an adapter even if monitoring is on for the multipathing group.
You must maintain at least one public network connection for each node in the cluster. The cluster is inaccessible without a public network connection.
To view the status of IP Network Multipathing groups on a cluster, use the command scstat(1M) with the -i option.
For more information about IP Network Multipathing, see the appropriate documentation in the Solaris system administration documentation set.
Table 6–3 Task Map: Administering the Public Network
Solaris Operating Environment Release |
For Instructions, Go To... |
---|---|
Solaris 8 operating environment |
IP Network Multipathing Administration Guide |
Solaris 9 operating environment |
“IP Network Multipathing Topics” inSystem Administration Guide: IP Series |
For cluster software installation procedures, see the Sun Cluster Software Installation Guide for Solaris OS. For procedures about servicing public networking hardware components, see the Sun Cluster Hardware Administration Manual for Solaris OS.
There are a few issues you must consider when completing dynamic reconfiguration (DR) operations on public network interfaces in a cluster.
All of the requirements, procedures, and restrictions that are documented for the Solaris DR feature also apply to Sun Cluster DR support (except for the operating environment quiescence operation). Therefore, review the documentation for the Solaris DR feature before using the DR feature with Sun Cluster software. You should review in particular the issues that affect non-network IO devices during a DR detach operation.
DR remove-board operations can succeed only when public network interfaces are not active. Before removing an active public network interface, switch the IP addresses from the adapter to be removed to another adapter in the multipathing group, using the if_mpadm(1M) command.
If you try to remove a public network interface card without having properly disabled it as an active network interface, Sun Cluster rejects the operation and identifies the interface that would be affected by the operation.
For multipathing groups with two adapters, if the remaining network adapter fails while you are performing the DR remove operation on the disabled network adapter, availability is impacted. The remaining adapter has no place to fail over for the duration of the DR operation.
Complete the following procedures in the order indicated when performing DR operations on public network interfaces.
Table 6–4 Task Map: Dynamic Reconfiguration with Public Network Interfaces
Task |
For Instructions, Go To... |
---|---|
1. Switch the IP addresses from the adapter to be removed to another adapter in the multipathing group, using the if_mpadm. |
if_mpadm(1M) man page. The appropriate Solaris documentation: Solaris 8: IP Network Multipathing Administration Guide Solaris 9: “IP Network Multipathing Topics” in System Administration Guide: IP Services |
2. Remove the adapter from the multipathing group using the ifconfig command. |
The appropriate Solaris documentation: Solaris 8: IP Network Multipathing Administration Guide Solaris 9: “IP Network Multipathing Topics” in System Administration Guide: IP Services ifconfig(1M) man page |
3. Perform the DR operation on the public network interface. |
Sun Enterprise 10000 DR Configuration Guide and the Sun Enterprise 10000 Dynamic Reconfiguration Reference Manual (from the Solaris 8 on Sun Hardware and Solaris 9 on Sun Hardware collections) |