| Sun IB Switch 9P Administration Guide |     |
| Sun IB Switch 9P Administration Guide | |
| A P P E N D I X D |
|
Blueprints |
This appendix describes how to set up a blueprint for your switch topology. It contains the following sections:
A blueprint describes how the InfiniBand (IB) management software running on the switch should set up the IB network. In addition, it also defines the maximum number of nodes in the topology.
In the initial release of the Sun IB switch 9p product, only a single predefined configuration was supported (that is, a single switch with up to 9 ports connected to host channel adapters). The reason for this was to allow the switch to be used in Sun Cluster configurations where this kind of connectivity is sufficient (that is, up to 9 hosts), and where the system administrators are not expected to have any knowledge about InfiniBand.
With a single predefined configuration, the switch could be used as a black-box with no need to configure IB specific parameters. However, to support redundant switch configurations (that is, dual, independent InfiniBand subnets), each subnet must have a unique subnet ID. Hence, despite the black-box approach, each switch would need to present a subnet ID that is different than the one used by the other switch in a redundant configuration.
Each switch is uniquely identified by its serial number. Hence, the switch serial number could have been used to present a unique subnet ID (that is, the IB subnet prefix). However, since a change of IB subnet prefix currently requires a manual re-configuration of the hosts attached to this subnet, this was not a desirable solution. Instead the IP address assigned to the Ethernet port of the switch is used as a basis for unique IB subnet IDs. This approach has the nice properties that unique values are assigned to include the switches in the site-specific management network. Additionally, if a switch is replaced, the new switch will typically be assigned the same IP address as the former switch, since it is taking over the "personality" of the former switch. However, if for some reason the IP address is changed on a switch in an operational system or a switch is replaced without updating the IP address on the new switch, then the Solaris hosts attached to this switch will no longer be able to communicate until a re-configuration has been performed.
|
Note - These aspects were not explained in the documentation for the first release of the Sun IB switch 9p product. |
The second release of the switch firmware includes three important changes:
A predefined configuration is referred to as a blueprint. Hence, new commands have been introduced in order to define which blueprint is to be used as well as which role this switch is to have in configurations that involve more than a single switch instance in each InfiniBand subnet.
The same black-box model for single switch configurations are supported as in the initial release. The default configuration is the one outlined previously.
The currently supported blueprints have the following options: 9 node, 12 node, 18 node, none, or unmanaged. Use the setbp command to set up the blueprint of the switch (see setbp) Each of these blueprints is described in the following sections.
One parameter that is part of almost all of the blueprints is the IB subnet ID that must be used for the IB subnet. This parameter must follow two rules:
In topologies that consist of more than one switch, all switches must be set up with the same blueprint. In addition, a maximum of three connections can be setup between each pair of switches.
|
Note - A warning message will be issued if these guidelines are not followed. |
A 9-node blueprint is a single switch topology with IB management software active on the switch. This represents the default blueprint. It is the blueprint used if the setbp command hasn't been performed on the switch.
A 12-node blueprint is a dual switch topology with IB management software active on both switches. One switch acts as the master management node, the other as standby. Standard IB mechanisms determine which switch is selected as master.
In this configuration each switch connects to maximum of six Host Channel Adapters (HCAs). The remaining three ports can be used to connect the two switches.
An 18-node blueprint is a topology that consists of four switches all with active IB management software.
One switch acts as the master management node, the three others as standby.
In this topology three switches connect to a maximum of six HCAs while the three remaining ports can be connected to the fourth switch. The fourth switch can then only connect to the other switches through three connections to each.
The switch that only connects to other switches is denoted as the top switch and will always be the master management node. The other switches are denoted bottom switch. These terms are used when the blueprint settings are displayed through the showbp command.
The none blueprint does not have any topology size parameter. This blueprint can only be used temporarily before configuring for the proper topology. When the none blueprint is active, the management software will find out what the topology looks like but the endnodes will be setup so that they cannot communicate. Use the showib command to display the topology. The output can be used to determine which blueprint (9, 12, or 18 node) fits the actual topology.
The unmanaged blueprint turns off the IB management software on the switch. Use this blueprint if the switch is used in configurations other than those covered by the previous blueprints.
You can change from one configuration (blueprint) to another without re-configuring the hosts by ensuring that the IB subnet prefix is maintained when changing from one configuration to another. However, changing the blueprint on a switch cannot be done without re-booting the switch. Hence, the hosts connected to the rebooting switch will loose access to the subnet during this transition period.
In cascaded configurations, one switch can be replaced (or removed or powered down) without impacting the other switches in the configuration or the hosts that are connected to those switches. To construct cascaded configurations, switches can be configured and connected in any order as long as they are configured with consistent blueprint type, subnet prefix and switch role.
However, merging two operational configurations cannot be performed without performing a manual re-configuration of all the hosts in one of the original configurations. Hence, if two switches are both configured as part of a 12-node blueprint with the same subnet IDs, but are not connected to each others before they are booted and configured, then they can not be interconnected at a later time without having to perform a re-configuration of (some of) the connected hosts.
In order to avoid such problems, it is recommended to always connect and configure all switches that are supposed to be part of a subnet before connecting (or booting) any associated host. - In the case of replacing a switch in an operational configuration, the implication is that the new switch should be connected to the existing switch(es) before it is allowed to become operational (with connected, operational hosts).
If a subnet has been made operational with all associated switches present, then it is possible to disconnect and then re-connect the switches without having to re-configure anything. Also, it is possible to change connection of a host from one switch port to another port on another switch without having to perform any re-configuration.
Connecting switches with conflicting configurations will not impact existing operational connectivity for the switch, but no communication will be possible between the switches where the blueprint configurations does not match.
If for some reason it is decided to reduce the number of nodes in a configuration (e.g. split a 12-node configuration into two disjoint parts), then this can be done without changing the blueprint setting on any of the switches. However, if changing the blueprint is required in order to allow more hosts to be connected to a single switch, then this may imply that the hosts that are currently attached to the switch have to be re-configured.
Similarly, if an operational host is disconnected from one subnet and then connected to another subnet, then it will have to be re-configured to become operational on the new subnet.
| Sun IB Switch 9P Administration Guide | 819-7247-10 | |
Copyright © 2004, Sun Microsystems, Inc. All Rights Reserved.