This section contains the following topics:
The subnet manager (SM) manages all operational characteristics of the InfiniBand network, such as the following:
Discovering the network topology
Assigning a local identifier (LID) to all ports connected to the network
Calculating and programming switch forwarding tables
Programming Partition Key (PKEY) tables at HCAs and switches
Programming QoS tables (Service Level to Virtual Lane mapping tables, and Virtual Lane arbitration tables)
Monitoring changes in the fabric
The InfiniBand network typically has more than one SM, but only one SM is active at a time. The active SM is Master SM, others are Standby SMs. If the master SM shuts down or fails, a standby SM will automatically become the master SM.
Note:
In the Exalogic machine, the InfiniBand switches (both leaf and spine) are automatically configured to separate the IP over InfiniBand (IPoIB) traffic and the Ethernet over InfiniBand (EoIB) traffic.
Table 13-1 provides information about the switches on which the subnet manager should run in different rack configurations.
Table 13-1 Running the Subnet Manager in Different Rack Configurations
| Rack Configuration | SM Should Run On... | SM Priority |
|---|---|---|
|
Single Exalogic machine |
All leaf switches |
All leaf switches: 5 |
|
Two half- or full-rack Exalogic machines |
Spine switches |
Spine switch: 8 |
|
Two quarter-rack Exalogic machines |
All leaf switches |
All leaf switches: 5 |
|
Three or more Exalogic machines |
Spine switches |
Spine switch: 8 |
|
Half- or full-rack Exalogic machine connected to a half- or full-rack Exadata machine. See also: Run the SM in Configurations with Varying Switch Firmware Versions. |
Spine switches |
Spine switch: 8 |
|
Quarter-rack Exalogic machine connected to a quarter-rack Exadata machine. See also: Run the SM in Configurations with Varying Switch Firmware Versions. |
All leaf switches |
All leaf switches: 5 |
|
Two or more Exalogic machines connected to two or more Exadata machines. See also: Run the SM in Configurations with Varying Switch Firmware Versions. |
Spine switches |
Spine switch: 8 |
In a multirack configuration consisting of both Exalogic and Exadata machines, if firmware upgrades result in switches with varying firmware versions across the configuration, the SM should run on only the switches with the latest firmware version. This is necessary to benefit from the features of the latest firmware.
Note that the SM should run on at least two switches in the fabric.
Consider a configuration that consists of three or more spine switches—for example, two Exalogic machines connected to two Exadata machines—but with varying firmware versions.
If two or more of the available spine switches, across the configuration, have the highest firmware version, the SM should run on those spine switches, with the priority set to 8.
If only one of the spine switches in the entire configuration has the highest firmware version:
The SM should run on that spine switch. The SM priority should be set to 8.
In addition, the SM should run on one or more leaf switches having the latest firmware version. The SM priority of the leaf switches should be set to 5.
In this case, running the SM on one or more leaf switches, besides running it on the spine switch, is necessary to fulfill the requirement that at least two SMs should be running in the fabric.
For more information about running the subnet manager, see the following topics:
This section contains the following topics:
If you want to quickly determine your Subnet Manager's priority and state, you can use the sminfo command.
On the command-line interface (CLI), run the following command:
# sminfo
The output is displayed, as in the following example:
sminfo: sm lid 15 sm guid 0x21283a8389a0a0, activity count 32046 priority 8 state3 SMINFO_MASTER
In the example output, the Subnet Manager's hosting HCA has LID 15 and GUID 0x21283a8620b0f0. The Subnet Manager has a priority of 8 (high) and its state is 3 (master).
On the command-line interface (CLI), run the following command:
# getmaster -l
The output is displayed, as in the following example:
# getmaster -l Last ring buffer history listed: whereismaster-daemon is running 20091204 15:00:53 whereismaster started 20091204 15:00:55 No OpenSM Master seen in the system 20091204 15:06:19 OpenSM Master on Switch : 0x0002c9000100d050 ports 36 Sun DCS 36 QDR switch o4nm2-36p-2.norway.test.com enhanced port 0 lid 7 lmc 0
This section contains the following topics:
From any InfiniBand switch in the network (leaf switch or spine switch), log in as root and run the getmaster command to obtain the location of the master SM as follows:
# getmaster
This command displays the host name or IP address and the IP address of the switch where the master SM is running.
You are required to relocate the master SM from a leaf switch (Sun Network QDR InfiniBand Gateway Switch) to the spine switch (Sun Datacenter InfiniBand Switch 36) when you are connecting more than one Exalogic machine. This step is also necessary when you are connecting an Exalogic machine to an Oracle Exadata Database Machine.
Relocating the master SM does not affect the availability of the InfiniBand network. You can perform this task while normal workload is running.
To relocate the master SM from a leaf switch (Sun Network QDR InfiniBand Gateway Switch) to the spine switch (Sun Datacenter InfiniBand Switch 36):
root user to the leaf switch where the master SM is located.