6.3 Setting the Subnet Manager Master on Oracle Exadata Database Machine Full Rack and Oracle Exadata Database Machine Half Rack
Oracle Exadata X3-2 systems and Oracle Exadata X2-2 systems have three Sun Datacenter InfiniBand Switch 36 switches. Starting with Oracle Exadata X4-2, Oracle Exadata Systems have two Sun Datacenter InfiniBand Switch 36 switches.
Note:
This procedure does not apply to Oracle Exadata X8M racks with RoCE Network Fabric.The switch located in rack unit 1 (U1) is referred to as the spine switch. The other two switches are referred to as the leaf switches. The location of the leaf switches is as follows:
-
Oracle Exadata Two-Socket Systems (X3-2 and later): rack unit 20 (U20) and rack unit 22 (U22)
-
Oracle Exadata X2-2 racks: rack unit 20 (U20) and rack unit 24 (U24)
-
Oracle Exadata Eight-Socket Systems (X2-8 and later) Full Racks: Rack unit 21 (U21) and rack unit 23 (U23)
The spine switch is the Subnet Manager Master for the InfiniBand Network Fabric subnet. The Subnet Manager Master has priority 8, and can be verified using the following procedure:
-
Log in to the spine switch as the
root
user. -
Run the
setsmpriority list
command.The command should show that
smpriority
has a value of 8. Ifsmpriority
has a different value, then do the following:-
Use the
disablesm
command to stop the Subnet Manager. -
Use the
setsmpriority 8
command to set the priority to 8. -
Use the
enablesm
command to restart the Subnet Manager.
-
The leaf switches are the Standby Subnet Managers with a priority of 5. This can be verified using the preceding procedure, substituting a value of 5 in the setsmpriority
command above.
Note:
Oracle Exadata Half Rack with Sun Fire X4170 Oracle Database Servers include two Sun Datacenter InfiniBand Switch 36 switches, which are set to priority 5.To determine the Subnet Manager Master, log in as the root
user on any InfiniBand Network Fabric switch, and run the getmaster
command. The location of the Subnet Manager Master is displayed. The following is an example of the output from the getmaster
command:
# getmaster
20100701 11:46:38 OpenSM Master on Switch : 0x0021283a8516a0a0 ports 36 Sun DCS 36
QDR switch dm01sw-ib1.example.com enhanced port 0 lid 1 lmc 0
The preceding output shows the proper configuration. The Subnet Master Manager is running on spine switch dm01sw-ib1.example.com
.
If the spine switch is not the Subnet Manager Master, then do the following procedure to set the Subnet Manager Master:
-
Use the
getmaster
command to identify the current location of the Subnet Manager Master. -
Log in as the
root
user on the leaf switch that is the Subnet Manager Master. -
Disable Subnet Manager on the switch. The Subnet Manager Master relocates to another switch.
-
Use the
getmaster
command to identify the current location of the Subnet Manager Master. If the spine switch is not Subnet Manager Master, then repeat steps 2 and 3 until the spine switch is the Subnet Manager Master. -
Enable Subnet Manager on the leaf switches that were disabled during this procedure.
Note:
- If the InfiniBand Network Fabric network consists of four or more racks cabled together, then only the spine switches should run Subnet Manager. The leaf switches should have Subnet Manager disabled on them.
- Oracle Exadata Half Racks with Sun Fire X4170 Oracle Database Servers, and Oracle Exadata Quarter Racks have two Sun Datacenter InfiniBand Switch 36 switches, and both are set to priority 5. The master is the one with the lowest GUID.
See Also:
- "Enable the Subnet Manager" in Sun Datacenter InfiniBand Switch 36 User's Guide
- "Disable the Subnet Manager" in Sun Datacenter InfiniBand Switch 36 User's Guide
- Oracle Exadata Database Machine System Overview for hardware component information
- Cabling tables in Oracle Exadata Database Machine System Overview
Parent topic: Configuring Oracle Exadata Database Machine