1. Installation and Configuration Guide for Exadata Database Machine
  2. Configuring Oracle Exadata Database Machine
  3. Setting the Subnet Manager Master on Oracle Exadata Database Machine Full Rack and Oracle Exadata Database Machine Half Rack

6.3 Setting the Subnet Manager Master on Oracle Exadata Database Machine Full Rack and Oracle Exadata Database Machine Half Rack

Oracle Exadata X3-2 systems and Oracle Exadata X2-2 systems have three Sun Datacenter InfiniBand Switch 36 switches. Starting with Oracle Exadata X4-2, Oracle Exadata Systems have two Sun Datacenter InfiniBand Switch 36 switches.

Note:

This procedure does not apply to Oracle Exadata X8M racks with RoCE Network Fabric.

The switch located in rack unit 1 (U1) is referred to as the spine switch. The other two switches are referred to as the leaf switches. The location of the leaf switches is as follows:

  • Oracle Exadata Two-Socket Systems (X3-2 and later): rack unit 20 (U20) and rack unit 22 (U22)

  • Oracle Exadata X2-2 racks: rack unit 20 (U20) and rack unit 24 (U24)

  • Oracle Exadata Eight-Socket Systems (X2-8 and later) Full Racks: Rack unit 21 (U21) and rack unit 23 (U23)

The spine switch is the Subnet Manager Master for the InfiniBand Network Fabric subnet. The Subnet Manager Master has priority 8, and can be verified using the following procedure:

  1. Log in to the spine switch as the root user.

  2. Run the setsmpriority list command.

    The command should show that smpriority has a value of 8. If smpriority has a different value, then do the following:

    1. Use the disablesm command to stop the Subnet Manager.

    2. Use the setsmpriority 8 command to set the priority to 8.

    3. Use the enablesm command to restart the Subnet Manager.

The leaf switches are the Standby Subnet Managers with a priority of 5. This can be verified using the preceding procedure, substituting a value of 5 in the setsmpriority command above.

Note:

Oracle Exadata Half Rack with Sun Fire X4170 Oracle Database Servers include two Sun Datacenter InfiniBand Switch 36 switches, which are set to priority 5.

To determine the Subnet Manager Master, log in as the root user on any InfiniBand Network Fabric switch, and run the getmaster command. The location of the Subnet Manager Master is displayed. The following is an example of the output from the getmaster command: 

# getmaster
20100701 11:46:38 OpenSM Master on Switch : 0x0021283a8516a0a0 ports 36 Sun DCS 36
QDR switch dm01sw-ib1.example.com enhanced port 0 lid 1 lmc 0

The preceding output shows the proper configuration. The Subnet Master Manager is running on spine switch dm01sw-ib1.example.com.

If the spine switch is not the Subnet Manager Master, then do the following procedure to set the Subnet Manager Master:

  1. Use the getmaster command to identify the current location of the Subnet Manager Master.

  2. Log in as the root user on the leaf switch that is the Subnet Manager Master.

  3. Disable Subnet Manager on the switch. The Subnet Manager Master relocates to another switch.

  4. Use the getmaster command to identify the current location of the Subnet Manager Master. If the spine switch is not Subnet Manager Master, then repeat steps 2 and 3 until the spine switch is the Subnet Manager Master.

  5. Enable Subnet Manager on the leaf switches that were disabled during this procedure.

Note:

  • If the InfiniBand Network Fabric network consists of four or more racks cabled together, then only the spine switches should run Subnet Manager. The leaf switches should have Subnet Manager disabled on them.
  • Oracle Exadata Half Racks with Sun Fire X4170 Oracle Database Servers, and Oracle Exadata Quarter Racks have two Sun Datacenter InfiniBand Switch 36 switches, and both are set to priority 5. The master is the one with the lowest GUID.

See Also:

Parent topic: Configuring Oracle Exadata Database Machine