4.7.4 Using Access VLANs with Cisco Nexus 9336C-FX2 RoCE Network Fabric Switches

You can change the switchport access vlan ID setting to implement server-level isolation across the RoCE Network Fabric.

By default, Oracle Exadata uses Access VLAN ID 3888 for all RoCE Network Fabric private network traffic, on the server re0 and re1 interfaces.

If you change the default Access VLAN ID setting on the Cisco Nexus 9336C-FX2 RoCE Network Fabric switches, the corresponding server can no longer communicate with other database servers or storage servers using the default setting. The range of valid Access VLAN IDs is 2744-3967.

Use the following procedure to implement different Access VLANs on the RoCE Network Fabric.

Note:

It is recommended that you shut down the servers while modifying the switch configuration to avoid any outages. If you are changing the Access VLAN ID for a subset of the servers in the rack, then only the affected servers need to be shut down prior to modifying the switch.

1. Identify the RoCE Network Fabric switch host name and port that is connected to the re0 interface of the server that you are configuring.

   [root@dbm01adm02 ~]# /opt/oracle.SupportTools/ibdiagtools/utils/lldp_cap.py re0 
   | egrep 'SWITCH_PORT:|SWITCH_NAME:'
   SWITCH_PORT: Ethernet1/21
   SWITCH_PORT_DESCRIPTION: adm02

2. Log in to the lower leaf switch using the SWITCH_NAME identified in the previous step.

   # ssh admin@dbm01sw-rocea0
   User Access Verification
   Password: *******
   

3. Check the configuration for the switch port.

   To specify the interface, use the SWITCH_PORT identified in Step 1.

   dbm01sw-rocea0# show running-config interface ethernet 1/21
   !Running configuration last done at: Wed Nov 13 10:34:58 2019
   !Time: Wed Nov 13 14:55:48 2019
   
   version 7.0(3)I7(6) Bios:version 05.33 
   
   interface Ethernet1/21
     description adm02
     switchport access vlan 3888
     priority-flow-control mode on
     spanning-tree port type edge
     spanning-tree bpduguard enable
     mtu 2300
     speed 100000
     duplex full
     no negotiate auto
     service-policy type qos input QOS_MARKING no-stats  

4. Create the new VLAN ID on the switch.

   This configuration is only required once on each switch. However, it is harmless to repeat the VLAN configuration on a switch.

   For example, to create a new VLAN ID with the value 3889:

   dbm01sw-rocea0# configure terminal 
   Enter configuration commands, one per line. End with CNTL/Z.
   dbm01sw-rocea0(config)# vlan 3889
   dbm01sw-rocea0(config-vlan)# exit
   dbm01sw-rocea0(config)# exit
   dbm01sw-rocea0# 

5. Modify the switch port configuration to change the switchport access vlan setting.

   Specify the same interface as in the previous steps. Then, remove the old VLAN ID (for example, 3888), add the new VLAN ID (for example, 3889), and exit configuration mode.

   dbm01sw-rocea0# configure terminal 
   Enter configuration commands, one per line. End with CNTL/Z.
   dbm01sw-rocea0(config)# interface ethernet 1/21
   dbm01sw-rocea0(config-if)# no switchport access vlan 3888
   dbm01sw-rocea0(config-if)# switchport access vlan 3889
   dbm01sw-rocea0(config-if)# exit
   dbm01sw-rocea0(config)# exit
   dbm01sw-rocea0# 

6. Verify that the switch interface is using the new VLAN ID.

   Specify the same interface as in the previous steps.

   dbm01sw-rocea0# show running-config interface ethernet 1/21
    
   !Command: show running-config interface Ethernet1/21
   !Running configuration last done at: Wed Nov 20 23:53:38 2019
   !Time: Wed Nov 20 23:53:45 2019
    
   version 7.0(3)I7(6) Bios:version 05.33 
    
   interface Ethernet1/21
     description adm02
     switchport access vlan 3889
     priority-flow-control mode on
     spanning-tree port type edge
     spanning-tree bpduguard enable
     mtu 2300
     speed 100000
     duplex full
     no negotiate auto
     service-policy type qos input QOS_MARKING no-stats

7. Save the configuration.

   dbm01sw-rocea0# copy running-config startup-config 
   [########################################] 100%
   Copy complete, now saving to disk (please wait)...
   Copy complete.

8. Repeat Steps 1 to 7 for all of the database servers (bare metal servers or KVM hosts) and all of the storage servers that you want to change to the new VLAN ID.
9. Repeat Steps 1 to 8 for the re1 interfaces, which are connected to the upper leaf switch.