1. Owner's Guide
  2. Extending Oracle Zero Data Loss Recovery Appliance
  3. Extending a Rack by Adding Another Rack
  4. Cabling Two Racks Together
  5. Cabling Two RoCE Network Fabric Racks Together with Down Time
  6. Cabling Two RoCE Network Fabric Racks Together with Down Time using Oracle Exadata System Software Release 19.3 or Earlier

Cabling Two RoCE Network Fabric Racks Together with Down Time using Oracle Exadata System Software Release 19.3 or Earlier

Use this procedure to cable together two racks with RoCE Network Fabric where some down-time can be tolerated and you are using Oracle Exadata System Software Release 19.3, or earlier.

In this procedure, the existing rack is R1, and the new rack is R2.

  1. Ensure the new rack is near the existing rack.
    The RDMA Network Fabric cables must be able to reach the servers in each rack.
  2. Ensure you have a backup of the current switch configuration for each switch in the existing and new rack.

    See Backing Up Settings on the RoCE Network Fabric Switch in Oracle Exadata Database Machine Maintenance Guide.

  3. Shut down all servers on both the new rack (R2) and the existing rack (R1).
    The switches should remain available.
  4. Update the firmware to the latest available release on all of the RoCE Network Fabric switches.

    For this step, treat all of the switches as if they belong to a single rack system.

    See Updating RoCE Network Fabric Switch Firmware in Oracle Exadata Database Machine Maintenance Guide.

  5. Apply the multi-rack spine switch configuration to the two spine switches.
    1. Log in to the server that has downloaded the latest RDMA Network Fabric patch ZIP file.

      To find the available RDMA Network Fabric patches, search for 'RDMA network switch' in My Oracle Support document 888828.1. Download and use the latest patch for your Oracle Exadata System Software release.

    2. Unzip the RDMA Network Fabric patch ZIP file and change directories to the location of the patchmgr utility.
    3. Make a copy of the golden configuration file for each switch.

      Run these commands from patch directory:

      # cp roce_switch_templates/roce_spine_switch_multi.cfg roce_spine_switch_multi_R1SS.cfg
# cp roce_switch_templates/roce_spine_switch_multi.cfg roce_spine_switch_multi_R2SS.cfg
    4. Edit each copy of the spine switch configuration file.

      Using a text editor, replace each occurrence of %SPINE_LOOPBACK_IP0% with the correct IP address for the switch, as indicated in the table below.

      Switch SPINE_LOOPBACK_IP0
      Rack 1 spine switch (R1SS) 100.64.0.201
      Rack 2 spine switch (R2SS) 100.64.0.202

      The following example shows using IP address 100.64.0.201 as the SPINE_LOOPBACK_IP0 for the spine switch in rack 1 (R1SS). 

      ! Define loopback interface for underlay OSPF routing
interface loopback0
 description Routing loopback interface
 !ip address 100.64.0.201/32
 ip address 100.64.0.201/32
 ip router ospf UNDERLAY area 0.0.0.0

! Configure OSPF as the underlay network
router ospf UNDERLAY
 router-id 100.64.0.201

! change ECMP hash rotate value from default 32 to 40 for better
! router port utilization for upto parallel flows via the 8
! available router ports
ip load-sharing address source-destination port source-destination rotate 40

! Create BGP route reflector to exchange routes across VTEPs
! Use CIDR block of IPs for neighbor range
! - log-neighbor-changes: Enables the generation of logging messages
! generated when the status of a BGP neighbor changes.
! - address-family ipv4 unicast: Enters address family configuration
! mode and Specifies IP Version 4 unicast address prefixes.
! address
router bgp 65502
 router-id 100.64.0.201
 log-neighbor-changes
    5. Verify the three replacements in the spine switch configuration file.

      For example, for rack 1 (R1SS) check for IP address 100.64.0.201 in the spine switch configuration file:

      $ grep 100.64 roce_spine_switch_multi_R1SS.cfg |grep -v ‘neighbor’ |grep -v ‘!’
 ip address 100.64.0.201/32
 router-id 100.64.0.201
 router-id 100.64.0.201
    6. Apply the updated multi-rack configuration file to its corresponding spine switch.

      1. Log in to each spine switch, and remove the existing configuration file using the following command:

        delete bootflash:roce_spine_switch_multi.cfg

        For example:

        rack1sw-roces0# delete bootflash:roce_spine_switch_multi.cfg
Do you want to delete "/roce_spine_switch_multi.cfg" ? (yes/no/abort) [y] y
rack1sw-roces0#

      2. Log in to the server that contains the modified configuration files, and copy each file to its corresponding spine switch.

        # scp roce_spine_switch_multi_R1SS.cfg admin@R1SS_IP_Address:/
# scp roce_spine_switch_multi_R2SS.cfg admin@R2SS_IP_Address:/

      3. Verify the modified file was copied successfully to the spine switch. For example, for the first rack, log in to the spine switch again and use the following command:

        rack1sw-roces0# dir bootflash:roce_spine_switch_multi_R1SS.cfg
     27360 Nov 20 12:12:50 2019 roce_spine_switch_multi_R1SS.cfg
Usage for bootflash://sup-local
1829572608 bytes used
114893496320 bytes free
116723068928 bytes total

      4. Log in to each switch again, and copy the modified configuration into flash.

        On the spine switch for rack 1, you would use the following commands:

        rack1sw-roces0# run-script bootflash:roce_spine_switch_multi_R1SS.cfg | grep 'none'
rack1sw-roces0# copy running-config startup-config

        On the spine switch for rack 2, you would use the following commands:

        rack2sw-roces0# run-script bootflash:roce_spine_switch_multi_R2SS.cfg | grep 'none'
rack2sw-roces0# copy running-config startup-config

        Note:

        The run-script command for a spine switch can take approximately 2 minutes to complete.
  6. Apply the multi-rack leaf switch configuration to the four leaf switches.

    For each switch, complete the following steps, where SW# represents the values R1LL, R1UL, R2LL, or R2UL, depending on which switch you are configuring.

    1. Log in to the server that has downloaded the RDMA Network Fabric patch ZIP file (from an earlier step) for the Oracle Zero Data Loss Recovery Appliance Software release used by the existing racks.
    2. Change directories to the location of the patchmgr utility.
    3. Make a copy of the golden configuration file for each leaf switch.
      You can copy either the roce_leaf_switch_multi.cfg file or the roce_qinq_leaf_switch_multi.cfg file if you want to enable Exadata Secure RDMA Fabric Isolation.

      After extracting the patch ZIP file, run the following command four times from the patch directory, substituting for SW# the values R1LL, R1UL, R2LL, and R2UL. 

      # cp roce_switch_templates/roce_leaf_switch_multi.cfg roce_leaf_switch_multi_SW#.cfg
    4. Edit each copy of the leaf switch configuration file to replace the loopback IP addresses.

      Using a text editor, replace each occurrence of %LEAF_LOOPBACK_IP0% and %LEAF_LOOPBACK_IP1% with the correct IP addresses for the leaf switch, as indicated in the table below.

      The scheme used for loopback IP addresses for the leaf switches in a 2-rack system is:

      Switch LEAF_LOOPBACK_IP0 LEAF_LOOPBACK_IP1
      Rack 1 Lower Leaf switch (R1LL) 100.64.0.101 100.64.1.101
      Rack 1 Upper Leaf switch (R1UL) 100.64.0.102 100.64.1.102
      Rack 2 Lower Leaf switch (R2LL) 100.64.0.103 100.64.1.103
      Rack 2 Upper Leaf switch (R2UL) 100.64.0.104 100.64.1.104

      The following example shows using IP address 100.64.0.101 as the SPINE_LOOPBACK_IP0 for the lower leaf switch in rack 1 (R1LL). 

      ! Define loopback interface for IGP protocol for VTEP reachability
interface loopback0
 description Routing loopback interface
 !ip address 100.64.0.101/32
 ip address 100.64.0.101/32
 ip router ospf UNDERLAY area 0.0.0.0

! Define loopback interface for associating with local VTEP
interface loopback1
 description VTEP loopback interface
 !ip address 100.64.1.101/32
 ip address 100.64.1.101/32
 ip router ospf UNDERLAY area 0.0.0.0

! Configure OSPF as the underlay network
router ospf UNDERLAY
 router-id 100.64.0.101

! change ECMP hash rotate value from default 32 to 40 for better
! router port utilization for upto parallel flows via the 8
! available router ports
ip load-sharing address source-destination port source-destination rotate 40

! - Create BGP route reflector to exchange routes across VTEPs
! Define max config 8 neighbor spines using their loopback IPs
! - BGP peers are located in an autonomous system (AS) that uses
! 4-byte AS numbers. Cisco recommends to pick a high value such
! as 65502 to avoid conflict with future bgp peers.
! - Create a template ‘BasePolicy’ that defines a peer policy
! template to define attributes for a particular address family.
router bgp 65502
 router-id 100.64.0.101
 log-neighbor-changes
    5. Verify the IP address replacements in each leaf switch configuration file.

      For example, for the first rack check for IP addresses 100.64.0.101 and 100.64.1.101 in the lower leaf switch configuration file (R1LL), and for IP addresses 100.64.0.102 and 100.64.1.102 in the upper leaf switch configuration file (R1UL):

      $ grep 100.64. roce_leaf_switch_multi_R1LL.cfg | grep -v neighbor | grep -v ‘!’
 ip address 100.64.0.101/32
 ip address 100.64.1.101/32
 router-id 100.64.0.101
 router-id 100.64.0.101

$ grep 100.64. roce_leaf_switch_multi_R1UL.cfg | grep -v neighbor | grep -v ‘!’
 ip address 100.64.0.102/32
 ip address 100.64.1.102/32
 router-id 100.64.0.102
 router-id 100.64.0.102
    6. Apply the updated multi-rack configuration files to each corresponding leaf switch.

      1. Log in to each leaf switch, and remove the existing configuration file. For example:

        rack1sw-rocea0# delete bootflash:roce_leaf_switch.cfg
Do you want to delete “/roce_leaf_switch.cfg” ? (yes/no/abort) [y] y

rack1sw-rocea0# delete bootflash:roce_leaf_switch_multi.cfg
No such file or directory
        rack1sw-roceb0# delete bootflash:roce_leaf_switch.cfg
Do you want to delete “/roce_leaf_switch.cfg” ? (yes/no/abort) [y] y

rack1sw-roceb0# delete bootflash:roce_leaf_switch_multi.cfg
No such file or directory

      2. Log in to the server that contains the modified configuration files, and copy each file to its corresponding leaf switch. For example:

        # scp roce_leaf_switch_multi_R1LL.cfg admin@R1LL_IP_Address:/
# scp roce_leaf_switch_multi_R1UL.cfg admin@R1UL_IP_Address:/
# scp roce_leaf_switch_multi_R2LL.cfg admin@R2LL_IP_Address:/
# scp roce_leaf_switch_multi_R2UL.cfg admin@R2UL_IP_Address:/

      3. Log in to each leaf switch and verify that the modified files were copied successfully. For example:

        rack1sw-rocea0# dir bootflash:roce_leaf_switch_multi_R1LL.cfg
    171387 Nov 20 14:41:52 2019 roce_leaf_switch_multi_R1LL.cfg
Usage for bootflash://sup-local
2583580672 bytes used
114139488256 bytes free
116723068928 bytes total

rack1sw-roceb0# dir bootflash:roce_leaf_switch_multi_R1UL.cfg
    171387 Nov 20 21:41:50 2019 roce_leaf_switch_multi_R1UL.cfg
Usage for bootflash://sup-local
2579836928 bytes used
114143232000 bytes free
116723068928 bytes total
        rack2sw-rocea0# dir bootflash:roce_leaf_switch_multi_R2LL.cfg
    171387 Nov 20 14:41:52 2019 roce_leaf_switch_multi_R2LL.cfg
Usage for bootflash://sup-local
2583580672 bytes used
114139488256 bytes free
116723068928 bytes total

rack2sw-roceb0# dir bootflash:roce_leaf_switch_multi_R2UL.cfg
    171387 Nov 20 21:41:50 2019 roce_leaf_switch_multi_R2UL.cfg
Usage for bootflash://sup-local
2579836928 bytes used
114143232000 bytes free
116723068928 bytes total

      4. Copy the modified configuration file into flash. For example:

        rack1sw-rocea0# run-script bootflash:roce_leaf_switch_multi_R1LL.cfg | grep 'none'
rack1sw-rocea0# copy running-config startup-config
        rack1sw-roceb0# run-script bootflash:roce_leaf_switch_multi_R1UL.cfg | grep 'none'
rack1sw-roceb0# copy running-config startup-config
        rack2sw-rocea0# run-script bootflash:roce_leaf_switch_multi_R2LL.cfg | grep 'none'
rack2sw-rocea0# copy running-config startup-config
        rack2sw-roceb0# run-script bootflash:roce_leaf_switch_multi_R2UL.cfg | grep 'none'
rack2sw-roceb0# copy running-config startup-config

        Note:

        The run-script command for a leaf switch can take approximately 6 minutes to complete.
  7. Use patchmgr to verify the configuration of the RDMA Network Fabric switches against the golden configuration files.
    1. Log in to the server that has downloaded the RDMA Network Fabric patch ZIP file (from earlier step ).
    2. Change directories to the location of the patchmgr utility.
    3. Create a file that contains the host name or IP address of the leaf and spine switches on both racks.
      For example, create a file name switches.lst. The file must contain the host name or IP address for the 2 spine switches and 4 leaf switches, with each switch on a new line.
    4. Run patchmgr with the --verify_config option.

      In the following command, switches.lst is a file that contains the switches to be queried, one per line. 

      $ ./patchmgr --roceswitches switches.lst --verify-config

2019-11-20 14:12:27 -0800 :Working: Initiate config verify on RoCE switches from . Expect up to 6 minutes for each switch
                                                   

2019-11-20 14:12:30 -0800 1 of 15 :Verifying config on switch rack1sw-rocea0

2019-11-20 14:12:30 -0800: [INFO ] Dumping current running config locally as file: /tmp/run.rack1sw-rocea0.cfg
2019-11-20 14:12:33 -0800: [SUCCESS ] Backed up switch config successfully
2019-11-20 14:12:33 -0800: [INFO ] Validating running config against template [1/3]: /tmp/patch_switch_19.3.1.0.0.191018/roce_switch_templates/roce_leaf_switch.cfg
2019-11-20 14:12:33 -0800: [INFO ] Validating running config against template [2/3]: /tmp/patch_switch_19.3.1.0.0.191018/roce_switch_templates/roce_leaf_switch_multi.cfg
2019-11-20 14:12:33 -0800: [INFO ] Config matches template: /tmp/patch_switch_19.3.1.0.0.191018/roce_switch_templates/roce_leaf_switch_multi.cfg
2019-11-20 14:12:33 -0800: [SUCCESS ] Config validation successful!


2019-11-20 14:12:33 -0800 2 of 15 :Verifying config on switch rack1sw-roceb0
...
  8. Perform the physical cabling of the switches.
    1. In Rack 2, remove the eight existing inter-switch connections between the two leaf switches, R2UL and R2LL.
    2. In Rack 2, cable each leaf switch using the tables in Two-Rack Cabling for X8M Racks.
    3. In Rack 1, remove the eight existing inter-switch connections between the two leaf switches, R1UL and R1LL.
    4. In Rack 1, cable each leaf switch using the tables in Two-Rack Cabling for X8M Racks.
  9. Confirm each switch is available and connected.

    For each of the 6 switches, confirm the output from the show interface status command shows connected and 100G. In the following examples, the leaf switches are ports Eth1/4 to Eth1/7, and Eth1/30 to Eth1/33. The spine switches are ports Eth1/5 to Eth1/20.

    When run from a spine switch, the output should be similar to the following:

    rack1sw-roces0# show interface status
--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full    1000    -- 
--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
...
Eth1/5        RouterPort5        connected routed    full    100G    QSFP-100G-CR4
Eth1/6        RouterPort6        connected routed    full    100G    QSFP-100G-SR4
Eth1/7        RouterPort7        connected routed    full    100G    QSFP-100G-CR4
Eth1/8        RouterPort8        connected routed    full    100G    QSFP-100G-SR4
Eth1/9        RouterPort9        connected routed    full    100G    QSFP-100G-CR4
Eth1/10       RouterPort10       connected routed    full    100G    QSFP-100G-SR4
Eth1/11       RouterPort11       connected routed    full    100G    QSFP-100G-CR4
Eth1/12       RouterPort12       connected routed    full    100G    QSFP-100G-SR4
Eth1/13       RouterPort13       connected routed    full    100G    QSFP-100G-CR4
Eth1/14       RouterPort14       connected routed    full    100G    QSFP-100G-SR4
Eth1/15       RouterPort15       connected routed    full    100G    QSFP-100G-CR4
Eth1/16       RouterPort16       connected routed    full    100G    QSFP-100G-SR4
Eth1/17       RouterPort17       connected routed    full    100G    QSFP-100G-CR4
Eth1/18       RouterPort18       connected routed    full    100G    QSFP-100G-SR4
Eth1/19       RouterPort19       connected routed    full    100G    QSFP-100G-CR4
Eth1/20       RouterPort20       connected routed    full    100G    QSFP-100G-SR4
Eth1/21       RouterPort21       xcvrAbsen      routed    full    100G    --
...

    When run from a leaf switch, the output should be similar to the following:

    rack1sw-rocea0# show interface status
--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full    1000    -- 
--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
...
Eth1/4        RouterPort1        connected routed    full    100G    QSFP-100G-CR4
Eth1/5        RouterPort2        connected routed    full    100G    QSFP-100G-CR4
Eth1/6        RouterPort3        connected routed    full    100G    QSFP-100G-CR4
Eth1/7        RouterPort4        connected routed    full    100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full    100G    QSFP-100G-CR4
...
Eth1/29       celadm01           connected 3888      full    100G    QSFP-100G-CR4
Eth1/30       RouterPort5        connected routed    full    100G    QSFP-100G-SR4
Eth1/31       RouterPort6        connected routed    full    100G    QSFP-100G-SR4
Eth1/32       RouterPort7        connected routed    full    100G    QSFP-100G-SR4
Eth1/33       RouterPort8        connected routed    full    100G    QSFP-100G-SR4
...
  10. Check the neighbor discovery for every switch in racks R1 and R2.

    Log in to each switch and use the show lldp neighbors command. Make sure that all switches are visible and check the switch ports assignment (leaf switches: ports Eth1/4 - Eth1/7, Eth1/30 - Eth1/33; spine switches: ports Eth1/5 - Eth1/20) against the tables in Two-Rack Cabling for X8M Racks.

    A spine switch should see the two leaf switches in each rack, but not the other spine switch. The output for a spine switch should be similar to the following:

    Note:

    The interfaces output in the Port ID column are different for each switch based on the applicable cabling tables. 
    rack1sw-roces0# show lldp neighbors
...
Device ID            Local Intf      Hold-time  Capability  Port ID
rack1-adm0           mgmt0           120        BR          Ethernet1/47
rack1sw-roceb0       Eth1/5     120        BR          Ethernet1/5
rack2sw-roceb0       Eth1/6     120        BR          Ethernet1/5
rack1sw-roceb0       Eth1/7     120        BR          Ethernet1/7
rack2sw-roceb0       Eth1/8     120        BR          Ethernet1/7
rack1sw-roceb0       Eth1/9     120        BR          Ethernet1/4
rack2sw-roceb0       Eth1/10    120        BR          Ethernet1/4
rack1sw-roceb0       Eth1/11    120        BR          Ethernet1/6
rack2sw-roceb0       Eth1/12    120        BR          Ethernet1/6
rack1sw-rocea0       Eth1/13    120        BR          Ethernet1/5
rack2sw-rocea0       Eth1/14    120        BR          Ethernet1/5
rack1sw-rocea0       Eth1/15    120        BR          Ethernet1/7
rack2sw-rocea0       Eth1/16    120        BR          Ethernet1/7
rack1sw-rocea0       Eth1/17    120        BR          Ethernet1/4
rack2sw-rocea0       Eth1/18    120        BR          Ethernet1/4
rack1sw-rocea0       Eth1/19    120        BR          Ethernet1/6 
rack2sw-rocea0       Eth1/20    120        BR          Ethernet1/6
Total entries displayed: 17

    Each leaf switch should see the two spine switches, but not the other leaf switches. The output for a leaf switch should be similar to the following:

    Note:

    The interfaces output in the Port ID column are different for each switch based on the applicable cabling tables. 
    rack1sw-rocea0# show lldp neighbors
...
Device ID            Local Intf      Hold-time  Capability  Port ID
switch               mgmt0      120        BR          Ethernet1/46
rack1sw-roces0       Eth1/4     120        BR          Ethernet1/17
rack1sw-roces0       Eth1/5     120        BR          Ethernet1/13
rack1sw-roces0       Eth1/6     120        BR          Ethernet1/19
rack1sw-roces0       Eth1/7     120        BR          Ethernet1/15
rack2sw-roces0       Eth1/30    120        BR          Ethernet1/17
rack2sw-roces0       Eth1/31    120        BR          Ethernet1/13
rack2sw-roces0       Eth1/32    120        BR          Ethernet1/19
rack2sw-roces0       Eth1/33    120        BR          Ethernet1/15
rocetoi-ext-sw       Eth1/36    120        BR          Ethernet1/49
Total entries displayed: 10
  11. Power on all servers in racks R1 and R2.
  12. For each rack, confirm the multi-rack cabling by running the verify_roce_cables.py script.

    Refer to My Oracle Support Doc ID 2587717.1 for download and usage instructions.

    Check the output of the verify_roce_cables.py script against the tables in Two-Rack Cabling for X8M Racks. Also, check that output in the CABLE OK? columns contains the OK status.

    When running the script, two input files are used, one for nodes and one for switches. Each file should contain the servers or switches on separate lines. Use fully qualified domain names or IP addresses for each server and switch.

    The following output is a partial example of the command results:

    # ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
SWITCH PORT (EXPECTED PEER)  LEAF-1 (rack1sw-rocea0)     : CABLE OK?  LEAF-2 (rack1sw-roceb0)    : CABLE OK?
----------- --------------   --------------------------- : --------   -----------------------    : ---------
Eth1/4 (ISL peer switch)   : rack1sw-roces0 Ethernet1/17 : OK         rack1sw-roces0 Ethernet1/9 : OK
Eth1/5 (ISL peer switch)   : rack1sw-roces0 Ethernet1/13 : OK         rack1sw-roces0 Ethernet1/5 : OK
Eth1/6 (ISL peer switch)   : rack1sw-roces0 Ethernet1/19 : OK         rack1sw-roces0 Ethernet1/11: OK
Eth1/7 (ISL peer switch)   : rack1sw-roces0 Ethernet1/15 : OK         rack1sw-roces0 Ethernet1/7 : OK
Eth1/12 (celadm10)         : rack1celadm10 port-1        : OK         rack1celadm10 port-2       : OK
Eth1/13 (celadm09)         : rack1celadm09 port-1        : OK         rack1celadm09 port-2       : OK
Eth1/14 (celadm08)         : rack1celadm08 port-1        : OK         rack1celadm08 port-2       : OK
...
Eth1/15 (adm08)            : rack1dbadm08 port-1         : OK         rack1dbadm08 port-2        : OK
Eth1/16 (adm07)            : rack1dbadm07 port-1         : OK         rack1dbadm07 port-2        : OK
Eth1/17 (adm06)            : rack1dbadm06 port-1         : OK         rack1dbadm06 port-2        : OK
...
Eth1/30 (ISL peer switch)  : rack2sw-roces0 Ethernet1/17 : OK         rack2sw-roces0 Ethernet1/9 : OK
Eth1/31 (ISL peer switch)  : rack2sw-roces0 Ethernet1/13 : OK         rack2sw-roces0 Ethernet1/5 : OK
Eth1/32 (ISL peer switch)  : rack2sw-roces0 Ethernet1/19 : OK         rack2sw-roces0 Ethernet1/11: OK
Eth1/33 (ISL peer switch)  : rack2sw-roces0 Ethernet1/15 : OK         rack2sw-roces0 Ethernet1/7 : OK

    # ./verify_roce_cables.py -n nodes.rack2 -s switches.rack2
SWITCH PORT (EXPECTED PEER)  LEAF-1 (rack2sw-rocea0)     : CABLE OK?  LEAF-2 (rack2sw-roceb0)    : CABLE OK?
----------- --------------   --------------------------- : --------   -----------------------    : ---------
Eth1/4 (ISL peer switch)  :  rack1sw-roces0 Ethernet1/18 : OK         rack1sw-roces0 Ethernet1/10: OK
...
  13. Verify the RoCE Network Fabric operation by using the infinicheck command.

    Use the following recommended command sequence. In each command, hosts.lst is the name of an input file that contains a comma-delimited list of database server host names or RoCE Network Fabric IP addresses, and cells.lst is the name of an input file that contains a list of RoCE Network Fabric IP addresses for the storage servers.

    • Use infinicheck with the -z option to clear the files that were created during the last run of the infinicheck command. For example: 

      # /opt/oracle.SupportTools/ibdiagtools/infinicheck -g hosts.lst -c cells.lst -z

    • Use infinicheck with the -s option to set up user equivalence for password-less SSH across the RoCE Network Fabric. For example: 

      # /opt/oracle.SupportTools/ibdiagtools/infinicheck -g hosts.lst -c cells.lst -s

    • Finally, verify the RoCE Network Fabric operation by using infinicheck with the -b option, which is recommended on newly imaged machines where it is acceptable to suppress the cellip.ora and cellinit.ora configuration checks. For example: 

      # /opt/oracle.SupportTools/ibdiagtools/infinicheck -g hosts.lst -c cells.lst -b

INFINICHECK                    
        [Network Connectivity, Configuration and Performance]        
               
          ####  FABRIC TYPE TESTS  #### 
System type identified: RoCE
Verifying User Equivalance of user=root from all DBs to all CELLs.
     ####  RoCE CONFIGURATION TESTS  ####       
     Checking for presence of RoCE devices on all DBs and CELLs 
[SUCCESS].... RoCE devices on all DBs and CELLs look good
     Checking for RoCE Policy Routing settings on all DBs and CELLs 
[SUCCESS].... RoCE Policy Routing settings look good
     Checking for RoCE DSCP ToS mapping on all DBs and CELLs 
[SUCCESS].... RoCE DSCP ToS settings look good
     Checking for RoCE PFC settings and DSCP mapping on all DBs and CELLs
[SUCCESS].... RoCE PFC and DSCP settings look good
     Checking for RoCE interface MTU settings. Expected value : 2300
[SUCCESS].... RoCE interface MTU settings look good
     Verifying switch advertised DSCP on all DBs and CELLs ports ( )
[SUCCESS].... Advertised DSCP settings from RoCE switch looks good  
    ####  CONNECTIVITY TESTS  ####
    [COMPUTE NODES -> STORAGE CELLS] 
      (60 seconds approx.)       
    (Will walk through QoS values: 0-6) [SUCCESS]..........Results OK
[SUCCESS]....... All  can talk to all storage cells          
    [COMPUTE NODES -> COMPUTE NODES]               
...