4.7.6.1 Labeling the RoCE Network Fabric Links with configure_roce_hostinfo.sh

configure_roce_hostinfo.sh is a utility program that scans the RoCE Network Fabric and labels each RoCE switch port, identifying the hostname, IP address, and port number of the link target. This information makes it easy to understand what each RoCE switch port is connected to and can be used to construct a map of the RoCE Network Fabric.

Starting with Oracle Exadata System Software release 24.1.0, configure_roce_hostinfo.sh is run as part of the initial Exadata system configuration procedure. However, you can run the utility at any time to label (or re-label) the RoCE Network Fabric links.

To explicitly label (or re-label) the RoCE Network Fabric links, you must run configure_roce_hostinfo.sh on an Exadata database server as the root user and provide a file containing the host names for all of the Exadata database servers and Exadata storage servers connected to the RoCE Network Fabric. You must also have SSH equivalency configured across all the servers connected to the RoCE Network Fabric.

Following is an example that includes the output from configure_roce_hostinfo.sh. In this example, the target Exadata system contains 2 database servers, 3 storage servers, and 2 RoCE Network Fabric leaf switches.

[root@dbm01adm01 ~]# cat nodes.lst
dbm01adm01.example.com
dbm01adm02.example.com
dbm01celadm01.example.com
dbm01celadm02.example.com
dbm01celadm03.example.com
[root@dbm01adm01 ~]# /opt/oracle.SupportTools/ibdiagtools/configure_roce_hostinfo.sh nodes.lst
2024-04-12 19:01:08 -0700: Checking all nodes are online...[OK]
2024-04-12 19:01:08 -0700: Checking ssh equivalency on all nodes...[OK]
2024-04-12 19:01:08 -0700: Checking re0/re1 interfaces on all nodes...[OK]
2024-04-12 19:01:12 -0700: Getting lldp_cap data from all nodes...[OK]
2024-04-12 19:01:40 -0700: Checking ssh equivalency on leaf switches...[OK]
2024-04-12 19:01:43 -0700: Collecting lldp information from leaf switches...[OK]
2024-04-12 19:01:45 -0700: Updating all switches concurrently...[OK]
Leaf switch dbm01sw-rocea0.example.com ports have been updated with the following descriptions:
Ethernet1/4: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-roceb0.example.com 192.0.2.132 P4
Ethernet1/5: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-roceb0.example.com 192.0.2.132 P5
Ethernet1/6: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-roceb0.example.com 192.0.2.132 P6
Ethernet1/7: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-roceb0.example.com 192.0.2.132 P7
Ethernet1/23: dbm01adm01.example.com S 192.168.0.1,192.168.0.2 P1
Ethernet1/22: dbm01adm02.example.com S 192.168.0.3,192.168.0.4 P1
Ethernet1/29: dbm01celadm01.example.com C 192.168.0.5,192.168.0.6 P1
Ethernet1/28: dbm01celadm02.example.com C 192.168.0.7,192.168.0.8 P1
Ethernet1/27: dbm01celadm03.example.com C 192.168.0.9,192.168.0.10 P1
Ethernet1/30: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-roceb0.example.com 192.0.2.132 P30
Ethernet1/31: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-roceb0.example.com 192.0.2.132 P31
Ethernet1/32: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-roceb0.example.com 192.0.2.132 P32
Ethernet1/33: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-roceb0.example.com 192.0.2.132 P33
Leaf switch dbm01sw-roceb0.example.com ports have been updated with the following descriptions:
Ethernet1/4: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-rocea0.example.com 192.0.2.131 P4
Ethernet1/5: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-rocea0.example.com 192.0.2.131 P5
Ethernet1/6: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-rocea0.example.com 192.0.2.131 P6
Ethernet1/7: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-rocea0.example.com 192.0.2.131 P7
Ethernet1/23: dbm01adm01.example.com S 192.168.0.1,192.168.0.2 P2
Ethernet1/22: dbm01adm02.example.com S 192.168.0.3,192.168.0.4 P2
Ethernet1/29: dbm01celadm01.example.com C 192.168.0.5,192.168.0.6 P2
Ethernet1/28: dbm01celadm02.example.com C 192.168.0.7,192.168.0.8 P2
Ethernet1/27: dbm01celadm03.example.com C 192.168.0.9,192.168.0.10 P2
Ethernet1/30: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-rocea0.example.com 192.0.2.131 P30
Ethernet1/31: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-rocea0.example.com 192.0.2.131 P31
Ethernet1/32: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-rocea0.example.com 192.0.2.131 P32
Ethernet1/33: Cisco Nexus 9336C-FX2 QSF28 dbm01sw-rocea0.example.com 192.0.2.131 P33
2024-04-12 19:02:12 -0700: SUCCESS: Completed run of configure_roce_hostinfo.sh