Applying Golden Configuration Settings on Cisco Nexus 9336C-FX2 RoCE Network Fabric Switches

4.7.2 Applying Golden Configuration Settings on Cisco Nexus 9336C-FX2 RoCE Network Fabric Switches

The golden configuration settings are normally applied on the RoCE Network Fabric switches during initial deployment.

Caution:

Take care when performing this procedure, as misconfiguration of the RoCE Network Fabric will likely cause a system outage.
Do not apply the golden configuration settings to an active switch in the RoCE Network Fabric. Applying the golden configuration settings to an active switch may disrupt the RoCE Network Fabric and cause a system outage.

Starting with Oracle Exadata System Software release 20.1.0, you can use the following procedure to apply the golden configuration settings on the RoCE Network Fabric switches. For earlier releases, see Restoring Settings on a Cisco Nexus 9336C-FX2 RoCE Network Fabric Switch.

The following procedure applies the latest golden configuration settings to one or more switches in an Exadata single rack or multi-rack configuration. The switches must be powered on along with one server that has SSH access to the switches for accessing the switch configuration files.

Ensure you have a backup of the current switch configuration for each switch.

For each switch, complete the steps in Backing Up Settings on the Cisco Nexus 9336C-FX2 RoCE Network Fabric Switch.
Log in to a server that has SSH access to the switch, and contains the latest RDMA Network Fabric patch ZIP file.

To find the available RDMA Network Fabric patches, search for 'RDMA network switch' in My Oracle Support document 888828.1. Download and use the latest patch for your Oracle Exadata System Software release.
Unzip the RDMA Network Fabric patch ZIP file and change directories to the location of the patchmgr utility.
Create a switch list file to drive the configuration of the RoCE Network Fabric switches.
1. Create a file that contains the host name or IP address of the switches that you want to configure. Place each switch on a separate line.
  For example, create a file named switches.lst, which contains the host name of each switch on separate lines. On a single rack system, with only two leaf switches, the file might contain switch host name entries like:
```
rack1sw-rocea0
rack1sw-roceb0
```
2. Tag each line to specify the configuration type for each switch.
  To specify the configuration type for each switch, append a colon (:) and tag to each switch host name or IP address in the switch list file. The following tags are supported:
  - leaf - Identifies a leaf switch in a single rack system. This configuration type is assumed if no tag is specified.
  - mspine - Identifies a spine switch. Note that one spine switch configuration supports all spine switches on single and multi-rack systems, with and without Exadata Secure RDMA Fabric Isolation.
  - mleaf - Identifies a leaf switch in a multi-rack X8M system.
  - sfleaf - Identifies a leaf switch in a single rack system that is enabled to support Exadata Secure RDMA Fabric Isolation.
  - msfleaf - Identifies a leaf switch in a multi-rack X8M system that is enabled to support Exadata Secure RDMA Fabric Isolation.
  - leaf23 - Identifies a leaf switch in a single rack system that is configured with 23 host ports. This configuration is required only for 8-socket systems (X8M-8 and later) with 3 database servers and 11 storage servers.
  - mleaf23 - Identifies a leaf switch in a multi-rack system that is configured with 23 host ports. This configuration is required only for 8-socket X8M-8 systems with 3 database servers and 11 storage servers.
  - mleaf_u14 - Identifies a leaf switch in a multi-rack system that is configured with 14 inter-switch links. This is the typical multi-rack leaf switch configuration for X9M and later model systems.
  - msfleaf_u14 - Identifies a leaf switch in a multi-rack system that is enabled to support Exadata Secure RDMA Fabric Isolation and is configured with 14 inter-switch links. This configuration is required for X9M and later model systems with Secure Fabric enabled.
  - mleaf23_u13 - Identifies a leaf switch in a multi-rack system that is configured with 23 host ports and 13 inter-switch links. This configuration is required only for 8-socket X9M-8 systems with three database servers and 11 storage servers.
  For example:
```
rack1sw-rocea0:leaf
rack1sw-roceb0:leaf
```
3. For multi-rack configurations only, specify a unique loopback octet for each switch.
  The loopback octet is the last octet of the switch loopback address, which uniquely identifies a switch.
  
  To specify the loopback octet for each switch, append a period (.) and numeric loopback octet value to each entry in the switch list file.
  
  Caution:
  
  Every switch in a multi-rack configuration must have a unique loopback octet. If multiple switches use the same loopback octet, the RoCE Network Fabric cannot function correctly, resulting in a system outage.
  
  For the leaf switches, start with 101 as the first loopback octet value and increment as follows:
  - 101 - Rack 1 lower leaf switch (rack1sw-rocea0 in the following example)
  - 102 - Rack 1 upper leaf switch (rack1sw-roceb0 in the following example)
  - 103 - Rack 2 lower leaf switch (rack2sw-rocea0 in the following example)
  - 104 - Rack 2 upper leaf switch (rack2sw-roceb0 in the following example)
  - 105 - Rack 3 lower leaf switch
  - 106 - Rack 3 upper leaf switch, and so on.
  For the spine switches, start with 201 as the first loopback octet value and increment as follows:
  - 201 - Rack 1 spine switch (rack1sw-roces0 in the following example)
  - 202 - Rack 2 spine switch (rack2sw-roces0 in the following example)
  - 203 - Rack 3 spine switch
  - 204 - Rack 4 spine switch, and so on.
  For example, the switch list file for a 2-rack Exadata X9M system might contain:
```
rack1sw-rocea0:mleaf_u14.101
rack1sw-roceb0:mleaf_u14.102
rack1sw-roces0:mspine.201
rack2sw-rocea0:mleaf_u14.103
rack2sw-roceb0:mleaf_u14.104
rack2sw-roces0:mspine.202
```
  Or, if you were adding a 5th rack to an existing 4-rack Exadata X9M system the switch list file might contain:
```
rack5sw-rocea0:mleaf_u14.109
rack5sw-roceb0:mleaf_u14.110
rack5sw-roces0:mspine.205
```
Use patchmgr to apply the latest golden configuration settings to the RoCE Network Fabric switches in the switch list file.
For example:
```
# ./patchmgr --roceswitches switches.lst --apply-config –log_dir log-directory
```
See Also:
Patchmgr Syntax for RoCE Network Fabric Switches.
Use patchmgr to verify the configuration of the RoCE Network Fabric switches in the switch list file.
For example:
```
# ./patchmgr --roceswitches switches.lst --verify-config –log_dir log-directory
```

Related Topics

Exadata Database Machine and Exadata Storage Server Supported Versions (My Oracle Support Doc ID 888828.1)

Parent topic: Maintaining the RoCE Network Fabric