Configuring the Cisco Nexus 9336C-FX2 Switch

6.2.1 Configuring the Cisco Nexus 9336C-FX2 Switch

The RoCE Network Fabric switch supplied with the engineered system rack is minimally configured during installation.

During initial system configuration, you can reset and configure the switch.

Connect from the RoCE Network Fabric switch serial console to a laptop or similar device using the available RJ45 cable.
Ensure the terminal session is recorded on the laptop by logging the output.
The output can be used as a reference that the switch has been configured correctly.
Power on the switch.
Log in as the admin user.
```
User Access Verification
dbm0sw-rocea0 login: admin
Password: ********
```
Note:
If you do not have the password for the admin user, then contact Oracle Support Services.

Erase the existing configuration.

dbm0sw-rocea0# write erase

Warning: This command will erase the startup-configuration.

Do you wish to proceed anyway? (y/n)  [n] y

Restart the system so you can perform the automated setup.

dbm0sw-rocea0# reload

This command will reboot the system. (y/n)?  [n] y

2017 Aug 31 01:09:00 dbm0sw-rocea0 %$ VDC-1 %$ %PLATFORM-2-PFM_SYSTEM_RESET: Manual system restart from Command Line Interface


CISCO SWITCH Ver7.59
Device detected on 0:1:2 after 0 msecs  
...

Switch to normal setup and, when asked if you want to enforce secure password standard, enter no, then enter a new password for the admin user.

Running S93thirdparty-script...

Populating conf files for hybrid sysmgr ...
Starting hybrid sysmgr ...
inserting /isan/lib/modules/klm_cisco_nb.o ... done

Abort Auto Provisioning and continue with normal setup ? (yes/no) [n]: yes

         ---- System Admin Account Setup ----

Do you want to enforce secure password standard (yes/no) [y]: no

  Enter the password for "admin": 
  Confirm the password for "admin":

When the Basic System Configuration Dialog appears, choose to enter the basic configuration dialog.

        ---- Basic System Configuration Dialog VDC: 1 ----

This setup utility will guide you through the basic configuration of
the system. Setup configures only enough connectivity for management
of the system.

Please register Cisco Nexus9000 Family devices promptly with your
supplier. Failure to register may affect response times for initial
service calls. Nexus9000 devices must be registered to receive 
entitled support services.

Press Enter at anytime to skip a dialog. Use ctrl-c at anytime
to skip the remaining dialogs.

Would you like to enter the basic configuration dialog (yes/no): yes

In the basic configuration, you can use the default inputs until asked to enter the switch name.

In this example, the switch has a name of test123sw-rocea0.

  Create another login account (yes/no) [n]: 
  Configure read-only SNMP community string (yes/no) [n]: 
  Configure read-write SNMP community string (yes/no) [n]: 
  Enter the switch name : test123sw-rocea0

Respond yes when asked to configure Out-of-band management configuration, and specify appropriate network addresses when prompted.

Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: yes
     Mgmt0 IPv4 address : 100.104.10.21
     Mgmt0 IPv4 netmask : 255.255.248.0
  Configure the default gateway? (yes/no) [y]:
     IPv4 address of the default gateway : 100.104.10.1

Respond yes when asked to configure advanced IP options.
```
Configure advanced IP options? (yes/no) [n]: yes
```
Respond yes when asked to configure static route (this can be changed later).
```
Configure static route? (yes/no) [n]: yes
```

Enter the destination prefix and mask, and other values as prompted.

   Destination prefix : 10.100.100.0

   Destination prefix mask : 255.255.255.0

   Next hop IPv4 address : 10.100.100.1

Configure the DNS IPv4 addresses.

Configure the DNS IPv4 address? (yes/no) [n]: yes
   DNS IP address: 10.100.100.2

Skip configuring the default domain name (this will be configured later).
```
Configure the default domain name? (yes/no) [n]: no
```

Accept the default responses until asked to configure SSH and the NTP server.

Enable the telnet service? (yes/no) [n]: no
Enable the ssh service? (yes/no) [y]: yes
   Type of ssh key you would like to generate (dsa/rsa) [rsa]: rsa
   Number of rsa key bits <1024-2048> [1024]: 1024
 
Configure the NTP server? (yes/no) [n]: yes
     NTP server IPv4 address : 10.100.100.3

Accept the default responses until asked to specify the CoPP system profile. Enter strict.

 Configure default interface layer (L3/L2) [L2]: 
 Configure default switchport interface state (shut/noshut) [noshut]: 
 Configure CoPP system profile (strict/moderate/lenient/dense) [strict]: strict

After reviewing the configuration, save the configuration.

The following configuration will be applied:
   no password strength-check
   switchname test123sw-rocea0
   ip route 100.104.8.0 255.255.248.0 100.104.10.1
   vrf context management
   ip route 0.0.0.0/0 100.104.10.1
   exit
    no feature telnet
    ssh key rsa 1024 force
    feature ssh
    ntp server 100.104.10.1
    system default switchport
    no system default switchport shutdown
    copp profile strict
   interface mgmt0
   ip address 100.104.10.21 255.255.248.0
   no shutdown

Would you like to edit the configuration? (yes/no) [n]: 

Use this configuration and save it? (yes/no) [y]: yes

[########################################] 100%
Copy complete.

Enable the scp server feature on the switch.
```
test123sw-rocea0# feature scp-server
```

Save the running configuration to flash.

test123sw-rocea0# copy running-config startup-config
[########################################] 100%
Copy complete.

Apply the golden configuration on the switch.
- Starting with Oracle Exadata System Software release 20.1.0, use the procedure described in Applying Golden Configuration Settings on RoCE Network Fabric Switches, in Oracle Exadata Database Machine Maintenance Guide.
- Otherwise, use the following procedure to apply the golden configuration on the switch:
1. Delete the configuration file on the switch for the target configuration.
  
  Note:
  If you do not remove the file you are replacing, then when you attempt to overwrite the file you will get a 'permission denied' error.
  Log in to the switch, enter configuration mode, then run a command similar to the following:
```
test123sw-rocea0# delete bootflash:roce_leaf_switch.cfg
Do you want to delete "/roce_leaf_switch.cfg" ? (yes/no/abort) [y] y
test123sw-rocea0# 
```
2. Log in to a server that has SSH access to the switch, and contains the latest RDMA Network Fabric patch ZIP file.
  
  To find the available RDMA Network Fabric patches, search for 'RDMA network switch' in My Oracle Support document 888828.1. Download and use the latest patch for your Oracle Exadata System Software release.
3. Unzip the RDMA Network Fabric patch ZIP file and change directories to the location of the patchmgr utility.
4. Locate the golden configuration files in the RDMA Network Fabric patch bundle.
  The files are located within the roce_switch_templates directory.
  
  The golden configuration files are as follows:
  - Single rack leaf (leaf): roce_leaf_switch.cfg
  - Multi-rack spine (mspine): roce_spine_switch_multi.cfg
  - Multi-rack leaf (mleaf): roce_leaf_switch_multi.cfg
  - Single rack leaf with Secure Fabric support (sfleaf): roce_sf_leaf_switch.cfg
  - Multi-rack leaf with Secure Fabric support (msfleaf): roce_sf_leaf_switch_multi.cfg
  - Single rack leaf configured with 23 host ports (leaf23): roce_leaf_switch_23hosts.cfg
  - Multi-rack leaf configured with 23 host ports (mleaf23): roce_leaf_switch_23hosts_multi.cfg
  - Multi-rack leaf configured with 14 inter-switch links (mleaf_u14): roce_leaf_switch_14uplinks_multi.cfg
  - Multi-rack leaf configured with 14 inter-switch links and with Secure Fabric support (msfleaf_u14): roce_sf_leaf_switch_14uplinks_multi.cfg
  - Multi-rack leaf configured with 23 host ports and 13 inter-switch links (mleaf23_u13): roce_leaf_switch_23hosts_13uplinks_multi.cfg
5. Copy the golden configuration file to the switch.
  In the following example, 100.104.10.21 represents the IP address of the switch you are configuring.
```
# scp roce_leaf_switch.cfg admin@100.104.10.21:/
User Access Verification
Password:
roce_leaf_switch.cfg 100% 23KB 23.5KB/s 00:00
```
6. Apply the golden configuration file on the switch.
  Use the run-script command while connected directly to the switch.
```
test123sw-rocea0# run-script bootflash:roce_leaf_switch.cfg | grep 'none'
```
  Note:
  This command may take up to 1-2 minutes on a single-rack switch and up to 3-4 minutes on a multi-rack switch.
7. Verify the switch configuration.
  Use the patchmgr utility on the server that has SSH access to the switch, and contains the latest RDMA Network Fabric patch bundle.
  In the following command, roceswitch.lst is a file that contains the switch host name or IP address.
```
# ./patchmgr --roceswitches roceswitch.lst --verify-config
```
Backup up the switch configuration.

Follow the steps in Backing Up Settings on the ROCE Switch, in Oracle Exadata Database Machine Maintenance Guide.
Optional: Set the clock, using the same procedure as in Setting the Clock on the Cisco 93108-1G or 9348 Ethernet Switch.

Related Topics

Exadata Database Machine and Exadata Storage Server Supported Versions (My Oracle Support Doc ID 888828.1)

Parent topic: Configuring the RDMA Network Fabric Switch