- Owner's Guide
- Installation and Configuration
- Configuring a Recovery Appliance Rack
- Configuring the RDMA Network Fabric Switch
- Configuring the Cisco Nexus 9336C-FX2 Switch
Configuring the Cisco Nexus 9336C-FX2 Switch
The RoCE Network Fabric switch supplied with the engineered system rack is minimally configured during installation.
During initial system configuration, you can reset and configure the switch.
- Connect from the RoCE Network Fabric switch serial console to a laptop or similar device using the available RJ45 cable.
- Ensure the terminal session is recorded on the laptop by logging the output.The output can be used as a reference that the switch has been configured correctly.
- Power on the switch.
- Log in as the
admin
user.User Access Verification dbm0sw-rocea0 login: admin Password: ********
Note:
If you do not have the password for theadmin
user, then contact Oracle Support Services. - Erase the existing configuration.
dbm0sw-rocea0# write erase Warning: This command will erase the startup-configuration. Do you wish to proceed anyway? (y/n) [n] y
- Restart the system so you can perform the automated setup.
dbm0sw-rocea0# reload This command will reboot the system. (y/n)? [n] y 2017 Aug 31 01:09:00 dbm0sw-rocea0 %$ VDC-1 %$ %PLATFORM-2-PFM_SYSTEM_RESET: Manual system restart from Command Line Interface CISCO SWITCH Ver7.59 Device detected on 0:1:2 after 0 msecs ...
- Switch to normal setup and, when asked if you want to enforce secure password standard, enter
no
, then enter a new password for theadmin
user.Running S93thirdparty-script... Populating conf files for hybrid sysmgr ... Starting hybrid sysmgr ... inserting /isan/lib/modules/klm_cisco_nb.o ... done Abort Auto Provisioning and continue with normal setup ? (yes/no) [n]: yes ---- System Admin Account Setup ---- Do you want to enforce secure password standard (yes/no) [y]: no Enter the password for "admin": Confirm the password for "admin":
- When the Basic System Configuration Dialog appears, choose to enter the basic configuration dialog.
---- Basic System Configuration Dialog VDC: 1 ---- This setup utility will guide you through the basic configuration of the system. Setup configures only enough connectivity for management of the system. Please register Cisco Nexus9000 Family devices promptly with your supplier. Failure to register may affect response times for initial service calls. Nexus9000 devices must be registered to receive entitled support services. Press Enter at anytime to skip a dialog. Use ctrl-c at anytime to skip the remaining dialogs. Would you like to enter the basic configuration dialog (yes/no): yes
- In the basic configuration, you can use the default inputs until asked to enter the switch name.
In this example, the switch has a name of
test123sw-rocea0
.Create another login account (yes/no) [n]: Configure read-only SNMP community string (yes/no) [n]: Configure read-write SNMP community string (yes/no) [n]: Enter the switch name : test123sw-rocea0
- Respond
yes
when asked to configure Out-of-band management configuration, and specify appropriate network addresses when prompted.Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: yes Mgmt0 IPv4 address : 100.104.10.21 Mgmt0 IPv4 netmask : 255.255.248.0 Configure the default gateway? (yes/no) [y]: IPv4 address of the default gateway : 100.104.10.1
- Respond
yes
when asked to configure advanced IP options.Configure advanced IP options? (yes/no) [n]: yes
- Respond
yes
when asked to configure static route (this can be changed later).Configure static route? (yes/no) [n]: yes
- Enter the destination prefix and mask, and other values as prompted.
Destination prefix : 10.100.100.0 Destination prefix mask : 255.255.255.0 Next hop IPv4 address : 10.100.100.1
- Configure the DNS IPv4 addresses.
Configure the DNS IPv4 address? (yes/no) [n]: yes DNS IP address: 10.100.100.2
- Skip configuring the default domain name (this will be configured later).
Configure the default domain name? (yes/no) [n]: no
- Accept the default responses until asked to configure SSH and the NTP server.
Enable the telnet service? (yes/no) [n]: no Enable the ssh service? (yes/no) [y]: yes Type of ssh key you would like to generate (dsa/rsa) [rsa]: rsa Number of rsa key bits <1024-2048> [1024]: 1024 Configure the NTP server? (yes/no) [n]: yes NTP server IPv4 address : 10.100.100.3
- Accept the default responses until asked to specify the CoPP system profile. Enter
strict
.Configure default interface layer (L3/L2) [L2]: Configure default switchport interface state (shut/noshut) [noshut]: Configure CoPP system profile (strict/moderate/lenient/dense) [strict]: strict
- After reviewing the configuration, save the configuration.
The following configuration will be applied: no password strength-check switchname test123sw-rocea0 ip route 100.104.8.0 255.255.248.0 100.104.10.1 vrf context management ip route 0.0.0.0/0 100.104.10.1 exit no feature telnet ssh key rsa 1024 force feature ssh ntp server 100.104.10.1 system default switchport no system default switchport shutdown copp profile strict interface mgmt0 ip address 100.104.10.21 255.255.248.0 no shutdown Would you like to edit the configuration? (yes/no) [n]: Use this configuration and save it? (yes/no) [y]: yes [########################################] 100% Copy complete.
- Enable the scp server feature on the switch.
test123sw-rocea0# feature scp-server
- Save the running configuration to flash.
test123sw-rocea0# copy running-config startup-config [########################################] 100% Copy complete.
- Apply the golden configuration on the switch.
-
Starting with Oracle Exadata System Software release 20.1.0, use the procedure described in Applying Golden Configuration Settings on RoCE Network Fabric Switches, in Oracle Exadata Database Machine Maintenance Guide.
- Otherwise, use the following procedure to apply the golden configuration on the switch:
- Delete the configuration file on the switch for the target
configuration.
Note:
If you do not remove the file you are replacing, then when you attempt to overwrite the file you will get a 'permission denied' error.Log in to the switch, enter configuration mode, then run a command similar to the following:
test123sw-rocea0# delete bootflash:roce_leaf_switch.cfg Do you want to delete "/roce_leaf_switch.cfg" ? (yes/no/abort) [y] y test123sw-rocea0#
- Log in to a server that has SSH access to the switch, and contains the latest
RDMA Network Fabric patch ZIP file.
To find the available RDMA Network Fabric patches, search for 'RDMA network switch' in My Oracle Support document 888828.1. Download and use the latest patch for your Oracle Exadata System Software release.
- Unzip the RDMA Network Fabric patch ZIP file
and change directories to the location of the
patchmgr
utility. - Locate the golden configuration files in the RDMA Network Fabric patch bundle.
The files are located within the
roce_switch_templates
directory.The golden configuration files are as follows:
- Single rack leaf (
leaf
):roce_leaf_switch.cfg
- Multi-rack spine (
mspine
):roce_spine_switch_multi.cfg
- Multi-rack leaf (
mleaf
):roce_leaf_switch_multi.cfg
- Single rack leaf with Secure Fabric
support (
sfleaf
):roce_sf_leaf_switch.cfg
- Multi-rack leaf with Secure Fabric
support (
msfleaf
):roce_sf_leaf_switch_multi.cfg
- Single rack leaf configured with 23 host ports
(
leaf23
):roce_leaf_switch_23hosts.cfg
- Multi-rack leaf configured with 23 host ports
(
mleaf23
):roce_leaf_switch_23hosts_multi.cfg
- Multi-rack leaf configured with 14 inter-switch links
(
mleaf_u14
):roce_leaf_switch_14uplinks_multi.cfg
- Multi-rack leaf configured with 14 inter-switch links and with Secure Fabric support
(
msfleaf_u14
):roce_sf_leaf_switch_14uplinks_multi.cfg
- Multi-rack leaf configured with 23 host ports and 13 inter-switch
links (
mleaf23_u13
):roce_leaf_switch_23hosts_13uplinks_multi.cfg
- Single rack leaf (
- Copy the golden configuration file to the switch.
In the following example,
100.104.10.21
represents the IP address of the switch you are configuring.# scp roce_leaf_switch.cfg admin@100.104.10.21:/ User Access Verification Password: roce_leaf_switch.cfg 100% 23KB 23.5KB/s 00:00
- Apply the golden configuration file on the switch.Use the
run-script
command while connected directly to the switch.test123sw-rocea0# run-script bootflash:roce_leaf_switch.cfg | grep 'none'
Note:
This command may take up to 1-2 minutes on a single-rack switch and up to 3-4 minutes on a multi-rack switch. - Verify the switch configuration.Use the
patchmgr
utility on the server that has SSH access to the switch, and contains the latest RDMA Network Fabric patch bundle.In the following command,
roceswitch.lst
is a file that contains the switch host name or IP address.# ./patchmgr --roceswitches roceswitch.lst --verify-config
-
- Backup up the switch configuration.
Follow the steps in Backing Up Settings on the ROCE Switch, in Oracle Exadata Database Machine Maintenance Guide.
- Optional: Set the clock, using the same procedure as in Setting the Clock on the Cisco 93108-1G or 9348 Ethernet Switch.