1. Extending and Multi-Rack Cabling Guide
  2. Extending the Hardware

2 Extending the Hardware

You can extend Oracle Exadata Database Machine by adding database servers and storage servers within a rack. You can also cable together multiple racks.

All new equipment receives a Customer Support Identifier (CSI). Any new equipment for your Oracle Exadata Rack has a new CSI. Contact Oracle Support Services to reconcile the new CSI with the existing Oracle Exadata Rack CSI. Have the original instance numbers or serial numbers available, as well as the new numbers when contacting Oracle Support Services.

2.1 Extending an Eighth Rack

The process of extending an Oracle Exadata Database Machine Eighth Rack system is principally the same as extending any other Exadata rack. At a high level, the process involves the following stages:

  1. Verify the preexisting Eighth Rack system configuration.

  2. Perform any required hardware installation.

    During this stage, you may need to shut down servers to install some hardware components. Where required, servers would be shut down in a rolling manner to preserve overall system availability.

  3. Configure the additional hardware resources.

    During this stage:

    • New database server resources are added to a database cluster.

    • New storage server resources are used to create additional grid disks, which can be used to extend existing Oracle ASM disk groups.

The following sections contain additional information and procedures specific to Oracle Exadata Database Machine Eighth Rack:

  • Eighth Rack Expansion Options
    This section outlines the available expansion options for different Oracle Exadata Database Machine Eighth Rack models.
  • Eighth Rack Expansion Procedures
    This section contains specific procedures that may be required for expanding some Oracle Exadata Database Machine Eighth Rack models.

Parent topic: Extending the Hardware

2.1.1 Eighth Rack Expansion Options

This section outlines the available expansion options for different Oracle Exadata Database Machine Eighth Rack models.

Parent topic: Extending an Eighth Rack

2.1.1.1 X9M-2 Eighth Rack Upgrade Options

Upgrade of Oracle Exadata Database Machine X9M-2 Eighth Rack systems requires hardware modification. Compared with non-Eighth Rack systems, Eighth Rack database servers have one CPU, and less memory, and Eighth Rack storage servers have fewer CPU cores, less memory, less disk storage, and less flash storage.

You can upgrade Oracle Exadata Database Machine X9M-2 Eighth Rack systems as follows:

  • Add a CPU upgrade kit to the database servers, which adds 1 CPU to each database server.

    As part of adding the CPU upgrade kit, you must also:

    1. Physically move the network interface card from PCIe slot 2 to PCIe slot 1.

    2. Run /opt/oracle.cellos/ipconf.pl to reconfigure the OS network configuration files.

  • Add a memory upgrade kit to the database servers, which adds 4 x 32 GB DIMMs to each database server.

    On database servers with 384 GB of memory, you must install the memory upgrade kit if you want to install the CPU upgrade kit.

  • Add more database servers.

    You can expand processing capacity by adding standard (fully-configured) database servers. Though not mandatory, upgrading the original Eighth Rack database servers before adding more database servers is recommended to maximize consistency across the rack.

  • Add more storage servers.

    There is no upgrade kit to expand Eighth Rack High Capacity storage servers. However, you can expand storage capacity by adding more Eighth Rack High Capacity storage servers.

    You can also expand storage capacity by adding standard High Capacity (HC), Extreme Flash (EF), or Extended (XT) storage servers.

2.1.1.2 X7-2, X8-2, and X8M-2 Eighth Rack Upgrade Options

Upgrade of Oracle Exadata Database Machine X7-2, X8-2, or X8M-2 Eighth Rack systems requires hardware modification. Eighth Rack database servers have one of the CPUs removed, and all of the memory for CPU1 was moved to CPU0. Storage servers have half the cores enabled, and half the disks and flash cards were removed.

On Oracle Exadata Database Machine X7-2, X8-2, or X8M-2 Eighth Rack systems with Extreme Flash storage servers, you can add CPUs and flash cards to extend the system to a Quarter Rack.

For Oracle Exadata Database Machine X7-2, X8-2, or X8M-2 Eighth Rack systems with High Capacity storage servers, you can add the CPU and memory to the database servers and additional Eighth Rack High Capacity storage servers to expand the system.

Specifically:

  • To extend an Eighth Rack X7-2, X8-2, or X8M-2 database server:

    1. Install CPU1

    2. Move half of CPU0's memory to CPU1

    3. Move the 10/25GbE PCI card to PCIe slot 1

  • To extend an Eighth Rack X7-2, X8-2, or X8M-2 Extreme Flash storage server, install four F640/F640v2 flash cards in PCIe slots 2,3,8, and 9.

2.1.1.3 X6-2 Eighth Rack Upgrade Options

On Oracle Exadata Database Machine X6-2 Eighth Rack systems, you can add High Capacity (HC) storage by adding more Eighth Rack High Capacity storage servers. There is no upgrade kit to expand individual Eighth Rack High Capacity storage servers.

Additionally, software-only upgrade kits are available to enable the disabled hardware resources for X6-2 Eighth Rack database servers and X6-2 Eighth Rack Extreme Flash (EF) storage servers.

2.1.1.4 Eighth Rack X4-2 and X5-2 Expansion Options

Oracle Exadata Database Machine Eighth Rack X4-2 and X5-2 contains fully configured servers, with approximately half of the available system resources (CPU, RAM, and storage) disabled. Consequently, Eighth Rack to Quarter Rack expansion of X4-2 or X5-2 systems is done using software only to enable the disabled hardware resources. Expansion beyond a Quarter Rack requires additional expansion kits, which contain additional hardware.

2.1.2 Eighth Rack Expansion Procedures

This section contains specific procedures that may be required for expanding some Oracle Exadata Database Machine Eighth Rack models.

Some of the following procedures only apply in specific circumstances. So, if present, check the applicability note at the beginning of the procedure. Otherwise, the expansion of an Eighth Rack through the addition of database or storage servers uses the same general procedures as the elastic expansion of a non-Eighth Rack. See Extending Elastic Configurations and Configuring the New Hardware.

Also, in the following procedures:

  • The disk group names and sizes are examples. The values should be changed in the commands to match the actual system.

  • User equivalence (password-less SSH) is assumed to exist between the root user on the first database server and all other database servers, and to the celladmin user on all storage cells.

  • The text files cell_group and db_group should be created to contain lists of cell host names and database server host names, respectively.

Eighth Rack Expansion Procedures:

Parent topic: Extending an Eighth Rack

2.1.2.1 Reviewing and Validating Current Configuration of Eighth Rack Oracle Exadata Database Machine

The following procedure describes how to review and validate the current configuration.

  1. Log in as the root user on the first database server.

  2. Review the current configuration of the storage servers using the following command. The expected output is TRUE. 

    # dcli -g cell_group -l celladmin 'cellcli -e LIST CELL attributes eighthrack'

  3. Review the current CPU core count on the database servers using the following command:

    # dcli -g db_group -l root 'dbmcli -e list dbserver attributes coreCount'

    The following is an example of the expected output from an Oracle Exadata Database Machine X9M-2 Eighth Rack with all CPU cores enabled: 

    dm01db01: 32
dm01db02: 32

    Contact Oracle Support Services if the number of active database server CPU cores differs from the expected value.

2.1.2.2 Activating Database Server Cores in Oracle Exadata Database Machine Eighth Rack

The following procedure describes how to activate database server cores.

Note:

This procedure applies to:

  • Original database server CPU cores that are disabled

  • Additional CPU cores that are part of an approved CPU upgrade kit

The following is not required where database server hardware resources are expanded by adding more database servers.

  1. Log in as the root user on the first database server.

  2. Activate the database server cores using the following dcli utility command on the database server group:

    # dcli -g db_group -l root  'dbmcli  -e    \
ALTER DBSERVER pendingCoreCount = number_of_cores'

    In the preceding command, number_of_cores is the total number of cores to activate. The value includes the existing core count and the additional cores to be activated. The following command shows how to activate all the cores in Oracle Exadata Database Machine X5-2 Eighth Rack: 

    # dcli -g db_group -l root 'dbmcli -e ALTER DBSERVER pendingCoreCount = 36'

    For a description of the supported core counts for each server model, see Restrictions for Capacity-On-Demand on Oracle Exadata Database Machine

  3. Restart each database server.

    Note:

    If this procedure is done in a rolling fashion with the Oracle Database and Oracle Grid Infrastructure active, then ensure the following before restarting the database server:

    • All Oracle ASM grid disks are online.

    • There are no active Oracle ASM rebalance operations. You can query the V$ASM_OPERATION view for the status of the rebalance operation.

    • Shut down Oracle Database and Oracle Grid Infrastructure in a controlled manner, failing over services as needed.

  4. Verify the following items on the database server after the restart completes and before proceeding to the next server:

    • The Oracle Database and Oracle Grid Infrastructure services are active.

      See Using SRVCTL to Verify That Instances are Running in Oracle Real Application Clusters Administration and Deployment Guide and the crsctl status resource –w "TARGET = ONLINE" —t command.

    • The number of active cores is correct. Use the dbmcli -e list dbserver attributes coreCount command to verify the number of cores.

2.1.2.3 Activating Storage Server Cores and Disks in Oracle Exadata Database Machine Eighth Rack

The following procedure describes how to activate the storage server cores and disks.

Note:

This procedure applies only to the original storage servers in the following Oracle Exadata Database Machine Eighth Rack models: X4-2, X5-2, and X6-2 with Extreme Flash (EF) storage servers.

This procedure does not apply where storage server hardware resources are expanded by adding more storage servers.

  1. Log in as the root user on the first database server.

  2. Activate the cores on the storage server group using the following command. The command uses the dcli utility, and runs the command as the celladmin user. 

    # dcli -g cell_group -l celladmin cellcli -e "alter cell eighthRack=false"

  3. Create the cell disks using the following command:

    # dcli -g cell_group -l celladmin cellcli -e  "create celldisk all"

  4. Recreate the flash log using the following commands:

    # dcli -g cell_group -l celladmin cellcli -e  "drop flashlog all force"
# dcli -g cell_group -l celladmin cellcli -e  "create flashlog all"

  5. Expand the flash cache using the following command:

    # dcli -g cell_group -l celladmin cellcli -e  "alter flashcache all"
2.1.2.4 Creating Additional Grid Disks in Oracle Exadata Database Machine Eighth Rack

Additional grid disk creation must follow a specific order to ensure the proper offset.

Note:

This procedure applies only to the original storage servers in the following Oracle Exadata Database Machine Eighth Rack models: X4-2, X5-2, and X6-2 with Extreme Flash (EF) storage servers.

This procedure does not apply where storage server hardware resources are expanded by adding more storage servers.

The order of grid disk creation must follow the same sequence that was used during the initial grid disk creation process. For a standard deployment using Oracle Exadata Deployment Assistant (OEDA), the order is DATA, RECO, and DBFS_DG (if present). Create all DATA grid disks first, followed by the RECO grid disks, and then the DBFS_DG grid disks (if present).

The following procedure describes how to create the grid disks:

Note:

The commands shown in this procedure use the standard deployment grid disk prefix names of DATA, RECO, and DBFS_DG. The sizes being checked are on cell disk 02. Cell disk 02 is used because the disk layout for cell disks 00 and 01 are different from the other cell disks in the server.

  1. Check the size of the grid disks using the following commands. Each cell should return the same size for the grid disks starting with the same grid disk prefix.

    # dcli -g cell_group -l celladmin cellcli -e    \
"list griddisk attributes name, size where name like \'DATA.*_02_.*\'"

# dcli -g cell_group -l celladmin cellcli -e    \
"list griddisk attributes name, size where name like \'RECO.*_02_.*\'"

# dcli -g cell_group -l celladmin cellcli -e    \
"list griddisk attributes name, size where name like \'DBFS_DG.*_02_.*\'"

    The sizes shown are used during grid disk creation.

  2. Create the grid disks for the disk groups using the sizes shown in step 1. The following table shows the commands to create the grid disks based on rack type and disk group.

Table 2-1 Commands to Create Disk Groups When Extending Oracle Exadata Database Machine Eighth Rack

Rack Commands

Extreme Flash Oracle Exadata Database Machine

 
dcli -g cell_group -l celladmin "cellcli -e create griddisk         \
DATA_FD_04_\'hostname -s\' celldisk=FD_04_\'hostname -s\',size=datasize"

dcli -g cell_group -l celladmin "cellcli -e create griddisk         \
DATA_FD_05_\'hostname -s\' celldisk=FD_05_\'hostname -s\',size=datasize"

dcli -g cell_group -l celladmin "cellcli -e create griddisk         \
DATA_FD_06_\'hostname -s\' celldisk=FD_06_\'hostname -s\',size=datasize"

dcli -g cell_group -l celladmin "cellcli -e create griddisk         \
DATA_FD_07_\'hostname -s\' celldisk=FD_07_\'hostname -s\',size=datasize"

dcli -g cell_group -l celladmin "cellcli -e create griddisk          \
RECO_FD_04_\'hostname -s\' celldisk=FD_04_\'hostname -s\',size=recosize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk          \
RECO_FD_05_\'hostname -s\' celldisk=FD_05_\'hostname -s\',size=recosize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk          \
RECO_FD_06_\'hostname -s\' celldisk=FD_06_\'hostname -s\',size=recosize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk          \
RECO_FD_07_\'hostname -s\' celldisk=FD_07_\'hostname -s\',size=recosize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk           \
DBFS_DG_FD_04_\'hostname -s\' celldisk=FD_04_\'hostname -s\',size=dbfssize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk           \
DBFS_DG_FD_05_\'hostname -s\' celldisk=FD_05_\'hostname -s\',size=dbfssize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk           \
DBFS_DG_FD_06_\'hostname -s\' celldisk=FD_06_\'hostname -s\',size=dbfssize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk           \
DBFS_DG_FD_07_\'hostname -s\' celldisk=FD_07_\'hostname -s\',size=dbfssize, \
cachingPolicy=none"

High Capacity Oracle Exadata Database Machine

 
dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
DATA_CD_06_\'hostname -s\' celldisk=CD_06_\'hostname -s\',size=datasize"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
DATA_CD_07_\'hostname -s\' celldisk=CD_07_\'hostname -s\',size=datasize"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
DATA_CD_08_\'hostname -s\' celldisk=CD_08_\'hostname -s\',size=datasize"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
DATA_CD_09_\'hostname -s\' celldisk=CD_09_\'hostname -s\',size=datasize"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
DATA_CD_10_\'hostname -s\' celldisk=CD_10_\'hostname -s\',size=datasize"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
DATA_CD_11_\'hostname -s\' celldisk=CD_11_\'hostname -s\',size=datasize"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
RECO_CD_06_\'hostname -s\' celldisk=CD_06_\'hostname -s\',size=recosize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk             \
RECO_CD_07_\'hostname -s\' celldisk=CD_07_\'hostname -s\',size=recosize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
RECO_CD_08_\'hostname -s\' celldisk=CD_08_\'hostname -s\',size=recosize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
RECO_CD_09_\'hostname -s\' celldisk=CD_09_\'hostname -s\',size=recosize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
RECO_CD_10_\'hostname -s\' celldisk=CD_10_\'hostname -s\',size=recosize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
RECO_CD_11_\'hostname -s\' celldisk=CD_11_\'hostname -s\',size=recosize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
DBFS_DG_CD_06_\'hostname -s\' celldisk=CD_06_\'hostname -s\',size=dbfssize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
DBFS_DG_CD_07_\'hostname -s\' celldisk=CD_07_\'hostname -s\',size=dbfssize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
DBFS_DG_CD_08_\'hostname -s\' celldisk=CD_08_\'hostname -s\',size=dbfssize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
DBFS_DG_CD_09_\'hostname -s\' celldisk=CD_09_\'hostname -s\',size=dbfssize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
DBFS_DG_CD_10_\'hostname -s\' celldisk=CD_10_\'hostname -s\',size=dbfssize, \
cachingPolicy=none"

dcli -g cell_group -l celladmin "cellcli -e create griddisk            \
DBFS_DG_CD_11_\'hostname -s\' celldisk=CD_11_\'hostname -s\',size=dbfssize, \
cachingPolicy=none"
2.1.2.5 Adding Grid Disks to Oracle ASM Disk Groups in Oracle Exadata Database Machine Eighth Rack

The following procedure describes how to add the new grid disks to Oracle ASM disk groups.

Note:

This procedure applies only to the original storage servers in the following Oracle Exadata Database Machine Eighth Rack models: X4-2, X5-2, and X6-2 with Extreme Flash (EF) storage servers.

This procedure does not apply where storage server hardware resources are expanded by adding more storage servers.

The grid disks created in Creating Additional Grid Disks in Oracle Exadata Database Machine Eighth Rack must be added as Oracle ASM disks to their corresponding, existing Oracle ASM disk groups.

  1. Validate the following:

    • No rebalance operation is currently running.
    • All Oracle ASM disks are active.

  2. Log in to the first database server as the owner who runs the Oracle Grid Infrastructure software.

  3. Set the environment to access the +ASM instance on the server.

  4. Log in to the ASM instance as the sysasm user using the following command: 

    $ sqlplus / as sysasm

  5. Validate the current settings, as follows: 

    SQL> set lines 100
SQL> column attribute format a20
SQL> column value format a20
SQL> column diskgroup format a20
SQL> SELECT att.name attribute, upper(att.value) value, dg.name diskgroup
FROM V$ASM_ATTRIBUTE att, V$ASM_DISKGROUP DG
WHERE DG.group_number=att.group_number AND att.name LIKE '%appliance.mode%'
ORDER BY att.group_number;

    The output should be similar to the following:

    ATTRIBUTE            VALUE                DISKGROUP
-------------------- -------------------- --------------------
appliance.mode       TRUE                 DATAC1
appliance.mode       TRUE                 DBFS_DG
appliance.mode       TRUE                 RECOC1

  6. Disable the appliance.mode attribute for any disk group that shows TRUE using the following commands: 

    SQL> ALTER DISKGROUP data_diskgroup set attribute 'appliance.mode'='FALSE';
SQL> ALTER DISKGROUP reco_diskgroup set attribute 'appliance.mode'='FALSE';
SQL> ALTER DISKGROUP dbfs_dg_diskgroup set attribute 'appliance.mode'='FALSE';

    In the preceding commands, data_diskgroup, reco_diskgroup, and dbfs_dg_diskgroup are the names of the DATA, RECO, and DBFS_DG disk groups, respectively.

  7. Add the grid disks to the Oracle ASM disk groups. The following table shows the commands to create the grid disks based on rack type and disk group. Adding the new disks requires a rebalance of the system.

    Table 2-2 Commands to Add Disk Groups When Extending Eighth Rack Oracle Exadata Database Machine

    Rack Commands

    Extreme Flash Oracle Exadata Database Machine

    		 
    SQL> ALTER DISKGROUP data_diskgroup ADD DISK 'o/*/DATA_FD_0[4-7]*'      \
REBALANCE POWER 32;
 
SQL> ALTER DISKGROUP reco_diskgroup ADD DISK 'o/*/RECO_FD_0[4-7]*'      \
REBALANCE POWER 32;
 
SQL> ALTER DISKGROUP dbfs_dg_diskgroup ADD DISK 'o/*/DBFS_DG_FD_0[4-7]*'\
REBALANCE POWER 32;

    High Capacity Oracle Exadata Database Machine

    		 
    SQL> ALTER DISKGROUP data_diskgroup ADD DISK 'o/*/DATA_CD_0[6-9]*','    \
o/*/DATA_CD_1[0-1]*' REBALANCE POWER 32;
 
SQL> ALTER DISKGROUP reco_diskgroup ADD DISK 'o/*/RECO_CD_0[6-9]*','    \
o/*/RECO_CD_1[0-1]*' REBALANCE POWER 32;
 
SQL> ALTER DISKGROUP dbfs_dg_diskgroup ADD DISK '                       \
o/*/DBFS_DG_CD_0[6-9]*',' o/*/DBFS_DG_CD_1[0-1]*' REBALANCE POWER 32;

    The preceding commands return Diskgroup altered, if successful.

  8. (Optional) Monitor the current rebalance operation using the following command:

    SQL> SELECT * FROM  gv$asm_operation;

  9. Re-enable the appliance.mode attribute, if it was disabled in step 6 using the following commands: 

    SQL> ALTER DISKGROUP data_diskgroup set attribute 'appliance.mode'='TRUE';
SQL> ALTER DISKGROUP reco_diskgroup set attribute 'appliance.mode'='TRUE';
SQL> ALTER DISKGROUP dbfs_dg_diskgroup set attribute 'appliance.mode'='TRUE';
2.1.2.6 Validating Expansion of Oracle Exadata Database Machine

After expansion, validate the new configuration.

Note:

This procedure applies only to the original storage servers in the following Oracle Exadata Database Machine Eighth Rack models: X4-2, X5-2, X6-2 with Extreme Flash (EF) storage servers.

This procedure does not apply where hardware resources are expanded by adding more servers.

  1. Log in as the root user on the first database server.

  2. Check the database server core count using the following command:

    # dcli -g db_group -l root 'dbmcli -e list dbserver attributes coreCount'

  3. Review the database server configuration using the following command.

    # dcli -g db_group -l root 'dbmcli -e list dbserver attributes eighthrack'

    The output should show FALSE.

  4. Review the storage server configuration using the following command.

    # dcli -g cell_group -l celladmin 'cellcli -e list cell attributes eighthrack'

    The output should show FALSE.

  5. Review the appliance mode for each disk group using the following commands:

    SQL> set lines 100
SQL> column attribute format a20
SQL> column value format a20
SQL> column diskgroup format a20
SQL> SELECT att.name attribute, upper(att.value) value, dg.name diskgroup    \
FROM V$ASM_ATTRIBUTE att, V$ASM_DISKGROUP DG                                 \
WHERE DG.group_number = att.group_number AND                                 \
att.name LIKE '%appliance.mode%' ORDER BY DG.group_number;

  6. Validate the number of Oracle ASM disks using the following command:

    SQL> SELECT g.name,d.failgroup,d.mode_status,count(*)                      \
FROM v$asm_diskgroup g, v$asm_disk d                                       \
WHERE d.group_number=g.group_number                                        \
GROUP BY g.name,d.failgroup,d.mode_status;

NAME                      FAILGROUP                     MODE_ST   COUNT(*)
------------------------- ----------------------------- ------- ----------
DATAC1                    EXA01CELADM01                 ONLINE          12
DATAC1                    EXA01CELADM02                 ONLINE          12
DATAC1                    EXA01CELADM03                 ONLINE          12
RECOC1                    EXA01CELADM01                 ONLINE          12
RECOC1                    EXA01CELADM02                 ONLINE          12
RECOC1                    EXA01CELADM03                 ONLINE          12
RECOC2                    EXA01CELADM01                 ONLINE          12
RECOC2                    EXA01CELADM02                 ONLINE          12
RECOC2                    EXA01CELADM03                 ONLINE          12
DBFS_DG                   EXA01CELADM01                 ONLINE          10
DBFS_DG                   EXA01CELADM02                 ONLINE          10
DBFS_DG                   EXA01CELADM03                 ONLINE          10

    Each High Capacity (HC) storage server (non-Eighth Rack) contains 12 disks.

2.2 Extending Elastic Configurations

Oracle Exadata is available in Elastic Configurations that consist of a number of database and storage servers up to the capacity of the rack, as defined within Oracle Exadata Configuration Assistant (OECA).

Additional database and storage servers can be added if space is available; see OECA for details. The upgrade process includes adding new servers and cables

Note:

It is possible to extend the hardware while the machine is online, and with no downtime. However, extreme care should be taken. In addition, patch application to existing switches and servers should be done before extending the hardware.

Parent topic: Extending the Hardware

2.2.1 Removing the Doors

This procedure describes how to remove the doors on Oracle Exadata.

Parent topic: Extending Elastic Configurations

2.2.2 Adding New RDMA Network Fabric Switches

You can add individual new RDMA Network Fabric switches as needed to meet growing resource requirements.

The instructions are different for RoCE Network Fabric switches and InfiniBand Network Fabric switches.

Parent topic: Extending Elastic Configurations

2.2.2.1 Adding a RoCE Network Fabric Switch (Cisco Nexus 9336C-FX2)

This procedure only applies to systems with Cisco Nexus 9336C-FX2 RoCE Network Fabric switches.

Note:

The steps in this procedure are specific to Oracle Exadata. They are not the same as the steps in the Cisco Nexus manual.

  1. Unpack the Cisco Nexus switch components from the packing cartons. The following items should be in the packing cartons:

    • Cisco Nexus 9336C-FX2 Switch

    • Cable bracket and rack-mount kit

    • Cable management bracket and cover

    • Two rack rail assemblies

    • Assortment of screws and captive nuts

    • Cisco Nexus 9336C-FX2 Switch documentation

    The service label procedure on top of the switch includes descriptions of the preceding items.

  2. Remove the trough from the rack in RU1. Put the cables aside while installing the RoCE Network Fabric switch. The trough can be discarded.

  3. Install cage nuts in each rack rail in the appropriate holes.

  4. Attach the brackets with cutouts to the power supply side of the switch.

  5. Attach the C-brackets to the switch on the side of the ports.

  6. Slide the switch halfway into the rack from the front. Keep the switch to the left side of the rack as far as possible while pulling the two power cords through the C-bracket on the right side.

  7. Slide the server in rack location U2 out to the locked service position. This improves access to the rear of the switch during further assembly.

  8. Install the slide rails from the rear of the rack into the C-brackets on the switch, pushing them up to the rack rail.

  9. Attach an assembled cable arm bracket to the slide rail and using a No. 3 Phillips screwdriver, screw these together into the rack rail:

    1. Install the lower screw loosely with the cable arm bracket rotated 90 degrees downward. This allows better finger access to the screw.

    2. Rotate the cable arm bracket to the correct position.

    3. Install the upper screw.

    4. Tighten both screws.

    If available, a screwdriver with a long-shaft (16-inch / 400mm) will allow easier installation such that the handle is outside the rack and beyond the cabling.

  10. Push the switch completely into the rack from the front, routing the power cords through the cutout on the rail bracket.

  11. Secure the switch to the front rack rail with M6 16mm screws. Tighten the screws using the No. 3 Phillips screwdriver.

  12. Install the lower part of the cable management arm across the back of the switch.

  13. Connect the cables to the appropriate ports.

  14. Install the upper part of the cable management arm.

  15. Slide the server in rack location U2 back into the rack.

  16. Install power cords to the switch power supply slots on the front.

  17. Loosen the front screws to install the vented filler panel brackets. Tighten the screws, and snap on the vented filler panel in front of the switch.

See Also:

Parent topic: Adding New RDMA Network Fabric Switches

2.2.2.2 Adding an InfiniBand Network Fabric Switch (Sun Datacenter InfiniBand Switch 36)

This procedure only applies to systems with InfiniBand Network Fabric.

Note:

The steps in this procedure are specific to Oracle Exadata Database Machine. They are not the same as the steps in the Sun Datacenter InfiniBand Switch 36 manual.

  1. Unpack the Sun Datacenter InfiniBand Switch 36 switch components from the packing cartons. The following items should be in the packing cartons:

    • Sun Datacenter InfiniBand Switch 36 switch

    • Cable bracket and rackmount kit

    • Cable management bracket and cover

    • Two rack rail assemblies

    • Assortment of screws and captive nuts

    • Sun Datacenter InfiniBand Switch 36 documentation

    The service label procedure on top of the switch includes descriptions of the preceding items.

  2. X5 racks only: Remove the trough from the rack in RU1 and put the cables aside while installing the Sun Datacenter InfiniBand Switch 36 switch. The trough can be discarded.

  3. Install cage nuts in each rack rail in the appropriate holes.

  4. Attach the brackets with cutouts to the power supply side of the switch.

  5. Attach the C-brackets to the switch on the side of the Sun Datacenter InfiniBand Switch 36 ports.

  6. Slide the switch halfway into the rack from the front. You need to keep it to the left side of the rack as far as possible while pulling the two power cords through the C-bracket on the right side.

  7. Slide the server in rack location U2 out to the locked service position. This improves access to the rear of the switch during further assembly.

  8. Install the slide rails from the rear of the rack into the C-brackets on the switch, pushing them up to the rack rail.

  9. Attach an assembled cable arm bracket to the slide rail and using a No. 3 Phillips screwdriver, screw these together into the rack rail:

    1. Install the lower screw loosely with the cable arm bracket rotated 90 degrees downward. This allows better finger access to the screw.

    2. Rotate the cable arm bracket to the correct position.

    3. Install the upper screw.

    4. Tighten both screws.

    If available, a screwdriver with a long-shaft (16-inch / 400mm) will allow easier installation such that the handle is outside the rack and beyond the cabling.

  10. Push the switch completely into the rack from the front, routing the power cords through the cutout on the rail bracket.

  11. Secure the switch to the front rack rail with M6 16mm screws. Tighten the screws using the No. 3 Phillips screwdriver.

  12. Install the lower part of the cable management arm across the back of the switch.

  13. Connect the cables to the appropriate ports.

  14. Install the upper part of the cable management arm.

  15. Slide the server in rack location U2 back into the rack.

  16. Install power cords to the Sun Datacenter InfiniBand Switch 36 switch power supply slots on the front.

  17. Loosen the front screws to install the vented filler panel brackets. Tighten the screws, and snap on the vented filler panel in front of the switch.

See Also:

Parent topic: Adding New RDMA Network Fabric Switches

2.2.3 Adding New Servers

You can add new servers to an Oracle Exadata Rack that is not at full capacity.

You can add individual database servers or storage servers to meet growing resource requirements using the Elastic Configuration method. See Oracle Exadata Configuration Assistant (OECA) for details. The upgrade process includes adding new servers and cables. Additional hardware may be required.

Note:

  • Always load equipment into the rack from the bottom up, so that the rack does not become top-heavy and tip over. Extend the rack anti-tip bar to prevent the rack from tipping during equipment installation.

  • The new servers need to be configured manually.

Related Topics

Parent topic: Extending Elastic Configurations

2.2.3.1 Preparing to Install New Servers

Before you install a new server, prepare the rack unit for the server installation.

  1. Identify the rack unit where the server will be installed. Fill the first available unit, starting from the bottom of the rack.

  2. Remove and discard the trough, which attaches the cable harness when no server is installed in the unit.

  3. Remove and discard the solid filler.

Parent topic: Adding New Servers

2.2.3.2 Installing the Rack Assembly

After preparing for installation, you next install the rack assembly to hold the new servers.

  1. Position a mounting bracket against the chassis so that the slide-rail lock is at the server front, and the five keyhole openings on the mounting bracket are aligned with the five locating pins on the side of the chassis.

  2. Orient the slide-rail assembly so that the ball-bearing track is forward and locked in place.

  3. Starting on either side of the rack, align the rear of the slide-rail assembly against the inside of the rear rack rail, and push until the assembly locks into place with an audible click.

    Figure 2-1 Locking the Slide-Rail Assembly Against the Inside of the Rear Rack Rail

    Description of Figure 2-1 follows
    Description of "Figure 2-1 Locking the Slide-Rail Assembly Against the Inside of the Rear Rack Rail"

  4. Align the front of the slide-rail assembly against the outside of the front rack rail, and push until the assembly locks into place and you hear the click.

  5. Repeat steps 2 to 4 on the other side on the rack.

Parent topic: Adding New Servers

2.2.3.3 Installing the Server

After preparing for the installation and installing the rack assembly, you then install the new server.

WARNING:

  • Installing a server requires a minimum of two people or a lift because of the weight of each server. Attempting this procedure alone can result in equipment damage, personal injury, or both.

  • Always load equipment into the rack from the bottom up, so that the rack does not become top-heavy and tip over. Extend the rack anti-tip bar to prevent the rack from tipping during equipment installation.

  1. Read the service label on the top cover of the server before installing a server into the rack.

  2. Push the server into the slide rail assembly:

    1. Push the slide rails into the slide rail assemblies as far as possible.

    2. Position the server so the rear ends of the mounting brackets are aligned with the slide rail assemblies mounted in the equipment rack.

      Figure 2-2 Aligning the Rear Ends of the Mounting Brackets with the Slide Rail Assemblies in the Rack


      Description of Figure 2-2 follows
      Description of "Figure 2-2 Aligning the Rear Ends of the Mounting Brackets with the Slide Rail Assemblies in the Rack"

      The callouts in the preceding image highlight the following:

      1: Mounting bracket inserted into slide rail

      2: Slide-rail release lever

    3. Insert the mounting brackets into the slide rails, and push the server into the rack until the mounting brackets encounter the slide rail stops, approximately 30 cm (12 inches).

    4. Simultaneously push down and hold the slide rail release levers on each mounting bracket while pushing the server into the rack.

      Note:

      Oracle recommends that two people push the servers into the rack: one person to move the server in and out of the rack, and another person to watch the cables and cable management arm (CMA).

    5. Continue pushing until the slide rail locks on the front of the mounting brackets engage the slide rail assemblies, and you hear the click.

  3. Cable the new server as described in Cabling Exadata Storage Servers.

Parent topic: Adding New Servers

2.2.4 Cabling Database Servers

After the new database servers are installed, they need to be cabled to the existing equipment. The following procedure describes how to cable the new equipment in the rack. The images shown in the procedure are of a Sun Fire X4170 M2 Oracle Database Server.

Note:

  • The existing cable connections in the rack do not change.

  • The blue cables connect to Oracle Database servers, and the black cables connect to Exadata Storage Servers. These network cables are for the NET0 Ethernet interface port.

  • Attach and route the management cables on the CMA and rear panel one server at a time. Do not slide out more than one server at a time.

  • Start from the bottom of the rack, and work upward. Route the cables through the CMA with the dongle on the top and power cables on the bottom.

  • Longer hook and loop straps are needed when cabling three CAT5e cables or two TwinAx cables.

  1. Connect the CAT5e cables, AC power cables, and USB to their respective ports on the rear of the server. Ensure the flat side of the dongle is flush against the CMA inner rail.

    Figure 2-3 Cables at the Rear of the Server

    Description of Figure 2-3 follows
    Description of "Figure 2-3 Cables at the Rear of the Server"

  2. Adjust the green cable management arm (CMA) brackets

    Figure 2-4 Cable Management Arm (CMA) Brackets

    Description of Figure 2-4 follows
    Description of "Figure 2-4 Cable Management Arm (CMA) Brackets"

    Description of the CMA callouts in the preceding image"

    1. Connector A

    2. Front slide bar

    3. Velcro straps (6)

    4. Connector B

    5. Connector C

    6. Connector D

    7. Slide-rail latching bracket (used with connector D)

    8. Rear slide bar

    9. Cable covers

    10. Cable covers

  3. Attach the CMA to the server.

  4. Route the CAT5e and power cables through the wire clip.

    Figure 2-5 Cables Routed Through the Cable Management Arm

    Description of Figure 2-5 follows
    Description of "Figure 2-5 Cables Routed Through the Cable Management Arm"

  5. Bend the CAT5e and power cables to enter the CMA, while adhering to the bend radius minimums.

  6. Secure the CAT5e and power cables under the cable clasps.

    Figure 2-6 Cables Secured under the Cable Clasps

    Description of Figure 2-6 follows
    Description of "Figure 2-6 Cables Secured under the Cable Clasps"

  7. Route the cables through the CMA, and secure them with hook and loop straps at equal intervals.

    Figure 2-7 Cables Secured with Hook and Loop Straps at Regular Intervals

    Description of Figure 2-7 follows
    Description of "Figure 2-7 Cables Secured with Hook and Loop Straps at Regular Intervals"

  8. Connect the RDMA Network Fabric or TwinAx cables with the initial bend resting on the CMA. The TwinAx cables are for client access to the database servers.

    Figure 2-8 RDMA Network Fabric or TwinAx Cables Positioned on the CMA

    Description of Figure 2-8 follows
    Description of "Figure 2-8 RDMA Network Fabric or TwinAx Cables Positioned on the CMA"

  9. Secure the RDMA Network Fabric or TwinAx cables with hook and loop straps at equal intervals.

    Figure 2-9 RDMA Network Fabric or TwinAx Cables Secured with Hook and Loop Straps at Regular Intervals

    Description of Figure 2-9 follows
    Description of "Figure 2-9 RDMA Network Fabric or TwinAx Cables Secured with Hook and Loop Straps at Regular Intervals"

  10. Route the fiber core cables.

  11. Rest the cables over the green clasp on the CMA.

  12. Attach the red ILOM cables to the database server.

  13. Attach the network cables to the Oracle Database server.

  14. Attach the cables from Oracle Database server to the RDMA Network Fabric switches.

  15. Connect the orange Ethernet cable to the KVM switch.

  16. Connect the red and blue Ethernet cables to the Cisco switch.

  17. Verify operation of the slide rails and CMA for each server, as follows:

    Note:

    Oracle recommends that two people do this step. One person to move the server in and out of the rack, and another person to observe the cables and CMA.

    1. Slowly pull the server out of the rack until the slide rails reach their stops.

    2. Inspect the attached cables for any binding or kinks.

    3. Verify the CMA extends fully from the slide rails.

  18. Push the server back into the rack, as follows:

    1. Release the two sets of slide rail stops.

    2. Push in both levers simultaneously, and slide the server into the rack. The first stop in the set are levers located on the inside of each slide rail, just behind the back panel of the server. The levers are labeled PUSH. The server slides approximately 46 cm (18 inches) and stop.

    3. Verify the cables and CMA retract without binding.

    4. Simultaneously push or pull both slide rail release buttons, and push the server completely into the rack until both slide rails engage. The second stop in the set are the slide rail release buttons located near the front of each mounting bracket.

  19. Dress the cables, and then tie off the cables with the straps. Oracle recommends the cables should be dressed in bundles of eight or less.

  20. Extend and then fully retract the server to check cable travel by sliding each server out and back fully to ensure that the cables are not binding or catching.

  21. Repeat the procedure for the rest of the servers.

  22. Connect the power cables to the power distribution units (PDUs). Ensure the breaker switches are in the OFF position before connecting the power cables. Do not plug the power cables into the facility receptacles at this time.

See Also:

Parent topic: Extending Elastic Configurations

2.2.5 Cabling Storage Servers

After the new Storage Servers are installed, you need to connect them to the existing equipment.

The following procedure describes how to cable the new equipment in the rack.

Note:

  • The existing cable connections in the rack do not change.

  • The blue cables connect to Oracle Database servers, and the black cables connect to Exadata Storage Servers. These network cables are for the NET0 Ethernet interface port.

  • Attach and route the management cables on the CMA and rear panel one server at a time. Do not slide out more than one server at a time.

  • Start from the bottom of the rack, and work upward.

  • Longer hook and loop straps are needed when cabling three CAT5e cables or two TwinAx cables.

  1. Attach a CMA to the server.

  2. Insert the cables into their ports through the hook and loop straps, then route the cables into the CMA in this order:

    1. Power

    2. Ethernet

    3. RDMA Network Fabric

    Figure 2-10 Rear of the Server Showing Power and Network Cables

    Description of Figure 2-10 follows
    Description of "Figure 2-10 Rear of the Server Showing Power and Network Cables"

  3. Route the cables through the CMA and secure them with hook and loop straps on both sides of each bend in the CMA.

    Figure 2-11 Cables Routed Through the CMA and Secured with Hook and Loop Straps

    Description of Figure 2-11 follows
    Description of "Figure 2-11 Cables Routed Through the CMA and Secured with Hook and Loop Straps"

  4. Close the crossbar covers to secure the cables in the straightaway.

  5. Verify operation of the slide rails and the CMA for each server:

    Note:

    Oracle recommends that two people do this step: one person to move the server in and out of the rack, and another person to watch the cables and the CMA.

    1. Slowly pull the server out of the rack until the slide rails reach their stops.

    2. Inspect the attached cables for any binding or kinks.

    3. Verify that the CMA extends fully from the slide rails.

  6. Push the server back into the rack:

    1. Release the two sets of slide rail stops.

    2. Locate the levers on the inside of each slide rail, just behind the back panel of the server. They are labeled PUSH.

    3. Simultaneously push in both levers and slide the server into the rack, until it stops in approximately 46 cm (18 inches).

    4. Verify that the cables and CMA retract without binding.

    5. Locate the slide rail release buttons near the front of each mounting bracket.

    6. Simultaneously push in both slide rail release buttons and slide the server completely into the rack, until both slide rails engage.

  7. Dress the cables, and then tie off the cables with the straps. Oracle recommends that you dress the RDMA Network Fabric cables in bundles of eight or fewer.

  8. Slide each server out and back fully to ensure that the cables are not binding or catching.

  9. Repeat the procedure for all servers.

  10. Connect the power cables to the power distribution units (PDUs). Ensure the breaker switches are in the OFF position before connecting the power cables. Do not plug the power cables into the facility receptacles now.

See Also:

Multi-Rack Cabling Tables

Oracle Exadata Database Machine System Overview for the cabling tables for your system

Parent topic: Extending Elastic Configurations

2.2.6 Closing the Rack

After installing new equipment, you must replace the panels and close the rack.

There are two rack models in use with Oracle Exadata Racks. Refer to the appropriate documentation for the most up-to-date steps:

The following steps provide an overview of the process.

  1. Replace the rack front and rear doors as follows:

    1. Retrieve the doors, and place them carefully on the door hinges.

    2. Connect the front and rear door grounding strap to the frame.

    3. Close the doors.

    4. (Optional) Lock the doors. The keys are in the shipping kit.

  2. (Optional) Replace the side panels, if they were removed for the upgrade, as follows:

    1. Lift each side panel up and onto the side of the rack. The top of the rack should support the weight of the side panel. Ensure the panel fasteners line up with the grooves in the rack frame.

    2. Turn each side panel fastener one-quarter turn clockwise using the side panel removal tool. Turn the fasteners next to the panel lock clockwise. There are 10 fasteners per side panel.

    3. (Optional) Lock each side panel. The key is in the shipping kit. The locks are located on the bottom, center of the side panels.

    4. Connect the grounding straps to the side panels.

After closing the rack, proceed to Configuring the New Hardware to configure the new hardware.

Parent topic: Extending Elastic Configurations

2.3 Extending a Rack by Adding Another Rack

You can extend your Oracle Exadata Rack by adding another rack and configuring the racks together.

Parent topic: Extending the Hardware

2.3.1 Overview of Adding Another Rack to an Existing System

Review the following notes before cabling racks together.

  • The procedures for extending racks with RoCE Network Fabric (X8M and later) are different than the procedures for racks with InfiniBand Network Fabric (X8 and earlier.)

  • Racks with InfiniBand Network Fabric can be cabled together with no downtime. Depending on the procedure being used, racks with RoCE Network Fabric might require downtime when cabling racks together.

  • Cabling within a live network must be done carefully in order to avoid potentially serious disruptions.

  • There can be performance degradation while cabling the racks together. This degradation results from data retransmission due to packet loss and reduced network bandwidth when a cable is unplugged.

  • Redundancy with the RDMA Network Fabric can be compromised while cabling the racks together. This occurs whenever the RDMA Network Fabric ports or switches are taken offline and all traffic must use the remaining switches.

  • Only the existing racks are operational when adding racks. It is assumed that the servers on any new racks are initially powered down.

  • The software running on the systems cannot have problems related to RDMA Network Fabric restarts. To verify the configuration, run infinicheck separately on each rack before connecting multiple racks together.

  • It is assumed that each Oracle Exadata Rack has three RDMA Network Fabric switches already installed.

  • The new racks have been configured with the appropriate IP addresses to be migrated into the expanded system prior to any cabling, and there are no duplicate IP addresses.

  • Racks with RoCE Network Fabric use one loopback IP interface on each spine switch and two loopback IP interfaces on each leaf switch. The IP addressing scheme uses IANA 'Shared Address Space' 100.64.0.0/10. This ensures that there is no overlap with IPv4 addresses in the network using other schemes.
    • Leaf loopback0 IPs are assigned as 100.64.0.101, 100.64.0.102, 100.64.0.103, and so on.
    • Leaf loopback1 IPs are assigned as 100.64.1.101, 100.64.1.102, 100.64.1.103, and so on.
    • Spine loopback0 IPs are assigned as 100.64.0.201, 100.64.0.202, up to 100.64.0.208.

Parent topic: Extending a Rack by Adding Another Rack

2.3.2 Cabling Two Racks Together

Choose from the available methods based on your system specifications and operational requirements.

Parent topic: Extending a Rack by Adding Another Rack

2.3.2.1 Cabling Two RoCE Network Fabric Racks Together with No Down Time

If your operational requirements cannot tolerate any scheduled down time, then choose from the following procedures to extend your existing RoCE Network Fabric rack by adding another rack.

2.3.2.1.1 Extending an X9M or Later Model Rack with No Down Time by Adding Another X9M or Later Model Rack

WARNING:

Take time to read and understand this procedure before implementation. Pay careful attention to the instructions that surround the command examples. A system outage may occur if the procedure is not applied correctly.

Note:

For additional background information, see Understanding Multi-Rack Cabling for X9M and Later Model Racks.

Use this procedure to extend a typical X9M or later model rack by cabling it together with a second X9M or later model rack. The primary rack (designated R1) and all of the systems it supports remain online throughout the procedure. At the beginning of the procedure, the additional rack (designated R2) is shut down.

The following is an outline of the procedure:

  • Preparation (steps 1 and 2)

    In this phase, you prepare the racks, switches, and cables. Also, you install and cable the spine switches in both racks.

  • Configuration and Physical Cabling

    In this phase, you reconfigure the leaf switches and finalize the cabling to the spine switches. These tasks are carefully orchestrated to avoid downtime on the primary system, as follows:

    • Partially configure the lower leaf switches (step 3)

      In this step, you reconfigure the switch ports on the lower leaf switches. There is no physical cabling performed in this step.

    • Partially configure the upper leaf switches (step 4)

      In this step, you reconfigure the switch ports on the upper leaf switches, remove the inter-switch cables that connect the leaf switches in both racks and connect the cables between the upper leaf switches and the spine switches.

    • Finalize the lower leaf switches (step 5)

      In this step, you finalize the switch port configuration on the lower leaf switches. You also complete the physical cabling by connecting the cables between the lower leaf switches and the spine switches.

    • Finalize the upper leaf switches (step 6)

      In this step, you finalize the switch port configuration on the upper leaf switches.

  • Validation and Testing (steps 7 and 8)

    In this phase, you validate and test the RoCE Network Fabric across both of the interconnect racks.

After completing the procedure, both racks share the RoCE Network Fabric, and the combined system is ready for further configuration. For example, you can extend existing disk groups and Oracle RAC databases to consume resources across both racks.

Note:

  • This procedure applies only to typical rack configurations that initially have leaf switches with the following specifications:

    • The inter-switch ports are ports 4 to 7, and ports 30 to 33.

    • The storage server ports are ports 8 to 14, and ports 23 to 29.

    • The database server ports are ports 15 to 22.

    For other rack configurations (for example, X9M-8 systems with three database servers and 11 storage servers) a different procedure and different RoCE Network Fabric switch configuration files are required. Contact Oracle for further guidance.

  • The procedure uses the following naming abbreviations and conventions:

    • The abbreviation for the existing rack is R1, and the new rack is R2.

    • LL identifies a lower leaf switch and UL identifies an upper leaf switch.

    • SS identifies a spine switch.

    • A specific switch is identified by combining abbreviations. For example, R1LL identifies the lower leaf switch (LL) on the existing rack (R1).

  • Most operations must be performed in multiple locations. For example, step 1.h instructs you to update the firmware on all the RoCE Network Fabric leaf switches (R1LL, R1UL, R2LL, and R2UL). Pay attention to the instructions and keep track of your actions.

    Tip:

    When a step must be performed on multiple switches, the instruction contains a list of the applicable switches. For example, (R1LL, R1UL, R2LL, and R2UL). You can use this list as a checklist to keep track of your actions.

  • Perform operations sequentially, and complete every operation before proceeding. For example, run the entire command sequence at 3.a.i as one operation and complete it before proceeding.

  • All of commands that are run on a RoCE Network Fabric switch must be run while connected to the switch management interface as the switch administrator.

  1. Prepare the systems.
    1. Position the new rack (R2) so that it is physically near the existing rack (R1).

      The RDMA Network Fabric cables must be able to reach the switches in each rack.

      For the required cross-rack cabling information, see Two-Rack Cabling for X9M and Later Model Racks.

    2. Power on all of the servers and network switches in the new rack (R2).

      This includes the database servers, storage servers, RoCE Network Fabric leaf switches, and the Management Network Switch.

    3. Prepare the RoCE Network Fabric cables that you will use to interconnect the racks.

      Label both ends of every cable.

      For the required cross-rack cabling information, see Two-Rack Cabling for X9M and Later Model Racks.

    4. Connect the new rack (R2) to your existing management network.

      Ensure that there are no IP address conflicts across the racks and that you can access the management interfaces on the RoCE Network Fabric switches.

    5. Ensure that you have a backup of the current switch configuration for each RoCE Network Fabric switch (R1LL, R1UL, R2LL, and R2UL).

      See Backing Up Settings on the RoCE Network Fabric Switch in Oracle Exadata Database Machine Maintenance Guide.

    6. Download the required RoCE Network Fabric switch configuration files.

      This procedure requires specific RoCE Network Fabric switch configuration files, which you must download from My Oracle Support document 2704997.1.

      WARNING:

      You must use different switch configuration files depending on whether your system uses Exadata Secure RDMA Fabric Isolation. Ensure that you download the correct archive that matches your system configuration.

      For system configurations without Secure Fabric, download online_multi-rack_14uplinks.zip. For system configurations with Secure Fabric, download online_SF_enabled_multi-rack_14uplinks.zip.

      Download and extract the archive containing the required RoCE Network Fabric switch configuration files. Place the files on a server with access to the management interfaces on the RoCE Network Fabric switches.

    7. Copy the required RoCE Network Fabric switch configuration files to the leaf switches on both racks.

      You can use the following commands to copy the required configuration files to all of the RoCE Network Fabric switches on a system without Secure Fabric enabled: 

      1. # scp roce_multi_14uplinks_online_step3_R1_LL.cfg admin@R1LL_IP:/
      2. # scp roce_multi_14uplinks_online_step3_R2_LL.cfg admin@R2LL_IP:/
      3. # scp roce_multi_14uplinks_online_step4_R1_UL.cfg admin@R1UL_IP:/
      4. # scp roce_multi_14uplinks_online_step4_R2_UL.cfg admin@R2UL_IP:/
      5. # scp roce_multi_14uplinks_online_step5.cfg admin@R1LL_IP:/
      6. # scp roce_multi_14uplinks_online_step5.cfg admin@R2LL_IP:/

      On a system with Secure Fabric enabled, you can use the following commands: 

      1. # scp roce_SF_multi_14uplinks_online_step3_R1_LL.cfg admin@R1LL_IP:/
      2. # scp roce_SF_multi_14uplinks_online_step3_R2_LL.cfg admin@R2LL_IP:/
      3. # scp roce_SF_multi_14uplinks_online_step4_R1_UL.cfg admin@R1UL_IP:/
      4. # scp roce_SF_multi_14uplinks_online_step4_R2_UL.cfg admin@R2UL_IP:/
      5. # scp roce_SF_multi_14uplinks_online_step5.cfg admin@R1LL_IP:/
      6. # scp roce_SF_multi_14uplinks_online_step5.cfg admin@R2LL_IP:/

      In the above commands, substitute the appropriate IP address or host name where applicable. For example, in place of R1LL_IP, substitute the management IP address or host name for the lower leaf switch (LL) on the existing rack (R1).

      Note:

      The command examples in the rest of this procedure use the configuration files for a system configuration without Secure Fabric enabled. If required, adjust the commands to use the Secure Fabric-enabled switch configuration files.
    8. Update the firmware to the latest available release on all of the RoCE Network Fabric leaf switches (R1LL, R1UL, R2LL, and R2UL).

      See Updating RoCE Network Fabric Switch Firmware in Oracle Exadata Database Machine Maintenance Guide.

    9. Examine the RoCE Network Fabric leaf switches (R1LL, R1UL, R2LL, and R2UL) and confirm the port categories for the cabled ports.

      Run the show interface status command on every RoCE Network Fabric leaf switch: 

      1. R1LL# show interface status
      2. R1UL# show interface status
      3. R2LL# show interface status
      4. R2UL# show interface status

      Examine the output and confirm the port categories as follows:

      • Confirm that the inter-switch ports are ports 4 to 7, and ports 30 to 33.

      • Confirm that the storage server ports are ports 8 to 14, and ports 23 to 29.

      • Confirm that the database server ports are ports 15 to 22.

      For example:

      R1LL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected trunk     full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected trunk     full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected trunk     full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected trunk     full   100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           connected 3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           connected 3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           connected 3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           connected 3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           connected 3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           connected 3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              connected 3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              connected 3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              connected 3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              connected 3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              connected 3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              connected 3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              connected 3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              connected 3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           connected 3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           connected 3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           connected 3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           connected 3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           connected 3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           connected 3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           connected 3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected trunk     full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected trunk     full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected trunk     full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected trunk     full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --
    10. For each rack (R1 and R2), confirm the RoCE Network Fabric cabling by running the verify_roce_cables.py script.

      The verify_roce_cables.py script uses two input files; one for database servers and storage servers (nodes.rackN), and another for switches (switches.rackN). In each file, every server or switch must be listed on separate lines. Use fully qualified domain names or IP addresses for each server and switch.

      See My Oracle Support document 2587717.1 for download and detailed usage instructions.

      Run the verify_roce_cables.py script against both of the racks: 

      1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
      2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack2 -s switches.rack2

      Check that output in the CABLE OK? columns contains the OK status.

      The following example shows the expected command results:

      # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
SWITCH PORT (EXPECTED PEER)               LOWER LEAF (rack1sw-rocea0) : CABLE OK?             UPPER LEAF (rack1sw-roceb0) : CABLE OK?
----------- ---------------          -------------------------------- : --------         -------------------------------- : ---------
    Eth1/4 (ISL peer switch)   :           rack1sw-rocea0 Ethernet1/4 : OK                     rack1sw-roceb0 Ethernet1/4 : OK
    Eth1/5 (ISL peer switch)   :           rack1sw-rocea0 Ethernet1/5 : OK                     rack1sw-roceb0 Ethernet1/5 : OK
    Eth1/6 (ISL peer switch)   :           rack1sw-rocea0 Ethernet1/6 : OK                     rack1sw-roceb0 Ethernet1/6 : OK
    Eth1/7 (ISL peer switch)   :           rack1sw-rocea0 Ethernet1/7 : OK                     rack1sw-roceb0 Ethernet1/7 : OK
    Eth1/8 (RU39)              :                 rack1celadm14 port-1 : OK                           rack1celadm14 port-2 : OK
    Eth1/9 (RU37)              :                 rack1celadm13 port-1 : OK                           rack1celadm13 port-2 : OK
   Eth1/10 (RU35)              :                 rack1celadm12 port-1 : OK                           rack1celadm12 port-2 : OK
   Eth1/11 (RU33)              :                 rack1celadm11 port-1 : OK                           rack1celadm11 port-2 : OK
   Eth1/12 (RU31)              :                 rack1celadm10 port-1 : OK                           rack1celadm10 port-2 : OK
   Eth1/13 (RU29)              :                 rack1celadm09 port-1 : OK                           rack1celadm09 port-2 : OK
   Eth1/14 (RU27)              :                 rack1celadm08 port-1 : OK                           rack1celadm08 port-2 : OK
   Eth1/15 (RU26)              :                    rack1adm08 port-1 : OK                              rack1adm08 port-2 : OK
   Eth1/16 (RU25)              :                    rack1adm07 port-1 : OK                              rack1adm07 port-2 : OK
   Eth1/17 (RU24)              :                    rack1adm06 port-1 : OK                              rack1adm06 port-2 : OK
   Eth1/18 (RU23)              :                    rack1adm05 port-1 : OK                              rack1adm05 port-2 : OK
   Eth1/19 (RU19)              :                    rack1adm04 port-1 : OK                              rack1adm04 port-2 : OK
   Eth1/20 (RU18)              :                    rack1adm03 port-1 : OK                              rack1adm03 port-2 : OK
   Eth1/21 (RU17)              :                    rack1adm02 port-1 : OK                              rack1adm02 port-2 : OK
   Eth1/22 (RU16)              :                    rack1adm01 port-1 : OK                              rack1adm01 port-2 : OK
   Eth1/23 (RU14)              :                 rack1celadm07 port-1 : OK                           rack1celadm07 port-2 : OK
   Eth1/24 (RU12)              :                 rack1celadm06 port-1 : OK                           rack1celadm06 port-2 : OK
   Eth1/25 (RU10)              :                 rack1celadm05 port-1 : OK                           rack1celadm05 port-2 : OK
   Eth1/26 (RU08)              :                 rack1celadm04 port-1 : OK                           rack1celadm04 port-2 : OK
   Eth1/27 (RU06)              :                 rack1celadm03 port-1 : OK                           rack1celadm03 port-2 : OK
   Eth1/28 (RU04)              :                 rack1celadm02 port-1 : OK                           rack1celadm02 port-2 : OK
   Eth1/29 (RU02)              :                 rack1celadm01 port-1 : OK                           rack1celadm01 port-2 : OK
   Eth1/30 (ISL peer switch)   :          rack1sw-rocea0 Ethernet1/30 : OK                    rack1sw-roceb0 Ethernet1/30 : OK
   Eth1/31 (ISL peer switch)   :          rack1sw-rocea0 Ethernet1/31 : OK                    rack1sw-roceb0 Ethernet1/31 : OK
   Eth1/32 (ISL peer switch)   :          rack1sw-rocea0 Ethernet1/32 : OK                    rack1sw-roceb0 Ethernet1/32 : OK
   Eth1/33 (ISL peer switch)   :          rack1sw-rocea0 Ethernet1/33 : OK                    rack1sw-roceb0 Ethernet1/33 : OK
    11. For each rack (R1 and R2), verify the RoCE Network Fabric operation by using the infinicheck command.

      • Use infinicheck with the -z option to clear the files that were created during the last run of the infinicheck command.

      • Use infinicheck with the -s option to set up user equivalence for password-less SSH across the RoCE Network Fabric.

      • Finally, verify the RoCE Network Fabric operation by using infinicheck with the -b option, which is recommended on newly imaged machines where it is acceptable to suppress the cellip.ora and cellinit.ora configuration checks.

      In each command, the hosts input file (hosts.rack1 and hosts.rack2) contains a list of database server RoCE Network Fabric IP addresses (2 RoCE Network Fabric IP addresses for each database server), and the cells input file (cells.rack1 and cells.rack2) contains a list of RoCE Network Fabric IP addresses for the storage servers (2 RoCE Network Fabric IP addresses for each storage server).

      1. Use the following recommended command sequence on the existing rack (R1):

        1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack1 -c cells.rack1 -z
        2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack1 -c cells.rack1 -s
        3. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack1 -c cells.rack1 -b

      2. Use the following recommended command sequence on the new rack (R2):

        1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack2 -c cells.rack2 -z
        2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack2 -c cells.rack2 -s
        3. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack2 -c cells.rack2 -b

      The following example shows the expected command results for the final command in the sequence:

      # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rackN -c cells.rackN -b

INFINICHECK                    
        [Network Connectivity, Configuration and Performance]        
               
          ####  FABRIC TYPE TESTS  #### 
System type identified: RoCE
Verifying User Equivalance of user=root from all DBs to all CELLs.
     ####  RoCE CONFIGURATION TESTS  ####       
     Checking for presence of RoCE devices on all DBs and CELLs 
[SUCCESS].... RoCE devices on all DBs and CELLs look good
     Checking for RoCE Policy Routing settings on all DBs and CELLs 
[SUCCESS].... RoCE Policy Routing settings look good
     Checking for RoCE DSCP ToS mapping on all DBs and CELLs 
[SUCCESS].... RoCE DSCP ToS settings look good
     Checking for RoCE PFC settings and DSCP mapping on all DBs and CELLs
[SUCCESS].... RoCE PFC and DSCP settings look good
     Checking for RoCE interface MTU settings. Expected value : 2300
[SUCCESS].... RoCE interface MTU settings look good
     Verifying switch advertised DSCP on all DBs and CELLs ports ( )
[SUCCESS].... Advertised DSCP settings from RoCE switch looks good  
    ####  CONNECTIVITY TESTS  ####
    [COMPUTE NODES -> STORAGE CELLS] 
      (60 seconds approx.)       
    (Will walk through QoS values: 0-6) [SUCCESS]..........Results OK
[SUCCESS]....... All  can talk to all storage cells          
    [COMPUTE NODES -> COMPUTE NODES]               
...
  2. Install the spine switches (R1SS and R2SS).
    1. Physically install and power up the spine switches in the existing rack (R1SS) and the new rack (R2SS).

      1. Physically install each spine switch in RU1.

      2. For each spine switch, ensure that the management Ethernet interface is connected to the management network and then supply power.

      3. On each spine switch, perform the initial configuration steps outlined in Configuring the Cisco Nexus C9336C-FX2 Switch. Skip the step for applying the golden configuration settings as you will do this later.

      4. For each spine switch, perform a ping test to the management Ethernet interface to ensure that the switch is online and accessible.

    2. Apply the golden configuration settings to the new spine switches.

      See Applying Golden Configuration Settings on RoCE Network Fabric Switches in Oracle Exadata Database Machine Maintenance Guide.

      You can use the instance of patchmgr that you previously used to update the firmware on the leaf switches (in step 1.h).

      Use a switch list file (spines.lst) to apply the golden configuration settings to both spine switches using one patchmgr command: 

      # cat spines.lst
R1SS_IP:mspine.201
R2SS_IP:mspine.202

# ./patchmgr --roceswitches spines.lst --apply-config -log_dir /tmp/spinelogs

      Note:

      In the switch list file, R1SS_IP is the management IP address or host name for the spine switch on the existing rack (R1SS) and R2SS_IP is the management IP address or host name for the spine switch on the new rack (R2SS).

    3. Upgrade the firmware on the spine switches.

      See Updating RoCE Network Fabric Switch Firmware in Oracle Exadata Database Machine Maintenance Guide.

      You can use the instance of patchmgr that you used in the previous step.

      Use a switch list file (spines.lst) to perform the firmware upgrade on both spine switches using one patchmgr command: 

      # cat spines.lst
R1SS_IP:mspine.201
R2SS_IP:mspine.202

# ./patchmgr --roceswitches spines.lst --upgrade -log_dir /tmp/spinelogs

      Note:

      In the switch list file, R1SS_IP is the management IP address or host name for the spine switch on the existing rack (R1SS) and R2SS_IP is the management IP address or host name for the spine switch on the new rack (R2SS).

    4. Connect the RoCE Network Fabric cables to the spine switches (R1SS and R2SS).

      WARNING:

      At this stage, only connect the cables to the spine switches.

      To avoid later complications, ensure that each cable connects to the correct switch and port.

      DO NOT CONNECT ANY OF THE CABLES TO THE LEAF SWITCHES.

      Use the cables that you prepared earlier (in step 1.c).

      For the required cross-rack cabling information, see Two-Rack Cabling for X9M and Later Model Racks.

  3. Perform the first round of configuration on the lower leaf switches (R1LL and R2LL).

    Perform this step on the lower leaf switches (R1LL and R2LL) only.

    Note:

    During this step, the lower leaf switch ports are shut down. While the R1LL ports are down, R1UL exclusively supports the RoCE Network Fabric. During this time, there is no redundancy in the RoCE Network Fabric, and availability cannot be maintained if R1UL goes down.

    1. Shut down the switch ports on the lower leaf switches (R1LL and R2LL).

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# interface ethernet 1/1-36
R1LL(config-if-range)# shut
R1LL(config-if-range)# exit
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. Repeat the command sequence on R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# interface ethernet 1/1-36
R2LL(config-if-range)# shut
R2LL(config-if-range)# exit
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#
    2. Reconfigure the lower leaf switch ports (R1LL and R2LL) .

      For each switch, you must use the correct corresponding switch configuration file, which you earlier copied to the switch (in step 1.g).

      1. On R1LL, the switch configuration file name must end with step3_R1_LL.cfg: 

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# run-script bootflash:///roce_multi_14uplinks_online_step3_R1_LL.cfg | grep 'none'
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. On R2LL, the switch configuration file name must end with step3_R2_LL.cfg: 

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# run-script bootflash:///roce_multi_14uplinks_online_step3_R2_LL.cfg | grep 'none'
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#

      Note:

      This step can take approximately 5 to 8 minutes on each switch.

    3. Start the inter-switch ports on the lower leaf switches (R1LL and R2LL) .

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# interface ethernet 1/1-7, ethernet 1/30-36
R1LL(config-if-range)# no shut
R1LL(config-if-range)# exit
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. Repeat the command sequence on R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# interface ethernet 1/1-7, ethernet 1/30-36
R2LL(config-if-range)# no shut
R2LL(config-if-range)# exit
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#
    4. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    5. Verify the status of the inter-switch ports on the lower leaf switches (R1LL and R2LL) .

      Run the show interface status command on each lower leaf switch: 

      1. R1LL# show interface status
      2. R2LL# show interface status

      Examine the output to ensure that the inter-switch ports are connected.

      For example:

      R1LL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected trunk     full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected trunk     full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected trunk     full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected trunk     full   100G    QSFP-100G-CR4
Eth1/8        celadm14           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected trunk     full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected trunk     full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected trunk     full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected trunk     full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --
    6. Start the storage server ports on the lower leaf switches (R1LL and R2LL) .

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# interface ethernet 1/8-14, ethernet 1/23-29
R1LL(config-if-range)# no shut
R1LL(config-if-range)# exit
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. Repeat the command sequence on R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# interface ethernet 1/8-14, ethernet 1/23-29
R2LL(config-if-range)# no shut
R2LL(config-if-range)# exit
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#
    7. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    8. Verify the status of the storage server ports on the lower leaf switches (R1LL and R2LL).

      Run the show interface status command on each lower leaf switch: 

      1. R1LL# show interface status
      2. R2LL# show interface status

      Examine the output to ensure that the storage server ports are connected.

      For example:

      R1LL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected trunk     full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected trunk     full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected trunk     full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected trunk     full   100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           connected 3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           connected 3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           connected 3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           connected 3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           connected 3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           connected 3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           connected 3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           connected 3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           connected 3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           connected 3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           connected 3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           connected 3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           connected 3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected trunk     full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected trunk     full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected trunk     full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected trunk     full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --
    9. Start the database server ports on the lower leaf switches (R1LL and R2LL).

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# interface ethernet 1/15-22
R1LL(config-if-range)# no shut
R1LL(config-if-range)# exit
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. Repeat the command sequence on R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# interface ethernet 1/15-22
R2LL(config-if-range)# no shut
R2LL(config-if-range)# exit
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#
    10. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    11. Verify the status of the database server ports on the lower leaf switches (R1LL and R2LL).

      Run the show interface status command on each lower leaf switch: 

      1. R1LL# show interface status
      2. R2LL# show interface status

      Examine the output to ensure that the database server ports are connected.

      For example:

      R1LL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected trunk     full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected trunk     full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected trunk     full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected trunk     full   100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           connected 3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           connected 3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           connected 3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           connected 3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           connected 3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           connected 3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              connected 3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              connected 3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              connected 3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              connected 3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              connected 3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              connected 3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              connected 3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              connected 3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           connected 3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           connected 3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           connected 3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           connected 3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           connected 3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           connected 3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           connected 3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected trunk     full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected trunk     full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected trunk     full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected trunk     full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --

    Note:

    Before proceeding, ensure that you have completed all of the actions in step 3 on both lower leaf switches (R1LL and R2LL). If not, then ensure that you go back and perform the missing actions.

  4. Perform the first round of configuration on the upper leaf switches (R1UL and R2UL).

    Perform this step on the upper leaf switches (R1UL and R2UL) only.

    Note:

    At the start of this step, the upper leaf switch ports are shut down. While the R1UL ports are down, R1LL exclusively supports the RoCE Network Fabric on the existing rack. During this time, there is no redundancy in the RoCE Network Fabric, and availability cannot be maintained if R1LL goes down.

    1. Shut down the upper leaf switch ports (R1UL and R2UL).

      1. On R1UL:

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# interface ethernet 1/1-36
R1UL(config-if-range)# shut
R1UL(config-if-range)# exit
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. Repeat the command sequence on R2UL:

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# interface ethernet 1/1-36
R2UL(config-if-range)# shut
R2UL(config-if-range)# exit
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#
    2. On both racks, remove the inter-switch links between the leaf switches (R1LL to R1UL, and R2LL to R2UL).

      On every leaf switch, remove the cables for the inter-switch links:

      1. On R1LL, disconnect the inter-switch links from ports 04, 05, 06, 07, 30, 31, 32, and 33.

      2. On R1UL, disconnect the inter-switch links from ports 04, 05, 06, 07, 30, 31, 32, and 33.

      3. On R2LL, disconnect the inter-switch links from ports 04, 05, 06, 07, 30, 31, 32, and 33.

      4. On R2UL, disconnect the inter-switch links from ports 04, 05, 06, 07, 30, 31, 32, and 33.

    3. On both racks, cable the upper leaf switch to both of the spine switches (R1UL and R2UL to R1SS and R2SS).

      Connect the cables from the spine switches that you prepared earlier (in step 2.d).

      Cable the switches as described in Two-Rack Cabling for X9M and Later Model Racks:

      1. On R1UL, cable ports 01, 02, 03, 04, 05, 06, 07, 30, 31, 32, 33, 34, 35, and 36 to R1SS and R2SS.

      2. On R2UL, cable ports 01, 02, 03, 04, 05, 06, 07, 30, 31, 32, 33, 34, 35, and 36 to R1SS and R2SS.

      Note:

      Ensure that each cable connects to the correct switch and port at both ends. In addition to physically checking each connection, you can run the show lldp neighbors command on each network switch and examine the output to confirm correct connections. You can individually check each cable connection to catch and correct errors quickly.

    4. Reconfigure the upper leaf switch ports (R1UL and R2UL).

      For each switch, you must use the correct corresponding switch configuration file, which you earlier copied to the switch (in step 1.g):

      1. On R1UL, the switch configuration file name must end with step4_R1_UL.cfg: 

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# run-script bootflash:///roce_multi_14uplinks_online_step4_R1_UL.cfg | grep 'none'
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. On R2UL, the switch configuration file name must end with step4_R2_UL.cfg: 

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# run-script bootflash:///roce_multi_14uplinks_online_step4_R2_UL.cfg | grep 'none'
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#

      Note:

      This step can take approximately 5 to 8 minutes on each switch.

    5. Check the status of the RoCE Network Fabric ports on the upper leaf switches (R1UL and R2UL).

      Run the show interface status command on each upper leaf switch: 

      1. R1UL# show interface status
      2. R2UL# show interface status

      Examine the output to ensure that all of the cabled ports are disabled.

      For example:

      R1UL# show interface status

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
mgmt0         --                 connected routed    full 1000    -- 

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
Eth1/1        RouterPort1        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/2        RouterPort2        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/3        RouterPort3        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/4        RouterPort4        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/5        RouterPort5        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/6        RouterPort6        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/7        RouterPort7        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/8        celadm14           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/9        celadm13           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/10       celadm12           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/11       celadm11           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/12       celadm10           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/13       celadm09           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/14       celadm08           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/15       adm08              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/16       adm07              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/17       adm06              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/18       adm05              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/19       adm04              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/20       adm03              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/21       adm02              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/22       adm01              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/23       celadm07           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/24       celadm06           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/25       celadm05           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/26       celadm04           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/27       celadm03           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/28       celadm02           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/29       celadm01           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/30       RouterPort8        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/31       RouterPort9        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/32       RouterPort10       disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/33       RouterPort11       disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/34       RouterPort12       disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/35       RouterPort13       disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/36       RouterPort14       disabled  routed    full 100G    QSFP-100G-CR4 
Lo0           Routing loopback i connected routed    auto auto    -- 
Lo1           VTEP loopback inte connected routed    auto auto    -- 
Vlan1         --                 down      routed    auto auto    -- 
nve1          --                 connected --        auto auto    --

      Note:

      Before proceeding, ensure that you have completed all of the actions to this point in step 4 on both upper leaf switches (R1UL and R2UL). If not, then ensure that you go back and perform the missing actions.

    6. Verify the configuration of the upper leaf switches.

      You can use the instance of patchmgr that you previously used to update the switch firmware (in step 1.h).

      Use a switch list file (ul.lst) to check both upper leaf switches using one patchmgr command: 

      # cat ul.lst
R1UL_IP:mleaf_u14.102
R2UL_IP:mleaf_u14.104

      On a system with Secure Fabric enabled, use the msfleaf_u14 tag in the switch list file: 

      # cat ul.lst
R1UL_IP:msfleaf_u14.102
R2UL_IP:msfleaf_u14.104

      The following shows the recommended command and an example of the expected results:

      # ./patchmgr --roceswitches ul.lst --verify-config -log_dir /tmp/log
2020-08-10 13:40:09 -0700        :Working: Initiating config verification... Expect up to 6 minutes for each switch
Mon Aug 10 13:40:13 PDT 2020 1 of 4 :Verifying config on switch ...
...
Mon Aug 10 13:40:32 PDT 2020:	  [INFO     ] Config matches template: ...
Mon Aug 10 13:40:32 PDT 2020:	  [SUCCESS  ] Config validation successful!
2020-08-10 13:40:32 -0700        
 Config check on RoCE switch(es)
2020-08-10 13:40:32 -0700        
 Completed run of command: ./patchmgr --roceswitches ul.lst --verify-config -log_dir /tmp/log
2020-08-10 13:40:32 -0700        :INFO   : config attempted on nodes in file ul.lst: [R1UL_IP R2UL_IP]
2020-08-10 13:40:32 -0700        :INFO   : For details, check the following files in /tmp/log:
2020-08-10 13:40:32 -0700        :INFO   :  - updateRoceSwitch.log
2020-08-10 13:40:32 -0700        :INFO   :  - updateRoceSwitch.trc
2020-08-10 13:40:32 -0700        :INFO   :  - patchmgr.stdout
2020-08-10 13:40:32 -0700        :INFO   :  - patchmgr.stderr
2020-08-10 13:40:32 -0700        :INFO   :  - patchmgr.log
2020-08-10 13:40:32 -0700        :INFO   :  - patchmgr.trc
2020-08-10 13:40:32 -0700        :INFO   : Exit status:0
2020-08-10 13:40:32 -0700        :INFO   : Exiting.

      In the command output, verify that the switch configuration is good for both upper leaf switches. You can ignore messages about the ports that are down.

  5. Finalize the configuration of the lower leaf switches (R1LL and R2LL).

    Perform this step on the lower leaf switches (R1LL and R2LL) only.

    1. Reconfigure the lower leaf switch ports (R1LL and R2LL).

      Run the following command sequence on both of the lower leaf switches (R1LL and R2LL).

      You must use the correct switch configuration file, which you earlier copied to the switch (in step 1.g). In this step, the configuration file name must end with step5.cfg.

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# run-script bootflash:///roce_multi_14uplinks_online_step5.cfg | grep 'none'
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. Repeat the command sequence on R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# run-script bootflash:///roce_multi_14uplinks_online_step5.cfg | grep 'none'
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#

      Note:

      This step can take approximately 5 to 8 minutes on each switch.

    2. On both racks, cable the lower leaf switch to both of the spine switches (R1LL and R2LL to R1SS and R2SS).

      Connect the cables from the spine switches that you prepared earlier (in step 2.d).

      Cable the switches as described in Two-Rack Cabling for X9M and Later Model Racks:

      1. On R1LL, cable ports 01, 02, 03, 04, 05, 06, 07, 30, 31, 32, 33, 34, 35, and 36 to R1SS and R2SS.

      2. On R2LL, cable ports 01, 02, 03, 04, 05, 06, 07, 30, 31, 32, 33, 34, 35, and 36 to R1SS and R2SS.

      Note:

      Ensure that each cable connects to the correct switch and port at both ends. In addition to physically checking each connection, you can run the show lldp neighbors command on each network switch and examine the output to confirm correct connections. You can individually check each cable connection to catch and correct errors quickly.

    3. On the lower leaf switches, verify that all of the cabled RoCE Network Fabric ports are connected (R1LL and R2LL).

      Run the show interface status command on each lower leaf switch: 

      1. R1LL# show interface status
      2. R2LL# show interface status

      Examine the output to ensure that all of the cabled ports are connected.

      For example:

      R1LL# show interface status

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
mgmt0         --                 connected routed    full 1000    -- 

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
Eth1/1        RouterPort1        connected routed    full 100G    QSFP-100G-CR4 
Eth1/2        RouterPort2        connected routed    full 100G    QSFP-100G-CR4 
Eth1/3        RouterPort3        connected routed    full 100G    QSFP-100G-CR4 
Eth1/4        RouterPort4        connected routed    full 100G    QSFP-100G-CR4 
Eth1/5        RouterPort5        connected routed    full 100G    QSFP-100G-CR4 
Eth1/6        RouterPort6        connected routed    full 100G    QSFP-100G-CR4 
Eth1/7        RouterPort7        connected routed    full 100G    QSFP-100G-CR4 
Eth1/8        celadm14           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/9        celadm13           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/10       celadm12           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/11       celadm11           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/12       celadm10           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/13       celadm09           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/14       celadm08           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/15       adm08              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/16       adm07              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/17       adm06              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/18       adm05              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/19       adm04              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/20       adm03              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/21       adm02              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/22       adm01              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/23       celadm07           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/24       celadm06           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/25       celadm05           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/26       celadm04           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/27       celadm03           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/28       celadm02           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/29       celadm01           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/30       RouterPort8        connected routed    full 100G    QSFP-100G-CR4 
Eth1/31       RouterPort9        connected routed    full 100G    QSFP-100G-CR4 
Eth1/32       RouterPort10       connected routed    full 100G    QSFP-100G-CR4 
Eth1/33       RouterPort11       connected routed    full 100G    QSFP-100G-CR4 
Eth1/34       RouterPort12       connected routed    full 100G    QSFP-100G-CR4 
Eth1/35       RouterPort13       connected routed    full 100G    QSFP-100G-CR4 
Eth1/36       RouterPort14       connected routed    full 100G    QSFP-100G-CR4 
Lo0           Routing loopback i connected routed    auto auto    -- 
Lo1           VTEP loopback inte connected routed    auto auto    -- 
Vlan1         --                 down      routed    auto auto    -- 
nve1          --                 connected --        auto auto    --

      Note:

      Before proceeding, ensure that you have completed all of the actions to this point in step 5 on both lower leaf switches (R1LL and R2LL). If not, then ensure that you go back and perform the missing actions.

    4. Verify the configuration of the lower leaf switches.

      You can use the instance of patchmgr that you previously used to update the switch firmware (in step 1.h).

      Use a switch list file (ll.lst) to check both lower leaf switches using one patchmgr command: 

      # cat ll.lst
R1LL_IP:mleaf_u14.101
R2LL_IP:mleaf_u14.103

      On a system with Secure Fabric enabled, use the msfleaf_u14 tag in the switch list file: 

      # cat ll.lst
R1LL_IP:msfleaf_u14.101
R2LL_IP:msfleaf_u14.103

      The following shows the recommended command and an example of the expected results:

      # ./patchmgr --roceswitches ll.lst --verify-config -log_dir /tmp/log
2020-08-10 13:45:09 -0700        :Working: Initiating config verification... Expect up to 6 minutes for each switch
Mon Aug 10 13:45:13 PDT 2020 1 of 4 :Verifying config on switch ...
...
Mon Aug 10 13:45:32 PDT 2020:	  [INFO     ] Config matches template: ...
Mon Aug 10 13:45:32 PDT 2020:	  [SUCCESS  ] Config validation successful!
2020-08-10 13:45:32 -0700        
 Config check on RoCE switch(es)
2020-08-10 13:45:32 -0700        
 Completed run of command: ./patchmgr --roceswitches ll.lst --verify-config -log_dir /tmp/log
2020-08-10 13:45:32 -0700        :INFO   : config attempted on nodes in file ll.lst: [R1LL_IP R2LL_IP]
2020-08-10 13:45:32 -0700        :INFO   : For details, check the following files in /tmp/log:
2020-08-10 13:45:32 -0700        :INFO   :  - updateRoceSwitch.log
2020-08-10 13:45:32 -0700        :INFO   :  - updateRoceSwitch.trc
2020-08-10 13:45:32 -0700        :INFO   :  - patchmgr.stdout
2020-08-10 13:45:32 -0700        :INFO   :  - patchmgr.stderr
2020-08-10 13:45:32 -0700        :INFO   :  - patchmgr.log
2020-08-10 13:45:32 -0700        :INFO   :  - patchmgr.trc
2020-08-10 13:45:32 -0700        :INFO   : Exit status:0
2020-08-10 13:45:32 -0700        :INFO   : Exiting.

      In the command output, verify that the switch configuration is good for both lower leaf switches.

    5. Verify that nve is up on the lower leaf switches (R1LL and R2LL).

      Run the following command on each lower leaf switch and examine the output:

      1. R1LL# show nve peers
      2. R2LL# show nve peers

      At this point, you should see one nve peer with State=Up.

      For example:

      R1LL# show nve peers
Interface         Peer-IP State LearnType   Uptime        Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1         100.64.1.103    Up        CP 00:04:29               n/a
    6. Verify that BGP is up on the lower leaf switches (R1LL and R2LL).

      Run the following command on each lower leaf switch and examine the output:

      1. R1LL# show logging log | grep BGP
      2. R2LL# show logging log | grep BGP

      Look for two entries with Up in the rightmost column that are associated with different IP addresses.

      For example:

      R1LL# show logging log | grep BGP
2020 Aug 10 13:47:13 R1LL %BGP-5-ADJCHANGE: bgp- [29342] (default) neighbor 100.64.0.201 Up 
2020 Aug 10 13:47:24 R1LL %BGP-5-ADJCHANGE: bgp- [29342] (default) neighbor 100.64.0.202 Up
  6. Finalize the configuration of the upper leaf switches (R1UL and R2UL).

    Perform this step on the upper leaf switches (R1UL and R2UL) only.

    1. Start the inter-switch ports on the upper leaf switches (R1UL and R2UL).

      1. On R1UL:

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# interface ethernet 1/1-7, ethernet 1/30-36
R1UL(config-if-range)# no shut
R1UL(config-if-range)# exit
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. Repeat the command sequence on R2UL:

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# interface ethernet 1/1-7, ethernet 1/30-36
R2UL(config-if-range)# no shut
R2UL(config-if-range)# exit
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#
    2. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    3. Verify the status of the inter-switch ports on the upper leaf switches (R1UL and R2UL).

      Run the show interface status command on each upper leaf switch: 

      1. R1UL# show interface status
      2. R2UL# show interface status

      Examine the output to ensure that the inter-switch ports are connected.

      For example:

      R1UL# show interface status

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
mgmt0         --                 connected routed    full 1000    -- 

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
Eth1/1        RouterPort1        connected routed    full 100G    QSFP-100G-CR4 
Eth1/2        RouterPort2        connected routed    full 100G    QSFP-100G-CR4 
Eth1/3        RouterPort3        connected routed    full 100G    QSFP-100G-CR4 
Eth1/4        RouterPort4        connected routed    full 100G    QSFP-100G-CR4 
Eth1/5        RouterPort5        connected routed    full 100G    QSFP-100G-CR4 
Eth1/6        RouterPort6        connected routed    full 100G    QSFP-100G-CR4 
Eth1/7        RouterPort7        connected routed    full 100G    QSFP-100G-CR4 
Eth1/8        celadm14           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/9        celadm13           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/10       celadm12           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/11       celadm11           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/12       celadm10           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/13       celadm09           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/14       celadm08           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/15       adm08              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/16       adm07              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/17       adm06              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/18       adm05              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/19       adm04              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/20       adm03              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/21       adm02              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/22       adm01              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/23       celadm07           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/24       celadm06           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/25       celadm05           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/26       celadm04           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/27       celadm03           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/28       celadm02           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/29       celadm01           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/30       RouterPort8        connected routed    full 100G    QSFP-100G-CR4 
Eth1/31       RouterPort9        connected routed    full 100G    QSFP-100G-CR4 
Eth1/32       RouterPort10       connected routed    full 100G    QSFP-100G-CR4 
Eth1/33       RouterPort11       connected routed    full 100G    QSFP-100G-CR4 
Eth1/34       RouterPort12       connected routed    full 100G    QSFP-100G-CR4 
Eth1/35       RouterPort13       connected routed    full 100G    QSFP-100G-CR4 
Eth1/36       RouterPort14       connected routed    full 100G    QSFP-100G-CR4 
Lo0           Routing loopback i connected routed    auto auto    -- 
Lo1           VTEP loopback inte connected routed    auto auto    -- 
Vlan1         --                 down      routed    auto auto    -- 
nve1          --                 connected --        auto auto    --
    4. Start the storage server ports on the upper leaf switches (R1UL and R2UL).

      1. On R1UL:

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# interface ethernet 1/8-14, ethernet 1/23-29
R1UL(config-if-range)# no shut
R1UL(config-if-range)# exit
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. Repeat the command sequence on R2UL:

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# interface ethernet 1/8-14, ethernet 1/23-29
R2UL(config-if-range)# no shut
R2UL(config-if-range)# exit
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#
    5. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    6. Verify the status of the storage server ports on the upper leaf switches (R1UL and R2UL).

      Run the show interface status command on each upper leaf switch: 

      1. R1UL# show interface status
      2. R2UL# show interface status

      Examine the output to ensure that the storage server ports are connected.

      For example:

      R1UL# show interface status

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
mgmt0         --                 connected routed    full 1000    -- 

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
Eth1/1        RouterPort1        connected routed    full 100G    QSFP-100G-CR4 
Eth1/2        RouterPort2        connected routed    full 100G    QSFP-100G-CR4 
Eth1/3        RouterPort3        connected routed    full 100G    QSFP-100G-CR4 
Eth1/4        RouterPort4        connected routed    full 100G    QSFP-100G-CR4 
Eth1/5        RouterPort5        connected routed    full 100G    QSFP-100G-CR4 
Eth1/6        RouterPort6        connected routed    full 100G    QSFP-100G-CR4 
Eth1/7        RouterPort7        connected routed    full 100G    QSFP-100G-CR4 
Eth1/8        celadm14           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/9        celadm13           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/10       celadm12           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/11       celadm11           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/12       celadm10           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/13       celadm09           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/14       celadm08           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/15       adm08              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/16       adm07              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/17       adm06              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/18       adm05              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/19       adm04              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/20       adm03              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/21       adm02              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/22       adm01              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/23       celadm07           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/24       celadm06           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/25       celadm05           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/26       celadm04           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/27       celadm03           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/28       celadm02           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/29       celadm01           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/30       RouterPort8        connected routed    full 100G    QSFP-100G-CR4 
Eth1/31       RouterPort9        connected routed    full 100G    QSFP-100G-CR4 
Eth1/32       RouterPort10       connected routed    full 100G    QSFP-100G-CR4 
Eth1/33       RouterPort11       connected routed    full 100G    QSFP-100G-CR4 
Eth1/34       RouterPort12       connected routed    full 100G    QSFP-100G-CR4 
Eth1/35       RouterPort13       connected routed    full 100G    QSFP-100G-CR4 
Eth1/36       RouterPort14       connected routed    full 100G    QSFP-100G-CR4 
Lo0           Routing loopback i connected routed    auto auto    -- 
Lo1           VTEP loopback inte connected routed    auto auto    -- 
Vlan1         --                 down      routed    auto auto    -- 
nve1          --                 connected --        auto auto    --
    7. Start the database server ports on the upper leaf switches (R1UL and R2UL).

      1. On R1UL:

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# interface ethernet 1/15-22
R1UL(config-if-range)# no shut
R1UL(config-if-range)# exit
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. Repeat the command sequence on R2UL:

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# interface ethernet 1/15-22
R2UL(config-if-range)# no shut
R2UL(config-if-range)# exit
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#
    8. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    9. Verify the status of the database server ports on the upper leaf switches (R1UL and R2UL).

      Run the show interface status command on each upper leaf switch: 

      1. R1UL# show interface status
      2. R2UL# show interface status

      Examine the output to ensure that the database server ports are connected.

      For example:

      R1UL# show interface status

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
mgmt0         --                 connected routed    full 1000    -- 

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
Eth1/1        RouterPort1        connected routed    full 100G    QSFP-100G-CR4 
Eth1/2        RouterPort2        connected routed    full 100G    QSFP-100G-CR4 
Eth1/3        RouterPort3        connected routed    full 100G    QSFP-100G-CR4 
Eth1/4        RouterPort4        connected routed    full 100G    QSFP-100G-CR4 
Eth1/5        RouterPort5        connected routed    full 100G    QSFP-100G-CR4 
Eth1/6        RouterPort6        connected routed    full 100G    QSFP-100G-CR4 
Eth1/7        RouterPort7        connected routed    full 100G    QSFP-100G-CR4 
Eth1/8        celadm14           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/9        celadm13           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/10       celadm12           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/11       celadm11           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/12       celadm10           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/13       celadm09           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/14       celadm08           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/15       adm08              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/16       adm07              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/17       adm06              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/18       adm05              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/19       adm04              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/20       adm03              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/21       adm02              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/22       adm01              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/23       celadm07           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/24       celadm06           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/25       celadm05           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/26       celadm04           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/27       celadm03           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/28       celadm02           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/29       celadm01           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/30       RouterPort8        connected routed    full 100G    QSFP-100G-CR4 
Eth1/31       RouterPort9        connected routed    full 100G    QSFP-100G-CR4 
Eth1/32       RouterPort10       connected routed    full 100G    QSFP-100G-CR4 
Eth1/33       RouterPort11       connected routed    full 100G    QSFP-100G-CR4 
Eth1/34       RouterPort12       connected routed    full 100G    QSFP-100G-CR4 
Eth1/35       RouterPort13       connected routed    full 100G    QSFP-100G-CR4 
Eth1/36       RouterPort14       connected routed    full 100G    QSFP-100G-CR4 
Lo0           Routing loopback i connected routed    auto auto    -- 
Lo1           VTEP loopback inte connected routed    auto auto    -- 
Vlan1         --                 down      routed    auto auto    -- 
nve1          --                 connected --        auto auto    --
    10. Verify that nve is up on the leaf switches (R1LL, R1UL, R2LL, and R2UL).

      Run the following command on each leaf switch and examine the output:

      1. R1LL# show nve peers
      2. R1UL# show nve peers
      3. R2LL# show nve peers
      4. R2UL# show nve peers

      In the output, you should see three nve peers with State=Up.

      For example:

      R1UL# show nve peers
Interface         Peer-IP State LearnType   Uptime        Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1         100.64.1.101    Up        CP 00:04:29               n/a
nve1         100.64.1.103    Up        CP 00:07:48               n/a
nve1         100.64.1.104    Up        CP 00:04:10               n/a
    11. Verify that BGP is up on the upper leaf switches (R1UL and R2UL).

      Run the following command on each upper leaf switch and examine the output:

      1. R1UL# show logging log | grep BGP
      2. R2UL# show logging log | grep BGP

      In the output, look for two entries with Up in the rightmost column that are associated with different IP addresses.

      For example:

      R1UL# show logging log | grep BGP
2020 Aug 10 13:57:13 R1UL %BGP-5-ADJCHANGE: bgp- [32782] (default) neighbor 100.64.0.201 Up 
2020 Aug 10 13:57:24 R1UL %BGP-5-ADJCHANGE: bgp- [32782] (default) neighbor 100.64.0.202 Up
  7. For each rack (R1 and R2), confirm the multi-rack cabling by running the verify_roce_cables.py script.

    The verify_roce_cables.py script uses two input files; one for database servers and storage servers (nodes.rackN), and another for switches (switches.rackN). In each file, every server or switch must be listed on separate lines. Use fully qualified domain names or IP addresses for each server and switch.

    See My Oracle Support document 2587717.1 for download and detailed usage instructions.

    Run the verify_roce_cables.py script against both of the racks: 

    1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
    2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack2 -s switches.rack2

    Check the output of the verify_roce_cables.py script against the tables in Two-Rack Cabling for X9M and Later Model Racks. Also, check that output in the CABLE OK? columns contains the OK status.

    The following examples show extracts of the expected command results:

    # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
SWITCH PORT (EXPECTED PEER)  LOWER LEAF (rack1sw-rocea0) : CABLE OK?  UPPER LEAF (rack1sw-roceb0) : CABLE OK?
----------- ---------------  --------------------------- : ---------  --------------------------- : ---------
...
    # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack2 -s switches.rack2
SWITCH PORT (EXPECTED PEER)  LOWER LEAF (rack2sw-rocea0) : CABLE OK?  UPPER LEAF (rack2sw-roceb0) : CABLE OK?
----------- ---------------  --------------------------- : ---------  --------------------------- : ---------
...
  8. Verify the RoCE Network Fabric operation across both interconnected racks by using the infinicheck command.

    Use the following recommended command sequence to verify the RoCE Network Fabric operation across both racks.

    In each command, hosts.all contains a list of database server RoCE Network Fabric IP addresses from both racks (2 RoCE Network Fabric IP addresses for each database server), and cells.all contains a list of RoCE Network Fabric IP addresses for the storage servers from both racks (2 RoCE Network Fabric IP addresses for each storage server). 

    1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.all -c cells.all -z
    2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.all -c cells.all -s
    3. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.all -c cells.all -b

    See step 1.k for most information about each infinicheck command.

    The following example shows the expected command results for the final command in the sequence:

    # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.all -c cells.all -b

INFINICHECK                    
        [Network Connectivity, Configuration and Performance]        
               
          ####  FABRIC TYPE TESTS  #### 
System type identified: RoCE
Verifying User Equivalance of user=root from all DBs to all CELLs.
     ####  RoCE CONFIGURATION TESTS  ####       
     Checking for presence of RoCE devices on all DBs and CELLs 
[SUCCESS].... RoCE devices on all DBs and CELLs look good
     Checking for RoCE Policy Routing settings on all DBs and CELLs 
[SUCCESS].... RoCE Policy Routing settings look good
     Checking for RoCE DSCP ToS mapping on all DBs and CELLs 
[SUCCESS].... RoCE DSCP ToS settings look good
     Checking for RoCE PFC settings and DSCP mapping on all DBs and CELLs
[SUCCESS].... RoCE PFC and DSCP settings look good
     Checking for RoCE interface MTU settings. Expected value : 2300
[SUCCESS].... RoCE interface MTU settings look good
     Verifying switch advertised DSCP on all DBs and CELLs ports ( )
[SUCCESS].... Advertised DSCP settings from RoCE switch looks good  
    ####  CONNECTIVITY TESTS  ####
    [COMPUTE NODES -> STORAGE CELLS] 
      (60 seconds approx.)       
    (Will walk through QoS values: 0-6) [SUCCESS]..........Results OK
[SUCCESS]....... All  can talk to all storage cells          
    [COMPUTE NODES -> COMPUTE NODES]               
...

At this point, both racks share the RoCE Network Fabric, and the combined system is ready for further configuration.

See Configuring the New Hardware.

2.3.2.1.2 Extending an X8M Rack with No Down Time by Adding an X9M or Later Model Rack

WARNING:

Take time to read and understand this procedure before implementation. Pay careful attention to the instructions that surround the command examples. A system outage may occur if the procedure is not applied correctly.

Note:

This procedure assumes that the RoCE Network Fabric switches on the existing X8M rack contain the golden configuration settings from Oracle Exadata System Software 20.1.0 or later. Otherwise, before using this procedure, you must update the Oracle Exadata System Software and update the golden configuration settings on the RoCE Network Fabric switches. Downtime is required to update the golden configuration settings on the RoCE Network Fabric switches.

Note:

For additional background information, see Understanding Multi-Rack Cabling for X8M Racks and Understanding Multi-Rack Cabling for X9M and Later Model Racks.

Use this procedure to extend a typical X8M rack without down-time by cabling it together with an X9M or later model rack. The primary rack (designated R1) and all of the systems it supports remain online throughout the procedure. At the beginning of the procedure, the additional rack (designated R2) is shut down.

The following is an outline of the procedure:

  • Preparation (steps 1 and 2)

    In this phase, you prepare the racks, switches, and cables. Also, you install and cable the spine switches in both racks.

  • Configuration and Physical Cabling

    In this phase, you reconfigure the leaf switches and finalize the cabling to the spine switches. These tasks are carefully orchestrated to avoid downtime on the primary system, as follows:

    • Partially configure the lower leaf switches (step 3)

      In this step, you reconfigure the switch ports on the lower leaf switches. There is no physical cabling performed in this step.

    • Partially configure the upper leaf switches (step 4)

      In this step, you reconfigure the switch ports on the upper leaf switches, remove the inter-switch cables that connect the leaf switches in both racks and connect the cables between the upper leaf switches and the spine switches.

    • Finalize the lower leaf switches (step 5)

      In this step, you finalize the switch port configuration on the lower leaf switches. You also complete the physical cabling by connecting the cables between the lower leaf switches and the spine switches.

    • Finalize the upper leaf switches (step 6)

      In this step, you finalize the switch port configuration on the upper leaf switches.

  • Validation and Testing (steps 7 and 8)

    In this phase, you validate and test the RoCE Network Fabric across both of the interconnect racks.

After completing the procedure, both racks share the RoCE Network Fabric, and the combined system is ready for further configuration. For example, you can extend existing disk groups and Oracle RAC databases to consume resources across both racks.

Note:

  • This procedure applies only to typical rack configurations that initially have leaf switches with the following specifications:

    • The inter-switch ports are ports 4 to 7, and ports 30 to 33.

    • The storage server ports are ports 8 to 14, and ports 23 to 29.

    • The database server ports are ports 15 to 22.

    For other rack configurations (for example, 8-socket systems with three database servers and 11 storage servers) a different procedure and different RoCE Network Fabric switch configuration files are required. Contact Oracle for further guidance.

  • The procedure uses the following naming abbreviations and conventions:

    • The abbreviation for the existing X8M rack is R1, and the new X9M or later model rack is R2.

    • LL identifies a lower leaf switch and UL identifies an upper leaf switch.

    • SS identifies a spine switch.

    • A specific switch is identified by combining abbreviations. For example, R1LL identifies the lower leaf switch (LL) on the existing rack (R1).

  • Most operations must be performed in multiple locations. For example, step 1.h instructs you to update the firmware on all the RoCE Network Fabric leaf switches (R1LL, R1UL, R2LL, and R2UL). Pay attention to the instructions and keep track of your actions.

    Tip:

    When a step must be performed on multiple switches, the instruction contains a list of the applicable switches. For example, (R1LL, R1UL, R2LL, and R2UL). You can use this list as a checklist to keep track of your actions.

  • Perform operations sequentially, and complete every operation before proceeding. For example, run the entire command sequence at 3.a.i as one operation and complete it before proceeding.

  • All of commands that are run on a RoCE Network Fabric switch must be run while connected to the switch management interface as the switch administrator.

  1. Prepare the systems.
    1. Position the new X9M or later model rack (R2) so that it is physically near the existing X8M rack (R1).

      The RDMA Network Fabric cables must be able to reach the switches in each rack.

      For the required cross-rack cabling information, see Two-Rack Cabling for a System Combining an X8M Rack and a Later Model Rack.

    2. Power on all of the servers and network switches in the new rack (R2).

      This includes the database servers, storage servers, RoCE Network Fabric leaf switches, and the Management Network Switch.

    3. Prepare the RoCE Network Fabric cables that you will use to interconnect the racks.

      Label both ends of every cable.

      For the required cross-rack cabling information, see Two-Rack Cabling for a System Combining an X8M Rack and a Later Model Rack.

    4. Connect the new rack (R2) to your existing management network.

      Ensure that there are no IP address conflicts across the racks and that you can access the management interfaces on the RoCE Network Fabric switches.

    5. Ensure that you have a backup of the current switch configuration for each RoCE Network Fabric switch (R1LL, R1UL, R2LL, and R2UL).

      See Backing Up Settings on the RoCE Network Fabric Switch in Oracle Exadata Database Machine Maintenance Guide.

    6. Download the required RoCE Network Fabric switch configuration files.

      This procedure requires specific RoCE Network Fabric switch configuration files, which you must download from My Oracle Support document 2704997.1.

      WARNING:

      You must use different switch configuration files depending on whether your system uses Exadata Secure RDMA Fabric Isolation. Ensure that you download the correct archive that matches your system configuration.

      For system configurations without Secure Fabric, download online_multi-rack_8and14uplinks.zip. For system configurations with Secure Fabric, download online_SF_enabled_multi-rack_8and14uplinks.zip.

      Download and extract the archive containing the required RoCE Network Fabric switch configuration files. Place the files on a server with access to the management interfaces on the RoCE Network Fabric switches.

    7. Copy the required RoCE Network Fabric switch configuration files to the leaf switches on both racks.

      You can use the following commands to copy the required configuration files to all of the RoCE Network Fabric switches on a system without Secure Fabric enabled: 

      1. # scp roce_multi_online_step3_R1_LL.cfg admin@R1LL_IP:/
      2. # scp roce_multi_14uplinks_online_step3_R2_LL.cfg admin@R2LL_IP:/
      3. # scp roce_multi_online_step4_R1_UL.cfg admin@R1UL_IP:/
      4. # scp roce_multi_14uplinks_online_step4_R2_UL.cfg admin@R2UL_IP:/
      5. # scp roce_multi_online_step5.cfg admin@R1LL_IP:/
      6. # scp roce_multi_14uplinks_online_step5.cfg admin@R2LL_IP:/

      On a system with Secure Fabric enabled, you can use the following commands: 

      1. # scp roce_SF_multi_online_step3_R1_LL.cfg admin@R1LL_IP:/
      2. # scp roce_SF_multi_14uplinks_online_step3_R2_LL.cfg admin@R2LL_IP:/
      3. # scp roce_SF_multi_online_step4_R1_UL.cfg admin@R1UL_IP:/
      4. # scp roce_SF_multi_14uplinks_online_step4_R2_UL.cfg admin@R2UL_IP:/
      5. # scp roce_SF_multi_online_step5.cfg admin@R1LL_IP:/
      6. # scp roce_SF_multi_14uplinks_online_step5.cfg admin@R2LL_IP:/

      In the above commands, substitute the appropriate IP address or host name where applicable. For example, in place of R1LL_IP, substitute the management IP address or host name for the lower leaf switch (LL) on the existing rack (R1).

      Note:

      The command examples in the rest of this procedure use the configuration files for a system configuration without Secure Fabric enabled. If required, adjust the commands to use the Secure Fabric-enabled switch configuration files.
    8. Update the firmware to the latest available release on all of the RoCE Network Fabric leaf switches (R1LL, R1UL, R2LL, and R2UL).

      See Updating RoCE Network Fabric Switch Firmware in Oracle Exadata Database Machine Maintenance Guide.

    9. Examine the RoCE Network Fabric leaf switches (R1LL, R1UL, R2LL, and R2UL) and confirm the port categories for the cabled ports.

      Run the show interface status command on every RoCE Network Fabric leaf switch: 

      1. R1LL# show interface status
      2. R1UL# show interface status
      3. R2LL# show interface status
      4. R2UL# show interface status

      Examine the output and confirm the port categories as follows:

      • Confirm that the inter-switch ports are ports 4 to 7, and ports 30 to 33.

      • Confirm that the storage server ports are ports 8 to 14, and ports 23 to 29.

      • Confirm that the database server ports are ports 15 to 22.

      For example:

      R1LL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected trunk     full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected trunk     full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected trunk     full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected trunk     full   100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           connected 3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           connected 3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           connected 3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           connected 3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           connected 3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           connected 3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              connected 3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              connected 3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              connected 3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              connected 3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              connected 3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              connected 3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              connected 3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              connected 3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           connected 3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           connected 3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           connected 3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           connected 3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           connected 3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           connected 3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           connected 3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected trunk     full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected trunk     full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected trunk     full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected trunk     full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --
    10. For each rack (R1 and R2), confirm the RoCE Network Fabric cabling by running the verify_roce_cables.py script.

      The verify_roce_cables.py script uses two input files; one for database servers and storage servers (nodes.rackN), and another for switches (switches.rackN). In each file, every server or switch must be listed on separate lines. Use fully qualified domain names or IP addresses for each server and switch.

      See My Oracle Support document 2587717.1 for download and detailed usage instructions.

      Run the verify_roce_cables.py script against both of the racks: 

      1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
      2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack2 -s switches.rack2

      Check that output in the CABLE OK? columns contains the OK status.

      The following example shows the expected command results:

      # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
SWITCH PORT (EXPECTED PEER)               LOWER LEAF (rack1sw-rocea0) : CABLE OK?             UPPER LEAF (rack1sw-roceb0) : CABLE OK?
----------- ---------------          -------------------------------- : --------         -------------------------------- : ---------
    Eth1/4 (ISL peer switch)   :           rack1sw-rocea0 Ethernet1/4 : OK                     rack1sw-roceb0 Ethernet1/4 : OK
    Eth1/5 (ISL peer switch)   :           rack1sw-rocea0 Ethernet1/5 : OK                     rack1sw-roceb0 Ethernet1/5 : OK
    Eth1/6 (ISL peer switch)   :           rack1sw-rocea0 Ethernet1/6 : OK                     rack1sw-roceb0 Ethernet1/6 : OK
    Eth1/7 (ISL peer switch)   :           rack1sw-rocea0 Ethernet1/7 : OK                     rack1sw-roceb0 Ethernet1/7 : OK
    Eth1/8 (RU39)              :                 rack1celadm14 port-1 : OK                           rack1celadm14 port-2 : OK
    Eth1/9 (RU37)              :                 rack1celadm13 port-1 : OK                           rack1celadm13 port-2 : OK
   Eth1/10 (RU35)              :                 rack1celadm12 port-1 : OK                           rack1celadm12 port-2 : OK
   Eth1/11 (RU33)              :                 rack1celadm11 port-1 : OK                           rack1celadm11 port-2 : OK
   Eth1/12 (RU31)              :                 rack1celadm10 port-1 : OK                           rack1celadm10 port-2 : OK
   Eth1/13 (RU29)              :                 rack1celadm09 port-1 : OK                           rack1celadm09 port-2 : OK
   Eth1/14 (RU27)              :                 rack1celadm08 port-1 : OK                           rack1celadm08 port-2 : OK
   Eth1/15 (RU26)              :                    rack1adm08 port-1 : OK                              rack1adm08 port-2 : OK
   Eth1/16 (RU25)              :                    rack1adm07 port-1 : OK                              rack1adm07 port-2 : OK
   Eth1/17 (RU24)              :                    rack1adm06 port-1 : OK                              rack1adm06 port-2 : OK
   Eth1/18 (RU23)              :                    rack1adm05 port-1 : OK                              rack1adm05 port-2 : OK
   Eth1/19 (RU19)              :                    rack1adm04 port-1 : OK                              rack1adm04 port-2 : OK
   Eth1/20 (RU18)              :                    rack1adm03 port-1 : OK                              rack1adm03 port-2 : OK
   Eth1/21 (RU17)              :                    rack1adm02 port-1 : OK                              rack1adm02 port-2 : OK
   Eth1/22 (RU16)              :                    rack1adm01 port-1 : OK                              rack1adm01 port-2 : OK
   Eth1/23 (RU14)              :                 rack1celadm07 port-1 : OK                           rack1celadm07 port-2 : OK
   Eth1/24 (RU12)              :                 rack1celadm06 port-1 : OK                           rack1celadm06 port-2 : OK
   Eth1/25 (RU10)              :                 rack1celadm05 port-1 : OK                           rack1celadm05 port-2 : OK
   Eth1/26 (RU08)              :                 rack1celadm04 port-1 : OK                           rack1celadm04 port-2 : OK
   Eth1/27 (RU06)              :                 rack1celadm03 port-1 : OK                           rack1celadm03 port-2 : OK
   Eth1/28 (RU04)              :                 rack1celadm02 port-1 : OK                           rack1celadm02 port-2 : OK
   Eth1/29 (RU02)              :                 rack1celadm01 port-1 : OK                           rack1celadm01 port-2 : OK
   Eth1/30 (ISL peer switch)   :          rack1sw-rocea0 Ethernet1/30 : OK                    rack1sw-roceb0 Ethernet1/30 : OK
   Eth1/31 (ISL peer switch)   :          rack1sw-rocea0 Ethernet1/31 : OK                    rack1sw-roceb0 Ethernet1/31 : OK
   Eth1/32 (ISL peer switch)   :          rack1sw-rocea0 Ethernet1/32 : OK                    rack1sw-roceb0 Ethernet1/32 : OK
   Eth1/33 (ISL peer switch)   :          rack1sw-rocea0 Ethernet1/33 : OK                    rack1sw-roceb0 Ethernet1/33 : OK
    11. For each rack (R1 and R2), verify the RoCE Network Fabric operation by using the infinicheck command.

      • Use infinicheck with the -z option to clear the files that were created during the last run of the infinicheck command.

      • Use infinicheck with the -s option to set up user equivalence for password-less SSH across the RoCE Network Fabric.

      • Finally, verify the RoCE Network Fabric operation by using infinicheck with the -b option, which is recommended on newly imaged machines where it is acceptable to suppress the cellip.ora and cellinit.ora configuration checks.

      In each command, the hosts input file (hosts.rack1 and hosts.rack2) contains a list of database server RoCE Network Fabric IP addresses (2 RoCE Network Fabric IP addresses for each database server), and the cells input file (cells.rack1 and cells.rack2) contains a list of RoCE Network Fabric IP addresses for the storage servers (2 RoCE Network Fabric IP addresses for each storage server).

      1. Use the following recommended command sequence on the existing rack (R1):

        1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack1 -c cells.rack1 -z
        2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack1 -c cells.rack1 -s
        3. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack1 -c cells.rack1 -b

      2. Use the following recommended command sequence on the new rack (R2):

        1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack2 -c cells.rack2 -z
        2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack2 -c cells.rack2 -s
        3. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack2 -c cells.rack2 -b

      The following example shows the expected command results for the final command in the sequence:

      # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rackN -c cells.rackN -b

INFINICHECK                    
        [Network Connectivity, Configuration and Performance]        
               
          ####  FABRIC TYPE TESTS  #### 
System type identified: RoCE
Verifying User Equivalance of user=root from all DBs to all CELLs.
     ####  RoCE CONFIGURATION TESTS  ####       
     Checking for presence of RoCE devices on all DBs and CELLs 
[SUCCESS].... RoCE devices on all DBs and CELLs look good
     Checking for RoCE Policy Routing settings on all DBs and CELLs 
[SUCCESS].... RoCE Policy Routing settings look good
     Checking for RoCE DSCP ToS mapping on all DBs and CELLs 
[SUCCESS].... RoCE DSCP ToS settings look good
     Checking for RoCE PFC settings and DSCP mapping on all DBs and CELLs
[SUCCESS].... RoCE PFC and DSCP settings look good
     Checking for RoCE interface MTU settings. Expected value : 2300
[SUCCESS].... RoCE interface MTU settings look good
     Verifying switch advertised DSCP on all DBs and CELLs ports ( )
[SUCCESS].... Advertised DSCP settings from RoCE switch looks good  
    ####  CONNECTIVITY TESTS  ####
    [COMPUTE NODES -> STORAGE CELLS] 
      (60 seconds approx.)       
    (Will walk through QoS values: 0-6) [SUCCESS]..........Results OK
[SUCCESS]....... All  can talk to all storage cells          
    [COMPUTE NODES -> COMPUTE NODES]               
...
  2. Install the spine switches (R1SS and R2SS).
    1. Physically install and power up the spine switches in the existing rack (R1SS) and the new rack (R2SS).

      1. Physically install each spine switch in RU1.

      2. For each spine switch, ensure that the management Ethernet interface is connected to the management network and then supply power.

      3. On each spine switch, perform the initial configuration steps outlined in Configuring the Cisco Nexus C9336C-FX2 Switch. Skip the step for applying the golden configuration settings as you will do this later.

      4. For each spine switch, perform a ping test to the management Ethernet interface to ensure that the switch is online and accessible.

    2. Apply the golden configuration settings to the new spine switches.

      See Applying Golden Configuration Settings on RoCE Network Fabric Switches in Oracle Exadata Database Machine Maintenance Guide.

      You can use the instance of patchmgr that you previously used to update the firmware on the leaf switches (in step 1.h).

      Use a switch list file (spines.lst) to apply the golden configuration settings to both spine switches using one patchmgr command: 

      # cat spines.lst
R1SS_IP:mspine.201
R2SS_IP:mspine.202

# ./patchmgr --roceswitches spines.lst --apply-config -log_dir /tmp/spinelogs

      Note:

      In the switch list file, R1SS_IP is the management IP address or host name for the spine switch on the existing rack (R1SS) and R2SS_IP is the management IP address or host name for the spine switch on the new rack (R2SS).

    3. Upgrade the firmware on the spine switches.

      See Updating RoCE Network Fabric Switch Firmware in Oracle Exadata Database Machine Maintenance Guide.

      You can use the instance of patchmgr that you used in the previous step.

      Use a switch list file (spines.lst) to perform the firmware upgrade on both spine switches using one patchmgr command: 

      # cat spines.lst
R1SS_IP:mspine.201
R2SS_IP:mspine.202

# ./patchmgr --roceswitches spines.lst --upgrade -log_dir /tmp/spinelogs

      Note:

      In the switch list file, R1SS_IP is the management IP address or host name for the spine switch on the existing rack (R1SS) and R2SS_IP is the management IP address or host name for the spine switch on the new rack (R2SS).

    4. Connect the RoCE Network Fabric cables to the spine switches (R1SS and R2SS).

      WARNING:

      At this stage, only connect the cables to the spine switches.

      To avoid later complications, ensure that each cable connects to the correct switch and port.

      DO NOT CONNECT ANY OF THE CABLES TO THE LEAF SWITCHES.

      Use the cables that you prepared earlier (in step 1.c).

      For the required cross-rack cabling information, see Two-Rack Cabling for a System Combining an X8M Rack and a Later Model Rack.

  3. Perform the first round of configuration on the lower leaf switches (R1LL and R2LL).

    Perform this step on the lower leaf switches (R1LL and R2LL) only.

    Note:

    During this step, the lower leaf switch ports are shut down. While the R1LL ports are down, R1UL exclusively supports the RoCE Network Fabric. During this time, there is no redundancy in the RoCE Network Fabric, and availability cannot be maintained if R1UL goes down.

    1. Shut down the switch ports on the lower leaf switches (R1LL and R2LL).

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# interface ethernet 1/4-33
R1LL(config-if-range)# shut
R1LL(config-if-range)# exit
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. On R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# interface ethernet 1/1-36
R2LL(config-if-range)# shut
R2LL(config-if-range)# exit
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#
    2. Reconfigure the lower leaf switch ports (R1LL and R2LL) .

      For each switch, you must use the correct corresponding switch configuration file, which you earlier copied to the switch (in step 1.g).

      1. On R1LL, the switch configuration file name must end with step3_R1_LL.cfg: 

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# run-script bootflash:///roce_multi_online_step3_R1_LL.cfg | grep 'none'
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. On R2LL, the switch configuration file name must end with step3_R2_LL.cfg: 

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# run-script bootflash:///roce_multi_14uplinks_online_step3_R2_LL.cfg | grep 'none'
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#

      Note:

      This step can take approximately 5 to 8 minutes on each switch.

    3. Start the inter-switch ports on the lower leaf switches (R1LL and R2LL) .

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# interface ethernet 1/4-7, ethernet 1/30-33
R1LL(config-if-range)# no shut
R1LL(config-if-range)# exit
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. On R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# interface ethernet 1/1-7, ethernet 1/30-36
R2LL(config-if-range)# no shut
R2LL(config-if-range)# exit
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#
    4. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    5. Verify the status of the inter-switch ports on the lower leaf switches (R1LL and R2LL) .

      Run the show interface status command on each lower leaf switch: 

      1. R1LL# show interface status
      2. R2LL# show interface status

      Examine the output to ensure that the inter-switch ports are connected.

      For example:

      R1LL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected trunk     full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected trunk     full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected trunk     full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected trunk     full   100G    QSFP-100G-CR4
Eth1/8        celadm14           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected trunk     full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected trunk     full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected trunk     full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected trunk     full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --
    6. Start the storage server ports on the lower leaf switches (R1LL and R2LL) .

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# interface ethernet 1/8-14, ethernet 1/23-29
R1LL(config-if-range)# no shut
R1LL(config-if-range)# exit
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. Repeat the command sequence on R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# interface ethernet 1/8-14, ethernet 1/23-29
R2LL(config-if-range)# no shut
R2LL(config-if-range)# exit
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#
    7. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    8. Verify the status of the storage server ports on the lower leaf switches (R1LL and R2LL).

      Run the show interface status command on each lower leaf switch: 

      1. R1LL# show interface status
      2. R2LL# show interface status

      Examine the output to ensure that the storage server ports are connected.

      For example:

      R1LL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected trunk     full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected trunk     full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected trunk     full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected trunk     full   100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           connected 3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           connected 3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           connected 3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           connected 3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           connected 3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           connected 3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           connected 3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           connected 3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           connected 3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           connected 3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           connected 3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           connected 3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           connected 3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected trunk     full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected trunk     full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected trunk     full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected trunk     full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --
    9. Start the database server ports on the lower leaf switches (R1LL and R2LL).

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# interface ethernet 1/15-22
R1LL(config-if-range)# no shut
R1LL(config-if-range)# exit
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. Repeat the command sequence on R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# interface ethernet 1/15-22
R2LL(config-if-range)# no shut
R2LL(config-if-range)# exit
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#
    10. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    11. Verify the status of the database server ports on the lower leaf switches (R1LL and R2LL).

      Run the show interface status command on each lower leaf switch: 

      1. R1LL# show interface status
      2. R2LL# show interface status

      Examine the output to ensure that the database server ports are connected.

      For example:

      R1LL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected trunk     full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected trunk     full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected trunk     full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected trunk     full   100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           connected 3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           connected 3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           connected 3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           connected 3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           connected 3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           connected 3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              connected 3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              connected 3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              connected 3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              connected 3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              connected 3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              connected 3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              connected 3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              connected 3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           connected 3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           connected 3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           connected 3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           connected 3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           connected 3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           connected 3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           connected 3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected trunk     full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected trunk     full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected trunk     full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected trunk     full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --

    Note:

    Before proceeding, ensure that you have completed all of the actions in step 3 on both lower leaf switches (R1LL and R2LL). If not, then ensure that you go back and perform the missing actions.

  4. Perform the first round of configuration on the upper leaf switches (R1UL and R2UL).

    Perform this step on the upper leaf switches (R1UL and R2UL) only.

    Note:

    At the start of this step, the upper leaf switch ports are shut down. While the R1UL ports are down, R1LL exclusively supports the RoCE Network Fabric on the existing rack. During this time, there is no redundancy in the RoCE Network Fabric, and availability cannot be maintained if R1LL goes down.

    1. Shut down the upper leaf switch ports (R1UL and R2UL).

      1. On R1UL:

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# interface ethernet 1/4-33
R1UL(config-if-range)# shut
R1UL(config-if-range)# exit
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. On R2UL:

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# interface ethernet 1/1-36
R2UL(config-if-range)# shut
R2UL(config-if-range)# exit
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#
    2. On both racks, remove the inter-switch links between the leaf switches (R1LL to R1UL, and R2LL to R2UL).

      On every leaf switch, remove the cables for the inter-switch links:

      1. On R1LL, disconnect the inter-switch links from ports 04, 05, 06, 07, 30, 31, 32, and 33.

      2. On R1UL, disconnect the inter-switch links from ports 04, 05, 06, 07, 30, 31, 32, and 33.

      3. On R2LL, disconnect the inter-switch links from ports 04, 05, 06, 07, 30, 31, 32, and 33.

      4. On R2UL, disconnect the inter-switch links from ports 04, 05, 06, 07, 30, 31, 32, and 33.

    3. On both racks, cable the upper leaf switch to both of the spine switches (R1UL and R2UL to R1SS and R2SS).

      Connect the cables from the spine switches that you prepared earlier (in step 2.d).

      Cable the switches as described in Two-Rack Cabling for a System Combining an X8M Rack and a Later Model Rack:

      1. On R1UL, cable ports 04, 05, 06, 07, 30, 31, 32, and 33 to R1SS and R2SS.

      2. On R2UL, cable ports 01, 02, 03, 04, 05, 06, 07, 30, 31, 32, 33, 34, 35, and 36 to R1SS and R2SS.

      Note:

      Ensure that each cable connects to the correct switch and port at both ends. In addition to physically checking each connection, you can run the show lldp neighbors command on each network switch and examine the output to confirm correct connections. You can individually check each cable connection to catch and correct errors quickly.

    4. Reconfigure the upper leaf switch ports (R1UL and R2UL).

      For each switch, you must use the correct corresponding switch configuration file, which you earlier copied to the switch (in step 1.g):

      1. On R1UL, the switch configuration file name must end with step4_R1_UL.cfg: 

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# run-script bootflash:///roce_multi_online_step4_R1_UL.cfg | grep 'none'
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. On R2UL, the switch configuration file name must end with step4_R2_UL.cfg: 

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# run-script bootflash:///roce_multi_14uplinks_online_step4_R2_UL.cfg | grep 'none'
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#

      Note:

      This step can take approximately 5 to 8 minutes on each switch.

    5. Check the status of the RoCE Network Fabric ports on the upper leaf switches (R1UL and R2UL).

      Run the show interface status command on each upper leaf switch: 

      1. R1UL# show interface status
      2. R2UL# show interface status

      Examine the output to ensure that all of the cabled ports are disabled.

      The following example shows the expected output on the X9M or later model rack (R2UL). On the X8M rack (R1UL), ports 01, 02, 03, 34, 35, and 36 are not physically connected. 

      R2UL# show interface status

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
mgmt0         --                 connected routed    full 1000    -- 

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
Eth1/1        RouterPort1        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/2        RouterPort2        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/3        RouterPort3        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/4        RouterPort4        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/5        RouterPort5        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/6        RouterPort6        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/7        RouterPort7        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/8        celadm14           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/9        celadm13           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/10       celadm12           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/11       celadm11           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/12       celadm10           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/13       celadm09           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/14       celadm08           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/15       adm08              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/16       adm07              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/17       adm06              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/18       adm05              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/19       adm04              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/20       adm03              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/21       adm02              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/22       adm01              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/23       celadm07           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/24       celadm06           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/25       celadm05           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/26       celadm04           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/27       celadm03           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/28       celadm02           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/29       celadm01           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/30       RouterPort8        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/31       RouterPort9        disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/32       RouterPort10       disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/33       RouterPort11       disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/34       RouterPort12       disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/35       RouterPort13       disabled  routed    full 100G    QSFP-100G-CR4 
Eth1/36       RouterPort14       disabled  routed    full 100G    QSFP-100G-CR4 
Lo0           Routing loopback i connected routed    auto auto    -- 
Lo1           VTEP loopback inte connected routed    auto auto    -- 
Vlan1         --                 down      routed    auto auto    -- 
nve1          --                 connected --        auto auto    --

      Note:

      Before proceeding, ensure that you have completed all of the actions to this point in step 4 on both upper leaf switches (R1UL and R2UL). If not, then ensure that you go back and perform the missing actions.

    6. Verify the configuration of the upper leaf switches.

      You can use the instance of patchmgr that you previously used to update the switch firmware (in step 1.h).

      Use a switch list file (ul.lst) to check both upper leaf switches using one patchmgr command: 

      # cat ul.lst
R1UL_IP:mleaf.102
R2UL_IP:mleaf_u14.104

      On a system with Secure Fabric enabled, use the msfleaf and msfleaf_u14 tags in the switch list file: 

      # cat ul.lst
R1UL_IP:msfleaf.102
R2UL_IP:msfleaf_u14.104

      The following shows the recommended command and an example of the expected results:

      # ./patchmgr --roceswitches ul.lst --verify-config -log_dir /tmp/log
2020-08-10 13:40:09 -0700        :Working: Initiating config verification... Expect up to 6 minutes for each switch
Mon Aug 10 13:40:13 PDT 2020 1 of 4 :Verifying config on switch ...
...
Mon Aug 10 13:40:32 PDT 2020:	  [INFO     ] Config matches template: ...
Mon Aug 10 13:40:32 PDT 2020:	  [SUCCESS  ] Config validation successful!
2020-08-10 13:40:32 -0700        
 Config check on RoCE switch(es)
2020-08-10 13:40:32 -0700        
 Completed run of command: ./patchmgr --roceswitches ul.lst --verify-config -log_dir /tmp/log
2020-08-10 13:40:32 -0700        :INFO   : config attempted on nodes in file ul.lst: [R1UL_IP R2UL_IP]
2020-08-10 13:40:32 -0700        :INFO   : For details, check the following files in /tmp/log:
2020-08-10 13:40:32 -0700        :INFO   :  - updateRoceSwitch.log
2020-08-10 13:40:32 -0700        :INFO   :  - updateRoceSwitch.trc
2020-08-10 13:40:32 -0700        :INFO   :  - patchmgr.stdout
2020-08-10 13:40:32 -0700        :INFO   :  - patchmgr.stderr
2020-08-10 13:40:32 -0700        :INFO   :  - patchmgr.log
2020-08-10 13:40:32 -0700        :INFO   :  - patchmgr.trc
2020-08-10 13:40:32 -0700        :INFO   : Exit status:0
2020-08-10 13:40:32 -0700        :INFO   : Exiting.

      In the command output, verify that the switch configuration is good for both upper leaf switches. You can ignore messages about the ports that are down.

  5. Finalize the configuration of the lower leaf switches (R1LL and R2LL).

    Perform this step on the lower leaf switches (R1LL and R2LL) only.

    1. Reconfigure the lower leaf switch ports (R1LL and R2LL).

      Run the following command sequence on both of the lower leaf switches (R1LL and R2LL).

      You must use the correct switch configuration file, which you earlier copied to the switch (in step 1.g). In this step, the configuration file name must end with step5.cfg.

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# run-script bootflash:///roce_multi_online_step5.cfg | grep 'none'
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. On R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# run-script bootflash:///roce_multi_14uplinks_online_step5.cfg | grep 'none'
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#

      Note:

      This step can take approximately 5 to 8 minutes on each switch.

    2. On both racks, cable the lower leaf switch to both of the spine switches (R1LL and R2LL to R1SS and R2SS).

      Connect the cables from the spine switches that you prepared earlier (in step 2.d).

      Cable the switches as described in Two-Rack Cabling for a System Combining an X8M Rack and a Later Model Rack:

      1. On R1LL, cable ports 04, 05, 06, 07, 30, 31, 32, and 33 to R1SS and R2SS.

      2. On R2LL, cable ports 01, 02, 03, 04, 05, 06, 07, 30, 31, 32, 33, 34, 35, and 36 to R1SS and R2SS.

      Note:

      Ensure that each cable connects to the correct switch and port at both ends. In addition to physically checking each connection, you can run the show lldp neighbors command on each network switch and examine the output to confirm correct connections. You can individually check each cable connection to catch and correct errors quickly.

    3. On the lower leaf switches, verify that all of the cabled RoCE Network Fabric ports are connected (R1LL and R2LL).

      Run the show interface status command on each lower leaf switch: 

      1. R1LL# show interface status
      2. R2LL# show interface status

      Examine the output to ensure that all of the cabled ports are connected.

      The following example shows the expected output on the X9M or later model rack (R2LL). On the X8M rack (R1LL), ports 01, 02, 03, 34, 35, and 36 are not physically connected. 

      R2LL# show interface status

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
mgmt0         --                 connected routed    full 1000    -- 

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
Eth1/1        RouterPort1        connected routed    full 100G    QSFP-100G-CR4 
Eth1/2        RouterPort2        connected routed    full 100G    QSFP-100G-CR4 
Eth1/3        RouterPort3        connected routed    full 100G    QSFP-100G-CR4 
Eth1/4        RouterPort4        connected routed    full 100G    QSFP-100G-CR4 
Eth1/5        RouterPort5        connected routed    full 100G    QSFP-100G-CR4 
Eth1/6        RouterPort6        connected routed    full 100G    QSFP-100G-CR4 
Eth1/7        RouterPort7        connected routed    full 100G    QSFP-100G-CR4 
Eth1/8        celadm14           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/9        celadm13           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/10       celadm12           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/11       celadm11           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/12       celadm10           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/13       celadm09           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/14       celadm08           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/15       adm08              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/16       adm07              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/17       adm06              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/18       adm05              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/19       adm04              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/20       adm03              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/21       adm02              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/22       adm01              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/23       celadm07           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/24       celadm06           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/25       celadm05           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/26       celadm04           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/27       celadm03           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/28       celadm02           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/29       celadm01           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/30       RouterPort8        connected routed    full 100G    QSFP-100G-CR4 
Eth1/31       RouterPort9        connected routed    full 100G    QSFP-100G-CR4 
Eth1/32       RouterPort10       connected routed    full 100G    QSFP-100G-CR4 
Eth1/33       RouterPort11       connected routed    full 100G    QSFP-100G-CR4 
Eth1/34       RouterPort12       connected routed    full 100G    QSFP-100G-CR4 
Eth1/35       RouterPort13       connected routed    full 100G    QSFP-100G-CR4 
Eth1/36       RouterPort14       connected routed    full 100G    QSFP-100G-CR4 
Lo0           Routing loopback i connected routed    auto auto    -- 
Lo1           VTEP loopback inte connected routed    auto auto    -- 
Vlan1         --                 down      routed    auto auto    -- 
nve1          --                 connected --        auto auto    --

      Note:

      Before proceeding, ensure that you have completed all of the actions to this point in step 5 on both lower leaf switches (R1LL and R2LL). If not, then ensure that you go back and perform the missing actions.

    4. Verify the configuration of the lower leaf switches.

      You can use the instance of patchmgr that you previously used to update the switch firmware (in step 1.h).

      Use a switch list file (ll.lst) to check both lower leaf switches using one patchmgr command: 

      # cat ll.lst
R1LL_IP:mleaf.101
R2LL_IP:mleaf_u14.103

      On a system with Secure Fabric enabled, use the msfleaf and msfleaf_u14 tags in the switch list file: 

      # cat ll.lst
R1LL_IP:msfleaf.101
R2LL_IP:msfleaf_u14.103

      The following shows the recommended command and an example of the expected results:

      # ./patchmgr --roceswitches ll.lst --verify-config -log_dir /tmp/log
2020-08-10 13:45:09 -0700        :Working: Initiating config verification... Expect up to 6 minutes for each switch
Mon Aug 10 13:45:13 PDT 2020 1 of 4 :Verifying config on switch ...
...
Mon Aug 10 13:45:32 PDT 2020:	  [INFO     ] Config matches template: ...
Mon Aug 10 13:45:32 PDT 2020:	  [SUCCESS  ] Config validation successful!
2020-08-10 13:45:32 -0700        
 Config check on RoCE switch(es)
2020-08-10 13:45:32 -0700        
 Completed run of command: ./patchmgr --roceswitches ll.lst --verify-config -log_dir /tmp/log
2020-08-10 13:45:32 -0700        :INFO   : config attempted on nodes in file ll.lst: [R1LL_IP R2LL_IP]
2020-08-10 13:45:32 -0700        :INFO   : For details, check the following files in /tmp/log:
2020-08-10 13:45:32 -0700        :INFO   :  - updateRoceSwitch.log
2020-08-10 13:45:32 -0700        :INFO   :  - updateRoceSwitch.trc
2020-08-10 13:45:32 -0700        :INFO   :  - patchmgr.stdout
2020-08-10 13:45:32 -0700        :INFO   :  - patchmgr.stderr
2020-08-10 13:45:32 -0700        :INFO   :  - patchmgr.log
2020-08-10 13:45:32 -0700        :INFO   :  - patchmgr.trc
2020-08-10 13:45:32 -0700        :INFO   : Exit status:0
2020-08-10 13:45:32 -0700        :INFO   : Exiting.

      In the command output, verify that the switch configuration is good for both lower leaf switches.

    5. Verify that nve is up on the lower leaf switches (R1LL and R2LL).

      Run the following command on each lower leaf switch and examine the output:

      1. R1LL# show nve peers
      2. R2LL# show nve peers

      At this point, you should see one nve peer with State=Up.

      For example:

      R1LL# show nve peers
Interface         Peer-IP State LearnType   Uptime        Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1         100.64.1.103    Up        CP 00:04:29               n/a
    6. Verify that BGP is up on the lower leaf switches (R1LL and R2LL).

      Run the following command on each lower leaf switch and examine the output:

      1. R1LL# show logging log | grep BGP
      2. R2LL# show logging log | grep BGP

      Look for two entries with Up in the rightmost column that are associated with different IP addresses.

      For example:

      R1LL# show logging log | grep BGP
2020 Aug 10 13:47:13 R1LL %BGP-5-ADJCHANGE: bgp- [29342] (default) neighbor 100.64.0.201 Up 
2020 Aug 10 13:47:24 R1LL %BGP-5-ADJCHANGE: bgp- [29342] (default) neighbor 100.64.0.202 Up
  6. Finalize the configuration of the upper leaf switches (R1UL and R2UL).

    Perform this step on the upper leaf switches (R1UL and R2UL) only.

    1. Start the inter-switch ports on the upper leaf switches (R1UL and R2UL).

      1. On R1UL:

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# interface ethernet 1/4-7, ethernet 1/30-33
R1UL(config-if-range)# no shut
R1UL(config-if-range)# exit
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. On R2UL:

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# interface ethernet 1/1-7, ethernet 1/30-36
R2UL(config-if-range)# no shut
R2UL(config-if-range)# exit
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#
    2. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    3. Verify the status of the inter-switch ports on the upper leaf switches (R1UL and R2UL).

      Run the show interface status command on each upper leaf switch: 

      1. R1UL# show interface status
      2. R2UL# show interface status

      Examine the output to ensure that the inter-switch ports are connected.

      The following example shows the expected output on the X9M or later model rack (R2UL). On the X8M rack (R1UL), ports 01, 02, 03, 34, 35, and 36 are not physically connected. 

      R2UL# show interface status

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
mgmt0         --                 connected routed    full 1000    -- 

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
Eth1/1        RouterPort1        connected routed    full 100G    QSFP-100G-CR4 
Eth1/2        RouterPort2        connected routed    full 100G    QSFP-100G-CR4 
Eth1/3        RouterPort3        connected routed    full 100G    QSFP-100G-CR4 
Eth1/4        RouterPort4        connected routed    full 100G    QSFP-100G-CR4 
Eth1/5        RouterPort5        connected routed    full 100G    QSFP-100G-CR4 
Eth1/6        RouterPort6        connected routed    full 100G    QSFP-100G-CR4 
Eth1/7        RouterPort7        connected routed    full 100G    QSFP-100G-CR4 
Eth1/8        celadm14           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/9        celadm13           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/10       celadm12           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/11       celadm11           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/12       celadm10           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/13       celadm09           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/14       celadm08           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/15       adm08              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/16       adm07              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/17       adm06              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/18       adm05              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/19       adm04              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/20       adm03              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/21       adm02              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/22       adm01              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/23       celadm07           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/24       celadm06           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/25       celadm05           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/26       celadm04           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/27       celadm03           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/28       celadm02           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/29       celadm01           disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/30       RouterPort8        connected routed    full 100G    QSFP-100G-CR4 
Eth1/31       RouterPort9        connected routed    full 100G    QSFP-100G-CR4 
Eth1/32       RouterPort10       connected routed    full 100G    QSFP-100G-CR4 
Eth1/33       RouterPort11       connected routed    full 100G    QSFP-100G-CR4 
Eth1/34       RouterPort12       connected routed    full 100G    QSFP-100G-CR4 
Eth1/35       RouterPort13       connected routed    full 100G    QSFP-100G-CR4 
Eth1/36       RouterPort14       connected routed    full 100G    QSFP-100G-CR4 
Lo0           Routing loopback i connected routed    auto auto    -- 
Lo1           VTEP loopback inte connected routed    auto auto    -- 
Vlan1         --                 down      routed    auto auto    -- 
nve1          --                 connected --        auto auto    --
    4. Start the storage server ports on the upper leaf switches (R1UL and R2UL).

      1. On R1UL:

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# interface ethernet 1/8-14, ethernet 1/23-29
R1UL(config-if-range)# no shut
R1UL(config-if-range)# exit
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. Repeat the command sequence on R2UL:

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# interface ethernet 1/8-14, ethernet 1/23-29
R2UL(config-if-range)# no shut
R2UL(config-if-range)# exit
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#
    5. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    6. Verify the status of the storage server ports on the upper leaf switches (R1UL and R2UL).

      Run the show interface status command on each upper leaf switch: 

      1. R1UL# show interface status
      2. R2UL# show interface status

      Examine the output to ensure that the storage server ports are connected.

      The following example shows the expected output on the X9M or later model rack (R2UL). On the X8M rack (R1UL), ports 01, 02, 03, 34, 35, and 36 are not physically connected. 

      R2UL# show interface status

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
mgmt0         --                 connected routed    full 1000    -- 

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
Eth1/1        RouterPort1        connected routed    full 100G    QSFP-100G-CR4 
Eth1/2        RouterPort2        connected routed    full 100G    QSFP-100G-CR4 
Eth1/3        RouterPort3        connected routed    full 100G    QSFP-100G-CR4 
Eth1/4        RouterPort4        connected routed    full 100G    QSFP-100G-CR4 
Eth1/5        RouterPort5        connected routed    full 100G    QSFP-100G-CR4 
Eth1/6        RouterPort6        connected routed    full 100G    QSFP-100G-CR4 
Eth1/7        RouterPort7        connected routed    full 100G    QSFP-100G-CR4 
Eth1/8        celadm14           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/9        celadm13           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/10       celadm12           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/11       celadm11           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/12       celadm10           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/13       celadm09           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/14       celadm08           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/15       adm08              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/16       adm07              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/17       adm06              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/18       adm05              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/19       adm04              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/20       adm03              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/21       adm02              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/22       adm01              disabled  3888      full 100G    QSFP-100G-CR4 
Eth1/23       celadm07           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/24       celadm06           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/25       celadm05           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/26       celadm04           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/27       celadm03           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/28       celadm02           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/29       celadm01           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/30       RouterPort8        connected routed    full 100G    QSFP-100G-CR4 
Eth1/31       RouterPort9        connected routed    full 100G    QSFP-100G-CR4 
Eth1/32       RouterPort10       connected routed    full 100G    QSFP-100G-CR4 
Eth1/33       RouterPort11       connected routed    full 100G    QSFP-100G-CR4 
Eth1/34       RouterPort12       connected routed    full 100G    QSFP-100G-CR4 
Eth1/35       RouterPort13       connected routed    full 100G    QSFP-100G-CR4 
Eth1/36       RouterPort14       connected routed    full 100G    QSFP-100G-CR4 
Lo0           Routing loopback i connected routed    auto auto    -- 
Lo1           VTEP loopback inte connected routed    auto auto    -- 
Vlan1         --                 down      routed    auto auto    -- 
nve1          --                 connected --        auto auto    --
    7. Start the database server ports on the upper leaf switches (R1UL and R2UL).

      1. On R1UL:

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# interface ethernet 1/15-22
R1UL(config-if-range)# no shut
R1UL(config-if-range)# exit
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. Repeat the command sequence on R2UL:

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# interface ethernet 1/15-22
R2UL(config-if-range)# no shut
R2UL(config-if-range)# exit
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#
    8. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    9. Verify the status of the database server ports on the upper leaf switches (R1UL and R2UL).

      Run the show interface status command on each upper leaf switch: 

      1. R1UL# show interface status
      2. R2UL# show interface status

      Examine the output to ensure that the database server ports are connected.

      The following example shows the expected output on the X9M or later model rack (R2UL). On the X8M rack (R1UL), ports 01, 02, 03, 34, 35, and 36 are not physically connected. 

      R2UL# show interface status

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
mgmt0         --                 connected routed    full 1000    -- 

-------------------------------------------------------------------------------- 
Port          Name               Status    Vlan      Duplex Speed   Type 
-------------------------------------------------------------------------------- 
Eth1/1        RouterPort1        connected routed    full 100G    QSFP-100G-CR4 
Eth1/2        RouterPort2        connected routed    full 100G    QSFP-100G-CR4 
Eth1/3        RouterPort3        connected routed    full 100G    QSFP-100G-CR4 
Eth1/4        RouterPort4        connected routed    full 100G    QSFP-100G-CR4 
Eth1/5        RouterPort5        connected routed    full 100G    QSFP-100G-CR4 
Eth1/6        RouterPort6        connected routed    full 100G    QSFP-100G-CR4 
Eth1/7        RouterPort7        connected routed    full 100G    QSFP-100G-CR4 
Eth1/8        celadm14           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/9        celadm13           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/10       celadm12           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/11       celadm11           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/12       celadm10           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/13       celadm09           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/14       celadm08           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/15       adm08              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/16       adm07              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/17       adm06              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/18       adm05              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/19       adm04              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/20       adm03              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/21       adm02              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/22       adm01              connected 3888      full 100G    QSFP-100G-CR4 
Eth1/23       celadm07           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/24       celadm06           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/25       celadm05           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/26       celadm04           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/27       celadm03           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/28       celadm02           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/29       celadm01           connected 3888      full 100G    QSFP-100G-CR4 
Eth1/30       RouterPort8        connected routed    full 100G    QSFP-100G-CR4 
Eth1/31       RouterPort9        connected routed    full 100G    QSFP-100G-CR4 
Eth1/32       RouterPort10       connected routed    full 100G    QSFP-100G-CR4 
Eth1/33       RouterPort11       connected routed    full 100G    QSFP-100G-CR4 
Eth1/34       RouterPort12       connected routed    full 100G    QSFP-100G-CR4 
Eth1/35       RouterPort13       connected routed    full 100G    QSFP-100G-CR4 
Eth1/36       RouterPort14       connected routed    full 100G    QSFP-100G-CR4 
Lo0           Routing loopback i connected routed    auto auto    -- 
Lo1           VTEP loopback inte connected routed    auto auto    -- 
Vlan1         --                 down      routed    auto auto    -- 
nve1          --                 connected --        auto auto    --
    10. Verify that nve is up on the leaf switches (R1LL, R1UL, R2LL, and R2UL).

      Run the following command on each leaf switch and examine the output:

      1. R1LL# show nve peers
      2. R1UL# show nve peers
      3. R2LL# show nve peers
      4. R2UL# show nve peers

      In the output, you should see three nve peers with State=Up.

      For example:

      R1UL# show nve peers
Interface         Peer-IP State LearnType   Uptime        Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1         100.64.1.101    Up        CP 00:04:29               n/a
nve1         100.64.1.103    Up        CP 00:07:48               n/a
nve1         100.64.1.104    Up        CP 00:04:10               n/a
    11. Verify that BGP is up on the upper leaf switches (R1UL and R2UL).

      Run the following command on each upper leaf switch and examine the output:

      1. R1UL# show logging log | grep BGP
      2. R2UL# show logging log | grep BGP

      In the output, look for two entries with Up in the rightmost column that are associated with different IP addresses.

      For example:

      R1UL# show logging log | grep BGP
2020 Aug 10 13:57:13 R1UL %BGP-5-ADJCHANGE: bgp- [32782] (default) neighbor 100.64.0.201 Up 
2020 Aug 10 13:57:24 R1UL %BGP-5-ADJCHANGE: bgp- [32782] (default) neighbor 100.64.0.202 Up
  7. For each rack (R1 and R2), confirm the multi-rack cabling by running the verify_roce_cables.py script.

    The verify_roce_cables.py script uses two input files; one for database servers and storage servers (nodes.rackN), and another for switches (switches.rackN). In each file, every server or switch must be listed on separate lines. Use fully qualified domain names or IP addresses for each server and switch.

    See My Oracle Support document 2587717.1 for download and detailed usage instructions.

    Run the verify_roce_cables.py script against both of the racks: 

    1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
    2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack2 -s switches.rack2

    Check the output of the verify_roce_cables.py script against the tables in Two-Rack Cabling for a System Combining an X8M Rack and a Later Model Rack. Also, check that output in the CABLE OK? columns contains the OK status.

    The following examples show extracts of the expected command results:

    # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
SWITCH PORT (EXPECTED PEER)  LOWER LEAF (rack1sw-rocea0) : CABLE OK?  UPPER LEAF (rack1sw-roceb0) : CABLE OK?
----------- ---------------  --------------------------- : ---------  --------------------------- : ---------
...
    # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack2 -s switches.rack2
SWITCH PORT (EXPECTED PEER)  LOWER LEAF (rack2sw-rocea0) : CABLE OK?  UPPER LEAF (rack2sw-roceb0) : CABLE OK?
----------- ---------------  --------------------------- : ---------  --------------------------- : ---------
...
  8. Verify the RoCE Network Fabric operation across both interconnected racks by using the infinicheck command.

    Use the following recommended command sequence to verify the RoCE Network Fabric operation across both racks.

    In each command, hosts.all contains a list of database server RoCE Network Fabric IP addresses from both racks (2 RoCE Network Fabric IP addresses for each database server), and cells.all contains a list of RoCE Network Fabric IP addresses for the storage servers from both racks (2 RoCE Network Fabric IP addresses for each storage server). 

    1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.all -c cells.all -z
    2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.all -c cells.all -s
    3. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.all -c cells.all -b

    See step 1.k for most information about each infinicheck command.

    The following example shows the expected command results for the final command in the sequence:

    # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.all -c cells.all -b

INFINICHECK                    
        [Network Connectivity, Configuration and Performance]        
               
          ####  FABRIC TYPE TESTS  #### 
System type identified: RoCE
Verifying User Equivalance of user=root from all DBs to all CELLs.
     ####  RoCE CONFIGURATION TESTS  ####       
     Checking for presence of RoCE devices on all DBs and CELLs 
[SUCCESS].... RoCE devices on all DBs and CELLs look good
     Checking for RoCE Policy Routing settings on all DBs and CELLs 
[SUCCESS].... RoCE Policy Routing settings look good
     Checking for RoCE DSCP ToS mapping on all DBs and CELLs 
[SUCCESS].... RoCE DSCP ToS settings look good
     Checking for RoCE PFC settings and DSCP mapping on all DBs and CELLs
[SUCCESS].... RoCE PFC and DSCP settings look good
     Checking for RoCE interface MTU settings. Expected value : 2300
[SUCCESS].... RoCE interface MTU settings look good
     Verifying switch advertised DSCP on all DBs and CELLs ports ( )
[SUCCESS].... Advertised DSCP settings from RoCE switch looks good  
    ####  CONNECTIVITY TESTS  ####
    [COMPUTE NODES -> STORAGE CELLS] 
      (60 seconds approx.)       
    (Will walk through QoS values: 0-6) [SUCCESS]..........Results OK
[SUCCESS]....... All  can talk to all storage cells          
    [COMPUTE NODES -> COMPUTE NODES]               
...

At this point, both racks share the RoCE Network Fabric, and the combined system is ready for further configuration.

See Configuring the New Hardware.

2.3.2.1.2.1 Two-Rack Cabling for a System Combining an X8M Rack and a Later Model Rack

This section provides the cabling details to connect an X8M rack with an X9M or later model rack, both of which use RoCE Network Fabric.

Note:

  • The following conventions are used in the cabling notation for connecting multiple racks together:

    • The abbreviation for the first (X8M) rack is R1, and the second (X9M or later) rack is R2.

    • LL identifies a lower leaf switch and UL identifies an upper leaf switch.

    • SS identifies the spine switch, which is located in U1 on all racks.

    • A specific switch is identified by combining abbreviations. For example, R1LL identifies the lower leaf switch (LL) on the first rack (R1).

  • The leaf switches are located as follows:

    • At rack unit 20 (U20) and 22 (U22) in 2-socket systems (Oracle Exadata Rack X8M-2 and later models).

    • At rack unit 21 (U21) and rack unit 23 (U23) in 8-socket systems (Oracle Exadata X8M-8 or X9M-8).

  • The cable lengths shown in the following lists assume that the racks are adjacent to each other, the cables are routed through a raised floor, and there are no obstacles in the routing between the racks. If the racks are not adjacent, or use overhead cabling trays, then they may require longer cable lengths. Cable lengths up to 100 meters are supported.

  • Only optical cables (with additional transceivers) are supported for lengths greater than 5 meters.

The following illustration shows the cable connections for the spine switches when cabling a two-rack hybrid system with an X8M rack and an X9M or later model rack:


Description of x8m-x9m-2-rack.eps follows
Description of the illustration x8m-x9m-2-rack.eps

The following tables contain details for all of the RoCE Network Fabric cabling connections in a two-rack hybrid system with an X8M rack and a later model rack.

Table 2-3 Leaf Switch Connections for the X8M Rack (R1)

Leaf Switch Connection Cable Length

R1UL to R1SS

R1UL-P5 to R1SS-P5

R1UL-P7 to R1SS-P7

R1UL-P4 to R1SS-P9

R1UL-P6 to R1SS-P11

3 meters

R1UL to R2SS

R1UL-P31 to R2SS-P5

R1UL-P33 to R2SS-P7

R1UL-P30 to R2SS-P9

R1UL-P32 to R2SS-P11

5 meters

R1LL to R1SS

R1LL-P5 to R1SS-P13

R1LL-P7 to R1SS-P15

R1LL-P4 to R1SS-P17

R1LL-P6 to R1SS-P19

3 meters

R1LL to R2SS

R1LL-P31 to R2SS-P13

R1LL-P33 to R2SS-P15

R1LL-P30 to R2SS-P17

R1LL-P32 to R2SS-P19

5 meters

Table 2-4 Leaf Switch Connections for the X9M or Later Model Rack (R2)

Switch Pair Connections Cable Length

R2UL to R1SS

R2UL-P1 to R1SS-P6

R2UL-P2 to R1SS-P8

R2UL-P3 to R1SS-P10

R2UL-P4 to R1SS-P12

R2UL-P5 to R1SS-P14

R2UL-P6 to R1SS-P16

R2UL-P7 to R1SS-P18

5 meters

R2UL to R2SS

R2UL-P30 to R2SS-P6

R2UL-P31 to R2SS-P8

R2UL-P32 to R2SS-P10

R2UL-P33 to R2SS-P12

R2UL-P34 to R2SS-P14

R2UL-P35 to R2SS-P16

R2UL-P36 to R2SS-P18

3 meters

R2LL to R1SS

R2LL-P1 to R1SS-P20

R2LL-P2 to R1SS-P22

R2LL-P3 to R1SS-P24

R2LL-P4 to R1SS-P26

R2LL-P5 to R1SS-P28

R2LL-P6 to R1SS-P30

R2LL-P7 to R1SS-P32

5 meters

R2LL to R2SS

R2LL-P30 to R2SS-P20

R2LL-P31 to R2SS-P22

R2LL-P32 to R2SS-P24

R2LL-P33 to R2SS-P26

R2LL-P34 to R2SS-P28

R2LL-P35 to R2SS-P30

R2LL-P36 to R2SS-P32

3 meters
2.3.2.1.3 Extending an X8M Rack with No Down Time by Adding Another X8M Rack

WARNING:

Take time to read and understand this procedure before implementation. Pay careful attention to the instructions that surround the command examples. A system outage may occur if the procedure is not applied correctly.

Note:

This procedure assumes that the RoCE Network Fabric switches on the X8M racks contain the golden configuration settings from Oracle Exadata System Software 20.1.0 or later. Otherwise, before using this procedure, you must update the Oracle Exadata System Software and update the golden configuration settings on the RoCE Network Fabric switches. Downtime is required to update the golden configuration settings on the RoCE Network Fabric switches.

Note:

For additional background information, see Understanding Multi-Rack Cabling for X8M Racks.

Use this procedure to extend a typical X8M rack without down-time by cabling it together with a second X8M rack. The primary rack (designated R1) and all of the systems it supports remain online throughout the procedure. At the beginning of the procedure, the additional rack (designated R2) is shut down.

The following is an outline of the procedure:

  • Preparation (steps 1 and 2)

    In this phase, you prepare the racks, switches, and cables. Also, you install and cable the spine switches in both racks.

  • Configuration and Physical Cabling

    In this phase, you reconfigure the leaf switches and finalize the cabling to the spine switches. These tasks are carefully orchestrated to avoid downtime on the primary system, as follows:

    • Partially configure the lower leaf switches (step 3)

      In this step, you reconfigure the switch ports on the lower leaf switches. There is no physical cabling performed in this step.

    • Partially configure the upper leaf switches (step 4)

      In this step, you reconfigure the switch ports on the upper leaf switches, remove the inter-switch cables that connect the leaf switches in both racks and connect the cables between the upper leaf switches and the spine switches.

    • Finalize the lower leaf switches (step 5)

      In this step, you finalize the switch port configuration on the lower leaf switches. You also complete the physical cabling by connecting the cables between the lower leaf switches and the spine switches.

    • Finalize the upper leaf switches (step 6)

      In this step, you finalize the switch port configuration on the upper leaf switches.

  • Validation and Testing (steps 7 and 8)

    In this phase, you validate and test the RoCE Network Fabric across both of the interconnect racks.

After completing the procedure, both racks share the RoCE Network Fabric, and the combined system is ready for further configuration. For example, you can extend existing disk groups and Oracle RAC databases to consume resources across both racks.

Note:

  • This procedure applies only to typical rack configurations that initially have leaf switches with the following specifications:

    • The inter-switch ports are ports 4 to 7, and ports 30 to 33.

    • The storage server ports are ports 8 to 14, and ports 23 to 29.

    • The database server ports are ports 15 to 22.

    For other rack configurations (for example, X8M-8 systems with three database servers and 11 storage servers) a different procedure and different RoCE Network Fabric switch configuration files are required. Contact Oracle for further guidance.

  • The procedure uses the following naming abbreviations and conventions:

    • The abbreviation for the existing rack is R1, and the new rack is R2.

    • LL identifies a lower leaf switch and UL identifies an upper leaf switch.

    • SS identifies a spine switch.

    • A specific switch is identified by combining abbreviations. For example, R1LL identifies the lower leaf switch (LL) on the existing rack (R1).

  • Most operations must be performed in multiple locations. For example, step 1.h instructs you to update the firmware on all the RoCE Network Fabric leaf switches (R1LL, R1UL, R2LL, and R2UL). Pay attention to the instructions and keep track of your actions.

    Tip:

    When a step must be performed on multiple switches, the instruction contains a list of the applicable switches. For example, (R1LL, R1UL, R2LL, and R2UL). You can use this list as a checklist to keep track of your actions.

  • Perform operations sequentially, and complete every operation before proceeding. For example, run the entire command sequence at 3.a.i as one operation and complete it before proceeding.

  • All of commands that are run on a RoCE Network Fabric switch must be run while connected to the switch management interface as the switch administrator.

  1. Prepare the systems.
    1. Position the new rack (R2) so that it is physically near the existing rack (R1).

      The RDMA Network Fabric cables must be able to reach the switches in each rack.

      For the required cross-rack cabling information, see Two-Rack Cabling for X8M Racks.

    2. Power on all of the servers and network switches in the new rack (R2).

      This includes the database servers, storage servers, RoCE Network Fabric leaf switches, and the Management Network Switch.

    3. Prepare the RoCE Network Fabric cables that you will use to interconnect the racks.

      Label both ends of every cable.

      For the required cross-rack cabling information, see Two-Rack Cabling for X8M Racks.

    4. Connect the new rack (R2) to your existing management network.

      Ensure that there are no IP address conflicts across the racks and that you can access the management interfaces on the RoCE Network Fabric switches.

    5. Ensure that you have a backup of the current switch configuration for each RoCE Network Fabric switch (R1LL, R1UL, R2LL, and R2UL).

      See Backing Up Settings on the RoCE Network Fabric Switch in Oracle Exadata Database Machine Maintenance Guide.

    6. Download the required RoCE Network Fabric switch configuration files.

      This procedure requires specific RoCE Network Fabric switch configuration files, which you must download from My Oracle Support document 2704997.1.

      WARNING:

      You must use different switch configuration files depending on whether your system uses Exadata Secure RDMA Fabric Isolation. Ensure that you download the correct archive that matches your system configuration.

      For system configurations without Secure Fabric, download online_multi-rack.zip. For system configurations with Secure Fabric, download online_SF_enabled_multi-rack.zip.

      Download and extract the archive containing the required RoCE Network Fabric switch configuration files. Place the files on a server with access to the management interfaces on the RoCE Network Fabric switches.

    7. Copy the required RoCE Network Fabric switch configuration files to the leaf switches on both racks.

      You can use the following commands to copy the required configuration files to all of the RoCE Network Fabric switches on a system without Secure Fabric enabled: 

      1. # scp roce_multi_online_step3_R1_LL.cfg admin@R1LL_IP:/
      2. # scp roce_multi_online_step3_R2_LL.cfg admin@R2LL_IP:/
      3. # scp roce_multi_online_step4_R1_UL.cfg admin@R1UL_IP:/
      4. # scp roce_multi_online_step4_R2_UL.cfg admin@R2UL_IP:/
      5. # scp roce_multi_online_step5.cfg admin@R1LL_IP:/
      6. # scp roce_multi_online_step5.cfg admin@R2LL_IP:/

      On a system with Secure Fabric enabled, you can use the following commands: 

      1. # scp roce_SF_multi_online_step3_R1_LL.cfg admin@R1LL_IP:/
      2. # scp roce_SF_multi_online_step3_R2_LL.cfg admin@R2LL_IP:/
      3. # scp roce_SF_multi_online_step4_R1_UL.cfg admin@R1UL_IP:/
      4. # scp roce_SF_multi_online_step4_R2_UL.cfg admin@R2UL_IP:/
      5. # scp roce_SF_multi_online_step5.cfg admin@R1LL_IP:/
      6. # scp roce_SF_multi_online_step5.cfg admin@R2LL_IP:/

      In the above commands, substitute the appropriate IP address or host name where applicable. For example, in place of R1LL_IP, substitute the management IP address or host name for the lower leaf switch (LL) on the existing rack (R1).

      Note:

      The command examples in the rest of this procedure use the configuration files for a system configuration without Secure Fabric enabled. If required, adjust the commands to use the Secure Fabric-enabled switch configuration files.
    8. Update the firmware to the latest available release on all of the RoCE Network Fabric leaf switches (R1LL, R1UL, R2LL, and R2UL).

      See Updating RoCE Network Fabric Switch Firmware in Oracle Exadata Database Machine Maintenance Guide.

    9. Examine the RoCE Network Fabric leaf switches (R1LL, R1UL, R2LL, and R2UL) and confirm the port categories for the cabled ports.

      Run the show interface status command on every RoCE Network Fabric leaf switch: 

      1. R1LL# show interface status
      2. R1UL# show interface status
      3. R2LL# show interface status
      4. R2UL# show interface status

      Examine the output and confirm the port categories as follows:

      • Confirm that the inter-switch ports are ports 4 to 7, and ports 30 to 33.

      • Confirm that the storage server ports are ports 8 to 14, and ports 23 to 29.

      • Confirm that the database server ports are ports 15 to 22.

      For example:

      R1LL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected trunk     full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected trunk     full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected trunk     full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected trunk     full   100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           connected 3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           connected 3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           connected 3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           connected 3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           connected 3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           connected 3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              connected 3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              connected 3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              connected 3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              connected 3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              connected 3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              connected 3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              connected 3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              connected 3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           connected 3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           connected 3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           connected 3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           connected 3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           connected 3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           connected 3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           connected 3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected trunk     full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected trunk     full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected trunk     full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected trunk     full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --
    10. For each rack (R1 and R2), confirm the RoCE Network Fabric cabling by running the verify_roce_cables.py script.

      The verify_roce_cables.py script uses two input files; one for database servers and storage servers (nodes.rackN), and another for switches (switches.rackN). In each file, every server or switch must be listed on separate lines. Use fully qualified domain names or IP addresses for each server and switch.

      See My Oracle Support document 2587717.1 for download and detailed usage instructions.

      Run the verify_roce_cables.py script against both of the racks: 

      1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
      2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack2 -s switches.rack2

      Check that output in the CABLE OK? columns contains the OK status.

      The following example shows the expected command results:

      # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
SWITCH PORT (EXPECTED PEER)               LOWER LEAF (rack1sw-rocea0) : CABLE OK?             UPPER LEAF (rack1sw-roceb0) : CABLE OK?
----------- ---------------          -------------------------------- : --------         -------------------------------- : ---------
    Eth1/4 (ISL peer switch)   :           rack1sw-rocea0 Ethernet1/4 : OK                     rack1sw-roceb0 Ethernet1/4 : OK
    Eth1/5 (ISL peer switch)   :           rack1sw-rocea0 Ethernet1/5 : OK                     rack1sw-roceb0 Ethernet1/5 : OK
    Eth1/6 (ISL peer switch)   :           rack1sw-rocea0 Ethernet1/6 : OK                     rack1sw-roceb0 Ethernet1/6 : OK
    Eth1/7 (ISL peer switch)   :           rack1sw-rocea0 Ethernet1/7 : OK                     rack1sw-roceb0 Ethernet1/7 : OK
    Eth1/8 (RU39)              :                 rack1celadm14 port-1 : OK                           rack1celadm14 port-2 : OK
    Eth1/9 (RU37)              :                 rack1celadm13 port-1 : OK                           rack1celadm13 port-2 : OK
   Eth1/10 (RU35)              :                 rack1celadm12 port-1 : OK                           rack1celadm12 port-2 : OK
   Eth1/11 (RU33)              :                 rack1celadm11 port-1 : OK                           rack1celadm11 port-2 : OK
   Eth1/12 (RU31)              :                 rack1celadm10 port-1 : OK                           rack1celadm10 port-2 : OK
   Eth1/13 (RU29)              :                 rack1celadm09 port-1 : OK                           rack1celadm09 port-2 : OK
   Eth1/14 (RU27)              :                 rack1celadm08 port-1 : OK                           rack1celadm08 port-2 : OK
   Eth1/15 (RU26)              :                    rack1adm08 port-1 : OK                              rack1adm08 port-2 : OK
   Eth1/16 (RU25)              :                    rack1adm07 port-1 : OK                              rack1adm07 port-2 : OK
   Eth1/17 (RU24)              :                    rack1adm06 port-1 : OK                              rack1adm06 port-2 : OK
   Eth1/18 (RU23)              :                    rack1adm05 port-1 : OK                              rack1adm05 port-2 : OK
   Eth1/19 (RU19)              :                    rack1adm04 port-1 : OK                              rack1adm04 port-2 : OK
   Eth1/20 (RU18)              :                    rack1adm03 port-1 : OK                              rack1adm03 port-2 : OK
   Eth1/21 (RU17)              :                    rack1adm02 port-1 : OK                              rack1adm02 port-2 : OK
   Eth1/22 (RU16)              :                    rack1adm01 port-1 : OK                              rack1adm01 port-2 : OK
   Eth1/23 (RU14)              :                 rack1celadm07 port-1 : OK                           rack1celadm07 port-2 : OK
   Eth1/24 (RU12)              :                 rack1celadm06 port-1 : OK                           rack1celadm06 port-2 : OK
   Eth1/25 (RU10)              :                 rack1celadm05 port-1 : OK                           rack1celadm05 port-2 : OK
   Eth1/26 (RU08)              :                 rack1celadm04 port-1 : OK                           rack1celadm04 port-2 : OK
   Eth1/27 (RU06)              :                 rack1celadm03 port-1 : OK                           rack1celadm03 port-2 : OK
   Eth1/28 (RU04)              :                 rack1celadm02 port-1 : OK                           rack1celadm02 port-2 : OK
   Eth1/29 (RU02)              :                 rack1celadm01 port-1 : OK                           rack1celadm01 port-2 : OK
   Eth1/30 (ISL peer switch)   :          rack1sw-rocea0 Ethernet1/30 : OK                    rack1sw-roceb0 Ethernet1/30 : OK
   Eth1/31 (ISL peer switch)   :          rack1sw-rocea0 Ethernet1/31 : OK                    rack1sw-roceb0 Ethernet1/31 : OK
   Eth1/32 (ISL peer switch)   :          rack1sw-rocea0 Ethernet1/32 : OK                    rack1sw-roceb0 Ethernet1/32 : OK
   Eth1/33 (ISL peer switch)   :          rack1sw-rocea0 Ethernet1/33 : OK                    rack1sw-roceb0 Ethernet1/33 : OK
    11. For each rack (R1 and R2), verify the RoCE Network Fabric operation by using the infinicheck command.

      • Use infinicheck with the -z option to clear the files that were created during the last run of the infinicheck command.

      • Use infinicheck with the -s option to set up user equivalence for password-less SSH across the RoCE Network Fabric.

      • Finally, verify the RoCE Network Fabric operation by using infinicheck with the -b option, which is recommended on newly imaged machines where it is acceptable to suppress the cellip.ora and cellinit.ora configuration checks.

      In each command, the hosts input file (hosts.rack1 and hosts.rack2) contains a list of database server RoCE Network Fabric IP addresses (2 RoCE Network Fabric IP addresses for each database server), and the cells input file (cells.rack1 and cells.rack2) contains a list of RoCE Network Fabric IP addresses for the storage servers (2 RoCE Network Fabric IP addresses for each storage server).

      1. Use the following recommended command sequence on the existing rack (R1):

        1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack1 -c cells.rack1 -z
        2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack1 -c cells.rack1 -s
        3. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack1 -c cells.rack1 -b

      2. Use the following recommended command sequence on the new rack (R2):

        1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack2 -c cells.rack2 -z
        2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack2 -c cells.rack2 -s
        3. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rack2 -c cells.rack2 -b

      The following example shows the expected command results for the final command in the sequence:

      # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.rackN -c cells.rackN -b

INFINICHECK                    
        [Network Connectivity, Configuration and Performance]        
               
          ####  FABRIC TYPE TESTS  #### 
System type identified: RoCE
Verifying User Equivalance of user=root from all DBs to all CELLs.
     ####  RoCE CONFIGURATION TESTS  ####       
     Checking for presence of RoCE devices on all DBs and CELLs 
[SUCCESS].... RoCE devices on all DBs and CELLs look good
     Checking for RoCE Policy Routing settings on all DBs and CELLs 
[SUCCESS].... RoCE Policy Routing settings look good
     Checking for RoCE DSCP ToS mapping on all DBs and CELLs 
[SUCCESS].... RoCE DSCP ToS settings look good
     Checking for RoCE PFC settings and DSCP mapping on all DBs and CELLs
[SUCCESS].... RoCE PFC and DSCP settings look good
     Checking for RoCE interface MTU settings. Expected value : 2300
[SUCCESS].... RoCE interface MTU settings look good
     Verifying switch advertised DSCP on all DBs and CELLs ports ( )
[SUCCESS].... Advertised DSCP settings from RoCE switch looks good  
    ####  CONNECTIVITY TESTS  ####
    [COMPUTE NODES -> STORAGE CELLS] 
      (60 seconds approx.)       
    (Will walk through QoS values: 0-6) [SUCCESS]..........Results OK
[SUCCESS]....... All  can talk to all storage cells          
    [COMPUTE NODES -> COMPUTE NODES]               
...
  2. Install the spine switches (R1SS and R2SS).
    1. Physically install and power up the spine switches in the existing rack (R1SS) and the new rack (R2SS).

      1. Physically install each spine switch in RU1.

      2. For each spine switch, ensure that the management Ethernet interface is connected to the management network and then supply power.

      3. On each spine switch, perform the initial configuration steps outlined in Configuring the Cisco Nexus C9336C-FX2 Switch. Skip the step for applying the golden configuration settings as you will do this later.

      4. For each spine switch, perform a ping test to the management Ethernet interface to ensure that the switch is online and accessible.

    2. Apply the golden configuration settings to the new spine switches.

      See Applying Golden Configuration Settings on RoCE Network Fabric Switches in Oracle Exadata Database Machine Maintenance Guide.

      You can use the instance of patchmgr that you previously used to update the firmware on the leaf switches (in step 1.h).

      Use a switch list file (spines.lst) to apply the golden configuration settings to both spine switches using one patchmgr command: 

      # cat spines.lst
R1SS_IP:mspine.201
R2SS_IP:mspine.202

# ./patchmgr --roceswitches spines.lst --apply-config -log_dir /tmp/spinelogs

      Note:

      In the switch list file, R1SS_IP is the management IP address or host name for the spine switch on the existing rack (R1SS) and R2SS_IP is the management IP address or host name for the spine switch on the new rack (R2SS).

    3. Upgrade the firmware on the spine switches.

      See Updating RoCE Network Fabric Switch Firmware in Oracle Exadata Database Machine Maintenance Guide.

      You can use the instance of patchmgr that you used in the previous step.

      Use a switch list file (spines.lst) to perform the firmware upgrade on both spine switches using one patchmgr command: 

      # cat spines.lst
R1SS_IP:mspine.201
R2SS_IP:mspine.202

# ./patchmgr --roceswitches spines.lst --upgrade -log_dir /tmp/spinelogs

      Note:

      In the switch list file, R1SS_IP is the management IP address or host name for the spine switch on the existing rack (R1SS) and R2SS_IP is the management IP address or host name for the spine switch on the new rack (R2SS).

    4. Connect the RoCE Network Fabric cables to the spine switches (R1SS and R2SS).

      WARNING:

      At this stage, only connect the cables to the spine switches.

      To avoid later complications, ensure that each cable connects to the correct switch and port.

      DO NOT CONNECT ANY OF THE CABLES TO THE LEAF SWITCHES.

      Use the cables that you prepared earlier (in step 1.c).

      For the required cross-rack cabling information, see Two-Rack Cabling for X8M Racks.

  3. Perform the first round of configuration on the lower leaf switches (R1LL and R2LL).

    Perform this step on the lower leaf switches (R1LL and R2LL) only.

    Note:

    During this step, the lower leaf switch ports are shut down. While the R1LL ports are down, R1UL exclusively supports the RoCE Network Fabric. During this time, there is no redundancy in the RoCE Network Fabric, and availability cannot be maintained if R1UL goes down.

    1. Shut down the switch ports on the lower leaf switches (R1LL and R2LL).

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# interface ethernet 1/4-33
R1LL(config-if-range)# shut
R1LL(config-if-range)# exit
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. Repeat the command sequence on R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# interface ethernet 1/4-33
R2LL(config-if-range)# shut
R2LL(config-if-range)# exit
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#
    2. Reconfigure the lower leaf switch ports (R1LL and R2LL) .

      For each switch, you must use the correct corresponding switch configuration file, which you earlier copied to the switch (in step 1.g).

      1. On R1LL, the switch configuration file name must end with step3_R1_LL.cfg: 

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# run-script bootflash:///roce_multi_online_step3_R1_LL.cfg | grep 'none'
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. On R2LL, the switch configuration file name must end with step3_R2_LL.cfg: 

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# run-script bootflash:///roce_multi_online_step3_R2_LL.cfg | grep 'none'
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#

      Note:

      This step can take approximately 5 to 8 minutes on each switch.

    3. Start the inter-switch ports on the lower leaf switches (R1LL and R2LL) .

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# interface ethernet 1/4-7, ethernet 1/30-33
R1LL(config-if-range)# no shut
R1LL(config-if-range)# exit
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. Repeat the command sequence on R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# interface ethernet 1/4-7, ethernet 1/30-33
R2LL(config-if-range)# no shut
R2LL(config-if-range)# exit
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#
    4. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    5. Verify the status of the inter-switch ports on the lower leaf switches (R1LL and R2LL) .

      Run the show interface status command on each lower leaf switch: 

      1. R1LL# show interface status
      2. R2LL# show interface status

      Examine the output to ensure that the inter-switch ports are connected.

      For example:

      R1LL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected trunk     full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected trunk     full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected trunk     full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected trunk     full   100G    QSFP-100G-CR4
Eth1/8        celadm14           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected trunk     full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected trunk     full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected trunk     full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected trunk     full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --
    6. Start the storage server ports on the lower leaf switches (R1LL and R2LL) .

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# interface ethernet 1/8-14, ethernet 1/23-29
R1LL(config-if-range)# no shut
R1LL(config-if-range)# exit
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. Repeat the command sequence on R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# interface ethernet 1/8-14, ethernet 1/23-29
R2LL(config-if-range)# no shut
R2LL(config-if-range)# exit
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#
    7. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    8. Verify the status of the storage server ports on the lower leaf switches (R1LL and R2LL).

      Run the show interface status command on each lower leaf switch: 

      1. R1LL# show interface status
      2. R2LL# show interface status

      Examine the output to ensure that the storage server ports are connected.

      For example:

      R1LL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected trunk     full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected trunk     full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected trunk     full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected trunk     full   100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           connected 3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           connected 3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           connected 3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           connected 3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           connected 3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           connected 3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           connected 3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           connected 3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           connected 3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           connected 3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           connected 3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           connected 3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           connected 3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected trunk     full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected trunk     full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected trunk     full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected trunk     full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --
    9. Start the database server ports on the lower leaf switches (R1LL and R2LL).

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# interface ethernet 1/15-22
R1LL(config-if-range)# no shut
R1LL(config-if-range)# exit
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. Repeat the command sequence on R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# interface ethernet 1/15-22
R2LL(config-if-range)# no shut
R2LL(config-if-range)# exit
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#
    10. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    11. Verify the status of the database server ports on the lower leaf switches (R1LL and R2LL).

      Run the show interface status command on each lower leaf switch: 

      1. R1LL# show interface status
      2. R2LL# show interface status

      Examine the output to ensure that the database server ports are connected.

      For example:

      R1LL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected trunk     full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected trunk     full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected trunk     full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected trunk     full   100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           connected 3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           connected 3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           connected 3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           connected 3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           connected 3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           connected 3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              connected 3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              connected 3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              connected 3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              connected 3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              connected 3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              connected 3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              connected 3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              connected 3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           connected 3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           connected 3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           connected 3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           connected 3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           connected 3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           connected 3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           connected 3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected trunk     full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected trunk     full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected trunk     full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected trunk     full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --

    Note:

    Before proceeding, ensure that you have completed all of the actions in step 3 on both lower leaf switches (R1LL and R2LL). If not, then ensure that you go back and perform the missing actions.

  4. Perform the first round of configuration on the upper leaf switches (R1UL and R2UL).

    Perform this step on the upper leaf switches (R1UL and R2UL) only.

    Note:

    At the start of this step, the upper leaf switch ports are shut down. While the R1UL ports are down, R1LL exclusively supports the RoCE Network Fabric on the existing rack. During this time, there is no redundancy in the RoCE Network Fabric, and availability cannot be maintained if R1LL goes down.

    1. Shut down the upper leaf switch ports (R1UL and R2UL).

      1. On R1UL:

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# interface ethernet 1/4-33
R1UL(config-if-range)# shut
R1UL(config-if-range)# exit
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. Repeat the command sequence on R2UL:

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# interface ethernet 1/4-33
R2UL(config-if-range)# shut
R2UL(config-if-range)# exit
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#
    2. On both racks, remove the inter-switch links between the leaf switches (R1LL to R1UL, and R2LL to R2UL).

      On every leaf switch, remove the cables for the inter-switch links:

      1. On R1LL, disconnect the inter-switch links from ports 04, 05, 06, 07, 30, 31, 32, and 33.

      2. On R1UL, disconnect the inter-switch links from ports 04, 05, 06, 07, 30, 31, 32, and 33.

      3. On R2LL, disconnect the inter-switch links from ports 04, 05, 06, 07, 30, 31, 32, and 33.

      4. On R2UL, disconnect the inter-switch links from ports 04, 05, 06, 07, 30, 31, 32, and 33.

    3. On both racks, cable the upper leaf switch to both of the spine switches (R1UL and R2UL to R1SS and R2SS).

      Connect the cables from the spine switches that you prepared earlier (in step 2.d).

      Cable the switches as described in Two-Rack Cabling for X8M Racks:

      1. On R1UL, cable ports 04, 05, 06, 07, 30, 31, 32, and 33 to R1SS and R2SS.

      2. On R2UL, cable ports 04, 05, 06, 07, 30, 31, 32, and 33 to R1SS and R2SS.

      Note:

      Ensure that each cable connects to the correct switch and port at both ends. In addition to physically checking each connection, you can run the show lldp neighbors command on each network switch and examine the output to confirm correct connections. You can individually check each cable connection to catch and correct errors quickly.

    4. Reconfigure the upper leaf switch ports (R1UL and R2UL).

      For each switch, you must use the correct corresponding switch configuration file, which you earlier copied to the switch (in step 1.g):

      1. On R1UL, the switch configuration file name must end with step4_R1_UL.cfg: 

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# run-script bootflash:///roce_multi_online_step4_R1_UL.cfg | grep 'none'
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. On R2UL, the switch configuration file name must end with step4_R2_UL.cfg: 

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# run-script bootflash:///roce_multi_online_step4_R2_UL.cfg | grep 'none'
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#

      Note:

      This step can take approximately 5 to 8 minutes on each switch.

    5. Check the status of the RoCE Network Fabric ports on the upper leaf switches (R1UL and R2UL).

      Run the show interface status command on each upper leaf switch: 

      1. R1UL# show interface status
      2. R2UL# show interface status

      Examine the output to ensure that all of the cabled ports are disabled.

      For example:

      R1UL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full    1000    -- 

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
...
Eth1/4        RouterPort1        disabled  routed    full    100G    QSFP-100G-CR4
Eth1/5        RouterPort2        disabled  routed    full    100G    QSFP-100G-CR4
Eth1/6        RouterPort3        disabled  routed    full    100G    QSFP-100G-CR4
Eth1/7        RouterPort4        disabled  routed    full    100G    QSFP-100G-CR4
Eth1/8        celadm14           disabled  3888      full    100G    QSFP-100G-CR4
Eth1/9        celadm13           disabled  3888      full    100G    QSFP-100G-CR4
Eth1/10       celadm12           disabled  3888      full    100G    QSFP-100G-CR4
Eth1/11       celadm11           disabled  3888      full    100G    QSFP-100G-CR4
Eth1/12       celadm10           disabled  3888      full    100G    QSFP-100G-CR4
Eth1/13       celadm09           disabled  3888      full    100G    QSFP-100G-CR4
Eth1/14       celadm08           disabled  3888      full    100G    QSFP-100G-CR4
Eth1/15       adm08              disabled  3888      full    100G    QSFP-100G-CR4
Eth1/16       adm07              disabled  3888      full    100G    QSFP-100G-CR4
Eth1/17       adm06              disabled  3888      full    100G    QSFP-100G-CR4
Eth1/18       adm05              disabled  3888      full    100G    QSFP-100G-CR4
...

      Note:

      Before proceeding, ensure that you have completed all of the actions to this point in step 4 on both upper leaf switches (R1UL and R2UL). If not, then ensure that you go back and perform the missing actions.

    6. Verify the configuration of the upper leaf switches.

      You can use the instance of patchmgr that you previously used to update the switch firmware (in step 1.h).

      Use a switch list file (ul.lst) to check both upper leaf switches using one patchmgr command: 

      # cat ul.lst
R1UL_IP:mleaf.102
R2UL_IP:mleaf.104

      On a system with Secure Fabric enabled, use the msfleaf tag in the switch list file: 

      # cat ul.lst
R1UL_IP:msfleaf.102
R2UL_IP:msfleaf.104

      The following shows the recommended command and an example of the expected results:

      # ./patchmgr --roceswitches ul.lst --verify-config -log_dir /tmp/log
2020-08-10 13:40:09 -0700        :Working: Initiating config verification... Expect up to 6 minutes for each switch
Mon Aug 10 13:40:13 PDT 2020 1 of 4 :Verifying config on switch ...
...
Mon Aug 10 13:40:32 PDT 2020:	  [INFO     ] Config matches template: ...
Mon Aug 10 13:40:32 PDT 2020:	  [SUCCESS  ] Config validation successful!
2020-08-10 13:40:32 -0700        
 Config check on RoCE switch(es)
2020-08-10 13:40:32 -0700        
 Completed run of command: ./patchmgr --roceswitches ul.lst --verify-config -log_dir /tmp/log
2020-08-10 13:40:32 -0700        :INFO   : config attempted on nodes in file ul.lst: [R1UL_IP R2UL_IP]
2020-08-10 13:40:32 -0700        :INFO   : For details, check the following files in /tmp/log:
2020-08-10 13:40:32 -0700        :INFO   :  - updateRoceSwitch.log
2020-08-10 13:40:32 -0700        :INFO   :  - updateRoceSwitch.trc
2020-08-10 13:40:32 -0700        :INFO   :  - patchmgr.stdout
2020-08-10 13:40:32 -0700        :INFO   :  - patchmgr.stderr
2020-08-10 13:40:32 -0700        :INFO   :  - patchmgr.log
2020-08-10 13:40:32 -0700        :INFO   :  - patchmgr.trc
2020-08-10 13:40:32 -0700        :INFO   : Exit status:0
2020-08-10 13:40:32 -0700        :INFO   : Exiting.

      In the command output, verify that the switch configuration is good for both upper leaf switches. You can ignore messages about the ports that are down.

  5. Finalize the configuration of the lower leaf switches (R1LL and R2LL).

    Perform this step on the lower leaf switches (R1LL and R2LL) only.

    1. Reconfigure the lower leaf switch ports (R1LL and R2LL).

      Run the following command sequence on both of the lower leaf switches (R1LL and R2LL).

      You must use the correct switch configuration file, which you earlier copied to the switch (in step 1.g). In this step, the configuration file name must end with step5.cfg.

      1. On R1LL:

        R1LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1LL(config)# run-script bootflash:///roce_multi_online_step5.cfg | grep 'none'
R1LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1LL(config)# <Ctrl-Z>
R1LL#

      2. Repeat the command sequence on R2LL:

        R2LL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2LL(config)# run-script bootflash:///roce_multi_online_step5.cfg | grep 'none'
R2LL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2LL(config)# <Ctrl-Z>
R2LL#

      Note:

      This step can take approximately 5 to 8 minutes on each switch.

    2. On both racks, cable the lower leaf switch to both of the spine switches (R1LL and R2LL to R1SS and R2SS).

      Connect the cables from the spine switches that you prepared earlier (in step 2.d).

      Cable the switches as described in Two-Rack Cabling for X8M Racks:

      1. On R1LL, cable ports 04, 05, 06, 07, 30, 31, 32, and 33 to R1SS and R2SS.

      2. On R2LL, cable ports 04, 05, 06, 07, 30, 31, 32, and 33 to R1SS and R2SS.

      Note:

      Ensure that each cable connects to the correct switch and port at both ends. In addition to physically checking each connection, you can run the show lldp neighbors command on each network switch and examine the output to confirm correct connections. You can individually check each cable connection to catch and correct errors quickly.

    3. On the lower leaf switches, verify that all of the cabled RoCE Network Fabric ports are connected (R1LL and R2LL).

      Run the show interface status command on each lower leaf switch: 

      1. R1LL# show interface status
      2. R2LL# show interface status

      Examine the output to ensure that all of the cabled ports are connected.

      For example:

      R1LL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full    1000    -- 

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
...
Eth1/4        RouterPort1        connected routed    full    100G    QSFP-100G-CR4
Eth1/5        RouterPort2        connected routed    full    100G    QSFP-100G-CR4
Eth1/6        RouterPort3        connected routed    full    100G    QSFP-100G-CR4
Eth1/7        RouterPort4        connected routed    full    100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full    100G    QSFP-100G-CR4
Eth1/9        celadm13           connected 3888      full    100G    QSFP-100G-CR4
Eth1/10       celadm12           connected 3888      full    100G    QSFP-100G-CR4
Eth1/11       celadm11           connected 3888      full    100G    QSFP-100G-CR4
Eth1/12       celadm10           connected 3888      full    100G    QSFP-100G-CR4
Eth1/13       celadm09           connected 3888      full    100G    QSFP-100G-CR4
Eth1/14       celadm08           connected 3888      full    100G    QSFP-100G-CR4
Eth1/15       adm08              connected 3888      full    100G    QSFP-100G-CR4
Eth1/16       adm07              connected 3888      full    100G    QSFP-100G-CR4
Eth1/17       adm06              connected 3888      full    100G    QSFP-100G-CR4
Eth1/18       adm05              connected 3888      full    100G    QSFP-100G-CR4
...

      Note:

      Before proceeding, ensure that you have completed all of the actions to this point in step 5 on both lower leaf switches (R1LL and R2LL). If not, then ensure that you go back and perform the missing actions.

    4. Verify the configuration of the lower leaf switches.

      You can use the instance of patchmgr that you previously used to update the switch firmware (in step 1.h).

      Use a switch list file (ll.lst) to check both lower leaf switches using one patchmgr command: 

      # cat ll.lst
R1LL_IP:mleaf.101
R2LL_IP:mleaf.103

      On a system with Secure Fabric enabled, use the msfleaf tag in the switch list file: 

      # cat ll.lst
R1LL_IP:msfleaf.101
R2LL_IP:msfleaf.103

      The following shows the recommended command and an example of the expected results:

      # ./patchmgr --roceswitches ll.lst --verify-config -log_dir /tmp/log
2020-08-10 13:45:09 -0700        :Working: Initiating config verification... Expect up to 6 minutes for each switch
Mon Aug 10 13:45:13 PDT 2020 1 of 4 :Verifying config on switch ...
...
Mon Aug 10 13:45:32 PDT 2020:	  [INFO     ] Config matches template: ...
Mon Aug 10 13:45:32 PDT 2020:	  [SUCCESS  ] Config validation successful!
2020-08-10 13:45:32 -0700        
 Config check on RoCE switch(es)
2020-08-10 13:45:32 -0700        
 Completed run of command: ./patchmgr --roceswitches ll.lst --verify-config -log_dir /tmp/log
2020-08-10 13:45:32 -0700        :INFO   : config attempted on nodes in file ll.lst: [R1LL_IP R2LL_IP]
2020-08-10 13:45:32 -0700        :INFO   : For details, check the following files in /tmp/log:
2020-08-10 13:45:32 -0700        :INFO   :  - updateRoceSwitch.log
2020-08-10 13:45:32 -0700        :INFO   :  - updateRoceSwitch.trc
2020-08-10 13:45:32 -0700        :INFO   :  - patchmgr.stdout
2020-08-10 13:45:32 -0700        :INFO   :  - patchmgr.stderr
2020-08-10 13:45:32 -0700        :INFO   :  - patchmgr.log
2020-08-10 13:45:32 -0700        :INFO   :  - patchmgr.trc
2020-08-10 13:45:32 -0700        :INFO   : Exit status:0
2020-08-10 13:45:32 -0700        :INFO   : Exiting.

      In the command output, verify that the switch configuration is good for both lower leaf switches.

    5. Verify that nve is up on the lower leaf switches (R1LL and R2LL).

      Run the following command on each lower leaf switch and examine the output:

      1. R1LL# show nve peers
      2. R2LL# show nve peers

      At this point, you should see one nve peer with State=Up.

      For example:

      R1LL# show nve peers
Interface         Peer-IP State LearnType   Uptime        Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1         100.64.1.103    Up        CP 00:04:29               n/a
    6. Verify that BGP is up on the lower leaf switches (R1LL and R2LL).

      Run the following command on each lower leaf switch and examine the output:

      1. R1LL# show logging log | grep BGP
      2. R2LL# show logging log | grep BGP

      Look for two entries with Up in the rightmost column that are associated with different IP addresses.

      For example:

      R1LL# show logging log | grep BGP
2020 Aug 10 13:47:13 R1LL %BGP-5-ADJCHANGE: bgp- [29342] (default) neighbor 100.64.0.201 Up 
2020 Aug 10 13:47:24 R1LL %BGP-5-ADJCHANGE: bgp- [29342] (default) neighbor 100.64.0.202 Up
  6. Finalize the configuration of the upper leaf switches (R1UL and R2UL).

    Perform this step on the upper leaf switches (R1UL and R2UL) only.

    1. Start the inter-switch ports on the upper leaf switches (R1UL and R2UL).

      1. On R1UL:

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# interface ethernet 1/4-7, ethernet 1/30-33
R1UL(config-if-range)# no shut
R1UL(config-if-range)# exit
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. Repeat the command sequence on R2UL:

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# interface ethernet 1/4-7, ethernet 1/30-33
R2UL(config-if-range)# no shut
R2UL(config-if-range)# exit
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#
    2. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    3. Verify the status of the inter-switch ports on the upper leaf switches (R1UL and R2UL).

      Run the show interface status command on each upper leaf switch: 

      1. R1UL# show interface status
      2. R2UL# show interface status

      Examine the output to ensure that the inter-switch ports are connected.

      For example:

      R1UL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected routed    full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected routed    full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected routed    full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected routed    full   100G    QSFP-100G-CR4
Eth1/8        celadm14           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           disabled  3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected routed    full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected routed    full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected routed    full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected routed    full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --
    4. Start the storage server ports on the upper leaf switches (R1UL and R2UL).

      1. On R1UL:

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# interface ethernet 1/8-14, ethernet 1/23-29
R1UL(config-if-range)# no shut
R1UL(config-if-range)# exit
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. Repeat the command sequence on R2UL:

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# interface ethernet 1/8-14, ethernet 1/23-29
R2UL(config-if-range)# no shut
R2UL(config-if-range)# exit
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#
    5. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    6. Verify the status of the storage server ports on the upper leaf switches (R1UL and R2UL).

      Run the show interface status command on each upper leaf switch: 

      1. R1UL# show interface status
      2. R2UL# show interface status

      Examine the output to ensure that the storage server ports are connected.

      For example:

      R1UL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected routed    full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected routed    full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected routed    full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected routed    full   100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           connected 3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           connected 3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           connected 3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           connected 3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           connected 3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           connected 3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              disabled  3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           connected 3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           connected 3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           connected 3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           connected 3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           connected 3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           connected 3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           connected 3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected routed    full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected routed    full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected routed    full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected routed    full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --
    7. Start the database server ports on the upper leaf switches (R1UL and R2UL).

      1. On R1UL:

        R1UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R1UL(config)# interface ethernet 1/15-22
R1UL(config-if-range)# no shut
R1UL(config-if-range)# exit
R1UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R1UL(config)# <Ctrl-Z>
R1UL#

      2. Repeat the command sequence on R2UL:

        R2UL# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R2UL(config)# interface ethernet 1/15-22
R2UL(config-if-range)# no shut
R2UL(config-if-range)# exit
R2UL(config)# copy running-config startup-config
[########################################] 100%
Copy complete, now saving to disk (please wait)...
Copy complete
R2UL(config)# <Ctrl-Z>
R2UL#
    8. Wait for 5 minutes to ensure that the ports you just started are fully operational before continuing.
    9. Verify the status of the database server ports on the upper leaf switches (R1UL and R2UL).

      Run the show interface status command on each upper leaf switch: 

      1. R1UL# show interface status
      2. R2UL# show interface status

      Examine the output to ensure that the database server ports are connected.

      For example:

      R1UL# show interface status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full   1000    --

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 xcvrAbsen 1         auto   auto    --
Eth1/2        --                 xcvrAbsen 1         auto   auto    --
Eth1/3        --                 xcvrAbsen 1         auto   auto    --
Eth1/4        ISL1               connected routed    full   100G    QSFP-100G-CR4
Eth1/5        ISL2               connected routed    full   100G    QSFP-100G-CR4
Eth1/6        ISL3               connected routed    full   100G    QSFP-100G-CR4
Eth1/7        ISL4               connected routed    full   100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full   100G    QSFP-100G-CR4
Eth1/9        celadm13           connected 3888      full   100G    QSFP-100G-CR4
Eth1/10       celadm12           connected 3888      full   100G    QSFP-100G-CR4
Eth1/11       celadm11           connected 3888      full   100G    QSFP-100G-CR4
Eth1/12       celadm10           connected 3888      full   100G    QSFP-100G-CR4
Eth1/13       celadm09           connected 3888      full   100G    QSFP-100G-CR4
Eth1/14       celadm08           connected 3888      full   100G    QSFP-100G-CR4
Eth1/15       adm08              connected 3888      full   100G    QSFP-100G-CR4
Eth1/16       adm07              connected 3888      full   100G    QSFP-100G-CR4
Eth1/17       adm06              connected 3888      full   100G    QSFP-100G-CR4
Eth1/18       adm05              connected 3888      full   100G    QSFP-100G-CR4
Eth1/19       adm04              connected 3888      full   100G    QSFP-100G-CR4
Eth1/20       adm03              connected 3888      full   100G    QSFP-100G-CR4
Eth1/21       adm02              connected 3888      full   100G    QSFP-100G-CR4
Eth1/22       adm01              connected 3888      full   100G    QSFP-100G-CR4
Eth1/23       celadm07           connected 3888      full   100G    QSFP-100G-CR4
Eth1/24       celadm06           connected 3888      full   100G    QSFP-100G-CR4
Eth1/25       celadm05           connected 3888      full   100G    QSFP-100G-CR4
Eth1/26       celadm04           connected 3888      full   100G    QSFP-100G-CR4
Eth1/27       celadm03           connected 3888      full   100G    QSFP-100G-CR4
Eth1/28       celadm02           connected 3888      full   100G    QSFP-100G-CR4
Eth1/29       celadm01           connected 3888      full   100G    QSFP-100G-CR4
Eth1/30       ISL5               connected routed    full   100G    QSFP-100G-CR4
Eth1/31       ISL6               connected routed    full   100G    QSFP-100G-CR4
Eth1/32       ISL7               connected routed    full   100G    QSFP-100G-CR4
Eth1/33       ISL8               connected routed    full   100G    QSFP-100G-CR4
Eth1/34       --                 xcvrAbsen 1         auto   auto    --
Eth1/35       --                 xcvrAbsen 1         auto   auto    --
Eth1/36       --                 xcvrAbsen 1         auto   auto    --
Po100         --                 connected trunk     full   100G    --
Lo0           Routing loopback i connected routed    auto   auto    --
Lo1           VTEP loopback inte connected routed    auto   auto    --
Vlan1         --                 down      routed    auto   auto    --
nve1          --                 connected --        auto   auto    --
    10. Verify that nve is up on the leaf switches (R1LL, R1UL, R2LL, and R2UL).

      Run the following command on each leaf switch and examine the output:

      1. R1LL# show nve peers
      2. R1UL# show nve peers
      3. R2LL# show nve peers
      4. R2UL# show nve peers

      In the output, you should see three nve peers with State=Up.

      For example:

      R1UL# show nve peers
Interface         Peer-IP State LearnType   Uptime        Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1         100.64.1.101    Up        CP 00:04:29               n/a
nve1         100.64.1.103    Up        CP 00:07:48               n/a
nve1         100.64.1.104    Up        CP 00:04:10               n/a
    11. Verify that BGP is up on the upper leaf switches (R1UL and R2UL).

      Run the following command on each upper leaf switch and examine the output:

      1. R1UL# show logging log | grep BGP
      2. R2UL# show logging log | grep BGP

      In the output, look for two entries with Up in the rightmost column that are associated with different IP addresses.

      For example:

      R1UL# show logging log | grep BGP
2020 Aug 10 13:57:13 R1UL %BGP-5-ADJCHANGE: bgp- [32782] (default) neighbor 100.64.0.201 Up 
2020 Aug 10 13:57:24 R1UL %BGP-5-ADJCHANGE: bgp- [32782] (default) neighbor 100.64.0.202 Up
  7. For each rack (R1 and R2), confirm the multi-rack cabling by running the verify_roce_cables.py script.

    The verify_roce_cables.py script uses two input files; one for database servers and storage servers (nodes.rackN), and another for switches (switches.rackN). In each file, every server or switch must be listed on separate lines. Use fully qualified domain names or IP addresses for each server and switch.

    See My Oracle Support document 2587717.1 for download and detailed usage instructions.

    Run the verify_roce_cables.py script against both of the racks: 

    1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
    2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack2 -s switches.rack2

    Check the output of the verify_roce_cables.py script against the tables in Two-Rack Cabling for X8M Racks. Also, check that output in the CABLE OK? columns contains the OK status.

    The following examples show extracts of the expected command results:

    # cd /opt/oracle.SupportTools/ibdiagtools
# ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
SWITCH PORT (EXPECTED PEER)  LOWER LEAF (rack1sw-rocea0) : CABLE OK?  UPPER LEAF (rack1sw-roceb0) : CABLE OK?
----------- ---------------  --------------------------- : ---------  --------------------------- : ---------
Eth1/4 (ISL peer switch)   : rack1sw-roces0 Ethernet1/17 : OK         rack1sw-roces0 Ethernet1/9  : OK
Eth1/5 (ISL peer switch)   : rack1sw-roces0 Ethernet1/13 : OK         rack1sw-roces0 Ethernet1/5  : OK
Eth1/6 (ISL peer switch)   : rack1sw-roces0 Ethernet1/19 : OK         rack1sw-roces0 Ethernet1/11 : OK
Eth1/7 (ISL peer switch)   : rack1sw-roces0 Ethernet1/15 : OK         rack1sw-roces0 Ethernet1/7  : OK
Eth1/12 (celadm10)         : rack1celadm10 port-1        : OK         rack1celadm10 port-2        : OK
Eth1/13 (celadm09)         : rack1celadm09 port-1        : OK         rack1celadm09 port-2        : OK
Eth1/14 (celadm08)         : rack1celadm08 port-1        : OK         rack1celadm08 port-2        : OK
... 
Eth1/15 (adm08)            : rack1dbadm08 port-1         : OK         rack1dbadm08 port-2         : OK
Eth1/16 (adm07)            : rack1dbadm07 port-1         : OK         rack1dbadm07 port-2         : OK
Eth1/17 (adm06)            : rack1dbadm06 port-1         : OK         rack1dbadm06 port-2         : OK
... 
Eth1/30 (ISL peer switch)  : rack2sw-roces0 Ethernet1/17 : OK         rack2sw-roces0 Ethernet1/9  : OK
Eth1/31 (ISL peer switch)  : rack2sw-roces0 Ethernet1/13 : OK         rack2sw-roces0 Ethernet1/5  : OK
Eth1/32 (ISL peer switch)  : rack2sw-roces0 Ethernet1/19 : OK         rack2sw-roces0 Ethernet1/11 : OK
Eth1/33 (ISL peer switch)  : rack2sw-roces0 Ethernet1/15 : OK         rack2sw-roces0 Ethernet1/7  : OK
    # ./verify_roce_cables.py -n nodes.rack2 -s switches.rack2
SWITCH PORT (EXPECTED PEER)  LOWER LEAF (rack2sw-rocea0) : CABLE OK?  UPPER LEAF (rack2sw-roceb0) : CABLE OK?
----------- ---------------  --------------------------- : ---------  --------------------------- : ---------
Eth1/4 (ISL peer switch)   : rack1sw-roces0 Ethernet1/18 : OK         rack1sw-roces0 Ethernet1/10 : OK
...
  8. Verify the RoCE Network Fabric operation across both interconnected racks by using the infinicheck command.

    Use the following recommended command sequence to verify the RoCE Network Fabric operation across both racks.

    In each command, hosts.all contains a list of database server RoCE Network Fabric IP addresses from both racks (2 RoCE Network Fabric IP addresses for each database server), and cells.all contains a list of RoCE Network Fabric IP addresses for the storage servers from both racks (2 RoCE Network Fabric IP addresses for each storage server). 

    1. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.all -c cells.all -z
    2. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.all -c cells.all -s
    3. # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.all -c cells.all -b

    See step 1.k for most information about each infinicheck command.

    The following example shows the expected command results for the final command in the sequence:

    # cd /opt/oracle.SupportTools/ibdiagtools
# ./infinicheck -g hosts.all -c cells.all -b

INFINICHECK                    
        [Network Connectivity, Configuration and Performance]        
               
          ####  FABRIC TYPE TESTS  #### 
System type identified: RoCE
Verifying User Equivalance of user=root from all DBs to all CELLs.
     ####  RoCE CONFIGURATION TESTS  ####       
     Checking for presence of RoCE devices on all DBs and CELLs 
[SUCCESS].... RoCE devices on all DBs and CELLs look good
     Checking for RoCE Policy Routing settings on all DBs and CELLs 
[SUCCESS].... RoCE Policy Routing settings look good
     Checking for RoCE DSCP ToS mapping on all DBs and CELLs 
[SUCCESS].... RoCE DSCP ToS settings look good
     Checking for RoCE PFC settings and DSCP mapping on all DBs and CELLs
[SUCCESS].... RoCE PFC and DSCP settings look good
     Checking for RoCE interface MTU settings. Expected value : 2300
[SUCCESS].... RoCE interface MTU settings look good
     Verifying switch advertised DSCP on all DBs and CELLs ports ( )
[SUCCESS].... Advertised DSCP settings from RoCE switch looks good  
    ####  CONNECTIVITY TESTS  ####
    [COMPUTE NODES -> STORAGE CELLS] 
      (60 seconds approx.)       
    (Will walk through QoS values: 0-6) [SUCCESS]..........Results OK
[SUCCESS]....... All  can talk to all storage cells          
    [COMPUTE NODES -> COMPUTE NODES]               
...

At this point, both racks share the RoCE Network Fabric, and the combined system is ready for further configuration.

See Configuring the New Hardware.

2.3.2.2 Cabling Two RoCE Network Fabric Racks Together with Down Time Allowed

Use this simpler procedure to cable together two racks with RoCE Network Fabric where some down-time can be tolerated.

This procedure is for systems with RoCE Network Fabric (X8M or later) using Oracle Exadata System Software Release 20.1.0 or later.

In this procedure, the existing rack is R1, and the new rack is R2.

Use the applicable cabling tables depending on your system configuration:

  1. Ensure the new rack is near the existing rack.
    The RDMA Network Fabric cables must be able to reach the servers in each rack.
  2. Ensure you have a backup of the current switch configuration for each switch in the existing and new rack.

    See Backing Up Settings on the RoCE Network Fabric Switch in Oracle Exadata Database Machine Maintenance Guide.

  3. Shut down all servers on both the new rack (R2) and the existing rack (R1).
    The switches should remain available.
  4. Update the firmware to the latest available release on all of the RoCE Network Fabric switches.

    For this step, treat all of the switches as if they belong to a single rack system.

    See Updating RoCE Network Fabric Switch Firmware in Oracle Exadata Database Machine Maintenance Guide.

  5. Apply the multi-rack golden configuration settings on the RoCE Network Fabric switches.

    Use the procedure described in Applying Golden Configuration Settings on RoCE Network Fabric Switches, in Oracle Exadata Database Machine Maintenance Guide.

  6. Enable the leaf switch server ports.

    The leaf switch server ports may be disabled as a consequence of applying the multi-rack golden configuration settings in the previous step.

    To ensure that the leaf switch server ports are enabled, log in to each of the four leaf switches and run the following commands on each leaf switch:

    rack1sw-rocea0# config term
rack1sw-rocea0# int eth1/8-30
rack1sw-rocea0# no shut
rack1sw-rocea0# copy running-config startup-config
  7. Perform the physical cabling of the switches.
    1. In Rack 2, remove the existing inter-switch connections between the two leaf switches, R2UL and R2LL.
    2. In Rack 2, cable each leaf switch to the spine switches using the applicable cabling tables.
    3. In Rack 1, remove the existing inter-switch connections between the two leaf switches, R1UL and R1LL.
    4. In Rack 1, cable each leaf switch to the spine switches using the applicable cabling tables.
  8. Confirm each switch is available and connected.

    For each of the six switches (two spine switches and four leaf switches), confirm the output from the show interface status command shows connected and 100G for each connected inter-switch port. Use the appropriate cabling tables to identify the ports that should be connected.

    In the following examples, the leaf switches are ports Eth1/4 to Eth1/7, and Eth1/30 to Eth1/33. The spine switches are ports Eth1/5 to Eth1/20.

    When run from a spine switch, the output should be similar to the following:

    rack1sw-roces0# show interface status
--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full    1000    -- 
--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
...
Eth1/5        RouterPort5        connected routed    full    100G    QSFP-100G-CR4
Eth1/6        RouterPort6        connected routed    full    100G    QSFP-100G-SR4
Eth1/7        RouterPort7        connected routed    full    100G    QSFP-100G-CR4
Eth1/8        RouterPort8        connected routed    full    100G    QSFP-100G-SR4
Eth1/9        RouterPort9        connected routed    full    100G    QSFP-100G-CR4
Eth1/10       RouterPort10       connected routed    full    100G    QSFP-100G-SR4
Eth1/11       RouterPort11       connected routed    full    100G    QSFP-100G-CR4
Eth1/12       RouterPort12       connected routed    full    100G    QSFP-100G-SR4
Eth1/13       RouterPort13       connected routed    full    100G    QSFP-100G-CR4
Eth1/14       RouterPort14       connected routed    full    100G    QSFP-100G-SR4
Eth1/15       RouterPort15       connected routed    full    100G    QSFP-100G-CR4
Eth1/16       RouterPort16       connected routed    full    100G    QSFP-100G-SR4
Eth1/17       RouterPort17       connected routed    full    100G    QSFP-100G-CR4
Eth1/18       RouterPort18       connected routed    full    100G    QSFP-100G-SR4
Eth1/19       RouterPort19       connected routed    full    100G    QSFP-100G-CR4
Eth1/20       RouterPort20       connected routed    full    100G    QSFP-100G-SR4
Eth1/21       RouterPort21       xcvrAbsen      routed    full    100G    --
...

    When run from a leaf switch, the output should be similar to the following:

    rack1sw-rocea0# show interface status
--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full    1000    -- 
--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
...
Eth1/4        RouterPort1        connected routed    full    100G    QSFP-100G-CR4
Eth1/5        RouterPort2        connected routed    full    100G    QSFP-100G-CR4
Eth1/6        RouterPort3        connected routed    full    100G    QSFP-100G-CR4
Eth1/7        RouterPort4        connected routed    full    100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full    100G    QSFP-100G-CR4
...
Eth1/29       celadm01           connected 3888      full    100G    QSFP-100G-CR4
Eth1/30       RouterPort5        connected routed    full    100G    QSFP-100G-SR4
Eth1/31       RouterPort6        connected routed    full    100G    QSFP-100G-SR4
Eth1/32       RouterPort7        connected routed    full    100G    QSFP-100G-SR4
Eth1/33       RouterPort8        connected routed    full    100G    QSFP-100G-SR4
...
  9. Check the neighbor discovery for every switch in racks R1 and R2.

    Log in to each switch and use the show lldp neighbors command. Make sure that all switches are visible and check the switch ports assignment against the applicable cabling tables.

    A spine switch should see the two leaf switches in each rack, but not the other spine switch. The output for a spine switch should be similar to the following:

    Note:

    The interfaces output in the Port ID column are different for each switch based on the applicable cabling tables. 
    rack1sw-roces0# show lldp neighbors
...
Device ID            Local Intf      Hold-time  Capability  Port ID
rack1-adm0           mgmt0           120        BR          Ethernet1/47
rack1sw-roceb0       Eth1/5     120        BR          Ethernet1/5
rack2sw-roceb0       Eth1/6     120        BR          Ethernet1/5
rack1sw-roceb0       Eth1/7     120        BR          Ethernet1/7
rack2sw-roceb0       Eth1/8     120        BR          Ethernet1/7
rack1sw-roceb0       Eth1/9     120        BR          Ethernet1/4
rack2sw-roceb0       Eth1/10    120        BR          Ethernet1/4
rack1sw-roceb0       Eth1/11    120        BR          Ethernet1/6
rack2sw-roceb0       Eth1/12    120        BR          Ethernet1/6
rack1sw-rocea0       Eth1/13    120        BR          Ethernet1/5
rack2sw-rocea0       Eth1/14    120        BR          Ethernet1/5
rack1sw-rocea0       Eth1/15    120        BR          Ethernet1/7
rack2sw-rocea0       Eth1/16    120        BR          Ethernet1/7
rack1sw-rocea0       Eth1/17    120        BR          Ethernet1/4
rack2sw-rocea0       Eth1/18    120        BR          Ethernet1/4
rack1sw-rocea0       Eth1/19    120        BR          Ethernet1/6 
rack2sw-rocea0       Eth1/20    120        BR          Ethernet1/6
Total entries displayed: 17

    Each leaf switch should see the two spine switches, but not the other leaf switches. The output for a leaf switch should be similar to the following:

    Note:

    The interfaces output in the Port ID column are different for each switch based on the applicable cabling tables. 
    rack1sw-rocea0# show lldp neighbors
...
Device ID            Local Intf      Hold-time  Capability  Port ID
switch               mgmt0      120        BR          Ethernet1/46
rack1sw-roces0       Eth1/4     120        BR          Ethernet1/17
rack1sw-roces0       Eth1/5     120        BR          Ethernet1/13
rack1sw-roces0       Eth1/6     120        BR          Ethernet1/19
rack1sw-roces0       Eth1/7     120        BR          Ethernet1/15
rack2sw-roces0       Eth1/30    120        BR          Ethernet1/17
rack2sw-roces0       Eth1/31    120        BR          Ethernet1/13
rack2sw-roces0       Eth1/32    120        BR          Ethernet1/19
rack2sw-roces0       Eth1/33    120        BR          Ethernet1/15
rocetoi-ext-sw       Eth1/36    120        BR          Ethernet1/49
Total entries displayed: 10
  10. Power on all servers in racks R1 and R2.
  11. For each rack, confirm the multi-rack cabling by running the verify_roce_cables.py script.

    Refer to My Oracle Support Doc ID 2587717.1 for download and usage instructions.

    Check the output of the verify_roce_cables.py script against the applicable cabling tables. Also, check that output in the CABLE OK? columns contains the OK status.

    When running the script, two input files are used, one for nodes and one for switches. Each file should contain the servers or switches on separate lines. Use fully qualified domain names or IP addresses for each server and switch.

    The following output is a partial example of the command results:

    # ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
SWITCH PORT (EXPECTED PEER)  LEAF-1 (rack1sw-rocea0)     : CABLE OK?  LEAF-2 (rack1sw-roceb0)    : CABLE OK?
----------- --------------   --------------------------- : --------   -----------------------    : ---------
Eth1/4 (ISL peer switch)   : rack1sw-roces0 Ethernet1/17 : OK         rack1sw-roces0 Ethernet1/9 : OK
Eth1/5 (ISL peer switch)   : rack1sw-roces0 Ethernet1/13 : OK         rack1sw-roces0 Ethernet1/5 : OK
Eth1/6 (ISL peer switch)   : rack1sw-roces0 Ethernet1/19 : OK         rack1sw-roces0 Ethernet1/11: OK
Eth1/7 (ISL peer switch)   : rack1sw-roces0 Ethernet1/15 : OK         rack1sw-roces0 Ethernet1/7 : OK
Eth1/12 (celadm10)         : rack1celadm10 port-1        : OK         rack1celadm10 port-2       : OK
Eth1/13 (celadm09)         : rack1celadm09 port-1        : OK         rack1celadm09 port-2       : OK
Eth1/14 (celadm08)         : rack1celadm08 port-1        : OK         rack1celadm08 port-2       : OK
...
Eth1/15 (adm08)            : rack1dbadm08 port-1         : OK         rack1dbadm08 port-2        : OK
Eth1/16 (adm07)            : rack1dbadm07 port-1         : OK         rack1dbadm07 port-2        : OK
Eth1/17 (adm06)            : rack1dbadm06 port-1         : OK         rack1dbadm06 port-2        : OK
...
Eth1/30 (ISL peer switch)  : rack2sw-roces0 Ethernet1/17 : OK         rack2sw-roces0 Ethernet1/9 : OK
Eth1/31 (ISL peer switch)  : rack2sw-roces0 Ethernet1/13 : OK         rack2sw-roces0 Ethernet1/5 : OK
Eth1/32 (ISL peer switch)  : rack2sw-roces0 Ethernet1/19 : OK         rack2sw-roces0 Ethernet1/11: OK
Eth1/33 (ISL peer switch)  : rack2sw-roces0 Ethernet1/15 : OK         rack2sw-roces0 Ethernet1/7 : OK

    # ./verify_roce_cables.py -n nodes.rack2 -s switches.rack2
SWITCH PORT (EXPECTED PEER)  LEAF-1 (rack2sw-rocea0)     : CABLE OK?  LEAF-2 (rack2sw-roceb0)    : CABLE OK?
----------- --------------   --------------------------- : --------   -----------------------    : ---------
Eth1/4 (ISL peer switch)  :  rack1sw-roces0 Ethernet1/18 : OK         rack1sw-roces0 Ethernet1/10: OK
...
  12. Verify the RoCE Network Fabric operation by using the infinicheck command.

    Use the following recommended command sequence. In each command, hosts.lst contains a list of database server RoCE Network Fabric IP addresses (2 RoCE Network Fabric IP addresses for each database server), and cells.lst contains a list of RoCE Network Fabric IP addresses for the storage servers (2 RoCE Network Fabric IP addresses for each storage server).

    • Use infinicheck with the -z option to clear the files that were created during the last run of the infinicheck command. For example: 

      # /opt/oracle.SupportTools/ibdiagtools/infinicheck -g hosts.lst -c cells.lst -z

    • Use infinicheck with the -s option to set up user equivalence for password-less SSH across the RoCE Network Fabric. For example: 

      # /opt/oracle.SupportTools/ibdiagtools/infinicheck -g hosts.lst -c cells.lst -s

    • Finally, verify the RoCE Network Fabric operation by using infinicheck with the -b option, which is recommended on newly imaged machines where it is acceptable to suppress the cellip.ora and cellinit.ora configuration checks. For example: 

      # /opt/oracle.SupportTools/ibdiagtools/infinicheck -g hosts.lst -c cells.lst -b

INFINICHECK                    
        [Network Connectivity, Configuration and Performance]        
               
          ####  FABRIC TYPE TESTS  #### 
System type identified: RoCE
Verifying User Equivalance of user=root from all DBs to all CELLs.
     ####  RoCE CONFIGURATION TESTS  ####       
     Checking for presence of RoCE devices on all DBs and CELLs 
[SUCCESS].... RoCE devices on all DBs and CELLs look good
     Checking for RoCE Policy Routing settings on all DBs and CELLs 
[SUCCESS].... RoCE Policy Routing settings look good
     Checking for RoCE DSCP ToS mapping on all DBs and CELLs 
[SUCCESS].... RoCE DSCP ToS settings look good
     Checking for RoCE PFC settings and DSCP mapping on all DBs and CELLs
[SUCCESS].... RoCE PFC and DSCP settings look good
     Checking for RoCE interface MTU settings. Expected value : 2300
[SUCCESS].... RoCE interface MTU settings look good
     Verifying switch advertised DSCP on all DBs and CELLs ports ( )
[SUCCESS].... Advertised DSCP settings from RoCE switch looks good  
    ####  CONNECTIVITY TESTS  ####
    [COMPUTE NODES -> STORAGE CELLS] 
      (60 seconds approx.)       
    (Will walk through QoS values: 0-6) [SUCCESS]..........Results OK
[SUCCESS]....... All  can talk to all storage cells          
    [COMPUTE NODES -> COMPUTE NODES]               
...
  13. After cabling the racks together, proceed to Configuring the New Hardware to finish the configuration of the new rack.
2.3.2.3 Cabling Two InfiniBand Network Fabric Racks Together

Use this procedure to cable together two racks with InfiniBand Network Fabric.

This procedure assumes that the racks are adjacent to each other.
In the procedure, the existing rack is R1, and the new rack is R2.

  1. Set the priority of the current, active Subnet Manager Master to 10 on the spine switch, as follows:

    1. Log in to any RDMA Network Fabric switch on the active system.

    2. Use the getmaster command to determine that the Subnet Manager Master is running on the spine switch. If it is not, then follow the procedure Setting the Subnet Manager Master on Oracle Exadata Database Machine Full Rack and Oracle Exadata Database Machine Half Rack in Oracle Exadata Database Machine Installation and Configuration Guide.

    3. Log in to the spine switch.

    4. Use the disablesm command to stop Subnet Manager.

    5. Use the setsmpriority 10 command to set the priority to 10.

    6. Use the enablesm command to restart Subnet Manager.

    7. Repeat step 1.b to ensure the Subnet Manager Master is running on the spine switch.

  2. Ensure the new rack is near the existing rack. The RDMA Network Fabric cables must be able to reach the servers in each rack.

  3. Completely shut down the new rack (R2).

  4. Cable the two leaf switches R2 IB2 and R2 IB3 in the new rack according to Two-Rack Cabling with InfiniBand Network Fabric. Note that you need to first remove the seven existing inter-switch connections between each leaf switch, as well as the two connections between the leaf switches and the spine switch in the new rack R2, not in the existing rack R1.

  5. Verify both RDMA Network Fabric interfaces are up on all database nodes and storage cells. You can do this by running the ibstat command on each node and verifying both interfaces are up.

  6. Power off leaf switch R1 IB2. This causes all the database servers and Exadata Storage Servers to fail over their RDMA Network Fabric traffic to R1 IB3.

  7. Disconnect all seven inter-switch links between R1 IB2 and R1 IB3, as well as the one connection between R1 IB2 and the spine switch R1 IB1.

  8. Cable leaf switch R1 IB2 according to Two-Rack Cabling with InfiniBand Network Fabric.

  9. Power on leaf switch R1 IB2.

  10. Wait for three minutes for R1 IB2 to become completely operational.

    To check the switch, log in to the switch and run the ibswitches command. The output should show three switches, R1 IB1, R1 IB2, and R1 IB3.

  11. Verify both RDMA Network Fabric interfaces are up on all database nodes and storage cells. You can do this by running the ibstat command on each node and verifying both interfaces are up.

  12. Power off leaf switch R1 IB3. This causes all the database servers and storage servers to fail over their RDMA Network Fabric traffic to R1 IB2.

  13. Disconnect the one connection between R1 IB3 and the spine switch R1 IB1.

  14. Cable leaf switch R1 IB3 according to Two-Rack Cabling with InfiniBand Network Fabric.

  15. Power on leaf switch R1 IB3.

  16. Wait for three minutes for R1 IB3 to become completely operational.

    To check the switch, log in to the switch and run the ibswitches command. The output should show three switches, R1 IB1, R1 IB2, and R1 IB3.

  17. Power on all the InfiniBand switches in R2.

  18. Wait for three minutes for the switches to become completely operational.

    To check the switch, log in to the switch and run the ibswitches command. The output should show six switches, R1 IB1, R1 IB2, R1 IB3, R2 IB1, R2 IB2, and R2 IB3.

  19. Ensure the Subnet Manager Master is running on R1 IB1 by running the getmaster command from any switch.

  20. Power on all servers in R2.

  21. Log in to spine switch R1 IB1, and lower its priority to 8 as follows:

    1. Use the disablesm command to stop Subnet Manager.

    2. Use the setsmpriority 8 command to set the priority to 8.

    3. Use the enablesm command to restart Subnet Manager.

  22. Ensure Subnet Manager Master is running on one of the spine switches.

After cabling the racks together, proceed to Configuring the New Hardware to configure the racks.

2.3.3 Cabling Several Racks Together

By using the following procedures, you can add another rack to an existing multi-rack system.

The procedures assume that the racks are adjacent to each other. The existing racks are R1, R2, ... Rn, and the new rack is Rn+1. For example, if you have four racks and you are adding a fifth rack, the existing racks are designated R1, R2, R3, and R4 and the new rack is designated R5.

Starting with Oracle Exadata X9M models, you can cable up to 12 racks together without additional switches. For prior models (up to X8M), you can cable up to 18 racks together without additional switches.

The procedures differ for systems that use RoCE Network Fabric (X8M and later) and InfiniBand Network Fabric (X8 and earlier).

Parent topic: Extending a Rack by Adding Another Rack

2.3.3.1 Cabling Several RoCE Network Fabric Racks Together

Use this procedure to add another rack to an existing multi-rack system with RoCE Network Fabric.

This procedure is for systems with RoCE Network Fabric (X8M or later) using Oracle Exadata System Software Release 20.1.0 or later.

WARNING:

Take time to read and understand this procedure before implementation. Pay careful attention to all the instructions, not just the command examples. A system outage may occur if the instructions are not applied correctly.

In this procedure, the existing racks are R1, R2, … ,Rn, and the new rack is Rn+1.

Note:

Cabling three or more racks together requires no downtime for the existing racks R1, R2, …, Rn. Only the new rack, Rn+1, is powered down.

Use the applicable cabling tables depending on your system:

In the following steps, these example switch names are used for the new rack (Rn+1):

  • rack5sw-roces0: Rack 5 spine switch (R5SS)
  • rack5sw-rocea0: Rack 5 lower leaf switch (R5LL)
  • rack5sw-roceb0: Rack 5 upper leaf switch (R5UL)
  1. Ensure the new rack is near the existing racks (R1, R2, …, Rn).
    The RDMA Network Fabric cables must be able to reach the servers in each rack.
  2. Ensure you have a backup of the current switch configuration for each switch in the existing racks and the new rack.
    For each switch, complete the steps in the Oracle Exadata Database Machine Maintenance Guide, section Backing Up Settings on the RoCE Network Fabric Switch.
  3. Shut down all servers in the new rack (Rn+1).
    Refer to Powering Off Oracle Exadata Rack. The switches must remain online and available.
  4. Verify the configuration of the existing RoCE Network Fabric switches.

    Before you configure the RoCE Network Fabric switches in the new rack (Rn+1), check the configuration of the RoCE Network Fabric switches in the existing racks (R1, R2, …, Rn). You must do this to ensure that every switch uses a unique loopback octet. The loopback octet is the last octet of the switch loopback IP address.

    1. Connect to an existing RoCE Network Fabric leaf switches and determine the loopback octet for the switch.

      Use the command shown in the following example.

      rack1sw-rocea0# show interface loopback 1 | grep Address

Internet Address is 192.128.10.101/32

      In the example, the loopback octet is 101.

    2. Determine the loopback octet for every other leaf switch.

      Use the command shown in the following example.

      rack1sw-rocea0# show nve peers

Interface Peer-IP                                State LearnType
--------- -------------------------------------- ----- ---------
nve1      192.128.10.102                         Up    CP
nve1      192.128.10.103                         Up    CP
nve1      192.128.10.104                         Up    CP
nve1      192.128.10.105                         Up    CP
nve1      192.128.10.106                         Up    CP
nve1      192.128.10.107                         Up    CP
nve1      192.128.10.108                         Up    CP

      In the example, the output shows seven other leaf switches having loopback octet values from 102 to 108. This output is consistent with an existing system containing four racks.

    3. Determine the loopback octet for every spine switch.

      Use the command shown in the following example.

      rack1sw-rocea0# show bgp l2vpn evpn summary | egrep -v
"BGP|Idle|I|Neighbor|memory"

192.128.10.201 4 65502 9161 581 75716 0 0 08:53:23 3687
192.128.10.202 4 65502 9160 582 75716 0 0 08:34:20 3687
192.128.10.203 4 65502 9162 582 75716 0 0 08:41:22 3687
192.128.10.204 4 65502 9163 582 75716 0 0 08:50:27 3687

      In the example, the output shows four spine switches having loopback octet values from 201 to 204. This output is also consistent with an existing system containing four racks.

    4. Validate the configuration of the existing RoCE Network Fabric switches.

      Check the information gathered from the existing RoCE Network Fabric switches to ensure that every switch uses a unique loopback octet value and that all the values are as expected.

      Verify that the information gathered from the existing RoCE Network Fabric switches conforms to the following conventions:

      • On the leaf switches, the overall range of loopback octet values should start with 101 and increase incrementally (by 1) for each leaf switch.

        According to the best-practice convention, the loopback octet value for each leaf switch should be configured as follows:

        • 101 - Rack 1 lower leaf switch (R1LL)

        • 102 - Rack 1 upper leaf switch (R1UL)

        • 103 - Rack 2 lower leaf switch (R2LL)

        • 104 - Rack 2 upper leaf switch (R2UL)

        • 105 - Rack 3 lower leaf switch (R3LL)

        • 106 - Rack 3 upper leaf switch (R3UL), and so on.

      • On the spine switches, the range of loopback octet values should start with 201 and increase incrementally (by 1) for each spine switch.

        According to the best-practice convention, the loopback octet value for each spine switch should be configured as follows:

        • 201 - Rack 1 spine switch (R1SS)

        • 202 - Rack 2 spine switch (R2SS)

        • 203 - Rack 3 spine switch (R3SS)

        • 204 - Rack 4 spine switch (R4SS), and so on.

      Caution:

      If the switches in the existing racks (R1, R2, …, Rn) don't conform to the above conventions, then you must take special care to assign unique loopback octet values to the switches in the new rack (Rn+1) as part of applying their golden configuration settings (in the next step).

      If multiple switches use the same loopback octet, the RoCE Network Fabric cannot function correctly, resulting in a system outage.

  5. Apply the golden configuration settings on the RoCE Network Fabric switches in the new rack (Rn+1).

    Combine the information about the existing RoCE Network Fabric switches you gathered in the previous step and the procedure described in Applying Golden Configuration Settings on RoCE Network Fabric Switches (in Oracle Exadata Database Machine Maintenance Guide).

    Caution:

    Take care when performing this step, as misconfiguration of the RoCE Network Fabric will likely cause a system outage.

    For example, every switch in a multi-rack configuration must have a unique loopback octet. If multiple switches use the same loopback octet, the RoCE Network Fabric cannot function correctly, resulting in a system outage.

  6. Enable the leaf switch server ports on the RoCE Network Fabric leaf switches in the new rack (Rn+1).

    The leaf switch server ports may be disabled as a consequence of applying the multi-rack golden configuration settings in the previous step.

    To ensure that the leaf switch server ports are enabled, log in to each of the leaf switches in the new rack and run the following commands on each switch:

    rack5sw-rocea0# config term
rack5sw-rocea0# int eth1/8-30
rack5sw-rocea0# no shut
rack5sw-rocea0# copy running-config startup-config
  7. Perform the physical cabling of the switches in the new rack (Rn+1).

    Caution:

    Cabling within a live network must be done carefully in order to avoid potentially serious disruptions.
    1. Remove the eight existing inter-switch connections (ports 4, 5, 6, 7 and 30, 31, 32, 33) between each leaf switch in the new rack (Rn+1).
    2. Cable the leaf switches in the new rack according to the applicable cabling table.

      For example, if you are adding a 5th rack to a system using Exadata X9M (or later model) racks, then use "Table 4-17 Leaf Switch Connections for the Fifth Rack in a Five-Rack System".

  8. Add the new rack to the switches in the existing racks (R1 to Rn).
    1. For an existing rack (Rx), cable the lower leaf switch RxLL according to the applicable cabling table.
    2. For the same rack, cable the upper leaf switch RxUL according to the applicable cabling table.
    3. Repeat these steps for each existing rack, R1 to Rn.
  9. Confirm each switch is available and connected.

    For each switch in racks R1, R2, …, Rn, Rn+1, confirm the output for the switch show interface status command shows connected and 100G.

    When run from a spine switch, the output should be similar to the following:

    rack1sw-roces0# show interface status
--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full    1000    -- 
--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
...
Eth1/5        RouterPort5        connected routed    full    100G    QSFP-100G-CR4
Eth1/6        RouterPort6        connected routed    full    100G    QSFP-100G-SR4
Eth1/7        RouterPort7        connected routed    full    100G    QSFP-100G-CR4
Eth1/8        RouterPort8        connected routed    full    100G    QSFP-100G-SR4
Eth1/9        RouterPort9        connected routed    full    100G    QSFP-100G-CR4
Eth1/10       RouterPort10       connected routed    full    100G    QSFP-100G-SR4
Eth1/11       RouterPort11       connected routed    full    100G    QSFP-100G-CR4
Eth1/12       RouterPort12       connected routed    full    100G    QSFP-100G-SR4
Eth1/13       RouterPort13       connected routed    full    100G    QSFP-100G-CR4
Eth1/14       RouterPort14       connected routed    full    100G    QSFP-100G-SR4
Eth1/15       RouterPort15       connected routed    full    100G    QSFP-100G-CR4
Eth1/16       RouterPort16       connected routed    full    100G    QSFP-100G-SR4
Eth1/17       RouterPort17       connected routed    full    100G    QSFP-100G-CR4
Eth1/18       RouterPort18       connected routed    full    100G    QSFP-100G-SR4
Eth1/19       RouterPort19       connected routed    full    100G    QSFP-100G-CR4
Eth1/20       RouterPort20       connected routed    full    100G    QSFP-100G-SR4
Eth1/21       RouterPort21       xcvrAbsen      routed    full    100G    --
...

    When run from a leaf switch, the output should be similar to the following:

    rack1sw-rocea0# show interface status
--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
mgmt0         --                 connected routed    full    1000    -- 
--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
...
Eth1/4        RouterPort1        connected routed    full    100G    QSFP-100G-CR4
Eth1/5        RouterPort2        connected routed    full    100G    QSFP-100G-CR4
Eth1/6        RouterPort3        connected routed    full    100G    QSFP-100G-CR4
Eth1/7        RouterPort4        connected routed    full    100G    QSFP-100G-CR4
Eth1/8        celadm14           connected 3888      full    100G    QSFP-100G-CR4
...
Eth1/29       celadm01           connected 3888      full    100G    QSFP-100G-CR4
Eth1/30       RouterPort5        connected routed    full    100G    QSFP-100G-SR4
Eth1/31       RouterPort6        connected routed    full    100G    QSFP-100G-SR4
Eth1/32       RouterPort7        connected routed    full    100G    QSFP-100G-SR4
Eth1/33       RouterPort8        connected routed    full    100G    QSFP-100G-SR4
...
  10. Check the neighbor discovery for every switch in racks R1, R2, …, Rn, Rn+1.
    Log in to each switch and use the show lldp neighbors command. Make sure that all switches are visible and check the switch ports assignment (leaf switches: ports Eth1/4 - Eth1/7, Eth1/30 - Eth1/33; spine switches: ports Eth1/5 - Eth1/20) against the applicable cabling tables.

    Each spine switch should see all the leaf switches in each rack, but not the other spine switches. The output for a spine switch should be similar to the following:

    Note:

    The interfaces in the rightmost output column (for example, Ethernet1/5) are different for each switch based on the applicable cabling tables. 
    rack1sw-roces0# show lldp neighbors | grep roce
rack1sw-roceb0 Eth1/5 120 BR Ethernet1/5
rack2sw-roceb0 Eth1/6 120 BR Ethernet1/5
rack1sw-roceb0 Eth1/7 120 BR Ethernet1/7
rack2sw-roceb0 Eth1/8 120 BR Ethernet1/7
rack1sw-roceb0 Eth1/9 120 BR Ethernet1/4
rack2sw-roceb0 Eth1/10 120 BR Ethernet1/4
rack3sw-roceb0 Eth1/11 120 BR Ethernet1/5
rack3sw-roceb0 Eth1/12 120 BR Ethernet1/7
rack1sw-rocea0 Eth1/13 120 BR Ethernet1/5
rack2sw-rocea0 Eth1/14 120 BR Ethernet1/5
rack1sw-rocea0 Eth1/15 120 BR Ethernet1/7
rack2sw-rocea0 Eth1/16 120 BR Ethernet1/7
rack3sw-rocea0 Eth1/17 120 BR Ethernet1/5
rack2sw-rocea0 Eth1/18 120 BR Ethernet1/4
rack3sw-rocea0 Eth1/19 120 BR Ethernet1/7
rack3sw-rocea0 Eth1/20 120 BR Ethernet1/4 
...

    Each leaf switch should see the spine switch in every rack, but not the other leaf switches. The output for a leaf switch should be similar to the following:

    Note:

    The interfaces in the rightmost output column (for example, Ethernet1/13) are different for each switch based on the applicable cabling tables. 
    rack1sw-rocea0# show lldp neighbors | grep roce
rack3sw-roces0 Eth1/4 120 BR Ethernet1/13
rack1sw-roces0 Eth1/5 120 BR Ethernet1/13
rack3sw-roces0 Eth1/6 120 BR Ethernet1/15
rack1sw-roces0 Eth1/7 120 BR Ethernet1/15
rack2sw-roces0 Eth1/30 120 BR Ethernet1/17
rack2sw-roces0 Eth1/31 120 BR Ethernet1/13
rack3sw-roces0 Eth1/32 120 BR Ethernet1/17
rack2sw-roces0 Eth1/33 120 BR Ethernet1/15
...
  11. Power on all the servers in the new rack (Rn+1).
  12. For each rack, confirm the multi-rack cabling by running the verify_roce_cables.py script.

    Refer to My Oracle Support Doc ID 2587717.1 for download and usage instructions.

    Check the output of the verify_roce_cables.py script against the applicable cabling tables. Also, check that output in the CABLE OK? columns contains the OK status.

    When running the script, two input files are used, one for nodes and one for switches. Each file should contain the servers or switches on separate lines. Use fully qualified domain names or IP addresses for each server and switch.

    The following output is a partial example of the command results:

    # ./verify_roce_cables.py -n nodes.rack1 -s switches.rack1
SWITCH PORT (EXPECTED PEER)  LEAF-1 (rack1sw-rocea0)     : CABLE OK?  LEAF-2 (rack1sw-roceb0)    : CABLE OK?
----------- --------------   --------------------------- : --------   -----------------------    : ---------
Eth1/4 (ISL peer switch)   : rack1sw-roces0 Ethernet1/17 : OK         rack1sw-roces0 Ethernet1/9 : OK
Eth1/5 (ISL peer switch)   : rack1sw-roces0 Ethernet1/13 : OK         rack1sw-roces0 Ethernet1/5 : OK
Eth1/6 (ISL peer switch)   : rack1sw-roces0 Ethernet1/19 : OK         rack1sw-roces0 Ethernet1/11: OK
Eth1/7 (ISL peer switch)   : rack1sw-roces0 Ethernet1/15 : OK         rack1sw-roces0 Ethernet1/7 : OK
Eth1/12 (celadm10)         : rack1celadm10 port-1        : OK         rack1celadm10 port-2       : OK
Eth1/13 (celadm09)         : rack1celadm09 port-1        : OK         rack1celadm09 port-2       : OK
Eth1/14 (celadm08)         : rack1celadm08 port-1        : OK         rack1celadm08 port-2       : OK
...
Eth1/15 (adm08)            : rack1dbadm08 port-1         : OK         rack1dbadm08 port-2        : OK
Eth1/16 (adm07)            : rack1dbadm07 port-1         : OK         rack1dbadm07 port-2        : OK
Eth1/17 (adm06)            : rack1dbadm06 port-1         : OK         rack1dbadm06 port-2        : OK
...
Eth1/30 (ISL peer switch)  : rack2sw-roces0 Ethernet1/17 : OK         rack2sw-roces0 Ethernet1/9 : OK
Eth1/31 (ISL peer switch)  : rack2sw-roces0 Ethernet1/13 : OK         rack2sw-roces0 Ethernet1/5 : OK
Eth1/32 (ISL peer switch)  : rack2sw-roces0 Ethernet1/19 : OK         rack2sw-roces0 Ethernet1/11: OK
Eth1/33 (ISL peer switch)  : rack2sw-roces0 Ethernet1/15 : OK         rack2sw-roces0 Ethernet1/7 : OK
  13. Verify the RoCE Network Fabric operation by using the infinicheck command.

    Use the following recommended command sequence. In each command, hosts.lst contains a list of database server RoCE Network Fabric IP addresses (2 RoCE Network Fabric IP addresses for each database server), and cells.lst contains a list of RoCE Network Fabric IP addresses for the storage servers (2 RoCE Network Fabric IP addresses for each storage server).

    • Use infinicheck with the -z option to clear the files that were created during the last run of the infinicheck command. For example: 

      # /opt/oracle.SupportTools/ibdiagtools/infinicheck -g hosts.lst -c cells.lst -z

    • Use infinicheck with the -s option to set up user equivalence for password-less SSH across the RoCE Network Fabric. For example: 

      # /opt/oracle.SupportTools/ibdiagtools/infinicheck -g hosts.lst -c cells.lst -s

    • Finally, verify the RoCE Network Fabric operation by using infinicheck with the -b option, which is recommended on newly imaged machines where it is acceptable to suppress the cellip.ora and cellinit.ora configuration checks. For example: 

      # /opt/oracle.SupportTools/ibdiagtools/infinicheck -g hosts.lst -c cells.lst -b

INFINICHECK                    
        [Network Connectivity, Configuration and Performance]        
               
          ####  FABRIC TYPE TESTS  #### 
System type identified: RoCE
Verifying User Equivalance of user=root from all DBs to all CELLs.
     ####  RoCE CONFIGURATION TESTS  ####       
     Checking for presence of RoCE devices on all DBs and CELLs 
[SUCCESS].... RoCE devices on all DBs and CELLs look good
     Checking for RoCE Policy Routing settings on all DBs and CELLs 
[SUCCESS].... RoCE Policy Routing settings look good
     Checking for RoCE DSCP ToS mapping on all DBs and CELLs 
[SUCCESS].... RoCE DSCP ToS settings look good
     Checking for RoCE PFC settings and DSCP mapping on all DBs and CELLs
[SUCCESS].... RoCE PFC and DSCP settings look good
     Checking for RoCE interface MTU settings. Expected value : 2300
[SUCCESS].... RoCE interface MTU settings look good
     Verifying switch advertised DSCP on all DBs and CELLs ports ( )
[SUCCESS].... Advertised DSCP settings from RoCE switch looks good  
    ####  CONNECTIVITY TESTS  ####
    [COMPUTE NODES -> STORAGE CELLS] 
      (60 seconds approx.)       
    (Will walk through QoS values: 0-6) [SUCCESS]..........Results OK
[SUCCESS]....... All  can talk to all storage cells          
    [COMPUTE NODES -> COMPUTE NODES]               
...
  14. After cabling the racks together, proceed to Configuring the New Hardware to finish the configuration of the new rack.
2.3.3.2 Cabling Several InfiniBand Network Fabric Racks Together

Use this procedure to add another rack to an existing multi-rack system with InfiniBand Network Fabric.

This procedure is for systems with InfiniBand Network Fabric (X8, or earlier).

  1. Set the priority of the current active Subnet Manager Master to 10 on a spine switch.
    1. Log in to any InfiniBand switch on the active system.
    2. Use the getmaster command to determine that the Subnet Manager Master is running on the spine switch.

      The following example shows that the Subnet Manager Master is running on the spine switch dm01sw-ib1. 

      # getmaster
20100701 11:46:38 OpenSM Master on Switch : 0x0021283a8516a0a0 ports 36 Sun DCS 36
QDR switch dm01sw-ib1.example.com enhanced port 0 lid 1 lmc 0
      If the Subnet Manager Master is not running on a spine switch, then perform the following steps:

      1. Use the getmaster command to identify the current location of the Subnet Manager Master.

      2. Log in as the root user on the leaf switch that is the Subnet Manager Master.

      3. Disable Subnet Manager on the switch. The Subnet Manager Master relocates to another switch.

      4. Use the getmaster command to identify the current location of the Subnet Manager Master. If a spine switch is not the Subnet Manager Master, then repeat steps 1.b.ii and 1.b.iii until a spine switch is the Subnet Manager Master.

      5. Enable Subnet Manager on the leaf switches that were disabled during this procedure.

    3. Log in to the Subnet Manager Master spine switch.
    4. Use the disablesm command to stop the Subnet Manager.
    5. Use the setsmpriority 10 command to set the priority to 10.
    6. Use the enablesm command to restart the Subnet Manager.
    7. Repeat step 1.b to ensure that the Subnet Manager Master is running on the spine switch.
  2. Ensure the new rack is near the existing rack.
    The InfiniBand cables must be able to reach the servers in each rack.
  3. Completely shut down the new rack (Rn+1).
  4. Cable the leaf switch in the new rack according to the appropriate table in Multi-Rack Cabling Tables for Oracle Exadata Rack Models with InfiniBand Network Fabric (X2 to X8).

    For example, if rack Rn+1 is R4, then use Table 6-9.

    Caution:

    Cabling within a live network must be done carefully in order to avoid potentially serious disruptions.

    The cabling table that you use for your new InfiniBand topology tells you how to connect ports on the leaf switches to ports on spine switches in order to connect the racks. Some of these ports on the spine switches might be already in use to support the existing InfiniBand topology. In these cases, connect only the cable on the leaf switch in the new rack and stop there for now. Make note of which cables you were not able to terminate.

    Do not unplug any cables on the spine switch in the existing rack at this point. Step 5 describes how to re-cable the leaf switches on the existing racks (one leaf switch after the other - while the leaf switch being re-cabled will be powered off), which will free up these currently in-use ports. At that point, you can connect the other end of the cable from the leaf switch in the new rack to the spine switch in the existing rack as indicated in the table.

  5. Complete the following procedure for each of the original racks:
    In these steps, Rx represents a rack number from R1 to Rn.
    1. Power off leaf switch Rx IB2.
      This causes all servers in the rack to fail over their InfiniBand traffic to Rx IB3.
    2. Cable leaf switch Rx IB2 according to Multi-Rack Cabling Tables for Oracle Exadata Rack Models with InfiniBand Network Fabric (X2 to X8).
    3. Power on leaf switch Rx IB2.
    4. Wait at least three minutes for Rx IB2 to become completely operational.

      To check the switch, log in to the switch and run the ibswitches command. The output should show n*3 switches for IB1, IB2, and IB3 in racks R1, R2, ... Rn.

    5. Power off leaf switch Rx IB3.
      This causes all servers in the rack to fail over their InfiniBand traffic to Rx IB2.
    6. Cable leaf switch Rx IB3 according to Multi-Rack Cabling Tables for Oracle Exadata Rack Models with InfiniBand Network Fabric (X2 to X8).
    7. Power on leaf switch Rx IB3.
    8. Wait at least three minutes for Rx IB3 to become completely operational.

      To check the switch, log in to the switch and run the ibswitches command. The output should show n*3 switches for IB1, IB2, and IB3 in racks R1, R2, ... Rn.

  6. Power on all the InfiniBand switches in the new rack.
  7. Wait three minutes for the switches to become completely operational.

    To check the switch, log in to the switch and run the ibswitches command. The output should show (n+1)*3 switches for IB1, IB2, and IB3 in racks R1, R2, ... Rn+1.

  8. Ensure that the Subnet Manager Master is running on R1 IB1 by running the getmaster command from any switch.
  9. Power on all servers in the new rack (Rn+1).
  10. Log in to spine switch R1 IB1, and lower its priority to 8.
    1. Use the disablesm command to stop Subnet Manager.
    2. Use the setsmpriority 8 command to set the priority to 8.
    3. Use the enablesm command to restart Subnet Manager.
  11. Ensure that the Subnet Manager Master is running on one of the spine switches using the getmaster command from any switch.
  12. Ensure that the Subnet Manager is running on every spine switch by entering the following command from any switch:
    ibdiagnet -r

    Each spine switch should show as running in the Summary Fabric SM-state-priority section of the output. If a spine switch is not running, then log in to the switch and enable the Subnet Manager using the enablesm command.

  13. If there are now four or more racks, then log in to the leaf switches in each rack and disable Subnet Manager using the disablesm command.