Verify NVMe Cable Connections

Language:

When NVMe cables are removed or replaced between the storage drive backplane and NVMe switch cards, you must perform the procedure in this section to confirm that all NVMe cable connections are correct. If NVMe cable connections are not correct, the storage server operating system should not be allowed to boot, as it could cause a problem with disk drive mapping.

Before beginning this procedure, ensure that the storage server has been returned to operation. See Step 4 in Install NVMe Cables. This procedure requires access to the Oracle ILOM service processor (SP). The BIOS Setup Utility is used to halt the system so NVMe cable connection tests can be run for NVMe storage drives.

You can access the BIOS Setup Utility screens from the following interfaces:

Use a USB keyboard and VGA monitor connected directly to the server. (A mouse is not required to access the BIOS Setup Utility.)
Use a terminal (or terminal emulator connected to a computer) through the serial port on the back panel of the server.
Connect to the server using the Oracle ILOM Remote System Console Plus application.

Power on the server.
For example, to power on the server:
- From the local server, press the Power button on the front panel of the server to power on the server.
- From the Oracle ILOM web interface, click Host Management → Power Control and select Power On from the Select Action list.
- From the Oracle ILOM CLI, type start /System
The power-on self-test (POST) sequence begins.

To enter the BIOS Setup Utility, press the F2 key (Ctrl+E from a serial connection) when prompted and while the BIOS is running the power-on self-tests (POST).
This step halts the system so NVMe cable connection tests can be run for the NVMe storage drives.

Open a terminal window. Using SSH (secure shell), log in to the Oracle ILOM command-line interface (CLI) on the server SP.

To access the Oracle ILOM Restricted Shell, type the set SESSION mode=restricted command, and then click Enter.

Type y to confirm that you want to start the Oracle ILOM Restricted Shell.
The # prompt appears.

To run the NVMe cable connection tests, type the hwdiag io nvme_test command, and then click Enter.

Your output should be similar to the following example:

HWdiag - Build Number 107051 (Jan 24 2016, 11:40:48)
         Current Date/Time: Mar 08 2016, 11:44:28
    Checking NVME drive fru contents...
        checking fru on drive NVMe 0              OK
        checking fru on drive NVMe 1              OK
        checking fru on drive NVMe 3              OK
        checking fru on drive NVMe 4              OK
        checking fru on drive NVMe 6              OK
        checking fru on drive NVMe 7              OK
        checking fru on drive NVMe 9              OK
        checking fru on drive NVMe 10             OK
    NVME drives fru check:                        PASSED

    Checking NVME drive pcie links...
        checking pcie link on drive NVMe 0        OK
        checking pcie link on drive NVMe 1        OK
        checking pcie link on drive NVMe 3        OK
        checking pcie link on drive NVMe 4        OK
        checking pcie link on drive NVMe 6        OK
        checking pcie link on drive NVMe 7        OK
        checking pcie link on drive NVMe 9        OK
        checking pcie link on drive NVMe 10       OK
    NVME drives pcie link check:                  PASSED

    Checking NVME drive DSN...
        checking DSN on drive NVMe 0              OK
        checking DSN on drive NVMe 1              OK
        checking DSN on drive NVMe 3              OK
        checking DSN on drive NVMe 4              OK
        checking DSN on drive NVMe 6              OK
        checking DSN on drive NVMe 7              OK
        checking DSN on drive NVMe 9              OK
        checking DSN on drive NVMe 10             OK
    NVME drives DSN check:                        PASSED 

    Checking NVME cabling...
        Cables associated with Switch Card in PCIe Slot 6 verified
        Cables associated with Switch Card in PCIe Slot 5 verified
        Cables associated with Switch Card in PCIe Slot 2 verified
        Cables associated with Switch Card in PCIe Slot 1 verified
    NVME cable check:                             PASSED

NVME test PASSED

If any of the tests fail, there could be a problem with the cabling. For example:

If a cable is crossed (plugged into the incorrect port), then the NVMe drive DSN (Drive Serial Number) test should report two drives where the DSN does not match. The cabling for those drives should be inspected to ensure that it is correct.
If multiple NVMe drive PCIe links are not active, or multiple NVMe drive DSNs are not OK, then check the Port 2 connection on the NVMe switch card. If Port 2 is incorrectly cabled, it could affect all three drives that are connected to that NVMe switch card.

Once issues are resolved, power cycle the system by removing the power cords from the power supplies, waiting 30 seconds, and then reattaching the power cords.

Retest the system by repeating Step 1 through Step 6.
Once all NVMe cable connection tests have passed, the storage server operating system can be booted.