3.12 NVMe device hotplug unplug procedure change

Since UEK R5 adds support for NVMe subsystems and multipathing, enumerated device names generated by the kernel are not stable. This means that the procedure for identifying and unplugging NVMe devices using hotplug functionality is slightly different to the procedure that you may have followed using other kernel releases. This note describes the steps that you should take to identify, power down and unplug the appropriate device.

  1. Use the lsblk command to identify the disk that you wish to remove according to its WWN or UUID. For example:

    # lsblk -o +UUID,WWN,MODEL

    Take note of the enumerated kernel instance name that has been assigned to the device. For example, this may be nvme0n1. It is very important to understand that the device name does not necessarily map onto the controller or PCIe bridge that it is attached to. See Section 3.11, “NVMe device names change across reboots” for more information.

  2. Search for the device path to obtain the PCI domain identifier for the device:

    # find /sys/devices -iname nvme0n1
    /sys/devices/pci0000:85/0000:85:01.0/0000:8d:00.0/nvme/nvme1/nvme0n1

    Note that 0000:8d:00.0 in the returned path for the device is the PCI domain identifier for the device. You need this information to proceed.

  3. Obtain the physical slot number for the NVMe drive. Under UEK R5, the slot is bound to the NVMe device directly and not to the PCIe controller. You can find the slot number for the NVMe device by running the lspci command and querying the PCI domain identifier for the device in verbose mode:

    # lspci -s 0000:8d:00.0 -vvv
    8d:00.0 Non-Volatile memory controller: Intel Corporation Express Flash NVMe
    P4500 (prog-if 02 [NVM Express])
            Subsystem: Oracle/SUN Device 4871
            Physical Slot: 104-1
    … 

    Note that the Physical Slot number for the device in this example is 104-1. Take note of this value to proceed.

  4. Use the Physical Slot number for the device to find its bus interface:

    # find /sys -iname "104-1"
    /sys/bus/pci/slots/104-1
  5. Use the returned bus interface path to power off the NVMe drive:

    # echo 0 > /sys/bus/pci/slots/104-1/power

    Depending on your hardware, the blue disk LED may display on the front panel of the system may display to indicate that you can safely remove the disk drive.