Prepare an NVMe Storage Drive for Removal

Before You Begin

• Linux NVMe hot plug requires the kernel boot argument "pci=pcie_bus_perf" be set in order to get proper MPS (MaxPayloadSize) and MRR (MaxReadRequest). Fatal errors will occur without this argument.

• For a list of the virtual PCIe slots of NVMe drives as seen by the operating system, see NVMe Storage Drive Virtual PCIe Slot Designation. Note that the virtual PCIe slot name is not the same as the name on the server front panel label.

1. Log in to Oracle Linux that is running on the server.
2. Obtain information about available NVMe storage devices.
   1. Obtain the PCIe addresses (Bus Device Function) of enabled NVMe drives. Type:

      # find /sys/devices |egrep 'nvme[0-9][0-9]?$'

      This commands returns output similar to the following:

      /sys/devices/pci0000:00/0000:00:02.2/0000:10:00.0/0000:11:04.0/0000:12:00.0/misc/nvme0
      /sys/devices/pci0000:00/0000:00:02.2/0000:10:00.0/0000:11:05.0/0000:13:00.0/misc/nvme1
      /sys/devices/pci0000:00/0000:00:02.2/0000:10:00.0/0000:11:06.0/0000:14:00.0/misc/nvme2
      /sys/devices/pci0000:00/0000:00:02.2/0000:10:00.0/0000:11:07.0/0000:15:00.0/misc/nvme3

      For example, 0000:12:00.0 matches the PCIe address of the drive labeled NVMe0 on the system front panel.

   2. Obtain the PCIe virtual slot number (APIC ID). Type:

      # egrep -H '.*' /sys/bus/pci/slots/*/address

      This commands returns output similar to the following:

      /sys/bus/pci/slots/100/address:0000:12:00
      /sys/bus/pci/slots/101/address:0000:13:00
      /sys/bus/pci/slots/102/address:0000:14:00
      /sys/bus/pci/slots/103/address:0000:15:00

      For example, the PCIe address 0000:12:00.0 matches the PCIe slot number (100) for the drive labeled NVMe0 on the system front panel.

3. Remove the NVMe storage device path.
   1. Use the umount command to unmount any file systems that are mounted on the device.

      In Linux, NVMe drives do not use the standard block device labeling, such as /dev/sd*. For example, NVMe drive 0 that has a single namespace block device would be /dev/nvme0n1. If you formatted and partitioned that namespace with a single partition, that would be /dev/nvme0n1p1.

   2. Remove the device from any multiple device (md) and Logical Volume Manager (LVM) volume using it.

      If the device is a member of an LVM Volume group, then it may be necessary to move data off the device using the pvmove command, then use the vgreduce command to remove the physical volume, and (optionally) pvremove to remove the LVM meta data from the disk.

   3. If the device uses multipathing, run multipath -l and note all the paths to the device. Then, remove the multipathed device using the multipath -f device command.
   4. Run the blockdev --flushbufs device command to flush any outstanding I/O on all paths to the device.
4. Prepare the NVMe drive for removal by powering off the NVMe drive slot. Type:

   # echo 0 > /sys/bus/pci/slots/slot_number/power

   Where slot_number is the PCIe slot number (e.g., 100, which represents the drive labeled NVMe0 on the system front panel).

5. Verify that the blue OK to Remove indicator on the NVMe drive is lit.