1. Oracle Server X9-2L Service Manual
  2. Servicing Storage Drives (CRU)

4 Servicing Storage Drives (CRU)

This section describes how to service storage drives. Storage drives are customer-replaceable units (CRUs) that do not require you to power off the server. For more information about CRUs, see Illustrated Parts Breakdown and Customer-Replaceable Units.

Servicing SAS Cables (FRU)
Servicing NVMe Cables (FRU)

Storage Drives Hot-Plug Conditions

The hard-disk drives (HDDs), solid-state drives (SSDs), or NVMe SSD drives that are installed in the server are in most cases hot-pluggable. The hot-plug capability depends on how the drives are configured and whether the drive is an NVMe device. To hot-plug a drive you must take the drive offline before you can remove it. When you take the drive offline, you prevent any application from accessing the drive and remove the logical software links to the drive. For an NVMe storage drive, you must not only take the drive offline, but also power down the drive slot.

The following conditions inhibit the ability to perform hot-plugging of a drive:

  • The drive provides the operating system, and the operating system is not mirrored on another drive.

  • The drive cannot be logically isolated from the online operations of the server.

  • The operating system does not support hot plug for the drive.

If any of these disk drive conditions are true, then you must shut down the system before you replace the drive. See Powering Down the Server.

Note:

Replacing a drive does not require removing the server from a rack.

Storage Drive Failure and RAID

A single storage drive failure does not cause a data failure if you configured the storage drives as a mirrored RAID 1 volume (optional). The storage drive can be removed, and when a new storage drive is inserted, the contents are automatically rebuilt from the rest of the array with no need to reconfigure the RAID parameters. If you configured the replaced storage drive as a hot-spare, the new HDD is automatically configured as a new hot-spare.

For instructions for configuring RAID on the server, refer to Configuring Storage Devices for Operating System Installation in Oracle Servers X9-2 and X9-2L Installation Guide at Oracle Server X9-2L Documentation.

Storage Drive Locations and Numbering

The following illustration shows the location of the drives and the internal system software designations for the drives. For information on storage drive status indicators (LEDs), see Storage Drive Status Indicators.


Figure showing the location and numbering of drives on the server.
Callout Description

1

Storage drive 0 HDD/NVMe

2

Storage drive 4 HDD/NVMe

3

Storage drive 8 HDD/NVMe

4

Storage drive 1 HDD/NVMe

5

Storage drive 5 HDD/NVMe

6

Storage drive 9 HDD/NVMe

7

Storage drive 2 HDD/NVMe

8

Storage drive 6 HDD/NVMe

9

Storage drive 10 HDD/NVMe

10

Storage drive 3 HDD/NVMe

11

Storage drive 7 HDD/NVMe

12

Storage drive 11 HDD/NVMe

Note:

The NVMe storage drives are labeled NVMe0 through NVMe11; however, operating systems installed on the server assign different names to the NVMe storage drives. For the corresponding names assigned to the NVMe storage drives by server operating systems, see Server Operating System Names for the NVMe Storage Drives.

Removing and Replacing a Storage Drive

The following sections describe how to remove and replace an HDD or SSD storage drive.

Remove a Storage Drive

  1. Prepare the system for the drive removal.
  2. Identify the location of the drive that you want to remove.

    For storage drive locations, see Storage Drive Locations and Numbering.

  3. Remove the storage drive.
  4. Push the latch release button to open the drive latch [1, 2]. latch release button, storage drive storage drives latch release button

    Figure showing the location of the storage drive release button and latch.
    Callout Description

    1

    Pressing the latch release button.

    2

    Opening the latch.

    Caution:

    The latch is not an ejector. Do not open the latch too far to the right. Doing so can damage the latch.
  5. Grasp the latch and pull the drive out of the drive slot.

    Figure showing a storage drive being removed from the server.
  6. Consider your next steps:

    • If you are replacing the drive, continue to Install a Storage Drive.

    • If you are not replacing the drive, install a filler panel in the empty drive slot to maintain proper airflow and perform administrative tasks to configure the server to operate without the drive.

      For information on how to install a storage drive filler panel, see Remove and Install Filler Panels.

Install a Storage Drive

  1. Remove the replacement drive from its packaging, and place the drive on an antistatic mat.
  2. If necessary, remove the drive filler panel.
  3. Align the replacement drive with the drive slot.

    The drive is physically addressed according to the slot in which it is installed. It is important to install a replacement drive in the same slot as the drive that you removed.

  4. Slide the drive into the slot until the drive is fully seated.

    Figure showing a storage drive being installed in the server.
  5. Close the drive latch to lock the drive in place.
  6. Perform administrative procedures to reconfigure the drive.

    The procedures that you perform at this point depend on how your data is configured. You might need to partition the drive, create file systems, load data from backups, or have the drive updated from a RAID configuration.

Removing and Replacing an NVMe Storage Drive Using Oracle Solaris

The following sections describe how to remove and replace an NVMe storage drive on a server that is running the Oracle Solaris operating system.

Unmount an NVMe Storage Drive

  1. Log in to Oracle Solaris that is running on the server.
  2. To find the NVMe drive slot number, type:

    # hotplug list -lc

    This command produces output similar to the following for each of the NVMe drives installed in the server:

    hotplug list -lc
Connection           State           Description Path
________________________________________________________________________________
Slot0                EMPTY           PCIe-Native /pci@0,0/pci8086,a190@1c
/SYS/DBP/HDD1/NVME   EMPTY           PCIe-Native /pci@4,0/pci8086,347a@2
/SYS/DBP/HDD5/NVME   EMPTY           PCIe-Native /pci@4,0/pci8086,347b@3
/SYS/DBP/HDD0/NVME   ENABLED         PCIe-Native /pci@4,0/pci8086,347c@4
/SYS/DBP/HDD4/NVME   EMPTY           PCIe-Native /pci@4,0/pci8086,347d@5
/SYS/DBP/HDD3/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347a@2
/SYS/DBP/HDD7/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347b@3
/SYS/DBP/HDD2/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347c@4
/SYS/DBP/HDD6/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347d@5
  3. To prepare the NVMe drive for removal, that is, to detach the device driver and power off the drive slot, type the following commands:

    # hotplug poweroff /SYS/DBP/HDD0/NVME

    # hotplug list -lc

    The following output appears for the NVMe drive that has been unmounted:

    # 
# hotplug list -lc
Connection           State           Description Path
________________________________________________________________________________
Slot0                EMPTY           PCIe-Native /pci@0,0/pci8086,a190@1c
/SYS/DBP/HDD1/NVME   EMPTY           PCIe-Native /pci@4,0/pci8086,347a@2
/SYS/DBP/HDD5/NVME   EMPTY           PCIe-Native /pci@4,0/pci8086,347b@3
/SYS/DBP/HDD0/NVME   PRESENT         PCIe-Native /pci@4,0/pci8086,347c@4
/SYS/DBP/HDD4/NVME   EMPTY           PCIe-Native /pci@4,0/pci8086,347d@5
/SYS/DBP/HDD3/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347a@2
/SYS/DBP/HDD7/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347b@3
/SYS/DBP/HDD2/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347c@4
/SYS/DBP/HDD6/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347d@5
  4. Verify that the OK to Remove indicator (LED) on the NVMe storage drive is lit.

Remove an NVMe Storage Drive

Perform this procedure to physically remove an NVMe storage drive from the server.

  1. Identify the physical location of the NVMe drive that you want to remove.

    For storage drive locations, see Remove a Storage Drive.

  2. Verify that the OK to Remove indicator (LED) on the NVMe storage drive is lit.
  3. On the drive that you plan to remove, push the latch release button to open the drive latch.
  4. Grasp the latch and pull the drive out of the drive slot.
  5. Consider your next steps:

Verify Removal of an NVMe Storage Drive

  1. To verify that the NVMe drive has been removed, type:

    # hotplug list -lc

    The following output appears for the NVMe drive that you removed:

    # hotplug list -lc
Connection           State           Description Path
________________________________________________________________________________
Slot0                EMPTY           PCIe-Native /pci@0,0/pci8086,a190@1c
/SYS/DBP/HDD1/NVME   EMPTY           PCIe-Native /pci@4,0/pci8086,347a@2
/SYS/DBP/HDD5/NVME   EMPTY           PCIe-Native /pci@4,0/pci8086,347b@3
/SYS/DBP/HDD0/NVME   EMPTY           PCIe-Native /pci@4,0/pci8086,347c@4
/SYS/DBP/HDD4/NVME   EMPTY           PCIe-Native /pci@4,0/pci8086,347d@5
/SYS/DBP/HDD3/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347a@2
/SYS/DBP/HDD7/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347b@3
/SYS/DBP/HDD2/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347c@4
/SYS/DBP/HDD6/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347d@5

Install an NVMe Storage Drive

Perform this procedure to physically install an NVMe storage drive into the server.

Note:

After you physically remove an NVMe storage drive from the server, wait at least 10 seconds before installing a replacement drive.
  1. Remove the replacement drive from its packaging and place the drive on an antistatic mat.
  2. If necessary, remove the drive filler panel.
  3. Align the replacement drive with the drive slot.

    The drive is physically addressed according to the slot in which it is installed. It is important to install a replacement drive in the same slot as the drive that you removed.

  4. Slide the drive into the slot until the drive is fully seated.
  5. Close the drive latch to lock the drive in place.

Power On an NVMe Storage Drive and Attach a Device Driver

  1. Type: # hotplug enable /SYS/DBP/HDD0/NVME
  2. Type: # hotplug list -lc

    The following status is displayed for the NVMe storage drive.

    # hotplug enable /SYS/DBP/HDD0/NVME
hotplug list -lc
Connection           State           Description Path
________________________________________________________________________________
Slot0                EMPTY           PCIe-Native /pci@0,0/pci8086,a190@1c
/SYS/DBP/HDD1/NVME   EMPTY           PCIe-Native /pci@4,0/pci8086,347a@2
/SYS/DBP/HDD5/NVME   EMPTY           PCIe-Native /pci@4,0/pci8086,347b@3
/SYS/DBP/HDD0/NVME   ENABLED         PCIe-Native /pci@4,0/pci8086,347c@4
/SYS/DBP/HDD4/NVME   EMPTY           PCIe-Native /pci@4,0/pci8086,347d@5
/SYS/DBP/HDD3/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347a@2
/SYS/DBP/HDD7/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347b@3
/SYS/DBP/HDD2/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347c@4
/SYS/DBP/HDD6/NVME   EMPTY           PCIe-Native /pci@b,0/pci8086,347d@5
  3. To check the NVMe drive health, firmware level, temperature, get error log, SMART data, security erase, low level format, and other data, type:

    # nvmeadm --help

Removing and Replacing an NVMe Storage Drive Using Oracle Linux

The following sections describe how to remove and replace an NVMe storage drive on a server that is running the Oracle Linux operating system.

Unmount an NVMe Storage Drive

  1. Log in to Oracle Linux that is running on the server.
  2. Remove the NVMe storage device path.
    1. To find the PCIe addresses (Bus Device Function), type:

      # find /sys/devices |egrep ‘nvme[0-9][0-9]?$’

      This command returns output similar to the following example, with the PCIe addresses in bold text:

      /sys/devices/pci0000:80/0000:80:03.0/0000:b0:00.0/0000:b1:04.0/0000:b2:00.0/misc/nvme0
/sys/devices/pci0000:80/0000:80:03.0/0000:b0:00.0/0000:b1:05.0/0000:b4:00.0/misc/nvme1
    2. To obtain the slot number (APIC ID) for the bus address, type the following command to list the PCIe slot numbers with corresponding bus addresses:

      # egrep -H ‘.*’ /sys/bus/pci/slots/*/address

      This commands returns output similar to the following example, with the bus addresses for the corresponding NVMe instances in bold text.:

      Note:

      In the following output, notice that the instance names for the NVMe drives do not correspond to the NVMe drive labels on the front of the server. That is, pci/slots/12/address: 0000:b2:00 corresponds to instance nvme0; however, on the front of the server, this drive is labeled NVMe2. For a table that shows the relationship between the pci/slot# and the NVMe storage drive label on the front of the server, see Server Operating System Names for the NVMe Storage Drives. 
          /sys/bus/pci/slots/10/address:0000:b8:00
    /sys/bus/pci/slots/11/address:0000:b6:00
    /sys/bus/pci/slots/12/address:0000:b2:00 (instance nvme0, pcie slot 12, drive label nvme2)
    /sys/bus/pci/slots/13/address:0000:b4:00 (instance nvme1, pcie slot 13, drive label nvme3)
    3. Disconnect all users from the NVMe drive and back up the NVMe drive data, as needed.
      1. Use the umount command to unmount any file systems that are mounted on the device.
      2. Remove the device from any multiple device (md) and Logical Volume Manager (LVM) volume using the device.
      3. If the device uses multipathing, run multipath -l and note all the paths to the device. Then, remove the multipathed device using the multipath -f device command.
      4. Run the blockdev --flushbufs device command to flush any outstanding I/O to all paths to the device.
  3. To prepare the NVMe drive for removal, that is, to detach the NVMe device driver and power off the NVMe drive slot, type:

    # echo 0 > /sys/bus/pci/slots/$slot/power

    Where $slot is the slot number obtained in Step 2.b above.

  4. Verify that the OK to Remove indicator (LED) on the NVMe drive is lit.

Remove an NVMe Storage Drive

Perform this procedure to physically remove an NVMe storage drive from the server.

  1. Identify the location of the NVMe drive that you want to remove.

    For storage drive locations, see Remove a Storage Drive.

  2. Verify that the OK to Remove indicator (LED) on the NVMe drive is lit.
  3. On the NVMe drive you plan to remove, push the latch release button to open the drive latch.
  4. Grasp the latch and pull the drive out of the drive slot.
  5. Consider your next steps:

Verify Removal of an NVMe Storage Drive

  1. To check the NVMe drive enumeration to verify that NVMe drive has been removed, type:

    # lspci -nnd :0a54

  2. View the command output and verify that the entry for the slot number that was disabled no longer appears.

    This command returns output similar to the following:

    86:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:0a54]
8d:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:0a54]
d9:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:0a54]
e0:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:0a54]

Install an NVMe Storage Drive

Perform this procedure to physically install an NVMe storage drive into the server.

Note:

After you physically remove an NVMe drive from the server, wait at least 10 seconds before installing a replacement drive.
  1. Remove the replacement drive from its packaging and place the drive on an antistatic mat.
  2. If necessary, remove the drive filler panel.
  3. Align the replacement drive with the drive slot.

    The drive is physically addressed according to the slot in which it is installed. It is important to install a replacement drive in the same slot as the drive that was removed.

  4. Slide the drive into the slot until the drive is fully seated.
  5. Close the drive latch to lock the drive in place.

Power On an NVMe Storage Drive and Attach a Device Driver

  1. To power on the slot and attach the device driver, type:

    # echo 1 > /sys/bus/pci/slots/$slot/power

    Where $slot is the slot number for the NVMe storage drive.

Verify Operation of an NVMe Storage Drive

  1. To verify that an NVMe drive is operating properly, do one of the following:

    • Check the /var/log/messages log file.

    • Type: ls -l /dev/nvme*

Removing and Replacing an NVMe Storage Drive Using Microsoft Windows Server

The following sections describe how to remove and replace an NVMe storage drive on a server that is running the Microsoft Windows Server operating system.

Note:

NVMe storage drive hot plug is not supported for an Oracle Server X9-2L running Microsoft Windows Server. The system must be powered down before removing and replacing an NVMe storage drive.

Remove an NVMe Storage Drive

Perform this procedure to physically remove an NVMe storage drive from the server.

  1. Power down the server that contains the storage drive to be removed.
  2. On the NVMe drive you plan to remove, push the latch release button to open the drive latch.
  3. Grasp the latch and pull the drive out of the drive slot.
  4. Consider your next steps:

    • If you are replacing the drive, continue to Install an NVMe Storage Drive.

    • If you are not replacing the drive, install a filler panel in the empty drive slot to maintain proper airflow and perform administrative tasks to configure the server to operate without the drive.

      For information on how to install a storage drive filler panel, see Remove and Install Filler Panels.

Install an NVMe Storage Drive

Perform this procedure to physically install an NVMe storage drive into the server.

  1. Remove the replacement drive from its packaging and place the drive on an antistatic mat.
  2. If necessary, remove the drive filler panel.
  3. Align the replacement drive with the drive slot.

    The drive is physically addressed according to the slot in which it is installed. It is important to install a replacement drive in the same slot as the drive that was removed.

  4. Slide the drive into the slot until the drive is fully seated.
  5. Close the drive latch to lock the drive in place.
  6. Power on the server.