5 Securely Erasing Oracle Exadata Database Machine

If you are repurposing or removing an Oracle Exadata Database Machine from your environment, it is critical to securely erase all the information on the servers.

5.1 Overview of Secure Eraser

Oracle Exadata System Software release 12.2.1.1.0 or later provides a secure erasure solution, called Secure Eraser, for every component within Oracle Exadata Database Machine.

Oracle Exadata Database Machine consists of the following components:

  • Oracle Exadata Database Servers

  • Oracle Exadata Storage Servers

  • InfiniBand switches

  • Ethernet switches

  • Power distribution units

Secure Eraser is a comprehensive solution that covers all Oracle Exadata Database Machines V2 or higher, including both 2-socket and 8-socket servers. The solution securely erases all data on both database servers and storage servers, and resets InfiniBand switches, Ethernet switches, and power distribution units back to factory default.

To achieve the best possible performance, secure erasure is performed in parallel at every layer on an Oracle Exadata Database Machine. Oracle Exadata Database Servers and Oracle Exadata Storage Servers are securely erased in parallel. Within a server, all device types (such as hard drives, flash devices, and internal USBs) are securely erased in parallel. For each device type, all devices are further securely erased in parallel.

Secure Eraser automatically detects the hardware capability of a storage device and picks the best erasure method supported by the device. Crypto erase is used whenever possible to provide better security and faster speed.

Secure Eraser comes with flexible options. You can choose to initiate a secure erasure either through PXE or through an external USB. The entire process can be completely automated without any user intervention. Or, you can choose to do it interactively and choose to erase specific types of storage devices.

Secure Eraser periodically generates a progress report every 10 seconds so that you can easily monitor the progress.

When the secure erasure is completed, a certificate is generated for each server with a list of devices that have been securely erased. The following figure shows a sample certificate from Secure Eraser.

Figure 5-1 Sample Certificate from Secure Eraser

Description of Figure 5-1 follows
Description of "Figure 5-1 Sample Certificate from Secure Eraser"

5.2 Securely Erasing Database Servers and Storage Servers

Oracle Exadata System Software 12.2.1.1.0 or later comes with a utility called Secure Eraser which securely erases data on hard drives, flash devices, and internal USBs, and resets ILOM to factory default.

In earlier versions of Exadata, you can securely erase user data through CellCLI commands such as DROP CELL ERASE, DROP CELLDISK ERASE, or DROP GRIDDISK ERASE. These DROP commands only cover user data on hard drives and flash devices. Secure Eraser, on the other hand, sanitizes all content, not only user data but also operating system, Oracle Exadata System Software, and user configurations. In addition, it covers a wider range of hardware components including hard drives, flash devices, internal USBs, and ILOMs.

Caution:

The server will become unbootable after the system devices are securely erased, and ILOM will no longer be remotely accessible after being reset to factory default. ILOM will remain accessible through serial console.

The Secure Eraser utility works on both database servers and storage servers and covers all Oracle Exadata Database Machines V2 or higher.

Based on hardware capabilities, different secure erasure methods are applied. In general, Secure Eraser has two types of erasure methods: 3-pass erase and crypto erase. The 3-pass erase method overwrites all addressable locations with a character, its complement, then a random character, and finally verifies the results. The crypto erase method erases all user data present on instant secure erase (ISE) devices by deleting the encryption keys with which the user data was previously encrypted.

The following table gives a summary of the secure erasure methods used and their approximate time. Note that the time for 3-pass erase varies from drives to drives based on their size and speed. It is approximately equal to the time required to overwrite the entire device three times and read it one more time. Hard drives, flash devices, and internal USBs are securely erased in parallel: the time required to erase one device is the same as that required for erasing multiple devices of the same kind.

Table 5-1 Time Required to Securely Erase Various Devices

Component Make Model Erasure Method Approximate Time

Hard drive

  • 8 TB hard drives on Oracle Exadata Database Machine X5

  • All hard drives on Oracle Exadata Database Machine X6 or later

Crypto erase

1 minute

Hard drive

All other hard drives

3-pass erase

600 GB: 7 hours

1.2 TB: 12 hours

2 TB: 35 hours

3 TB: 49 hours

4 TB: 56 hours

10 TB: 150 hours

Flash device

Flash devices on Oracle Exadata Database Machine X3 or later

Crypto erase

1 minute

Flash device

All other flash devices

3-pass erase

22.875 GB: 20 minutes

Internal USB

All platforms

3-pass erase

4 GB: 30 minutes

8 GB: 1 hour

M.2

Oracle Exadata Database Machine X7-2 or later

Crypto erase

1 minute

ILOM

All platforms

Factory reset

1 minute

5.3 Automatic Secure Eraser through PXE Boot

In this procedure, you configure Secure Eraser to run automatically when you reboot the nodes.

Before you begin:

  • Review My Oracle Support note 2180963.1 for the latest Secure Eraser information and to download the Secure Eraser package.

  • Make sure you have access to a Preboot Execution Environment (PXE) server where the nodes to be erased can boot from.

  • Make sure you have access to a NFS server that is accessible from all the nodes to be erased.

  • Make sure you have access to one of the nodes to be erased.

  1. Copy the PXE image files initrd (initrd-<version>) and kernel (vmlinux-<version>) from the Secure Eraser package to the /tftpboot directory on the PXE server.

  2. Create a file containing the names of the database servers and storage servers you want to erase.

    To generate this file, you can run the following command from one of the nodes to be erased, and verify the nodes in the files are the ones to be erased.

    # ibhosts | awk '/S [0-9\.\,]*/ || /C [0-9\.\,]*/ {print $6}' | 
    sed "s/\"//g" > nodes_to_be_erased
    

    If you only want to erase one server, enter the name of the server into the nodes_to_be_erased file, for example Exa01celadm04.

  3. Copy the dcli utility from the Secure Eraser package and the nodes_to_be_erased file generated in step 2 to the PXE server.

  4. Create a PXE configuration template called pxe_cfg.template to contain the following lines:

    • For all models before X7 — grub:

      default linux
      label linux
      kernel vmlinux-nfs-12.2.1.1.0-161015-cell
      append initrd=initrd-nfs-12.2.1.1.0-161015-cell.img dhcp pxe quiet loglevel=
      0 secureeraser bootarea=diagnostics console=tty1 console=ttyS0,115200n8 logp
      ath=10.133.42.221:/export/exadata_secure_eraser_certificate_dir
      
      • The first line (default) indicates that the default label to use is called linux.

      • The second line (label) defines the linux label.

      • The third line (kernel) identifies the kernel file to load. In this case it is the file copied over in step 1.

      • The fourth line (append) adds more options to the kernel command line. The append statement must be on a single line in the configuration file.

        • The initrd option specifies the initrd file to load. In this case it is the initrd file copied over in step 1.

        • The dhcp option specifies to use DHCP to discover the eth0 interface.

        • The pxe option suppresses search for the image on virtual CD and USB devices.

        • The quiet option disables excessive kernel log messages.

        • The loglevel=0 option suppresses non-critical kernel messages.

        • The secureeraser option indicates PXE boot will automatically trigger the Secure Eraser utility to sanitize all media including hard drives, flash devices, internal USBs, and ILOM on the node.

        • The bootarea option indicates that the boot mode is diagnostic and not imaging install or rescue.

        • The console options indicate standard output and standard error messages are printed to both ILOM web console and serial console.

        • The logpath option specifies the NFS share directory where Secure Eraser will save the certificate.

    • For Oracle Exadata Database Machine X7-2 and newer models —grub2 / Secure Boot:

      Note:

      In the following example, the following parameters must be updated to match your environment:
      • kernel (the vmlinux file)

      • initrd (the initrd*.img file)

      • logpath

      set default 0
      set timeout=10
      menuentry 'ExadataLinux' {
        echo "Loading efi/vmlinuz"   
       linuxefi efi/vmlinux-nfs-18.1.0.0.0-170915.1 dhcp pxe boot-from=uefi 
      quiet loglevel=0 secureeraser bootarea=diagnostics console=ttyS0,115200n8 
      logpath=10.133.42.221:/export/exadata_secure_eraser_certificate_dir   
        echo "Loading efi/initrd.img"   
        initrdefi efi/initrd-nfs-18.1.0.0.0-170915.1.img 
        echo "Booting installation kernel"
      }
      
      • The first line (default) identifies a menu entry that should be selected by default, after the timeout value specified by the second line.

      • The third line (menuentry) represents the Linux kernel that will be used in the Secure Eraser environment.

      • The fifth line (linuxefi) indicates the kernel is on an UEFI-based system. The linuxefi statement must be on a single line in the configuration file.

        • The dhcp option specifies to use DHCP to discover the eth0 interface.

        • The pxe option suppresses search for the image on virtual CD and USB devices.

        • The boot-from=uefi option indicates the system is booting from UEFI.

        • The quiet option disables excessive kernel log messages.

        • The loglevel=0 option suppresses non-critical kernel messages.

        • The secureeraser option indicates PXE boot will automatically trigger the Secure Eraser utility to sanitize all media including hard drives, flash devices, internal USBs, and ILOM on the node.

        • The bootarea option indicates that the boot mode is diagnostic and not imaging install or rescue.

        • The console options indicate standard output and standard error messages are printed to both ILOM web console and serial console.

        • The logpath option specifies the NFS share directory where Secure Eraser will save the certificate.

      • The seventh line (initrdefi) specifies the initrd file to load. In this case it is the initrd file copied over in step 1.

    By default, the examples shown above cause Secure Eraser to erase all components. You can use secureeraser-options to specify command-line options for Secure Eraser to change the default behavior and securely erase certain components only. For example, to erase hard drives and USBs only during the PXE boot, the template would look like this:

    For grub:

    default linux
    label linux
    kernel vmlinux-nfs-12.2.1.1.0-161015-cell
    append initrd=initrd-nfs-12.2.1.1.0-161015-cell.img dhcp pxe quiet loglevel=0 
    secureeraser secureeraser-options="--hdd --usb" bootarea=diagnostics 
    console=tty1 console=ttyS0,115200n8 logpath=10.133.42.221:/export/exadata_
    secure_eraser_certificate_dir
    

    For grub2 / SecureBoot:

    Note:

    In the following example, the following parameters must be updated to match your environment:
    • kernel (the vmlinux file)

    • initrd (the initrd*img file)

    • logpath

    set default 0 
    set timeout=10
    menuentry 'ExadataLinux' {
       echo "Loading efi/vmlinuz" 
       linuxefi efi/vmlinux-nfs-18.1.0.0.0-170915.1 stit dhcp pxe boot-from=
    uefi quiet loglevel=0 secureeraser secureeraser-options="--hdd --usb" 
    bootarea=diagnostics console=ttyS0,115200n8 logpath=10.133.42
    .221:/export/exadata_secure_eraser_certificate_dir
       echo "Loading efi/initrd.img"
      initrdefi efi/initrd-nfs-18.1.0.0.0-170915.1.img
       echo "Booting installation kernel"
    }
    
  5. On the PXE server, use the template file to generate a PXE configuration file in the /tftpboot/pxelinux.cfg/ directory for each of the nodes to be erased.

    The PXE configuration file name is the dash-separated MAC address of the node with the prefix 01-.

    If the nodes to be erased are accessible, use the following steps to automatically generate a PXE configuration file for each node based on the template.

    1. Set up SSH equivalence with the nodes to be erased from the PXE server. The command will prompt for the root password of each node.

      pxe_server# dcli -g nodes_to_be_erased -k -l root
      
    2. Create PXE configuration files, one for each node to be erased based on the configuration template.

      pxe_server# dcli -g nodes_to_be_erased -l root "ip addr show eth0" | 
      awk '/link\/ether/ {print "01:"$3}' | sed "s/:/-/g" | 
      xargs -I {} cp pxe_cfg.template {}
      

    If the nodes are not accessible, use the following step to generate a PXE configuration file for each node to be erased:

    1. Manually collect the MAC address of the eth0 interface from each node and write them into a text file called mac_addresses. Write one MAC address per line. For example:

      00:10:e0:62:c4:fa
      00:10:e0:62:c2:8a
      00:10:e0:62:b8:7c
      00:10:e0:62:b8:3a
      00:10:e0:62:c6:bc
      
    2. Use the following command to create a list of PXE configuration files, one for each node to be erased based on the configuration template.

      pxe_server# cat mac_addresses | sed "s/:/-/g;s/^/01-/g" | 
      xargs -I {} cp pxe_cfg.template {}
      

    In both cases, you should get a list of PXE configuration files, one for each node to be erased. For example, if the MAC addresses of the nodes in a quarter rack are 00:10:e0:62:c4:fa, 00:10:e0:62:c2:8a, 00:10:e0:62:b8:7c, 00:10:e0:62:b8:3a, and 00:10:e0:62:c6:bc, you should get the following files:

    01-00-10-e0-62-c4-fa
    01-00-10-e0-62-c2-8a
    01-00-10-e0-62-b8-7c
    01-00-10-e0-62-b8-3a
    01-00-10-e0-62-c6-bc
    

    The files have the same content as the configuration template.

    Check your specific PXE server requirements. Your PXE server may need slightly different names or settings.

  6. Configure the nodes to boot from PXE and reboot the nodes.

    If the nodes to be erased are accessible, run the following commands:

    pxe_server# dcli -g nodes_to_be_erased -l root "ipmitool chassis bootdev pxe”
    
    pxe_server# dcli -g nodes_to_be_erased -l root "reboot”
    

    If the nodes to be erased are not remotely accessible but the ILOMs are, use the following steps

    1. Create a file called iloms_to_be_reset containing the names of ILOMs. For example:

      db1-ilom
      db2-ilom
      cell1-ilom
      cell2-ilom
      cell3-ilom
      
    2. Configure the nodes to boot from PXE through ILOMs. The command will prompt for ILOM root password.

      pxe_server# cat iloms_to_be_reset | xargs -I {} ipmitool -I lanplus -H 
      {} -U root chassis bootdev pxe
      
    3. Reboot the nodes from ILOMs. The command will prompt for ILOM root password.

      pxe_server# cat iloms_to_be_reset | xargs -I {} ipmitool -I lanplus -H 
      {} -U root chassis power cycle
      

    If neither host nor ILOM is remotely accessible, log into ILOM using a serial console and run the following commands

    ILOM> set /HOST/boot_device=pxe
    
    ILOM> reset /SYS
    
  7. The Secure Eraser utility will be automatically called to sanitize all storage media including hard drives, flash devices, and internal USBs, and to reset ILOM to factory default for all nodes in parallel.

    Secure Eraser creates a file called secureeraser_node_chassis_number_date_time.certificate in the specified logpath location. node_chassis_number is the ID attribute of the storage server or database server in CellCLI or DBMCLI.

    The file contains a progress report that is updated every 10 seconds. The progress report is also output to the console on each node. The following is an example of the progress report:

    ID  Type   Model                            Serial Number     Size      Status
    1   Flash  Flash Accelerator F80 PCIe Card  FL00A96H          200.00GB  Being Erased (6%)
    2   Flash  Flash Accelerator F80 PCIe Card  FL00A84Y          200.00GB  Being Erased (5%)
    3   Flash  Flash Accelerator F80 PCIe Card  FL00A7D4          200.00GB  Being Erased (5%)
    4   Flash  Flash Accelerator F80 PCIe Card  FL00A6WG          200.00GB  Being Erased (6%)
    5   Flash  Flash Accelerator F80 PCIe Card  FL008KSE          200.00GB  Being Erased (5%)
    6   Flash  Flash Accelerator F80 PCIe Card  FL008KS3          200.00GB  Being Erased (5%)
    7   Flash  Flash Accelerator F80 PCIe Card  FL008KL7          200.00GB  Being Erased (5%)
    8   Flash  Flash Accelerator F80 PCIe Card  FL008KQR          200.00GB  Being Erased (6%)
    9   Flash  Flash Accelerator F80 PCIe Card  FL00A812          200.00GB  Being Erased (5%)
    10  Flash  Flash Accelerator F80 PCIe Card  FL00A79G          200.00GB  Being Erased (5%)
    11  Flash  Flash Accelerator F80 PCIe Card  FL00A80C          200.00GB  Being Erased (6%)
    12  Flash  Flash Accelerator F80 PCIe Card  FL00A79F          200.00GB  Being Erased (6%)
    13  Flash  Flash Accelerator F80 PCIe Card  FL00A5WD          200.00GB  Being Erased (5%)
    14  Flash  Flash Accelerator F80 PCIe Card  FL00A5XS          200.00GB  Being Erased (5%)
    15  Flash  Flash Accelerator F80 PCIe Card  FL00A7N1          200.00GB  Being Erased (5%)
    16  Flash  Flash Accelerator F80 PCIe Card  FL00A62G          200.00GB  Being Erased (5%)
    17  Disk   H7240AS60SUN4.0T                 1352E5XHWX        4.00TB    Being Erased (1%)
    18  Disk   H7240AS60SUN4.0T                 1352E60SYX        4.00TB    Being Erased (1%)
    19  Disk   H7240AS60SUN4.0T                 1352E60U4X        4.00TB    Being Erased (1%)
    20  Disk   H7240AS60SUN4.0T                 1352E5UPAX        4.00TB    Being Erased (1%)
    21  Disk   H7240AS60SUN4.0T                 1352E5XK3X        4.00TB    Being Erased (1%)
    22  Disk   H7240AS60SUN4.0T                 1352E62M7X        4.00TB    Being Erased (1%)
    23  Disk   H7240AS60SUN4.0T                 1352E5PSPX        4.00TB    Being Erased (1%)
    24  Disk   H7240AS60SUN4.0T                 1352E60TJX        4.00TB    Being Erased (1%)
    25  Disk   H7240AS60SUN4.0T                 1352E5LYDX        4.00TB    Being Erased (1%)
    26  Disk   H7240AS60SUN4.0T                 1352E602WX        4.00TB    Being Erased (1%)
    27  Disk   H7240AS60SUN4.0T                 1352E5LY9X        4.00TB    Being Erased (1%)
    28  Disk   H7240AS60SUN4.0T                 1352E5VX4X        4.00TB    Being Erased (1%)
    29  USB    SSM                              1900638EA8BFB749  8.00GB    Being Erased (5%)
    30  ILOM                                    1403NM50CA                        To Be Reset
    
    

    As the sample progress report shows, Secure Eraser erases all storage devices in parallel. After the storage devices are securely erased, Secure Eraser will reset the ILOM to the factory default. This is to ensure that in the case that secure erasure fails on some storage device, the web console is still accessible for remote debugging, and ILOM is still accessible to control the host.

    Once secure erasure is complete, a certificate called secureeraser_node_chassis_number_date_time.certificate.pdf is generated at the NFS share location specified by the logpath option in step 4. If secure erasure is successful, the nodes will be shut down automatically. If Secure Eraser does not succeed on some components, then the node will be left in diagnostic shell for further debugging. Assuming all previous steps are successful, and you have resolved the issue, you can go back to step 6 and rerun Secure Eraser.

5.4 Interactive Secure Eraser through PXE Boot

On Exadata systems prior to Oracle Exadata Database Machine X7-2, you can use Preboot Execution Environment (PXE) Boot when performing a Secure Eraser.

Before you begin:

  • Review My Oracle Support note 2180963.1 for the latest information and to download the Secure Eraser package

  • Make sure you have access to a PXE server where the nodes to be erased can boot from.

  • Make sure you have access to a NFS server that is accessible from all the nodes to be erased.

  • Make sure you have access to one of the nodes to be erased.

  1. Copy the cell PXE image files initrd (initrd-version) and kernel (vmlinux-version) from the Secure Eraser package to the /tftpboot directory on the PXE server. For Oracle Exadata Database Machine X7 systems, the directory is /tftpboot/efi for grub2 and Secure Boot.

  2. Create a file containing the names of the database servers and storage servers to be erased.

    To generate this file, you can run the following command from one of the nodes to be erased, and verify the nodes in the files are the ones to be erased.

    # ibhosts | awk '/S [0-9\.\,]*/ || /C [0-9\.\,]*/ {print $6}' | sed "s/\"//g" 
    > nodes_to_be_erased
    

    If you only want to erase one server, enter the name of the server into the nodes_to_be_erased file, for example Exa01celadm04.

  3. Copy the dcli utility from the Secure Eraser package along with the file generated in step 2 to the PXE server.

  4. Create a PXE configuration template called pxe_cfg.template to contain the following lines:

    • For all models before Oracle Exadata Database Machine X7-2grub:

      default linux
      label linux
      kernel vmlinux-nfs-12.2.1.1.0-161015-cell
      append initrd=initrd-nfs-12.2.1.1.0-161015-cell.img dhcp pxe quiet loglevel=0 
      bootarea=diagnostics console=tty1 console=ttyS0,115200n8
      
    • For Oracle Exadata Database Machine X7-2 and newer models —grub2 / Secure Boot:

      Note:

      In the following example, the following parameters must be updated to match your environment:

      • kernel (the vmlinux file)

      • initrd (the initrd*.img file)

      • logpath

      set default 0
      set timeout=10
      menuentry 'ExadataLinux' {
        echo "Loading efi/vmlinuz"   
       linuxefi efi/vvmlinux-nfs-18.1.0.0.0-170915.1 stit dhcp pxe boot-from=uefi quiet
       loglevel=0 bootarea=diagnostics console=ttyS0,115200n8 
      logpath=10.133.42.221:/export/exadata_secure_eraser_certificate_dir   
        echo "Loading efi/initrd.img"   
        initrdefi efi/initrd-nfs-18.1.0.0.0-170915.1.img 
        echo "Booting installation kernel"
      }
      

    This configuration file differs from the one in Automatic Secure Eraser through PXE Boot in that the secureeraser option is left out to indicate that no Secure Eraser shall be triggered automatically. The rest of the file is the same.

  5. On the PXE server, use the template file to generate a PXE configuration file. For all systems up to Oracle Exadata Database Machine X6-2, save the file in the /tftpboot/pxelinux.cfg/ directory for each of the nodes to be erased. For Oracle Exadata Database Machine X7-2 and newer systems, save the file in the /tftpboot/efi/pxelinux.cfg/ directory for each of the nodes to be erased.

    The PXE configuration file name is the dash-separated MAC address of the node with the prefix 01-.

    If the nodes to be erased are accessible, perform the following steps to automatically generate a PXE configuration file for each node based on the template:

    1. Set up SSH equivalence with the nodes to be erased from the PXE server. The command will prompt for the root password of each node.

      pxe_server# dcli -g nodes_to_be_erased -k -l root
      
    2. Create a list of PXE configuration files, one for each node to be erased based on the configuration template.

      pxe_server# dcli -g nodes_to_be_erased -l root "ip addr show eth0" | 
      awk '/link\/ether/ {print "01:"$3}' | sed "s/:/-/g" | xargs -I {} 
      cp pxe_cfg.template {}
      

    If the nodes are not accessible, perform the following steps to generate a PXE configuration file for each node.

    1. Manually collect the MAC address of the eth0 interface from each node and write them into a text file called mac_addresses. Write one MAC address per line. For example:

      00:10:e0:62:c4:fa
      00:10:e0:62:c2:8a
      00:10:e0:62:b8:7c
      00:10:e0:62:b8:3a
      00:10:e0:62:c6:bc
      
    2. Use the following command to create a list of PXE configuration file, one for each node to be erased based on the configuration template.

      pxe_server# cat mac_addresses | sed "s/:/-/g;s/^/01-/g" | xargs -I {} cp 
      pxe_cfg.template {}
      

    In both cases, you should have a list of PXE configuration files, one for each node to be erased. For example, if the MAC addresses of the nodes in a quarter rack are 00:10:e0:62:c4:fa, 00:10:e0:62:c2:8a, 00:10:e0:62:b8:7c, 00:10:e0:62:b8:3a, and 00:10:e0:62:c6:bc, then you should get the following files:

    01-00-10-e0-62-c4-fa
    01-00-10-e0-62-c2-8a
    01-00-10-e0-62-b8-7c
    01-00-10-e0-62-b8-3a
    01-00-10-e0-62-c6-bc
    

    The files have the same content as the configuration template.

    Check your specific PXE server requirements. Your PXE server may need slightly different names or settings.

  6. Configure the nodes to boot from PXE and reboot the nodes.

    If the nodes to be erased are accessible, run the following commands:

    pxe_server# dcli -g nodes_to_be_erased -l root "ipmitool chassis bootdev pxe”
    
    pxe_server# dcli -g nodes_to_be_erased -l root "reboot”
    

    If the nodes are not accessible, then perform the following steps:

    1. Create a file called iloms_to_be_reset containing the names of ILOMs. For example:

      db1-ilom
      db2-ilom
      cell1-ilom
      cell2-ilom
      cell3-ilom
      
    2. Configure the nodes to boot from PXE through ILOMs. The command will prompt for ILOM root password.

      pxe_server# cat iloms_to_be_reset | xargs -I {} ipmitool -I lanplus -H {} 
      -U root chassis bootdev pxe
      
    3. Reboot the nodes from ILOMs. The command will prompt for ILOM root password.

      pxe_server# cat iloms_to_be_reset | xargs -I {} ipmitool -I lanplus -H {} 
      -U root chassis power cycle
      
  7. If you get the following prompt on the remote or serial console, enter e at the prompt to enter the diagnostic shell:

    Choose from following by typing letter in '()':
    (e)nter interactive diagnostics shell. Must use credentials from Oracle
    support to login (reboot or power cycle to exit the shell),
    Select:e
    
  8. Log in to the system as the root user.

    You will be prompted for the password for the root user of diagnostic shell. The password can be retrieved from Oracle Support Services.

    localhost login: root
    Password: *********
    -sh-3.1#
    
  9. Run the Secure Eraser utility to sanitize all devices or one type of device.

    -sh-3.1# /usr/sbin/secureeraser -–erase --all --output=REMOTE_NFS_LOCATION
    

    REMOTE_NFS_LOCATION is the remote NFS location in the format of IP:FILE_PATH. The Secure Eraser utility will automatically mount the remote NFS location and save the certificate there.

    For example, to erase all devices including hard drives, flash devices, internal USBs, and ILOM, and save the certificate at this NFS location: 10.133.42.221:/export/exadata_secure_eraser_certificate_dir:

    -sh-3.1# /usr/sbin/secureeraser --erase --all --output=10.133.42.221:/export
    /exadata_secure_eraser_certificate_dir
    

    To erase just the hard drives:

    -sh-3.1# /usr/sbin/secureeraser --erase --hdd --output=10.133.42.221:/export
    /exadata_secure_eraser_certificate_dir
    

    Note that it is important to point the output option to an NFS location so that the certificate can be saved properly.

    You will be prompted with a list of devices to be erased and to confirm that you want to proceed with Secure Eraser.

    A progress report, as shown in step 7 of the Automatic Secure Eraser through PXE Boot, will be printed to the console every 10 seconds.

    In interactive mode, the server will be left on after the specified devices are securely erased. You can power off the node from the diagnostic shell.

    The web console will no longer be accessible if ILOM is reset. You can power off the server from the serial console or with the power button.

5.5 Interactive Secure Eraser through Network Boot

Starting with Oracle Exadata Database Machine X7-2, you can use EFI Network Boot when using Secure Eraser.

Before you begin:

  • Review My Oracle Support Doc ID 2180963.1 for the latest information

  • Make sure you have access to a NFS server that is accessible from all the nodes to be erased.

  • Make sure you have access to one of the nodes to be erased.

  1. Copy the cell initrd file (initrd-version) and kernel file (vmlinux-version) from the Secure Eraser package to the /tftpboot/efi on the network boot server.

    The location does not have to be /tftpboot/efi/. The location is determined by the TFTP Server configuration.

  2. Create a file containing the names of the database servers and storage servers to be erased.

    To generate this file, you can run the following command from one of the nodes to be erased, and verify the nodes in the files are the ones to be erased.

    # ibhosts | awk '/S [0-9\.\,]*/ || /C [0-9\.\,]*/ {print $6}' | sed "s/\"//g" 
    > nodes_to_be_erased
    

    If you only want to erase one server, enter the name of the server into the nodes_to_be_erased file, for example Exa01celadm04.

  3. Copy the dcli utility from the Secure Eraser package along with the file generated in step 2 to the network boot server.

  4. Create a configuration template called pxe_cfg.template to contain the following lines for grub2 and Secure Boot on Oracle Exadata Database Machine X7-2 and newer systems:

    Note:

    In the following example, the following parameters must be updated to match your environment:

    • kernel (the vmlinux file)

    • initrd (the initrd*.img file)

    • logpath

    set default 0
    set timeout=10
    menuentry 'ExadataLinux' {
      echo "Loading efi/vmlinuz"   
     linuxefi efi/vmlinux-nfs-18.1.0.0.0-170915.1 stit dhcp pxe boot-from=uefi quiet 
     loglevel=0 bootarea=diagnostics console=ttyS0,115200n8 
     logpath=10.133.42.221:/export/exadata_secure_eraser_certificate_dir   
      echo "Loading efi/initrd.img"   
      initrdefi efi/initrd-nfs-18.1.0.0.0-170915.1.img 
      echo "Booting installation kernel"
    }
    
    • The first line (default) identifies a menu entry that should be selected by default, after the timeout value specified by the second line.

    • The third line (menuentry) represents the Linux kernel that will be used in the Secure Eraser environment.

    • The fifth line (linuxefi) indicates the kernel is on an UEFI-based system. The linuxefi statement must be on a single line in the configuration file.

      • The stit option specifies INSTALL imaging mode, plus BARE METAL conditions, plus ERASING ADD DATA PARTITIONS

      • The dhcp option specifies to use DHCP to discover the eth0 interface.

      • The pxe option suppresses search for the image on virtual CD and USB devices.

      • The boot-from=uefi option indicates the system is booting from UEFI.

      • The quiet option disables excessive kernel log messages.

      • The loglevel=0 option suppresses non-critical kernel messages.

      • The secureeraser option indicates the network boot will automatically trigger the Secure Eraser utility to sanitize all media including hard drives, flash devices, internal USBs, and ILOM on the node.

      • The bootarea option indicates that the boot mode is diagnostic and not imaging install or rescue.

      • The console options indicate standard output and standard error messages are printed to both the ILOM web console and the serial console.

      • The logpath option specifies the NFS share directory where Secure Eraser will save the certificate.

    • The seventh line (initrdefi) specifies the initrd file to load. In this case it is the initrd file copied over in step 1.

  5. On the network boot server, use the template file to generate a network boot configuration file in the /tftpboot/efi/pxelinux.cfg/ directory (Oracle Exadata Database Machine X7-2 and newer) for each of the nodes to be erased.

    The network boot configuration file name is the dash-separated MAC address of the node with the prefix 01-.

    If the nodes to be erased are accessible, perform the following steps to automatically generate a network boot configuration file for each node based on the template:

    1. Set up SSH equivalence with the nodes to be erased from the network boot server. The command will prompt for the root password of each node.

      pxe_server# dcli -g nodes_to_be_erased -k -l root
      
    2. Create a list of network boot configuration files, one for each node to be erased based on the configuration template.

      pxe_server# dcli -g nodes_to_be_erased -l root "ip addr show eth0" | 
      awk '/link\/ether/ {print "01:"$3}' | sed "s/:/-/g" | xargs -I {} 
      cp pxe_cfg.template {}
      

    If the nodes are not accessible, perform the following steps to generate a network boot configuration file for each node.

    1. Manually collect the MAC address of the eth0 interface from each node and write them into a text file called mac_addresses. Write one MAC address per line. For example:

      00:10:e0:62:c4:fa
      00:10:e0:62:c2:8a
      00:10:e0:62:b8:7c
      00:10:e0:62:b8:3a
      00:10:e0:62:c6:bc
      
    2. Use the following command to create a list of network boot configuration file, one for each node to be erased based on the configuration template.

      pxe_server# cat mac_addresses | sed "s/:/-/g;s/^/01-/g" | xargs -I {} cp 
      pxe_cfg.template {}
      

    In both cases, you should have a list of network boot configuration files, one for each node to be erased. For example, if the MAC addresses of the nodes in a quarter rack are 00:10:e0:62:c4:fa, 00:10:e0:62:c2:8a, 00:10:e0:62:b8:7c, 00:10:e0:62:b8:3a, and 00:10:e0:62:c6:bc, then you should get the following files:

    01-00-10-e0-62-c4-fa
    01-00-10-e0-62-c2-8a
    01-00-10-e0-62-b8-7c
    01-00-10-e0-62-b8-3a
    01-00-10-e0-62-c6-bc
    

    The files have the same content as the configuration template.

    Check your specific network boot server requirements. Your network boot server may need slightly different names or settings.

  6. Configure the nodes to boot from the network boot server and reboot the nodes.

    If the nodes to be erased are accessible, run the following commands:

    pxe_server# dcli -g nodes_to_be_erased -l root "ipmitool chassis bootdev pxe”
    
    pxe_server# dcli -g nodes_to_be_erased -l root "reboot”
    

    If the nodes are not accessible, then perform the following steps:

    1. Create a file called iloms_to_be_reset containing the names of ILOMs. For example:

      db1-ilom
      db2-ilom
      cell1-ilom
      cell2-ilom
      cell3-ilom
      
    2. Configure the nodes to boot from the network boot server through ILOMs. The command will prompt for ILOM root password.

      pxe_server# cat iloms_to_be_reset | xargs -I {} ipmitool -I lanplus -H {} 
      -U root chassis bootdev pxe
      
    3. Reboot the nodes from ILOMs. The command will prompt for ILOM root password.

      pxe_server# cat iloms_to_be_reset | xargs -I {} ipmitool -I lanplus -H {} 
      -U root chassis power cycle
      
  7. If you get the following prompt on the remote or serial console, enter e at the prompt to enter the diagnostic shell:

    Choose from following by typing letter in '()':
    (e)nter interactive diagnostics shell. Must use credentials from Oracle
    support to login (reboot or power cycle to exit the shell),
    Select:e
    
  8. Log in to the system as the root user.

    You will be prompted for the password for the root user of diagnostic shell. The password can be retrieved from Oracle Support Services.

    localhost login: root
    Password: *********
    -sh-3.1#
    
  9. Run the Secure Eraser utility to sanitize all devices or one type of device.

    -sh-3.1# /usr/sbin/secureeraser -–erase --all --output=REMOTE_NFS_LOCATION
    

    REMOTE_NFS_LOCATION is the remote NFS location in the format of IP:FILE_PATH. The Secure Eraser utility will automatically mount the remote NFS location and save the certificate there.

    For example, to erase all devices including hard drives, flash devices, internal USBs, and ILOM, and save the certificate at this NFS location: 10.133.42.221:/export/exadata_secure_eraser_certificate_dir:

    -sh-3.1# /usr/sbin/secureeraser --erase --all --output=10.133.42.221:/export
    /exadata_secure_eraser_certificate_dir
    

    To erase just the hard drives:

    -sh-3.1# /usr/sbin/secureeraser --erase --hdd --output=10.133.42.221:/export
    /exadata_secure_eraser_certificate_dir
    

    Note that it is important to point the output option to an NFS location so that the certificate can be saved properly.

    You will be prompted with a list of devices to be erased and to confirm that you want to proceed with the Secure Eraser.

    A progress report, as shown in step 7 of the Automatic Secure Eraser through PXE Boot, will be printed to the console every 10 seconds.

    In interactive mode, the server will be left on after the specified devices are securely erased. You can power off the node from the diagnostic shell.

    The web console will no longer be accessible if ILOM is reset. You can power off the server from the serial console or with the power button.

5.6 Interactive Secure Eraser through External USB

You can securely erase a node using an external USB drive.

Before you begin:

  • Review My Oracle Support note 2180963.1 for the latest information and to download the Secure Eraser package.

  • Make sure you have an external USB.

  • Make sure you have physical access to the nodes to be erased.

  1. Copy the diagnostic image from the Secure Eraser package to an external USB.
    # dd if=PATH_TO_DIAGNOSTIC_IMAGE of=USB_DEVICE
    

    For example:

    # dd if=image_diagnostics_12.2.1.1.0_LINUX.X64_161015-1.x86_64.usb of=/dev/sdm
    
  2. Insert the external USB to the node to be securely erased.

    External USB slots are located at both the front panel and the back panel of 2-socket database servers and storage servers. On 8-socket database servers, external USB slots are located at the back panel.

  3. Reboot the node from the external USB by pressing CTRL+P after BIOS initialization splash screens and selecting the inserted external USB device.
  4. Login to the server.
    1. If you are erasing a storage server, then login to the diagnostic shell.

      Enter e to enter the diagnostic shell as follows:

      Choose from following by typing letter in '()':
      (e)nter interactive diagnostics shell. Must use credentials from Oracle
      support to login (reboot or power cycle to exit the shell),
      Select:e
      
    2. If you are erasing database servers, then log in as the root user.

      You will be prompted for the password for the root user of diagnostic shell. The password can be retrieved from Oracle Support Services.

      localhost login: root
      Password: *********
      -sh-3.1#
      
  5. Run the Secure Eraser utility to sanitize all devices or one type of device.

    For example, to erase all devices:

    -sh-3.1# /usr/sbin/secureeraser --erase --all --output=/mnt/iso
    

    To erase just the hard drives:

    -sh-3.1# /usr/sbin/secureeraser --erase --hdd --output=/mnt/iso
    

    By default, /mnt/iso is the mount point for the external USB when system is booted from the diagnostic ISO on the external USB. It is important to point the output option to the external USB mount point /mnt/iso so that the certificate can be saved properly.

  6. Secure Eraser prompts you with a list of devices to be erased. Confirm that you want to proceed with the secure erasure.

    A progress report, as shown in step 7 of Automatic Secure Eraser through PXE Boot, is printed to the console every 10 seconds.

In interactive mode, the server will be left on after the specified devices are securely erased. You can power off the node from the diagnostic shell.

The web console will no longer be accessible if ILOM is reset. You need to power off the server from the serial console or with the power button.

5.7 Secure Eraser Syntax

The Secure Eraser utility has several options.

Syntax:

secureeraser [options]

Table 5-2 Command-Line Options for Secure Eraser

Command-Line Option Description

--list

List devices (hard drives, flash devices, USB devices, and ILOM) on the system.

--erase

Perform a secure erase of data.

-q, --quiet

Quietly skip prompts.

--all

Perform the action (--list or --erase) on all devices on the system. Devices include hard drives, flash devices, USB devices, and ILOM.

--hdd

Erase all hard drives.

--hdd_erasure_method=HDD_ERASURE_METHOD

Erase all hard drives using the specified method. The following values are supported:

  • 3pass

  • 7pass

--flash

Erase all flash devices.

--flash_erasure_method=FLASH_ERASURE_METHOD

Erase all flash devices using the specified method. The following values are supported:

  • 3pass

  • 7pass

—m2

Erase all M.2 devices.

--m2_erasure_method=M2_ERASURE_METHOD

Erase all M.2 devices using the specified method. The following values are supported:

  • 3pass

  • 7pass

--usb

Erase all internal USB devices.

--usb_erasure_method=USB_ERASURE_METHOD

Erase all internal USB devices using the specified method. The following values are supported:

  • 3pass

  • 7pass

--ilom

Reset ILOM to factory default.

-t TECHNICIAN_NAME, --technician=TECHNICIAN_NAME

Specify the name of the technician performing the erasure. This name will be recorded in the certificate.

-w WITNESS_NAME, --witness=WITNESS_NAME

Specify the name of the person witnessing the erasure. This name will be recorded in the certificate.

-o, --output=CERTIFICATE_DIRECTORY

Specify a full path to the directory for the certificate output location. The default is /var/log/cellos.

-h, --help

Show this help message and exit.

Examples

List all devices (hard drives, flash devices, USB devices, and ILOM) on the system.

secureeraser --list --all

List all hard drives.

secureeraser --list --hdd

Securely erase all devices, and enter the names of the technician and witness in the certificate.

secureeraser --erase --all --technician="jdoe" --witness="jsmith"

Reset ILOM to factory default.

secureeraser --erase --ilom

Securely erase all hard drives.

secureeraser --erase --hdd

Securely erase all hard drives, all flash devices, and all internal USB devices. Force "3-pass" method on flash devices.

secureeraser --erase --hdd --flash --usb --flash_erasure_method 3pass

5.8 Resetting InfiniBand Switches, Ethernet Switch, and Power Distribution Units to Factory Default

Before you begin:

  • Review My Oracle Support note 2180963.1 for the latest information and to download the Secure Eraser package

  • Print out the Exadata Factory Reset Certificate template in the Secure Eraser package.

The following figure shows the Factory Reset certificate:

Figure 5-2 Factory Reset Certificate

Description of Figure 5-2 follows
Description of "Figure 5-2 Factory Reset Certificate"

5.8.1 Resetting InfiniBand Switches to Factory Default

To reset Sun Datacenter InfiniBand Switch 36 to factory default, refer to My Oracle Support note 2180877.1.

Record the serial numbers of the switches that have been reset to factory default in the Exadata Factory Reset certificate template. Sign and date the entries.

You can identify the serial number of an InfiniBand switch by running the following command on the switch:

[root@switch1 ~]# version | grep "Serial Number"

5.8.2 Resetting Ethernet Switch to Factory Default

You can reset the Ethernet switch configuration to the original default factory settings.

For Oracle Exadata Database Machine X6-2 or earlier

To reset the Ethernet switch to factory default, refer to "Reset Catalyst Switches Running Cisco IOS Software" in the Cisco Troubleshooting TechNotes Document ID: 24328.

Record the serial number of the Ethernet switch that has been reset to factory default in the Exadata Factory Reset certificate template. Sign and date the entry.

The serial number of an Ethernet switch can be identified by the “Processor board ID” field in the “show version” command output.

Switch# show version

For Oracle Exadata Database Machine X7-2 or later

To reset the Ethernet switch to factory default:

  1. Display the start up configuration.

    # show startup-config
    
  2. Display the boot configuration.

    # show boot
    
  3. Display the debug configuration.

    # show debug
    
  4. Erase the startup-configuration file.

    # write erase 
    
    Warning: This command will erase the startup-configuration.
    
    Do you wish to proceed anyway? (y/n)  [n] y
    
  5. Erase the boot variable definitions.

    # write erase boot
    
    Warning: This command will erase the boot variables and the ip configuration of interface mgmt 0
    
    Do you wish to proceed anyway? (y/n)  [n] y
    
  6. Erase the debugging configuration.

    # write erase debug
    
  7. Reload the Cisco Nexus 93108-1G or Cisco Nexus 9348 Ethernet switch.

    # reload
    
    This command will reboot the system. (y/n)  [n] y
    

See Also:

"Erasing a Configuration" in Cisco Nexus 9000 Series NX-OS Fundamentals Configuration Guide, Release 6.x

5.8.3 Resetting Power Distribution Units to Factory Default

You can reset the power distribution units (PDUs) configuration to the original default factory settings.

There are two types of power distribution units (PDUs): original PDUs and enhanced PDUs. Enhanced PDUs have SER MGT port that can be connected to a host using an RS-232 cable, whereas the original PDUs do not have SER MGT port. Typically, Oracle Exadata Database Machine V2 to Oracle Exadata Database Machine X3 racks have the original PDUs, and Oracle Exadata Database Machine X4-2 and later have the enhanced PDUs.

You can reset both the original power distribution units and the enhanced power distribution units, as described in the Sun Rack II Power Distribution Units User’s Guideor the Oracle Rack Cabinet 1242 Power Distribution Units User's Guide (for Oracle Exadata Database Machine X7-2 and later systems).

Record the serial numbers of the power distribution units that have been reset to factory default in the Exadata Factory Reset certificate template. Sign and date the entries.

The serial number can be found on the “View Module Information” topic of the appropriate Power Distribution Units User's Guide.

For enhanced PDUs, the serial number can also be retrieved through the following CLI command:

pducli -> get pdu_serial_number

See Also:

5.9 Actions After Using Secure Eraser

After performing a secure erase, the system is ready for return or re-imaging.

If you plan to re-image the machine, you must perform the following steps:

  1. Connect to and configure ILOM. See "Oracle ILOM – Quick Start" in the Oracle ILOM Getting Started Guide Firmware Release 4.0.x.

    Perform the following tasks:

    1. Connect to Oracle ILOM

    2. Log In to Oracle ILOM

    3. Modify Default Network Connectivity Settings

  2. Re-image the system. Refer to Imaging a New System in Oracle Exadata Database Machine Installation and Configuration Guide.

If you are preparing to return the machine, refer to http://www.oracle.com/us/products/servers-storage/take-back-and-recycling/index.html