3.8.5 Dropping a Storage Server from an Existing Disk Group or Storage Grid

You can remove a storage server from an existing Oracle Exadata Rack.

1. Drop the disks belonging to the storage server to be removed from Oracle Automatic Storage Management (Oracle ASM).

   Note:

   For Oracle Exadata VM deployments, the substeps below need to be performed on all of the Oracle VM clusters.
   1. Log in to any node in the cluster.
   2. Query the list of grid disks being used by the cluster for the targeted storage server.

      Grid_home/bin/asmcmd  lsdsk --suppressheader | grep cellName_being_removed | awk -F'/' '{print $NF}'

      Note:

      Make sure the available free space in every disk group that contains disks from the storage server being removed is at least 15% of the allocated storage for that disk group.
   3. Drop the Oracle ASM disks returned by the command above from their respective disk groups.

      SQL> ALTER DISKGROUP diskgroup_name DROP DISKS IN FAILGROUP cellName_being_removed;

   4. The disk drop operation above kicks off a rebalance operation at the default power level. Monitor for the rebalance using the following command:

      SQL> SELECT * FROM gv$asm_operation;

      Wait until the rebalance completes, that is, wait until gv$asm_operation returns no rows.

   5. Verify that all the disk groups do not have any references to the disks from the storage server being removed.

      SQL> SELECT path, name, header_status, mode_status, mount_status, state,
       failgroup FROM v$asm_disk ORDER BY path;
      

      The header_status column for all the disks belonging to the storage server being removed should show FORMER.

      Reminder:

      For Exadata Oracle VM deployments, the substeps above need to be performed on all of the Oracle VM clusters.
2. Clean up the storage server being removed.

   Log in to the storage server as celladmin and run the following commands. Run the following commands for each set of grid disks:

   1. Drop the grid disks.

      cellcli -e drop griddisk all prefix=prefix_of_the_grid_disk

   2. If flash cache exists and the storage server is in WriteBack flash cache mode, you must first flush the flash cache before dropping it.

      cellcli -e alter flashcache all flush

      Wait for the command to return.

   3. Drop the flash cache.

      cellcli -e drop flashcache all
      

   4. Drop the cell disks.

      cellcli -e drop celldisk all
      

      If you need to erase data securely, you can run the DROP CELLDISK command with the erase option, or the DROP CELL with the erase option.

      The time required to complete the erase operation is listed in the table under the DROP CELL command.

3. Remove the entry of the storage server being removed from /etc/oracle/cell/network-config/cellip.ora on all the database server nodes in the cluster.
   Run the following steps on any database server node in the cluster:
   1. Make a backup copies of the cellip.ora file.

      cd /etc/oracle/cell/network-config
      cp cellip.ora cellip.ora.orig
      cp cellip.ora cellip.ora-bak

   2. Remove the entries for the storage server being removed from /etc/oracle/cell/network-config/cellip.ora-bak.
   3. Use dcli to copy the updated cellip.ora-bak file to the other database servers.

      In the following command database_nodes refers to a file containing the names of each database server in the cluster. Each database server name is on a separate line in the file.

      /usr/local/bin/dcli -g database_nodes -l root -f cellip.ora-bak -d 
      /etc/oracle/cell/network-config/cellip.ora

4. Download and run the latest version of Exachk to ensure that the resulting configuration implements the latest best practices for Oracle Exadata.