2.14.6 Reconfigure Quorum Disk After Restoring a Database Server

After restoring a database server, lvdisplay shows the quorum disk was not restored.

When you restore a database server, Exadata image rescue mode restores the layout of disks and file systems, with the exception of custom partitions, including quorum disks. These files must be recreated after being restored from backup.

The logical volumes created for quorum disks are in /dev/VGExaDb and have the name-prefix LVDbVd*.

1. Using the configuration backed up under /etc/lvm/archive, make a logical volume (LV) for the quorum disk on every node.

   For example, you would use a command similar to the following, but using values from the backup configuration information.

   # lvcreate -L 128MB -n <LVName> VGExaDb

2. Reboot all database servers.

   # shutdown -r now

3. After the servers restart, verify the quorum disks are restored.

   # /opt/oracle.SupportTools/quorumdiskmgr --list --config
   Owner: grid
   Group: dba
   ifaces: exadata_ib1 exadata_ib0
   
   # /opt/oracle.SupportTools/quorumdiskmgr --list --target
   Name: iqn.2015-05.com.oracle:QD_DATAC1_DB01
   Host name: DB01
   ASM disk group name: DATAC1
   Size: 128 MB
   Visible to: 192.168.10.45, 192.168.10.46
   Discovered by: 192.168.10.45, 192.168.10.46
   
   # /opt/oracle.SupportTools/quorumdiskmgr --list --device
   Device path: /dev/exadata_quorum/QD_DATAC1_DB01
   Host name: DB01
   ASM disk group name: DATAC1
   Size: 128 MB
   
   Device path: /dev/exadata_quorum/QD_DATAL1_DB01
   Host name: DB01
   ASM disk group name: DATAC1
   Size: 128 MB

4. Query the cluster to see if all voting disks are available.

   # crsctl query css votedisk
   ##  STATE    File Universal Id                  File Name                               Disk group
   --  -----    -----------------                  ------------------------------------    -----------
    1. ONLINE   ca2f1b57873f4ff4bf1dfb78824f2912   (o/192.168.10.42/DATAC1_CD_09_celadm12) [DATAC1]
    2. ONLINE   a8c3609a3dd44f53bf17c89429c6ebe6   (o/192.168.10.43/DATAC1_CD_09_celadm13) [DATAC1]
    3. ONLINE   4948b73db0514f47bf94ee53b98fdb51   (/dev/exadata_quorum/QD_ DATAC1_DB02)   [DATAC1]
    4. ONLINE   cafb7e95a5be4f00bf10bc094469cad9   (o/192.168.10.44/DATAC1_CD_09_celadm14) [DATAC1]
   Located 4 voting disk(s).

   Notice that there is one voting disk missing, for the recovered database server (DB01).

   If you query V$ASM_DISK, you can see that the quorum disk was offlined by the recovery process.

   SQL> SELECT label, path, mode_status, header_status, mount_status 
        FROM v$asm_disk
        WHERE path LIKE '/dev/%';
   
   LABEL                PATH                           MODE_STATUS          
   HEADER_STATUS                  MOUNT_S
   -------------------- ------------------------------ --------------------
   ------------------------------ -------
   QD_DATAC1_DB01       /dev/exadata_quorum/QD_DATAC1_ ONLINE              
   CANDIDATE                      CLOSED
   
   QD_DATAC1_DB02       /dev/exadata_quorum/QD_DATAC1_ ONLINE              
   MEMBER                         CACHED

5. Drop the unavailable quorum disk from the Oracle ASM disk group using the FORCE option.

   This step is only required if the previous step shows the quorum disk as part of the ASM disk group in the V$ASM_DISK query output.

   SQL> alter diskgroup DATA_C1 drop quorum disk QD_DATAC1_DB01 force;

6. Add the quorum disk back to the Oracle ASM disk group.

   SQL> alter diskgroup DATA_C1 add quorum failgroup DB01 
        disk '/dev/exadata_quorum/QD_DATAC1_DB01';

   If the command fails with an error indicating that the disk already belongs to the disk group, run the command with the FORCE option.

   SQL> alter diskgroup DATA_C1 add quorum failgroup DB01 
        disk '/dev/exadata_quorum/QD_DATAC1_DB01' force;

7. Requery V$ASM_DISK to verify that the quorum disks are available.

   SQL> SELECT label, path, mode_status, header_status, mount_status 
        FROM v$asm_disk
        WHERE path LIKE '/dev/%';
   
   LABEL                PATH                           MODE_STATUS          
   HEADER_STATUS                  MOUNT_S
   -------------------- ------------------------------ --------------------
   ------------------------------ -------
   QD_DATAC1_DB01       /dev/exadata_quorum/QD_DATAC1_ ONLINE              
   MEMBER                         CACHED
   
   QD_DATAC1_DB02       /dev/exadata_quorum/QD_DATAC1_ ONLINE              
   MEMBER                         CACHED

8. Query the cluster to verify all voting disks are now available.

   # crsctl query css votedisk
   ##  STATE    File Universal Id                  File Name                               Disk group
   --  -----    -----------------                  ------------------------------------    -----------
    1. ONLINE   ca2f1b57873f4ff4bf1dfb78824f2912   (o/192.168.10.42/DATAC1_CD_09_celadm12) [DATAC1]
    2. ONLINE   a8c3609a3dd44f53bf17c89429c6ebe6   (o/192.168.10.43/DATAC1_CD_09_celadm13) [DATAC1]
    3. ONLINE   4948b73db0514f47bf94ee53b98fdb51   (/dev/exadata_quorum/QD_ DATAC1_DB02)   [DATAC1]
    4. ONLINE   cafb7e95a5be4f00bf10bc094469cad9   (o/192.168.10.44/DATAC1_CD_09_celadm14) [DATAC1]
    5. ONLINE   4dca8fb7bd594f6ebf8321ac23e53434   (/dev/exadata_quorum/QD_ DATAC1_DB01)   [DATAC1]
   Located 5 voting disk(s).