Adding the Disk Expansion Kit to Database Servers: X8M-2 and Prior

2.4.2 Adding the Disk Expansion Kit to Database Servers: X8M-2 and Prior

Note the following restrictions and requirements:

The disk expansion kit is supported only on 2-socket systems starting with Oracle Exadata Database Machine X5-2.
Oracle Exadata System Software release 12.1.2.3.0 or later is required.
For systems running Oracle Linux 6 (OL6), a reboot is required for the Linux kernel to recognize a newly added disk partition.
If you are adding the disk expansion kit to an Oracle Exadata Database Machine X7-2 system, and you are using an Oracle Exadata System Software release before 18.1.11, then ensure that the following symbolic link is present on the database server before proceeding:
```
# ls -l /opt/MegaRAID/MegaCli/MegaCli64 
lrwxrwxrwx 1 root root 31 Jun  4 03:40 /opt/MegaRAID/MegaCli/MegaCli64 -> /opt/MegaRAID/storcli/storcli64
```
If the symbolic link is not present, then use the following commands to create it:
```
# mkdir -p /opt/MegaRAID/MegaCli
# ln -s /opt/MegaRAID/storcli/storcli64 /opt/MegaRAID/MegaCli/MegaCli64
```

To add the disk expansion kit to an Oracle Exadata Database Server:

Remove the plastic filler panels that cover the vacant drive bays and insert the four drives that are contained in the disk expansion kit.

The server should be powered on so that the disk controller can sense the new drives.

The drives may be installed in any order. All four drives must be installed at the same time (within 30 minutes) so that the disk controller can sense the new drives before any of them enter a power-saving mode.

When the disk controller senses the new drives, the RAID reconstruction process automatically begins.
Monitor the server alert history. Ensure that the RAID reconstruction process completes successfully before proceeding.
The RAID reconstruction process may take several hours to complete (7 hours in the following example). Look out for the clear message (message 1_2 below), which indicates that the RAID reconstruction process is completed.
```
# dbmcli -e list alerthistory

         1_1     2016-02-15T14:01:00-08:00       warning         "A disk
 expansion kit was installed. The additional physical drives were automatically
 added to the existing RAID5 configuration, and reconstruction of the
 corresponding virtual drive was automatically started."

         1_2     2016-02-15T21:01:01-08:00       clear           "Virtual drive
 reconstruction due to disk expansion was completed."
```
At the end of the RAID reconstruction process, the virtual drive at /dev/sda includes the additional storage space from the disk expansion kit.

If you are adding the disk expansion kit as part of deploying a new system, then proceed with this step. Otherwise, skip to the next step.

This section uses reclaimdisks.sh to extend the VGExaDb volume group so that it consumes the additional storage space provided by the disk expansion kit.

reclaimdisks.sh works only during initial deployment, before installation of the database software.

Run /opt/oracle.SupportTools/reclaimdisks.sh -extend-vgexadb to extend the VGExaDb volume group.

If prompted to fix the GUID Partition Table (GPT) or to continue with the current settings, enter F to fix the GPT.

For example:

# /opt/oracle.SupportTools/reclaimdisks.sh -extend-vgexadb
Model is ORACLE SERVER X6-2
Number of LSI controllers: 1
Physical disks found: 8 (252:0 252:1 252:2 252:3 252:4 252:5 252:6 252:7)
Logical drives found: 1
Linux logical drive: 0
RAID Level for the Linux logical drive: 5
Physical disks in the Linux logical drive: 8 (252:0 252:1 252:2 252:3 252:4 252:5 252:6 252:7)
Dedicated Hot Spares for the Linux logical drive: 0
Global Hot Spares: 0
Valid. Disks configuration: RAID5 from 8 disks with no global and dedicated hot spare disks.
Valid. Booted: Linux. Layout: Linux + DOM0.
[INFO     ] Size of system block device /dev/sda: 4193GB
[INFO     ] Last partition on /dev/sda ends on: 1797GB
[INFO     ] Unused space detected on the system block device: /dev/sda
[INFO     ] Label of partition table on /dev/sda: gpt
[INFO     ] Adjust the partition table to use all of the space on /dev/sda
[INFO     ] Respond to the following prompt by typing 'F'
Warning: Not all of the space available to /dev/sda appears to be used, you can fix the GPT to use all of the space (an extra 4679680000 blocks) or
continue with the current setting?
Fix/Ignore? F
Model: LSI MR9361-8i (scsi)
Disk /dev/sda: 4193GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
 
Number  Start   End     Size    File system  Name     Flags
 1      32.8kB  537MB   537MB   ext4         primary  boot
 2      537MB   123GB   122GB                primary  lvm
 3      123GB   1690GB  1567GB               primary
 4      1690GB  1797GB  107GB                primary  lvm
 
[INFO     ] Check for Linux with inactive DOM0 system disk
[INFO     ] Valid Linux with inactive DOM0 system disk is detected
[INFO     ] Number of partitions on the system device /dev/sda: 4
[INFO     ] Higher partition number on the system device /dev/sda: 4
[INFO     ] Last sector on the system device /dev/sda: 8189440000
[INFO     ] End sector of the last partition on the system device /dev/sda: 3509759000
[INFO     ] Unmount /u01 from /dev/mapper/VGExaDbOra-LVDbOra1
[INFO     ] Remove inactive system logical volume /dev/VGExaDb/LVDbSys3
[INFO     ] Remove xen files from /boot
[INFO     ] Remove logical volume /dev/VGExaDbOra/LVDbOra1
[INFO     ] Remove volume group VGExaDbOra
[INFO     ] Remove physical volume /dev/sda4
[INFO     ] Remove partition /dev/sda4
[INFO     ] Remove device /dev/sda4
[INFO     ] Remove partition /dev/sda3
[INFO     ] Remove device /dev/sda3
[INFO     ] Create primary partition 3 using 240132160 8189439966
[INFO     ] Set lvm flag for the primary partition 3 on device /dev/sda
[INFO     ] Add device /dev/sda3
[INFO     ] Primary LVM partition /dev/sda3 has size 7949307807 sectors
[INFO     ] Create physical volume on partition /dev/sda3
[INFO     ] LVM Physical Volume /dev/sda3 has size 3654340511 sectors
[INFO     ] Size of LVM physical volume less than size of device /dev/sda3
[INFO     ] Remove LVM physical volume /dev/sda3
[INFO     ] Reboot is required to apply the changes in the partition table

Examine the end of the output from the previous command. If a reboot is not required, then skip to the next substep. If a reboot a required, then reboot the server and re-run /opt/oracle.SupportTools/reclaimdisks.sh -extend-vgexadb.
For example:
```
# shutdown -r now
```
Then, after system reboot:
```
# /opt/oracle.SupportTools/reclaimdisks.sh -extend-vgexadb
```

Run /opt/oracle.SupportTools/reclaimdisks.sh with no arguments. In the output, confirm that there are no errors and that the output references the additional disks from the disk expansion kit.

# /opt/oracle.SupportTools/reclaimdisks.sh
Model is ORACLE SERVER X6-2
Number of LSI controllers: 1
Physical disks found: 8 (252:0 252:1 252:2 252:3 252:4 252:5 252:6 252:7)
Logical drives found: 1
Linux logical drive: 0
RAID Level for the Linux logical drive: 5
Physical disks in the Linux logical drive: 8 (252:0 252:1 252:2 252:3 252:4 252:5 252:6 252:7)
Dedicated Hot Spares for the Linux logical drive: 0
Global Hot Spares: 0
Valid. Disks configuration: RAID5 from 8 disks with no global and dedicated hot spare disks.
Valid. Booted: Linux. Layout: Linux.

You can now continue with deploying the system and use the additional storage space provided by the disk expansion kit. Do not perform the next step.

If you are adding the disk expansion kit to a previously deployed system, then proceed with this step.

This step uses Operating System commands to consume the additional storage space provided by the disk expansion kit.
1. Run parted to view the sector information for /dev/sda.
  If you see a request to fix the GPT, respond with F.
```
# parted /dev/sda 
GNU Parted 2.1Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit s 
(parted) print
Warning: Not all of the space available to /dev/sda appears to be used, you can
fix the GPT to use all of the space (an extra 4679680000 blocks) or continue
with the current setting? Fix/Ignore? F  

Model: LSI MR9361-8i (scsi) 
Disk /dev/sda: 8189440000s 
Sector size (logical/physical): 512B/512B 
Partition Table: gpt 

Number  Start       End           Size         File system  Name     Flags 
1       64s         1046591s      1046528s     ext3         primary  boot 
4       1046592s    1048639s      2048s                     primary  bios_grub
2       1048640s    240132159s    239083520s                primary  lvm 

(parted) q
```
  Examine the output and note the disk size. Note also the largest end sector value, which should be the end sector of the last partition. In the preceding example, the disk size is 8189440000 sectors, and the largest end sector value is 240132159. You will use these values in the next step.
2. Create a new partition in /dev/sda.
  
  The command requires a start sector and an end sector, which you must derive from the values that you noted previously.
  
  For the start sector, add 1 to largest end sector value from the previous step. For example: 240132159 + 1 = 240132160.
  
  For the end sector, subtract 34 from the disk size value. For example: 8189440000 - 34 = 8189439966.
```
# parted -s /dev/sda mkpart primary 240132160s 8189439966s
```
  This command produces no output.
3. Review the updated partition table and take note of the partition number for the new partition.
  In this example, the new partition number is 3. You will use this value in the following commands.
```
# parted -s /dev/sda unit s print
Model: LSI MR9361-8i (scsi)
Disk /dev/sda: 8189440000s
Sector size (logical/physical): 512B/512B
Partition Table: gpt 
Number  Start        End          Size         File system  Name     Flags
1       64s         1046591s      1046528s     ext4         primary  boot 
4       1046592s    1048639s      2048s                     primary  bios_grub
2       1048640s    240132159s    239083520s                primary  lvm 
3       240132160s  8189439966s   7949307807s               primary  
```
4. Set the LVM flag for the new partition.
  
  In this example, the new partition number is 3. Use the partition number that you observed in the previous step.
```
# parted -s /dev/sda set 3 lvm on
```
5. Examine the output from the previous step, and reboot the server if required.
  For systems running Oracle Linux 6 (OL6), a server reboot is required for the Linux kernel to recognize the newly added disk partition.
  
  Reboot the server if the output from the previous step contains a message similar to the following:
```
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or
 resource busy).  As a result, it may not reflect all of your changes until after reboot.
```
6. Create an LVM physical volume (PV) on the newly created partition.
  
  In this example, the new partition number is 3, so the physical volume is /dev/sda3. Adjust the command based on the partition number that you observed previously.
```
# lvm pvcreate --force /dev/sda3
  Physical volume "/dev/sda3" successfully created
```
7. Extend the LVM volume group VGExaDb to use the newly created physical volume.
```
# lvm vgextend VGExaDb /dev/sda3
  Volume group "VGExaDb" successfully extended
```
You can now use the additional storage space provided by the disk expansion kit to extend various storage volumes and file systems on the server.