Expanding /EXAVMIMAGES on Management Domain on Releases Earlier than 12.2.x

5.14.3 Expanding /EXAVMIMAGES on Management Domain on Releases Earlier than 12.2.x

If you are using a release of Oracle Exadata System Software starting with release 12.1.2.3.0 but earlier than release 12.2.x, then use this procedure to expand the /EXAVMIMAGES directory on the management domain following the addition of a disk expansion kit.

During deployment, all available disk space on a database server will be allocated in the management domain with the majority of the space allocated to /EXAVMIMAGES for user domain storage. The /EXAVMIMAGES file system is created on /dev/sda3.

In the example below, dm01db01 is the name of the management domain, and dm01db01vm01 is a user domain.

Ensure reclaimdisks.sh has been run by using the -check option.

Note that the last line reads "Layout: DOM0". If reclaimdisks.sh was not run, it would read "Layout: DOM0 + Linux".

[root@dm01db01 ~]# /opt/oracle.SupportTools/reclaimdisks.sh -check
Model is ORACLE SERVER X5-2
Number of LSI controllers: 1
Physical disks found: 4 (252:0 252:1 252:2 252:3)
Logical drives found: 1
Linux logical drive: 0
RAID Level for the Linux logical drive: 5
Physical disks in the Linux logical drive: 4 (252:0 252:1 252:2 252:3)
Dedicated Hot Spares for the Linux logical drive: 0
Global Hot Spares: 0
Valid. Disks configuration: RAID5 from 4 disks with no global and dedicated hot spare disks.
Valid. Booted: DOM0. Layout: DOM0.

Add the disk expansion kit to the database server.
The kit consists of 4 additional hard drives to be installed in the 4 available slots. Remove the filler panels and install the drives. The drives may be installed in any order.

Verify that the RAID reconstruction is completed by seeing the warning and clear messages in the alert history.

This may take several hours to complete. The example below shows that it took approximately 7 hours. Once the clear message (message 1_2 below) is present, the reconstruction is completed and it is safe to proceed.

[root@dm01db01 ~]# dbmcli -e list alerthistory

         1_1     2016-02-15T14:01:00-08:00       warning         "A disk
 expansion kit was installed. The additional physical drives were automatically
 added to the existing RAID5 configuration, and reconstruction of the
 corresponding virtual drive was automatically started."

         1_2     2016-02-15T21:01:01-08:00       clear           "Virtual drive
 reconstruction due to disk expansion was completed."

Collect information about the current environment.

[root@dm01db01 ~]# cat /proc/partitions |grep sda
   8        0 4094720000 sda
   8        1     524288 sda1
   8        2  119541760 sda2
   8        3 1634813903 sda3

[root@dm01db01 ~]# df -h /EXAVMIMAGES
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             1.6T   44G  1.5T   3% /EXAVMIMAGES

[root@dm01db01 ~]# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  8192     4     r-----  94039.1
dm01db01vm01.example.com                   4 16384     2     -b----   3597.3

Stop all user domain guests by running the command xm shutdown –a –w from the management domain.

After all user domain guests are shut down, only Domain-0 (dom0) should be listed.

[root@dm01db01 ~]# xm shutdown –a -w
Domain dm01db01vm01.example.com terminated 
All domains terminated

[root@dm01db01 ~]# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  8192     4     r-----  94073.4

Run parted to verify the partition size.

If you see a request to fix the GPT, respond with F.

root@dm01db01 ~]# parted /dev/sda 
GNU Parted 2.1Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Warning: Not all of the space available to /dev/sda appears to be used, you can
fix the GPT to use all of the space (an extra 4679680000 blocks) or continue
with the current setting? Fix/Ignore? F  

Model: LSI MR9361-8i (scsi)
Disk /dev/sda: 4193GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      32.8kB  537MB   537MB   ext3         primary  boot 
 2      537MB   123GB   122GB                primary  lvm  
 3      123GB   1797GB  1674GB               primary       

(parted) q

The partition table shown above lists partition 3 as 1674 GB. The size of the disk that contains this partition (/dev/sda) is 4193 GB.

Run parted to view the sector start and end values.

root@dm01db01 ~]# parted -s /dev/sda unit s print
Model: LSI MR9361-8i (scsi) 
Disk /dev/sda: 8189440000s 
Sector size (logical/physical): 512B/512B 
Partition Table: gpt 

Number  Start       End          Size         File system  Name     Flags 
1       64s         1048639s     1048576s     ext3         primary  boot 
2       1048640s    240132159s   239083520s                primary  lvm 
3       240132160s  3509759965s  3269627806s               primary

The partition table shown above lists partition 3 as starting at sector 240132160 and disk size as 8189440000. You will use these values in step 10.

Dismount the /EXAVMIMAGES file system on the management domain.
```
[root@dm01db01 ~]# umount /EXAVMIMAGES
```
Remove partition 3.
```
[root@dm01db01 ~]# parted -s /dev/sda rm 3
```
This command produces no output.
Re-create the partition specifying the same starting sector and the new end of the partition sector. The new end of partition sector is calculated by subtracting 34 from the disk size, for example: 8189440000 - 34 = 8189439966
```
[root@dm01db01 ~]# parted -s /dev/sda mkpart primary 240132160s 8189439966s 
```
You might encounter the following warning:
```
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda 
(Device or resource busy).  As a result, it may not reflect all of your changes until after 
reboot.
```
If you encounter this error, restart the Exadata database server to apply the changes in the partition table.

Mount the /EXAVMIMAGES partition again and view the file system disk space usage for this partition.

[root@dm01db01 ~]# mount /EXAVMIMAGES

[root@dm01db01 ~]# df -h /EXAVMIMAGES
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             1.6T   44G  1.5T   3% /EXAVMIMAGES

Note that the size of the file system is still the same, 1.6 TB, as in step 4.

Verify that the partition table as seen by the kernel shows the updated size for partition 3.

The output for sda3 should now be larger compared to the output observed earlier in step 4.
```
[root@dm01db01 ~]# cat /proc/partitions |grep sda
   8        0 4094720000 sda
   8        1     524288 sda1
   8        2  119541760 sda2
   8        3 3974653903 sda3
```
Expand the file system.

You can do this while the file system is mounted and processes are running. Note the updated file system size, compared to the value in step 4. The tunefs.ocfs2 command typically runs very quickly and should have no output normally.
```
[root@dm01db01 ~]# tunefs.ocfs2 -S /dev/sda3

[root@dm01db01 ~]# df -h /EXAVMIMAGES
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             3.8T   44G  3.7T   2% /EXAVMIMAGES
```
Restart the user domains.