Expanding /EXAVMIMAGES on Management Domain on Release 18.1.x or Later

5.14.1 Expanding /EXAVMIMAGES on Management Domain on Release 18.1.x or Later

If you are using a release of Oracle Exadata System Software release 18c (18.1.0) or later, then use this procedure to expand the /EXAVMIMAGES file system on the management domain following the addition of a disk expansion kit.

During deployment, all available disk space on a database server will be allocated in the management domain (dom0) with the majority of the space allocated to /EXAVMIMAGES for user domain storage. The /EXAVMIMAGES file system is created on /dev/VGExaDb/LVDbExaVMImages.

In the example below, dm01db01 is the name of the management domain, and dm01db01vm01 is a user domain.

Ensure reclaimdisks.sh has been run in the management domain by using the -check option.

Note that the last line reads "Layout: DOM0". If reclaimdisks.sh was not run, it would read "Layout: DOM0 + Linux".

[root@dm01db01 ~]# /opt/oracle.SupportTools/reclaimdisks.sh -check
Model is ORACLE SERVER X6-2
Number of LSI controllers: 1
Physical disks found: 4 (252:0 252:1 252:2 252:3)
Logical drives found: 1
Linux logical drive: 0
RAID Level for the Linux logical drive: 5
Physical disks in the Linux logical drive: 4 (252:0 252:1 252:2 252:3)
Dedicated Hot Spares for the Linux logical drive: 0
Global Hot Spares: 0
Valid. Disks configuration: RAID5 from 4 disks with no global and dedicated hot spare disks.
Valid. Booted: DOM0. Layout: DOM0.

Add the disk expansion kit to the database server.
The kit consists of 4 additional hard drives to be installed in the 4 available slots. Remove the filler panels and install the drives. The drives may be installed in any order.

Verify that the RAID reconstruction is completed by seeing the warning and clear messages in the alert history.

This may take several hours to complete. The example below shows that it took approximately 7 hours. Once the clear message (message 1_2 below) is present, the reconstruction is completed and it is safe to proceed.

[root@dm01db01 ~]# dbmcli -e list alerthistory

         1_1     2016-02-15T14:01:00-08:00       warning         "A disk
 expansion kit was installed. The additional physical drives were automatically
 added to the existing RAID5 configuration, and reconstruction of the
 corresponding virtual drive was automatically started."

         1_2     2016-02-15T21:01:01-08:00       clear           "Virtual drive
 reconstruction due to disk expansion was completed."

Collect information about the current environment.

[root@dm01db01 ~]# df -h /EXAVMIMAGES
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             1.6T   44G  1.5T   3% /EXAVMIMAGES

[root@dm01db01 ~]# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  8192     4     r-----  94039.1
dm01db01vm01.example.com                   4 16384     2     -b----   3597.3

Stop all user domains by running the command xm shutdown –a –w from the management domain.

After all user domains are shut down, only Domain-0 (the management domain) should be listed.

[root@dm01db01 ~]# xm shutdown –a -w
Domain dm01db01vm01.example.com terminated 
All domains terminated

[root@dm01db01 ~]# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  8192     4     r-----  94073.4

Run parted to view the sector start and end values.

Check the size of the disk against the end of the third partition. If you see a request to fix the GPT, respond with F.

root@dm01db01 ~]# parted /dev/sda 
GNU Parted 2.1Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit s 
(parted) print
Warning: Not all of the space available to /dev/sda appears to be used, you can
fix the GPT to use all of the space (an extra 4679680000 blocks) or continue
with the current setting? Fix/Ignore? F  

Model: LSI MR9361-8i (scsi) 
Disk /dev/sda: 8189440000s 
Sector size (logical/physical): 512B/512B 
Partition Table: gpt 

Number  Start       End           Size         File system  Name     Flags 
1       64s         1046591s      1046528s     ext3         primary  boot 
4       1046592s    1048639s      2048s                     primary  bios_grub
2       1048640s    240132159s    239083520s                primary  lvm 

(parted) q

The partition table shown above lists partition 2 as ending at sector 240132159 and disk size as 8189440000 sectors. You will use these values in step 7.

Create a fourth partition.
The start sector is the end of the third partition from step 6 plus 1 sector (240132159+1=240132160). The end sector of the fourth partition is the size of the disk minus 34 (8189440000-34=8189439966).
```
[root@dm01db01 ~]# parted -s /dev/sda mkpart primary 240132160s 8189439966s 
```
This command produces no output.

Set the LVM flag for the fourth partition.

[root@dm01db01 ~]# parted -s /dev/sda set 3 lvm on
Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or
 resource busy).  As a result, it may not reflect all of your changes until after reboot.

Review the updated partition table.

[root@dm01db01 ~]# parted -s /dev/sda unit s print
Model: LSI MR9361-8i (scsi)
Disk /dev/sda: 8189440000s
Sector size (logical/physical): 512B/512B
Partition Table: gpt 
Number  Start        End          Size         File system  Name     Flags
1       64s         1046591s      1046528s     ext4         primary  boot 
4       1046592s    1048639s      2048s                     primary  bios_grub
2       1048640s    240132159s    239083520s                primary  lvm 
3       240132160s  8189439966s   7949307807s               primary  lvm

Restart the Exadata server.
```
[root@dm01db01 ~]# shutdown -r now
```

Check the size of the disk against the end of the fourth partition.

[root@dm01db01 ~]# parted -s /dev/sda unit s print
Model: LSI MR9361-8i (scsi)
Disk /dev/sda: 8189440000s
Sector size (logical/physical): 512B/512B
Partition Table: gpt 
Number  Start        End          Size         File system  Name     Flags
1       64s          1048639s     1048576s     ext4         primary  boot 
4       1048640s     3509759966s  3508711327s               primary  lvm 
2       3509759967s  8189439966s  4679680000s               primary  lvm
3

Create a LVM physical volume (PV) on the newly created fourth partition.

[root@dm01db01 ~]# lvm pvcreate --force /dev/sda3
  Physical volume "/dev/sda3" successfully created

Extend the LVM volume group VGExaDb to the newly created third partition.

[root@dm01db01 ~]# lvm vgextend VGExaDb /dev/sda3
  Volume group "VGExaDb" successfully extended

Dismount the /EXAVMIMAGES OCFS2 partition.
```
[root@dm01db01 ~]# umount /EXAVMIMAGES/
```

Extend the logical volume that contains the OCFS2 partition to include the rest of the free space.

[root@dm01db01 ~]# lvm lvextend -l +100%FREE /dev/VGExaDb/LVDbExaVMImages
Size of logical volume VGExaDb/LVDbExaVMImages changed from 1.55 TiB (406549 extents) to 
3.73 TiB (977798 extents).  
Logical volume LVDbExaVMImages successfully resized.

Resize the OCFS2 file system to the rest of the logical volume.

The tunefs.ocfs2 command typically runs very quickly and does not produce output.
```
[root@dm01db01 ~]# tunefs.ocfs2 -S /dev/VGExaDb/LVDbExaVMImages
```

Mount the OCFS2 partition and then view the file system disk space usage for this partition.

[root@dm01db01 ~]# mount -a

[root@dm01db01 ~]# ls -al /EXAVMIMAGES/
total 4518924
drwxr-xr-x  3 root root        3896 Jul 18 18:01 .
drwxr-xr-x 26 root root        4096 Jul 24 14:50 ..
drwxr-xr-x  2 root root        3896 Jul 18 17:51 lost+found
-rw-r-----  1 root root 26843545600 Jul 18 18:01 System.first.boot.12.2.1.1.8.180510.1.img

[root@dm01db01 ~]# df -h /EXAVMIMAGES/
Filesystem                            Size  Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbExaVMImages   3.8T  9.0G  3.8T   1% /EXAVMIMAGES

Restart the user domains.