6 Managing Oracle Linux KVM Guests

Starting with Oracle Exadata Database Machine X8M-2, Oracle Linux KVM is the virtualization technology for systems that use RoCE Network Fabric.

6.1 Oracle Linux KVM and Oracle Exadata Database Machine

When deploying Oracle Exadata Database Machine X8M-2, you can decide to implement Oracle Linux KVM on the database servers.

A KVM host and one or more guests are installed on every database server. You can configure Oracle Linux KVM environments on your initial deployment using scripts created by Oracle Exadata Deployment Assistant (OEDA) or you can migrate an existing environment to Oracle Linux KVM.

Note:

Oracle Linux KVM is not supported on 8-socket servers, such as X8M-8.

6.1.1 About Oracle Linux KVM

Oracle Linux KVM enables you to deploy the Oracle Linux operating system and application software within a supported virtual environment that is managed by KVM.

If you use Oracle Linux KVM on Oracle Exadata Database Machine, then Oracle Linux KVM provides CPU, memory, operating system, and system administration isolation for your workloads. You can combine virtual machines (VMs) with network and I/O prioritization to achieve full stack isolation. For consolidation, you can create multiple trusted databases or pluggable databases using Oracle Linux KVM, allowing resources to be shared more dynamically.

Starting with Oracle Exadata System Software release 19.3.0, KVM is the virtualization technology used with Oracle Exadata Database Machine systems configured with RDMA over Converged Ethernet (RoCE) interconnects. An Oracle Linux KVM environment consists of a management server (the KVM host), virtual machines, and resources. A KVM host is a managed virtual environment providing a lightweight, secure, server platform which runs VMs, also known as guests.

The KVM host is installed on a bare metal computer. The hypervisor on each KVM host is an extremely small-footprint VM manager and scheduler. It is designed so that it is the only fully privileged entity in the system. It controls only the most basic resources of the system, including CPU and memory usage, privilege checks, and hardware interrupts.

The hypervisor securely executes multiple VMs on one host computer. Each VM runs in its own guest and has its own operating system. The KVM host also runs as a guest on top of the hypervisor. The KVM host has privileged access to the hardware and device drivers. This is the environment from where you manage all the guests.

A guest is an unprivileged VM that can access the RoCE interface. The guest is started and managed on the KVM host. Because a guest operates independently of other VMs, a configuration change applied to the virtual resources of a guest does not affect any other guests. A failure of the guest does not impact any other guests.

The terms "guest", "domain", and "virtual machine" are often used interchangeably.

In general, each KVM host supports up to 12 guests. However, the limit is 8 guests on servers that contain 384 GB of RAM and are configured to support Exadata Secure RDMA Fabric Isolation.

Each guest is started alongside the KVM host. The guests never interact with the KVM host directly. Their requirements are handled by the hypervisor itself. The KVM host only provides a means to administer the hypervisor.

You use Oracle Exadata Deployment Assistant (OEDA) to create and configure Oracle Linux KVMs on Oracle Exadata Database Machine.

6.1.2 Oracle Linux KVM Deployment Specifications and Limits

This topic describes the deployment specifications and limits for using Oracle Linux KVM on Oracle Exadata Database Machine.

Table 6-1 Oracle Linux KVM Deployment Specifications and Limits

Attribute Value for X8M-2

Maximum number of Oracle Linux KVM guests on each database server

On servers that contain 384 GB of RAM and are configured to support Exadata Secure RDMA Fabric Isolation: 8

Otherwise: 12

Total physical memory on each database server

Default: 384 GB

Maximum: 1.5 TB

Memory for each Oracle Linux KVM guest

Minimum: 16 GB

Maximum: 1390 GB

Total CPU cores (vCPUs) on each database server

48 (96)

CPU cores (vCPUs) for each Oracle Linux KVM guest

Minimum: 2 (4)

Maximum: 46 (92)

Total usable disk storage for Oracle Linux KVM guests on each database server

3.15 TB

Note:

1 CPU core = 1 OCPU = 2 vCPUs = 2 hyper-threads

6.1.3 Supported Operations in the KVMHost

Manually modifying the KVM host can result in configuration issues, which can degrade performance or cause a loss of service.

Caution:

Oracle does not support any changes that are made to the KVM host beyond what is documented. Third-party applications can be installed on the KVM host and guests, but if there are issues with the Oracle software, then Oracle Support Services may request the removal of the third-party software while troubleshooting the cause.

If you are in doubt whether an operation on the KVM host is supported, contact Oracle Support Services.

6.1.4 Oracle Linux KVM Resources

Two fundamental parts of the Oracle Linux KVM infrastructure – networking and storage – are configured outside of Oracle Linux KVM.

Networking

When specifying the configuration details for your Oracle Exadata Rack using Oracle Exadata Deployment Assistant (OEDA), you provide input on how the required network IP addresses for Oracle Linux KVM environments should be created. The generated OEDA setup files are transferred to the Oracle Exadata Rack and used to create the network addresses.

Storage

Oracle Linux KVM always requires a location to store environment resources that are essential to the creation and management of virtual machines (VMs). These resources include ISO files (virtual DVD images), VM configuration files and VM virtual disks. The location of such a group of resources is called a storage repository.

On Oracle Exadata Database Machine, storage for the Oracle Linux KVMs uses an XFS file system.

6.2 Migrating a Bare Metal Oracle RAC Cluster to an Oracle RAC Cluster in Oracle Linux KVM

You can move an existing Oracle Real Application Clusters (Oracle RAC) cluster into a virtual environment that is managed by KVM.

Note:

This topic applies only to two-socket x86 servers. It does not apply to eight-socket servers such as Oracle Exadata Database Machine X8M-8.

The migration of a bare metal Oracle RAC cluster to an Oracle RAC cluster in Oracle Linux KVM can be achieved in the following ways:

  • Migrate to Oracle RAC cluster in Oracle Linux KVM using the existing bare metal Oracle RAC cluster with zero downtime.

  • Migrate to Oracle RAC cluster in Oracle Linux KVM by creating a new Oracle RAC cluster in Oracle Linux KVM with minimal downtime.

  • Migrate to Oracle RAC cluster in Oracle Linux KVM using Oracle Data Guard with minimal downtime.

  • Migrate to Oracle RAC cluster in Oracle Linux KVM using Oracle Recovery Manager (RMAN) backup and restore with complete downtime.

The conversion of a bare metal Oracle RAC cluster to an Oracle RAC cluster in Oracle Linux KVM has the following implications:

  • Each of the database servers will be converted to an Oracle Linux KVM server on which a KVM host is created along with one or more guests, depending on the number of Oracle RAC clusters being deployed. Each guest on a database server will belong to a particular Oracle RAC cluster.

  • As part of the conversion procedure, the bare metal Oracle RAC cluster will be converted to one Oracle RAC cluster in Oracle Linux KVM to start with. There will be one guest per database server.

  • At the end of the conversion, the cell disk and grid disk configuration of the storage cells are the same as they were at the beginning of the conversion.

  • The KVM host will use a small portion of the system resources on each database server. Typically a KVM host uses 16 GB or 6% of the available machine RAM, whichever is more. A KVM host also uses 4 virtual CPUs. These resource requirements have to be taken into consideration when sizing the SGA of the databases running on the Oracle RAC cluster in Oracle Linux KVM.

  • Refer to My Oracle Support note 2099488.1 for the complete instructions.

6.3 Showing Running Domains

Use the vm_maker utility to list the running domains.

  1. Connect to the management domain.
  2. Run the command /opt/exadata_ovm/vm_maker --list-domains to list the domains.
    # /opt/exadata_ovm/vm_maker --list-domains
    dm01db01vm01.example.com(55)      : running
    dm01db01vm02.example.com(57)      : running
    dm01db01vm03.example.com(59)      : running

    To view memory or CPU distribution for the domains, there are separate commands:

    • /opt/exadata_ovm/vm_maker --list --memory
    • /opt/exadata_ovm/vm_maker --list --vcpu

6.4 Starting a Guest

You can start a guest manually, or configure the guest to start automatically when the KVM host is started.

  1. Connect to the KVM host.
  2. To manually start a guest, use vm_maker to start the guest.

    In the following example, db01_guest01.example.com is the name of the guest.

    # /opt/exadata_ovm/vm_maker --start-domain db01_guest01.example.com
    [INFO] Running 'virsh start db01_guest01.example.com...
    Domain db01_guest01.example.com started
    
    [INFO] The domain has been started but may not have network connectivity for 
    several minutes.
  3. To configure autostart for a guest, use the vm_maker --autostart command.

    In the following example, db01_guest01.example.com is the name of the guest.

    # /opt/exadata_ovm/vm_maker --autostart db01_guest01.example.com --enable
    [INFO] Running 'virsh autostart db01_guest01.example.com'...
    Domain db01_guest01.example.com marked as autostarted

6.5 Monitoring a Guest Console During Startup

To see Oracle Linux boot messages during guest startup, use the --console option with the vm_maker --start-domain command.

  1. Connect as the root user to the KVM host.
  2. Obtain the guest name using the /opt/exadata_ovm/vm_maker --list-domains command.
  3. Use the following command to attach to the guest console, as part of starting the guest:

    In the following command, GuestName is the name of the guest.

    # vm_maker --start-domain GuestName --console
  4. Press CTRL+] to disconnect from the console.

6.6 Managing Automatic Startup of Oracle Linux KVM Guests

By default, when you create a guest, it is configured to automatically start when the KVM host is started. You can enable and disable this feature as needed.

6.6.1 Enabling Guest Automatic Start

You can configure a guest to automatically start when the KVM host is started.

  1. Connect to the KVM host.
  2. Use vm_maker to enable autostart for the guest.

    In the following example, db01_guest01.example.com is the name of the guest.

    # /opt/exadata_ovm/vm_maker --autostart db01_guest01.example.com --enable
    [INFO] Running 'virsh autostart db01_guest01.example.com --enable'...
    Domain db01_guest01.example.com marked as autostarted

6.6.2 Disabling Guest Automatic Start

You can disable a guest from automatically starting when the KVM host is started.

  1. Connect to the KVM host.
  2. Use vm_maker to disable autostart for the guest.

    In the following example, db01_guest01.example.com is the name of the guest.

    # /opt/exadata_ovm/vm_maker --autostart db01_guest01.example.com --disable
    [INFO] Running 'virsh autostart db01_guest01.example.com --disable'...
    Domain db01_guest01.example.com unmarked as autostarted

6.7 Shutting Down a User Domain From Within the User Domain

The following procedure describes how to shut down a user domain from within a user domain:

  1. Connect as the root user to the user domain.
  2. Use the following command to shut down the domain:
    # shutdown -h now
    

6.8 Shutting Down a Guest From Within the KVM host

You can shut down a guest from within a KVM host.

  1. Connect as the root user to the KVM host.
  2. Use the following command to shut down the guest, where GuestName is the name of the guest:
    # /opt/exadata_ovm/vm_maker --stop-domain GuestName

    To shut down all guests within the KVM host, use the following command:

    # /opt/exadata_ovm/vm_maker --stop-domain --all

    The following is an example of the output:

    [INFO] Running 'virsh shutdown db01_guest01.example.com'...
    Domain db01_guest01.example.com is being shutdown

6.9 Backing up the KVM host and Guests

In an Oracle Linux KVM deployment, you need to back up the KVM host and the guests.

Backups are required to restore and recover from a database server physical or logical data issue where you need to restore database server operating system files.

6.9.1 Backing up the KVM host Using Snapshot-Based Backup

This procedure describes how to take a snapshot-based backup of the KVM host.

The logical volume /dev/VGExaDb/LVDoNotRemoveOrUse is a placeholder to make sure there is always free space available to create a snapshot. If you run dbserver_backup.sh, then the placeholder logical volume is removed by the script, the free space is used for creating a snapshot, and the logical volume is re-created after the snapshot is backed up and removed. If you use the following manual procedure, then you must perform these tasks manually.

The values shown in the following steps are examples and you may need to substantiate different values to match your situation.

All steps must be performed as the root user.

  1. Prepare a destination to hold the backup.

    The destination should reside outside of the local machine, such as a writable NFS location, and be large enough to hold the backup tar files. For non-customized partitions, the space needed for holding the backup is approximately 50 GB.

    You can use the following commands to prepare a backup destination using NFS.

    # mkdir -p /remote_FS
    # mount -t nfs -o rw,intr,soft,proto=tcp,nolock ip_address:/nfs_location/ /remote_FS

    In the mount command, ip_address is the IP address of the NFS server, and nfs_location is the NFS location holding the backups.

  2. Remove the LVDoNotRemoveOrUse logical volume.

    You can use the following script to check for the existence of the LVDoNotRemoveOrUse logical volume and remove it if present.

    lvm lvdisplay --ignorelockingfailure /dev/VGExaDb/LVDoNotRemoveOrUse
    if [ $? -eq 0 ]; then
      # LVDoNotRemoveOrUse logical volume exists.
      lvm lvremove -f /dev/VGExaDb/LVDoNotRemoveOrUse
      if [ $? -ne 0 ]; then
        echo "Unable to remove logical volume: LVDoNotRemoveOrUse. Unable to proceed with backup"
      fi
    fi

    If the LVDoNotRemoveOrUse logical volume does not exist, then do not proceed with the remaining steps and determine the reason.

  3. Determine the current system partition by examining the output from the imageinfo command.

    For example:

    # imageinfo -sys
    /dev/mapper/VGExaDb-LVDbSys1

    There are two system partitions, LVDbSys1 and LVDbSys2. One partition is active and mounted (LVDbSys1 in the example). The other partition is inactive and used as a backup location during upgrade.

  4. Create a snapshot for the logical volume hosting the / (root) file system and back it up.

    In the example, the snapshot is named LVDbSys1_snap. The example also assumes that LVDbSys1 is the active logical volume. Substitute LVDbSys2 if that is the active logical volume.

    1. Create the snapshot.
      # lvcreate -L1G -s -n LVDbSys1_snap /dev/VGExaDb/LVDbSys1
    2. Label the snapshot.
      # xfs_admin -L DBSYS_SNAP /dev/VGExaDb/LVDbSys1_snap
    3. Mount the snapshot.
      # mkdir -p /root/mnt/sys
      # mount -o nouuid /dev/VGExaDb/LVDbSys1_snap /root/mnt/sys
    4. Change to the directory for the backup.
      # cd /root/mnt/sys
    5. Create the backup file, which contains the root file system.
      # tar -pjcvf /remote_FS/rootfs-boot.tar.bz2 * /boot > /tmp/backup_tar.stdout 2> /tmp/backup_tar.stderr
    6. Check the /tmp/backup_tar.stderr file for any significant errors.

      You can ignore errors about failing to tar open sockets, and other similar errors.

    7. Unmount and remove the snapshot.
      # cd /
      # umount /root/mnt/sys
      # /bin/rmdir /root/mnt/sys
      # lvremove -f /dev/VGExaDb/LVDbSys1_snap
  5. Create a snapshot for the logical volume hosting the /var file system and back it up.

    In the example, the snapshot is named LVDbVar1_snap. The example also assumes that LVDbVar1 is the active logical volume. If LVDbSys2 is the current active system partition, then use LVDbVar2.

    1. Create the snapshot.
      # lvcreate -L1G -s -n LVDbVar1_snap /dev/VGExaDb/LVDbVar1
    2. Label the snapshot.
      # xfs_admin -L VAR_SNAP /dev/VGExaDb/LVDbVar1_snap
    3. Mount the snapshot.
      # mkdir -p /root/mnt/var
      # mount -o nouuid /dev/VGExaDb/LVDbVar1_snap /root/mnt/var
    4. Change to the directory for the backup.
      # cd /root/mnt/var
    5. Create the backup file which will contain the /var file system.
      # tar -pjcvf /remote_FS/var.tar.bz2 * > /tmp/var_tar.stdout 2>
       /tmp/var_tar.stderr
    6. Check the /tmp/var_tar.stderr file for any significant errors.

      You can ignore errors about failing to tar open sockets, and other similar errors.

    7. Unmount and remove the snapshot.
      # cd /
      # umount /root/mnt/var
      # /bin/rmdir /root/mnt/var
      # lvremove -f /dev/VGExaDb/LVDbVar1_snap
  6. Create a snapshot for the logical volume hosting the /var/log file system and back it up.

    The logical volume hosting the /var/log file system is /dev/VGExaDb/LVDbVarLog.

    In the example, the snapshot is named LVDbVarLog_snap.

    1. Create the snapshot.
      # lvcreate -L1G -s -n LVDbVarLog_snap /dev/VGExaDb/LVDbVarLog
    2. Label the snapshot.
      # xfs_admin -L VARLOG_SNAP /dev/VGExaDb/LVDbVarLog_snap
    3. Mount the snapshot.
      # mkdir -p /root/mnt/varlog
      # mount -o nouuid /dev/VGExaDb/LVDbVarLog_snap /root/mnt/varlog
    4. Change to the directory for the backup.
      # cd /root/mnt/varlog
    5. Create the backup file which will contain the /var/log file system.
      # tar -pjcvf /remote_FS/varlog.tar.bz2 * > /tmp/varlog_tar.stdout 2> /tmp/varlog_tar.stderr
    6. Check the /tmp/varlog_tar.stderr file for any significant errors.

      You can ignore errors about failing to tar open sockets, and other similar errors.

    7. Unmount and remove the snapshot.
      # cd /
      # umount /root/mnt/varlog
      # /bin/rmdir /root/mnt/varlog
      # lvremove -f /dev/VGExaDb/LVDbVarLog_snap
  7. Create a snapshot for the logical volume hosting the /var/log/audit file system and back it up.

    The logical volume hosting the /var/log/audit file system is /dev/VGExaDb/LVDbVarLogAudit.

    In the example, the snapshot is named LVDbVarLogAudit_snap.

    1. Create the snapshot.
      # lvcreate -L1G -s -n LVDbVarLogAudit_snap /dev/VGExaDb/LVDbVarLogAudit
    2. Label the snapshot.
      # xfs_admin -L VARLOGAUDIT_SNAP /dev/VGExaDb/LVDbVarLogAudit_snap
    3. Mount the snapshot.
      # mkdir -p /root/mnt/varlogaudit
      # mount -o nouuid /dev/VGExaDb/LVDbVarLogAudit_snap /root/mnt/varlogaudit
    4. Change to the directory for the backup.
      # cd /root/mnt/varlogaudit
    5. Create the backup file which will contain the /var/log/audit file system.
      # tar -pjcvf /remote_FS/varlogaudit.tar.bz2 * > /tmp/varlogaudit_tar.stdout 2> /tmp/varlogaudit_tar.stderr
    6. Check the /tmp/varlogaudit_tar.stderr file for any significant errors.

      You can ignore errors about failing to tar open sockets, and other similar errors.

    7. Unmount and remove the snapshot.
      # cd /
      # umount /root/mnt/varlogaudit
      # /bin/rmdir /root/mnt/varlogaudit
      # lvremove -f /dev/VGExaDb/LVDbVarLogAudit_snap
  8. Create a snapshot for the logical volume hosting the /home file system and back it up.

    The logical volume hosting the /home file system is /dev/VGExaDb/LVDbHome.

    In the example, the snapshot is named LVDbHome_snap.

    1. Create the snapshot.
      # lvcreate -L1G -s -n LVDbHome_snap /dev/VGExaDb/LVDbHome
    2. Label the snapshot.
      # xfs_admin -L LVDbHome_SNAP /dev/VGExaDb/LVDbHome_snap
    3. Mount the snapshot.
      # mkdir -p /root/mnt/home
      # mount -o nouuid /dev/VGExaDb/LVDbHome_snap /root/mnt/home
    4. Change to the directory for the backup.
      # cd /root/mnt/home
    5. Create the backup file which will contain the /home file system.
      # tar -pjcvf /remote_FS/home.tar.bz2 * > /tmp/home_tar.stdout 2> /tmp/home_tar.stderr
    6. Check the /tmp/home_tar.stderr file for any significant errors.

      You can ignore errors about failing to tar open sockets, and other similar errors.

    7. Unmount and remove the snapshot.
      # cd /
      # umount /root/mnt/home
      # /bin/rmdir /root/mnt/home
      # lvremove -f /dev/VGExaDb/LVDbHome_snap
  9. Unmount the NFS share.
    # umount /remote_FS
  10. Recreate the /dev/VGExaDb/LVDoNotRemoveOrUse logical volume.
    # lvm lvcreate -n LVDoNotRemoveOrUse -L2G VGExaDb -y

6.9.2 Backing up the Oracle Linux KVM Guests

You can back up Oracle Linux KVM guests by using the following procedures.

There are three ways to back up the guests:

Table 6-2 Oracle Linux KVM guest backup approaches

Method Description Managed By Best For
Method 1: Back Up All of the KVM Guests

From the KVM host, back up all guests in the /EXAVMIMAGES storage repository using XFS reflinks to get a consistent backup.

KVM host administrator

Recovering all guests after a compute node failure, which renders the guests unbootable.

Method 2: Back Up an Individual Guest

From the KVM host, selectively back up a guest in the /EXAVMIMAGES storage repository using XFS reflinks to get a consistent backup.

KVM host administrator

Selective recovery of a guest after a compute node failure that renders the guest unbootable but does not affect all of the other guests.

Method 3: Back Up a Guest Internally

Back up a guest using a snapshot-based backup procedure that is run inside the guest.

Guest administrator

Recovery of a guest after a failure where the guest is still bootable and allows root login. This method also enables selective recovery of specific files.

6.9.2.1 Method 1: Back Up All of the KVM Guests

You can back up all of the guests by backing up the storage repository under /EXAVMIMAGES.

The backup destination should be separate from the KVM host server, such as a writable NFS location, and be large enough to hold the backup. The space needed for the backup is proportional to the number of guests deployed on the system. The space needed for each guest backup is approximately 200 GB.

  1. Prepare the guest images.

    Use the following script to prepare the guest image backups under /EXAVMIMAGES/Backup.

    #!/bin/bash
    
    ScriptStarttime=$(date +%s)
    printf "This script is going to remove the directory /EXAVMIMAGES/Backup if it exists.  If that is not acceptable, exit the script by typing n, manually remove /EXAVMIMAGES/Backup and come back to rerun the script.  Otherwise, press y to continue :"
    read proceed
    if [[ ${proceed} == "n" ]] || [[ ${proceed} == "N" ]]
    then
      exit 0
    elif [[ ${proceed} != "n" ]] && [[ ${proceed} != "N" ]] && [[ ${proceed} != "y" ]] && [[ ${proceed} != "Y" ]]
    then
      echo "Invalid input"
      exit 1
    fi
    rm -rf /EXAVMIMAGES/Backup
    
    ## Create the Backup Directory
    
    mkdirStartTime=$(date +%s)
    find /EXAVMIMAGES -type d|grep -v 'lost+found'| awk '{print "mkdir -p /EXAVMIMAGES/Backup"$1}'|sh
    mkdir -p /EXAVMIMAGES/Backup/XML
    mkdirEndTime=$(date +%s)
    mkdirTime=$(expr ${mkdirEndTime} - ${mkdirStartTime})
    echo "Backup Directory creation time :" ${mkdirTime}" seconds"
    
    ## Create reflinks for files not in /EXAVMIMAGES/GuestImages
    
    relinkothesStartTime=$(date +%s)
    
    find /EXAVMIMAGES/ -not -path "/EXAVMIMAGES/GuestImages/*" -not -path "/EXAVMIMAGES/Backup/*" -type f|awk '{print "cp --reflink",$0,"/EXAVMIMAGES/Backup"$0}'|sh
    
    relinkothesEndTime=$(date +%s)
    reflinkothesTime=$(expr ${relinkothesEndTime} - ${relinkothesStartTime})
    
    echo "Reflink creation time for files other than in /EXAVMIMAGES/GuestImages :" ${reflinkothesTime}" seconds"
    
    cp -r /etc/libvirt/qemu/* /EXAVMIMAGES/Backup/XML
    
    for hostName in $(virsh list|egrep -v 'Id|^-'|awk '{print $2}'|sed '/^$/d')
    do
    
    ## Pause the guests
    
      PauseStartTime=$(date +%s)
      virsh suspend ${hostName}
      PauseEndTime=$(date +%s)
      PauseTime=$(expr ${PauseEndTime} - ${PauseStartTime})
      echo "SuspendTime for guest - ${hostName} :" ${PauseTime}" seconds"
    
    ## Create reflinks for all the files in /EXAVMIMAGES/GuestImages
    
      relinkStartTime=$(date +%s)
    
      find /EXAVMIMAGES/GuestImages/${hostName} -type f|awk '{print "cp --reflink", $0,"/EXAVMIMAGES/Backup"$0}'|sh
    
      relinkEndTime=$(date +%s)
      reflinkTime=$(expr ${relinkEndTime} - ${relinkStartTime})
      echo "Reflink creation time for guest - ${hostName} :" ${reflinkTime}" seconds"
    
    ## Unpause the guest
    
      unPauseStartTime=$(date +%s)
      virsh resume ${hostName}
      unPauseEndTime=$(date +%s)
      unPauseTime=$(expr ${unPauseEndTime} - ${unPauseStartTime})
      echo "ResumeTime for guest - ${hostName} :" ${unPauseTime}" seconds"
    
    done
    
    ScriptEndtime=$(date +%s)
    ScriptRunTime=$(expr ${ScriptEndtime} - ${ScriptStarttime})
    echo ScriptRunTime ${ScriptRunTime}" seconds"
  2. Create a backup of the guest images.

    Back up all of the reflink files under /EXAVMIMAGES/Backup to a remote location. The backup enables restoration if the KVM host is permanently damaged or lost.

    For example:

    # mkdir -p /remote_FS
    # mount -t nfs -o rw,intr,soft,proto=tcp,nolock ip_address:/nfs_location/ /remote_FS
    # cd /EXAVMIMAGES/Backup
    # tar -pjcvf /remote_FS/exavmimages.tar.bz2 * > /tmp/exavmimages_tar.stdout 2> /tmp/exavmimages_tar.stderr

    In the mount command, ip_address is the IP address of the NFS server, and nfs_location is the NFS location holding the backup.

    After the backup completes, check for any significant errors from the tar command. In the previous example, the tar command writes errors to the file at /tmp/exavmimages_tar.stderr.

  3. Remove the /EXAVMIMAGES/Backup directory and its contents.

    For example:

    # cd /
    # rm -rf /EXAVMIMAGES/Backup
6.9.2.2 Method 2: Back Up an Individual Guest

You can back up an individual guest by backing up its specific folder under /EXAVMIMAGES.

The backup destination should be separate from the KVM host server, such as a writable NFS location, and be large enough to hold the backup. The space needed for an individual guest backup is approximately 200 GB.

  1. Prepare the guest image.

    Use the following script to prepare the guest image backup under /EXAVMIMAGES/Backup.

    #!/bin/bash
    
    ScriptStarttime=$(date +%s)
    
    printf "This script is going to remove the directory /EXAVMIMAGES/Backup if it exists. If that is not acceptable, exit the script by typing n, manually remove /EXAVMIMAGES/Backup and come back to rerun the script.  Otherwise, press y to continue :"
    
    read proceed
    
    if [[ ${proceed} == "n" ]] || [[ ${proceed} == "N" ]]
    then
      exit 0
    elif [[ ${proceed} != "n" ]] && [[ ${proceed} != "N" ]] && [[ ${proceed} != "y" ]] && [[ ${proceed} != "Y" ]]
    then
      echo "Invalid input"
      exit 1
    fi
    
    rm -rf /EXAVMIMAGES/Backup
    
    printf "Enter the name of the KVM guest to be backed up :"
    
    read KVMGuestName
    
    ## Create the Backup Directory
    
    if [ ! -d /EXAVMIMAGES/GuestImages/${KVMGuestName} ]
    then
      echo "Guest ${KVMGuestName} does not exist"
      exit 1
    fi
    
    mkdirStartTime=$(date +%s)
    
    find /EXAVMIMAGES/GuestImages/${KVMGuestName} -type d|grep -v 'lost+found'|awk '{print "mkdir -p /EXAVMIMAGES/Backup"$1}'|sh
    
    mkdir -p /EXAVMIMAGES/Backup/XML
    
    mkdirEndTime=$(date +%s)
    mkdirTime=$(expr ${mkdirEndTime} - ${mkdirStartTime})
    echo "Backup Directory creation time :" ${mkdirTime}" seconds"
    
    cp -r /etc/libvirt/qemu/* /EXAVMIMAGES/Backup/XML
    
    ## Pause the guest
    
    PauseStartTime=$(date +%s)
    virsh suspend ${KVMGuestName}
    PauseEndTime=$(date +%s)
    PauseTime=$(expr ${PauseEndTime} - ${PauseStartTime})
    echo "PauseTime for guest - ${KVMGuestName} :" ${PauseTime}" seconds"
    
    ## Create reflinks for all the files in /EXAVMIMAGES/GuestImages/${KVMGuestName}
    
    relinkStartTime=$(date +%s)
    
    find /EXAVMIMAGES/GuestImages/${KVMGuestName} -type f|awk '{print "cp --reflink", $0,"/EXAVMIMAGES/Backup"$0}'|sh
    
    relinkEndTime=$(date +%s)
    reflinkTime=$(expr ${relinkEndTime} - ${relinkStartTime})
    echo "Reflink creation time for guest - ${KVMGuestName} :" ${reflinkTime}" seconds"
    
    ## Unpause the guest
    
    unPauseStartTime=$(date +%s)
    virsh resume ${KVMGuestName}
    unPauseEndTime=$(date +%s)
    unPauseTime=$(expr ${unPauseEndTime} - ${unPauseStartTime})
    echo "unPauseTime for guest - ${KVMGuestName} :" ${unPauseTime}" seconds"
    
    ScriptEndtime=$(date +%s)
    ScriptRunTime=$(expr ${ScriptEndtime} - ${ScriptStarttime})
    echo ScriptRunTime ${ScriptRunTime}" seconds"
  2. Create a backup of the guest image.

    Back up the reflink files under /EXAVMIMAGES/Backup to a remote location. The backup enables restoration of the specific guest if the KVM host is permanently damaged or lost.

    For example:

    # mkdir -p /remote_FS
    # mount -t nfs -o rw,intr,soft,proto=tcp,nolock ip_address:/nfs_location/ /remote_FS
    # cd /EXAVMIMAGES/Backup
    # tar -pjcvf /remote_FS/exavmimages.tar.bz2 * > /tmp/exavmimages_tar.stdout 2> /tmp/exavmimages_tar.stderr

    In the mount command, ip_address is the IP address of the NFS server, and nfs_location is the NFS location holding the backup.

    After the backup completes, check for any significant errors from the tar command. In the previous example, the tar command writes errors to the file at /tmp/exavmimages_tar.stderr.

  3. Remove the /EXAVMIMAGES/Backup directory and its contents.

    For example:

    # cd /
    # rm -rf /EXAVMIMAGES/Backup
6.9.2.3 Method 3: Back Up a Guest Internally

You can take a snapshot-based backup of a guest from inside the guest.

All steps are performed from inside the guest.

Note:

This backup method is performed internally within the guest and uses logical volume snapshots. Compared with other backup methods, this method provides more limited recovery options because the backup is only useful when the guest is bootable and allows root user login.

This procedure backs up the following:

  • primary root file system (LVDbSys1 or LVDbSys2 logical volume) and the /boot partition
  • /var file system (LVDbVar1 logical volume)
  • /var/log file system (LVDbVarLog logical volume)
  • /var/log/audit file system (LVDbVarLogAudit logical volume)
  • /u01 file system (LVDbOra1 logical volume), which includes the Oracle Grid Infrastructure and Oracle Database home directories
  • /home file system (LVDbHome1 logical volume)

The logical volume /dev/VGExaDb/LVDoNotRemoveOrUse is a placeholder to make sure there is always free space available to create a snapshot. If you run dbserver_backup.sh, then the placeholder logical volume is removed by the script, the free space is used for creating a snapshot, and the logical volume is re-created after the snapshot is backed up and removed. If you use the following manual procedure, then you must perform these tasks manually.

The values shown in the following steps are examples and you may need to substantiate different values to match your situation.

All steps must be performed as the root user.

  1. Prepare a destination to hold the backup.

    The destination should reside outside of the local machine, such as a writable NFS location, and be large enough to hold the backup tar files. For non-customized partitions, the space needed for holding the backup is approximately 60 GB.

    You can use the following commands to prepare a backup destination using NFS.

    # mkdir -p /remote_FS
    # mount -t nfs -o rw,intr,soft,proto=tcp,nolock ip_address:/nfs_location/ /remote_FS

    In the mount command, ip_address is the IP address of the NFS server, and nfs_location is the NFS location holding the backups.

  2. Remove the LVDoNotRemoveOrUse logical volume.

    You can use the following script to check for the existence of the LVDoNotRemoveOrUse logical volume and remove it if present.

    lvm lvdisplay --ignorelockingfailure /dev/VGExaDb/LVDoNotRemoveOrUse
    if [ $? -eq 0 ]; then 
      # LVDoNotRemoveOrUse logical volume exists. 
      lvm lvremove -f /dev/VGExaDb/LVDoNotRemoveOrUse 
      if [ $? -ne 0 ]; then 
        echo "Unable to remove logical volume: LVDoNotRemoveOrUse.Unable to proceed with backup" 
      fi
    fi

    If the LVDoNotRemoveOrUse logical volume does not exist, then do not proceed with the remaining steps and determine the reason.

  3. Determine the current system partition by examining the output from the imageinfo command.

    For example:

    # imageinfo -sys
    /dev/mapper/VGExaDb-LVDbSys1

    There are two system partitions, LVDbSys1 and LVDbSys2. One partition is active and mounted (LVDbSys1 in the example). The other partition is inactive and used as a backup location during upgrade.

  4. Create a snapshot for the logical volume hosting the / (root) file system and back it up.

    In the example, the snapshot is named LVDbSys1_snap. The example also assumes that LVDbSys1 is the active logical volume. Substitute LVDbSys2 if that is the active logical volume.

    1. Create the snapshot.
      # lvcreate -L1G -s -n LVDbSys1_snap /dev/VGExaDb/LVDbSys1
    2. Label the snapshot.
      # xfs_admin -L DBSYS_SNAP /dev/VGExaDb/LVDbSys1_snap
    3. Mount the snapshot.
      # mkdir -p /root/mnt/sys
      # mount -o nouuid /dev/VGExaDb/LVDbSys1_snap /root/mnt/sys
    4. Change to the directory for the backup.
      # cd /root/mnt/sys
    5. Create the backup file, which contains the root file system.
      # tar -pjcvf /remote_FS/rootfs-boot.tar.bz2 * /boot > /tmp/backup_tar.stdout 2> /tmp/backup_tar.stderr
    6. Check the /tmp/backup_tar.stderr file for any significant errors.

      You can ignore errors about failing to tar open sockets, and other similar errors.

    7. Unmount and remove the snapshot.
      # cd /
      # umount /root/mnt/sys
      # /bin/rmdir /root/mnt/sys
      # lvremove -f /dev/VGExaDb/LVDbSys1_snap
  5. Create a snapshot for the logical volume hosting the /var file system and back it up.

    The logical volume hosting the /var file system is /dev/VGExaDb/LVDbVar1.

    In the example, the snapshot is named LVDbVar1_snap.

    1. Create the snapshot.
      # lvcreate -L1G -s -n LVDbVar1_snap /dev/VGExaDb/LVDbVar1
    2. Label the snapshot.
      # xfs_admin -L VAR_SNAP /dev/VGExaDb/LVDbVar1_snap
    3. Mount the snapshot.
      # mkdir -p /root/mnt/var
      # mount -o nouuid /dev/VGExaDb/LVDbVar1_snap /root/mnt/var
    4. Change to the directory for the backup.
      # cd /root/mnt/var
    5. Create the backup file, which contains the /var file system.
      # tar -pjcvf /remote_FS/var.tar.bz2 * > /tmp/var_tar.stdout 2> /tmp/var_tar.stderr
    6. Check the /tmp/var_tar.stderr file for any significant errors.

      You can ignore errors about failing to tar open sockets, and other similar errors.

    7. Unmount and remove the snapshot.
      # cd /
      # umount /root/mnt/var
      # /bin/rmdir /root/mnt/var
      # lvremove -f /dev/VGExaDb/LVDbVar1_snap
  6. Create a snapshot for the logical volume hosting the /var/log file system and back it up.

    The logical volume hosting the /var/log file system is /dev/VGExaDb/LVDbVarLog.

    In the example, the snapshot is named LVDbVarLog_snap.

    1. Create the snapshot.
      # lvcreate -L1G -s -n LVDbVarLog_snap /dev/VGExaDb/LVDbVarLog
    2. Label the snapshot.
      # xfs_admin -L VARLOG_SNAP /dev/VGExaDb/LVDbVarLog_snap
    3. Mount the snapshot.
      # mkdir -p /root/mnt/varlog
      # mount -o nouuid /dev/VGExaDb/LVDbVarLog_snap /root/mnt/varlog
    4. Change to the directory for the backup.
      # cd /root/mnt/varlog
    5. Create the backup file, which contains the /var/log file system.
      # tar -pjcvf /remote_FS/varlog.tar.bz2 * > /tmp/varlog_tar.stdout 2> /tmp/varlog_tar.stderr
    6. Check the /tmp/varlog_tar.stderr file for any significant errors.

      You can ignore errors about failing to tar open sockets, and other similar errors.

    7. Unmount and remove the snapshot.
      # cd /
      # umount /root/mnt/varlog
      # /bin/rmdir /root/mnt/varlog
      # lvremove -f /dev/VGExaDb/LVDbVarLog_snap
  7. Create a snapshot for the logical volume hosting the /var/log/audit file system and back it up.

    The logical volume hosting the /var/log/audit file system is /dev/VGExaDb/LVDbVarLogAudit.

    In the example, the snapshot is named LVDbVarLogAudit_snap.

    1. Create the snapshot.
      # lvcreate -L1G -s -n LVDbVarLogAudit_snap /dev/VGExaDb/LVDbVarLogAudit
    2. Label the snapshot.
      # xfs_admin -L VARLOGAUDIT_SNAP /dev/VGExaDb/LVDbVarLogAudit_snap
    3. Mount the snapshot.
      # mkdir -p /root/mnt/varlogaudit
      # mount -o nouuid /dev/VGExaDb/LVDbVarLogAudit_snap /root/mnt/varlogaudit
    4. Change to the directory for the backup.
      # cd /root/mnt/varlogaudit
    5. Create the backup file, which contains the /var/log/audit file system.
      # tar -pjcvf /remote_FS/varlogaudit.tar.bz2 * > /tmp/varlogaudit_tar.stdout 2> /tmp/varlogaudit_tar.stderr
    6. Check the /tmp/varlogaudit_tar.stderr file for any significant errors.

      You can ignore errors about failing to tar open sockets, and other similar errors.

    7. Unmount and remove the snapshot.
      # cd /
      # umount /root/mnt/varlogaudit
      # /bin/rmdir /root/mnt/varlogaudit
      # lvremove -f /dev/VGExaDb/LVDbVarLogAudit_snap
  8. Create a snapshot for the logical volume hosting the /u01 file system and back it up.

    The logical volume hosting the /u01 file system is /dev/VGExaDb/LVDbOra1.

    In the example, the snapshot is named LVDbOra1_snap.

    1. Create the snapshot.
      # lvcreate -L1G -s -n LVDbOra1_snap /dev/VGExaDb/LVDbOra1
    2. Label the snapshot.
      # xfs_admin -L LVDbOra1_SNAP /dev/VGExaDb/LVDbOra1_snap
    3. Mount the snapshot.
      # mkdir -p /root/mnt/u01
      # mount -o nouuid /dev/VGExaDb/LVDbOra1_snap /root/mnt/u01
    4. Change to the directory for the backup.
      # cd /root/mnt/u01
    5. Create the backup file, which contains the /u01 file system.
      # tar -pjcvf /remote_FS/u01.tar.bz2 * > /tmp/u01_tar.stdout 2> /tmp/u01_tar.stderr
    6. Check the /tmp/u01_tar.stderr file for any significant errors.

      You can ignore errors about failing to tar open sockets, and other similar errors.

    7. Unmount and remove the snapshot.
      # cd /
      # umount /root/mnt/u01
      # /bin/rmdir /root/mnt/u01
      # lvremove -f /dev/VGExaDb/LVDbOra1_snap
  9. Create a snapshot for the logical volume hosting the /home file system and back it up.

    The logical volume hosting the /home file system is /dev/VGExaDb/LVDbHome.

    In the example, the snapshot is named LVDbHome_snap.

    1. Create the snapshot.
      # lvcreate -L1G -s -n LVDbHome_snap /dev/VGExaDb/LVDbHome
    2. Label the snapshot.
      # xfs_admin -L LVDbHome_SNAP /dev/VGExaDb/LVDbHome_snap
    3. Mount the snapshot.
      # mkdir -p /root/mnt/home
      # mount -o nouuid /dev/VGExaDb/LVDbHome_snap /root/mnt/home
    4. Change to the directory for the backup.
      # cd /root/mnt/home
    5. Create the backup file, which contains the /home file system.
      # tar -pjcvf /remote_FS/home.tar.bz2 * > /tmp/home_tar.stdout 2> /tmp/home_tar.stderr
    6. Check the /tmp/home_tar.stderr file for any significant errors.

      You can ignore errors about failing to tar open sockets, and other similar errors.

    7. Unmount and remove the snapshot.
      # cd /
      # umount /root/mnt/home
      # /bin/rmdir /root/mnt/home
      # lvremove -f /dev/VGExaDb/LVDbHome_snap
  10. Unmount the NFS share.
    # umount /remote_FS
  11. Recreate the /dev/VGExaDb/LVDoNotRemoveOrUse logical volume.
    # lvm lvcreate -n LVDoNotRemoveOrUse -L2G VGExaDb -y

6.10 Backing Up and Restoring Oracle Databases on KVM Guests

Backing up and restoring Oracle databases on KVM guests is the same as backing up and restoring Oracle databases on physical nodes.

6.11 Modifying the Memory Allocated to a Guest

You can modify the memory allocated to a guest using vm_maker.

This operation requires a guest restart. You can let vm_maker restart the guest after changing the memory configuration.

  1. Connect to the KVM host.
  2. If you are increasing the amount of memory used by the guest, then use the following command to determine the amount of free memory available:
    # /opt/exadata_ovm/vm_maker --list --memory

    In the output, the lowest value between Available memory (now) and Available memory (delayed) is the limit for free memory.

    Note:

    When assigning free memory to a guest, reserve approximately 1% to 2% of free memory for storing metadata and control structures.
  3. If you are decreasing the amount of memory used by the guest, then you must first review and adjust the memory usage of the databases running in the guest.
    1. Review the SGA size of databases and reduce if necessary.
    2. Review the huge pages operating system configuration for the databases and reduce if necessary.
    If you do not first reduce the memory requirements of the databases running in the guest, then the guest might fail to restart because too much memory is reserved for huge pages when the Oracle Linux operating system attempts to boot. See My Oracle Support Doc ID 361468.1 for details.
  4. Specify a new size for the memory.

    For example, if you want to increase the memory used to 32 GB for the db01_guest01.example.com guest, you would use the following command:

    # /opt/exadata_ovm/vm_maker --set --memory 32G --domain db01_guest01.example.com --restart-domain

    This command shuts down the guest, modifies the memory settings, and then restarts the guest.

6.12 Modifying the Number of Virtual CPUs Allocated to a Guest

You can dynamically modify the number of virtual CPUs allocated to a guest with the vm_maker --set --vcpu command.

All actions to modify the number of vCPUs allocated to a guest are performed in the KVM host.

It is possible to over-commit vCPUs such that the total number of vCPUs assigned to all guests exceeds the number of physical CPUs on the system. However, over-committing CPUs should be done only when competing workloads for oversubscribed resources are well understood and concurrent demand does not exceed physical capacity.

  1. Determine the number of physical CPUs.
    # /opt/exadata_ovm/vm_maker --list --vcpu --domain db01_guest01.example.com
  2. Modify the number of allocated vCPUs.
    The number of vCPUs must be a multiple of 2.

    For example, if you want to change the number of vCPUs allocated to 4 for the db01_guest01.example.com guest, you would use the following command:

    # /opt/exadata_ovm/vm_maker --set --vcpu 4 --domain db01_guest01.example.com

6.13 Increasing the Disk Space in a Guest

The KVM guest local space can be extended after initial deployment by adding local disk images.

6.13.1 Adding a New LVM Disk to a Guest

You can add a new LVM disk to an Oracle Linux KVM guest to increase the amount of usable disk space in a guest.

You might add an LVM disk to a guest so that the size of a file system or swap space can be increased. The system remains online while you perform this procedure.

Note:

During this procedure you perform actions in both the KVM host and in the guest.

Run all steps in this procedure as the root user.

  1. In the KVM host, verify that there is sufficient free disk space in /EXAVMIMAGES. For example:
    # df -h /EXAVMIMAGES
    Filesystem                           Size  Used Avail Use%  Mounted on
    /dev/mapper/VGExaDb-LVDbExaVMImages  1.5T   39G  1.5T   3%  /EXAVMIMAGES
  2. In the KVM host, create a new disk image and attach it to the guest.

    For example, the following command adds the disk image pv2_vgexadb.img to the guest dm01db01vm01.example.com:

    # /opt/exadata_ovm/vm_maker --create --disk-image /EXAVMIMAGES/pv2_vgexadb.img
     --attach --domain dm01db01vm01.example.com
    [INFO] Allocating an image for /EXAVMIMAGES/pv2_vgexadb.img, size 50.000000G...
    [INFO] Running 'qemu-img create /EXAVMIMAGES/pv2_vgexadb.img 50.000000G '...
    [INFO] Create label gpt on /EXAVMIMAGES/pv2_vgexadb.img.
    [INFO] Running 'parted -a none -s /EXAVMIMAGES/pv2_vgexadb.img mklabel gpt'...
    [INFO] Running 'losetup -P -f /EXAVMIMAGES/pv2_vgexadb.img'...
    [INFO] Finding loop device...
    [INFO] loop device is /dev/loop0
    [INFO] Finding number of sectors...
    [INFO] 104857600 sectors
    [INFO] Releasing loop device /dev/loop0...
    [INFO] Removing device maps for /dev/loop0...
    [INFO] Running 'kpartx -d -v /dev/loop0'...
    [INFO] Removing loop device /dev/loop0...
    [INFO] ##
    [INFO] ## Finished .
    [INFO] ##
    [INFO] Created image /EXAVMIMAGES/pv2_vgexadb.img
    [INFO] File /EXAVMIMAGES/GuestImages/dm01db01vm01.example.com/pv2_vgexadb.img is a reflink from 
    /EXAVMIMAGES/pv2_vgexadb.img and added as disk to domain dm01db01vm01.example.com
    [INFO] -------- MANUAL STEPS TO BE COMPLETED FOR MOUNTING THE DISK WITHIN DOMU dm01db01vm01
    .example.com --------
    [INFO] 1. Check a disk with name /dev/VGExaDbDisk.pv2_vgexadb.img/LVDBDisk exists.
    [INFO] - Run the command 'lvdisplay' to verify a disk with name '/dev/VGExaDbDisk.pv2_vgexadb.img/
    LVDBDisk' exists.
    [INFO] 2. Create a directory that will to be used for mounting the new disk.
    [INFO] 3. Add the following line to /etc/fstab: /dev/VGExaDbDisk.pv2_vgexadb.img/LVDBDisk <mount_
    point_from_step_2> <fstype> defaults 1 1
    [INFO] 4. Mount the newly added disk to mount point through the command: mount -a.

    Do not perform the manual steps described in the output. However, take note of the logical volume path identified in manual step number 1.

    In general, the logical volume path has the form: /dev/VolumeGroupName/LogicalVolumeName.

    In the example, the logical volume path is /dev/VGExaDbDisk.pv2_vgexadb.img/LVDBDisk.

  3. On the KVM host, list the available disk images for the guest and verify the creation of the new disk image.

    In the example in the previous step, the disk image file is identified as /EXAVMIMAGES/GuestImages/dm01db01vm01.example.com/pv2_vgexadb.img. This image should now appear in the list of disk images for the guest. For example:

    # /opt/exadata_ovm/vm_maker --list --disk-image --domain dm01db01vm01.example.com
    File /EXAVMIMAGES/GuestImages/dm01db01vm01.example.com/System.img
    File /EXAVMIMAGES/GuestImages/dm01db01vm01.example.com/grid19.2.0.0.0.img
    File /EXAVMIMAGES/GuestImages/dm01db01vm01.example.com/db19.2.0.0.0-3.img
    File /EXAVMIMAGES/GuestImages/dm01db01vm01.example.com/pv1_vgexadb.img
    File /EXAVMIMAGES/GuestImages/dm01db01vm01.example.com/pv2_vgexadb.img
  4. On the guest, identify the newly added disk.

    Use the lvdisplay command along with the logical volume path noted earlier.

    # lvdisplay /dev/VGExaDbDisk.pv2_vgexadb.img/LVDBDisk
      LV Path /dev/VGExaDbDisk.pv2_vgexadb.img/LVDBDisk
      LV Name LVDBDisk
      VG Name VGExaDbDisk.pv2_vgexadb.img
      LV UUID u3RBKF-UmCK-JQxc-iFf5-6WqS-GWAw-3nLjdn
      LV Write Access read/write
      LV Creation host, time dm01db01vm01.example.com, 2019-10-28 04:11:28 -0700
      LV Status available
      # open 0
      LV Size <50.00 GiB
      Current LE 12799
      Segments 1
      Allocation inherit
      Read ahead sectors auto
      - currently set to 256
      Block device 252:14
  5. In the guest, remove the logical volume and volume group that were created for the added disk.
    You must perform this step in order to use the newly created disk to extend an existing volume group.
    1. Remove the logical volume.

      In this example, the logical volume path is /dev/VGExaDbDisk.pv2_vgexadb.img/LVDBDisk.

      # lvremove /dev/VGExaDbDisk.pv2_vgexadb.img/LVDBDisk
      Do you really want to remove active logical volume VGExaDbDisk.pv2_vgexadb.img/LVDBDisk? [y/n]: y
        Logical volume "LVDBDisk" successfully removed
    2. Remove the volume group that came with the logical volume.

      In this example, the volume group name is VGExaDbDisk.pv2_vgexadb.img.

      # vgremove VGExaDbDisk.pv2_vgexadb.img
        Volume group "VGExaDbDisk.pv2_vgexadb.img" successfully removed
    At this point, all that is left is the physical volume with no logical volume and no volume group.
  6. In the guest, identify the physical volume device for the newly added disk.

    The physical volume identifies itself as a NEW Physical volume in pvdisplay output. For example:

    # pvdisplay
    ...  
        "/dev/sdf1" is a new physical volume of "<50.00 GiB"
      --- NEW Physical volume ---
      PV Name /dev/sdf1
      VG Name
      PV Size <50.00 GiB
      Allocatable NO
      PE Size 0
      Total PE 0
      Free PE 0
      Allocated PE 0
      PV UUID tfb8lM-eHe9-SPch-8UAu-pkHe-dAYx-ez3Sru
    ...
  7. In the guest, you can now use the physical volume to extend an existing volume group.

    In the following example, the volume group VGExaDb is extended using the physical volume device /dev/sdf1.

    # vgdisplay -s
      "VGExaDb" 88.00 GiB [88.00 GiB used / 0 free]
      "VGExaDbDisk.grid-klone-Linux-x86-64-190000.50.img" <50.00 GiB [<50.00 GiB used / 0 free]
      "VGExaDbDisk.db-klone-Linux-x86-64-190000.50.img" <50.00 GiB [<50.00 GiB used / 0 free]
    
    # vgextend VGExaDb /dev/sdf1
      Volume group "VGExaDb" successfully extended
    
    # vgdisplay -s
      "VGExaDb" <139.24 GiB [88.00 GiB used / <51.24 GiB free]
      "VGExaDbDisk.grid-klone-Linux-x86-64-190000.50.img" <50.00 GiB [<50.00 GiB used / 0 free]
      "VGExaDbDisk.db-klone-Linux-x86-64-190000.50.img" <50.00 GiB [<50.00 GiB used / 0 free]

To increase the size of various file systems, using the additional space added to the volume group by this procedure, refer to the following topics:

6.13.2 Increasing the Size of the root File System

This procedure describes how to increase the size of the system partition and / (root) file system.

This procedure is performed while the file system remains online.

Note:

There are two system partitions, LVDbSys1 and LVDbSys2. One partition is active and mounted. The other partition is inactive and used as a backup location during upgrade. The size of both system partitions must be equal.

Keep at least 1 GB of free space in the VGExaDb volume group. The free space is used for the LVM snapshot created by the dbnodeupdate.sh utility during software maintenance. If you make snapshot-based backups of the / (root) and /u01 directories as described in Creating a Snapshot-Based Backup of Oracle Linux Database Server, then keep at least 6 GB of free space in the VGExaDb volume group.

This task assumes that additional disk space is available to be used. If that is not the case, then complete the task Adding a New LVM Disk to a Guest before starting this procedure.
  1. Collect information about the current environment.
    1. Use the df command to identify the amount of free and used space in the root partition (/)
      # df -h /

      The following is an example of the output from the command:

      Filesystem                     Size  Used  Avail Use% Mounted on
      /dev/mapper/VGExaDb-LVDbSys1   12G   5.1G  6.2G  46%  / 
      

      Note:

      The active root partition may be either LVDbSys1 or LVDbSys2, depending on previous maintenance activities.
    2. Use the lvs command to display the current logical volume configuration.
      # lvs -o lv_name,lv_path,vg_name,lv_size

      The following is an example of the output from the command:

      LV        Path                   VG      LSize 
      LVDbOra1  /dev/VGExaDb/LVDbOra1  VGExaDb 10.00g
      LVDbSwap1 /dev/VGExaDb/LVDbSwap1 VGExaDb  8.00g
      LVDbSys1  /dev/VGExaDb/LVDbSys1  VGExaDb 12.00g
      LVDbSys2  /dev/VGExaDb/LVDbSys2  VGExaDb 12.00g 
      
  2. Verify there is available space in the volume group VGExaDb using the vgdisplay command.
    # vgdisplay VGExaDb -s

    The following is an example of the output from the command:

    "VGExaDb" 53.49 GiB [42.00 GiB used / 11.49 GiB free]

    The volume group must contain enough free space to increase the size of both system partitions, and maintain at least 1 GB of free space for the LVM snapshot created by the dbnodeupdate.sh utility during upgrade. If there is not sufficient free space in the volume group, then add a new disk to LVM.

  3. Resize both LVDbSys1 and LVDbSys2 logical volumes using the lvextend command.
    # lvextend -L +size /dev/VGExaDb/LVDbSys1
    # lvextend -L +size /dev/VGExaDb/LVDbSys2

    In the preceding command, size is the amount of space to be added to the logical volume. The amount of space added to each system partition must be the same.

    The following example extends the logical volumes by 10 GB:

    # lvextend -L +10G /dev/VGExaDb/LVDbSys1
    # lvextend -L +10G /dev/VGExaDb/LVDbSys2
  4. Resize the partition using the xfs_growfs command.
    # xfs_growfs /
  5. Verify the space was extended for the active system partition using the df command.
    # df -h /

6.13.3 Increasing the Size of the /u01 File System

This procedure describes how to increase the size of the /u01 file system in Oracle Linux KVM.

This procedure is performed while the file system remains online.

Note:

Keep at least 1 GB of free space in the VGExaDb volume group. The free space is used for the LVM snapshot created by the dbnodeupdate.sh utility during software maintenance. If you make snapshot-based backups of the / (root) and /u01 directories as described in Creating a Snapshot-Based Backup of Oracle Linux Database Server, then keep at least 6 GB of free space in the VGExaDb volume group
This task assumes that additional disk space is available to be used. If that is not the case, then complete the task Adding a New LVM Disk to a Guest before starting this procedure.
  1. Collect information about the current environment.
    1. Use the df command to identify the amount of free and used space in the /u01 partition.
      # df -h /u01
      

      The following is an example of the output from the command:

      Filesystem            Size  Used Avail Use% Mounted on
      /dev/mapper/VGExaDb-LVDbOra1
                            9.9G  1.7G  7.8G  18% /u01
      
    2. Use the lvs command to display the current logical volume configuration used by the /u01 file system.
      # lvs -o lv_name,lv_path,vg_name,lv_size /dev/VGExaDb/LVDbOra1
      

      The following is an example of the output from the command:

      LV        Path                  VG       LSize 
      LVDbOra1 /dev/VGExaDb/LVDbOra1  VGExaDb 10.00g
      
  2. Verify there is available space in the volume group VGExaDb using the vgdisplay command.
    # vgdisplay VGExaDb -s
    

    The following is an example of the output from the command:

    "VGExaDb" 53.49 GiB [42.00 GiB used / 11.49 GiB free]
    

    If the output shows there is less than 1 GB of free space, then neither the logical volume nor file system should be extended. Maintain at least 1 GB of free space in the VGExaDb volume group for the LVM snapshot created by the dbnodeupdate.sh utility during an upgrade. If there is not sufficient free space in the volume group, then add a new disk to LVM.

  3. Resize the logical volume using the lvextend command.
    # lvextend -L +sizeG /dev/VGExaDb/LVDbOra1
    

    In the preceding command, size is the amount of space to be added to the logical volume.

    The following example extends the logical volume by 10 GB:

    # lvextend -L +10G /dev/VGExaDb/LVDbOra1
    
  4. Resize the partition using the xfs_growfs command.
    # xfs_growfs /u01
    
  5. Verify the space was extended using the df command.
    # df -h /u01
    

6.13.4 Increasing the Size of the Grid Infrastructure Home or Database Home File System

You can increase the size of the Oracle Grid Infrastructure or Oracle Database home file system in a Oracle Linux KVM guest.

The Oracle Grid Infrastructure software home and the Oracle Database software home are created as separate disk image files in the KVM host. The disk image files are located in the /EXAVMIMAGES/GuestImages/DomainName/ directory. The disk image files are attached to the guest automatically during virtual machine startup, and mounted as separate, non-LVM file systems in the guest.

This task assumes that additional disk space is available to be used.

  1. Connect to the guest, and check the file system size using the df command. For example:
    # df -h
    Filesystem  
       Size Used Avail Use% Mounted on
    ...
    /dev/mapper/VGExaDbDisk.grid--klone--Linux--x86--64--190000.50.img-LVDBDisk 
       50G  5.9G 45G   12%  /u01/app/19.0.0.0/grid
    /dev/mapper/VGExaDbDisk.db--klone--Linux--x86--64--190000.50.img-LVDBDisk 
       50G  6.5G 44G   13%  /u01/app/oracle/product/19.0.0.0/DbHome_3
    ...
  2. Connect to the KVM host, and then shut down the guest.
    In the example, the guest name is dm01db01vm01.example.com.
    # /opt/exadata_ovm/vm_maker --stop-domain dm01db01vm01.example.com
    
  3. In the KVM host, create a new disk image and attach it to the guest.

    For example, the following command adds the disk image db03.img to the guest dm01db01vm01.example.com:

    # /opt/exadata_ovm/vm_maker --create --disk-image /EXAVMIMAGES/db03.img --attach 
    --domain dm01db01vm01.example.com
    [INFO] Allocating an image for /EXAVMIMAGES/db03.img, size 50.000000G...
    [INFO] Running 'qemu-img create /EXAVMIMAGES/db03.img 50.000000G '...
    [INFO] Create label gpt on /EXAVMIMAGES/db03.img.
    [INFO] Running 'parted -a none -s /EXAVMIMAGES/db03.img mklabel gpt'...
    [INFO] Running 'losetup -P -f /EXAVMIMAGES/rk02.img'...
    [INFO] Finding loop device...
    [INFO] loop device is /dev/loop0
    [INFO] Finding number of sectors...
    [INFO] 104857600 sectors
    [INFO] Releasing loop device /dev/loop0...
    [INFO] Removing device maps for /dev/loop0...
    [INFO] Running 'kpartx -d -v /dev/loop0'...
    [INFO] Removing loop device /dev/loop0...
    [INFO] ##
    [INFO] ## Finished .
    [INFO] ##
    [INFO] Created image /EXAVMIMAGES/db03.img
    [INFO] File /EXAVMIMAGES/GuestImages/dm01db01vm01.example.com/db03.img is a reflink from 
    /EXAVMIMAGES/db03.img and added as disk to domain dm01db01vm01.example.com
    [INFO] -------- MANUAL STEPS TO BE COMPLETED FOR MOUNTING THE DISK WITHIN DOMU dm01db01vm01
    .example.com --------
    [INFO] 1. Check a disk with name /dev/VGExaDbDisk.db03.img/LVDBDisk exists.
    [INFO] - Run the command 'lvdisplay' to verify a disk with name '/dev/VGExaDbDisk.db03.img/
    LVDBDisk' exists.
    [INFO] 2. Create a directory that will to be used for mounting the new disk.
    [INFO] 3. Add the following line to /etc/fstab: /dev/VGExaDbDisk.db03.img/LVDBDisk <mount_
    point_from_step_2> <fstype> defaults 1 1
    [INFO] 4. Mount the newly added disk to mount point through the command: mount -a.

    Do not perform the manual steps described in the output. However, take note of the logical volume path identified in manual step number 1.

    In general, the logical volume path has the form: /dev/VolumeGroupName/LogicalVolumeName.

    In the example, the logical volume path is /dev/VGExaDbDisk.db03.img/LVDBDisk.

  4. Restart the guest.
    For example:
    # /opt/exadata_ovm/vm_maker --start-domain dm01db01vm01.example.com --console
  5. In the guest, confirm the newly added disk device.

    Use the lvdisplay command along with the logical volume path noted earlier.

    # lvdisplay /dev/VGExaDbDisk.db03.img/LVDBDisk
      LV Path /dev/VGExaDbDisk.db03.img/LVDBDisk
      LV Name LVDBDisk
      VG Name VGExaDbDisk.db03.img
      LV UUID u3RBKF-UmCK-JQxc-iFf5-6WqS-GWAw-3nLjdn
      LV Write Access read/write
      LV Creation host, time dm01db01vm01.example.com, 2019-10-28 04:11:28 -0700
      LV Status available
      # open 0
      LV Size <50.00 GiB
      Current LE 12799
      Segments 1
      Allocation inherit
      Read ahead sectors auto
      - currently set to 256
      Block device 252:14
  6. In the guest, remove the logical volume and volume group that were created for the added disk.
    You must perform this step in order to use the newly created disk to extend an existing volume group.
    1. Remove the logical volume.

      In this example, the logical volume path is /dev/VGExaDbDisk.db03.img/LVDBDisk.

      # lvremove /dev/VGExaDbDisk.db03.img/LVDBDisk
      Do you really want to remove active logical volume VGExaDbDisk.db03.img/LVDBDisk? [y/n]: y
        Logical volume "LVDBDisk" successfully removed
    2. Remove the volume group that came with the logical volume.

      In this example, the volume group name is VGExaDbDisk.db03.img.

      # vgremove VGExaDbDisk.db03.img
        Volume group "VGExaDbDisk.db03.img" successfully removed
    At this point, all that is left is the physical volume with no logical volume and no volume group.
  7. In the guest, identify the physical volume device for the newly added disk.

    The physical volume identifies itself as a NEW Physical volume in pvdisplay output. For example:

    # pvdisplay
    ...  
        "/dev/sdf4" is a new physical volume of "<50.00 GiB"
      --- NEW Physical volume ---
      PV Name /dev/sdf4
      VG Name
      PV Size <50.00 GiB
      Allocatable NO
      PE Size 0
      Total PE 0
      Free PE 0
      Allocated PE 0
      PV UUID tfb8lM-eHe9-SPch-8UAu-pkHe-dAYx-ru3Sez
    ...
  8. In the guest, identify the volume group for the file system that you want to extend.
    Use the vgdisplay command. The volume group name contains grid for Oracle Grid Infrastructure or db for Oracle Database. For example:
    # vgdisplay -s
    ...
      "VGExaDbDisk.grid-klone-Linux-x86-64-190000.50.img" <50.00 GiB [<50.00 GiB used / 0 free]
      "VGExaDbDisk.db-klone-Linux-x86-64-190000.50.img" <50.00 GiB [<50.00 GiB used / 0 free]
    ...
  9. In the guest, extend the volume group, then verify the additional space in the volume group.
    Use the vgextend command and specify the volume group name and physical volume device that you identified previously. For example:
    # vgextend VGExaDbDisk.db-klone-Linux-x86-64-190000.50.img /dev/sdf4
      Volume group "VGExaDbDisk.db-klone-Linux-x86-64-190000.50.img" successfully extended
    Use the vgdisplay command to verify that the volume group now contains some free space. For example:
    # vgdisplay -s
    ...
      "VGExaDbDisk.grid-klone-Linux-x86-64-190000.50.img" <50.00 GiB [<50.00 GiB used / 0 free]
      "VGExaDbDisk.db-klone-Linux-x86-64-190000.50.img" <101.24 GiB [<50.00 GiB used / <51.24 GiB free]
    ...
  10. In the guest, resize the logical volume using the following lvextend command:
    # lvextend -L +sizeG LogicalVolumePath

    The following example extends the logical volume by 10 GB:

    # lvextend -L +10G /dev/VGExaDbDisk.db-klone-Linux-x86-64-190000.50.img/LVDBDisk
  11. In the guest, resize the file system partition using the xfs_growfs command.
    # xfs_growfs /u01/app/oracle/product/19.0.0.0/DbHome_3
  12. In the guest, verify the file system size was increased. For example:
    # df -h
    Filesystem                                                
       Size Used Avail Use% Mounted on
    ...
    /dev/mapper/VGExaDbDisk.db--klone--Linux--x86--64--190000.50.img-LVDBDisk 
       60G  6.5G 53G   10%  /u01/app/oracle/product/19.0.0.0/DbHome_3
    ...
  13. Connect to the KVM host, and remove the backup image.

    Use a command similar to the following where pre_resize.db19.0.0.img is the name of the backup image file created in step 3:

    # cd /EXAVMIMAGES/GuestImages/DomainName
    # rm pre_resize.db19.0.0.img

6.13.5 Increasing the Size of the Swap Area

You can increase the amount of swap configured in a guest.

  1. In the KVM host, create a new disk image and attach it to the guest.

    For example, the following command adds the disk image swap2.img to the guest dm01db01vm01.example.com:

    # /opt/exadata_ovm/vm_maker --create --disk-image /EXAVMIMAGES/swap2.img
     --attach --domain dm01db01vm01.example.com
    [INFO] Allocating an image for /EXAVMIMAGES/swap2.img, size 50.000000G...
    [INFO] Running 'qemu-img create /EXAVMIMAGES/swap2.img 50.000000G '...
    [INFO] Create label gpt on /EXAVMIMAGES/swap2.img.
    [INFO] Running 'parted -a none -s /EXAVMIMAGES/swap2.img mklabel gpt'...
    [INFO] Running 'losetup -P -f /EXAVMIMAGES/swap2.img'...
    [INFO] Finding loop device...
    [INFO] loop device is /dev/loop0
    [INFO] Finding number of sectors...
    [INFO] 104857600 sectors
    [INFO] Releasing loop device /dev/loop0...
    [INFO] Removing device maps for /dev/loop0...
    [INFO] Running 'kpartx -d -v /dev/loop0'...
    [INFO] Removing loop device /dev/loop0...
    [INFO] ##
    [INFO] ## Finished .
    [INFO] ##
    [INFO] Created image /EXAVMIMAGES/swap2.img
    [INFO] File /EXAVMIMAGES/GuestImages/dm01db01vm01.example.com/swap2.img is a reflink from 
    /EXAVMIMAGES/swap2.img and added as disk to domain dm01db01vm01.example.com
    [INFO] -------- MANUAL STEPS TO BE COMPLETED FOR MOUNTING THE DISK WITHIN DOMU dm01db01vm01
    .example.com --------
    [INFO] 1. Check a disk with name /dev/VGExaDbDisk.swap2.img/LVDBDisk exists.
    [INFO] - Run the command 'lvdisplay' to verify a disk with name '/dev/VGExaDbDisk.swap2.img/
    LVDBDisk' exists.
    [INFO] 2. Create a directory that will to be used for mounting the new disk.
    [INFO] 3. Add the following line to /etc/fstab: /dev/VGExaDbDisk.swap2.img/LVDBDisk <mount_
    point_from_step_2> <fstype> defaults 1 1
    [INFO] 4. Mount the newly added disk to mount point through the command: mount -a.

    Do not perform the manual steps described in the output. However, take note of the logical volume path identified in manual step number 1.

    In general, the logical volume path has the form: /dev/VolumeGroupName/LogicalVolumeName.

    In the example, the logical volume path is /dev/VGExaDbDisk.swap2.img/LVDBDisk.

  2. In the guest, configure the new logical volume as a swap device.

    Use the mkswap command, and configure the new logical volume with a unique label, which is not currently in use in the /etc/fstab file.

    In the following example, the swap device label is SWAP2 and the logical volume path is /dev/VGExaDbDisk.swap2.img/LVDBDisk.

    # mkswap -L SWAP2 /dev/VGExaDbDisk.swap2.img/LVDBDisk
  3. In the guest, enable the new swap device.

    Use the swapon command with the -L option and specify the label of the newly created swap device.

    For example:

    # swapon -L SWAP2
  4. In the guest, verify that the new swap device is enabled by using the swapon -s command.

    For example:

    # swapon -s
    Filename                   Type            Size      Used     Priority
    /dev/dm-3                  partition       8388604   306108   -1
    /dev/VGExaDb/LVDbSwap2     partition       8388604   0        -2
    
  5. In the guest, edit the /etc/fstab file to include the new swap device.

    You can copy the existing swap entry, and then change the LABEL value in the new entry to the label used to create the new swap device.

    In the following example, the new swap device is added to the /etc/fstab file using LABEL=SWAP2.

    # cat /etc/fstab
    LABEL=DBSYS   /                                            ext4    defaults        1 1
    LABEL=BOOT    /boot                                        ext4    defaults,nodev  1 1
    tmpfs         /dev/shm                                     tmpfs   defaults,size=7998m 0
    devpts        /dev/pts                                     devpts  gid=5,mode=620  0 0
    sysfs         /sys                                         sysfs   defaults        0 0
    proc          /proc                                        proc    defaults        0 0
    LABEL=SWAP    swap                                         swap    defaults        0 0
    LABEL=SWAP2   swap                                         swap    defaults        0 0
    LABEL=DBORA   /u01                                         ext4    defaults        1 1
    /dev/xvdb     /u01/app/12.1.0.2/grid                       ext4    defaults        1 1
    /dev/xvdc     /u01/app/oracle/product/12.1.0.2/dbhome_1    ext4    defaults        1 1
    

6.14 Adding an Oracle VM Cluster

You can use Oracle Exadata Deployment Assistant (OEDA) to create a new Oracle VM cluster on an existing Oracle Exadata Database Machine.

6.15 Adding an Oracle Linux KVM Cluster

You can use Oracle Exadata Deployment Assistant (OEDA) to create a new Oracle Linux KVM cluster on an existing Oracle Exadata Database Machine.

6.16 Expanding an Oracle RAC Cluster in Oracle Linux KVM Using OEDACLI

You can expand an existing Oracle RAC cluster on Oracle Linux KVM by adding guests using the Oracle Exadata Deployment Assistant command-line interface (OEDACLI).

OEDACLI is the preferred method if you have a known, good version of the OEDA XML file for your cluster.

Note:

During the execution of this procedure, the existing Oracle RAC cluster nodes along with their database instances incur zero downtime.

Use cases for this procedure include:

  • You have an existing Oracle RAC cluster that uses only a subset of the database servers of an Oracle Exadata Rack, and now the nodes not being used by the cluster have become candidates for use.
  • You have an existing Oracle RAC cluster on Oracle Exadata Database Machine that was recently extended with additional database servers.
  • You have an existing Oracle RAC cluster that had a complete node failure and the node was removed and replaced with a newly re-imaged node.

Before preforming the steps in this section, the new database servers should have been set up as detailed in Adding a New Database Server to the Cluster, including the following:

  • The new database server is installed and configured on the network with a KVM host.
  • Download the latest Oracle Exadata Deployment Assistant (OEDA); ensure the version you download is the July 2019 release, or later.
  • You have an OEDA configuration XML file that accurately reflects the existing cluster configuration. You can validate the XML file by generating an installation template from it and comparing it to the current configuration. See the OEDACLI command SAVE FILES.
  • Review the OEDA Installation Template report for the current system configuration to obtain node names and IP addresses for existing nodes. You will need to have new host names and IP addresses for the new nodes being added. The new host names and IP addresses required are:
    • Administration host names and IP addresses (referred to as ADMINNET) for the KVM host and the guests.
    • Private host names and IP addresses (referred to as PRIVNET) for the KVM host and the guests.
    • Integrated Lights Out Manager (ILOM) host names and IP addresses for the KVM host.
    • Client host names and IP addresses (referred to as CLIENTNET) for the guests.
    • Virtual IP (VIP) host names and IP addresses (referred to as VIPNET) for the guests.
    • Physical rack number and location of the new node in the rack (in terms of U number)
  • Each KVM host has been imaged or patched to the same image in use on the existing database servers. The current system image must match the version of the /EXAVMIMAGES/ System.first.boot.*.img file on the new KVM host node.

    Note:

    The ~/dom0_group file referenced below is a text file that contains the host names of the KVM hosts for all existing and new nodes being added.

    Check that the image version across all KVM hosts are the same.

    dcli -g ~/dom0_group -l root "imageinfo -ver"
    
    exa01adm01: 19.2.0.0.0.190225
    exa01adm02: 19.2.0.0.0.190225
    exa01adm03: 19.2.0.0.0.190225

    If any image versions differ, you must upgrade the nodes as needed so that they match.

    Ensure that the System.first.boot version across all KVM hosts matches the image version retrieved in the previous step.

    dcli -g ~/dom0_group -l root "ls  -1 /EXAVMIMAGES/System.first.boot*.img" 
    exa01adm01:  /EXAVMIMAGES/System.first.boot.19.2.0.0.0.190225.img
    exa01adm02:  /EXAVMIMAGES/System.first.boot.19.2.0.0.0.190225.img
    exa01adm03:  /EXAVMIMAGES/System.first.boot.19.2.0.0.0.190225.img

    If any nodes are missing the System.first.boot.img file that corresponds to the current image, then obtain the required file. See the “Supplemental README note” for your Exadata release in My Oracle Support Doc ID 888828.1 and look for the patch file corresponding to this description, “DomU System.img OS image for V.V.0.0.0 VM creation on upgraded dom0s”

  • Place the klone.zip files (gi-klone*.zip and db-klone*.zip) in the /EXAVMIMAGES location on the freshly imaged KVM host node you are adding to the cluster. These files can be found in the/EXAVMIMAGES directory on the KVM host node from where the system was initially deployed.

The steps here show how to add a new KVM host node called exa01adm03 that will have a new guest called exa01adm03vm01. The steps show how to extend an existing Oracle RAC cluster onto the guest using OEDACLI commands. The existing cluster has KVM host nodes named exa01adm01 and exa01adm02 and guest nodes named exa01adm01vm01 and exa01adm02vm01.

  1. Add the KVM host information to the OEDA XML file using the CLONE COMPUTE command.

    In the examples below, the OEDA XML file is assumed to be in: unzipped_OEDA_location/ExadataConfigurations.

    OEDACLI> LOAD FILE NAME=exa01_original_deployment.xml 
    
    OEDACLI> CLONE COMPUTE SRCNAME  = exa01adm01 TGTNAME = exa01adm03
    SET ADMINNET NAME=exa01adm03,IP=xx.xx.xx.xx
    SET PRIVNET NAME1=exa01adm03-priv1,IP1=  xx.xx.xx.xx, 
    SET PRIVNET NAME2=exa01adm03-priv2,IP2=  xx.xx.xx.xx
    SET ILOMNET NAME=exa01adm03-c,IP=xx.xx.xx.xx
    SET RACK NUM=NN,ULOC=XX 
    
    OEDACLI> SAVE ACTION
    OEDACLI> MERGE ACTIONS FORCE
    OEDACLI> SAVE FILE NAME=exa01_plus_adm03_node.xml

    At this point we have a new XML file that has the new compute node KVM host in the configuration. This file will be used by the subsequent steps.

  2. Add the new guest information to the OEDA XML file using the CLONE GUEST command and deploy the guest.
    OEDACLI> LOAD FILE NAME=exa01_plus_adm03_node.xml 
    
    OEDACLI> CLONE GUEST SRCNAME  = exa01adm01vm01 TGTNAME = exa01adm03vm01
    WHERE STEPNAME=CREATE_GUEST
    SET PARENT NAME = exa01adm03
    SET ADMINNET NAME=exa01adm03vm01,IP=xx.xx.xx.xx
    SET PRIVNET NAME1=exa01db03vm01-priv1,IP1=  xx.xx.xx.xx, 
    SET PRIVNET NAME2=exa01db03vm01-priv2,IP2=  xx.xx.xx.xx
    SET CLIENTNET NAME=exa01client03vm01,IP=xx.xx.xx.xx
    SET VIPNET NAME=exa01client03vm01-vip,IP=xx.xx.xx.xx
    
    
    OEDACLI> SAVE ACTION
    OEDACLI> MERGE ACTIONS
    OEDACLI> DEPLOY ACTIONS

    If you prefer that OEDACLI runs all steps automatically, omit the following clause above, WHERE STEPNAME=CREATE_GUEST and skip step 3 below.

    At this point we have a guest created on our new compute node.

  3. Use OEDACLI to extend the cluster to the new guest.

    Note:

    Continue using the same XML file, exa01_plus_adm03_node.xml in this example. You will continue to update this file as you proceed through these steps. At the very end of the procedure, this XML file will properly reflect the new state of the clusters.
    OEDACLI> CLONE GUEST TGTNAME=exa01adm03vm01 WHERE STEPNAME = CREATE_USERS

    OEDACLI> SAVE ACTION
    OEDACLI> MERGE ACTIONS
    OEDACLI> DEPLOY ACTIONS
    
    OEDACLI> CLONE GUEST TGTNAME=exa01adm03vm01 WHERE STEPNAME = CELL_CONNECTIVITY

    OEDACLI> SAVE ACTION
    OEDACLI> MERGE ACTIONS
    OEDACLI> DEPLOY ACTIONS
    
    OEDACLI> CLONE GUEST TGTNAME=exa01adm03vm01 WHERE STEPNAME = ADD_NODE

    OEDACLI> SAVE ACTION
    OEDACLI> MERGE ACTIONS
    OEDACLI> DEPLOY ACTIONS
    
    OEDACLI> CLONE GUEST TGTNAME=exa01adm03vm01 WHERE STEPNAME = EXTEND_DBHOME

    OEDACLI> SAVE ACTION
    OEDACLI> MERGE ACTIONS
    OEDACLI> DEPLOY ACTIONS
    
    OEDACLI> CLONE GUEST TGTNAME=exa01adm03vm01 WHERE STEPNAME = ADD_INSTANCE

    OEDACLI> SAVE ACTION
    OEDACLI> MERGE ACTIONS
    OEDACLI> DEPLOY ACTIONS

    OEDACLI prints out messages similar to the following as each step completes:

    Deploying Action ID : 39 CLONE GUEST TGTNAME=exa01adm03vm01 where STEPNAME = ADD_INSTANCE 
    Deploying CLONE GUEST 
    Cloning Guest 
    Cloning Guest  :  exa01adm03vm01.example.com_id 
    Adding new instance for database [dbm] on exa01adm03vm01.example.com 
    Setting up Huge Pages for Database..[dbm] 
    Adding instance dbm3 on host exa01adm03vm01.example.com 
    Successfully completed adding database instance on the new node [elapsed Time [Elapsed = 
    249561 mS [4.0  minutes] Fri Jun 28 13:35:52 PDT 2019]] 
    Done...
    Done
  4. Save the current state of the configuration and generate configuration information.
    OEDACLI> SAVE FILES LOCATION=/tmp/exa01_plus_adm03_config

    The above command writes all the configuration files to the directory /tmp/exa01_plus_adm03_config. Save a copy of these files in a safe place since they now reflect the changes made to your cluster.

  5. Gather an Oracle EXAchk report and examine it to ensure the cluster is in good health.

6.17 Moving a Guest to a Different Database Server

Guests can move to different database servers.

The target Oracle Exadata Database Machine database server must meet the following requirements:

  • The target database server must have the same Oracle Exadata System Software release installed with Oracle Linux KVM.

  • The target database server must have the same network visibility.

  • The target database server must have access to the same Oracle Exadata Database Machine storage servers.

  • The target database server must have sufficient free resources (CPU, memory, and local disk storage) to operate the guest.

    • It is possible to over-commit virtual CPUs such that the total number of virtual CPUs assigned to all domains exceeds the number of physical CPUs on the system. Over-committing CPUs can be done only when the competing workloads for over-subscribed resources are well understood and the concurrent demand does not exceed physical capacity.

    • It is not possible to over-commit memory.

    • Copying disk images to the target database server may increase space allocation of the disk image files because the copied files are no longer able to benefit from the disk space savings gained by using reflinks.

  • The guest name must not be already in use on the target database server.

The following procedure moves a guest to a new database server in the same Oracle Exadata System Software configuration. All steps in this procedure are performed in the KVM host.

  1. Shut down the guest.
    # /opt/exadata_ovm/vm_maker --stop-domain GuestName
  2. Copy the guest disk image and configuration files to the target database server.

    In the following examples, replace GuestName with the name of the guest.

    # scp -r /EXAVMIMAGES/GuestImages/GuestName/ target:/EXAVMIMAGES/GuestImages
    
  3. Obtain the UUID of the guest.
    # grep ^uuid /EXAVMIMAGES/GuestImages/GuestName/vm.cfg
    

    An example of the guest UUID is 49ffddce4efe43f5910d0c61c87bba58.

  4. Using the UUID of the guest, copy the guest symbolic links from /OVS/Repositories to the target database server.
    # tar cpvf - /OVS/Repositories/UUID/ | ssh target_db_server "tar xpvf - -C /"
    
  5. Start the guest on the target database server.
    # /opt/exadata_ovm/vm_maker --start-domain GuestName

6.18 Recovering a KVM Deployment

A KVM host can be recovered from backups and guests can be recovered from snapshot backups.

A KVM host can be recovered from a snapshot-based backup when severe disaster conditions damage the Oracle KVM host, or when the server hardware is replaced to such an extent that it amounts to new hardware.

For example, replacing all hard disks leaves no trace of original software on the system. This is similar to replacing the complete system as far as the software is concerned.

The recovery procedures described in this section do not include backup or recovery of Exadata storage servers or the data in an Oracle Database. Oracle recommends testing the backup and recovery procedures on a regular basis.

6.18.1 Overview of Snapshot-Based Recovery of KVM Hosts

The recovery of a KVM host consists of a series of tasks.

The recovery of a KVM consists of a series of tasks.

The recovery procedures use the diagnostics.iso image as a virtual CD-ROM to restart the KVM host in rescue mode using the Integrated Lights Out Manager (ILOM). At a high-level, the steps are:

  1. Re-create the following:
    • Boot partitions
    • Physical volumes
    • Volume groups
    • Logical volumes
    • File system
    • Swap partition
  2. Activate the swap partition
  3. Ensure the /boot partition is the active boot partition
  4. Restore the data
  5. Reconfigure GRUB
  6. Restart the server

6.18.2 KVM System Recovery Scenarios

How to recover a KVM system deployment.

The following scenarios are applicable to a KVM system recovery:

6.18.2.1 Recovering a KVM Host and the Guests from Backup

This procedure recovers the KVM host and all its guest from a backup of the KVM host and a backup of the guests from the KVM host.

A KVM host can be recovered from a snapshot-based backup using the steps below when severe disaster conditions damage the management domain, or when the server hardware is replaced to such an extent that it amounts to new hardware.

Prepare an NFS server to host the backup archives created in Backing up the KVM host Using Snapshot-Based Backup

The NFS server must be accessible by IP address. For example, on an NFS server with the IP address nfs_ip, where the directory /Backup contains the backup archives.

Attach the diagnostics iso file as a virtual media to the ILOM of the KVM host to be restored. This is can be done in 2 ways.

The following example shows how to set up a virtual CD-ROM using the ILOM interface:

  1. Attach the diagnostics.iso file as a virtual media to the ILOM of the KVM host to be restored. This is can be done in 2 ways.

    Method 1 he following example shows how to set up a virtual CD-ROM using the ILOM interface:

    1. Obtain the diagnostics iso for the Exadata software release of the KVM host.
    2. Setup the machine that will be using the ILOM interface using: Oracle® ILOM Administrator's Guide for Configuration and Maintenance Firmware Release 4.0.x
    3. Copy the diagnostics iso file to a directory on the machine that will be using the ILOM interface.
    4. Log in to the ILOM web interface.
    5. In the Oracle ILOM web interface, click Remote Control, and then click Redirection.
    6. Select Use Video Redirection
    7. After the console launches, click Storage in the KVMS menu.

      To add a storage image, such as a DVD image, to the Storage Devices dialog box, click Add.

      1. Open the diagnostics.iso file.
      2. To redirect storage media from the Storage Device dialog box, select the storage media and click Connect.
      3. After a connection to the device has been established, the label on the Connect button in the Storage Device dialog box changes to Disconnect.
      4. Select Host Control from the Host Management tab.
      5. Select CDROM as the next boot device from the list of values.
      6. Click Save.

    When the system is booted, the diagnostics.iso image is used for the boot process.

    Method 2: This method needs an NFS server for sourcing the diagnostics.iso file.

    1. Login to the ILOM CLI using root.
      -> set /SP/cli timeout=0                             
      -> cd /SP/services/kvms/host_storage_device
      -> cd remote
      -> set server_URI=nfs://<NFS server IP >:<full_path_to_diagnostics_iso>
      -> cd .. 
      -> set mode=remote  
      -> set /HOST boot_device=cdrom 
      Verify that mode is set to remote & that status is operational:
      -> show
      /SP/services/kvms/host_storage_device    
       Targets:
              remote  
       Properties:
              mode = remote
              status =
              operational    
       Commands:
              cd
              show 
      If it is not operational, but stays in 'connecting' then do "set mode=disabled" followed by "set mode=remote" to re-start the service.
  2. Restart the system from the ISO image file using one of the following methods:.
    • Choose the CD-ROM as the boot device during start up.
    • Preset the boot device by running the ipmitool command from any other machine that can reach the ILOM of the management domain to be restored:
      # ipmitool -H ILOM_ip_address_or_hostname  -U root chassis bootdev cdrom
      # ipmitool -H ILOM_ip_address_or_hostname  -U root chassis power cycle
    Either of these will force boot the machine from cdrom. Start the serial console and perform steps 3-15 below from the serial console. The serial console can be started using the below command from the ILOM CLI.
    ->  Start /SP/console -script
  3. Log in to the diagnostics shell as the root user.
    Choose from following by typing letter in '()':
    (e)nter interactive diagnostics shell. Must use credentials 
    from Oracle support to login (reboot or power cycle to exit
    the shell),
    (r)estore system from NFS backup archive, 
    Type e to enter the diagnostics shell and log in as the root user.
    Note: If you do not have the password for the root user, then contact Oracle Support Services.
  4. If required, use /opt/MegaRAID/MegaCli/MegaCli64 to configure the disk controller to set up the disks.
  5. Remove the logical volumes, the volume group, and the physical volume, in case they still exist after the disaster.
    # lvm vgremove VGExaDb  --force
    # lvm pvremove /dev/sda3 –force
  6. Remove the existing partitions, then verify all partitions were removed. The below script can be used.
    sh-4.2# for v_partition in $(parted -s /dev/sda print|awk '/^ / {print $1}')
    do
      parted -s /dev/sda rm ${v_partition}
    done
     
    sh-4.2#  parted  -s /dev/sda unit s print
    Model: AVAGO MR9[ 2783.921605]  sda:361-16i (scsi)
    Disk /dev/sda: 3509760000s
    Sector size (logical/physical): 512B/512B
    Partition Table: gpt
    Disk Flags:  
    
    Number  Start   End  Size  File system Name  Flags
  7. Create the three partitions on /dev/sda.Get the end sector for the disk /dev/sda from a surviving KVM host and store it in a variable:
    # end_sector_logical=$(parted -s /dev/sda unit s print|perl -ne '/^Disk\s+\S+:\s+(\d+)s/
              and print $1')
    # end_sector=$( expr $end_sector_logical - 34 )
    # echo $end_sector
    The values for the start and end sectors in the commands below were taken from a surviving KVM host. Because these values can change over time, it is recommended that these values are checked from a KVM host at the time of performing this procedure. For example, for an Oracle Exadata Database Machine X8M-2 database server the following might be seen:
    # parted -s /dev/sda  unit s print
    Model:  AVAGO MR9361-16i (scsi)
    Disk  /dev/sda: 7025387520s
    Sector  size (logical/physical): 512B/512B
    Partition  Table: gpt
    Disk  Flags:  
    Number   Start     End         Size         File system   Name     Flags  
    1        64s       1048639s    1048576s     xfs           primary  boot  
    2        1048640s  1572927s    524288s      fat32         primary  boot  
    3        1572928s  7025387486s 7023814559s                primary  lvm
    1. Create the boot partition, /dev/sda1.
      # parted -s /dev/sda  mklabel gpt mkpart primary 64s 1048639s set 1 boot on
    2. Create the efi boot partition , /dev/sda2.
      # parted -s /dev/sda  mkpart primary fat32 1048640s 1572927s set 2 boot on
    3. Create the partition that will hold the logival volumes, /dev/sda3.
      # parted -s /dev/sda mkpart primary 1572928s ${end_sector}s set 3 lvm on
    4. Verify all the partitions have been created.
      sh-4.2# parted -s /dev/sda unit s print
      Model: AVAGO MR9[2991.834796]  sda: sda1 sda2 sda3
      361-16i(scsi)
      Disk /dev/sda:3509760000s
      Sector size(logical/physical): 512B/512B
      Partition Table:gpt
      Disk Flags:
      Number  Start    End            Size          File system    Name    Flags 
      1       64s      1048639s       1048576s      xfs            primary boot 
      2       1048640s 1572927s       524288s       fat32          primary boot   
      3       1572928s 3509759966s    3508187039s                  primary lvm
  8. Create logical volumes and file systems.
    1. Create the physical volume and the volume group.
      # lvm pvcreate /dev/sda3
      # lvm vgcreate VGExaDb /dev/sda3
    2. Create and label the logical volume for the file system that will contain the first system partition and build a xfs file system on it.
      #  lvm lvcreate -n LVDbSys1 -L15G VGExaDb -y
      # mkfs.xfs -L DBSYS /dev/VGExaDb/LVDbSys1 -f
    3. Create and label the logical volume for the swap directory.
      # lvm lvcreate -n LVDbSwap1  -L16G VGExaDb -y
      # mkswap -L SWAP /dev/VGExaDb/LVDbSwap1
    4. Create the logical volume for the second system partition and build a xfs file system on it.
      # lvm lvcreate -n LVDbSys2 -L15G VGExaDb -y
      # mkfs.xfs /dev/VGExaDb/LVDbSys2
    5. Create and label the logical volume for the HOME partition and build a xfs file system on it.
      #  lvm lvcreate -n LVDbHome -L4G VGExaDb -y
      #  mkfs.xfs -L HOME /dev/VGExaDb/LVDbHome
    6. Create the logical volume for the tmp partition and build a xfs file system on it.
      # lvm lvcreate -n LVDbTmp -L3G VGExaDb -y
      # mkfs.xfs -L TMP /dev/VGExaDb/LVDbTmp -f
    7. Create the logical volume for the first var partition and build a xfs file system on it.
      # lvm lvcreate -n LVDbVar1 -L2G VGExaDb -y
      # mkfs.xfs -L VAR /dev/VGExaDb/LVDbVar1 -f
    8. Create the logical volume for the second var partition and build a xfs file system on it.
      # lvm lvcreate -n LVDbVar2 -L2G VGExaDb -y
      # mkfs.xfs /dev/VGExaDb/LVDbVar2 -f
    9. Create and label the logical volume for the LVDbVarLog partition and build a xfs file system on it.
      # lvm lvcreate -n LVDbVarLog -L18G VGExaDb -y
      # mkfs.xfs -L DIAG /dev/VGExaDb/LVDbVarLog -f
    10. Create and label the logical volume for the LVDbVarLogAudit partition and build a xfs file system on it.
      # lvm lvcreate -n LVDbVarLogAudit -L1G VGExaDb -y
      # mkfs.xfs -L AUDIT /dev/VGExaDb/LVDbVarLogAudit -f
    11. Create the LVDoNotRemoveOrUse logical volume.
      # lvm lvcreate -n LVDoNotRemoveOrUse -L2G VGExaDb -y
    12. Create the logical volume for the guest storage repository and build a xfs file system on it.
      # lvm lvcreate -n LVDbExaVMImages -L1500G VGExaDb -y
      # mkfs.xfs -L EXAVMIMAGES /dev/VGExaDb/LVDbExaVMImages -f
    13. Create a file system on the /dev/sda1 partition, and label it.
      # mkfs.xfs -L BOOT /dev/sda1 -f
    14. Create a file system on the /dev/sda2 partition, and label it.
      # mkfs.vfat -v -c -F 32 -s 2 /dev/sda2
      # dosfslabel /dev/sda2 ESP
  9. Create mount points for all the partitions and mount the respective partitions.

    For example, if /mnt is used as the top-level directory, the mounted list of partitions might look like:

    /dev/VGExaDb/LVDbSys1 on /mnt
    /dev/sda1 on /mnt/boot
    /dev/sda2 on /mnt/boot/efi
    /dev/VGExaDb/LVDbHome on /mnt/home
    /dev/VGExaDb/LVDbVar1 on /mnt/var
    /dev/VGExaDb/LVDbVarLog on /mnt/var/log
    /dev/VGExaDb/LVDbVarLogAudit on /mnt/var/log/audit
    /dev/VGExaDb/LVDbExaVMImages on /mnt/EXAVMIMAGES
    The following example mounts the system partition and creates 2 mount points for the boot partitions
    # mount /dev/VGExaDb/LVDbSys1 /mnt -t xfs
    # mkdir /mnt/boot
    # mount /dev/sda1 /mnt/boot -t xfs
    # mkdir /mnt/boot/efi
    #  mount /dev/sda2 /mnt/boot/efi -t vfat
    #  mkdir /mnt/home
    #  mount /dev/VGExaDb/LVDbHome /mnt/home -t xfs
    # mkdir /mnt/var
    # mount /dev/VGExaDb/LVDbVar1 /mnt/var -t xfs
    # mkdir /mnt/var/log
    # mount /dev/VGExaDb/LVDbVarLog /mnt/var/log -t xfs
    # mkdir /mnt/var/log/audit
    # mount /dev/VGExaDb/LVDbVarLogAudit /mnt/var/log/audit -t xfs
    # mkdir /mnt/EXAVMIMAGES
    # mount /dev/VGExaDb/LVDbExaVMImages /mnt/EXAVMIMAGES -t xfs
  10. Bring up the network on eth0 and (if not using DHCP) assign the host's IP address and netmask to it. If using DHCP then manually configure the IP address for the host.
    # ip link set eth0 up
    # ip address add ip_address_for_eth0/netmask_for_eth0 dev eth0
    # ip route add default via gateway_ip_address dev eth0
  11. Mount the NFS server holding the backups.
    #  mkdir -p /root/mnt
    #  mount -t nfs -o ro,intr,soft,proto=tcp,nolock nfs_ip:/<location_of_backup>  /root/mnt
    From the backup which was created in section Backing up the KVM host Using Snapshot-Based Backup, restore the following:
    • The root directory (/) and the boot file system
      # tar -pjxvf /root/mnt/rootfs-boot.tar.bz2 -C /mnt
    • The /var directory
      # tar -pjxvf /root/mnt/var.tar.bz2 -C /mnt/var
    • The /var/log directory
      .# tar -pjxvf /root/mnt/varlog.tar.bz2 -C /mnt/var/log
    • The /var/log/audit directory
      # tar -pjxvf /root/mnt/varlogaudit.tar.bz2 -C /mnt/var/log/audit
    • The /home directory
      # tar -pjxvf /root/mnt/home.tar.bz2 -C /mnt/home
    • Create the directory for kdump service
      # mkdir /mnt/EXAVMIMAGES/crashfiles
  12. Set the boot device using efibootmgr.
    1. Disable and delete the Oracle Linux boot device.

      If the entry ExadataLinux_1 is seen, then remove this entry and recreate it. Example:

      # efibootmgr
      BootCurrent:  0000
      Timeout:  1 seconds
      BootOrder: 0000,0001,000A,000B,0007,0008,0004,0005
      Boot0000*  ExadataLinux_1
      Boot0001*  NET0:PXE IP4 Intel(R) I210 Gigabit Network Connection
      Boot0004*  PCIE1:PXE IP4 Oracle dual 25Gb Ethernet Adapter or dual 10Gb Ethernet Adapter
      Boot0005*  PCIE1:PXE IP4 Oracle dual 25Gb Ethernet Adapter or dual 10Gb Ethernet  Adapter
      Boot0007*  NET1:PXE IP4 Oracle Dual Port 10Gb/25Gb SFP28 Ethernet Controller
      Boot0008*  NET2:PXE IP4 Oracle Dual Port 10Gb/25Gb SFP28 Ethernet Controller
      Boot000A   PCIE2:PXE IP4 Mellanox Network Adapter - 50:6B:4B:CB:EF:F2
      Boot000B   PCIE2:PXE IP4 Mellanox Network Adapter - 50:6B:4B:CB:EF:F3
      MirroredPercentageAbove4G:  0.00
      MirrorMemoryBelow4GB:  false    
      In this example, ExadataLinux_1 (Boot000) would be disabled and removed. Use the commands below to disable and delete the boot device, disavle old 'ExadataLinux_1':
      # efibootmgr -b 0000 -A
      Delete old 'ExadataLinux_1':
      # efibootmgr -b 0000 -B
    2. Recreate the boot entry for ExadataLinux_1 and then view the boot order entries.
      # efibootmgr -c -d /dev/sda  -p 2 -l '\EFI\REDHAT\SHIM.EFI' -L 'ExadataLinux_1'
      # efibootmgr
      BootCurrent:  0000
      Timeout:  1 seconds
      BootOrder: 0000,0001,0007,0008,0004,0005,000B,000C
      Boot0000*  ExadataLinux_1
      Boot0001*  NET0:PXE IP4 Intel(R) I210 Gigabit Network Connection
      Boot0004*  PCIE1:PXE IP4 Oracle dual 25Gb Ethernet Adapter or dual 10Gb Ethernet  Adapter
      Boot0005*  PCIE1:PXE IP4 Oracle dual 25Gb Ethernet Adapter or dual 10Gb Ethernet  Adapter
      Boot0007*  NET1:PXE IP4 Oracle Dual Port 10Gb/25Gb SFP28 Ethernet Controller
      Boot0008*  NET2:PXE IP4 Oracle Dual Port 10Gb/25Gb SFP28 Ethernet Controller
      Boot000B*  PCIE2:PXE IP4 Mellanox Network Adapter - EC:0D:9A:CC:1E:46
      Boot000C*  PCIE2:PXE IP4 Mellanox Network Adapter - EC:0D:9A:CC:1E:47
      MirroredPercentageAbove4G: 0.00
      MirrorMemoryBelow4GB: false
      In the output from the efibootmgr command, make note of the boot order number for ExadataLinux_1 and use that value in the following commands
      #  efibootmgr -b (entry number) -A
      #  efibootmgr -b (entry number) -a
      For example, in the previous output shown in step 12.b, ExadataLinux_1 was listed as (Boot0000). So you would use the following commands:
      #  efibootmgr -b 0000 -A
      #  efibootmgr -b 0000 -a
    3. Set the correct boot order. Set ExadataLinux_1 as the first boot device. The remaining devices should stay in the same boot order
      .# efibootmgr -o 0000,0001,0004,0005,0007,0008,000B,000C
      The boot order should now look like the following:
      # efibootmgr
      BootCurrent: 0000
      Timeout: 1 seconds
      BootOrder: 0000,0001,0004,0005,0007,0008,000B,000C
      Boot0000* ExadataLinux_1
      Boot0001* NET0:PXE IP4 Intel(R) I210 Gigabit Network Connection
      Boot0004* PCIE1:PXE IP4 Oracle dual 25Gb Ethernet Adapter or dual 10Gb Ethernet Adapter
      Boot0005* PCIE1:PXE IP4 Oracle dual 25Gb Ethernet Adapter or dual 10Gb Ethernet Adapter
      Boot0007* NET1:PXE IP4 Oracle Dual Port 10Gb/25Gb SFP28 Ethernet Controller
      Boot0008* NET2:PXE IP4 Oracle Dual Port 10Gb/25Gb SFP28 Ethernet Controller
      Boot000B* PCIE2:PXE IP4 Mellanox Network Adapter - EC:0D:9A:CC:1E:46
      Boot000C* PCIE2:PXE IP4 Mellanox Network Adapter - EC:0D:9A:CC:1E:47
      MirroredPercentageAbove4G: 0.00
      MirrorMemoryBelow4GB: false 
      Check the boot order using the ubiosconfig command.
      # ubiosconfig export all -x /tmp/ubiosconfig.xml
      Make sure the ExadataLinux_1 entry is the first child element of boot_order.
      <boot_order>
       <boot_device>
        <description>ExadataLinux_1</description>
        <instance>1</instance> </boot_device>
       <boot_device>
        <description>NET0:PXE IP4 Intel(R) I210 Gigabit Network Connection</description>
        <instance>1</instance>
       </boot_device>
       <boot_device>
        <description>PCIE1:PXE IP4 Oracle dual 25Gb Ethernet Adapter or dual 10Gb Ethernet Adapter</description>
        <instance>1</instance> 
       </boot_device>
       <boot_device>
        <description>PCIE1:PXE IP4 Oracle dual 25Gb Ethernet Adapter or dual 10Gb Ethernet Adapter</description>
        <instance>2</instance> 
       </boot_device>
       <boot_device>
        <description>NET1:PXE IP4 Oracle Dual Port 10Gb/25Gb SFP28 Ethernet Controller</description>
        <instance>1</instance>
       </boot_device>
       <boot_device>
        <description>NET2:PXE IP4 Oracle Dual Port 10Gb/25Gb SFP28 Ethernet Controller</description>
        <instance>1</instance> 
       </boot_device>
       <boot_device>
        <description>PCIE2:PXE IP4 Mellanox Network Adapter - EC:0D:9A:CC:1E:46</description>
        <instance>1</instance>
       </boot_device>
       <boot_device> <description>PCIE2:PXE IP4 Mellanox Network Adapter - EC:0D:9A:CC:1E:47</description>
        <instance>1</instance> </boot_device> </boot_order>
    4. Check the restored /etc/fstab file and remove any reference to /EXAVMIMAGES.
      # cd /mnt/etc
      Comment out any line that references /EXAVMIMAGES.
  13. Detach the diagnostics.iso file. This can be done by clicking Disconnect on the ILOM web interface console. Connect was clicked in Method 1step vii.2 managing-oracle-vm-guests-kvm.html#GUID-0C6AB3FE-CDC9-4AE2-B066-B644D9E6BA02__OL_RPR_DHV_KLBto attach the DVD ISO image.
  14. Unmount the restored /dev/sda1 partitions so /dev/sda1 can be remounted on /boot.
    # umount /mnt/boot/efi
    # umount /mnt/boot
    # umount /mnt/boot/efi
    # umount /mnt/boot
    # umount /mnt/home
    # umount /mnt/var/log/audit
    # umount /mnt/var/log/
    # umount /mnt/var 
    # umount /mnt/var
    # umount /mnt/EXAVMIMAGES/
    # umount /mnt
    # umount /root/mnt
  15. Restart the system.
    # reboot
    When the server comes back up, complete the following steps.
  16. Convert to Eighth Rack, if required.

    If the recovery is on an Oracle Exadata Database Machine Eighth Rack, then perform the procedure described in Configuring Oracle Exadata Database Machine Eighth Rack Oracle Linux Database Server After Recovery.

  17. Mount the backup NFS server that holds the guest storage repository (/EXAVMIMAGES) backup to restore the /EXAVMIMAGES file system. This backup procedure is described in section Method 1: Back Up All of the KVM Guests
    # mkdir -p /root/mnt
    # mount -t nfs -o ro,intr,soft,proto=tcp,nolock nfs_ip:/location_of_backup /root/mnt
  18. Restore the /EXAVMIMAGES file system.

    Use the command below to restore all guests:

    # tar -Spxvf /root/mnt/exavmimages.tar -C /EXAVMIMAGES
    Use the command below to restore a single guest from the backup.
    # tar -Spxvf /root/mnt/backup-of-exavmimages.tar -C /EXAVMIMAGES EXAVMIMAGES/<user-domain-name-to-be-restored>
  19. Bring up each guest
    # /opt/exadata_ovm/vm_maker --start-domain <guestname>
    At this point all the guests should come up along with Oracle Grid Infrastructure and the Oracle Database instances.
6.18.2.2 Re-imaging a KVM Host and Restoring the Guests from Backup

This procedure re-images the management domain and reconstructs all the user domains from a backup of the KVM guests. There is no backup of the KVM host available in this scenario.

This procedure re-images the KVM host and reconstructs all the guests.

The following procedure can be used when the KVM host is damaged beyond repair and no backup exists for the KVM host, but there is a backup available of the storage repository (/EXAVMIMAGES file system) containing all the guests.

  1. Re-image the KVM host with the image used in the other hosts in the rack using the procedure described in Re-Imaging the Oracle Exadata Database Server.
  2. If the recovery is on Oracle Exadata Database Machine eighth rack, then perform the procedure described in Configuring Oracle Exadata Database Machine Eighth Rack Oracle Linux Database Server After Recovery.
  3. Create bondedinterface and the bonded network bridge. The command below uses vmbondeth0 as the name of the network bridge.
    # /opt/exadata_ovm/vm_maker --add-bonded-bridge vmbondeth0 --first-slave eth1 --second-slave eth2
  4. Mount the backup NFS server containing the guest backups
    # mkdir -p /remote_FS
    # mount -t nfs -o ro,intr,soft,proto=tcp,nolock nfs_ip:/location_of_backup /remote_FS
  5. Restore the /EXAVMIMAGES file system.
    # tar -Spxvf /remote_FS/backup-of-exavmimages.tar -C /EXAVMIMAGES
    Note:The restore process of storage repository restores the guest specific files (files under /EXAVMINAGES/GuestImages/guest/) as regular files and not as xfs reflinks, which is what these files in the storage repository were originally at the time of the guest creation. Consequently, the space usage in /EXAVMINAGES may go up after the restoration process when compared to the original space usage at the time of the backup.
  6. Restore the guest specific files for the KVM hypervisor
    # /usr/bin/cp -Rp /remote_FS/XML/* /etc/libvirt/qemu/
    Note: The files under /remote_FS/XML referenced above will be created by the backup procedure described in Method 1: Back Up All of the KVM Guests OR Method 2: Back Up an Individual Guest.
  7. Restart the libvirtd service.
    # systemctl stop libvirtd
    # systemctl start libvirtd
  8. Restart each user domain.
    # /opt/exadata_ovm/vm_maker --start-domain <guestname>
    At this point all the guests should start along with the Oracle Grid Infrastructure and the database instances.
6.18.2.3 Restoring and Recovering Guests from Snapshot Backups

This procedure can be used to restore lost or damaged files of a KVM guest using a snapshot-based backup of the guest taken from inside the guest.

Use this procedure to restore lost/damaged files of a KVM guest by using a snapshot-based backup created from within the guest as described in section Method 3: Back Up a Guest Internally .

Log in to the KVM guest as the root user.

  1. Mount the backup NFS server to restore the damaged or lost files.
    # mkdir -p /root/mnt
    # mount -t nfs -o ro,intr,soft,proto=tcp,nolock nfs_ip:/location_of_backup /root/mnt
  2. Extract the damaged or lost files from the backup to a staging area. Prepare a staging area to hold the extracted files. The backup logical volume LVDbSys2 may be used for this:
     # mkdir /backup-LVM 
    # mount /dev/mapper/VGExaDb-LVDbSys2 /backup-LVM
    # mkdir /backup-LVM/tmp_restore 
    # tar -pjxvf /root/mnt/tar_file_name -C /backup-LVM/tmp_restore absolute_path_of_file_to_be_restored
  3. Restore the damaged or lost files from the temporary staging area as needed.
  4. Restart the KVM guest if needed.

6.19 Removing a Guest

You can remove a guest in Oracle Linux KVM using either OEDACLI or the vm_maker utility.

6.19.1 Removing a Guest Using vm_maker

You can use the vm_maker utility to remove a guest.

The following procedure removes a guest from a cluster. If the guest is not part of a cluster, then you can skip the commands related to Oracle Clusterware.

  1. If your cluster uses Quorum failure groups, these need to be deleted first.
    1. List the existing quorum devices.
      ~]# /opt/oracle.SupportTools/quorumdiskmgr --list --target
    2. Delete the quorum devices.
      ~]# /opt/oracle.SupportTools/quorumdiskmgr --delete --target
  2. Remove the database instance on the guest from the cluster node.
    Refer to Deleting Instances from Oracle RAC Databases for the instructions.
  3. Remove the guest node from the cluster.
  4. Delete the guest using vm_maker.
    # /opt/exadata_ovm/vm_maker --remove-domain guest_name
  5. View the list of existing guests on the KVM host.
    # /opt/exadata_ovm/vm_maker --list-domains

    The guest you just removed should not be listed.

6.19.2 Removing a Guest from an Oracle RAC Cluster Using OEDACLI

You can use OEDACLI to remove a guest from a cluster.

The following procedure removes a guest from a cluster. If the guest is not part of a cluster, then you can skip the cluster-related commands.

  1. Load the XML configuration file (es.xml for example) into OEDACLI.
    ./oedacli -c full_path_to_XML_file
  2. Use the following OEDACLI commands to delete the database instance from the cluster node.
    DELETE GUEST WHERE srcname=guest_FQDN stepname=ADD_INSTANCE
    SAVE ACTION
    MERGE ACTIONS
    DEPLOY ACTIONS
  3. Use the following OEDACLI commands to delete the guest node from the cluster.
    DELETE GUEST WHERE srcname=guest_FQDN stepname=ADD_NODE
    SAVE ACTION
    MERGE ACTIONS
    DEPLOY ACTIONS
  4. Use the following OEDACLI commands to remove connectivity to the storage servers and delete the users on the guest.
    DELETE GUEST WHERE srcname=guest_FQDN stepname=CELL_CONNECTIVITY
    SAVE ACTION
    DELETE GUEST WHERE srcname=guest_FQDN stepname=CREATE_USERS
    SAVE ACTION
    MERGE ACTIONS
    DEPLOY ACTIONS
  5. Use the following OEDACLI commands to remove the guest.
    DELETE GUEST WHERE srcname=guest_FQDN stepname=CREATE_GUEST
    SAVE ACTION
    MERGE ACTIONS
    DEPLOY ACTIONS
  6. Save the updated configuration information.
    Specify the full path to a directory where the updated configuration information will be saved.
    SAVE FILES LOCATION=output_directory
  7. View the list of existing guests on the KVM host.
    # /opt/exadata_ovm/vm_maker --list-domains

    The guest you just removed should not be listed.

6.20 Using Client Network VLAN Tagging with Oracle Linux KVM

This topic describes the implementation of tagged VLAN interfaces for the client network in conjunction with Oracle Linux KVM.

Oracle databases running in Oracle Linux KVM guests on Oracle Exadata Database Machine are accessed through the client Ethernet network defined in the Oracle Exadata Deployment Assistant (OEDA) configuration tool. Client network configuration in both the KVM host and guests is done automatically when the OEDA installation tool creates the first guest during initial deployment.

The following figure shows a default bonded client network configuration:

Figure 6-1 NIC Layout in an Oracle Virtual Environment

Description of Figure 6-1 follows
Description of "Figure 6-1 NIC Layout in an Oracle Virtual Environment"

The network has the following configuration:

  1. In the KVM host, eth slave interfaces (for example, eth1 and eth2, or eth4 and eth5) that allow access to the guest client network defined in OEDA are discovered, configured, and brought up, but no IP is assigned.

  2. In the KVM host, bondeth0 master interface is configured and brought up, but no IP is assigned.

  3. In the KVM host, bridge interface vmbondeth0 is configured, but no IP is assigned.

  4. In the KVM host, one virtual backend interface (VIF) per guest that maps to that particular guest's bondeth0 interface is configured and brought up, but no IP is assigned. These VIFs are configured on top of the bridge interface vmbondeth0, and the mapping between the KVM host VIF interface and its corresponding guest interface bondeth0 is defined in the guest configuration file called vm.cfg, located in /EXAVMIMAGES/GuestImages/guest name.

For default installations, a single bondeth0 and a corresponding vmbondeth0 bridge interface is configured in the KVM host as described above. This bondeth0 interface is based on the default Access VLAN. The ports on the switch used by the slave interfaces making up bondeth0 are configured for Access VLAN.

Using VLAN Tagging

If there is a need for virtual deployments on Exadata to access additional VLANs on the client network, such as enabling network isolation across guests, then 802.1Q-based VLAN tagging is a solution. The following figure shows a client network configuration with VLAN tagging.

Figure 6-2 NIC Layout for Oracle Virtual Environments with VLAN Tagging

Description of Figure 6-2 follows
Description of "Figure 6-2 NIC Layout for Oracle Virtual Environments with VLAN Tagging"

Note:

Commencing with the March 2020 OEDA release, the bridge names now have the form vmbethXY.VLANID, where X and Y are the numeric identifiers associated with the slave interface, and VLANID is the VLAN ID.

This avoids a potential naming conflict that could previously occur in some cases.

For example, under the new naming scheme the bridges in the previous diagram would be named vmbeth45.3005, vmbeth45.3006, and vmbeth45.3007.

6.21 Using Exadata Secure RDMA Fabric Isolation with Oracle Linux KVM

This topic describes the implementation of Exadata Secure RDMA Fabric Isolation in conjunction with Oracle Linux KVM.

Secure Fabric enables secure consolidation and strict isolation between multiple tenants on Oracle Exadata Database Machine. Each tenant resides in their own dedicated virtual machine (VM) cluster, using database server VMs running on Oracle Linux KVM.

With Secure Fabric, each database cluster uses a dedicated network partition and VLAN ID for cluster networking between the database servers, which supports Oracle RAC inter-node messaging. In this partition, all of the database servers are full members. They can communicate freely within the partition but cannot communicate with database servers in other partitions.

Another partition, with a separate VLAN ID, supports the storage network partition. The storage servers are full members in the storage network partition, and every database server VM is also a limited member. By using the storage network partition:

  • Each database server can communicate with all of the storage servers.
  • Each storage server can communicate with all of the database servers that they support.
  • Storage servers can communicate directly with each other to perform cell-to-cell operations.

Figure 6-3 Secure Fabric Network Partitions

Description of Figure 6-3 follows
Description of "Figure 6-3 Secure Fabric Network Partitions"

To support the cluster network partition and the storage network partition, each database server VM is plumbed with 4 virtual interfaces:

  • clre0 and clre1 support the cluster network partition.
  • stre0 and stre1 support the storage network partition.

    Corresponding stre0 and stre1 interfaces are also plumbed on each storage server.

On each server, the RoCE network interface card acts like a switch on the hypervisor, which performs VLAN tag enforcement. Since this is done at the KVM host level, cluster isolation cannot be bypassed by any software exploits or misconfiguration on the database server VMs.

In this release, you can only enable Secure Fabric as part of the initial system deployment using Oracle Exadata Deployment Assistant (OEDA). You cannot enable Secure Fabric on an existing X8M system without wiping the system and re-deploying it using OEDA. When enabled, Secure Fabric applies to all servers and clusters that share the same RoCE Network Fabric.

6.22 Using Oracle EXAchk in Oracle Linux KVM Environments

Oracle EXAchk version 12.1.0.2.2 and higher supports virtualization on Oracle Exadata Database Machine.

6.22.1 Running Oracle EXAchk in Oracle Linux KVM Environments

To perform the complete set of Oracle EXAchk audit checks in an Oracle Exadata Database Machine Oracle Linux KVM environment, Oracle EXAchk must be installed in and run from multiple locations.

  1. Run Oracle EXAchk from one KVM host.
  2. Run Oracle EXAchk from one guest in each Oracle Real Application Clusters (Oracle RAC) cluster running in Oracle Linux KVM.

For example, an Oracle Exadata Database Machine Quarter Rack with two database servers containing 4 Oracle RAC clusters (2 nodes per cluster for a total of 8 guests across both database servers) requires running Oracle EXAchk five separate times, as follows:

  1. Run Oracle EXAchk in the first guest for the first cluster.

  2. Run Oracle EXAchk in the first guest for the second cluster.

  3. Run Oracle EXAchk in the first guest for the third cluster.

  4. Run Oracle EXAchk in the first guest for the fourth cluster.

  5. Run Oracle EXAchk in the first KVM host.

6.22.2 Audit Checks Performed by Oracle EXAchk

Oracle EXAchk runs different audit checks on the KVM host and the guests.

When you install and run Oracle EXAchk on the KVM host, it performs the following hardware and operating system level checks:

  • Database servers (KVM hosts)
  • Storage servers
  • RDMA Network Fabric
  • RDMA Network Fabric switches

When you install and run Oracle EXAchk on the guest, it performs operating system checks for guests, and checks for Oracle Grid Infrastructure and Oracle Database.

6.22.3 Oracle EXAchk Command Line Options for Oracle Exadata Database Machine

Oracle EXAchk requires no special command line options. It automatically detects that it is running in an Oracle Exadata Database Machine Oracle Linux KVM environment. However, you can use command line options to run Oracle EXAchk on a subset of servers or switches.

Oracle EXAchk automatically detects whether it is running in a KVM host or guest and performs the applicable audit checks. For example, in the simplest case, you can run Oracle EXAchk with no command line options:

./exachk

When Oracle EXAchk is run in the KVM host, it performs audit checks on all database servers, storage servers, and RDMA Network Fabric switches accessible through the RDMA Network Fabric network.

To run Oracle EXAchk on a subset of servers or switches, use the following command line options:

Options

  • -clusternodes: Specifies a comma-separated list of database servers.

  • -cells: Specifies a comma-separated list of storage servers.

In release 19.3, support was added for RoCE Network Fabric. There are no checks for InfiniBand Network Fabric switches and fabrics when run on Exadata X8M systems. If the -profile switch is used, then Oracle EXAchk throws an error saying it is not a supported profile.

Example 6-1 Running Oracle EXAchk on a Subset of Nodes and Switches

For example, for an Oracle Exadata Database Machine Full Rack where only the first Quarter Rack is configured for virtualization, but all components are accessible through the RDMA Network Fabric network, you can run a command similar to the following from the database server dm01adm01:

./exachk -clusternodes dm01adm01,dm01adm02
         -cells dm01celadm01,dm01celadm02,dm01celadm03