Recover a KVM Guest Stored on Exascale Volumes

6.19.2.4 Recover a KVM Guest Stored on Exascale Volumes

Recover an Oracle Linux KVM guest whose virtual disks are stored on Exascale volumes from a complete backup set.

Use this procedure with a complete backup set created using the procedure described in Back Up a KVM Guest Stored on Exascale Volumes.

The procedure restores each persistent-volume backup as a new Exascale volume, adds the restored volumes to a new guest volume group, attaches the volumes to the target KVM host, mounts the restored configuration-files volume, updates the guest XML definition, redefines the guest, and starts the guest.

Before you begin:

Perform this procedure as an administrator (root) on the KVM host where you want to recover the guest.
Ensure that the KVM host is operational and configured as an Exascale Direct Volume (EDV) client.
Ensure that you have a wallet for connecting to Exascale and sufficient privileges to manage the Exascale volumes that host the guest image files. Ask your Exascale administrator to provide the wallet and grant you sufficient privileges. See also Provision a User and Associated Wallet in Oracle Exadata Exascale User's Guide.
Identify the guest backup set. The backup set resides in a unique Exascale location, using a path that contains the guest name and backup timestamp, such as:
```
@backupvault/guest-backups/guest-name/backup-timestamp/
```
For example:
```
@backupvault/guest-backups/exa01adm01vm01.example.com/20260826T120000Z/
```
Identify an Exascale vault for the restored operational guest volumes. All restored volumes that belong to the guest volume group must reside in the same vault.

Assumptions:

This procedure assumes that the backup set is complete and was created in accordance with the companion backup procedure.
For each volume backup, this procedure assumes that the backup’s name attribute combines the original Exascale volume name and source volume ID in the following format:
```
original-volume-name_source-volume-ID
```
This procedure relies on the guest-name and backup-timestamp values that were used by the backup procedure and encoded into the backup location.

For each volume backup, the backup name preserves the original Exascale volume name and ID. During recovery, these values are used to restore the backup into a new Exascale volume using the original volume name, construct a new EDV device name using the new volume ID, and update the guest XML to use the new /dev/exc/ path.

Note:

Restoring a volume backup creates a new Exascale volume. The restored volume is independent of the persistent volume backup and the original source volume.

Identify and verify the guest backup set.
Connect to the Exascale command line interface (ESCLI) using your Exascale wallet:
```
# escli --wallet wallet-location
```
In ESCLI, list the contents of the selected Exascale backup location:
```
@> ls -l @backupvault/guest-backups/guest-name/backup-timestamp/
```
List the Exascale volume backups and identify the backups whose destination is the selected backup-set location:
```
@> lsvolumebackup --filter destination=@backupvault/guest-backups/guest-name/backup-timestamp/* --attributes id,name,volume,destination,state
```
Verify that every required volume backup is in the AVAILABLE state. Do not combine volume backups from different backup-set locations.

Stop the guest, if the guest still exists on the KVM host.

Display the guest state:

# /opt/exadata_ovm/vm_maker --list --domain guest-name --detail

If the guest is running, stop it:

# /opt/exadata_ovm/vm_maker --stop-domain guest-name

If the guest does not stop gracefully, stop it forcefully or destroy it if necessary:

# /opt/exadata_ovm/vm_maker --stop-domain guest-name --force

# /opt/exadata_ovm/vm_maker --stop-domain guest-name --destroy

Prepare the existing guest configuration-files mount, if present.
Check whether the guest configuration-files volume is mounted:
```
# mountpoint /EXAVMIMAGES/GuestImages/guest-name
```
If it is mounted, unmount it:
```
# umount /EXAVMIMAGES/GuestImages/guest-name
```
Check whether /etc/fstab contains an entry for the guest configuration-files mount:
```
# grep -F '/EXAVMIMAGES/GuestImages/guest-name' /etc/fstab
```
If an entry exists, edit /etc/fstab and remove the entry.
Remove volume attachments and rename the existing guest volume group, if present.
Guest volume groups typically use the following name format:
```
sysvolgrp_short-guest-name
```
For example:
```
sysvolgrp_exa01adm01vm01
```
In ESCLI, check whether a volume group exists that is still associated with the guest:
```
@> lsvolumegroup --filter name=sysvolgrp_short-guest-name --detail
```
If the group exists, record the associated volumes.

Then, for each volume, locate and remove the associated volume attachment:
```
@> lsvolumeattachment --filter volume=volume-ID
@> rmvolumeattachment attachment-ID
```
After all associated volume attachments are removed, rename the volume group:
```
@> chvolumegroup existing-volume-group-ID --attributes name=sysvolgrp_short-guest-name_saved_timestamp
```
Build and validate the volume recovery mapping.
Display the backup name and source volume ID for each volume backup:
```
@> lsvolumebackup --filter destination=@backupvault/guest-backups/guest-name/backup-timestamp/* --attributes id,name,volume
```
The backup name contains the original volume name and ID. The volume attribute is the original source volume ID.

Derive the original volume name by removing the trailing underscore and source volume ID from the backup name. For example:
```
backup name:           exa01adm01vm01_sys_vol0019_acb70712ce824e8fbfef0205a05dd5fc
source volume ID:      vol0019_acb70712ce824e8fbfef0205a05dd5fc
original volume name:  exa01adm01vm01_sys
```
Record the following values for every volume backup:
- Volume backup ID
- Source volume ID
- Original volume name
Identify the guest configuration-files volume. For a standard Exadata guest, the volume name ends with _cfg.

Note:

If a backup name does not end with the source volume ID reported by the volume backup resource, or if the mapping is otherwise ambiguous, stop the recovery and investigate the backup set.
Restore every persistent volume backup into your selected operational vault.
For each volume backup, create a new Exascale volume using the original volume name derived earlier:
```
@> mkvolume --attributes volumeBackup=volume-backup-ID,vault=operational-vault,name=original-volume-name
```
Record the new restored volume ID returned by each command.

Repeat the operation for every volume backup in the selected guest backup set.
Wait for every restored volume to become available.
Monitor each restored volume:
```
@> lsvolume restored-volume-ID --attributes state
```
While the restore is in progress, the volume reports RESTORING. After successful completion, the volume reports AVAILABLE.

Do not continue until all the restored volumes are AVAILABLE.
Create a new volume group for the recovered guest.
Create the standard guest volume group:
```
@> mkvolumegroup sysvolgrp_short-guest-name
```
Record the new volume group ID returned by mkvolumegroup.
Add every restored volume to the new guest volume group.
For each restored volume:
```
@> chvolume restored-volume-ID --attributes volumeGroup=restored-volume-group-ID
```
Verify the resulting group membership:
```
@> lsvolume --filter volumeGroup=restored-volume-group-ID --attributes id,name,state,vault
```
Verify that the number of restored volumes matches the number of persistent volume backups in the selected backup set. Also, compare the returned names with the original volume names recorded earlier.
Identify the EDV initiator and associated Exascale user for the target KVM host.
List the EDV initiators:
```
@> lsinitiator --attributes id,hostName,user
```
Identify the entry whose hostName corresponds to the target KVM host. Record the initiator ID and associated Exascale user.
Ensure that the EDV initiator user can access every restored volume.
Display the owners of each restored volume:
```
@> lsvolume --filter volumeGroup=restored-volume-group-ID --attributes owners
```
If the Exascale user associated with the EDV initiator is not already an owner, add it:
```
@> chvolume restored-volume-ID --attributes owners=+initiator-user
```
Construct a new EDV device name for every restored volume.
Construct each new device name by concatenating the original volume name, an underscore, and the new restored volume ID:
```
new-device-name=original-volume-name_restored-volume-ID
```
For example:
```
original volume name: exa01adm01vm01_sys
restored volume ID:   vol0042_7dc4614e603841289115964a8c4ba60c
new device name:      exa01adm01vm01_sys_vol0042_7dc4614e603841289115964a8c4ba60c
```
Record the original device name and new device name for every restored volume.

Note:

This step preserves the device-naming relationship expected by the companion backup procedure. A later backup of the recovered guest can therefore use the same recovery method.
Attach every restored volume to the target KVM host using its new device name.
For each restored volume:
```
@> mkvolumeattachment restored-volume-ID new-device-name --attributes initiator=initiator-ID
```
The attachment creates the corresponding device on the KVM host.
```
/dev/exc/new-device-name
```
Verify the restored EDV devices on the KVM host.
Wait for device discovery and list the devices:
```
# udevadm settle
# ls -l /dev/exc/
```
Verify that all the expected /dev/exc/new-device-name devices exist and record them for later use in the guest definition.
Mount the restored guest configuration-files volume and set its persistent mount entry.
Use the mapping created earlier to identify the new device name for the volume whose original volume name ends with _cfg.

Create the mount point if necessary and mount the restored configuration-files volume:
```
# mkdir -p /EXAVMIMAGES/GuestImages/guest-name
# mount /dev/exc/new-cfg-device-name /EXAVMIMAGES/GuestImages/guest-name
```
Verify the mount:
```
# mountpoint /EXAVMIMAGES/GuestImages/guest-name
```
Add a persistent mount entry for the restored guest configuration-files volume into /etc/fstab. Use the entry saved in /EXAVMIMAGES/GuestImages/guest-name/configbackup/guest-name.fstab, replacing /dev/exc/old-cfg-device-name with /dev/exc/new-cfg-device-name.

To enable the appropriate mount options, unmount and remount the volume:
```
# umount /EXAVMIMAGES/GuestImages/guest-name
# systemctl daemon-reload
# mount /EXAVMIMAGES/GuestImages/guest-name
```
Retrieve the backup-time guest XML definition.
Verify the backup-time guest XML definition, which is stored on the restored configuration-files volume:
```
# cat /EXAVMIMAGES/GuestImages/guest-name/configbackup/guest-name.xml
```
Create a working copy of the backup-time XML:
```
# cp -p /EXAVMIMAGES/GuestImages/guest-name/configbackup/guest-name.xml /var/tmp/guest-name.restore.xml
```
Update the guest XML definition to reference the new EDV device names.
Edit the working copy of the guest XML definition created in the previous step.

For every EDV disk in the guest XML definition, replace the original device path (/dev/exc/...) with the corresponding new device path using the new device name you constructed earlier.

For example:
```
old device path: /dev/exc/exa01adm01vm01_sys_vol0019_acb70712ce824e8fbfef0205a05dd5fc
new device path: /dev/exc/exa01adm01vm01_sys_vol0042_7dc4614e603841289115964a8c4ba60c
```
Verify the resulting EDV paths:
```
# grep '/dev/exc/' /var/tmp/guest-name.restore.xml
```
Verify that every referenced path exists under /dev/exc/.
Define the recovered guest in the KVM hypervisor.
Define or redefine the guest using the updated XML:
```
# virsh define /var/tmp/guest-name.restore.xml
```
Configure autostart for the recovered guest.
With autostart enabled, the guest starts automatically when the KVM host starts.

Disable or enable autostart as desired by running one of the following commands:
```
# /opt/exadata_ovm/vm_maker --autostart guest-name --disable
# /opt/exadata_ovm/vm_maker --autostart guest-name --enable
```
Verify the recovered guest configuration before starting the guest.
Display the guest definition:
```
# /opt/exadata_ovm/vm_maker --list --domain guest-name --detail
```
Display the guest disk images:
```
# /opt/exadata_ovm/vm_maker --list --disk-image --domain guest-name --detail
```
Verify that every expected guest disk references the corresponding newly restored /dev/exc/ device.
Start and validate the recovered guest.
Start the guest:
```
# /opt/exadata_ovm/vm_maker --start-domain guest-name
```
Verify that the guest starts successfully, all expected disks are visible, required file systems mount, and the required Oracle Grid Infrastructure and Oracle Database resources operate as expected, if applicable.
Remove obsolete items associated with the recovered guest.
- If it still exists, remove the Exascale volume group formerly associated with the guest. The volume attachments for the volumes in this group must already have been removed, as described earlier. Inspect the volume group to identify its volumes, then remove the volumes and finally remove the volume group.
- Remove any temporary working files that you created in /var/tmp/ on the KVM host.

The guest uses new Exascale volumes restored from the selected persistent volume backups. Each restored volume retains the original Exascale volume name but has a new volume ID and a corresponding new EDV device name. The guest XML and the configuration-files mount reference the new EDV device names.

Related Topics

About the vm_maker Utility

Parent topic: KVM System Recovery Scenarios