Migrating from Virtualized Platform to KVM on Oracle Database Appliance

17 Migrating from Virtualized Platform to KVM on Oracle Database Appliance

Understand how you can migrate your Oracle Database Appliance deployment from virtualized platform to KVM.

About Migrating Oracle Database Appliance from Virtualized Platform to KVM
Understand how you can migrate your virtualized platform deployment to KVM.
Migrating from Virtualized Platform to KVM on Oracle Database Appliance
Understand the procedure to migrate from Virtualized Platform to KVM on Oracle Database Appliance.

About Migrating Oracle Database Appliance from Virtualized Platform to KVM

Understand how you can migrate your virtualized platform deployment to KVM.

You can migrate your Oracle Database Appliance virtualized platform deployment to KVM using the Data Preserving Reprovisioning feature.

Migration Considerations

Following are migration considerations:

Virtualized platform (Oracle VM) local repositories are not migrated. Any objects in the local repositories, such as VM or vdisks, must be backed up and manually migrated after the migration.
Mount point for repositories (Oracle ACFS file system) is changed from /u01/app/sharedrepo to /u05/app/sharedrepo.
VM templates are migrated as offline VMs. They can be used as templates to clone VMs using the command:
```
 odacli clone-vm -n source_vm_name -cn cloned_vm_name
```
If not needed after the migration, then these VM templates can be deleted using the command odacli delete-vm -n vm_name.
CPU pools odaBaseCpuPool and default-unpinned-pool are not migrated.
The net1 bridge on dom0, usually used as the public network on OVM and priv1, used as the private network for the ODA_BASE are not migrated to KVM vnetwork.
VM with attached VLAN without bridge is migrated as pubnet vnetwork.
Manual customizations made directly in vm.cfg are not migrated.
By default, VM after migration uses virtio. Earlier Linux versions VM such as Oracle Linux 5 which do not have the required virtio support fail to bootstrap correctly. You can change the bus type from virtio to IDE, and then restart the VM.
```
odacli modify-vm -n vm_name -dev bus=ide
```
To revert it back to the default virtio:
```
odacli modify-vm -n vm_name -dev bus=virtio
```
This issue can also occur on Microsoft Windows VMs that do not have virtio installed.
If migration fails while migrating any infrastructure resources such as repo, cpu pool, vlan, vdisk and vmtemplate), the created resources are reverted back. After the issue is fixed, rerun the command odacli migrate-ovm2kvm for migration.
If migration fails while migrating VMs, the migration continues and an error is displayed when you run the command odacli describe-ovm2kvm for each failed VM migration.
After migration, the actual device name inside the VM could change. For example, /dev/xvda on the virtualized platform deployment could change to /dev/vda after migration to KVM. If the device name, for example, /dev/xvda is listed in /etc/fstab inside the VM, then the VM can fail to start due to device name change after the migration. If the virtualized platform deployment uses the device name in /etc/fstab, then rename with the label or UUID before migration. For more information about using label or UUID in the file system mount table, that is, /etc/fstab, then refer to the Oracle Linux 7 Managing File Systems Guide at https://docs.oracle.com/en/operating-systems/oracle-linux/7/fsadmin/ol7-fsadmin.html#ol7-s1-fsadmin.
The network configuration of the VM could change after the migration. If you cannot connect to the VM after migration, login to VM from the console using the command virsh console vm_name and review and reconfigure the network, if necessary. The configuration steps are specific to the operating system. For details, refer to the operating system documentation.
Before migrating a Microsoft Windows VM, install the virtio driver for Windows. Refer to the My Oracle Support Note 2773840.1 Steps for MS Windows guests on ODA Bare Metal for the configuration procedure.

Steps in the Migration Process

Following are the steps in this process:

Run the odaupgradeutil.sh utility.
Reimage the appliance on both nodes.
Run the command odacli restore-node -g to reprovision with Oracle Grid infrastructure and odacli restore-node -d to reprovision the saved databases.
Run the odacli migrate-ovm2kvm command to migrate from virtualized platform to KVM.
Run the command odacli describe-ovm2kvm and examine the migration report for any errors. Fix the issues and then retry the operation.

Parent topic: Migrating from Virtualized Platform to KVM on Oracle Database Appliance

Migrating from Virtualized Platform to KVM on Oracle Database Appliance

Understand the procedure to migrate from Virtualized Platform to KVM on Oracle Database Appliance.

WARNING:

Do not run cleanup.pl at any time during the migration process. That will erase all the Oracle ASM disk groups on the storage and you will not be able to reprovision your Oracle Database Appliance system with all the Oracle virtualized platform resources intact.

Note:

After migration, the actual device name inside the VM could change. For example, /dev/xvda on the virtualized platform deployment could change to /dev/vda after migration to KVM. If the device name, for example, /dev/xvda is listed in /etc/fstab inside the VM, then the VM can fail to start due to device name change after the migration. If the virtualized platform deployment uses the device name in /etc/fstab, then rename with the label or UUID before migration. For more information about using label or UUID in the file system mount table, that is, /etc/fstab, then refer to the Oracle Linux 7 Managing File Systems Guide at https://docs.oracle.com/en/operating-systems/oracle-linux/7/fsadmin/ol7-fsadmin.html#ol7-s1-fsadmin.

Note:

Run odaupgradeutil on ODA_BASE at the first node (node0) first. After steps 3-5 complete successfully on node0, run steps 3-5 on ODA_BASE at the second node (node1).

Follow these steps. Run the commands on both nodes.

Download the Oracle Database Appliance odaupgradeutil utility to ODA_BASE.
Unzip the software and save it to /opt/oracle directory.
For example:
```
cd /opt/oracle
unzip p33594115_1913000_Linux-x86-64.zip
```
The utility is saved to the /opt/oracle/odaupgradeutil location.

The supported Oracle Database appliance releases for the migration from virtualized platform to KVM are 19.13.0.0.0, 18.8.0.0.0, 12.2.1.4.0, and 12.1.2.12.0.
Run pre-checks to evaluate whether the system is ready for upgrade. Ensure that the shared repositories on the system are online so that odaupgradeutil can collect the required information on the OVM platform.
```
$ ./odaupgradeutil run-prechecks
```
Note:
The commands for odaupgradeutil must be run on ODA_BASE and not dom0.

The command runs pre-checks related to Oracle Grid Infrastructure, databases, OAK, firmware, and other components. These checks determine whether the current node of the Oracle Database Appliance can be successfully detached. If there are failures reported, then review the failures in the report and take appropriate action. The precheck report is generated in the location /opt/oracle/oak/restore/prechecks/precheck_report.json. The log of the precheck operation is saved at /opt/oracle/oak/restore/log/odaupgradeutil_prechecks_timestamp.log. The odaupgradeutil utility logs are stored at opt/oracle/oak/restore/log.
For example, run prechecks on the first node:
```
[root@node0 odaupgradeutil]# ./odaupgradeutil run-prechecks
########################## ODAUPGRADEUTIL - PRECHECKS - BEGIN ##########################
Please check /opt/oracle/oak/restore/log/odaupgradeutil_prechecks_25-10-2021_14:54:48.log for details.
System Version precheck...BEGIN
System Version precheck...PASSED
Required Files precheck...BEGIN
Required Files precheck...PASSED
OAK precheck...BEGIN
OAK precheck...PASSED
ASM precheck...BEGIN
ASM precheck...PASSED
Audit Files precheck...BEGIN
Need to discover DB homes
Get Database homes...BEGIN
  Database Home: /u01/app/oracle/product/19.0.0.0/dbhome_1, Database Home Name: OraDb19000_home1, Database Home Version: 19.9.0.0.201020
Get Database homes...SUCCESS
DB home discovery...DONE
Audit Files precheck...WARNING
Custom rpms precheck...BEGIN
Custom rpms precheck...PASSED
OVM Repo precheck...BEGIN
OVM Repos precheck...PASSED
OVM Local Resources precheck...BEGIN
OVM Local Resources precheck...WARNING
########################## ODAUPGRADEUTIL - PRECHECKS - END ##########################
Use 'odaupgradeutil describe-precheck-report [-j]' to view the precheck report.
```

Review the prechecks report.

[root@node0 odaupgradeutil]# ./odaupgradeutil describe-precheck-report

For example:

[root@node0 odaupgradeutil]# ./odaupgradeutil describe-precheck-report
COMPONENT       STATUS  MESSAGE                                                               ACTION
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
SYSTEM VERSION  PASSED  PASSED                                                                NONE
 
REQUIRED FILES  PASSED  PASSED                                                                NONE
 
OAK             PASSED  PASSED                                                                NONE
 
ASM             PASSED  PASSED                                                                NONE
 
AUDIT FILES     WARNING Audit files found under ['/u01/app/oracle/product/19.0.0.0/dbhome_    These files will be lost after reimage, advise is to backup if nec
                        1/rdbms/audit', '/u01/app/oracle/admin', '/var/log']                  essary
 
OSRPMS          PASSED  PASSED                                                                NONE
 
OVMREPOS        PASSED  PASSED                                                                NONE
 
OVMLOCALRES     WARNING The following resources: ol7localvm ol7localvmare on local Repos      Please backup these resources and restore them manually

If there are failures in the precheck report, take corrective action as suggested in the ACTION column. After fixing the failures, rerun the precheck as explained in step 3 in this procedure. If there are no failures in the precheck report, then run the command to detach the node:

[root@node0 odaupgradeutil]# ./odaupgradeutil detach-node
########################## ODAUPGRADEUTIL - SAVECONF - BEGIN ##########################
Please check /opt/oracle/oak/restore/log/odaupgradeutil_saveconf_25-10-2021_15:04:22.log for details.
Setting up passwordless SSH login on node1...BEGIN
root@node1's password:
Setting up passwordless SSH login...SUCCESS
Setting up passwordless SSH login on node 0 DOM0(192.168.xx.xx)...BEGIN
The authenticity of host '192.168.16.24 (192.168.xx.xx)' can't be established.
RSA key fingerprint is SHA256:itz6YJz5MOTaPHiHwSlwT+SqygySeyaQOVAdKokbdOw.
RSA key fingerprint is MD5:b4:03:10:b6:07:2a:d2:e1:a6:cf:52:a6:b7:dd:54:6f.
Are you sure you want to continue connecting (yes/no)? yes
root@192.168.xx.xx's password:
Setting up passwordless SSH login...SUCCESS
Setting up passwordless SSH login on node 1 DOM0(192.168.xx.xx)...BEGIN
The authenticity of host '192.168.xx.xx (192.168.xx.xx)' can't be established.
RSA key fingerprint is SHA256:vRmREMGwtsDzLjDUqnuUEnK7jGRdT/wbcYk+AyI+xLg.
RSA key fingerprint is MD5:40:09:ab:26:f6:0b:53:ee:0e:ee:5d:50:2f:5e:d8:79.
Are you sure you want to continue connecting (yes/no)? yes
root@192.168.xx.xx's password:
Setting up passwordless SSH login...SUCCESS
Backup files to /opt/oracle/oak/restore/bkp...BEGIN
Backup files to /opt/oracle/oak/restore/bkp...SUCCESS
Get provision instance...BEGIN
Get provision instance...SUCCESS
Get network configuration...BEGIN
Get network configuration...SUCCESS
Get databases...BEGIN
  Database Name: gui
  Oracle Home: /u01/app/oracle/product/19.0.0.0/dbhome_1
Get databases...SUCCESS
Get Database homes...BEGIN
  Database Home: /u01/app/oracle/product/19.0.0.0/dbhome_1, Database Home Name: OraDb19000_home1, Database Home Version: 19.9.0.0.201020
  Get Unified Auditing for DB Homes
  Checking Unified Auditing for dbhome [/u01/app/oracle/product/19.0.0.0/dbhome_1]
  Unified Auditing is set to FALSE
Get Database homes...SUCCESS
Get Database storages...BEGIN
  Database Name: gui
        DATA destination: +DATA
        RECO destination: +RECO
        REDO destination: +RECO
        Flash Cache destination:
Get Database storages...SUCCESS
Get Volumes...BEGIN
Get Volumes...SUCCESS
Get Filesystems...BEGIN
Get Filesystems...SUCCESS
SAVECONF: SUCCESS
########################## ODAUPGRADEUTIL - SAVECONF - END ##########################
Saving OVM data
INFO 2021-10-25 15:05:14.534 - The script is running on Node 0
INFO 2021-10-25 15:05:14.537 - The script is running on ODA platform: X8_2_ODA_HA
INFO 2021-10-25 15:05:15.194 - create SSH equivalence login to remote node and DOM0
INFO 2021-10-25 15:05:15.196 - setting up passwordless SSH login to "remote node 1(192.168.xx.xx)"
INFO 2021-10-25 15:05:15.344 - SSH login to "192.168.xx.xx" already setup
 
INFO 2021-10-25 15:05:15.345 - setting up passwordless SSH login to "node 0 DOM0(192.168.xx.xx)"
INFO 2021-10-25 15:05:15.579 - SSH login to "192.168.xx.xx" already setup
 
INFO 2021-10-25 15:05:15.580 - setting up passwordless SSH login to "node 1 DOM0(192.168.xx.xx)"
INFO 2021-10-25 15:05:15.771 - SSH login to "192.168.xx.xx" already setup
 
INFO 2021-10-25 15:05:15.773 - collect data for odabase
INFO 2021-10-25 15:05:20.047 - backup ODA base vm.cfg files from DOM0
INFO 2021-10-25 15:05:20.515 - collect data for share repo
WARNING 2021-10-25 15:05:21.758 - Please backup the data in the following local repos:
odarepo1
odarepo2
INFO 2021-10-25 15:05:25.673 - collect data for vlan
INFO 2021-10-25 15:05:27.307 - backup network-script files from DOM0
INFO 2021-10-25 15:05:27.799 - collect data for vdisk
INFO 2021-10-25 15:05:33.913 - collect data for cpupool
INFO 2021-10-25 15:05:39.566 - collect data for vm
INFO 2021-10-25 15:05:46.278 - collect data for vm template
INFO 2021-10-25 15:05:50.558 - Fix the network interface name from OVM to BM
INFO 2021-10-25 15:05:50.562 - Interface "eth1" is used for provision
INFO 2021-10-25 15:05:50.572 - Change the network interface from "eth1" to "btbond1" for provision after migration
Warning: Permanently added '192.168.xx.xx' (ECDSA) to the list of known hosts.
OVM data has been saved
Stopping OVM resources
Stopping the Resources on the OVM platform
OVM resources has been stopped
Process share repo volume information
Process share repo volume information finish
########################## ODAUPGRADEUTIL - DETACHNODE - BEGIN ##########################
Please check /opt/oracle/oak/restore/log/odaupgradeutil_detachnode_25-10-2021_15:13:07.log for details.
Deconfigure databases...BEGIN
  Database Name: gui
  Local Instance: gui1
  Local Instance Status: RUNNING
  Stopping instance [gui1]...
  Removing instance [gui1] from CRS...
Deconfigure databases...SUCCESS
Deconfigure Grid Infrastructure...BEGIN
Deconfigure Grid Infrastructure...SUCCESS
Delete local node configuration from remote node 'node1'...BEGIN
Local Node 'node1' deleted from cluster. Cluster nodes now [node1]
DETACHNODE: SUCCESS
[CRITICAL] Server data archive file(s) generated at /opt/oracle/oak/restore/out . Please ensure the file(s) are copied outside the ODA system and preserved.
########################## ODAUPGRADEUTIL - DETACHNODE - END ##########################

This command saves all the metadata. The current node is detached from the Oracle Grid Infrastructure cluster by deinstalling database and Oracle Grid Infrastructure. The deinstallation does not affect Oracle ASM disk groups and the stored files such as datafiles, control files, and archive logs. Run all the steps in this procedure on each node separately, one node at a time. In high-availability systems, when the command is run on the first node, the services are brought down and software is deinstalled on that node. At this time, the services continue to run on second node, but all the VMs running on the share repositories are shut down. The database, for example, can still be connected to, and queries can be issued. The second node remains functional, until detached, at which point, there is a full downtime of the database and Oracle Grid Infrastructure.

On successful completion of the command on both nodes, the following zip files are generated:
For High-Availability systems, three zip files are generated:
```
/opt/oracle/oak/restore/out/serverarchive_node0_hostname.zip,
/opt/oracle/oak/restore/out/serverarchive_node1_hostname.zip, and 
/opt/oracle/oak/restore/out/serverarchive_cluster_name_common.zip
```
The serverarchive_node0_hostname.zip and serverarchive_node1_hostname.zip contain the file configure-firstnet.rsp. This file contains the values that you need to provide when running odacli configure-firstnet after reimaging the system.
Copy the files to a location outside of the Oracle Database Appliance system, to prepare the environment for reimage.
Download the Oracle Database Appliance release 19.15 bare metal ISO image, reimage the appliance as described in the topic Reimaging an Oracle Database Appliance Baremetal System, and plumb the network by running the odacli configure-firstnet command on both nodes.

WARNING:
Do not run cleanup.pl at any time during the migration process. That will erase all the Oracle ASM disk groups on the storage and you will not be able to reprovision your Oracle Database Appliance system with all the Oracle virtualized platform resources intact.
Download the Oracle Database Appliance Server Patch for the ODACLI/DCS stack and update the repository with the server software file as described in the topic Patching Oracle Database Appliance Bare Metal Systems Using the Command-Line:
```
[root@oda1 opt]# /opt/oracle/dcs/bin/odacli update-repository -f /tmp/software_file
```
Create the pre-patch report for the odacli update-server command by specifying the -s option.
```
[root@oda1 opt]# /opt/oracle/dcs/bin/odacli create-prepatchreport -v 19.15.0.0.0 -s
```
Fix the warnings and errors mentioned in the report and proceed with the server patching.
Update the server:
```
[root@oda1 opt]# /opt/oracle/dcs/bin/odacli update-server -v version
```
Updating the server in this step updates only the Oracle ILOM and boot disk firmware since the appliance is not yet reprovisioned.

Update the storage:

[root@oda1 opt]# /opt/oracle/dcs/bin/odacli update-storage -v version

Update the repository with the server archive files generated in Step 6 of this migration process.
For High-Availability systems, specify the three zip files generated in step 5 when the nodes are detached.
```
[root@oda1 opt]# /opt/oracle/dcs/bin/odacli update-repository -f serverarchive_node0_hostname.zip,serverarchive_node1_hostname.zip,serverarchive_cluster_name_common.zip
```

Reprovision the appliance:

[root@oda1 opt]# /opt/oracle/dcs/bin/odacli restore-node -g

This command reconfigures networks, operating system users, and operating system groups and installs the latest Oracle Grid Infrastructure. At this step, the installation reuses the existing Oracle ASM disk groups instead of creating new ones.

For example:

[root@oda1 opt]# /opt/oracle/dcs/bin/odacli restore-node -g
Job details                                                      
----------------------------------------------------------------
                     ID:  89c005e3-cf5e-4887-968a-7532bf9f993b
            Description:  Restore node service creation
                 Status:  Success
                Created:  September 13, 2021 11:33:38 AM UTC
                Message:  

Task Name                                Start Time                          End
Time                            Status    
---------------------------------------- -----------------------------------
----------------------------------- ----------
Restore node service creation            September 13, 2021 11:33:53 AM UTC  September 13, 2021 12:05:43 PM UTC  Success   
Setting up Network                       September 13, 2021 11:33:56 AM UTC  September 13, 2021 11:33:56 AM UTC  Success   
Setting up Vlan                          September 13, 2021 11:34:15 AM UTC  September 13, 2021 11:34:15 AM UTC  Success   
Setting up Network                       September 13, 2021 11:34:34 AM UTC  September 13, 2021 11:34:34 AM UTC  Success   
Setting up Network                       September 13, 2021 11:34:57 AM UTC  September 13, 2021 11:34:57 AM UTC  Success   
Setting up Vlan                          September 13, 2021 11:35:16 AM UTC  September 13, 2021 11:35:16 AM UTC  Success   
Setting up Network                       September 13, 2021 11:35:35 AM UTC  September 13, 2021 11:35:35 AM UTC  Success   
network update                           September 13, 2021 11:35:58 AM UTC  September 13, 2021 11:36:18 AM UTC  Success   
updating network                         September 13, 2021 11:35:58 AM UTC  September 13, 2021 11:36:18 AM UTC  Success   
Setting up Network                       September 13, 2021 11:35:58 AM UTC  September 13, 2021 11:35:58 AM UTC  Success   
network update                           September 13, 2021 11:36:18 AM UTC  September 13, 2021 11:36:37 AM UTC  Success   
updating network                         September 13, 2021 11:36:18 AM UTC  September 13, 2021 11:36:37 AM UTC  Success   
Setting up Network                       September 13, 2021 11:36:18 AM UTC  September 13, 2021 11:36:18 AM UTC  Success   
OS usergroup 'asmdba'creation            September 13, 2021 11:36:37 AM UTC  September 13, 2021 11:36:37 AM UTC  Success   
OS usergroup 'asmoper'creation           September 13, 2021 11:36:37 AM UTC  September 13, 2021 11:36:37 AM UTC  Success   
OS usergroup 'asmadmin'creation          September 13, 2021 11:36:37 AM UTC  September 13, 2021 11:36:38 AM UTC  Success   
OS usergroup 'dba'creation               September 13, 2021 11:36:38 AM UTC  September 13, 2021 11:36:38 AM UTC  Success   
OS usergroup 'dbaoper'creation           September 13, 2021 11:36:38 AM UTC  September 13, 2021 11:36:38 AM UTC  Success   
OS usergroup 'oinstall'creation          September 13, 2021 11:36:38 AM UTC  September 13, 2021 11:36:38 AM UTC  Success   
OS user 'grid'creation                   September 13, 2021 11:36:38 AM UTC  September 13, 2021 11:36:38 AM UTC  Success   
OS user 'oracle'creation                 September 13, 2021 11:36:38 AM UTC  September 13, 2021 11:36:38 AM UTC  Success   
Default backup policy creation           September 13, 2021 11:36:38 AM UTC  September 13, 2021 11:36:39 AM UTC  Success   
Backup config metadata persist           September 13, 2021 11:36:38 AM UTC  September 13, 2021 11:36:39 AM UTC  Success   
Grant permission to RHP files            September 13, 2021 11:36:39 AM UTC  September 13, 2021 11:36:39 AM UTC  Success   
Add SYSNAME in Env                       September 13, 2021 11:36:40 AM UTC  September 13, 2021 11:36:40 AM UTC  Success   
SSH equivalance setup                    September 13, 2021 11:36:40 AM UTC  September 13, 2021 11:36:40 AM UTC  Success   
Install oracle-ahf                       September 13, 2021 11:36:42 AM UTC  September 13, 2021 11:37:46 AM UTC  Success   
Unzipping storage configuration files    September 13, 2021 11:37:47 AM UTC  September 13, 2021 11:37:47 AM UTC  Success   
Reloading multipath devices              September 13, 2021 11:37:49 AM UTC  September 13, 2021 11:37:51 AM UTC  Success   
restart oakd                             September 13, 2021 11:38:02 AM UTC  September 13, 2021 11:38:14 AM UTC  Success   
Reloading multipath devices              September 13, 2021 11:39:17 AM UTC  September 13, 2021 11:39:20 AM UTC  Success   
restart oakd                             September 13, 2021 11:39:30 AM UTC  September 13, 2021 11:39:41 AM UTC  Success   
Creating GI home directories             September 13, 2021 11:39:41 AM UTC  September 13, 2021 11:39:41 AM UTC  Success   
Extract GI clone                         September 13, 2021 11:39:41 AM UTC  September 13, 2021 11:41:14 AM UTC  Success   
Extract GI clone                         September 13, 2021 11:41:16 AM UTC  September 13, 2021 11:42:48 AM UTC  Success   
Creating wallet for Root User            September 13, 2021 11:42:51 AM UTC  September 13, 2021 11:42:54 AM UTC  Success   
Creating wallet for ASM Client           September 13, 2021 11:42:54 AM UTC  September 13, 2021 11:43:01 AM UTC  Success   
GI Restore with RHP                      September 13, 2021 11:43:02 AM UTC  September 13, 2021 11:55:52 AM UTC  Success   
Updating GIHome version                  September 13, 2021 11:55:52 AM UTC  September 13, 2021 11:55:54 AM UTC  Success   
Updating GIHome version                  September 13, 2021 11:55:52 AM UTC  September 13, 2021 11:55:54 AM UTC  Success   
Mounting disk group DATA                 September 13, 2021 11:55:56 AM UTC  September 13, 2021 11:55:59 AM UTC  Success   
Mounting disk group RECO                 September 13, 2021 11:56:07 AM UTC  September 13, 2021 11:56:09 AM UTC  Success   
Mounting disk group REDO                 September 13, 2021 11:56:17 AM UTC  September 13, 2021 11:56:19 AM UTC  Success   
Post cluster OAKD configuration          September 13, 2021 11:56:19 AM UTC  September 13, 2021 12:01:47 PM UTC  Success   
Setting ACL for disk groups              September 13, 2021 12:01:47 PM UTC  September 13, 2021 12:01:52 PM UTC  Success   
Modify DB file attributes                September 13, 2021 12:01:52 PM UTC  September 13, 2021 12:02:06 PM UTC  Success   
Register Scan and Vips to Public Network September 13, 2021 12:02:11 PM UTC  September 13, 2021 12:02:16 PM UTC  Success   
Starting FileSystem                      September 13, 2021 12:02:53 PM UTC  September 13, 2021 12:03:04 PM UTC  Success   
Enabling Volume(s)                       September 13, 2021 12:03:32 PM UTC  September 13, 2021 12:03:42 PM UTC  Success   
Adding Volume COMMONSTORE to Clusterware September 13, 2021 12:03:42 PM UTC  September 13, 2021 12:03:47 PM UTC  Success   
Adding Volume DATTDB to Clusterware      September 13, 2021 12:04:01 PM UTC  September 13, 2021 12:04:04 PM UTC  Success   
Adding Volume RECO to Clusterware        September 13, 2021 12:04:17 PM UTC  September 13, 2021 12:04:20 PM UTC  Success   
Adding Volume REDO to Clusterware        September 13, 2021 12:04:33 PM UTC  September 13, 2021 12:04:37 PM UTC  Success   
Starting FileSystem                      September 13, 2021 12:04:50 PM UTC  September 13, 2021 12:05:00 PM UTC  Success   
Starting FileSystem                      September 13, 2021 12:05:00 PM UTC  September 13, 2021 12:05:11 PM UTC  Success   
Starting FileSystem                      September 13, 2021 12:05:11 PM UTC  September 13, 2021 12:05:22 PM UTC  Success   
Starting FileSystem                      September 13, 2021 12:05:22 PM UTC  September 13, 2021 12:05:32 PM UTC  Success   
Delete public key                        September 13, 2021 12:05:37 PM UTC  September 13, 2021 12:05:37 PM UTC  Success   
configure Zk Quorum TLS                  September 13, 2021 12:05:38 PM UTC  September 13, 2021 12:05:39 PM UTC  Success   
Create Trust Store                       September 13, 2021 12:05:39 PM UTC  September 13, 2021 12:05:40 PM UTC  Success   
Delete Trust Store                       September 13, 2021 12:05:40 PM UTC  September 13, 2021 12:05:40 PM UTC  Success   
Create Trust Store                       September 13, 2021 12:05:40 PM UTC  September 13, 2021 12:05:41 PM UTC  Success   
Provisioning service creation            September 13, 2021 12:05:42 PM UTC  September 13, 2021 12:05:42 PM UTC  Success   
persist new agent state entry            September 13, 2021 12:05:42 PM UTC  September 13, 2021 12:05:42 PM UTC  Success   
persist new agent state entry            September 13, 2021 12:05:42 PM UTC  September 13, 2021 12:05:42 PM UTC  Success   
persist new agent state entry            September 13, 2021 12:05:42 PM UTC  September 13, 2021 12:05:42 PM UTC  Success   
persist new agent state entry            September 13, 2021 12:05:42 PM UTC  September 13, 2021 12:05:42 PM UTC  Success   
Restart Zookeeper and DCS Agent          September 13, 2021 12:05:42 PM UTC  September 13, 2021 12:05:43 PM UTC  Success

Update the repository with the Oracle Database clones as described in /opt/oracle/oak/restore/metadata/dbVersions.list:

[root@oda1 opt]# /opt/oracle/dcs/bin/odacli update-repository -f /tmp/DB_software_file

For example:

[root@oda1 opt]# cat /opt/oracle/oak/restore/metadata/dbVersions.list
# List of all db versions found, to be used for downloading required clones before DB restore
11.2.0.4.180417
12.1.0.2.180417

Configure the storage location and size for database homes on Oracle ACFS. For information about creating database homes on Oracle ACFS, see the topic About Creating Database Homes on Oracle ACFS Storage in this guide.
```
[root@oda1 opt]# /opt/oracle/dcs/bin/odacli configure-dbhome-storage -dg DATA -s 80
```
The database files are intact on Oracle ASM disk groups. However, the database software must be reinstalled and the database instances must be restarted. Run the following command to create the database homes on Oracle ACFS and then restart the instances on the nodes. Single-instance Oracle databases and Oracle RAC One Node databases have only one instance running. For Oracle RAC deployments on high-availability systems, the database instances are restarted on both nodes.
```
[root@oda1 opt]# /opt/oracle/dcs/bin/odacli restore-node -d
```
This command restores the databases to the same versions previous to the migration, on the bare metal system. The databases are restarted and added to the cluster.
Migrate virtualized platform VMs to VMs on KVM:
```
[root@oda1 opt]# odacli migrate-ovm2kvm
```
Check the virtualized platform VMs to KVM migration completed successfully:
```
[root@oda1 opt]# odacli describe-ovm2kvm -i migration_job_id
```
Verify that all VMs bootstrap correctly. Refer to the Managing an Oracle Database Appliance KVM Deployment chapter in this guide.
Check all vdisks and vnetworks are attached correctly and are visible within the guest VM.

Note:

You have completed migration of your databases from Oracle Database Appliance virtualized platform to the bare metal system. To migrate the databases from Oracle Database Appliance bare metal system to DB system, refer to My Oracle Support Note 2869506.1 at https://support.oracle.com/rs?type=doc&id=2869506.1.

Parent topic: Migrating from Virtualized Platform to KVM on Oracle Database Appliance