Upgrading Your Environment to 4.5

5 Upgrading Your Environment to 4.5

You can upgrade Oracle Linux Virtualization Manager environment from the latest version of 4.4 to 4.5 by upgrading your engine or self-hosted engine and KVM hosts. Upgrading from 4.4 to 4.5 with Gluster 8 storage in your environment is supported.

Important:

Before you upgrade, be aware of the following preprequisites.

The engine, self-hosted engine and KVM hosts must be updated to the latest version of 4.4. See Updating engine or self-hosted engine to latest version of 4.4 and Updating KVM hosts to the latest version of 4.4.
All Oracle Linux 7 KVM hosts must be upgraded to Oracle Linux 8 before you upgrade your environment to 4.5.
Upgrading from 4.3 to 4.5 is not supported. You must first upgrade to the latest version of 4.4. For more information, see the Oracle Linux Virtualization Manager 4.4: Administration Guide.

Considerations

Plan for any necessary virtual machine downtime. After you update the clusters' compatibility versions, a new hardware configuration is automatically applied to each virtual machine once it reboots. You must reboot any running or suspended virtual machines as soon as possible to apply the configuration changes.
Ensure your environment meets the requirements for 4.5. See Requirements and Scalability Limits in the Oracle Linux Virtualization Manager: Architecture and Planning Guide.
Clusters must have a minimum of two (2) KVM hosts (in the same cluster) to perform a live migration of virtual machines when you are upgrading the specific KVM host that is in Maintenance mode.

Note:

Connected hosts and virtual machines can continue to work while you upgrade the engine.

Before You Begin

Before you begin the upgrade process, be aware of the following:

There are upgrade prerequisites that are common to both a standard environment and a self-hosted engine environment.
There are additional prerequisites for a self-hosted engine environment.
Your engine or self-hosted engine and KVM hosts must be updated to the latest version of 4.4 before you begin the upgrade process. However, you cannot upgrade your KVM hosts until after the engine or self-hosted engine upgrade is completed. See Updating engine or self-hosted engine to latest version of 4.4 and Updating KVM hosts to the latest version of 4.4.
All Oracle Linux 7 KVM hosts must be upgraded to Oracle Linux 8 before you begin the upgrade process.

Prerequisites

The operating system for the engine host must be Oracle Linux 8.8 (or later Oracle Linux 8 release). If it is not, update the engine host to Oracle Linux 8.8 (or later Oracle Linux 8 release) before you upgrade to the engine to 4.5.
All data centers and clusters in the environment must have the cluster compatibility level set to version 4.4 or higher.
All virtual machines in the environment must have the same compatibility level as their cluster.
For ULN registered hosts or if you are using Oracle Linux Manager, ensure you have subscribed to:
- ol8_x86_64_ovirt45
- ol8_x86_64_ovirt45_extras
If you use an external CA to sign HTTPS certificates, follow the steps in Replacing the Oracle Linux Virtualization Manager Apache SSL Certificate. The backup and restore include the 3rd-party certificate, so you should be able to log in to the Administration portal after the upgrade. Ensure the CA certificate is added to system-wide trust stores of all clients to ensure the foreign menu of virt-viewer works.
Additionally, for self-hosted engine environments:
- Make note of the MAC address of the self-hosted engine if you are using DHCP and want to use the same IP address. The deploy script prompts you for this information.
- Set the cluster scheduling policy to cluster_maintenance in order to prevent automatic virtual machine migration during the upgrade.

Updating engine or self-hosted engine to latest version of 4.4

Before upgrading to 4.5, you must update the engine or self-hosted engine to the latest version of 4.4.

(Self-hosted engine only) Migrate virtual machines and enable global maintenance mode.
- Migrate all other virtual machines off the host that contains the self-hosted engine virtual machine. Move the virtual machines to another host within the same cluster. During the upgrade, the host can only contain the self-hosted engine virtual machine (no other virtual machines can be on the host). Use Live Migration to minimize virtual machine down-time. See Migrating Virtual Machines between Hosts.
- Enable global maintenance mode:
  1. Log in to the KVM host where the self-hosted engine is running or any KVM host configured to run the self-hosted engine.
  2. Enable global maintenance mode:
```
# hosted-engine --set-maintenance --mode=global
```
  3. Confirm that the environment is in global maintenance mode before proceeding:
```
# hosted-engine --vm-status
```
  4. You should see the following message:
```
!! Cluster is in GLOBAL MAINTENANCE mode !!
```
On the engine or self-hosted engine machine, update to the latest Oracle Linux Virtualization Manager Release 4.4 package.
```
# dnf update oracle-ovirt-release-el8                    
```
Check for updated packages:
```
# engine-upgrade-check
```
Update the setup packages:
```
# dnf update ovirt\*setup\*
```
Update the engine or self-hosted engine:
```
# engine-setup
```
Important:
The update process might take some time. Do not stop the process before it completes.

The engine-setup script:
- Prompts you with some configuration questions
  
  For more information, see Engine Configuration Options in the Oracle Linux Virtualization Manager: Getting Started.
- Stops the ovirt-engine service
- Downloads and installs the updated packages
- Backs up and updates the database
- Performs post-installation configuration
- Starts the ovirt-engine service
Note:
The engine-setup script displays stored configuration values supplied during the initial engine installation process. These stored values display when previewing the configuration and may not be up to date if you ran engine-config after installation. However, engine-setup will not overwrite your updated values.
For example, if you ran engine-config to update SANWipeAfterDelete to true after installation, engine-setup outputs Default SAN wipe after delete: False in the configuration preview. However, engine-setup will not apply this value; rather, it will keep the SANWipeAfterDelete to true setting.

If the update is successful, you will see:
```
Execution of setup completed successfully
```
If the update fails, the engine-setup command attempts to rollback your installation to its previous state. If you encounter a failed update, detailed instructions display explaining how to restore your installation.
Update the base operating system and any optional packages installed on the engine:
```
# dnf update
```
If any kernel packages were updated, reboot the machine to complete the update.
You are now ready to update your KVM hosts.

Updating KVM hosts to the latest version of 4.4

In the Administration portal, go to Compute and then click Hosts.
In the Hosts pane, click a blank or non-linked cell for a host to select it.
Click Installation and then Check for Upgrade.
From the Upgrade Host window, click OK.

The engine checks the KVM host to see if it requires an update.
Using your mouse, hover over the icon next to the host name to see if an update is available.
To proceed with the update, click Installation and then Upgrade.
From the Upgrade Host window, click OK to begin the update process.

Repeat these steps to update the rest of the KVM hosts in the same cluster, one by one, until they are all updated to the latest version of 4.4.

If you have finished updating your engine or self-hosted engine and KVM hosts, continue to Upgrading the Engine or Self-Hosted Engine.

Upgrading the Engine or Self-Hosted Engine

Before you start the upgrade process,

Review Upgrading Your Environment to 4.5.
Verify that you have satisfied the prerequisites.
Ensure you have completed Updating engine or self-hosted engine to latest version of 4.4.
Ensure you have completed Updating KVM hosts to the latest version of 4.4.

On the engine or self-hosted engine machine, install the Oracle Linux Virtualization Manager Release 4.5 package.
```
# dnf install oracle-ovirt-release-45-el8
```
Verify that the ovirt-4.5 and ovirt-4.5-extras repositories are enabled.
```
# dnf repolist                 
```
If either repository is not enabled, use the dnf config-manager --enable repository command to enable them.
Check for updated packages:
```
# engine-upgrade-check
```
Update the setup packages:
```
# dnf update ovirt\*setup\*
```
Update the engine or self-hosted engine:
```
# engine-setup
```
Proceed to Upgrading KVM Hosts.

Upgrading KVM Hosts

During the upgrade process, you must migrate all virtual machines to another host in your environment. Doing so minimizes the downtime of virtual machines in your environment. After the upgrade, you can reattach the KVM host to the engine and migrate the virtual machines back to the host.

To upgrade your KVM hosts, you must update them to the latest version of 4.4 before you complete the engine or self-hosted engine upgrade. You also must upgrade any Oracle Linux 7 KVM hosts to Oracle Linux 8. See Before You Begin.

Attention:

When installing or reinstalling the host’s operating system, Oracle strongly recommends that you first detach any existing non-OS storage from the host to avoid potential data loss from accidental initialization of these disks.

Verify the 4.5 engine is installed and running.
(Optional) Verify that all data centers and clusters in the environment are at the same compatibility level.
Pick a host to upgrade and migrate the host’s virtual machines to another host in the same cluster. Any CPU-pinned virtual machines must be shut down and booted into another available KVM host before live migration.
You can use Live Migration to minimize virtual machine downtime. See Migrating Virtual Machines between Hosts.
For ULN registered hosts or if you are using Oracle Linux Manager, ensure you have subscribed to:
- ol8_x86_64_ovirt45
- ol8_x86_64_ovirt45_extras
Install the Oracle Linux Virtualization Manager Release 4.5 package:
```
# dnf install oracle-ovirt-release-45-el8
```
If your host is running UEK R7:
1. Install the Extra kernel modules package.
```
# dnf install kernel-uek-modules-extra
```
2. Reboot the host.
  
  Important:
  If the KVM host you rebooted is in a hyperconverged self-hosted engine environment, review the following information before you continue the upgrade process.
  - After the reboot a KVM host, the glusterd service takes time to check devices and prerequisites. This can take a significant amount of time to complete because it varies based on the number of volumes. And, the volume size influences whether a self-healing process starts.
  - Before attempting to manually activate LVM volumes or services or start healing a volume, check the Gluster logs for errors and failures.
  - Make sure all Gluster storage can be connected: hosted-engine --connect-storage.
  - Check that Gluster volumes and ovirt-ha-agent/ovirt-ha-broker services are in good status on all the KVM nodes:
```
gluster volume status
systemctl status ovirt-ha-agent
systemctl status ovirt-ha-agent
```
Upgrade the KVM host to 4.5:
1. In the Administration portal, go to Compute and then click Hosts.
2. In the Hosts pane, click a blank or non-linked cell for a host to select it.
3. Click Installation and then Check for Upgrade.
4. From the Upgrade Host window, click OK.
  
  The engine checks the KVM host to see if it requires an update.
5. Using your mouse, hover over the icon next to the host name to see if an update is available.
6. To proceed with the update, click Installation and then Upgrade.
7. From the Upgrade Host window, click OK to begin the update process.
Repeat these steps to upgrade the rest of the KVM hosts in the same cluster, one by one, until all are running 4.5.
Update the compatibility version to 4.7 to use all features of release 4.5.

See Changing Data Center and Cluster Compatibility Versions After Upgrading.

Rebooting KVM hosts in a hyperconverged self-hosted engine environment

After the reboot a KVM host, the glusterd service takes time to check devices and prerequisites. This can take a significant amount of time to complete because it varies based on the number of volumes. And, the volume size influences whether a self-healing process starts.

Changing Data Center and Cluster Compatibility Versions After Upgrading

Oracle Linux Virtualization Manager data centers and clusters have a compatibility version. The data center compatibility version indicates the version of Oracle Linux Virtualization Manager that the data center is intended to be compatible with. The cluster compatibility version indicates the features supported by all of the hosts in the cluster. The cluster compatibility is set according to the version of the least capable host operating system in the cluster.

The preferred approach after upgrading your engine to 4.5 is to upgrade all hosts to 4.5 and then change the cluster compatibility to 4.7. You can then add new hosts as 4.5 hosts.

Important:

Although the Oracle Linux Virtualization Manager version is 4.5, the corresponding compatibility version is 4.7.

Compatibility Version Restrictions

Consider these restrictions to ensure you do not have issues with compatibility versions after you upgrade.

Data Center Compatibility Versions

The data center compatibility level is the minimum version you can use for all clusters in your data center. For example:

If your data center compatibility level is 4.7, you can only have clusters with compatibility level 4.7.
If your data center compatibility level is 4.4, you can have 4.4 or higher compatibility level clusters.

Cluster Compatibility Versions

The cluster compatibility level is the minimum version of any host you add to the cluster. For example:

If you have a 4.6 compatibility version cluster, you can add 4.4 or 4.5 hosts.
If you have a 4.7 compatibility version cluster, you can only add 4.5 hosts.

Possible Errors When Changing Compatibility Versions

If you try to change the data center compatibility version from 4.6 to 4.7 when you have a 4.4 compatibility version cluster, you get the following error:

Cannot update Data Center compatibility version to a value that is greater than its 
cluster's version. The following clusters should be upgraded: [clustername]

If you try to change the cluster compatibility version from 4.6 to 4.7 when you have 4.4 hosts running, you get the following error:

Error while executing action: Cannot change Cluster Compatibility Version to higher version 
when there are active Hosts with lower version. -Please move Host [hostname] with lower 
version to maintenance first.

When you put a 4.4 host in maintenance mode, you can change the cluster and then data center compatibility version to 4.7. However, the host shows non-operational with the following event:
```
Host [hostname] is compatible with versions ([version levels]) and cannot join Cluster 
[clustername] which is set to version [version level].
```

Changing Cluster Compatibility Versions

To change the cluster compatibility version, you must have first upgraded all the hosts in your cluster to a level that supports your desired compatibility level.

Verify all hosts are running a version level that supports your desired compatibility level. See Compatibility Version Restrictions.
In the Administration Portal, go to Compute and click Clusters.
Select the cluster to change and click Edit.
From the Edit Cluster dialog box, select General.
For Compatibility Version, select desired value and click OK.
On the Change Cluster Compatibility Version confirmation window, click OK.

Important:

You might get an error message warning that some virtual machines and templates are incorrectly configured. To fix this error, edit each virtual machine manually. The Edit Virtual Machine window provides additional validations and warnings that show what to correct. Sometimes the issue is automatically corrected and the virtual machine’s configuration just needs to be saved again. After editing each virtual machine, you will be able to change the cluster compatibility version.
Update the cluster compatibility version of all running or suspended virtual machines by restarting them from within the Administration Portal.

Note:

Virtual machines continue to run in the previous cluster compatibility level until you restart them. The Next-Run icon (triangle with an exclamation mark) indentifies virtual machines that require a restart. However, the self-hosted engine virtual machine does not need to be restarted.

You cannot change the cluster compatibility version of a virtual machine snapshot that is in preview. You must first commit or undo the preview.

Changing Data Center Compatibility Versions

After updating the compatibility version of all clusters in a data center, you can change the compatibility version of the data center itself.

Verify that all clusters are at the proper compatibility version. If not, change the version of the clusters, see Changing Cluster Compatibility Versions.
In the Administration Portal, go to Compute and click Data Centers.
Select the data center to change and click Edit.
From the Edit Data Center dialog box, change the Compatibility Version to the desired value and then click OK.
On the Change Data Center Compatibility Version confirmation window, click OK.