Upgrading to Oracle Linux 8 using Leapp

5 Upgrading to Oracle Linux 8 using Leapp

This chapter describes how to upgrade nodes in an Oracle Cloud Native Environment from Oracle Linux 7 to Oracle Linux 8 using the Leapp utility. The Leapp utility upgrades the operating system to Oracle Linux 8 and maintains the Oracle Cloud Native Environment installation.

For more information about using Leap, see Oracle Linux 8: Upgrading Systems With Leapp.

Important:

If you want to upgrade the nodes to Oracle Linux 8, you must first install or upgrade to Oracle Cloud Native Environment Release 1.6 on Oracle Linux 7.

Upgrade Overview

To upgrade control plane and worker nodes, you remove a node from the cluster, upgrade the operating system, and then add the node back into the cluster. The basic steps to upgrade your cluster from Oracle Linux 7 to Oracle Linux 8 are:

Upgrade the operator node (where the Platform API server is located).
Remove control plane and worker nodes(s) from the cluster and upgrade them using Leapp.
Replace the nodes in the cluster.

It is recommended that during the upgrade of a highly available Kubernetes cluster, the cluster should always have:

A minimum of three control plane nodes. A minimum of five control plane nodes in the cluster during the upgrade is recommended.
A minimum of three workers nodes.
The workload in your cluster should have enough resources to allow you to remove 1/3 of your worker nodes. Make sure the cluster has enough resources to continue running your applications during the upgrade.

Oracle recommends that you do not run the cluster with mixed operating systems for longer than required to perform the upgrade.

Setting up a Remote Console

The Leapp utility requires some set up on each node before you perform the upgrade. The basics of this are included in this section. It is recommended that you read the Leapp documentation before you perform these steps as you may need to perform other steps on your nodes before you upgrade them. For more information about using Leap, see Oracle Linux 8: Upgrading Systems With Leapp.

Set up a method to connect remotely to the system console. This allows you to monitor the upgrade as it occurs over system reboots. For example:

Oracle Cloud Infrastructure, use the Cloud Shell connection console.
Oracle X86 servers, use Oracle Integrated Lights Out Manager (ILOM).
Oracle Private Cloud Appliance, use the Instance Console Connection.
Oracle Linux Kernel-based Virtual Machines (KVM), use the appropriate virtualization management software for your deployment, such as Cockpit Web Console, virt-viewer or virt-manager.

If you connect to the system using SSH in a terminal, you are disconnected while the upgrade is being performed and cannot log in again until the upgrade is completed.

Upgrading the Operator Node

This section shows you how to upgrade the operator node from Oracle Linux 7 to Oracle Linux 8 using Leapp. You should upgrade the operator node before you upgrade the nodes in the Kubernetes cluster.

To upgrade the operator node:

On the system you want to upgrade, update the operating system to the latest packages.
```
sudo yum update --disablerepo ol7_olcne16
```
Reboot the system.
```
sudo reboot
```
When the system restarts, install the Leapp upgrade software package:
```
sudo yum install -y leapp-upgrade --enablerepo ol7_leapp,ol7_latest
```
Grant the root user SSH login permissions. Edit the /etc/ssh/sshd_config file to remove the comment for the CopyPermitRootLogin yes entry.
```
sudo sed -i 's/#PermitRootLogin yes/PermitRootLogin yes/g' /etc/ssh/sshd_config
```
There may be preupgrade steps listed in the Leapp documentation that apply to your systems which are not listed here, such as unmounting any filesystems, or disabling Secure Boot. Understand and perform any further steps to prepare for an upgrade of the system as described in Oracle Linux 8: Upgrading Systems With Leapp.
On the node that you want to upgrade, run the Leapp upgrade preinstallation check script. This script checks whether the system is ready for the upgrade and generates a report you can review prior to upgrading.

If this is an Oracle Cloud Infrastructure instance, use:
```
sudo leapp preupgrade --oci --enablerepo ol8_olcne16
```
For all other Oracle Linux systems, use:
```
sudo leapp preupgrade --oraclelinux --enablerepo ol8_olcne16
```
The host is checked to make sure the upgrade can proceed, and an answerfile is generated for when you perform the upgrade.
Review the contents of the generated answerfile and resolve any issues.
```
sudo cat /var/log/leapp/answerfile
```
You can safely remove the section from the answerfile for the blocking module check for remove_pam_pkcs11_module_check.
```
sudo leapp answer --section remove_pam_pkcs11_module_check.confirm=True
```
More information is available on analyzing the Leapp answerfile in the Leapp documentation.
Review any issues that you may need to resolve with the upgrade by reviewing the Leapp upgrade report file.
```
sudo cat /var/log/leapp/leapp-report.txt
```
More information is available on analyzing the Leapp upgrade report in the Leapp documentation.
Run the leapp preupgrade command again to make sure all upgrade issues have been resolved.

If this is an Oracle Cloud Infrastructure instance, use:
```
sudo leapp preupgrade --oci --enablerepo ol8_olcne16
```
For all other Oracle Linux systems, use:
```
sudo leapp preupgrade --oraclelinux --enablerepo ol8_olcne16
```
The output should report that there are no blocking issues that need to be resolved.
You can review any risks with the upgrade by reading the upgrade report file.
```
sudo cat /var/log/leapp/leapp-report.txt
```
Run the Leapp upgrade script.

It is recommended that you run this upgrade commands using the remote console you set up or is available to your environment so you can monitor the upgrade as it occurs, and interract with the command line without losing the connection to the host. If you use an SSH terminal connection to the system, you can run the upgrade script, but your session is disconnected when the system is restarted.

If this is an Oracle Cloud Infrastructure instance, use:
```
sudo leapp upgrade --oci --enablerepo ol8_olcne16
```
For all other Oracle Linux systems, use:
```
sudo leapp upgrade --oraclelinux --enablerepo ol8_olcne16
```
A transaction check is performed and the output displayed.
Reboot the system to start the upgrade.
```
sudo reboot
```
The upgrade is performed and this may take some time. You can monitor the upgrade using the system console. When the system is upgraded you can log into the system again using SSH if you need.

When the upgrade is completed, reboot the system again.
```
sudo reboot
```
After the system restarts, you can confirm the operating system is upgraded using:
```
cat /etc/oracle-release 
```
You should see the latest version of Oracle Linux 8 listed, for example:
```
Oracle Linux Server release 8.version
```
You can confirm the Oracle Cloud Native Environment software packages are updated using:
```
rpm -qa | egrep olcne | sort
```
The output should list the installed Oracle Cloud Native Environment software packages, which should now be for Oracle Linux 8, and look similar to:
```
olcne-api-server-version.el8.x86_64
olcnectl-version.el8.x86_64
olcne-selinux-version.el8.x86_64
olcne-utils-version.el8.x86_64
oracle-olcne-release-el8-version.el8.x86_64
```

Upgrading Kubernetes Nodes

This section shows you how to upgrade the Kubernetes nodes from Oracle Linux 7 to Oracle Linux 8 using Leapp.

You should upgrade the control plane nodes first, then the worker nodes. You need to scale out the nodes from the cluster, perform the upgrade to Oracle Linux 8, then scale the nodes back into the cluster.

Make sure you maintain high availability requirements for the cluster if required. For a highly available cluster, it is recommended that you have a cluster with at least five control plane nodes, and at least three worker nodes. The number of control plane nodes in a cluster must be an odd number equal to or greater than three, for example, 3, 5, or 7. As it is recommended to always have at least three control plane nodes in the cluster, with a recommended minimum of five nodes, this is likely to be the most common upgrade scenario.

It is recommended that you scale down the cluster to remove two control plane nodes at a time, then replace them with two upgraded Oracle Linux 8 nodes.

Important:

Scaling down and up two nodes at a time, even if only one node needs to be upgraded to Oracle Linux 8, maintains a cluster quorum if a network outage occurs.

Worker nodes should be replaced in the cluster one at a time, to allow the applications running on the nodes to migrate to other nodes. If you have less than four worker nodes in the cluster, consider finding additional resources so that you do not enter a state where there are less than three worker nodes in the cluster. This may mean that you temporarily recruit another server to increase the worker node count to at least four during the upgrade process. Otherwise, your applications running on the worker nodes may not work as expected.

To upgrade the Kubernetes nodes:

On the operator node, use the olcnectl module update command to scale down the cluster to remove the Oracle Linux 7 node(s) you want to upgrade to Oracle Linux 8.

Use the following syntax to scale down (remove) control plane nodes:
```
olcnectl module update \
--environment-name environment_name \  
--name cluster_name \
--control-plane-nodes node_list
```
Use the following syntax to scale down worker nodes:
```
olcnectl module update \
--environment-name environment_name \  
--name cluster_name \
--worker-nodes node_list
```
The node_list option should contain the list the nodes that should remain in the cluster, so omit the nodes to remove from this list.
Tip:

You can see which nodes are in the cluster, by running the kubectl get nodes command on a control plane node where you have kubectl set up:
```
kubectl get nodes -o wide
```
On the system you want to upgrade, update the operating system to the latest packages.
```
sudo yum update --disablerepo ol7_olcne16
```
Reboot the system.
```
sudo reboot
```
When the system restarts, install the Leapp upgrade software package:
```
sudo yum install -y leapp-upgrade --enablerepo ol7_leapp,ol7_latest
```
Grant the root user SSH login permissions. Edit the /etc/ssh/sshd_config file to remove the comment for the CopyPermitRootLogin yes entry.
```
sudo sed -i 's/#PermitRootLogin yes/PermitRootLogin yes/g' /etc/ssh/sshd_config
```
There may be preupgrade steps listed in the Leapp documentation that apply to your systems which are not listed here, such as unmounting any filesystems, or disabling Secure Boot. Understand and perform any further steps to prepare for an upgrade of the system as described in Oracle Linux 8: Upgrading Systems With Leapp.
On the node that you want to upgrade, run the Leapp upgrade preinstallation check script. This script checks whether the system is ready for the upgrade and generates a report you can review prior to upgrading.

If this is an Oracle Cloud Infrastructure instance, use:
```
sudo leapp preupgrade --oci --enablerepo ol8_olcne16,ol8_kvm_appstream
```
For all other Oracle Linux systems, use:
```
sudo leapp preupgrade --oraclelinux --enablerepo ol8_olcne16,ol8_kvm_appstream
```
The host is checked to make sure the upgrade can proceed, and an answerfile is generated for when you perform the upgrade.
Review the contents of the generated answerfile and resolve any issues.
```
sudo cat /var/log/leapp/answerfile
```
You can safely remove the section from the answerfile for the blocking module check for remove_pam_pkcs11_module_check.
```
sudo leapp answer --section remove_pam_pkcs11_module_check.confirm=True
```
More information is available on analyzing the Leapp answerfile in the Leapp documentation.
Review any issues that you may need to resolve with the upgrade by reviewing the Leapp upgrade report file.
```
sudo cat /var/log/leapp/leapp-report.txt
```
More information is available on analyzing the Leapp upgrade report in the Leapp documentation.
Run the leapp preupgrade command again to make sure all upgrade issues have been resolved.

If this is an Oracle Cloud Infrastructure instance, use:
```
sudo leapp preupgrade --oci --enablerepo ol8_olcne16,ol8_kvm_appstream
```
For all other Oracle Linux systems, use:
```
sudo leapp preupgrade --oraclelinux --enablerepo ol8_olcne16,ol8_kvm_appstream
```
The output should report that there are no blocking issues that need to be resolved.
Run the Leapp upgrade script.

It is recommended that you run this upgrade commands using the remote console you set up or is available to your environment so you can monitor the upgrade as it occurs, and interract with the command line without losing the connection to the host. If you use an SSH terminal connection to the system, you can run the upgrade script, but your session is disconnected when the system is restarted.

If this is an Oracle Cloud Infrastructure instance, use:
```
sudo leapp upgrade --oci --enablerepo ol8_olcne16,ol8_kvm_appstream
```
For all other Oracle Linux systems, use:
```
sudo leapp upgrade --oraclelinux --enablerepo ol8_olcne16,ol8_kvm_appstream
```
A transaction check is performed and the output displayed.
Reboot the system to start the upgrade.
```
sudo reboot
```
The upgrade is performed and this may take some time. You can monitor the upgrade using the system console. When the system is upgraded you can log into the system again using SSH if you need.

When the upgrade is completed, reboot the system again.
```
sudo reboot
```
After the system restarts, you can confirm the operating system is upgraded using:
```
cat /etc/oracle-release 
```
You should see the latest version of Oracle Linux 8 listed, for example:
```
Oracle Linux Server release 8.version
```
You can confirm the Oracle Cloud Native Environment software packages are updated using:
```
rpm -qa | egrep 'olcne|kube|conmon|kata|cri-o|kvm' | sort
```
The output should list the installed Oracle Cloud Native Environment software packages, which should now be for Oracle Linux 8, and look similar to:
```
conmon-version.el8.x86_64
cri-o-version.el8.x86_64
kata-version.el8.x86_64
...
```
After the upgrade, SELinux is set to permissive. If SELinux was set to enforcing before the upgrade, update the SELinux configuration file to set it to enforcing again:
```
sudo sed -i s/^SELINUX=.*$/SELINUX=enforcing/ /etc/selinux/config; sudo setenforce 1
```

Run the post upgrade script:

sudo /etc/olcne/scripts/olcne-ol8-post-leapp-upgrade

On the operator node, use the olcnectl module update command to scale up the cluster to add the node(s) you have upgraded to Oracle Linux 8.

Use the following syntax to scale up (add) control plane nodes:
```
olcnectl module update \
--environment-name environment_name \  
--name cluster_name \
--control-plane-nodes node_list
```
Use the following syntax to scale up worker nodes:
```
olcnectl module update \
--environment-name environment_name \  
--name cluster_name \
--worker-nodes node_list
```
The node_list option should contain the list the nodes that should be in the cluster.
Tip:

You can see which nodes are in the cluster, by running the kubectl get nodes command on a control plane node where you have kubectl set up:
```
kubectl get nodes -o wide
```