JavaScript must be enabled to correctly display this content

The software described in this documentation is either no longer supported or is in extended support.
Oracle recommends that you upgrade to a current supported release.

Chapter 4 Upgrading to Oracle Linux 8

This chapter describes how to upgrade nodes in an Oracle Cloud Native Environment from Oracle Linux 7 to Oracle Linux 8.

Important

If you want to upgrade the nodes to Oracle Linux 8, you must first install or upgrade to Oracle Cloud Native Environment Release 1.4 on Oracle Linux 7.

You cannot upgrade the operating system from Oracle Linux 7 to Oracle Linux 8 on the Oracle Cloud Native Environment nodes. You can either create a new set of nodes running Oracle Linux 8 and swap them for the existing Oracle Linux 7 nodes, or remove a node from the cluster, upgrade it, and then add it back to the cluster. The method you choose depends on how many server resources you have in your environment.

It is recommended that during the upgrade your Kubernetes cluster should always have:

A minimum of three control plane nodes. A minimum of five control plane nodes in the cluster during the upgrade is recommended.
A minimum of three workers nodes.
The workload in your cluster should have enough resources to allow you to remove 1/3 of your worker nodes. Make sure the cluster has enough resources to continue running your applications during the upgrade.

Oracle recommends that you do not run the cluster with mixed operating systems for longer than required to perform the upgrade.

The basic steps to upgrade your cluster from Oracle Linux 7 to Oracle Linux 8 are:

Upgrade the operator node.
Prepare new Oracle Linux 8 nodes if possible. If you do not have the server resources to use new nodes, you can upgrade the nodes as they are removed from the cluster.
If the cluster uses an external load balancer, add any new nodes to it.
Replace the nodes in the cluster.

4.1 Upgrading the Operator Node

This section shows you how to migrate the operator node from Oracle Linux 7 to Oracle Linux 8.

To upgrade the operator node:

Install Oracle Linux 8 on a new node and prepare it to use as an operator node. For information on preparing a node to use as an operator node, see Getting Started.
Copy the Oracle Cloud Native Environment data from the /var/olcne directory of the Oracle Linux 7 operator node to the same location on the new Oracle Linux 8 node. Alternatively, if you are using external storage for the /var/olcne directory, mount the shared storage on the new node.
Set up the X.509 certificate for the new node and start the Platform API Server service.
- If you are using Vault to authenticate nodes in the cluster, start the Platform API Server service using the Vault token. For information on starting the Platform API Server service with a Vault token, see Getting Started.
- If you are using CA certificates to authenticate nodes in the cluster, copy the certificate information to the node and start the Platform API Server service using the certificate. For information on starting the Platform API Server service with a CA certificate, see Getting Started.
- If you are using private CA certificates to authenticate the nodes in the cluster, copy the private key and the CA certificate used to generate new certificates for nodes from the Oracle Linux 7 node to the new Oracle Linux 8 node.
  
  If you used the gen-certs-helper.sh script to generate the certificates, the location for these is most likely:
  
  /etc/olcne/configs/certificates/production/ca.cert
  /etc/olcne/configs/certificates/production/ca.key
  
  On the new Oracle Linux 8 node, generate a new certificate for the new operator node using the copied certificate and key, and copy the new certificate information to the node itself. For more information on generating and transferring certificates to nodes, see Getting Started.
  
  Start the Platform API Server service using the private certificate. For information on starting the Platform API Server with a CA certificate, see Getting Started.

Restart the Platform API Server.

sudo systemctl restart olcne-api-server.service

4.2 Preparing an Oracle Linux 8 Node

This section shows you how to prepare an Oracle Linux 8 node to use as a Kubernetes node in the cluster. When the node is prepared it can be scaled into the cluster.

If you have the server resources, you can prepare a whole new set of nodes to add to the cluster in advance, or you can prepare the nodes as you remove them from an Oracle Linux 7 cluster.

To prepare an Oracle Linux 8 node:

Install Oracle Linux 8 on the node and prepare it to use as a Kubernetes node. For information on preparing a node to use as a Kubernetes node, see Getting Started.
Set up the X.509 certificate for the new node and start the Platform Agent service.
- If you are using Vault to authenticate nodes in the cluster, start the Platform Agent service using the Vault token. For information on starting the Platform Agent service with a Vault token, see Getting Started.
- If you are using CA certificates to authenticate nodes in the cluster, copy the certificate information to the node and start the Platform Agent service using the certificate. For information on starting the Platform Agent service with a CA certificate, see Getting Started.
- If you are using private CA certificates to authenticate the nodes in the cluster, generate a new certificate and copy the new certificate information to the node. For more information on generating and transferring certificates to nodes, see Getting Started.
  
  Start the Platform Agent service using the private certificate. For information on starting the Platform Agent with a CA certificate, see Getting Started.

4.3 Adding New Nodes to Your Load Balancer

If you are using an external load balancer for the Kubernetes cluster (set with the --load-balancer option when you created the Kubernetes module), add any new control plane nodes to it. If you are reusing nodes and keeping the same IP address and hostname, you do not need to add them to the load balancer, only any new nodes. If you are creating any new worker nodes and you have the Istio module deployed, you should also add them to your load balancer if required.

If you are using an Oracle Cloud Infrastructure load balancer, add any new control plane nodes to the appropriate backend set and set the port for the control plane nodes to 6443. If you are using Istio with a load balancer, also add any new worker nodes to the the appropriate backend set.

If you are using the load balancer deployed by the Platform CLI (set with the --virtual-ip option when you created the Kubernetes module), you do not need to add the control plane nodes to it. This is done automatically when you scale the nodes into the cluster.

4.4 Upgrading the Kubernetes Cluster

This section shows you how to replace the nodes in a Kubernetes cluster to upgrade them from Oracle Linux 7 to Oracle Linux 8. The first option in this section upgrades the operating system in the cluster using the recommended cluster size of five control plane nodes and three worker nodes. If you do not have this recommended cluster size, alternative options are provided for different cluster types.

4.4.1 Replacing the Nodes

It is recommended that you have a cluster with at least five control plane nodes, and at least three worker nodes. This section shows you how to replace the Kubernetes cluster nodes in this scenario, with the addition of one extra worker node so that the minimum worker nodes requirement is maintained.

You can also use this method if you have more than five control plane nodes and more than three worker nodes.

The number of control plane nodes in a cluster must be an odd number equal to or greater than three, for example, 3, 5, or 7. As it is recommended to always have at least three control plane nodes in the cluster, with a recommended minimum of five nodes, this is likely to be the most common upgrade scenario.

It is recommended that you scale down the cluster to remove two of the five control plane nodes, then replace them with two Oracle Linux 8 nodes. You can either use new nodes you prepared in advance, or upgrade the two removed nodes and scale them back in, depending on your server resources.

Important

Scaling down and up two nodes at a time, even if only one node needs to be upgraded to Oracle Linux 8, maintains a cluster quorum if a network outage occurs.

Worker nodes should be replaced in the cluster one at a time, to allow the applications running on the nodes to migrate to other nodes. If you have less than four worker nodes in the cluster, consider finding additional resources so that you do not enter a state where there are less than three worker nodes in the cluster. This may mean that you temporarily recruit another server to increase the worker node count to at least four during the upgrade process. Otherwise, your applications running on the worker nodes may not work as expected.

To replace the Kubernetes cluster nodes:

From a control plane node, use the kubectl get nodes command to see the control plane and worker nodes in the cluster. In this example, there are five control plane and four worker nodes running Oracle Linux 7.

kubectl get nodes

NAME                   STATUS   ROLE     AGE     VERSION
control1.example.com   Ready    master   26h     v1.21.x+x.x.x.el7  
control2.example.com   Ready    master   26h     v1.21.x+x.x.x.el7
control3.example.com   Ready    master   26h     v1.21.x+x.x.x.el7
control4.example.com   Ready    master   26h     v1.21.x+x.x.x.el7
control5.example.com   Ready    master   26h     v1.21.x+x.x.x.el7
worker1.example.com    Ready    <none>   26h     v1.21.x+x.x.x.el7
worker2.example.com    Ready    <none>   26h     v1.21.x+x.x.x.el7
worker3.example.com    Ready    <none>   26h     v1.21.x+x.x.x.el7
worker4.example.com    Ready    <none>   26h     v1.21.x+x.x.x.el7

On the operator node, scale down the cluster to remove two Oracle Linux 7 control plane nodes. This example removes the control1.example.com and control2.example.com nodes.
```
olcnectl module update \
--environment-name myenvironment \  
--name mycluster \
--master-nodes control3.example.com:8090,control4.example.com:8090,control5.example.com:8090
```
The cluster now has the minimum required number of control plane nodes, which is three.

Allocate two of the new Oracle Linux 8 nodes, or upgrade the two nodes that have been removed from the cluster. For information on preparing nodes, see Section 4.2, “Preparing an Oracle Linux 8 Node”.

Scale up the cluster to add the two Oracle Linux 8 control plane nodes. This example adds the control1-ol8.example.com and control2-ol8.example.com nodes as control plane nodes.
```
olcnectl module update \
--environment-name myenvironment \  
--name mycluster \
--master-nodes control1-ol8.example.com:8090,control2-ol8.example.com:8090,control3.example.com:8090,control4.example.com:8090,control5.example.com:8090
```
The cluster now has the recommended number of control plane nodes, which is five.

Repeat this process until all Oracle Linux 7 control plane nodes are replaced with Oracle Linux 8 nodes.

As there are an odd number of control plane nodes in the cluster, there will be one final node that has not been upgraded. You should scale down and up two control plane nodes at a time, even though one of the nodes is already upgraded to Oracle Linux 8. This maintains an odd number of control plane nodes.
When all control plane nodes in the cluster are replaced with Oracle Linux 8 nodes, you can do the same for the worker nodes. Replace the worker nodes one at a time.

Scale down the cluster to remove an Oracle Linux 7 worker node. This example removes the worker1.example.com node.
```
olcnectl module update \
--environment-name myenvironment \  
--name mycluster \
--worker-nodes worker2.example.com:8090,worker3.example.com:8090,worker4.example.com:8090
```
There are three worker nodes left in the cluster, which is the recommended minimum.

Allocate one of the new Oracle Linux 8 nodes, or upgrade the node that has been removed from the cluster.

Scale up the cluster to add the Oracle Linux 8 worker node. This example adds the worker1-ol8.example.com node as a worker node.
```
olcnectl module update \
--environment-name myenvironment \  
--name mycluster \
--worker-nodes worker1-ol8.example.com:8090,worker2.example.com:8090,worker3.example.com:8090,worker4.example.com:8090
```
Repeat this process until all Oracle Linux 7 worker nodes are replaced with the new Oracle Linux 8 nodes.

On a control plane node, use the kubectl get nodes command to verify the cluster contains the new Oracle Linux 8 control plane and worker nodes and all the Oracle Linux 7 nodes have been removed. For example:

kubectl get nodes

NAME                      STATUS  ROLE     AGE     VERSION
control1-ol8.example.com  Ready   master   20m34s  v1.21.x+x.x.x.el8  
control2-ol8.example.com  Ready   master   18m12s  v1.21.x+x.x.x.el8
control3-ol8.example.com  Ready   master   14m12s  v1.21.x+x.x.x.el8
control4-ol8.example.com  Ready   master   14m12s  v1.21.x+x.x.x.el8
control5-ol8.example.com  Ready   master   14m12s  v1.21.x+x.x.x.el8
worker1-ol8.example.com   Ready   <none>   9m40s   v1.21.x+x.x.x.el8
worker2-ol8.example.com   Ready   <none>   6m28s   v1.21.x+x.x.x.el8
worker3-ol8.example.com   Ready   <none>   4m28s   v1.21.x+x.x.x.el8
worker4-ol8.example.com   Ready   <none>   1m28s   v1.21.x+x.x.x.el8

4.4.2 Replacing the Nodes in a Three Control Plane Node Cluster

If you have a cluster with three control plane nodes, you should create an extra two Oracle Linux 8 control plane nodes using new servers to use during the upgrade. You add these two new control plane nodes to the cluster then remove two Oracle Linux 7 control plane nodes to upgrade the cluster. You repeat this until all control plane nodes are replaced. This means you can maintain the minimum of three control plane nodes in the cluster during the upgrade.

Tip

If you do not have servers to use as a temporary control plane nodes, you could use worker nodes. Scale down the cluster to remove two worker nodes, prepare them as Oracle Linux 8 nodes, and scale up the cluster to add them as a control plane nodes. When the upgrade is completed, remove the two control plane nodes and add them back into the cluster as worker nodes.

The basic steps in this process are below. For details of these steps, see Section 4.4.1, “Replacing the Nodes”.

To upgrade a three control plane node cluster:

Create two Oracle Linux 8 nodes. The nodes are used as a temporary control plane nodes during the upgrade to maintain a cluster quorum.
Scale up the cluster to add the two new nodes as control plane nodes. The cluster now has five control plane nodes.
Scale down the cluster to remove two Oracle Linux 7 control plane nodes. The cluster now has three control plane nodes.
Upgrade the removed nodes to Oracle Linux 8 and prepare them to use as a new nodes.
Repeat the scaling up and down until all control plane nodes are replaced with Oracle Linux 8 nodes.
Upgrade the worker nodes by scaling down the cluster to remove a worker node, upgrading it, then scaling up to add it into the cluster again.

4.4.3 Replacing the Nodes in a Single Control Plane Node Cluster

If you have cluster with one control plane node and a load balancer, use the same procedure as described in Section 4.4.2, “Replacing the Nodes in a Three Control Plane Node Cluster”. You do not need to create two extra control plane nodes in this situation, one extra control plane node is sufficient. A single control plane node cluster does not meet the requirements for a cluster quorum, but creating an extra control plane node to add to the cluster while you upgrade the original node means your cluster remains up during the upgrade.

If you have a cluster with one control plane node and you have not set a load balancer, perform the following steps. During the upgrade the cluster is taken offline while the control plane node is replaced.

Important

During the upgrade you must use the same IP address and hostname for the control plane node you used when creating the cluster.

To migrate a single node cluster with no load balancer:

If the Platform API Server is also on the control plane node, you must migrate it to another node. For information on migrating the Platform API Server, see Section 4.1, “Upgrading the Operator Node”.
On the operator node, back up the cluster using the olcnectl module backup command. For example:
```
olcnectl module backup \
--environment-name myenvironment \
--name mycluster
```
Upgrade the control plane node to Oracle Linux 8. For information on preparing nodes, see Section 4.2, “Preparing an Oracle Linux 8 Node”.
On the operator node, restart the Platform API Server.
```
sudo systemctl restart olcne-api-server.service
```
Validate the nodes are set up correctly using the olcnectl module validate command. For example:
```
olcnectl module validate \
--environment-name myenvironment \
--name mycluster
```
Restore the cluster using the olcnectl module restore command. For example:
```
olcnectl module restore \
--environment-name myenvironment \
--name mycluster
```
When the control plane node is replaced with an Oracle Linux 8 node, you can replace the worker nodes. Details of replacing worker nodes is provided in Section 4.4.1, “Replacing the Nodes”.