Scale a Kubernetes Cluster on Oracle Cloud Native Environment

Introduction

This tutorial shows you how to scale an existing Kubernetes cluster in Oracle Cloud Native Environment Release 1.4.

You scale up a Kubernetes cluster by adding nodes, and you scale down by removing nodes. Nodes can be either control plane or worker nodes.

Oracle recommends that you should not scale the cluster up and down at the same time. You should scale up, then scale down, in two separate commands. To avoid split-brain scenarios, scale your Kubernetes cluster control plane nodes in odd numbers. For example, 3, 5, or 7 control plane nodes ensures the reliability of your cluster.

This tutorial assumes you have an existing Highly Available Kubernetes cluster running on Oracle Cloud Native Environment. This tutorial builds upon the tutorial to deploy a highly Available Oracle Cloud Native Environment at Deploy a Highly Available Oracle Cloud Native Environment .

Objectives

This tutorial steps you through configuring and adding two new control plane nodes and a worker node to the cluster. The tutorial then shows you how to scale down the cluster by removing the nodes from the cluster.

In this tutorial, X.509 Private CA Certificates are used to secure communication between the nodes. There are other methods to manage and deploy the certificates, such as by using HashiCorp Vault secrets manager, or by using your own certificates, signed by a trusted Certificate Authority (CA). These other methods are not included in this tutorial.

Prerequisites

In addition to the requirement of a Highly Available Kubernetes cluster running on Oracle Cloud Native Environment, you need a number of other hosts. You need:

Set up the New Kubernetes Nodes

On the new control plane nodes and the new worker node, make sure the systems prerequisites listed in the Prerequisites section of this tutorial are met.

In this tutorial, systems named control4.example.com and control5.example.com are the new control plane nodes, and a system named worker3.example.com is the new worker node. On both the new control plane nodes and the new worker node, install and enable the Platform Agent.

If you are using Oracle Linux 7, do the following:

sudo yum install olcne-agent olcne-utils
sudo systemctl enable olcne-agent.service

If you are using Oracle Linux 8, do the following:

sudo dnf install olcne-agent olcne-utils
sudo systemctl enable olcne-agent.service

If you use a proxy server, configure it with CRI-O. On each Kubernetes node, create a CRI-O systemd configuration directory. Create a file named proxy.conf in the directory and add the proxy server information.

sudo mkdir /etc/systemd/system/crio.service.d
sudo vi /etc/systemd/system/crio.service.d/proxy.conf

Substitute the appropriate proxy values for those in your environment using the example proxy.conf file:

[Service]
Environment="HTTP_PROXY=proxy.example.com:80"
Environment="HTTPS_PROXY=proxy.example.com:80"
Environment="NO_PROXY=.example.com,192.0.2.*"

If the docker service is running, or if the containerd service is running, stop and disable them.

sudo systemctl disable --now docker.service
sudo systemctl disable --now containerd.service

Set up X.509 Private CA Certificates

Set up X.509 Private CA Certificates for the new control plane nodes and the worker node. Use the /etc/olcne/gen-certs-helper.sh script to generate a private CA and certificates for the nodes. Run the script from the /etc/olcne directory on the operator node. Use the --byo-ca-cert option to specify the location of the existing CA Certificate, and the --byo-ca-key option to specify the location of the existing CA Key. Use the --nodes option and provide the FQDN of the new control plane and worker nodes.

cd /etc/olcne
sudo ./gen-certs-helper.sh \
--cert-dir /etc/olcne/configs/certificates/ \
--byo-ca-cert /etc/olcne/configs/certificates/production/ca.cert \
--byo-ca-key /etc/olcne/configs/certificates/production/ca.key \
--nodes control4.example.com,control5.example.com,worker3.example.com

Transfer Certificates

Run the script to transfer the certificate files from the operator node to the new nodes. Make sure the operator node has passwordless ssh access to the new control plane and worker nodes (not shown in this tutorial), then run the following command from the operator node to transfer the certificates to the new nodes.

bash -ex /etc/olcne/configs/certificates/olcne-tranfer-certs.sh

Configure the Platform Agent to Use the Certificates

Configure the Platform Agent to use the certificates for the new control plane nodes and the new worker node. On each new Kubernetes node, run the /etc/olcne/bootstrap-olcne.sh script as shown to configure the Platform Agent to use the certificates.

sudo /etc/olcne/bootstrap-olcne.sh \
--secret-manager-type file \
--olcne-node-cert-path /etc/olcne/configs/certificates/production/node.cert \
--olcne-ca-path /etc/olcne/configs/certificates/production/ca.cert \
--olcne-node-key-path /etc/olcne/configs/certificates/production/node.key \
--olcne-component agent

View the Kubernetes Nodes

On a control plane node, use the following command to view the Kubernetes nodes in the cluster. Note that there are three control plane nodes and two worker nodes.

kubectl get nodes

The output should look similar to:

NAME                   STATUS   ROLES    AGE   VERSION
control1.example.com   Ready    master   ...   ...
control2.example.com   Ready    master   ...   ...
control3.example.com   Ready    master   ...   ...
worker1.example.com    Ready    <none>   ...   ...
worker2.example.com    Ready    <none>   ...   ...

Add Control Plane Nodes to the Deployment Configuration File

On the operator node, edit the YAML deployment configuration file to include the new nodes you want to add to the cluster. The filename for the configuration file in this tutorial is myenvironment.yaml and includes three control plane and two worker nodes:

environments:
  - environment-name: myenvironment
    globals:
      api-server: 127.0.0.1:8091
      secret-manager-type: file
      olcne-ca-path: /etc/olcne/configs/certificates/production/ca.cert
      olcne-node-cert-path: /etc/olcne/configs/certificates/production/node.cert
      olcne-node-key-path:  /etc/olcne/configs/certificates/production/node.key
      update-config: true         
   modules:
     - module: kubernetes
       name: mycluster
       args:
         container-registry: container-registry.oracle.com/olcne
         virtual-ip: 192.0.2.137            
         master-nodes: 
           - control1.example.com:8090
           - control2.example.com:8090
           - control3.example.com:8090              
         worker-nodes: 
           - worker1.example.com:8090
           - worker2.example.com:8090           
         selinux: enforcing
         restrict-service-externalip: true            
         restrict-service-externalip-ca-cert: /etc/olcne/configs/certificates/restrict_external_ip/production/ca.cert
         restrict-service-externalip-tls-cert: /etc/olcne/configs/certificates/restrict_external_ip/production/node.cert
         restrict-service-externalip-tls-key: /etc/olcne/configs/certificates/restrict_external_ip/production/node.key

Add the FQDN and Platform Agent access port (8090) for all control plane nodes to be included in the cluster to the master-nodes section. The configuration file now includes the new control plane nodes (control4.example.com and control5.example.com). The control plane nodes you include in this option specifies all the control plane nodes that should be in the cluster after it is updated.

...
         master-nodes: 
           - control1.example.com:8090
           - control2.example.com:8090
           - control3.example.com:8090
           - control4.example.com:8090
           - control5.example.com:8090              
         worker-nodes: 
           - worker1.example.com:8090
           - worker2.example.com:8090
...              

Scale Up the Control Plane Nodes

On the operator node, run the olcnectl module update command with the --config-file option to specify the location of the configuration file. The Platform API Server validates the configuration file with the state of the cluster and recognises there are more nodes that should be added to the cluster. Answer yes when prompted.

olcnectl module update --config-file myenvironment.yaml

The update may take several minutes to complete. Note that the update was successful.

On a control plane node, run the following command to confirm that the new control plane nodes are added to the cluster.

kubectl get nodes

The output should look similar to:

NAME                   STATUS   ROLES    AGE   VERSION
control1.example.com   Ready    master   ...   ...
control2.example.com   Ready    master   ...   ...
control3.example.com   Ready    master   ...   ...
control4.example.com   Ready    master   ...   ...
control5.example.com   Ready    master   ...   ...
worker1.example.com    Ready    <none>   ...   ...
worker2.example.com    Ready    <none>   ...   ...

You can see the new control plane nodes named control4.example.com and control5.example.com are now included in the cluster.

Add Worker Nodes to the Configuration File

On the operator node, edit the deployment configuration file to include the new nodes you want to add to the cluster.

Add the FQDN and Platform Agent access port (8090) for all worker nodes to be included in the cluster to the worker-nodes section. The configuration file now includes the new worker node (worker3.example.com). The worker nodes you include in this option specifies all the worker nodes that should be in the cluster after it is updated.

...
         master-nodes: 
           - control1.example.com:8090
           - control2.example.com:8090
           - control3.example.com:8090
           - control4.example.com:8090
           - control5.example.com:8090              
         worker-nodes: 
           - worker1.example.com:8090
           - worker2.example.com:8090
           - worker3.example.com:8090
...              

Scale Up the Worker Nodes

On the operator node, run the olcnectl module update command with the --config-file option to specify the location of the configuration file. The Platform API Server validates the configuration file with the state of the cluster and recognises there are more nodes that should be added to the cluster.

This example includes the --force option to bypass the confirmation prompt.

Tip: You can also include the force: true option in the Kubernetes module section of the configuration file so you do not need to use the --force option to suppress the prompt.

modules:
  - module: kubernetes
    name: mycluster
    force: true
olcnectl module update --config-file myenvironment.yaml --force

On a control plane node, run the following command to confirm that the new worker node is added to the cluster.

kubectl get nodes

The output should look similar to:

NAME                   STATUS   ROLES    AGE   VERSION
control1.example.com   Ready    master   ...   ...
control2.example.com   Ready    master   ...   ...
control3.example.com   Ready    master   ...   ...
control4.example.com   Ready    master   ...   ...
control5.example.com   Ready    master   ...   ...
worker1.example.com    Ready    <none>   ...   ...
worker2.example.com    Ready    <none>   ...   ...
worker3.example.com    Ready    <none>   ...   ...

You can see the new worker node named worker3.example.com is now included in the cluster.

Scale Down the Control Plane Nodes

To remove nodes from the cluster, again, you update the configuration file with the nodes you want in the cluster, then run the olcnectl module update command.

To scale the cluster back down to the original three control plane nodes, remove the control4.example.com and control5.example.com control plane nodes from the configuration file:

...
         master-nodes: 
           - control1.example.com:8090
           - control2.example.com:8090
           - control3.example.com:8090              
         worker-nodes: 
           - worker1.example.com:8090
           - worker2.example.com:8090
           - worker3.example.com:8090
...              

Then run the command to update the cluster:

olcnectl module update --config-file myenvironment.yaml --force

The control plane nodes are removed from the cluster by the Platform API Server.

On a control plane node, run the following command to confirm that only the initial control plane nodes are included in the cluster and that control4.example.com and control5.example.com have been removed.

kubectl get nodes

The output should look similar to:

NAME                   STATUS   ROLES    AGE   VERSION
control1.example.com   Ready    master   ...   ...
control2.example.com   Ready    master   ...   ...
control3.example.com   Ready    master   ...   ...
worker1.example.com    Ready    <none>   ...   ...
worker2.example.com    Ready    <none>   ...   ...
worker3.example.com    Ready    <none>   ...   ...

Scale Down the Worker Nodes

To scale the cluster back down to the original two worker nodes, remove the worker3.example.com worker node from the configuration file:

...
         master-nodes: 
           - control1.example.com:8090
           - control2.example.com:8090
           - control3.example.com:8090              
         worker-nodes: 
           - worker1.example.com:8090
           - worker2.example.com:8090
...              

Then run the command to update the cluster:

olcnectl module update --config-file myenvironment.yaml --force

On a control plane node, run the following command to confirm that only the initial worker nodes are included in the cluster and that worker3.example.com node has been removed.

kubectl get nodes

The output should look similar to:

NAME                   STATUS   ROLES    AGE   VERSION
control1.example.com   Ready    master   ...   ...
control2.example.com   Ready    master   ...   ...
control3.example.com   Ready    master   ...   ...
worker1.example.com    Ready    <none>   ...   ...
worker2.example.com    Ready    <none>   ...   ...

For More Information

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.