JavaScript must be enabled to correctly display this content

The software described in this documentation is either no longer supported or is in extended support.
Oracle recommends that you upgrade to a current supported release.

Chapter 6 Scaling a Kubernetes Cluster

A Kubernetes cluster may consist of either a single or multiple control plane node and worker nodes. The more applications that you run in a cluster, the more resources (nodes) that you need. So, what do you do if you need additional resources to handle a high amount of workload or traffic, or if you want to deploy more services to the cluster? You add additional nodes to the cluster. Or, what happens if there are faulty nodes in your cluster? You remove them.

Scaling a Kubernetes cluster is updating the cluster by adding nodes to it or removing nodes from it. When you add nodes to a Kubernetes cluster, you are scaling up the cluster, and when you remove nodes from the cluster, you are scaling down the cluster.

If you want to replace a node in a cluster, first scale up the cluster (add the new node) and then scale down the cluster (remove the old node).

Note

Oracle recommends that you should not scale the cluster up and down at the same time. You should scale up, then scale down, in two separate commands. To avoid split-brain scenarios, scale your Kubernetes cluster control plane nodes in odd numbers. For example, 3, 5, or 7 control plane nodes ensures the reliability of your cluster.

If you used the --apiserver-advertise-address option when you created a Kubernetes module, then you cannot scale up from a cluster with a single control plane node to a highly available (HA) cluster with multiple control plane nodes. However, if you used the --virtual-ip or the --load-balancer options, then you can scale up, even if you have only a single control plane node cluster.

Important

The --apiserver-advertise-address option has been deprecated. Use the --master-nodes option.

When you scale a Kubernetes cluster, the following actions are completed:

A back up is taken of the cluster. In case something goes wrong during scaling up or scaling down, you can revert to the previous state so that you can restore the cluster. For more information about backing up and restoring a Kubernetes cluster, see Chapter 7, Backing up and Restoring a Kubernetes Cluster.
Any nodes that you want to add to the cluster are validated. If the nodes have any validation issues, such as firewall issues, then the update to the cluster cannot proceed, and the nodes cannot be added to the cluster. You are prompted for what to do to resolve the validation issues so that the nodes can be added to the cluster.
The control plane node and worker nodes are added to or removed from the cluster.
The cluster is checked to make sure all nodes are healthy. After validation of the cluster is completed, the cluster is scaled and you can access it.

Tip

The examples in this chapter show you how to scale up and down by changing the control plane node and worker nodes at the same time by providing all the nodes to be included in the cluster using the --master-nodes and --worker-nodes options. If you only want to scale control plane nodes, you only need to provide the list of control plane nodes to include in the cluster using the --master-nodes option (you do not need to provide all worker nodes). Similarly, if you only want to scale worker nodes, you only need to provide the list of worker nodes using the --worker-nodes option.

6.1 Scaling Up a Kubernetes Cluster

Before you scale up a Kubernetes cluster, set up the new nodes so they can be added to the cluster.

To prepare a node:

Set up the node so it can be added to a Kubernetes cluster. For information on setting up a Kubernetes node see Getting Started.
If you are using private X.509 certificates for nodes, you need to copy the certificates to the node. You do not need to do anything if you are using Vault to provide certificates for nodes. For information using X.509 certificates see Getting Started.
Start the Platform Agent service. For information on starting the Platform Agent, see Getting Started.

After completing these actions, use the instructions in this procedure to add nodes to a Kubernetes cluster.

To scale up a Kubernetes cluster:

From a control plane node of the Kubernetes cluster, use the kubectl get nodes command to see the control plane node and worker nodes of the cluster.

kubectl get nodes

NAME                   STATUS   ROLE     AGE     VERSION
control1.example.com    Ready    master   26h     v1.18.x+x.x.x.el7  
control2.example.com    Ready    master   26h     v1.18.x+x.x.x.el7
control3.example.com    Ready    master   26h     v1.18.x+x.x.x.el7
worker1.example.com     Ready    <none>   26h     v1.18.x+x.x.x.el7
worker2.example.com     Ready    <none>   26h     v1.18.x+x.x.x.el7
worker3.example.com     Ready    <none>   26h     v1.18.x+x.x.x.el7

In this example, there are three control plane nodes in the Kubernetes cluster:

control1.example.com
control2.example.com
control3.example.com

There are also three worker nodes in the cluster:

worker1.example.com
worker2.example.com
worker3.example.com

Use the olcnectl module update command to scale up a Kubernetes cluster.

In this example, the Kubernetes cluster is scaled up so that it has four control plane nodes and five worker nodes. This example adds a new control plane node (control.example.com) and two new workers nodes (worker4.example.com and worker5.example.com) to the Kubernetes module named mycluster. From the operator node run:
```
olcnectl module update \
--environment-name myenvironment \  
--name mycluster \
--master-nodes control1.example.com:8090,control2.example.com:8090,control3.example.com:8090,\
control4.example.com:8090 \
--worker-nodes worker1.example.com:8090,worker2.example.com:8090,worker3.example.com:8090,\
worker4.example.com:8090,worker5.example.com:8090
```
Make sure that if you are scaling up from a single control plane node to a highly available cluster, you have specified a load balancer for the cluster. If you do not specify a load balancer, then you cannot scale up your control plane nodes. That is, you cannot move from a single control plane node to a highly available cluster without a load balancer.

You can optionally include the --generate-scripts option. This option generates scripts you can run for each node in the event of any validation failures encountered during scaling. A script is created for each node in the module, saved to the local directory, and named hostname:8090.sh.

You can also optionally included the --force option to suppress the prompt displayed to confirm you want to continue with scaling the cluster.

On a control plane node of the Kubernetes cluster, use the kubectl get nodes command to verify the cluster is scaled up to include the new control plane node and worker nodes.

kubectl get nodes

NAME                   STATUS   ROLE     AGE     VERSION
control1.example.com    Ready   master   26h     v1.18.x+x.x.x.el7  
control2.example.com    Ready   master   26h     v1.18.x+x.x.x.el7
control3.example.com    Ready   master   26h     v1.18.x+x.x.x.el7
control4.example.com    Ready   master   2m38s   v1.18.x+x.x.x.el7
worker1.example.com     Ready   <none>   26h     v1.18.x+x.x.x.el7
worker2.example.com     Ready   <none>   26h     v1.18.x+x.x.x.el7
worker3.example.com     Ready   <none>   26h     v1.18.x+x.x.x.el7
worker4.example.com     Ready   <none>   2m38s   v1.18.x+x.x.x.el7
worker5.example.com     Ready   <none>   2m38s   v1.18.x+x.x.x.el7

6.2 Scaling Down a Kubernetes Cluster

This procedure shows you how to remove nodes from a Kubernetes cluster.

Warning

Be careful if you are scaling down the control plane nodes of your cluster. If you have two control plane nodes and you scale down to have only one control plane node, then you would have only a single point of failure.

To scale down a Kubernetes cluster:

From a control plane node of the Kubernetes cluster, use the kubectl get nodes command to see the control plane node and worker nodes of the cluster.

kubectl get nodes

NAME                   STATUS   ROLE     AGE     VERSION
control1.example.com   Ready    master   26h     v1.18.x+x.x.x.el7  
control2.example.com   Ready    master   26h     v1.18.x+x.x.x.el7
control3.example.com   Ready    master   26h     v1.18.x+x.x.x.el7
control4.example.com   Ready    master   2m38s   v1.18.x+x.x.x.el7
worker1.example.com    Ready    <none>   26h     v1.18.x+x.x.x.el7
worker2.example.com    Ready    <none>   26h     v1.18.x+x.x.x.el7
worker3.example.com    Ready    <none>   26h     v1.18.x+x.x.x.el7
worker4.example.com    Ready    <none>   2m38s   v1.18.x+x.x.x.el7
worker5.example.com    Ready    <none>   2m38s   v1.18.x+x.x.x.el7

In this example, there are four control plane nodes in the Kubernetes cluster:

control1.example.com
control2.example.com
control3.example.com
control4.example.com

There are also five worker nodes in the cluster:

worker1.example.com
worker2.example.com
worker3.example.com
worker4.example.com
worker5.example.com

Use the olcnectl module update command to scale down a Kubernetes cluster.

In this example, the Kubernetes cluster is scaled down so that it has three control plane nodes and three worker nodes. This example removes a control plane node (control4.example.com) and two workers nodes (worker4.example.com and worker5.example.com) from the Kubernetes module named mycluster. As the nodes are no longer listed in the --master-nodes or --worker-nodes options, they are removed from the cluster. From the operator node run:
```
olcnectl module update \
--environment-name myenvironment \  
--name mycluster \
--master-nodes control1.example.com:8090,control2.example.com:8090,control3.example.com:8090 \
--worker-nodes worker1.example.com:8090,worker2.example.com:8090,worker3.example.com:8090
```

On a control plane node of the Kubernetes cluster, use the kubectl get nodes command to verify the cluster is scaled down to remove the control plane node and worker nodes.

kubectl get nodes

NAME                   STATUS   ROLE     AGE     VERSION
control1.example.com   Ready    master   26h     v1.18.x+x.x.x.el7  
control2.example.com   Ready    master   26h     v1.18.x+x.x.x.el7
control3.example.com   Ready    master   26h     v1.18.x+x.x.x.el7
worker1.example.com    Ready    <none>   26h     v1.18.x+x.x.x.el7
worker2.example.com    Ready    <none>   26h     v1.18.x+x.x.x.el7
worker3.example.com    Ready    <none>   26h     v1.18.x+x.x.x.el7