Note:

This tutorial is available in an Oracle-provided free lab environment.
It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.

Scale an Oracle Cloud Native Environment Cluster

Introduction

Scaling up a Kubernetes cluster means adding nodes; likewise, scaling down occurs by removing nodes. These nodes can be either control plane or worker nodes. Oracle recommends against scaling the cluster up and down simultaneously but instead performing a scale up and down in two separate commands.

Scaling the Kubernetes cluster control plane or worker nodes in odd numbers is recommended to avoid split-brain scenarios and maintain the quorum. For example, 3, 5, or 7 control plane nodes or worker nodes ensure the cluster’s reliability.

Objectives

In this tutorial, you’ll learn how to use kubectl scale and kubectl edit to:

Add a new control plane and worker nodes to a Kubernetes cluster
Remove control plane nodes and worker nodes from a Kubernetes cluster

Prerequisites

Minimum of one Oracle Linux instance
- Runs ocne and libvirt ephemeral cluster
OCI cluster creation requires access to the following resources in an Oracle Cloud Infrastructure tenancy:
- Virtual cloud network with four subnets
- Network load balancer
- Object Storage bucket with minimum 5 GiB available
- Compute Custom Image
- Compute Arm Shape for the control plane node
  - VM.Standard.A1.Flex with two OCPU and 12 memory
- Compute for each additional control plane and worker node
  - VM.Standard.E4.Flex with four OCPU and 64 memory

Configure Oracle Cloud Native Environment

Note: If running in your tenancy, read the linux-virt-labs GitHub project README.md and complete the prerequisites before deploying the lab environment.

Open a terminal on the Luna Desktop.

Clone the linux-virt-labs GitHub project.

git clone https://github.com/oracle-devrel/linux-virt-labs.git

Change into the working directory.
```
cd linux-virt-labs/ocne2
```

Install the required collections.

ansible-galaxy collection install -r requirements.yml

Increase the Boot volume size.

cat << EOF | tee instances.yml > /dev/null
compute_instances:
  1:
    instance_name: "ocne"
    type: "server"
    boot_volume_size_in_gbs: 128
install_ocne_rpm: true
install_type: "oci"
ocne_cluster_name: "mycluster"
create_ocne_oci_cluster: true
EOF

Deploy the lab environment.
```
ansible-playbook create_instance.yml -e localhost_python_interpreter="/usr/bin/python3.6" -e "@instances.yml"
```
The free lab environment requires the extra variable local_python_interpreter, which sets ansible_python_interpreter for plays running on localhost. This variable is needed because the environment installs the RPM package for the Oracle Cloud Infrastructure SDK for Python, located under the python3.6 modules.

The default deployment shape uses the AMD CPU and Oracle Linux 8. To use an Intel CPU or Oracle Linux 9, add -e instance_shape="VM.Standard3.Flex" or -e os_version="9" to the deployment command.

Important: Wait for the playbook to run successfully and reach the pause task. At this stage of the playbook, the installation of the Oracle Cloud Native Environment is complete, and the instances are ready. Take note of the previous play, which prints the public and private IP addresses of the nodes it deploys and any other deployment information needed while running the lab.

Confirm the Cluster is Running

Open a terminal and connect via SSH to the ocne instance.
```
ssh oracle@<ip_address_of_node>
```
Get a list of known clusters using the CLI.
```
ocne cluster list
```
Example Output:
```
[oracle@ocne ~]$ ocne cluster list
ocne-ephemeral
mycluster
```
The output shows two clusters are present, the one you just created (mycluster) and the bootstrap cluster (ocne-ephemeral).
Get the location of the kube configuration.
```
ocne cluster show -C mycluster
```
We use the -C to specify a specific cluster from the cluster list.

Set the KUBECONFIG environment variable.

export KUBECONFIG=$(ocne cluster show -C mycluster)

Wait for the cluster to stabilize and all nodes to report in a ready state.

watch kubectl get nodes -A

Example Output:

Every 2.0s: kubectl get nodes -A                               ocne: Wed Nov 13 16:40:08 2024

NAME                            STATUS   ROLES           AGE     VERSION
mycluster-control-plane-8dq98   Ready    control-plane   11m     v1.30.3+1.el8
mycluster-control-plane-dkscq   Ready    control-plane   8m      v1.30.3+1.el8
mycluster-control-plane-jr9gz   Ready    control-plane   9m      v1.30.3+1.el8
mycluster-md-0-8jr9k-8gnvw	    Ready    <none>          9m      v1.30.3+1.el8
mycluster-md-0-8jr9k-fn9mk	    Ready    <none>          9m      v1.30.3+1.el8
mycluster-md-0-8jr9k-kr9p6	    Ready    <none>          9m      v1.30.3+1.el8

Wait for the three worker and control plane nodes to show a STATUS of Ready, then type ctrl-c to exit the watch command.

Use the Scale Command to Scale the Control Plane and Worker Nodes

Kubernetes administrators typically use the kubectl scale command to scale resources deployed onto your Kubernetes cluster (e.g.: Deployments, StatefulSets, ReplicaSets, etc.). However, it can also scale the number of control and worker nodes in your Kubernetes cluster.

Scale Up the Control Plane and Worker Nodes

Confirm the bootstrap cluster is present.

sudo virsh list

Example Output:

[oracle@ocne ~]$ sudo virsh list
 Id   Name                             State
------------------------------------------------
 1    ocne-ephemeral-control-plane-1   running

Set the kubectl context to reference the bootstrap cluster.

export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local

Use the Cluster API Provider for OCI to confirm the number of nodes present.

watch kubectl get ocimachine -n ocne

Example Output:

Every 2.0s: kubectl get ocimachine -n ocne                                ocne: Wed Dec 18 11:29:08 2024

[oracle@ocne ~]$ kubectl get ocimachine -n ocne
NAME                            AGE
mycluster-control-plane-wl7n8   9m58s
mycluster-control-plane-dkscq   12m
mycluster-control-plane-jr9gz   15m
mycluster-md-0-fvtvm-6nzwb      10m
mycluster-md-0-fvtvm-qs4dl      10m
mycluster-md-0-fvtvm-ttpl5      10m

Wait until the output matches the cluster’s structure from the kubectl get nodes -A command. Then type ctrl-c to exit the watch command.

Get the name of the machinedeployments Custom Resource.

kubectl get machinedeployments -n ocne

Example Output:

[oracle@ocne ~]$ kubectl get machinedeployments -n ocne
NAME             CLUSTER     REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE    VERSION
mycluster-md-0   mycluster   3          3       3         0             Running   10m    v1.30.3

The NAME column shows the MachineDeployments, which reflect Kubernetes objects representing the worker nodes in the cluster.

Scale up a worker node from 3 to 4.

kubectl scale machinedeployment mycluster-md-0 --replicas=4 --namespace ocne

Scale up the control plane nodes from 3 to 5.

kubectl scale kubeadmcontrolplane mycluster-control-plane --replicas=5 --namespace ocne

Set the context back to the Workload cluster and confirm the new control plane nodes are present.
```
export KUBECONFIG=$(ocne cluster show -C mycluster)
watch kubectl get nodes -A
```
Wait for this to complete, but check for the existence of five control plane nodes and four worker nodes. Once they show as running, type ctrl-c to exit the watch command.

Scale Down the Control Plane and Worker Nodes

Set the kubectl context to reference the bootstrap cluster.

export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local

Scale down a worker node.

kubectl scale machinedeployment mycluster-md-0 --replicas=3 --namespace ocne

Scale down the control plane nodes.

kubectl scale kubeadmcontrolplane mycluster-control-plane --replicas=3 --namespace ocne

Set the context back to the Workload cluster and confirm the new control plane nodes are present.
```
export KUBECONFIG=$(ocne cluster show -C mycluster)
watch kubectl get nodes -A
```
Wait for the cluster to stabilize and the scale down to complete by leaving only three worker and three control plane nodes running. Then type ctrl-c to exit the watch command.

Use the Edit Command to Scale the Control Plane and Worker Nodes

The kubectl edit command provides both Kubernetes administrators and developers with a way to apply real-time direct edits and modification to a deployed Kubernetes API configuration or resource. By default, it opens the referenced Kubernetes API resource in vi on Linux or notepad on Windows. To use your favorite editor, set the KUBE_EDITOR or EDITOR variables within the same shell as you run the edit command. However, it is generally not recommended for use on production systems to avoid altering something accidentally. It is, however, a flexible way to test changes in development or test environments.

You can use the edit command to accomplish the same scale up and down actions as the scale command performs.

Scale Up a Worker Node

Confirm the bootstrap cluster is present.

sudo virsh list

Example Output:

[oracle@ocne ~]$ sudo virsh list
 Id   Name                             State
------------------------------------------------
 1    ocne-ephemeral-control-plane-1   running

Set the kubectl context to reference the bootstrap cluster.

export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local

Use the Cluster API Provider for OCI to confirm the number of nodes present.

kubectl get ocimachine -n ocne

Example Output:

[oracle@ocne ~]$ kubectl get ocimachine -n ocne
NAME                            AGE
mycluster-control-plane-dbrkt   5m45s
mycluster-control-plane-nrlh9   4m13s
mycluster-control-plane-wm7rg   8m34s
mycluster-md-0-9cczf-kspf9      9m11s
mycluster-md-0-9cczf-ngjgk      9m11s
mycluster-md-0-9cczf-z62zl      9m11s

Get the name of the machinedeployments Custom Resource.

kubectl get machinedeployments -n ocne

Example Output:

[oracle@ocne ~]$ kubectl get machinedeployments -n ocne
NAME             CLUSTER     REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE    VERSION
mycluster-md-0   mycluster   3          3       3         0             Running   10m    v1.30.3

Edit the Custom Resource.
```
kubectl edit machinedeployments --namespace ocne mycluster-md-0
```
This command opens the Custom Resource for editing.

Set the Replicas field to the number of the worker nodes you want to deploy.

For this example, change the replicas: 3 field to: replicas: 4. Then save and exit. You enter the total number of worker nodes, and the cluster will scale up or down accordingly.

Example Output:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  annotations:
...
...
spec:
  clusterName: mycluster
  minReadySeconds: 0
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 1
...
...

Note: This opens the Custom Resource in vi. Use the standard vi keystrokes to enter text, and to save the change and exit editor.

Confirm the creation of the new worker node.

kubectl get ocimachine --namespace ocne

Example Output:

[oracle@ocne ~]$ kubectl get ocimachine --namespace ocne
NAME                            AGE
mycluster-control-plane-lj9m9   11m
mycluster-control-plane-t6gc8   14m
mycluster-control-plane-w427z   12m
mycluster-md-0-fvtvm-bzfnc      8s
mycluster-md-0-fvtvm-qs4dl      38m
mycluster-md-0-fvtvm-ttpl5      38m
mycluster-md-0-fvtvm-xf4ch      22m

Set the context back to the Workload cluster and confirm the new worker node is present.
```
export KUBECONFIG=$(ocne cluster show -C mycluster)
watch kubectl get nodes -A
```
Wait for the new worker node to show a STATUS of Ready. Then type ctrl-c to exit the watch command.

Scale Up Control Plane Nodes

Set the kubectl context to reference the bootstrap cluster.

export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local

Get the name of the kubeadmcontrolplane Custom Resource.

kubectl get kubeadmcontrolplane -n ocne

Example Output:

[oracle@ocne ~]$ kubectl get kubeadmcontrolplane -n ocne
NAME                      CLUSTER     INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE   VERSION
mycluster-control-plane   mycluster   true          true                   3          3       3         0             13m   v1.30.3

Edit the Custom Resource.

kubectl edit kubeadmcontrolplane --namespace ocne mycluster-control-plane

Set the Replicas field to the new number of control plane nodes to deploy.

Change the replicas: 3 field to: replicas: 5, then save and exit.

Example Output:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be   
# reopened with the relevant failures.
#
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  creationTimestamp: "2024-11-05T17:03:52Z"
  finalizers:
  - kubeadm.controlplane.cluster.x-k8s.io
  generation: 3
  labels:
    cluster.x-k8s.io/cluster-name: mycluster
...
...
  machineTemplate:
   infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: OCIMachineTemplate
      name: mycluster-control-plane
      namespace: ocne
    metadata: {}
  replicas: 5
  rolloutStrategy:
...
...

Note: This opens the Custom Resource in vi. Use the standard vi keystrokes to enter text, and to save the change and exit editor.

Important: The number of control plane nodes set must always be an odd number or the scale request will not be completed.

Confirm the new control plane nodes deployed.

watch kubectl get ocimachine --namespace ocne

Example Output:

Every 2.0s: kubectl get ocimachine --namespace ocne                    ocne: Wed Dec 18 11:15:03 2024

NAME                            AGE
mycluster-control-plane-8wq8l   16s
mycluster-control-plane-lj9m9   16m
mycluster-control-plane-qzwmb   97s
mycluster-control-plane-t6gc8   19m
mycluster-control-plane-w427z   18m
mycluster-md-0-fvtvm-bzfnc      5m14s
mycluster-md-0-fvtvm-qs4dl      43m
mycluster-md-0-fvtvm-ttpl5      43m
mycluster-md-0-fvtvm-xf4ch      27m

Note: It will take a few minutes for both of the new control plane nodes to show. Then type ctrl-c to exit the watch command.

Set the context back to the Workload cluster and confirm the new control plane nodes are present.
```
export KUBECONFIG=$(ocne cluster show -C mycluster)
watch kubectl get nodes -A
```
Wait for both of the new control plane nodes to display under the NAME column with a ready STATUS. Then type ctrl-c to exit the watch command.

Scale Down a Worker Node

Set the kubectl context to reference the bootstrap cluster.

export KUBECONFIG=~/.kube/kubeconfig.ocne-ephemeral.local

Edit the Custom Resource.

kubectl edit machinedeployments --namespace ocne mycluster-md-0

Set the Replicas field to reduce the number of worker nodes.

Change the replicas: 4 field to: replicas: 3. Then save and exit.

Example Output:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  annotations:
...
...
spec:
  clusterName: mycluster
  minReadySeconds: 0
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 1
...
...

Note: This opens the Custom Resource in vi. Use the standard vi keystrokes to enter text, and to save the change and exit editor.

Scale Down the Control Plane Nodes

Edit the Custom Resource (using the NAME noted in the previous step).

kubectl edit kubeadmcontrolplane --namespace ocne mycluster-control-plane

Set the Replicas field to a lower number to reduce the number of control plane nodes.

Change the replicas: 5 field to: replicas: 3. Then save and exit.

Example Output:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be   
# reopened with the relevant failures.
#
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  creationTimestamp: "2024-11-05T17:03:52Z"
  finalizers:
  - kubeadm.controlplane.cluster.x-k8s.io
  generation: 3
  labels:
    cluster.x-k8s.io/cluster-name: mycluster
...
...
  machineTemplate:
   infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: OCIMachineTemplate
      name: mycluster-control-plane
      namespace: ocne
    metadata: {}
  replicas: 5
  rolloutStrategy:
...
...

Note: This opens the Custom Resource in vi. Use the standard vi keystrokes to enter text, and to save the change and exit editor.

Important: The number of control plane nodes set must always be an odd number or the scale request will not be completed.

Confirm the removal of the control plane nodes.

watch kubectl get ocimachine --namespace ocne

Example Output:

Every 2.0s: kubectl get ocimachine --namespace ocne                    ocne: Wed Dec 18 11:35:43 2024

NAME                            AGE
mycluster-control-plane-8wq8l   10m
mycluster-control-plane-qzwmb   12m
mycluster-control-plane-t6gc8   30m
mycluster-md-0-fvtvm-qs4dl	    54m
mycluster-md-0-fvtvm-ttpl5	    54m
mycluster-md-0-fvtvm-xf4ch	    37m

When the removal process is complete, you can type ctrl-c to exit the watch command.

Set the context back to the Workload cluster and confirm the removal of the control plane nodes.
```
export KUBECONFIG=$(ocne cluster show -C mycluster)
kubectl get nodes -A
```

Next Steps

That completes the demonstration of adding and removing nodes from a Kubernetes cluster. While this exercise demonstrated updating the control plane and worker nodes simultaneously, this is not the recommended approach to scaling up or down an Oracle Cloud Native Environment Kubernetes cluster. In production environments, administrators should undertake these tasks separately.

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.

Title and Copyright Information

Scale an Oracle Cloud Native Environment Cluster

G23482-01

December 2024

Oracle and/or its affiliates.

Scale an Oracle Cloud Native Environment Cluster

Introduction

Objectives

Prerequisites

Configure Oracle Cloud Native Environment

Confirm the Cluster is Running

Use the Scale Command to Scale the Control Plane and Worker Nodes

Scale Up the Control Plane and Worker Nodes

Scale Down the Control Plane and Worker Nodes

Use the Edit Command to Scale the Control Plane and Worker Nodes

Scale Up a Worker Node

Scale Up Control Plane Nodes

Scale Down a Worker Node

Scale Down the Control Plane Nodes

Next Steps

Related Links

More Learning Resources