Note:

This tutorial is available in an Oracle-provided free lab environment.
It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.

Deploy an Internal Load Balancer with Oracle Cloud Native Environment

Introduction

Oracle Cloud Native Environment is a fully integrated suite for developing and managing cloud native applications. The Kubernetes module is the core module. It deploys and manages containers and automatically installs and configures CRI-O and RunC. CRI-O manages the container runtime for a Kubernetes cluster, which defaults to RunC.

Objectives

At the end of this tutorial, you should be able to do the following:

Configure the Kubernetes cluster with an internal load balancer to enable high availability
Configure Oracle Cloud Native Environment on a 5-node cluster
Verify keepalived failover between the control plane nodes completes successfully

Support Note: Using the internal load balancer is NOT recommended for production deployments. Instead, please use a correctly configured (external) load balancer.

Prerequisites

The free lab environment uses the following host systems:

6 Oracle Linux instances for Oracle Cloud Native Environment:
- Operator node (ocne-operator)
- 3 Kubernetes control plane nodes (ocne-control-01, ocne-control-02, ocne-control-03 )
- 2 Kubernetes worker nodes (ocne-worker-01, ocne-worker-02)
Note: We recommend that production environments have a cluster with at least five control plane nodes and three worker nodes.
A virtual IP address for the primary control plane node. Do not use this IP address on any of the nodes. The load balancer dynamically sets the IP address to the control plane node assigned as the primary controller.

Note: If you are deploying to Oracle Cloud Infrastructure, your tenancy requires enabling a new feature introduced in OCI: Layer 2 Networking for VLANs within your virtual cloud networks (VCNs). The OCI Layer 2 Networking feature is not generally available, although the free lab environment’s tenancy enables this feature.

If you have a use case, please work with your technical team to get your tenancy listed to use this feature.
Configure each system with the following:
- The latest Oracle Linux with the Unbreakable Enterprise Kernel Release 7 (UEK R7)
- An oracle user account with sudo access
- Key-based SSH, also known as passwordless SSH, between the instances
- Install Oracle Cloud Native Environment ready for creating the environment and installing modules
- Create a VLAN and assign IPv4 addresses

Set Firewall Rules on Control Plane Nodes

Note: When using the free lab environment, see Oracle Linux Lab Basics for connection and other usage instructions.

Open a terminal and connect via ssh to the ocne-operator node.
```
ssh oracle@<ip_address_of_operator_node>
```
Set the firewall rules and enable the Virtual Router Redundancy Protocol (VRRP) protocol on each of the control plane nodes.
```
for host in ocne-control-01 ocne-control-02 ocne-control-03
do
  ssh $host "sudo firewall-cmd --zone=public --add-port=6444/tcp --permanent; sudo firewall-cmd --zone=public --add-protocol=vrrp --permanent; sudo firewall-cmd --reload"
done
```
Note: You must complete this step before proceeding to ensure the load balancer process can communicate between the control plane nodes.

Create a Platform CLI Configuration File

Administrators can use a configuration file to simplify creating and managing environments and modules. The configuration file, written in valid YAML syntax, includes all information about the environments and modules to create. Using a configuration file saves repeated entries of Platform CLI command options.

Note: If entering more than one control plane node in the myenvironment.yaml when configuring Oracle Cloud Native Environment, then olcnectl requires setting the virtual IP address option into the myenvironment.yaml file. Achieve this via entering a new argument (virtual-ip: <enter-your-ip-here>) into the myenvironment.yaml file.

During lab deployment, a configuration file is automatically generated and ready to use in the exercise. More information on manually creating a configuration file is in the documentation at Using a Configuration File.

Update the Configuration File

Add the virtual-ip value to the myenvironment.yaml file.

sed -i '14i\          virtual-ip: 10.0.12.111\' ~/myenvironment.yaml

Confirm the virtual-ip value has been added to the myenviroment.yaml file.
```
cat ~/myenvironment.yaml
```

Create the Environment and Kubernetes Module

Create the environment.

cd ~
olcnectl environment create --config-file myenvironment.yaml

Create the Kubernetes module.

olcnectl module create --config-file myenvironment.yaml

Validate the Kubernetes module.
```
olcnectl module validate --config-file myenvironment.yaml
```
In the free lab environment, there are no validation errors. The command’s output provides the steps required to fix the nodes if there are any errors.
Install the Kubernetes module.
```
olcnectl module install --config-file myenvironment.yaml
```
The deployment of Kubernetes to the nodes may take several minutes to complete.

Validate the deployment of the Kubernetes module.

olcnectl module instances --config-file myenvironment.yaml

Example Output:

[oracle@ocne-operator ~]$ olcnectl module instances --config-file myenvironment.yaml
INSTANCE        MODULE      STATE    
10.0.12.10:8090 node        installed
10.0.12.11:8090 node        installed
10.0.12.12:8090 node        installed
10.0.12.20:8090 node        installed
10.0.12.21:8090 node        installed
mycluster       kubernetes  installed
[oracle@ocne-operator ~]$

Set up kubectl

Set up the kubectl command on the control plane nodes.

for host in ocne-control-01 ocne-control-02 ocne-control-03
do
ssh $host /bin/bash <<EOF
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
  export KUBECONFIG=$HOME/.kube/config
  echo 'export KUBECONFIG=$HOME/.kube/config' >> $HOME/.bashrc
EOF
done

Repeating this step across each control plane node is essential due to the potential for a node going offline. In practice, install the kubectl utility in a separate node outside of the cluster.

Verify kubectl works and that the install completed successfully with all nodes listed as being in the ‘Ready status.

ssh ocne-control-01 kubectl get nodes

Example Output:

[oracle@ocne-control-01 ~]$ kubectl get nodes
NAME              STATUS   ROLES           AGE     VERSION
ocne-control-01   Ready    control-plane   12m     v1.28.3+3.el8
ocne-control-02   Ready    control-plane   10m     v1.28.3+3.el8
ocne-control-03   Ready    control-plane   9m28s   v1.28.3+3.el8
ocne-worker-01    Ready    <none>          8m36s   v1.28.3+3.el8
ocne-worker-02    Ready    <none>          8m51s   v1.28.3+3.el8

Confirm Failover Between Control Plane Nodes

The Oracle Cloud Native Environment installation with three control plane nodes behind an internal load balancer is complete.

The following steps confirm that the internal load balancer (using keepalived) will detect when the primary control plane node fails and passes control to one of the surviving control plane nodes. Likewise, when the ‘missing’ node recovers, it can automatically rejoin the cluster.

Locate the Primary Control Plane Node

Determine which control plane node currently holds the virtual IP address.

List each control plane node’s network devices and IP addresses.
```
for host in ocne-control-01 ocne-control-02 ocne-control-03
do
  ssh $host "echo -e '\n---- $host ----\n'; ip -br a"
done
```
The -br option summarizes the IP address information assigned to a device.

Look at the results, and find which host’s output contains the virtual IP associated with the ens5 NIC.

In the free lab environment, the virtual IP address used by the keepalived daemon is set to 10.0.12.111.

ens5             UP             10.0.12.11/24 10.0.12.111/32 fe80::41f6:2a0d:9a89:13d0/64 

Example Output:

---- ocne-control-01 ----

lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens3             UP             10.0.0.150/24 fe80::17ff:fe06:56dd/64 
ens5             UP             10.0.12.10/24 fe80::b5ee:51ab:8efd:92de/64 
flannel.1        UNKNOWN        10.244.0.0/32 fe80::e80c:9eff:fe27:e0b3/64 

---- ocne-control-02 ----

lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens3             UP             10.0.0.151/24 fe80::17ff:fe02:f7d0/64 
ens5             UP             10.0.12.11/24 fe80::d7b2:5f5b:704b:369/64 
flannel.1        UNKNOWN        10.244.1.0/32 fe80::48ae:67ff:fef1:862c/64 

---- ocne-control-03 ----

lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens3             UP             10.0.0.152/24 fe80::17ff:fe0b:7feb/64 
ens5             UP             10.0.12.12/24 10.0.12.111/32 fe80::ba57:2df0:d652:f8b0/64 
flannel.1        UNKNOWN        10.244.2.0/32 fe80::10db:c4ff:fee9:2121/64 
cni0             UP             10.244.2.1/24 fe80::bc0b:26ff:fe2d:383e/64 
veth1814a0d4@if2 UP             fe80::ec22:ddff:fe41:1695/64 
veth321409fc@if2 UP             fe80::485e:c4ff:feec:6914/64 

The ocne-control-03 node contains the virtual IP address in the example output.

Important: Take note of which control plane node currently holds the virtual IP address in your running environment.

(Optional) Confirm which control plane node holds the virtual IP address by querying the keepalived.service logs on each control plane node.
```
journalctl -u keeaplived
```
Example Output:
```
...
Aug 10 23:47:26 ocne-control01 Keepalived_vrrp[55605]: (VI_1) Entering MASTER STATE
...
```
This control plane node has the virtual IP address assigned.
Example Output:
```
...
Aug 10 23:54:59 ocne-control02 Keepalived_vrrp[59961]: (VI_1) Entering BACKUP STATE (init)
...
```
This control plane node does not.

Force the keepalived Daemon to Move to a Different Control Plane Node

Double-click the Luna Lab icon on the desktop in the free lab environment and navigate to the Luna Lab tab. Then click on the OCI Console hyperlink. Sign on using the provided User Name and Password values. After logging on, proceed with the following steps:

Start from this screen.
Click on the navigation menu in the page’s top-left corner, then Compute and Instances.
This displays the Instances page.

Locate the Compartment used from the drop-down list as instructed in Oracle Linux Lab Basics.
Click on the Instance name matching the one that contains the virtual IP address.
This shows the details for the Instance.
Click on the Stop button.

In the pop-up dialog, select the ‘Force stop the instance by immediately powering off’ checkbox, and click the ‘Force stop instance` button.

Note: Do NOT do this on a production system as it may incur data loss, corruption, or worse to the entire system.

Important Information related to Terminal session: After shutting down a specific control plane node, its terminal session will no longer be active or accessible.
Wait until the Instance details page confirms the instance is ‘Stopped’.
Switch from the browser back to the terminal session.
Open a terminal and connect via ssh to one of the running control plane nodes.

Confirm that the control plane node you just shut down is reporting as NotReady.

Note: You may need to repeat this step several times or notice a timeout until the status changes.

for host in ocne-control-01 ocne-control-02 ocne-control-03
do
  ssh -o ConnectTimeout=10 $host "echo -e '\n---- $host ----\n'; kubectl get nodes"
done

-o ConnectTimeout=<time-in-seconds> overrides the TCP timeout and will stop trying to connect to the system if it’s reported down or unreachable.

Example Output:

NAME              STATUS     ROLES           AGE   VERSION
ocne-control-01   Ready      control-plane   51m   v1.28.3+3.el8
ocne-control-02   Ready      control-plane   50m   v1.28.3+3.el8
ocne-control-03   NotReady   control-plane   49m   v1.28.3+3.el8
ocne-worker-01    Ready      <none>          48m   v1.28.3+3.el8
ocne-worker-02    Ready      <none>          48m   v1.28.3+3.el8

Determine which node has the running keepalived daemon.

for host in ocne-control-01 ocne-control-02 ocne-control-03
do
  ssh -o ConnectTimeout=10 $host "echo -e '\n---- $host ----\n'; ip -br a"
done

Example Output:

---- ocne-control-01 ----

lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens3             UP             10.0.0.150/24 fe80::17ff:fe06:56dd/64 
ens5             UP             10.0.12.10/24 fe80::b5ee:51ab:8efd:92de/64 
flannel.1        UNKNOWN        10.244.0.0/32 fe80::e80c:9eff:fe27:e0b3/64 

---- ocne-control-02 ----

lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens3             UP             10.0.0.151/24 fe80::17ff:fe02:f7d0/64 
ens5             UP             10.0.12.11/24 10.0.12.111/32 fe80::d7b2:5f5b:704b:369/64 
flannel.1        UNKNOWN        10.244.1.0/32 fe80::48ae:67ff:fef1:862c/64 
ssh: connect to host ocne-control-03 port 22: Connection timed out

The virtual IP is now associated with ocne-control02 per the sample output.

Switch back to the browser.
Start the Instance previously shut down in the Cloud Console by clicking the Start button.

Wait until the Status section confirms the Instance is ‘Running’.
Switch back to the terminal session.

Confirm that kubectl shows all control plane nodes as being Ready.

for host in ocne-control-01 ocne-control-02 ocne-control-03
do
  ssh -o ConnectTimeout=10 $host "echo -e '\n---- $host ----\n'; kubectl get nodes"
done

Note: It may be necessary to repeat this step several times until the status changes.

Example Output:

NAME              STATUS     ROLES           AGE   VERSION
ocne-control-01   Ready      control-plane   62m   v1.28.3+3.el8
ocne-control-02   Ready      control-plane   60m   v1.28.3+3.el8
ocne-control-03   Ready      control-plane   59m   v1.28.3+3.el8
ocne-worker-01    Ready      <none>          58m   v1.28.3+3.el8
ocne-worker-02    Ready      <none>          58m   v1.28.3+3.el8

Confirm the location of the active keepalived daemon.
```
for host in ocne-control-01 ocne-control-02 ocne-control-03
do
  ssh -o ConnectTimeout=10 $host "echo -e '\n---- $host ----\n'; ip -br a"
done
```
The keepalived daemon keeps the virtual IP with the currently active host, despite the original host restarting. This behavior happens because Oracle Cloud Native Environment sets each node’s weight equally in the keepalived configuration file.

Summary

These steps confirm that the Internal Load Balancer based on keepalived has been configured correctly and accepts requests successfully for Oracle Cloud Native Environment.

For More Information

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.

Title and Copyright Information

Deploy an Internal Load Balancer with Oracle Cloud Native Environment

F26338-20

February 2024