Note:
- This tutorial is available in an Oracle-provided free lab environment.
- It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.
Deploy an Internal Load Balancer with Oracle Cloud Native Environment
Introduction
Oracle Cloud Native Environment is a fully integrated suite for developing and managing cloud native applications. The Kubernetes module is the core module. It deploys and manages containers and automatically installs and configures CRI-O and RunC. CRI-O manages the container runtime for a Kubernetes cluster, which defaults to RunC.
Objectives
At the end of this tutorial, you should be able to do the following:
- Configure the Kubernetes cluster with an internal load balancer to enable high availability
- Configure Oracle Cloud Native Environment on a 5-node cluster
- Verify keepalived failover between the control plane nodes completes successfully
Support Note: Using the internal load balancer is NOT recommended for production deployments. Instead, please use a correctly configured (external) load balancer.
Prerequisites
The free lab environment uses the following host systems:
-
6 Oracle Linux instances for Oracle Cloud Native Environment:
- Operator node (ocne-operator)
- 3 Kubernetes control plane nodes (ocne-control-01, ocne-control-02, ocne-control-03 )
- 2 Kubernetes worker nodes (ocne-worker-01, ocne-worker-02)
Note: We recommend that production environments have a cluster with at least five control plane nodes and three worker nodes.
-
A virtual IP address for the primary control plane node. Do not use this IP address on any of the nodes. The load balancer dynamically sets the IP address to the control plane node assigned as the primary controller.
Note: If you are deploying to Oracle Cloud Infrastructure, your tenancy requires enabling a new feature introduced in OCI: Layer 2 Networking for VLANs within your virtual cloud networks (VCNs). The OCI Layer 2 Networking feature is not generally available, although the free lab environment’s tenancy enables this feature.
If you have a use case, please work with your technical team to get your tenancy listed to use this feature. -
Configure each system with the following:
- The latest Oracle Linux with the Unbreakable Enterprise Kernel Release 7 (UEK R7)
- An
oracle
user account withsudo
access - Key-based SSH, also known as passwordless SSH, between the instances
- Install Oracle Cloud Native Environment ready for creating the environment and installing modules
- Create a VLAN and assign IPv4 addresses
Set Firewall Rules on Control Plane Nodes
Note: When using the free lab environment, see Oracle Linux Lab Basics for connection and other usage instructions.
-
Open a terminal and connect via ssh to the ocne-operator node.
ssh oracle@<ip_address_of_operator_node>
-
Set the firewall rules and enable the Virtual Router Redundancy Protocol (VRRP) protocol on each of the control plane nodes.
for host in ocne-control-01 ocne-control-02 ocne-control-03 do ssh $host "sudo firewall-cmd --zone=public --add-port=6444/tcp --permanent; sudo firewall-cmd --zone=public --add-protocol=vrrp --permanent; sudo firewall-cmd --reload" done
Note: You must complete this step before proceeding to ensure the load balancer process can communicate between the control plane nodes.
Create a Platform CLI Configuration File
Administrators can use a configuration file to simplify creating and managing environments and modules. The configuration file, written in valid YAML syntax, includes all information about the environments and modules to create. Using a configuration file saves repeated entries of Platform CLI command options.
Note: If entering more than one control plane node in the myenvironment.yaml when configuring Oracle Cloud Native Environment, then olcnectl requires setting the virtual IP address option into the myenvironment.yaml file. Achieve this via entering a new argument (
virtual-ip: <enter-your-ip-here>
) into the myenvironment.yaml file.
During lab deployment, a configuration file is automatically generated and ready to use in the exercise. More information on manually creating a configuration file is in the documentation at Using a Configuration File.
Update the Configuration File
-
Add the virtual-ip value to the myenvironment.yaml file.
sed -i '14i\ virtual-ip: 10.0.12.111\' ~/myenvironment.yaml
-
Confirm the virtual-ip value has been added to the myenviroment.yaml file.
cat ~/myenvironment.yaml
Create the Environment and Kubernetes Module
-
Create the environment.
cd ~ olcnectl environment create --config-file myenvironment.yaml
-
Create the Kubernetes module.
olcnectl module create --config-file myenvironment.yaml
-
Validate the Kubernetes module.
olcnectl module validate --config-file myenvironment.yaml
In the free lab environment, there are no validation errors. The command’s output provides the steps required to fix the nodes if there are any errors.
-
Install the Kubernetes module.
olcnectl module install --config-file myenvironment.yaml
The deployment of Kubernetes to the nodes may take several minutes to complete.
-
Validate the deployment of the Kubernetes module.
olcnectl module instances --config-file myenvironment.yaml
Example Output:
[oracle@ocne-operator ~]$ olcnectl module instances --config-file myenvironment.yaml INSTANCE MODULE STATE 10.0.12.10:8090 node installed 10.0.12.11:8090 node installed 10.0.12.12:8090 node installed 10.0.12.20:8090 node installed 10.0.12.21:8090 node installed mycluster kubernetes installed [oracle@ocne-operator ~]$
Set up kubectl
-
Set up the
kubectl
command on the control plane nodes.for host in ocne-control-01 ocne-control-02 ocne-control-03 do ssh $host /bin/bash <<EOF mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config export KUBECONFIG=$HOME/.kube/config echo 'export KUBECONFIG=$HOME/.kube/config' >> $HOME/.bashrc EOF done
Repeating this step across each control plane node is essential due to the potential for a node going offline. In practice, install the
kubectl
utility in a separate node outside of the cluster. -
Verify
kubectl
works and that the install completed successfully with all nodes listed as being in the ‘Ready status.ssh ocne-control-01 kubectl get nodes
Example Output:
[oracle@ocne-control-01 ~]$ kubectl get nodes NAME STATUS ROLES AGE VERSION ocne-control-01 Ready control-plane 12m v1.28.3+3.el8 ocne-control-02 Ready control-plane 10m v1.28.3+3.el8 ocne-control-03 Ready control-plane 9m28s v1.28.3+3.el8 ocne-worker-01 Ready <none> 8m36s v1.28.3+3.el8 ocne-worker-02 Ready <none> 8m51s v1.28.3+3.el8
Confirm Failover Between Control Plane Nodes
The Oracle Cloud Native Environment installation with three control plane nodes behind an internal load balancer is complete.
The following steps confirm that the internal load balancer (using keepalived
) will detect when the primary control plane node fails and passes control to one of the surviving control plane nodes. Likewise, when the ‘missing’ node recovers, it can automatically rejoin the cluster.
Locate the Primary Control Plane Node
Determine which control plane node currently holds the virtual IP address.
-
List each control plane node’s network devices and IP addresses.
for host in ocne-control-01 ocne-control-02 ocne-control-03 do ssh $host "echo -e '\n---- $host ----\n'; ip -br a" done
The
-br
option summarizes the IP address information assigned to a device. -
Look at the results, and find which host’s output contains the virtual IP associated with the
ens5
NIC.In the free lab environment, the virtual IP address used by the
keepalived
daemon is set to 10.0.12.111.ens5 UP 10.0.12.11/24 10.0.12.111/32 fe80::41f6:2a0d:9a89:13d0/64
Example Output:
---- ocne-control-01 ---- lo UNKNOWN 127.0.0.1/8 ::1/128 ens3 UP 10.0.0.150/24 fe80::17ff:fe06:56dd/64 ens5 UP 10.0.12.10/24 fe80::b5ee:51ab:8efd:92de/64 flannel.1 UNKNOWN 10.244.0.0/32 fe80::e80c:9eff:fe27:e0b3/64 ---- ocne-control-02 ---- lo UNKNOWN 127.0.0.1/8 ::1/128 ens3 UP 10.0.0.151/24 fe80::17ff:fe02:f7d0/64 ens5 UP 10.0.12.11/24 fe80::d7b2:5f5b:704b:369/64 flannel.1 UNKNOWN 10.244.1.0/32 fe80::48ae:67ff:fef1:862c/64 ---- ocne-control-03 ---- lo UNKNOWN 127.0.0.1/8 ::1/128 ens3 UP 10.0.0.152/24 fe80::17ff:fe0b:7feb/64 ens5 UP 10.0.12.12/24 10.0.12.111/32 fe80::ba57:2df0:d652:f8b0/64 flannel.1 UNKNOWN 10.244.2.0/32 fe80::10db:c4ff:fee9:2121/64 cni0 UP 10.244.2.1/24 fe80::bc0b:26ff:fe2d:383e/64 veth1814a0d4@if2 UP fe80::ec22:ddff:fe41:1695/64 veth321409fc@if2 UP fe80::485e:c4ff:feec:6914/64
The ocne-control-03 node contains the virtual IP address in the example output.
Important: Take note of which control plane node currently holds the virtual IP address in your running environment.
-
(Optional) Confirm which control plane node holds the virtual IP address by querying the
keepalived.service
logs on each control plane node.journalctl -u keeaplived
Example Output:
... Aug 10 23:47:26 ocne-control01 Keepalived_vrrp[55605]: (VI_1) Entering MASTER STATE ...
This control plane node has the virtual IP address assigned.
Example Output:
... Aug 10 23:54:59 ocne-control02 Keepalived_vrrp[59961]: (VI_1) Entering BACKUP STATE (init) ...
This control plane node does not.
Force the keepalived Daemon to Move to a Different Control Plane Node
Double-click the Luna Lab icon on the desktop in the free lab environment and navigate to the Luna Lab tab. Then click on the OCI Console hyperlink. Sign on using the provided User Name and Password values. After logging on, proceed with the following steps:
-
Start from this screen.
-
Click on the navigation menu in the page’s top-left corner, then Compute and Instances.
-
This displays the Instances page.
Locate the Compartment used from the drop-down list as instructed in Oracle Linux Lab Basics.
-
Click on the Instance name matching the one that contains the virtual IP address.
-
This shows the details for the Instance.
-
Click on the Stop button.
In the pop-up dialog, select the ‘Force stop the instance by immediately powering off’ checkbox, and click the ‘Force stop instance` button.
Note: Do NOT do this on a production system as it may incur data loss, corruption, or worse to the entire system.
Important Information related to Terminal session: After shutting down a specific control plane node, its terminal session will no longer be active or accessible.
-
Wait until the Instance details page confirms the instance is ‘Stopped’.
-
Switch from the browser back to the terminal session.
-
Open a terminal and connect via ssh to one of the running control plane nodes.
-
Confirm that the control plane node you just shut down is reporting as NotReady.
Note: You may need to repeat this step several times or notice a timeout until the status changes.
for host in ocne-control-01 ocne-control-02 ocne-control-03 do ssh -o ConnectTimeout=10 $host "echo -e '\n---- $host ----\n'; kubectl get nodes" done
-o ConnectTimeout=<time-in-seconds>
overrides the TCP timeout and will stop trying to connect to the system if it’s reported down or unreachable.
Example Output:
NAME STATUS ROLES AGE VERSION ocne-control-01 Ready control-plane 51m v1.28.3+3.el8 ocne-control-02 Ready control-plane 50m v1.28.3+3.el8 ocne-control-03 NotReady control-plane 49m v1.28.3+3.el8 ocne-worker-01 Ready <none> 48m v1.28.3+3.el8 ocne-worker-02 Ready <none> 48m v1.28.3+3.el8
-
Determine which node has the running
keepalived
daemon.for host in ocne-control-01 ocne-control-02 ocne-control-03 do ssh -o ConnectTimeout=10 $host "echo -e '\n---- $host ----\n'; ip -br a" done
Example Output:
---- ocne-control-01 ---- lo UNKNOWN 127.0.0.1/8 ::1/128 ens3 UP 10.0.0.150/24 fe80::17ff:fe06:56dd/64 ens5 UP 10.0.12.10/24 fe80::b5ee:51ab:8efd:92de/64 flannel.1 UNKNOWN 10.244.0.0/32 fe80::e80c:9eff:fe27:e0b3/64 ---- ocne-control-02 ---- lo UNKNOWN 127.0.0.1/8 ::1/128 ens3 UP 10.0.0.151/24 fe80::17ff:fe02:f7d0/64 ens5 UP 10.0.12.11/24 10.0.12.111/32 fe80::d7b2:5f5b:704b:369/64 flannel.1 UNKNOWN 10.244.1.0/32 fe80::48ae:67ff:fef1:862c/64 ssh: connect to host ocne-control-03 port 22: Connection timed out
The virtual IP is now associated with ocne-control02 per the sample output.
-
Switch back to the browser.
-
Start the Instance previously shut down in the Cloud Console by clicking the Start button.
Wait until the Status section confirms the Instance is ‘Running’.
-
Switch back to the terminal session.
-
Confirm that
kubectl
shows all control plane nodes as being Ready.for host in ocne-control-01 ocne-control-02 ocne-control-03 do ssh -o ConnectTimeout=10 $host "echo -e '\n---- $host ----\n'; kubectl get nodes" done
Note: It may be necessary to repeat this step several times until the status changes.
Example Output:
NAME STATUS ROLES AGE VERSION ocne-control-01 Ready control-plane 62m v1.28.3+3.el8 ocne-control-02 Ready control-plane 60m v1.28.3+3.el8 ocne-control-03 Ready control-plane 59m v1.28.3+3.el8 ocne-worker-01 Ready <none> 58m v1.28.3+3.el8 ocne-worker-02 Ready <none> 58m v1.28.3+3.el8
-
Confirm the location of the active
keepalived
daemon.for host in ocne-control-01 ocne-control-02 ocne-control-03 do ssh -o ConnectTimeout=10 $host "echo -e '\n---- $host ----\n'; ip -br a" done
The
keepalived
daemon keeps the virtual IP with the currently active host, despite the original host restarting. This behavior happens because Oracle Cloud Native Environment sets each node’s weight equally in thekeepalived
configuration file.
Summary
These steps confirm that the Internal Load Balancer based on keepalived
has been configured correctly and accepts requests successfully for Oracle Cloud Native Environment.
For More Information
- Oracle Cloud Native Environment Documentation
- Oracle Cloud Native Environment Track
- Oracle Linux Training Station
More Learning Resources
Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.
For product documentation, visit Oracle Help Center.
Deploy an Internal Load Balancer with Oracle Cloud Native Environment
F26338-20
February 2024
Copyright © 2021, Oracle and/or its affiliates.