Note:

Deploy an Internal Load Balancer with Oracle Cloud Native Environment

Introduction

Oracle Cloud Native Environment is a fully integrated suite for the development and management of cloud native applications. The Kubernetes module is the core module. It is used to deploy and manage containers and also automatically installs and configures CRI-O, runC and Kata Containers. CRI-O manages the container runtime for a Kubernetes cluster. The runtime may be either runC or Kata Containers.

Objectives

This tutorial/lab demonstrates how to:

Support Note: Using the internal load balancer is NOT recommended for production deployments. Instead, please use a correctly configured (external) load-balancer.

Prerequisites

The host systems to perform the steps in this tutorial are listed in this section. You need:

Set Up Lab Environment

Note: When using the free lab environment, see Oracle Linux Lab Basics for connection and other usage instructions.

This lab involves multiple systems, each of which requires different steps to be performed. It is recommended to start by opening a terminal window or tab and connecting to each node. This avoids you having to repeatedly log in and out. The nodes are:

  1. Open a terminal and connect via ssh to each of the nodes.

    ssh oracle@<ip_address_of_ol_node>
    

Set Firewall Rules on Control Plane Nodes

(On all control plane nodes) Set the firewall rules and enable the Virtual Router Redundancy Protocol (VRRP) protocol for all of the control plane nodes.

sudo firewall-cmd --add-port=6444/tcp --zone=public --permanent
sudo firewall-cmd --add-protocol=vrrp --zone=public --permanent
sudo firewall-cmd --reload

Note: This must be completed before proceeding to ensure the load balancer process can communicate between the control plane nodes.

Create a Platform CLI Configuration File

Administrators can use a configuration file to simplify creating and managing environments and modules. The configuration file, written in valid YAML syntax, includes all information about the environments and modules to create. Using a configuration file saves repeated entries of Platform CLI command options.

Note: If more than one control plane node is entered in the myenvironment.yaml used to configure the Oracle Cloud Native Environment, then olcnectl requires that details of the virtual IP address are entered into the myenvironment.yaml file. Achieve this via entering a new argument (virtual-ip: <enter-your-ip-here>) into the myenvironment.yaml, for example:

Example Output:

[oracle@ocne-operator ~]$ cat myenvironment.yaml 
environments:
 - environment-name: myenvironment
   globals:
     api-server: 127.0.0.1:8091
     secret-manager-type: file
     olcne-ca-path: /etc/olcne/configs/certificates/production/ca.cert
     olcne-node-cert-path: /etc/olcne/configs/certificates/production/node.cert
     olcne-node-key-path:  /etc/olcne/configs/certificates/production/node.key
   modules:
     - module: kubernetes
       name: mycluster
       args:
         container-registry: container-registry.oracle.com/olcne
         virtual-ip: 10.0.12.111
         master-nodes: ocne-control01:8090,ocne-control02:8090,ocne-control03:8090
         worker-nodes: ocne-worker01:8090,ocne-worker02:8090
         selinux: enforcing
         restrict-service-externalip: true
         restrict-service-externalip-ca-cert: /etc/olcne/configs/certificates/restrict_external_ip/production/ca.cert
         restrict-service-externalip-tls-cert: /etc/olcne/configs/certificates/restrict_external_ip/production/node.cert
         restrict-service-externalip-tls-key: /etc/olcne/configs/certificates/restrict_external_ip/production/node.key

During lab deployment, a configuration file is automatically generated and ready to use in the exercise. More information on manually creating a configuration file is in the documentation at Using a Configuration File.

Update the Configuration File

  1. (On ocne-operator) View the configuration file contents.

    cat ~/myenvironment.yaml
    
  2. (On ocne-operator) Add the virtual-ip value to the myenvironment.yaml file.

    sed -i '14i\          virtual-ip: 10.0.12.111\' ~/myenvironment.yaml
    
  3. (On ocne-operator) Confirm the virtual-ip value has been added to the myenviroment.yaml file.

    cat ~/myenvironment.yaml
    

Create the Environment and Kubernetes Module

  1. (On ocne-operator) Create the environment.

    cd ~
    olcnectl environment create --config-file myenvironment.yaml
    

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl environment create --config-file myenvironment.yaml
    Environment myenvironment created.
    
  2. (On ocne-operator) Create the Kubernetes module.

    olcnectl module create --config-file myenvironment.yaml
    

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module create --config-file myenvironment.yaml
    Modules created successfully.
    
  3. (On ocne-operator) Validate the Kubernetes module.

    olcnectl module validate --config-file myenvironment.yaml
    

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module validate --config-file myenvironment.yaml
    Validation of module mycluster succeeded.
    

    In this example, there are no validation errors. If there are any errors, the commands required to fix the nodes are provided as output of this command.

  4. (On ocne-operator) Install the Kubernetes module.

    olcnectl module install --config-file myenvironment.yaml
    

    The deployment of Kubernetes to the nodes may take several minutes to complete.

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module install --config-file myenvironment.yaml
    Modules installed successfully.
    
  5. (On ocne-operator) Validate the deployment of the Kubernetes module.

    olcnectl module instances --config-file myenvironment.yaml
    

    Example Output:

    [oracle@ocne-operator ~]$ olcnectl module instances --config-file myenvironment.yaml
    INSTANCE       	MODULE    	STATE    
    10.0.12.11:8090	node      	installed
    10.0.12.12:8090	node      	installed
    10.0.12.13:8090	node      	installed
    10.0.12.21:8090	node      	installed
    10.0.12.22:8090	node      	installed
    mycluster      	kubernetes	installed
    [oracle@ocne-operator ~]$
    

Set up kubectl

  1. (On all control plane nodes) Set up the kubectl command.

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
    export KUBECONFIG=$HOME/.kube/config
    echo 'export KUBECONFIG=$HOME/.kube/config' >> $HOME/.bashrc
    
  2. (On any control plane node) Verify kubectl works, and that the install completed successfully with all nodes listed as being in the ‘Ready status.

    kubectl get nodes
    

    Example Output:

    [oracle@ocne-control01 ~]$ kubectl get nodes
    NAME             STATUS   ROLES                  AGE     VERSION
    ocne-control01   Ready    control-plane,master   9m3s    v1.23.7+1.el8
    ocne-control02   Ready    control-plane,master   7m47s   v1.23.7+1.el8
    ocne-control03   Ready    control-plane,master   6m41s   v1.23.7+1.el8
    ocne-worker01    Ready    <none>                 8m32s   v1.23.7+1.el8
    ocne-worker02    Ready    <none>                 8m26s   v1.23.7+1.el8
    [oracle@ocne-control01 ~]$
    

Confirm Failover Between Control Plane Nodes

The Oracle Cloud Native Environment installation with three control plane nodes behind an internal load balancer is complete.

The following steps confirm that the internal load balancer (using keepalived) will detect when the primary control plane node fails and passes control to one of the surviving control plane nodes. Likewise, when the ‘missing’ node recovers, it can automatically rejoin the cluster.

Locate the Primary Control Plane Node

Determine which control plane node currently holds the virtual IP address.

  1. (On all control plane nodes) List the present network devices and IP addresses.

    ip -br a
    

    The -br option passed to the ip command provides a brief summary of the IP address information assigned to a device.

  2. Look at each of the results, and find which host’s output contains the virtual IP associated with the ens5 NIC.

    In the free lab environment, the virtual IP address used by the keepalived daemon is set to 10.0.12.111.

    ens5             UP             10.0.12.11/24 10.0.12.111/32 fe80::41f6:2a0d:9a89:13d0/64 
    

    Example Output:

    [oracle@ocne-control01 ~]$ ip -br a
    lo               UNKNOWN        127.0.0.1/8 ::1/128 
    ens3             UP             10.0.0.150/24 fe80::200:17ff:fe00:215e/64 
    ens5             UP             10.0.12.11/24 10.0.12.111/32 fe80::41f6:2a0d:9a89:13d0/64 
    flannel.1        UNKNOWN        10.244.0.0/32 fe80::78ad:2dff:fef1:259b/64 
    cni0             UP             10.244.0.1/24 fe80::104e:f2ff:fece:343b/64 
    veth9f7ab71b@if2 UP             fe80::545c:e3ff:fee9:2809/64 
    [oracle@ocne-control01 ~]$ 
    

    This control plane node has the virtual IP address assigned as a secondary IP on the ens5 device.

    Example Output:

    [oracle@ocne-control02 ~]$ ip -br a
    lo               UNKNOWN        127.0.0.1/8 ::1/128 
    ens3             UP             10.0.0.151/24 fe80::17ff:fe00:e0b9/64 
    ens5             UP             10.0.12.12/24 fe80::feab:426b:206e:8870/64 
    flannel.1        UNKNOWN        10.244.1.0/32 fe80::68ba:77ff:fedc:f56f/64 
    cni0             UP             10.244.1.1/24 fe80::a449:7fff:feb2:9738/64 
    veth3729db25@if2 UP             fe80::b07d:f8ff:feb4:4830/64 
    vethd7048746@if2 UP             fe80::ec1b:49ff:fe90:838a/64 
    

    This control plane node does not.

    Important: Take note which control plane node currently holds the virtual IP address.

  3. (Optional) (On all control plane nodes) Confirm which control plane node holds the virtual IP address by querying the keepalived.service logs.

    journalctl -u keeaplived
    

    Example Output:

    ...
    Aug 10 23:47:26 ocne-control01 Keepalived_vrrp[55605]: (VI_1) Entering MASTER STATE
    ...
    

    This control plane node has the virtual IP address assigned.

    Example Output:

    ...
    Aug 10 23:54:59 ocne-control02 Keepalived_vrrp[59961]: (VI_1) Entering BACKUP STATE (init)
    ...
    

    This control plane node does not.

Force the keepalived Daemon to Move to a Different Control Plane Node

If using the free lab environment, double-click the Luna Lab icon on the desktop and navigate to the Luna Lab tab. Then click on the OCI Console hyperlink. Sign-on using the User Name and Password values provided. After logging on, proceed with the following steps:

  1. Start from this screen.

    public_ip_address

  2. Click on the hamburger menu (top-left), then Compute and Instances.

    public_ip_address

  3. This displays the Instances page.

    Locate the Compartment used from the drop-down list as instructed in Oracle Linux Lab Basics.

    public_ip_address

  4. Click on the Instance name previously identified (for example, ocne-control01).

    public_ip_address

  5. This shows the details for the Instance.

    public_ip_address

  6. Click on the Stop button.

    In the pop-up dialog, select the ‘Force stop the instance by immediately powering off’ check-box, and click the ‘Force stop instance` button.

    Note: Do NOT do this on a production system as it may incur data loss, corruption or worse to the entire system.

    Important Information related to Terminal session: Note that it will no longer be possible to use the Terminal session associated with the specific control plane node once it has been shut down.

    public_ip_address

  7. Wait until the Instance details page confirms the instance is ‘Stopped’.

    public_ip_address

  8. Switch from the browser, back to the Terminal.

  9. (On one of the surviving control plane nodes) Execute the following command to confirm that the control plane node just shut down is reporting as NotReady.

    Note: It may be necessary to repeat this step several times until the status changes

    kubectl get nodes
    

    Example Output:

    [oracle@ocne-control03 ~]$ kubectl get nodes
    ocne-control01   NotReady   control-plane,master   3h26m   v1.23.7+1.el8
    ocne-control02   Ready      control-plane,master   3h12m   v1.23.7+1.el8
    ocne-control03   Ready      control-plane,master   3h9m    v1.23.7+1.el8
    ocne-worker01    Ready      <none>                 3h9m    v1.23.7+1.el8
    ocne-worker02    Ready      <none>                 3h9m    v1.23.7+1.el8
    [oracle@ocne-control03 ~]$
    
  10. (On each of the surviving control plane nodes) Determine which node has the running keepalived daemon.

    ip -br a
    

    Example Output:

    [oracle@ocne-control03 ~]$ ip -br a
    lo               UNKNOWN        127.0.0.1/8 ::1/128 
    ens3             UP             10.0.0.152/24 fe80::200:17ff:fe00:6af4/64 
    ens5             UP             10.0.12.13/24 10.0.12.111/32 fe80::7c90:5770:5da:a00/64 
    flannel.1        UNKNOWN        10.244.2.0/32 fe80::70a8:6ff:fefd:16f7/64 
    [oracle@ocne-control03 ~]$
    

    The virtual IP is now associated with ocne-control03.

  11. Switch back to the browser.

  12. In the Cloud Console, start the Instance previously shut down by clicking on the Start button.

    Wait until the Status section confirms the Instance is ‘Running’.

    public_ip_address

  13. Switch back to the Terminal.

  14. Reconnect to the control plane node that was previously shut down.

  15. (On any of the control plane nodes) Confirm that kubectl shows the restarted control plane node as being Ready.

    kubectl get nodes
    

    Note: It may be necessary to repeat this step several times until the status changes.

    Example Output:

    [oracle@ocne-control03 ~]$ kubectl get nodes
    NAME             STATUS   ROLES                  AGE     VERSION
    ocne-control01   Ready    control-plane,master   3h28m   v1.23.7+1.el8
    ocne-control02   Ready    control-plane,master   3h26m   v1.23.7+1.el8
    ocne-control03   Ready    control-plane,master   3h24m   v1.23.7+1.el8
    ocne-worker01    Ready    <none>                 3h24m   v1.23.7+1.el8
    ocne-worker02    Ready    <none>                 3h23m   v1.23.7+1.el8
    [oracle@ocne-control03
    
  16. (On all control plane nodes) Confirm the location of the active keepalived daemon.

    Note: The keepalived daemon remained with the currently active host, despite the original host restarting. This occurs due to how Oracle Cloud Native Environment weights the nodes in the keepalived configuration.

    ip -br  a
    

    Example Output:

    [oracle@ocne-control03 ~]$ ip -br a
    lo               UNKNOWN        127.0.0.1/8 ::1/128 
    ens3             UP             10.0.0.152/24 fe80::200:17ff:fe00:6af4/64 
    ens5             UP             10.0.12.13/24 10.0.12.111/32 fe80::7c90:5770:5da:a00/64 
    flannel.1        UNKNOWN        10.244.2.0/32 fe80::70a8:6ff:fefd:16f7/64 
    

    Example Output:

    [oracle@ocne-control01 ~]$ ip -br a
    lo               UNKNOWN        127.0.0.1/8 ::1/128 
    ens3             UP             10.0.0.150/24 fe80::200:17ff:fe00:215e/64 
    ens5             UP             10.0.12.11/24 fe80::41f6:2a0d:9a89:13d0/64 
    flannel.1        UNKNOWN        10.244.0.0/32 fe80::e4a0:25ff:fea3:b636/64 
    

For More Information

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.