Note:

• This tutorial requires access to Oracle Cloud. To sign up for a free account, see Get started with Oracle Cloud Infrastructure Free Tier.
• It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.

Set up Oracle Cloud Infrastructure Container Engine for Kubernetes with Three Worker Nodes

Introduction

In this tutorial, we will explain how to set up a Kubernetes cluster consisting of the Kubernetes control plane and the data plane (node pool) using the Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE). We will also deploy and delete two sample applications on the Kubernetes platform to prove it works. This tutorial will set the stage for future tutorials that will dive into networking services offered inside Kubernetes for container-hosted applications.

The example OKE deployment models are:

• Example 1: Cluster with Flannel CNI Plugin, Public Kubernetes API Endpoint, Private Worker Nodes, and Public Load Balancers.

• Example 2: Cluster with Flannel CNI Plugin, Private Kubernetes API Endpoint, Private Worker Nodes, and Public Load Balancers.

• Example 3: Cluster with OCI CNI Plugin, Public Kubernetes API Endpoint, Private Worker Nodes, and Public Load Balancers.

• Example 4: Cluster with OCI CNI Plugin, Private Kubernetes API Endpoint, Private Worker Nodes, and Public Load Balancers.

For more information about the different OKE deployment models we can choose, see Example Network Resource Configurations.

In this tutorial, we will implement the Example 3 deployment model.

Objectives

• We will deploy a Kubernetes control cluster and worker nodes which are fully deployed and configured inside Oracle Cloud Infrastructure (OCI). This is what we call the Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE). We will deploy two sample applications in two different namespaces where one application will deploy using a Helm chart in a new namespace. In the end, we will clean up the applications or pods. We will not deploy any network services for Kubernetes-operated applications or pods.

Task 1: Create a New Kubernetes Cluster and Verify the Components

• Click the hamburger menu.

  

  1. Click Developer Services.
  2. Click Kubernetes Clusters (OKE).

  

• Click Create Cluster.

  

  1. Select Quick create.
  2. Click Submit.

  

• In the Create cluster (quick) page, enter the following information.

  1. Enter a cluster Name.
  2. Select a Compartment.
  3. Select the Kubernetes version.
  4. Select the Kubernetes API endpoint to be a Public endpoint.
  5. Select the Node Type to be Managed.
  6. Scroll down.

  

  1. Select the Kubernetes Worker nodes to be Private workers.
  2. Scroll down.

  

  1. Keep the Node count (Worker nodes) default 3.
  2. Click Next.

  

  1. Review the cluster parameters.
  2. Scroll down.

  

  1. Review the node pools parameters.
  2. Scroll down.

  

  1. Do not select the Create a Basic cluster checkbox.
  2. Click Create cluster.

  

• Review the status of the different components that are created.

  

  1. Make sure everything has a green check.
  2. Click Close.

  

  1. Review that the status is CREATING.
  2. Scroll down.

  

• Review the cluster and node pool creation status. The Kubernetes control cluster is being created and the worker node pool will be created later.

  

• After a few minutes the Kubernetes control cluster is successfully created.

  

• The worker node pool will now be created.

  

• After a few minutes the worker node pool is successfully created.

  

  1. Click Node Pools.
  2. Notice that the worker nodes in the pool are still being created.
  3. Click 3 of worker nodes.

  

• Notice that the all nodes has Not ready status.

  

• After a few minutes they will be Ready.

  

The Kubernetes control cluster and worker nodes are fully deployed and configured inside Oracle Cloud Infrastructure (OCI). This is what we call the Oracle Cloud Infrastructure Container Engine for Kubernetes.

Task 2: Verify the Deployed Kubernetes Cluster Components in the OCI Console

When we use OKE to create a Kubernetes cluster some resources will be created inside OCI to support this deployment.

The first and most important resource is the virtual cloud network (VCN). Because we have chosen the Quick Create option, a new VCN dedicated to OKE was created.

• Log in to the OCI Console and navigate to Networking, Virtual Cloud Networks (VCN), you will see the new VCN that was created. Click the VCN.

  

  Inside the VCN, you will see three subnets, one private and two public subnets to support the OKE deployment.

  

• Verify the resources.

  1. Click CIDR Blocks/Prefixes to review the CIDR of the VCN.
  2. Notice that 10.0.0.0/16 was assigned by OCI.

  

  1. Click Route Tables to review the routing tables.
  2. Notice that there are two routing tables created: route to private subnets, and route to public subnets.

  

  1. Click Internet Gateways to review the internet gateway that will provide internet connectivity using the public subnets to and from the internet.
  2. Notice that there is only one internet gateway.

  

  1. Click Security Lists to review the security lists that can be either Ingress or Egress rules to protect connectivity between the subnets.
  2. Note that there are three security lists, one used for Kubernetes worker node connectivity protection, second used for Kubernetes API endpoint protection, and third for Kubernetes services protection.

  

  1. Click NAT Gateways to review the NAT Gateway that will provide internet connectivity using the private subnets to the internet.
  2. Notice that there is only one NAT Gateway.

  

  1. Click Service Gateways to review the service gateway that will provide private access to specific Oracle services, without exposing the data to an internet gateway or NAT gateway.
  2. Notice that there is only one service gateway.

  

  1. Open the OCI Console, click the hamburger menu and navigate to Compute, Instances.
  2. Notice that three instances that are created that will be used as the three Kubernetes worker nodes we specified during the deployment.

  

  1. Open the OCI Console, click the hamburger menu and navigate to IP Management, Reserved public IPs.
  2. Notice that there is one public IP address ending with .166, reserved for the Kubernetes public API endpoint.

  

If we place every piece of information we have just collected and place that in a diagram then diagram will look like as shown in the following image.

• Tables with Configuration Details to Deploy OKE

  • VCN:

    Resource	Name
    VCN	• Name: oke-vcn-quick-IH-OKE-CLUSTER-af593850a • CIDR Block: 10.0.0.0/16 • DNS Resolution: Selected
    Internet Gateway	• Name: oke-igw-quick-IH-OKE-CLUSTER-af593850a
    NAT Gateway	• Name: oke-ngw-quick-IH-OKE-CLUSTER-af593850a
    Service Gateway	• Name: oke-sgw-quick-IH-OKE-CLUSTER-af593850a • Services: All region Services in Oracle Services Network
    DHCP Options	• DNS Type set to Internet and VCN Resolver

  • Subnets:

    Resource	Example
    Public Subnet for Kubernetes API Endpoint	Purpose: Kubernetes API endpoint with the following properties: • Type: Regional • CIDR Block: 10.0.0.0/28 • Route Table: oke-public-routetable-IH-OKE-CLUSTER-af593850a • Subnet Access: Public • DNS Resolution: Selected • DHCP Options: Default • Security List: oke-k8sApiEndpoint-quick-IH-OKE-CLUSTER-af593850a
    Private Subnet for Worker Nodes	Purpose: workernodes with the following properties: • Type: Regional • CIDR Block: 10.0.10.0/24 • Route Table: N/A • Subnet Access: Private • DNS Resolution: Selected • DHCP Options: Default • Security List: oke-nodeseclist-quick-IH-OKE-CLUSTER-af593850a
    Private Subnet for Pods	Purpose: pods with the following properties: • Type: Regional • CIDR Block: 10.96.0.0/16 • Route Table: oke-private-routetable-IH-OKE-CLUSTER-af593850a • Subnet Access: Private • DNS Resolution: Selected • DHCP Options: Default • Security List: N/A
    Public Subnet for Service Load Balancers	Purpose: Load balancers with the following properties: • Type: Regional • CIDR Block: 10.0.20.0/24 • Route Table: oke-private-routetable-IH-OKE-CLUSTER-af593850a • Subnet Access: Public • DNS Resolution: Selected • DHCP Options: Default • Security List: oke-svclbseclist-quick-IH-OKE-CLUSTER-af593850a

  • Route Tables:

    Resource	Example
    Route Table for Public Kubernetes API Endpoint Subnet	Purpose: routetable-Kubernetes API endpoint, with one route rule defined as follows: • Destination CIDR block: 0.0.0.0/0 • Target Type: Internet Gateway • Target: oke-igw-quick-IH-OKE-CLUSTER-af593850a
    Route Table for Private Pods Subnet	Purpose: routetable-pods, with two route rules defined as follows: • Rule for traffic to the internet: ◦ Destination CIDR block: 0.0.0.0/0 ◦ Target Type: NAT Gateway ◦ Target: oke-ngw-quick-IH-OKE-CLUSTER-af593850a • Rule for traffic to OCI services: ◦ Destination: All region Services in Oracle Services Network ◦ Target Type: Service Gateway ◦ Target: oke-sgw-quick-IH-OKE-CLUSTER-af593850a
    Route Table for Public Load Balancers Subnet	Purpose: routetable-serviceloadbalancers, with one route rule defined as follows: • Destination CIDR block: 0.0.0.0/0 • Target Type: Internet Gateway • Target: oke-igw-quick-IH-OKE-CLUSTER-af593850a

• Security List Rules for Public Kubernetes API Endpoint Subnet

  The oke-k8sApiEndpoint-quick-IH-OKE-CLUSTER-af593850a security list has the ingress and egress rules as shown in the following tables.

  • Ingress Rules:

    Stateless	Source	IP Protocol	Source Port Range	Destination Port Range	Type and Code	Allows	Description
    No	0.0.0.0/0	TCP	All	6443		TCP traffic for ports: 6443	External access to Kubernetes API endpoint
    No	10.0.10.0/24	TCP	All	6443		TCP traffic for ports: 6443	Kubernetes worker to Kubernetes API endpoint communication
    No	10.0.10.0/24	TCP	All	12250		TCP traffic for ports: 12250	Kubernetes worker to control plane communication
    No	10.0.10.0/24	ICMP			3, 4	ICMP traffic for: 3, 4 Destination Unreachable: Fragmentation Needed and Don’t Fragment was Set	Path discovery

  • Egress Rules:

    Stateless	Destination	IP Protocol	Source Port Range	Destination Port Range	Type and Code	Allows	Description
    No	All AMS Services In Oracle Services Network	TCP	All	443		TCP traffic for ports: 443 HTTPS	Allow Kubernetes Control Plane to communicate with OKE
    No	10.0.10.0/24	TCP	All	All		TCP traffic for ports: All	All traffic to worker nodes
    No	10.0.10.0/24	ICMP			3, 4	ICMP traffic for: 3, 4 Destination Unreachable: Fragmentation Needed and Don’t Fragment was Set	Path discovery

• Security List Rules for Private Worker Nodes Subnet

  The oke-nodeseclist-quick-IH-OKE-CLUSTER-af593850a security list has the ingress and egress rules as shown in the following tables.

  Ingress Rules:

  Stateless	Source	IP Protocol	Source Port Range	Destination Port Range	Type and Code	Allows	Description
  No	10.0.10.0/24	All Protocols				All traffic for all ports	Allow pods on one worker node to communicate with pods on other worker nodes
  No	10.0.0.0/28	ICMP			3, 4	ICMP traffic for: 3, 4 Destination Unreachable: Fragmentation Needed and Don’t Fragment was Set	Path discovery
  No	10.0.0.0/28	TCP	All	All		TCP traffic for ports: All	TCP access from Kubernetes Control Plane
  No	0.0.0.0/0	TCP	All	22		TCP traffic for ports: 22 SSH Remote Login Protocol	Inbound SSH traffic to worker nodes
  No	10.0.20.0/24	TCP	All	32291		TCP traffic for ports: 32291
  No	10.0.20.0/24	TCP	All	10256		TCP traffic for ports: 10256
  No	10.0.20.0/24	TCP	All	31265		TCP traffic for ports: 31265

  Egress Rules:

  Stateless	Destination	IP Protocol	Source Port Range	Destination Port Range	Type and Code	Allows	Description
  No	10.0.10.0/24	All Protocols				All traffic for all ports	Allow pods on one worker node to communicate with pods on other worker nodes
  No	10.0.0.0/28	TCP	All	6443		TCP traffic for ports: 6443	Access to Kubernetes API Endpoint
  No	10.0.0.0/28	TCP	All	12250		TCP traffic for ports: 12250	Kubernetes worker to control plane communication
  No	10.0.0.0/28	ICMP			3, 4	ICMP traffic for: 3, 4 Destination Unreachable: Fragmentation Needed and Don’t Fragment was Set	Path discovery
  No	All AMS Services In Oracle Services Network	TCP	All	443		TCP traffic for ports: 443 HTTPS	Allow nodes to communicate with OKE to ensure correct start-up and continued functioning
  No	0.0.0.0/0	ICMP			3, 4	ICMP traffic for: 3, 4 Destination Unreachable: Fragmentation Needed and Don’t Fragment was Set	ICMP Access from Kubernetes Control Plane
  No	0.0.0.0/0	All Protocols				All traffic for all ports	Worker Nodes access to Internet

• Security List Rules for Public Load Balancer Subnet

  The oke-svclbseclist-quick-IH-OKE-CLUSTER-af593850a security list has the ingress and egress rules as shown in the following tables.

  • Ingress Rules:

    Stateless	Source	IP Protocol	Source Port Range	Destination Port Range	Type and Code	Allows	Description
    No	0.0.0.0/0	TCP	All	80		TCP traffic for ports: 80

  • Egress Rules:

    Stateless	Destination	IP Protocol	Source Port Range	Destination Port Range	Type and Code	Allows	Description
    No	10.0.10.0/24	TCP	All	32291		TCP traffic for ports: 32291
    No	10.0.10.0/24	TCP	All	10256		TCP traffic for ports: 10256
    No	10.0.10.0/24	TCP	All	31265		TCP traffic for ports: 31265

Task 3: Verify the Kubernetes Cluster is Running using the CLI

• Open the OCI Console, click the hamburger menu and navigate to Developer Services, Kubernetes Clusters (OKE). Click the Kubernetes cluster created in Task 1.

  

  1. Scroll down.
  2. Click Quick Start.

  

• Click Access Cluster.

  

  1. Select Cloud Shell Access.
  2. Click Copy to copy the command to allow access to the Kubernetes cluster.
  3. Click Launch Cloud Shell.

  

  The following diagram shows how the connection will be made to perform management on the OKE cluster using OCI Cloud Shell.

  

• The OCI Cloud Shell will start.

  

  Some informational messages will be shown about what is happening in the background.

  

  In this case, it is possible to let OCI Cloud Shell run on different CPU architectures.

• Click Close to close this informational message.

  

• We are ready to use the OCI Cloud Shell to access to the Kubernetes cluster.

  

• Paste the command that was copied above in this task.

  

• Run the following command to get information about the Kubernetes cluster.

  kubectl cluster-info
  

  

• Run the following command to get information about the worker nodes.

  kubectl get nodes
  

  

• Run the following command to get more information about the worker nodes.

  kubectl get nodes -o wide
  

  

• Run the following command to clear the screen and start with a fresh new screen.

  clear
  

  

  1. Notice that the previous output has been cleared but, still accessible when you scroll up.
  2. Click minimize icon to minimize the OCI Cloud Shell window.

  

• Click Close to close the Access Your Cluster window.

  

The connection is made to perform management on the OKE cluster using OCI Cloud Shell.

Task 4: Deploy a Sample Nginx Application using kubectl

• Run the following commands.

  1. Make a note of the command to get the Kubernetes version.
  2. Make a note of the command to deploy a sample application.
  3. Click Restore to restore the OCI Cloud Shell window.

  

• Run the following command to get the Kubernetes version.

  kubectl version
  

  

• Run the following command to verify the current pods or applications that are deployed.

  kubectl get pods
  

  Notice that there are no resources found.

  

• Run the following command to deploy a new sample application.

  kubectl create -f https://k8s.io/examples/application/deployment.yaml
  

  

• Run the following command to verify the current pods or applications that are deployed.

  kubectl get pods
  

• Notice that there are pods in the RUNNING state. This means that the application we just deployed is running.

  

• Run the following command to get the IP addresses to access the application.

  kubectl get deploy,svc
  

• Notice that the newly deployed application does not have any IP addresses assigned and that only the Kubernetes cluster has a cluster IP service attached to it with an internal IP address.

  

• Run the following command to look at the attached (network) services for the newly deployed application specifically.

  kubectl get svc ngnix
  

• Notice that there are no (network) services deployed or attached for the deployed Nginx application. For this reason, we will not be able to access the application from another application or use the web browser to access the webpage in the Nginx web server. We will discuss this in another tutorial.

  

Task 5: Deploy a Sample MySQL Application using Helm Chart

• A Helm chart is a package that contains all the necessary resources to deploy an application to a Kubernetes cluster. Run the following commands to:

  1. Add the Bitnami repository for the MySQL database.

     helm repo add bitnami https://charts.bitnami.com/bitnami
     

  2. Deploy a MySQL database on the Kubernetes worker nodes and also create a new namespace mysql.

     helm install mysql bitnami/mysql -–namespace mysql --create-namespace
     

  

• To get the deployed applications, run the following command. This command will only display the deployed applications in the current (default) namespace.

  kubectl get pods
  

• Notice that only the Nginx application is showing up in the current (default) namespace. This command will now display the deployed applications cluster-wide (all namespaces).

  kubectl get pods -A -w
  

  1. Notice that the Nginx application is running in the default namespace.
  2. Notice that the MySQL application is running in the new mysql namespace.

  

Task 6: Clean up the Pods and Namespaces

We have deployed an application in the default namespace (Nginx) and another application in a new namespace (MySQL). Using Helm charts, let us clean up the environment so we can start fresh whenever we need to.

1. Use the following command to get all the worker nodes (cluster-wide).

   kubectl get nodes -o wide
   

2. Use the following command to get all the running pods in the current (default) namespace.

   kubectl get pods -o wide
   

3. Run the following command to get all the namespaces.

   kubectl get namespaces
   

4. Run the following command to get all the running pods in the current (default) namespace specifically.

   kubectl get pods --namespace=default
   

5. Run the following command to get all the running pods in the MySQL namespace specifically.

   kubectl get pods --namespace=mysql
   

1. Run the following command to delete all deployments or pods in the default namespace.

   kubectl delete --all deployments --namespace=default
   

2. Use the following command to verify if the deployments or pods are deleted.

   kubectl get pods --namespace=default
   

3. Use the following command to get all the running pods in the MySQL namespace specifically. Just verify if this still exists.

   kubectl get pods --namespace=mysql
   

1. Run the following command to delete all deployments or pods and the complete MySQL namespace.

   kubectl delete namespace mysql
   

2. Use this command to gather all the namespaces, and verify if the MySQL namespace is deleted.

   kubectl get namespaces    
   

Acknowledgments

• Author - Iwan Hoogendoorn (OCI Network Specialist)

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.

---

Title and Copyright Information

Set up Oracle Cloud Infrastructure Container Engine for Kubernetes with Three Worker Nodes

F95645-01

March 2024

Copyright ©2024,

Oracle and/or its affiliates.