Note:

Enforce internal Oracle Container Engine for Kubernetes Load Balancer services using Kyverno

Introduction

Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) is a fully managed, scalable, and highly available service that you can use to deploy your containerized applications to the cloud.

Applications running in the OKE cluster can be exposed using services of type LoadBalancer. For each service, the built-in OKE cloud-controller-manager will create an Oracle Cloud Infrastructure (OCI) Load Balancer and automatically configure the backend to include the OKE worker nodes. The OCI Load Balancer configuration can be controlled using annotations, and in case of their absence, the cloud-controller-manager will assume the following defaults:

Using Kyverno, we can take advantage of the Kubernetes Mutating Admission Webhooks and define a policy with a set of default annotations we intend to add to Services of type LoadBalancer created within the OKE cluster.

Objectives

Prerequisites

Task 1: Install Kyverno

The Kyverno webpage describes two methods to install Kyverno on a Kubernetes cluster, using:

In this tutorial, we will focus on how to install Kyverno using the Helm charts.

Task 1.1: Install helm

  1. Depending on the operating system you are on, go through this guide and install helm.

  2. Confirm if the installation is successful by executing the following command:

    $ helm version
    version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.5"}
    

Task 1.2: Install Kyverno in standalone mode

  1. Add the Kyverno Helm repository.

    $ helm repo add kyverno https://kyverno.github.io/kyverno/
    "kyverno" has been added to your repositories
    
  2. Scan the new repository for charts.

    $ helm repo update
    Hang tight while we grab the latest from your chart repositories...
    ...Successfully got an update from the "kyverno" chart repository
    Update Complete. ⎈Happy Helming!⎈
    
  3. Install Kyverno in kyverno namespace.

    $ helm install kyverno kyverno/kyverno -n kyverno --create-namespace
    NAME: kyverno
    LAST DEPLOYED: Fri Jun 16 17:50:48 2023
    NAMESPACE: kyverno
    STATUS: deployed
    REVISION: 1
    NOTES:
    Chart version: 3.0.1
    Kyverno version: v1.10.0
    
    Thank you for installing kyverno! Your release is named kyverno.
    
    The following components have been installed in your cluster:
    - CRDs
    - Admission controller
    - Reports controller
    - Cleanup controller
    - Background controller
    

Task 2: Define Service defaults using ClusterPolicy

  1. Create a file named enforce-internal-loadbalancer.yaml.

  2. Copy and paste the text below in the file.

    apiVersion: kyverno.io/v1
    # The `ClusterPolicy` kind applies to the resources deployed in any namespace.
    kind: ClusterPolicy
    metadata:
    name: mutate-oci-services
    spec:
    rules:
    # As part of this rule we intend to mutate Flexible Load Balancers.
    - name: mutate-lb-services
       # We look for all requests to Create or Update Kubernetes Services.
       match:
          resources:
          kinds:
          - Service
          operations:
          - CREATE
          - UPDATE
       # We exclude services with the label: "service-type: external".
       exclude:
          resources:
          selector:
             matchLabels:
                service-type: "external"
       # Out of all the services we are interested in those of type Load Balancer where the annotation oci.oraclecloud.com/load-balancer-type is not present or equal to "nlb".
       preconditions:
          all:
          - key: ""
          operator: Equals
          value: LoadBalancer
          - key: ""
          operator: NotEquals
          value: "nlb"
       # We mutate the request by appending the annotations required to create an internal Load Balancer.
       mutate:
          patchStrategicMerge:
          metadata:
             annotations:
                service.beta.kubernetes.io/oci-load-balancer-internal: "true"
                # service.beta.kubernetes.io/oci-load-balancer-subnet1: "ocid1.subnet.oc1...3gi7y5a"
    # As part of this rule we intend to mutate Network Load Balancers.
    - name: mutate-nlb-services
       # We look for all requests to Create or Update Kubernetes Services with the annotation oci.oraclecloud.com/load-balancer-type: "nlb".
       match:
          resources:
          kinds:
          - Service
          annotations:
             oci.oraclecloud.com/load-balancer-type: "nlb"
          operations:
          - CREATE
          - UPDATE
       # We exclude services with the label: "service-type: external".
       exclude:
          resources:
          selector:
             matchLabels:
                service-type: "external"
       # Out of all the services we are interested in those of type Load Balancer.
       preconditions:
          all:
          - key: ""
          operator: Equals
          value: LoadBalancer
       # We mutate the request by appending the annotations required to create an internal NetworkLoad Balancer.
       mutate:
          patchStrategicMerge:
          metadata:
             annotations:
                oci-network-load-balancer.oraclecloud.com/internal: "true"
                # oci-network-load-balancer.oraclecloud.com/subnet: "ocid1.subnet.oc1...3gi7y5a"
    
  3. Save the file.

    Note: For a list of all the supported annotations in OKE for the LoadBalancer type of services, see Summary of Annotations for Load Balancers and Network Load Balancers.

  4. Create the ClusterPolicy in the cluster and execute below command to enforce the policy:

    $ kubectl apply -f enforce-internal-loadbalancer.yaml
    clusterpolicy.kyverno.io/mutate-oci-services created
    

Task 3: Test

  1. Create a file named test-service.yaml containing the text below:

    apiVersion: v1
    kind: Service
    metadata:
    name: my-nginx-svc
    labels:
       app: nginx
       # service-type: external
    annotations:
       oci.oraclecloud.com/load-balancer-type: "lb"
    spec:
    type: LoadBalancer
    ports:
    - port: 80
    selector:
       app: nginx
    
  2. Validate if the required annotations to create an internal LoadBalancer are added using the command below:

    $ kubectl apply -f test_service.yaml --dry-run=server -o json | jq ".metadata.annotations"
    {
    "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{\"oci.oraclecloud.com/load-balancer-type\":\"lb\"},\"labels\":{\"app\":\"nginx\"},\"name\":\"my-nginx-svc\",\"namespace\":\"default\"},\"spec\":{\"ports\":[{\"port\":80}],\"selector\":{\"app\":\"nginx\"},\"type\":\"LoadBalancer\"}}\n",
    "oci.oraclecloud.com/load-balancer-type": "lb",
    "policies.kyverno.io/last-applied-patches": "mutate-lb-services.mutate-oci-services.kyverno.io: added /metadata/annotations/service.beta.kubernetes.io~1oci-load-balancer-internal\n",
    "service.beta.kubernetes.io/oci-load-balancer-internal": "true"
    }
    
  3. You can validate if the internal load balancer is successfully created by executing the command:

    $ kubectl apply -f test_service.yaml
    

Task 4: Clean up

  1. Delete the load balancer created during the test:

    $ kubectl delete -f test_service.yaml
    
  2. Uninstall Kyverno by executing the command:

    $ helm uninstall kyverno -n kyverno
    

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.