Note:
- This tutorial requires access to Oracle Cloud. To sign up for a free account, see Get started with Oracle Cloud Infrastructure Free Tier.
- It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.
Enforce internal Oracle Container Engine for Kubernetes Load Balancer services using Kyverno
Introduction
Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) is a fully managed, scalable, and highly available service that you can use to deploy your containerized applications to the cloud.
Applications running in the OKE cluster can be exposed using services of type LoadBalancer
. For each service, the built-in OKE cloud-controller-manager will create an Oracle Cloud Infrastructure (OCI) Load Balancer and automatically configure the backend to include the OKE worker nodes. The OCI Load Balancer configuration can be controlled using annotations, and in case of their absence, the cloud-controller-manager will assume the following defaults:
- internal load-balancer:
false
- shape:
fixed
- bandwidth:
100Mbps
- listener protocol:
TCP
Using Kyverno, we can take advantage of the Kubernetes Mutating Admission Webhooks and define a policy with a set of default annotations we intend to add to Services of type LoadBalancer created within the OKE cluster.
Objectives
-
Deploy Kyverno to the OKE cluster.
-
Learn how to use Kyverno to enforce creation of internal load balancers by default.
Prerequisites
-
Sign up or Sign in to your Oracle Cloud account.
-
Use an existing OKE cluster or use the Quick Create flow to create one.
-
kubectl
installed and configured to access the OKE cluster.
Task 1: Install Kyverno
The Kyverno webpage describes two methods to install Kyverno on a Kubernetes cluster, using:
- Helm charts
- YAMLs
In this tutorial, we will focus on how to install Kyverno using the Helm charts.
Task 1.1: Install helm
-
Depending on the operating system you are on, go through this guide and install
helm
. -
Confirm if the installation is successful by executing the following command:
$ helm version version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.5"}
Task 1.2: Install Kyverno in standalone mode
-
Add the Kyverno Helm repository.
$ helm repo add kyverno https://kyverno.github.io/kyverno/ "kyverno" has been added to your repositories
-
Scan the new repository for charts.
$ helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "kyverno" chart repository Update Complete. ⎈Happy Helming!⎈
-
Install Kyverno in
kyverno
namespace.$ helm install kyverno kyverno/kyverno -n kyverno --create-namespace NAME: kyverno LAST DEPLOYED: Fri Jun 16 17:50:48 2023 NAMESPACE: kyverno STATUS: deployed REVISION: 1 NOTES: Chart version: 3.0.1 Kyverno version: v1.10.0 Thank you for installing kyverno! Your release is named kyverno. The following components have been installed in your cluster: - CRDs - Admission controller - Reports controller - Cleanup controller - Background controller
Task 2: Define Service defaults using ClusterPolicy
-
Create a file named
enforce-internal-loadbalancer.yaml
. -
Copy and paste the text below in the file.
apiVersion: kyverno.io/v1 # The `ClusterPolicy` kind applies to the resources deployed in any namespace. kind: ClusterPolicy metadata: name: mutate-oci-services spec: rules: # As part of this rule we intend to mutate Flexible Load Balancers. - name: mutate-lb-services # We look for all requests to Create or Update Kubernetes Services. match: resources: kinds: - Service operations: - CREATE - UPDATE # We exclude services with the label: "service-type: external". exclude: resources: selector: matchLabels: service-type: "external" # Out of all the services we are interested in those of type Load Balancer where the annotation oci.oraclecloud.com/load-balancer-type is not present or equal to "nlb". preconditions: all: - key: "" operator: Equals value: LoadBalancer - key: "" operator: NotEquals value: "nlb" # We mutate the request by appending the annotations required to create an internal Load Balancer. mutate: patchStrategicMerge: metadata: annotations: service.beta.kubernetes.io/oci-load-balancer-internal: "true" # service.beta.kubernetes.io/oci-load-balancer-subnet1: "ocid1.subnet.oc1...3gi7y5a" # As part of this rule we intend to mutate Network Load Balancers. - name: mutate-nlb-services # We look for all requests to Create or Update Kubernetes Services with the annotation oci.oraclecloud.com/load-balancer-type: "nlb". match: resources: kinds: - Service annotations: oci.oraclecloud.com/load-balancer-type: "nlb" operations: - CREATE - UPDATE # We exclude services with the label: "service-type: external". exclude: resources: selector: matchLabels: service-type: "external" # Out of all the services we are interested in those of type Load Balancer. preconditions: all: - key: "" operator: Equals value: LoadBalancer # We mutate the request by appending the annotations required to create an internal NetworkLoad Balancer. mutate: patchStrategicMerge: metadata: annotations: oci-network-load-balancer.oraclecloud.com/internal: "true" # oci-network-load-balancer.oraclecloud.com/subnet: "ocid1.subnet.oc1...3gi7y5a"
-
Save the file.
Note: For a list of all the supported annotations in OKE for the LoadBalancer type of services, see Summary of Annotations for Load Balancers and Network Load Balancers.
-
Create the
ClusterPolicy
in the cluster and execute below command to enforce the policy:$ kubectl apply -f enforce-internal-loadbalancer.yaml clusterpolicy.kyverno.io/mutate-oci-services created
Task 3: Test
-
Create a file named
test-service.yaml
containing the text below:apiVersion: v1 kind: Service metadata: name: my-nginx-svc labels: app: nginx # service-type: external annotations: oci.oraclecloud.com/load-balancer-type: "lb" spec: type: LoadBalancer ports: - port: 80 selector: app: nginx
-
Validate if the required annotations to create an internal LoadBalancer are added using the command below:
$ kubectl apply -f test_service.yaml --dry-run=server -o json | jq ".metadata.annotations" { "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{\"oci.oraclecloud.com/load-balancer-type\":\"lb\"},\"labels\":{\"app\":\"nginx\"},\"name\":\"my-nginx-svc\",\"namespace\":\"default\"},\"spec\":{\"ports\":[{\"port\":80}],\"selector\":{\"app\":\"nginx\"},\"type\":\"LoadBalancer\"}}\n", "oci.oraclecloud.com/load-balancer-type": "lb", "policies.kyverno.io/last-applied-patches": "mutate-lb-services.mutate-oci-services.kyverno.io: added /metadata/annotations/service.beta.kubernetes.io~1oci-load-balancer-internal\n", "service.beta.kubernetes.io/oci-load-balancer-internal": "true" }
-
You can validate if the internal load balancer is successfully created by executing the command:
$ kubectl apply -f test_service.yaml
Task 4: Clean up
-
Delete the load balancer created during the test:
$ kubectl delete -f test_service.yaml
-
Uninstall Kyverno by executing the command:
$ helm uninstall kyverno -n kyverno
Related Links
- OKE documentation
- OKE services of type Load Balancer
- Oracle Cloud Free Tier
- Sign in to your Oracle Cloud Account
- Kyverno homepage
Acknowledgments
- Author - Andrei Ilas (Principal Cloud Architect)
More Learning Resources
Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.
For product documentation, visit Oracle Help Center.
Enforce internal Oracle Container Engine for Kubernetes Load Balancer services using Kyverno
F83798-01
July 2023
Copyright © 2023, Oracle and/or its affiliates.