Note:

Inject the ImagePullSecrets attribute into Kubernetes pods using Kyverno

Introduction

Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) is a fully managed, scalable, and highly available service that you can use to deploy your containerized applications to the cloud.

Oracle Cloud Infrastructure Registry (also known as Container Registry or OCIR) is an Oracle-managed registry that enables you to simplify your development to production workflow. Container Registry makes it easy for you as a developer to store, share, and manage container images.

You can create public or private container image repositories in OCIR:

To pull the image from the OCIR private registry, Kubernetes needs credentials. The spec.imagePullSecrets attribute in the Pod configuration file specifies a list of Kubernetes secrets with credentials that worker nodes should use to pull the container images from the registries.

To simplify the application manifests and deployment, and to decouple some of the OKE cluster-specific configuration when integrating with OCIR, we can use Kyverno. This incubating project under CNCF can validate, mutate, generate, and clean up Kubernetes resources using Policies definition. We can take advantage of this product to mutate the manifests we are submitting to the Kubernetes API when we are creating new pods and inject the imagePullSecrets attribute, required to pull the container images stored in OCIR.

Objectives

Prerequisites

Task 1: Install Kyverno

The Kyverno webpage describes two methods to install Kyverno on a Kubernetes cluster, using:

In this tutorial, we will focus on how to install Kyverno using the Helm charts.

Task 1.1: Install helm

  1. Depending on the operating system you are on, go through this guide and install helm.

  2. Confirm if the installation is successful by executing the following command.

    helm version
# version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.5"}

Task 1.2: Install Kyverno in standalone mode

  1. Add the Kyverno Helm repository.

    helm repo add kyverno https://kyverno.github.io/kyverno/
# "kyverno" has been added to your repositories

  2. Scan the new repository for charts.

    helm repo update
# Hang tight while we grab the latest from your chart repositories...
# ...Successfully got an update from the "kyverno" chart repository
# Update Complete. ⎈Happy Helming!⎈

  3. Install Kyverno in kyverno namespace.

    helm install kyverno kyverno/kyverno -n kyverno --create-namespace
# NAME: kyverno
# LAST DEPLOYED: Fri Aug 30 12:20:33 2023
# NAMESPACE: kyverno
# STATUS: deployed
# REVISION: 1
# NOTES:
# Chart version: 3.0.1
# Kyverno version: v1.10.0

# Thank you for installing kyverno! Your release is named kyverno.

# The following components have been installed in your cluster:
# - CRDs
# - Admission controller
# - Reports controller
# - Cleanup controller
# - Background controller

Task 2: Create Kubernetes ImagePull Secret

  1. Generate user Auth Token following the steps provided here: Getting an Auth Token

  2. Ensure the proper policies are configured to allow user access to OCIR: Policies to Control Repository Access

  3. Confirm the OCIR registry URL, username and password:

    Registry URL is based on the OCI region key: <region-key>.ocir.io

    Example: For OCI Phoenix region: phx.ocir.io

    Username is based on the tenancy namespace, oci username and IDP(if used): <tenancy-namespace>/<username> or <tenancy-namespace>/oracleidentitycloudservice/<username>

    Example:

    axaxnpcrorw5/jdoe@acme.com or axaxnpcrorw5/oracleidentitycloudservice/jdoe@acme.com

  4. Create Kubernetes secret named ocirsecret.

    kubectl create secret docker-registry ocirsecret --docker-server='<OCIR registry>' --docker-username='<username>' --docker-password='<auth-token>'

    Example:

    kubectl create secret docker-registry ocirsecret --docker-server='phx.ocir.io' --docker-username='axaxnpcrorw5/jdoe@acme.com' --docker-password='cxOY5NL<AnBN}<123{_6'

Task 3: Define required Kyverno ClusterPolicies

  1. Create a file named add-imagepullsecret.yaml.

  2. Copy and paste the text below in the file.

    ---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: inject-imagepullsecret-to-namespace
  annotations:
    policies.kyverno.io/title: Clone imagePullSecret secret to new namespaces
    policies.kyverno.io/subject: Namespace
    policies.kyverno.io/description: >-
      ImagePullSecrets must be present in the same namespace as the pods using them.
      This policy monitors for new namespaces being created (except kube-system and kyverno),
      and automatically clones into the namespace the `ocirsecret` from the `default` namespace.
spec:
  generateExisting: true
  rules:
  - name: inject-imagepullsecret-to-namespace
    match:
      any:
      - resources:
          kinds:
          - Namespace
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - kube-node-lease
          - kube-public
          - kyverno
    generate:
      apiVersion: v1
      kind: Secret
      name: ocirsecret
      namespace: "{{ request.object.metadata.name }}"
      synchronize: true
      clone:
        namespace: default
        name: ocirsecret
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-imagepullsecrets
  annotations:
    policies.kyverno.io/title: Add imagePullSecrets
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      Images coming from certain registries require authentication in order to pull them,
      and the kubelet uses this information in the form of an imagePullSecret to pull
      those images on behalf of your Pod. This policy searches pod spec for images coming from a
      registry which contains `phx.ocir.io/axaxnpcrorw5` and, if found, will mutate the Pod
      to add an imagePullSecret called `ocirsecret`.
spec:
  rules:
  - name: add-imagepullsecret
    match:
      any:
      - resources:
          kinds:
          - Pod
    context:
    - name: images_in_ocir
      variable:
        jmesPath: "[request.object.spec.containers[*].image.contains(@, 'phx.ocir.io/axaxnpcrorw5'), request.object.spec.initContainers[*].image.contains(@, 'phx.ocir.io/axaxnpcrorw5')][]"
        default: []
    preconditions:
      all:
      - key: true
        operator: In
        value: "{{ images_in_ocir }}"
    mutate:
      patchStrategicMerge:
        spec:
          imagePullSecrets:
          - name: ocirsecret

  3. Replace the default OCIR address phx.ocir.io/axaxnpcrorw5 with the one you are using.

  4. Create the ClusterPolicy in the cluster and execute below command to enforce the policy.

    kubectl apply -f add-imagepullsecret.yaml
# clusterpolicy.kyverno.io/inject-imagepullsecret-to-namespace created
# clusterpolicy.kyverno.io/add-imagepullsecrets created

Task 4: Test

  1. Create a file named test-pod.yaml containing the text below:

    Note: Fill-in the URL of the image from the private repository in the placeholder.

    ---
apiVersion: v1
kind: Namespace
metadata:
  name: testns
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: testpod
  name: testpod
  namespace: testns
spec:
  containers:
  - args:
    - /bin/sh
    - -c
    - sleep infinity
    image: <image_from_OCIR> # eg: phx.ocir.io/axaxnpcrorw5/nginx:latest
    name: test
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always

  2. Create the resources

    kubectl apply -f test-pod.yaml

  3. Validate if the pod is successfully created.

    kubectl get pods -n testns
# NAME      READY   STATUS    RESTARTS   AGE
# testpod   1/1     Running   0          2m4s

  4. Validate if the secret was cloned from the default namespace to the new created namespace.

    kubectl get secret -n testns
# NAME         TYPE                             DATA   AGE
# ocirsecret   kubernetes.io/dockerconfigjson   1      2m56s

Task 5: Clean up

  1. Delete the load balancer created during the test.

    kubectl delete -f test-pod.yaml

  2. Uninstall Kyverno by executing the command.

    helm uninstall kyverno -n kyverno

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.