Using Karpenter Provider for OCI (KPO)

Prerequisites

Before you install KPO, confirm that the cluster and networking configuration support Karpenter-provisioned worker nodes. Note that it's possible for installation to succeed, but for subsequent worker node provisioning to fail due to missing prerequisites.

• Kubernetes version running on cluster: Kubernetes version 1.31 or later.

• Worker node capacity: Run at least one existing managed node pool or self-managed worker nodes so KPO can run in the data plane.

• OCI VCN-Native Pod Networking CNI plugin cluster add-on (also known as the OCI VCN IP Native CNI add-on) version: If the cluster is using the OCI VCN IP Native CNI add-on, the add-on must be version 3.0.0 or later. We strongly recommend the use of version 3.2.0 or later. If the cluster is using a version of the add-on prior to version 3.2.0, secondary VNICs support a maximum of 16 IP addresses (ipCount must not exceed 16).

• Helm client installed: Install the Helm client in the environment where you want to run Helm commands.

Download the Helm Chart

Download the KPO Helm chart tarball from your approved distribution location, then review supported configuration values before you deploy KPO.

To download KPO, go to https://github.com/oracle/karpenter-provider-oci/releases.

To review supported chart values:

helm show values <path-to-kpo-chart.tgz>

Review and Configure Helm Values

Review the Helm configuration values (often simply referred to as Helm values) in the downloaded chart tarball by entering:

helm show values <path-to-kpo-chart.tgz>

The following table shows the most important Helm values:

repairPolicies:
  - conditionType: Ready
    conditionStatus: "False"
    tolerationDuration: 600000000000

You must set the required Helm values to enable KPO to resolve OCI resources and bootstrap worker nodes. You can also set the optional Helm values to match your networking model and your worker node strategy.

You set Helm values in a configuration file (referred to as the Helm values file).

Install KPO

Deploy the KPO controller into the OKE data plane. Install KPO into a namespace you can manage and monitor with your cluster operations tooling.

1. Choose the namespace where you want to deploy KPO.

2. Create a Helm values file in YAML format that includes required values and any optional values you want to override. For example:

   settings:
     clusterCompartmentId: "<cluster-compartment-id>"
     vcnCompartmentId: "<cluster-vcn-compartment-id>"
     ociVcnIpNative: false
     apiserverEndpoint: 10.0.0.14
   
   image:
     registry: "phx.ocir.io"

3. Install the chart by entering:

   helm install karpenter <path-to-chart-tarball> \
     --values <path-to-helm-values-file> \
     --namespace <karpenter-namespace> \
     --create-namespace

   If you do not specify a namespace in which to deploy KPO, the default namespace is used.

For more information about installing KPO, see the KPO installation documentation on Github.

Verify the Installation

Having installed KPO, confirm that the KPO controller starts successfully before you create OCINodeClass and NodePool resources.

1. Confirm that the pods are running by entering:

   kubectl get pods --namespace <karpenter-namespace>

2. If the pods are not running, inspect pod events and logs by entering:

   kubectl describe pod --namespace <karpenter-namespace> <pod-name>

   kubectl logs --namespace <karpenter-namespace> <pod-name>

Workload Identity Policy Pattern

Use the following pattern to specify workload identity conditions in IAM policies so OCI permissions apply only to the KPO workload in the cluster:

Allow any-user to <verb> <resource> in <location> where all {
  request.principal.type = 'workload',
  request.principal.namespace = '<namespace-name>',
  request.principal.service_account = '<service-account-name>',
  request.principal.cluster_id = '<cluster-ocid>'
}

Basic Policies Required for KPO Operation

Grant permissions to KPO to manage the compute, storage, and networking resources required for worker nodes by including policy statements similar to the following in an IAM policy (using the pattern shown in Workload Identity Policy Pattern):

Allow any-user to manage instance-family in compartment <compartment-name> where all { ... }
Allow any-user to manage volumes in compartment <compartment-name> where all { ... }
Allow any-user to manage volume-attachments in compartment <compartment-name> where all { ... }
Allow any-user to manage virtual-network-family in compartment <compartment-name> where all { ... }
Allow any-user to inspect compartments in compartment <compartment-name> where all { ... }

Note that if you enable optional features in OCINodeClass, you also have to define additional policies (see Optional Policies (Feature-Specific)).

Optional Policies (Feature-Specific)

In addition to the policy statements shown in Basic Policies Required for KPO Operation, if you enable optional features in OCINodeClass, you also have to define additional policies.

Only add the policies required by the features you enable in OCINodeClass.

OCINodeClassSpec

VolumeConfig

Use volumeConfig to control how KPO builds the boot volume for each worker node.

BootVolumeConfig

Use bootVolumeConfig to select the image and to configure boot volume sizing and performance.

ImageConfig

Use imageConfig to select the OKE image for worker nodes. Select an image by OCID or by filter. For more information about OKE images, see OKE Images.

KmsKeyConfig

Use kmsKeyConfig to encrypt boot volumes with your own key.

ImageSelectorTerm

Use ImageSelectorTerm to select an image by OS, version, compartment, and tags.

NetworkConfig

Use networkConfig to place worker node VNICs in the correct subnets and attach NSGs when needed.

PrimaryVnicConfig

Use primaryVnicConfig to define the primary subnet and optional NSGs for the worker node.

SubnetConfig

Select a subnet by OCID or by selector.

NetworkSecurityGroupConfigs

Select an NSG by OCID or by selector.

Ipv6AddressIpv6SubnetCidrPairDetails

Use this field when you need to control IPv6 subnet CIDR assignment.

OciResourceSelectorTerm

Use selector fields when you want to choose a subnet or NSG by name or by tags, instead of by OCID.

SecondaryVnicConfigs

Use secondaryVnicConfigs when the cluster uses the OCI VCN IP Native CNI add-on and you need a secondary VNIC for pod IP addressing. Has all of the PrimaryVnicConfig fields, and some additional fields.

ShapeConfigs

Use shapeConfigs to define sizing for flexible shapes and burstable configurations.

CapacityReservationConfigs

Use capacityReservationConfigs when you want worker nodes to run on capacity reserved in OCI.

ClusterPlacementGroupConfigs

Use clusterPlacementGroupConfigs when you need worker nodes placed close together for low-latency networking.

ComputeClusterConfig

Use computeClusterConfig when you want worker nodes to run in a compute cluster.

KubeletConfig

Use kubeletConfig to override kubelet settings on Karpenter-provisioned worker nodes. Karpenter Provider for OCI (KPO) applies these settings during node provisioning.

LaunchOptions

Use launchOptions to set instance launch options.

OCINodeClassStatus

Use status fields to troubleshoot validation and resolved configuration.

Scheduling Labels

Use node labels in NodePool.spec.template.spec.requirements and in workload scheduling rules (for example, node selectors and node affinity). Use standard Kubernetes labels and OCI-specific labels. KPO adds OCI-specific labels to nodes it provisions.

shapename.<X>o.<Y>g.<burstableRatio>b

VM.Standard.E5.Flex.2o.32g.1_1b

Required Taints and Tolerations

Use taints and tolerations to control where workloads run. Configure taints in the NodePool so Karpenter offers the corresponding capacity, then configure tolerations in workloads that must run on those worker nodes.

Be aware of the following special scenarios:

• Preemptible (Spot) Worker Nodes:

  Worker nodes launched as spot capacity use the taint oci.oraclecloud.com/oke-is-preemptible.

  To use spot capacity:

  1. Add the taint to the NodePool so Karpenter offers spot shapes.

  2. Add the corresponding toleration to workloads that must run on preemptible nodes.

  NodePool taint example:

  template:
    spec:
      taints:
        - effect: NoSchedule
          key: oci.oraclecloud.com/oke-is-preemptible
          value: present

  If the NodePool does not include the taint oci.oraclecloud.com/oke-is-preemptible, Karpenter does not offer preemptible shapes.

• GPU Worker Nodes:

  Worker nodes launched with GPU shapes use vendor-specific taints:

  • NVIDIA GPU shapes: nvidia.com/gpu

  • AMD GPU shapes: amd.com/gpu

  To use GPU shapes:

  1. Add the appropriate taint to the NodePool so Karpenter offers GPU shapes.

  2. Add the corresponding toleration to GPU workloads.

  NodePool taint example (NVIDIA):

  template:
    spec:
      taints:
        - effect: NoSchedule
          key: nvidia.com/gpu
          value: present

  If the NodePool does not include the appropriate taint, Karpenter does not offer the corresponding shapes.

Example 1: Flexible Shapes with an Explicit OKE Image OCID

Use this example when you want a NodePool that allows specific flexible shapes, and you want an OCINodeClass that defines sizing for those flexible shapes and selects an OKE image by OCID.

---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: my-nodepool
spec:
  template:
    spec:
      expireAfter: Never
      nodeClassRef:
        group: oci.oraclecloud.com
        kind: OCINodeClass
        name: my-ocinodeclass
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values:
            - on-demand
        - key: oci.oraclecloud.com/instance-shape #expand this list as needed
          operator: In
          values:
            - VM.Standard.E3.Flex
            - VM.Standard.E4.Flex
            - VM.Standard.E5.Flex
      terminationGracePeriod: 120m
  disruption:
    budgets:
      - nodes: 5%
    consolidateAfter: 60m
    consolidationPolicy: WhenEmpty
  limits:
    cpu: 64
    memory: 256Gi 
---
apiVersion: oci.oraclecloud.com/v1beta1
kind: OCINodeClass
metadata:
  name: my-ocinodeclass
spec:
  shapeConfigs:
    - ocpus: 2  
      memoryInGbs: 8
    - ocpus: 4
      memoryInGbs: 16
  volumeConfig:
    bootVolumeConfig:
      imageConfig:
        imageType: OKEImage
        imageId: <OKE-Image-OCID> 
  networkConfig:
    primaryVnicConfig:
      subnetConfig:
        subnetId: <Subnet-OCID> 

Example 2: Image Filter Selection with Drift Replacement

Use this example when you want KPO to use a filter to select an image automatically, and you want Karpenter to replace worker nodes when they drift from the selected image.

The image that is selected depends on the cluster's Kubernetes version and the available OKE images. When the cluster control plane is upgraded or new OKE images are released, the desired worker node image will also change. Nodes launched with an outdated image are considered to have "drifted". To minimize unexpected disruption during such events, we recommend that you configure an appropriate disruption budget in the Karpenter node pool, specifying reasons, disruption percentage, and schedule (for more information, see Disruption in the Karpenter documentation).

---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: my-nodepool
spec:
  template:
    spec:
      expireAfter: Never
      nodeClassRef:
        group: oci.oraclecloud.com
        kind: OCINodeClass
        name: my-ocinodeclass
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values:
            - on-demand
        - key: oci.oraclecloud.com/instance-shape #expand this list as needed
          operator: In
          values:
            - VM.Standard.E3.Flex
            - VM.Standard.E4.Flex
            - VM.Standard.E5.Flex
      terminationGracePeriod: 120m
  disruption:
    budgets:
      - nodes: 5%
        reasons: 
          - Drifted
        schedule: "@daily" #customize schedule for your own needs, following https://karpenter.sh/docs/concepts/disruption/#schedule 
        duration: 10m
    consolidateAfter: 60m
    consolidationPolicy: WhenEmpty
  limits:
    cpu: 64
    memory: 256Gi 
---
apiVersion: oci.oraclecloud.com/v1beta1
kind: OCINodeClass
metadata:
  name: my-ocinodeclass
spec:
  shapeConfigs:
    - ocpus: 2  
      memoryInGbs: 8
    - ocpus: 4
      memoryInGbs: 16
  volumeConfig:
    bootVolumeConfig:
      imageConfig:
        imageType: OKEImage
        imageFilter: 
          osFilter: "Oracle Linux"
          osVersionFilter: "8"
  networkConfig:
    primaryVnicConfig:
      subnetConfig:
        subnetId: <Subnet-OCID> 

Example 3: Secondary VNIC for OCI VCN IP Native CNI

Use this example when your cluster uses the OCI VCN IP Native CNI add-on and pods need IP addresses from a secondary VNIC subnet. Configure secondaryVnicConfigs to attach a secondary VNIC and allocate pod IP capacity.

---
apiVersion: oci.oraclecloud.com/v1beta1
kind: OCINodeClass
metadata:
  name: my-ocinodeclass
spec:
  shapeConfigs:
    - ocpus: 2  
      memoryInGbs: 8
    - ocpus: 4
      memoryInGbs: 16
  volumeConfig:
    bootVolumeConfig:
      imageConfig:
        imageType: OKEImage
        imageFilter: 
          osFilter: "Oracle Linux"
          osVersionFilter: "8"  
  networkConfig:
    primaryVnicConfig:
      subnetConfig:
        subnetId: <Subnet-OCID> 
    secondaryVnicConfigs:
      - subnetConfig:
          subnetId: <Subnet-OCID>  #pod subnet
        ipCount: 16

Additional Use Cases

Use the following table to map common OCI features to NodePool requirements or OCINodeClass fields.

requirements:
  - key: karpenter.sh/capacity-type
    operator: In
    values: ["spot"]

capacityReservationConfigs:
  - capacityReservationId: "<capacity-reservation-ocid>"

shapeConfigs:
  - ocpus: 2
    memoryInGbs: 16
    baselineOcpuUtilization: BASELINE_1_2

clusterPlacementGroupConfigs:
  - clusterPlacementGroupId: "<cluster-placement-group-ocid>"

computeClusterConfig:
  computeClusterId: "<compute-cluster-ocid>"

Identify Karpenter-managed worker nodes

Karpenter applies the karpenter.sh/nodepool label to the nodes it creates. To list the nodes created by Karpenter, and the NodePool resource that each node belongs to, enter:

kubectl get nodes -L karpenter.sh/nodepool

To find out whether Karpenter is creating NodeClaim resources, or whether NodeClaim resources are blocked for some reason (for example, failing to launch, failing to register, or waiting on capacity), enter:

kubectl get nodeclaims

kubectl describe nodeclaim <nodeclaim-name>

Identify OCI instances (and related resources) created for Karpenter nodes

Use OCI search tools (such as the Resource Explorer) and the following OCI freeform tags to find instances and related resources that KPO created:

• karpenterNodepool

• orcl-containerengine/cluster-id (when applicable)

For more information about locating OCI resources, see Querying Resources

Clean up resources

Before you delete a cluster, remove Karpenter-managed capacity so that instances do not remain running after the cluster has been deleted.

1. Delete Karpenter NodePool resources that create capacity by using kubectl delete and the NodePool name as follows:

   1. List NodePool resources by entering:

      kubectl get nodepools

   2. Delete a NodePool resource by entering:

      kubectl delete nodepool <nodepool-name>

   3. (Optional) Confirm that the NodePool resource is deleted by entering:

      kubectl get nodepools

2. Confirm that NodeClaim resources and nodes are removed.

   1. Watch NodeClaim resources until they are deleted by entering:

      kubectl get nodeclaims --watch

   2. Watch nodes created for Karpenter node pools until they are removed:

      kubectl get nodes -l karpenter.sh/nodepool --watch

   3. If a NodeClaim does not terminate, check its status, conditions, and events by entering:

      kubectl describe nodeclaim <nodeclaim-name>

3. Use the OCI Console, CLI, or SDK to confirm that OCI instances and attached resources are deleted.

   1. Locate the instances that KPO created using the karpenterNodepool freeform tag (and the orcl-containerengine/cluster-id tag, when applicable).
   2. Confirm that no such instances are present, or are in a terminated state and then disappear.
   3. Confirm that there are no remaining VNIC attachments for the instance, and that any VNICs that were created for the instance no longer exist.
   4. Confirm that there are no remaining volume attachments for the instance, and that the boot volume that was created for the instance is deleted (or is terminated and then removed).

   If any of the resources remain after the instance is terminated, treat them as potential orphaned resources and investigate whether:

   • the KPO controller is running and has IAM permissions to clean up resources
   • there are any deletion finalizers or errors in the KPO controller logs

What you can troubleshoot and where to look

When a workload can’t schedule or a worker node doesn’t join the cluster, check resources and log files, and metrics, and increase logging, as follows:

• Use kubectl to check Kubernetes resources (see Check Kubernetes resources).

• Use the OCI Console, CLI, or SDK to confirm that KPO created the expected instances, VNICs, NSGs, and boot volumes for worker nodes.

• Use kubectl to check KPO controller logs (see Check KPO controller logs).

• Increase logging (see Enable debug logging).

Check Kubernetes resources

Use kubectl commands to list key resources, as follows:

kubectl get nodepools

kubectl get ocinodeclasses

kubectl get nodeclaims

kubectl get nodes -l karpenter.sh/nodepool

Use kubectl commands to review details and conditions, as follows:

kubectl describe ocinodeclass <name>

kubectl describe nodeclaim <name>

Check KPO controller logs

Use kubectl commands to review the status of KPO pods and to review logs, as follows:

kubectl get pods --namespace <karpenter-namespace>

kubectl logs --namespace <karpenter-namespace> <pod-name>

Enable debug logging

Increase logging temporarily when you need more detail in KPO controller output, in one or both of the following ways:

• Set logLevel: debug in the Helm values file and update the KPO deployment (see How do you change and reapply Helm values to update the KPO deployment?). By default, logging is at the info level.

• Set OCI_GO_SDK_DEBUG=1 as an environment variable in the KPO deployment to enable OCI Go SDK debug logging.

Do special taints added by OKE bootstrapping apply to Karpenter-managed nodes?

Yes. Some worker node types require specific taints. Configure the taints in the NodePool so Karpenter offers the corresponding capacity. Then configure tolerations in workloads that must run on those worker nodes.

For more information about special taints, see Required Taints and Tolerations.

How do you configure secondary VNICs?

When a cluster uses the OCI VCN IP Native CNI add-on, you can attach secondary VNICs to worker nodes so pods receive VCN-routable IP addresses from the pod subnet. Plan subnet capacity carefully so you don’t exhaust available addresses.

If the cluster is using the OCI VCN IP Native CNI add-on, the add-on must be version 3.0.0 or later. We strongly recommend the use of version 3.2.0 or later. In OCI VCN IP Native CNI add-on version 3.2.0 (and later), secondary VNICs support a maximum of 256 IP addresses. If the cluster is using a version of the add-on prior to version 3.2.0, secondary VNICs support a maximum of 16 IP addresses. If the cluster has to use a version of the add-on prior to version 3.2.0, you must explicitly set ipCount to a value no greater than 16.

In addition to the maximum number of supported IP addresses, secondary VNICs are subject to the following restrictions:

• The number of assigned IP addresses must be a power of two, so set ipCount to a power of two.

• For IPv6-only (single stack) secondary VNICs, only 1, 16, or 256 assigned IP addresses are supported, so set ipCount to 1, 16, or 256.

• The aggregate total of all assigned IP addresses across all secondary VNICs within a node must not exceed 256.

• If ipCount is not set for a secondary VNIC, it defaults to 32 for IPv4 clusters and IPv6 dual stack clusters, and to 256 for IPv6 single stack clusters.

Note the following recommendations and guidelines:

• We recommend that you configure two CIDR blocks for the pod subnet used by secondary VNICs.
• If the pod subnet used by secondary VNICs has a single CIDR block, make sure that the subnet has a sufficient number of contiguous IP addresses to accommodate the required number of IP assignments.

How do you schedule workloads on a specific Karpenter node pool?

To force workloads onto a specific Karpenter node pool, target the node pool label karpenter.sh/nodepool. Use either node affinity or a node selector, depending on how strict you want placement to be.

• Node affinity example:

  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: karpenter.sh/nodepool
                operator: In
                values:
                  - <nodepool-name>

  For more information about node affinity, see Node affinity in the Kubernetes documentation.

• Node selector example:

  nodeSelector:
    karpenter.sh/nodepool: <nodepool-name>

  For more information about node selectors, see nodeSelector in the Kubernetes documentation.

How do you spread workloads across availability domains and fault domains with Karpenter Provider for OCI?

Karpenter Provider for OCI (KPO) honors pod-level topologySpreadConstraints when the topologyKey matches a scheduling label that KPO supports. Typically, you'll use these topology keys most often in OCI:

• topology.kubernetes.io/zone to spread across availability domains (ADs)
• oci.oraclecloud.com/fault-domain to spread across fault domains (FDs) within an AD

To spread Karpenter-provisioned capacity, target the intended node pool by using the karpenter.sh/nodepool label, and ensure that the selected NodePool allows the ADs or FDs that you want the scheduler to use. If you set whenUnsatisfiable to DoNotSchedule, pods that can’t satisfy the spread constraint remain pending, which gives Karpenter an opportunity to provision nodes that meet the spread requirement:

We recommend the following practices:

• Set replicas to a value equal to or greater than the number of topology domains you want to use.
• Use AD spreading when the workload must remain available across multiple ADs.
• Use FD spreading in a single-AD region.
• Ensure that NodePool requirements include all topology values that topologySpreadConstraints uses.

Example: Configure a NodePool to allow three ADs

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: example-ad-nodepool
spec:
  template:
    spec:
      requirements:
      - key: topology.kubernetes.io/zone
        operator: In
        values:
        - <AVAILABILITY_DOMAIN_1>
        - <AVAILABILITY_DOMAIN_2>
        - <AVAILABILITY_DOMAIN_3>

Example: Spread a Deployment across those three ADs

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-ad-spread
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example-ad-spread
  template:
    metadata:
      labels:
        app: example-ad-spread
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: karpenter.sh/nodepool
                operator: In
                values:
                - example-ad-nodepool
      topologySpreadConstraints:
      - maxSkew: 1
        minDomains: 3
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: example-ad-spread
        matchLabelKeys:
        - pod-template-hash
        nodeAffinityPolicy: Honor
        nodeTaintsPolicy: Honor
      containers:
      - name: app
        image: registry.k8s.io/pause:3.9
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: "1"

Example: Configure a NodePool to allow three FDs in a single AD

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: example-fd-nodepool
spec:
  template:
    spec:
      requirements:
      - key: topology.kubernetes.io/zone
        operator: In
        values:
        - <AVAILABILITY_DOMAIN>
      - key: oci.oraclecloud.com/fault-domain
        operator: In
        values:
        - FAULT-DOMAIN-1
        - FAULT-DOMAIN-2
        - FAULT-DOMAIN-3

Example: Spread a Deployment across those three FDs

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-fd-spread
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example-fd-spread
  template:
    metadata:
      labels:
        app: example-fd-spread
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: karpenter.sh/nodepool
                operator: In
                values:
                - example-fd-nodepool
      topologySpreadConstraints:
      - maxSkew: 1
        minDomains: 3
        topologyKey: oci.oraclecloud.com/fault-domain
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: example-fd-spread
        matchLabelKeys:
        - pod-template-hash
        nodeAffinityPolicy: Honor
        nodeTaintsPolicy: Honor
      containers:
      - name: app
        image: registry.k8s.io/pause:3.9
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: "1"

Notes

• Use whenUnsatisfiable: DoNotSchedule when you want Karpenter to provision capacity that satisfies the spread constraint.
• Ensure that labelSelector matches the pod labels. If it does not match, the scheduler does not calculate skew for the intended pod set.
• For more information about supported scheduling labels, see Scheduling, Labels, and Taints.

How do you list compatible OKE images for a cluster?

Use the OCI CLI to retrieve node pool options and filter the returned image sources. This approach enables you to identify image OCIDs that are compatible with the cluster Kubernetes version.

1. Set environment variables that specify the region and cluster OCID, and that define filters for the Kubernetes version, OS major version, and any architectures or image types to exclude:

   REGION="<region>"
   CLUSTER_OCID="<cluster-ocid>"
   OKE_VERSION="<kubernetes-version>"
   OS_MAJOR="<os-major-version>"
   EXCLUDE_PATTERN="<architecture-pattern>"

   where:

   • <cluster-ocid> is the OCID of the cluster.
   • <kubernetes-version> is the Kubernetes minor version string to match in the image source name (for example, 1.31).
   • <os-major-version> is the OS major version to match in the image source name (for example, 8 for Oracle Linux 8). Set this to an empty string to avoid filtering by OS version.
   • <architecture-pattern> is a regular expression pattern used to exclude image source names (for example, aarch64|arm64|GPU to exclude ARM or GPU images). Set this to an empty string to avoid exclusions.

   For example:

   REGION="us-phoenix-1"
   CLUSTER_OCID="ocid1.cluster.oc1.phx.aaaaaaaa______w5q"
   OKE_VERSION="1.31"
   OS_MAJOR="8"
   EXCLUDE_PATTERN="aarch64|arm64|GPU"

2. Run the following OCI CLI command to obtain the OCIDs of compatible images:

   oci ce node-pool-options get --region "${REGION}" --node-pool-option-id "${CLUSTER_OCID}" --output json | jq -r --arg ver "${OKE_VERSION:-}" --arg os "${OS_MAJOR:-}" --arg ex "${EXCLUDE_PATTERN:-}" '.data.sources[] | . as $src | ($src["source-name"] // "") as $name | select( ($ver == "" or ($name | test($ver))) and ($os == "" or ($name | test($os; "i"))) and ($ex == "" or ($name | test($ex; "i") | not)) ) | {id: $src["image-id"], source_name: $name}'

What to do if a flexible shape node pool does not provision, and you see: “skipping, nodepool requirements filtered out all instance types.”

Flexible shapes require a sizing configuration so that KPO can translate shape requirements into a concrete offering. Fix this by defining shapeConfigs in the referenced OCINodeClass or by defining defaults in the Helm values file under settings.flexibleShapeConfigs.

• Example: Defining shapeConfigs in the referenced OCINodeClass:

  apiVersion: oci.oraclecloud.com/v1beta1
  kind: OCINodeClass
  metadata:
    name: example-nodeclass
  spec:
    shapeConfigs:
      ocpus: 2
      memoryInGbs: 16
      baselineOcpuUtilization: BASELINE_1_2
  ...

• Example: Setting default shapeConfigs values globally in the Helm values file:

  settings:
    flexibleShapeConfigs:
      ocpus: 2
      memoryInGbs: 16

  You can override a default in the Helm values file by setting shapeConfigs in the OCINodeClass.

If you’re using a capacity reservation and facing this issue, confirm that flexibleShapeConfigs in the Helm values file (or shapeConfigs in the OCINodeClass, if present) matches the reservation exactly, with the same values for ocpus, memoryInGbs, and baselineOcpuUtilization.

How do you run the KPO controller with debug logging?

Increase log output to troubleshoot provisioning and OCI API calls in one or both of the following ways:

• Set logLevel: debug in the Helm values file and update the KPO deployment (see How do you change and reapply Helm values to update the KPO deployment?).

• Set OCI_GO_SDK_DEBUG=1 as an environment variable in the KPO deployment to enable OCI Go SDK debug logging.

How do you pass a custom cloud-init script?

You can execute specific commands or configurations during a node's startup process by injecting custom cloud-init scripts using the OCINodeClass resource.

Thoroughly test custom cloud-init scripts before deploying them, to ensure the scripts execute as expected and do not interfere with the standard node initialization process.

There are two primary methods:

• Option 1: Use preBootstrapInitScript and postBootstrapInitScript in OCINodeClass (recommended)

  You can run custom scripts before and after the default cloud-init script by using preBootstrapInitScript and postBootstrapInitScript in the OCINodeClass to specify the custom scripts, as follows:

  1. Prepare the custom cloud-init scripts:
     1. Create the scripts you want to run before and after the default node initialisation.
     2. Base64-encode each script.

  2. Add the base64-encoded scripts to the preBootstrapInitScript and/or postBootstrapInitScript fields in the OCINodeClass resource.

     For example:

     apiVersion: oci.oraclecloud.com/v1beta1
     kind: OCINodeClass
     metadata:
       name: example-nodeclass
     spec:
       ...
       preBootstrapInitScript: "IyEvYmluL2Jhc2gKZWNobyAiSSBhbSBhIHByZSBib290c3RyYXAgc2NyaXB0Ig=="
       postBootstrapInitScript: "IyEvYmluL2Jhc2gKZWNobyAiSSBhbSBhIHBvc3QgYm9vdHN0cmFwIHNjcmlwdCI="

• Option 2: Specify a full base64-encoded cloud-init script in metadata.user_data (advanced workflow)

  You can provide a complete custom cloud-init script by setting the user_data field in the metadata section of the OCINodeClass. This method gives you full control of the initialization process, but requires careful management to ensure compatibility with Kubernetes Engine.

  1. Prepare the custom cloud-init script:

     1. Create the custom cloud-init script, and include both the custom configurations and the necessary commands that join the node to the cluster.

        The following example reads configuration values from the OCI Instance Metadata Service (IMDS), which is available only from within the instance, and then runs the commands that join the node to the cluster.

        #!/usr/bin/env bash
        set -o errexit
        set -o nounset
        set -o pipefail
        # OCI Instance Metadata Service (IMDS): link-local, reachable only from within the instance.
        # Do not use IMDS URLs with untrusted input (SSRF risk).
        MD_URL="http://169.254.169.254/opc/v2/instance/metadata"
        AUTH_HDR="Authorization: Bearer Oracle"
        
        # Fetch a metadata key, returning empty on error/missing
        fetch_md() {
          local key="$1"
          curl -sfL --noproxy '*' -H "${AUTH_HDR}" --connect-timeout 2 --max-time 5 "${MD_URL}/${key}" 2>/dev/null || true
        }
        
        CLUSTER_DNS="$(fetch_md kubedns_svc_ip)"
        KUBELET_EXTRA_ARGS="$(fetch_md kubelet-extra-args)"
        APISERVER_ENDPOINT="$(fetch_md apiserver_host)"
        KUBELET_CA_CERT="$(fetch_md cluster_ca_cert)"
        
        # Export only when present to avoid surprising consumers with empty values
        [ -n "${CLUSTER_DNS}" ] && export CLUSTER_DNS
        [ -n "${KUBELET_EXTRA_ARGS}" ] && export KUBELET_EXTRA_ARGS
        [ -n "${APISERVER_ENDPOINT}" ] && export APISERVER_ENDPOINT
        [ -n "${KUBELET_CA_CERT}" ] && export KUBELET_CA_CERT
        
        # BEGIN OF CUSTOM SCRIPT BOOTSTRAP SCRIPT , REPLACE THIS SECTION WITH CUSTOM PRE BOOTSTRAP SCRIPT
        #echo "pre bootstrap script"
        #echo "CLUSTER_DNS: ${CLUSTER_DNS:-}"
        #echo "KUBELET_EXTRA_ARGS: ${KUBELET_EXTRA_ARGS:-}"
        #echo "APISERVER_ENDPOINT: ${APISERVER_ENDPOINT:-}"
        #echo "KUBELET_CA_CERT: ${KUBELET_CA_CERT:-}"
        # END OF CUSTOM SCRIPT BOOTSTRAP SCRIPT
        
        bash /etc/oke/oke-install.sh
        
        # BEGIN OF POST BOOTSTRAP SCRIPT, IF NEEDED
        #echo "post bootstrap script"
        #END OF POST BOOTSTRAP SCRIPT

        If you use this example, only insert your custom logic:

        • between # BEGIN OF CUSTOM SCRIPT BOOTSTRAP SCRIPT , REPLACE THIS SECTION WITH CUSTOM PRE BOOTSTRAP SCRIPT and # END OF CUSTOM SCRIPT BOOTSTRAP SCRIPT

        • between # BEGIN OF POST BOOTSTRAP SCRIPT, IF NEEDED and #END OF POST BOOTSTRAP SCRIPT

        Note that the parameters necessary to bootstrap nodes are available in the Instance Metadata Service (IMDS). When creating a custom script, we strongly recommend setting the CLUSTER_DNS, KUBELET_EXTRA_ARGS, APISERVER_ENDPOINT, and KUBELET_CA_CERT environment variables by retrieving their values from IMDS, as shown in the example script. Configuring these variables as shown is essential for the correct operation of KPO.

     2. Base64-encode the script.

  2. Add the base64-encoded script to the user_data field in the metadata section of the OCINodeClass.

     For example:

     apiVersion: oci.oraclecloud.com/v1beta1
     kind: OCINodeClass
     metadata:
       name: example-nodeclass
     spec:
      metadata:
         user_data: "IyEvdXNyL2Jpbi9lbnYgYmFzaAoKc2V0IC1vIGVycmV4aXQKc2V0IC1vIG5vdW5zZXQKc2V0IC1vIHBpcGVmYWlsCg=="
     ...

What should you consider for KubeletConfig maxPods and podsPerCore?

Use kubeletConfig to control the pod density of worker nodes, and keep it aligned with your networking capacity.

• Set podsPerCore to a value that does not exceed maxPods.

• For clusters that use the OCI VCN IP Native CNI add-on, set maxPods lower than the aggregate sum of ipCount values across secondary VNICs.

How do you configure the OKE prebuilt image compartment OCID in Karpenter Provider for OCI (KPO)?

When you want KPO to use a custom image based on an OKE image (by setting imageType: OKEImage), and the image is located in a different compartment, you can enable KPO to discover the image using the preBakedImageCompartmentId Helm value or the imageFilter of OCINodeClass.

If you use only Oracle-published OKE images or you reference images directly by OCID (imageId), this configuration is not necessary.

Prerequisites

• You know the compartment OCID that contains the images you want KPO to discover.

• A suitable policy exists to enable the KPO controller to read images in the different compartment. For example:

  Allow any-user to read instance-images in compartment <compartment-name> where all { ...workload identity conditions... }

Option 1: Configure the compartment for all OCINodeClass resources (using a Helm value)

1. Set the preBakedImageCompartmentId Helm value:

   settings: 
     preBakedImageCompartmentId: "<compartment-ocid-containing-images>"

2. Apply the updated configuration by running helm upgrade for the KPO deployment (see How do you change and reapply Helm values to update the KPO deployment?).

Option 2: Configure the compartment for a specific OCINodeClass (using imageFilter)

Set compartmentId in imageFilter:

volumeConfig:
  bootVolumeConfig:
    imageConfig:
      imageType: OKEImage
      imageFilter:
        compartmentId: "<compartment-ocid-containing-images>"
        osFilter: "Oracle Linux"
        osVersionFilter: "8"

Custom images based on OKE images (k8s_version requirement)

When you use a custom image based on an OKE image, ensure that the image has one or other of the following:

• a k8s_version freeform tag
• a BaseImageId value that points (directly or indirectly) to an ancestor OKE image that has the k8s_version tag

If neither is present, KPO cannot determine the Kubernetes version for the image, and image selection can fail with a missing k8s_version tag error.

How do you change and reapply Helm values to update the KPO deployment?

To change Helm values after installation, update the Helm values file, and then apply the new configuration.

1. Update the Helm values file (for example, add or change logLevel: debug).

2. Apply the updated configuration by entering:

   helm upgrade karpenter <path-to-chart-tarball> \
     --values <path-to-helm-values-file> \
     --namespace <karpenter-namespace>

3. Confirm that the updated configuration is applied and that the KPO controller pods restart successfully:

   kubectl get pods --namespace <karpenter-namespace>