Sample OpenTofu Scripts to Create an OKE Cluster

The sample script creates the following resources:

  • A new virtual cloud network (VCN) for the cluster.
  • Two load balancer subnets with security lists.
  • Three worker subnets with security lists.
  • A Kubernetes cluster with one node pool.
  • A kubeconfig file for access using kubectl.
  • A mount target and two file systems based on File Storage Service (FSS) for persistent volume (PV) use: one for the domain home and the other for the database volume.

Nodes and network settings are configured to allow SSH access, and the cluster networking policies allow NodePort services to be exposed. This cluster is intended for testing and development only. Do not use the sample OpenTofu scripts to create production clusters without further review.

All OCI Container Engine master nodes are highly available (HA) and fronted by load balancers.

Prerequisites

Before using the OpenTofu scripts, complete the following prerequisites:

  • Ensure that you have an existing tenancy with enough compute and networking resources for the desired cluster.
  • Ensure that an Identity and Access Management policy is in place within that tenancy to allow the OCI Container Engine for Kubernetes service to manage tenancy resources.
  • Ensure that you have a user defined within that tenancy.
  • Ensure that you have an API key for use with the OCI API. See Managing User Credentials.
  • Ensure that you have an SSH key pair for configuring SSH access to the cluster nodes.
  • Install kubectl. Its version must be within one minor version (older or newer) of the Kubernetes version running on the control plane nodes. See Install kubectl for installation instructions.
  • Download, install, and configure the OCI CLI. See Installing the CLI for installation instructions.

Copy oci.props.template to oci.props and provide the following required values:

user.ocid
The tenancy user's OCID. Obtain it from the user settings in the OCI Console.
tfvars.filename
The filename for the generated tfvars file.
okeclustername
The name of the OCI Container Engine for Kubernetes cluster.
tenancy.ocid
The target tenancy's OCID.
region
The name of the region in the target tenancy.
compartment.ocid
The target compartment's OCID. See Finding the OCID of a Compartment.
compartment.name
The target compartment's name.
ociapi.pubkey.fingerprint
The fingerprint of the OCI user's public key.
ocipk.path
The local path to the private key for the API key pair.
vcn.cidr.prefix
The VCN CIDR prefix used when creating subnets. Examine the target compartment to find an available CIDR.
vcn.cidr
The full CIDR for the VCN. It must be unique within the compartment, and its first two octets must match vcn_cidr_prefix.
nodepool.shape
A valid OCI VM shape for the cluster nodes.
k8s.version
The Kubernetes version.
nodepool.ssh.pubkey
The SSH public key contents as a string.
nodepool.imagename
A valid image OCID for node pool creation. See Supported Images.
tofu.installdir
The installation location for the tofu binaries.

Optionally, modify the node shape by editing node-pool.tf:

node_shape_config {
                    #Optional
                    memory_in_gbs = 48.0
                    ocpus = 4.0
                    }
                

Optionally, add more nodes to the cluster (two worker nodes are created by default). Modify vcn.tf to add worker subnets:

resource "oci_core_subnet" "oke-subnet-worker-3" {
                    availability_domain = data.oci_identity_availability_domains.ADs.availability_domains[2]["name"]
                    cidr_block          = "${var.vcn_cidr_prefix}.12.0/24"
                    display_name        = "${var.cluster_name}-WorkerSubnet03"
                    dns_label           = "workers03"
                    compartment_id      = var.compartment_ocid
                    vcn_id              = oci_core_virtual_network.oke-vcn.id
                    security_list_ids   = [oci_core_security_list.oke-worker-security-list.id]
                    route_table_id      = oci_core_virtual_network.oke-vcn.default_route_table_id
                    dhcp_options_id     = oci_core_virtual_network.oke-vcn.default_dhcp_options_id
                    }
                

Add the corresponding egress_security_rules and ingress_security_rules for the worker subnets:

 egress_security_rules {
                    destination = "${var.vcn_cidr_prefix}.12.0/24"
                    protocol    = "all"
                    stateless   = true
                    }
                
  ingress_security_rules {
                    stateless = true
                    protocol  = "all"
                    source    = "${var.vcn_cidr_prefix}.12.0/24"
                    }
                

Modify subnet_ids in node-pool.tf to add the new worker subnets to the pool.

subnet_ids = [oci_core_subnet.oke-subnet-worker-1.id, oci_core_subnet.oke-subnet-worker-2.id,
                    oci_core_subnet.oke-subnet-worker-3.id]

Run the script using the following command:

$ kubernetes/oke/samples/opentofu/oke.create.sh oci.props

The script reads the values from oci.props and performs the following steps:

  1. Creates a new tfvars file based on the values from the provided oci.props file.
  2. Downloads and installs the required binaries for OpenTofu and the OpenTofu OCI Provider, based on the operating system (macOS or Linux).
  3. Applies the configuration, creates an OKE cluster using OpenTofu, and generates a kubeconfig file based on the okeclustername property defined in oci.props.

If the configuration contains errors, oke.create.sh displays output such as the following:

If you ever set or change modules or backend configuration for tofu,
                    rerun this command to reinitialize your working directory. If you forget, other
                    commands will detect it and remind you to do so if necessary.
                    \u2577
                    \u2502 Error: Reference to undeclared resource
                    \u2502 
                    \u2502   on node-pool.tf line 12, in resource "oci_containerengine_node_pool" "fmw_node_pool":
                    \u2502   12:   subnet_ids = [oci_core_subnet.oke-subnet-worker-1.id, oci_core_subnet.oke-subnet-worker-2.id, oci_core_subnet.oke-subnet-worker-3.id, oci_core_subnet.oke-subnet-worker-4.id, oci_core_subnet.oke-subnet-worker-5.id]
                    \u2502 
                    \u2502 A managed resource "oci_core_subnet" "oke-subnet-worker-5" has not been declared in the root module.
                    \u2575
                    \u2577
                    \u2502 Error: Reference to undeclared resource
                    \u2502 
                    \u2502   on node-pool.tf line 12, in resource "oci_containerengine_node_pool" "fmw_node_pool":
                    \u2502   12:   subnet_ids = [oci_core_subnet.oke-subnet-worker-1.id, oci_core_subnet.oke-subnet-worker-2.id, oci_core_subnet.oke-subnet-worker-3.id, oci_core_subnet.oke-subnet-worker-4.id, oci_core_subnet.oke-subnet-worker-5.id]
                    \u2502 
                    \u2502 A managed resource "oci_core_subnet" "oke-subnet-worker-5" has not been declared in the root module.
                    \u2575
                

If the cluster is created successfully, output such as the following is displayed:

cluster_id = "ocid1.cluster.oc1.phx.axxxxxxx.....jlxa"
                    fmw1_export_path = "/fmw1"
                    fmw1_fs_ocid = "ocid1.filesystem.oc1.phx.aaaxxxxxx....fuzaaaaa"
                    fmw2_export_path = "/fmw2"
                    fmw2_fs_ocid = "ocid1.filesystem.oc1.phx.aaaaxxxxx....xxxxxxaaaa"
                    fmw_mount_target_ip = "10.1.11.43"
                    Confirm access to cluster...
                    NotReady
                    echo '[ERROR] Some Nodes in the Cluster are not in the Ready Status , sleep 10s more ...
                    Status is Ready Iter [1/100]
                    - able to access cluster
                    okecluster6 cluster is up and running
                

To access the cluster, set the KUBECONFIG environment variable or use the --kubeconfig option with the kubeconfig file generated from the okeclustername property in oci.props, for example, okeclustername6_kubeconfig.

$ kubectl get nodes --kubeconfig=okecluster6_kubeconfig
                    NAME          STATUS   ROLES   AGE     VERSION
                    10.1.10.2    Ready    node    2d11h   v1.29.1
                    10.1.11.3    Ready    node    2d11h   v1.29.1
                

To add nodes after the cluster is created, update vcn.tf and node-pool.tf, then run the following commands:

${tofu.installdir}/tofu plan -var-file=<tfvars.filename>
                    ${tofu.installdir}/tofu apply -var-file=<tfvars.filename>
                

To delete the cluster, run oke.delete.sh. The script reads oci.props from the current directory and deletes the cluster.

$ kubernetes/oke/samples/opentofu/oke.delete.sh oci.props

Deploy SOA on an OKE cluster using Helmfile

See OracleSOASuite/helm-charts/README.md for detailed instructions.

OCI NFS volume static provisioning

For SOA, reference the file system directly in the domain.storage section of values.yaml. For example:

storage:
                    capacity: 10Gi
                    reclaimPolicy: Retain
                    type: nfs 
                    path: /fmw1   #Export Path
                    nfs:
                    server: 10.0.10.156 #Mount Target IP Address
                

For the database, the following example defines a persistent volume (PV) using a file system (FS) and a mount target IP address:

apiVersion: v1
                    kind: PersistentVolume
                    metadata:
                    name: fmw2-pv-db
                    spec:
                    capacity:
                    storage: 10Gi
                    accessModes:
                    - ReadWriteOnce
                    persistentVolumeReclaimPolicy: Retain
                    csi:
                    driver: fss.csi.oraclecloud.com
                    volumeHandle: "<OCID of the file system>:<Mount Target IP Address>:/<Export Path>"
                

Update the oracledb.persistence section of values.yaml to use the created PV as a volume. For example:

   persistence:
                    storageClass: ""
                    size: 10Gi
                    accessMode: "ReadWriteOnce"
                    volumeName: "fmw2-pv-db"
                

Note:

  • An example volumeHandle for the preceding configuration:
    volumeHandle:"ocid1.filesystem.oc1.phx.aaaaanoxxxxx....aaaa:10.0.10.156:/fmw2"
  • Obtain the volumeHandle details from the console output after the cluster is created successfully.
  • When a mount target is provisioned in OCI, its Reported Size (GiB) values are very large. These values appear on the mount target page in the OCI Console. Applications can fail to install if a space requirements check reports too much available disk space. In the OCI Console, click the pencil icon beside the mount target's Reported Size parameter to specify, in gigabytes (GiB), the maximum capacity reported by file systems exported through this mount target. This setting does not limit the actual amount of data you can store. See Changing a File System's Reported Size.