Sample OpenTofu Scripts to Create an OKE Cluster
The sample script creates the following resources:
- A new virtual cloud network (VCN) for the cluster.
- Two load balancer subnets with security lists.
- Three worker subnets with security lists.
- A Kubernetes cluster with one node pool.
- A
kubeconfigfile for access usingkubectl. - A mount target and two file systems based on File Storage Service (FSS) for persistent volume (PV) use: one for the domain home and the other for the database volume.
Nodes and network settings are configured to allow SSH access, and the cluster networking policies allow NodePort services to be exposed. This cluster is intended for testing and development only. Do not use the sample OpenTofu scripts to create production clusters without further review.
All OCI Container Engine master nodes are highly available (HA) and fronted by load balancers.
Prerequisites
Before using the OpenTofu scripts, complete the following prerequisites:
- Ensure that you have an existing tenancy with enough compute and networking resources for the desired cluster.
- Ensure that an Identity and Access Management policy is in place within that tenancy to allow the OCI Container Engine for Kubernetes service to manage tenancy resources.
- Ensure that you have a user defined within that tenancy.
- Ensure that you have an API key for use with the OCI API. See Managing User Credentials.
- Ensure that you have an SSH key pair for configuring SSH access to the cluster nodes.
- Install
kubectl. Its version must be within one minor version (older or newer) of the Kubernetes version running on the control plane nodes. See Install kubectl for installation instructions. - Download, install, and configure the OCI CLI. See Installing the CLI for installation instructions.
Copy oci.props.template to oci.props and provide the following required
values:
user.ocid- The tenancy user's OCID. Obtain it from the user settings in the OCI Console.
tfvars.filename- The filename for the generated tfvars file.
okeclustername- The name of the OCI Container Engine for Kubernetes cluster.
tenancy.ocid- The target tenancy's OCID.
region- The name of the region in the target tenancy.
compartment.ocid- The target compartment's OCID. See Finding the OCID of a Compartment.
compartment.name- The target compartment's name.
ociapi.pubkey.fingerprint- The fingerprint of the OCI user's public key.
ocipk.path- The local path to the private key for the API key pair.
vcn.cidr.prefix- The VCN CIDR prefix used when creating subnets. Examine the target compartment to find an available CIDR.
vcn.cidr- The full CIDR for the VCN. It must be unique within the compartment, and its first two octets must match
vcn_cidr_prefix. nodepool.shape- A valid OCI VM shape for the cluster nodes.
k8s.version- The Kubernetes version.
nodepool.ssh.pubkey- The SSH public key contents as a string.
nodepool.imagename- A valid image OCID for node pool creation. See Supported Images.
tofu.installdir- The installation location for the
tofubinaries.
Optionally, modify the node shape by editing
node-pool.tf:
node_shape_config {
#Optional
memory_in_gbs = 48.0
ocpus = 4.0
}
Optionally, add more nodes to the cluster (two worker nodes are created by
default). Modify vcn.tf to add worker subnets:
resource "oci_core_subnet" "oke-subnet-worker-3" {
availability_domain = data.oci_identity_availability_domains.ADs.availability_domains[2]["name"]
cidr_block = "${var.vcn_cidr_prefix}.12.0/24"
display_name = "${var.cluster_name}-WorkerSubnet03"
dns_label = "workers03"
compartment_id = var.compartment_ocid
vcn_id = oci_core_virtual_network.oke-vcn.id
security_list_ids = [oci_core_security_list.oke-worker-security-list.id]
route_table_id = oci_core_virtual_network.oke-vcn.default_route_table_id
dhcp_options_id = oci_core_virtual_network.oke-vcn.default_dhcp_options_id
}
Add the corresponding egress_security_rules and ingress_security_rules for the worker
subnets:
egress_security_rules {
destination = "${var.vcn_cidr_prefix}.12.0/24"
protocol = "all"
stateless = true
}
ingress_security_rules {
stateless = true
protocol = "all"
source = "${var.vcn_cidr_prefix}.12.0/24"
}
Modify subnet_ids in node-pool.tf to add the new worker subnets to the
pool.
subnet_ids = [oci_core_subnet.oke-subnet-worker-1.id, oci_core_subnet.oke-subnet-worker-2.id,
oci_core_subnet.oke-subnet-worker-3.id]
Run the script using the following command:
$ kubernetes/oke/samples/opentofu/oke.create.sh oci.props
The script reads the values from oci.props and performs the following
steps:
- Creates a new tfvars file based on the values from the provided
oci.propsfile. - Downloads and installs the required binaries for OpenTofu and the OpenTofu OCI Provider, based on the operating system (macOS or Linux).
- Applies the configuration, creates an OKE cluster using OpenTofu, and generates a
kubeconfig file based on the
okeclusternameproperty defined inoci.props.
If the configuration contains errors, oke.create.sh displays output such as the following:
If you ever set or change modules or backend configuration for tofu,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
\u2577
\u2502 Error: Reference to undeclared resource
\u2502
\u2502 on node-pool.tf line 12, in resource "oci_containerengine_node_pool" "fmw_node_pool":
\u2502 12: subnet_ids = [oci_core_subnet.oke-subnet-worker-1.id, oci_core_subnet.oke-subnet-worker-2.id, oci_core_subnet.oke-subnet-worker-3.id, oci_core_subnet.oke-subnet-worker-4.id, oci_core_subnet.oke-subnet-worker-5.id]
\u2502
\u2502 A managed resource "oci_core_subnet" "oke-subnet-worker-5" has not been declared in the root module.
\u2575
\u2577
\u2502 Error: Reference to undeclared resource
\u2502
\u2502 on node-pool.tf line 12, in resource "oci_containerengine_node_pool" "fmw_node_pool":
\u2502 12: subnet_ids = [oci_core_subnet.oke-subnet-worker-1.id, oci_core_subnet.oke-subnet-worker-2.id, oci_core_subnet.oke-subnet-worker-3.id, oci_core_subnet.oke-subnet-worker-4.id, oci_core_subnet.oke-subnet-worker-5.id]
\u2502
\u2502 A managed resource "oci_core_subnet" "oke-subnet-worker-5" has not been declared in the root module.
\u2575
If the cluster is created successfully, output such as the following is displayed:
cluster_id = "ocid1.cluster.oc1.phx.axxxxxxx.....jlxa"
fmw1_export_path = "/fmw1"
fmw1_fs_ocid = "ocid1.filesystem.oc1.phx.aaaxxxxxx....fuzaaaaa"
fmw2_export_path = "/fmw2"
fmw2_fs_ocid = "ocid1.filesystem.oc1.phx.aaaaxxxxx....xxxxxxaaaa"
fmw_mount_target_ip = "10.1.11.43"
Confirm access to cluster...
NotReady
echo '[ERROR] Some Nodes in the Cluster are not in the Ready Status , sleep 10s more ...
Status is Ready Iter [1/100]
- able to access cluster
okecluster6 cluster is up and running
To access the cluster, set the KUBECONFIG environment variable or use the
--kubeconfig option with the kubeconfig file generated from the okeclustername
property in oci.props, for example, okeclustername6_kubeconfig.
$ kubectl get nodes --kubeconfig=okecluster6_kubeconfig
NAME STATUS ROLES AGE VERSION
10.1.10.2 Ready node 2d11h v1.29.1
10.1.11.3 Ready node 2d11h v1.29.1
To add nodes after the cluster is created, update vcn.tf and
node-pool.tf, then run the following commands:
${tofu.installdir}/tofu plan -var-file=<tfvars.filename>
${tofu.installdir}/tofu apply -var-file=<tfvars.filename>
To delete the cluster, run oke.delete.sh. The script reads oci.props from the
current directory and deletes the cluster.
$ kubernetes/oke/samples/opentofu/oke.delete.sh oci.props
Deploy SOA on an OKE cluster using Helmfile
See OracleSOASuite/helm-charts/README.md for detailed instructions.
OCI NFS volume static provisioning
For SOA, reference the file system directly in the domain.storage section of values.yaml.
For example:
storage:
capacity: 10Gi
reclaimPolicy: Retain
type: nfs
path: /fmw1 #Export Path
nfs:
server: 10.0.10.156 #Mount Target IP Address
For the database, the following example defines a persistent volume (PV) using a file system (FS) and a mount target IP address:
apiVersion: v1
kind: PersistentVolume
metadata:
name: fmw2-pv-db
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
csi:
driver: fss.csi.oraclecloud.com
volumeHandle: "<OCID of the file system>:<Mount Target IP Address>:/<Export Path>"
Update the oracledb.persistence section of values.yaml to use the created PV
as a volume. For example:
persistence:
storageClass: ""
size: 10Gi
accessMode: "ReadWriteOnce"
volumeName: "fmw2-pv-db"
Note:
- An example
volumeHandlefor the preceding configuration:volumeHandle:"ocid1.filesystem.oc1.phx.aaaaanoxxxxx....aaaa:10.0.10.156:/fmw2" - Obtain the
volumeHandledetails from the console output after the cluster is created successfully. - When a mount target is provisioned in OCI, its Reported Size (GiB) values are very large. These values appear on the mount target page in the OCI Console. Applications can fail to install if a space requirements check reports too much available disk space. In the OCI Console, click the pencil icon beside the mount target's Reported Size parameter to specify, in gigabytes (GiB), the maximum capacity reported by file systems exported through this mount target. This setting does not limit the actual amount of data you can store. See Changing a File System's Reported Size.