Terraform: Create a Kubernetes Cluster

In this tutorial, you use Terraform to set up a Kubernetes cluster in your Oracle Cloud Infrastructure account.

Key tasks include how to:

Copy your existing scripts from the Terraform Get Started tutorials.
Edit existing scripts for reuse.
Write new scripts for a Kubernetes cluster.

A diagram of the components needed to create a Kubernetes cluster with Terraform. From a local Linux environment, the user connects to an Oracle Cloud Infrastructure account and creates a cluster. The cluster creation includes creating network resources. The network resources include one private regional subnet for worker nodes. These nodes are compute nodes. There is also a public regional subnet created to use for load balancers. Each regional subnet spans three availability domains: AD1, AD2, and AD3. The public subnet is connected to the internet with a two-way arrow. The private subnet has two one-directional, outgoing arrows, one to the internet and the other to Oracle services network.

For additional information, see:

Before You Begin

To successfully perform this tutorial, you must have the following:

Requirements

A paid Oracle Cloud Infrastructure account. See Signing Up for Oracle Cloud Infrastructure.
A MacOS, Linux, or Windows computer.
Terraform tutorial resources:
- Go through all the steps in:
  - Terraform Tutorial: Set Up a Simple Infrastructure with OCI Terraform
- Keep the scripts you created in the following directory:
  - $HOME/tf-simple-infrastructure/

1. Gather Required Information

Gather information for the compute instances in the node pool.

Get Node Shape

This tutorial uses VM.Standard2.1 for the compute instances in the node pool..

Save the <node-shape>, VM.Standard2.1 in your notepad.
Go to VM Standard Shapes to learn more about the shape.

Get Image ID

In the Console's top navigation bar, find your region.
Go to Oracle Provided Images.
Click Oracle Linux 7.x.
Click the latest Oracle Linux 7.x-<date>. Don't select any images labeled Gen2-GPU.
Copy the Image OCID for your region.
Save the <image-ocid> in your notepad.

Note

Ensure that you select a commercial OCID without gov in its OCID.

2. Copy Existing Scripts

Copy the Terraform: Set Up a Simple Infrastructure tutorial scripts. Then, remove the scripts and outputs related to the compute instance. In the next section, you declare a node pool with compute instances.

Copy the Scripts

In your $HOME directory, create a directory called tf-cluster and change to that directory.
```
mkdir tf-cluster
```
```
cd tf-cluster
```
Copy all the files ending in .tf from the tf-simple-infrastructure directory.
```
cp ../tf-simple-infrastructure/*.tf .
```
Confirm that you have the following files in your directory.
```
ls
```
```
availability-domains.tf
compartment.tf
compute.tf
outputs.tf
private-security-list.tf
private-subnet.tf
provider.tf
public-security-list.tf
public-subnet.tf
vcn-module.tf
```
Note

Do not copy the terraform.tfstate and terraform.tfstate.backup files. They contain the state of the resources in the directory that they are located. You will get a new state file, after you run the scripts in your new directory.

Remove Irrelevant Scripts

Remove the compute.tf file from the tf-cluster directory.
```
rm compute.tf
```

In the outputs.tf file, remove all the outputs for the compute instance.



# Outputs for compute instance

output "public-ip-for-compute-instance" {
  value = oci_core_instance.ubuntu_instance.public_ip
}
output "instance-name" {
  value = oci_core_instance.ubuntu_instance.display_name
}

output "instance-OCID" {
  value = oci_core_instance.ubuntu_instance.id
}

output "instance-region" {
  value = oci_core_instance.ubuntu_instance.region
}

output "instance-shape" {
  value = oci_core_instance.ubuntu_instance.shape
}

output "instance-state" {
  value = oci_core_instance.ubuntu_instance.state
}

output "instance-OCPUs" {
  value = oci_core_instance.ubuntu_instance.shape_config[0].ocpus
}

output "instance-memory-in-GBs" {
  value = oci_core_instance.ubuntu_instance.shape_config[0].memory_in_gbs
}

output "time-created" {
  value = oci_core_instance.ubuntu_instance.time_created
}

3. Create Scripts

Create three scripts: one for a cluster, one for a node pool and one to print outputs.

Declare a Cluster

Create a file called cluster.tf.

Add the following code to cluster.tf.

Replace <your-cluster-name> with a name of your choice. Example: tf-cluster.

# Source from https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/containerengine_cluster

resource "oci_containerengine_cluster" "oke-cluster" {
    # Required
    compartment_id = oci_identity_compartment.tf-compartment.id
    kubernetes_version = "v1.21.5"
    name = "<your-cluster-name>"
    vcn_id = module.vcn.vcn_id

    # Optional
    options {
        add_ons{
            is_kubernetes_dashboard_enabled = false
            is_tiller_enabled = false
        }
        kubernetes_network_config {
            pods_cidr = "10.244.0.0/16"
            services_cidr = "10.96.0.0/16"
        }
        service_lb_subnet_ids = [oci_core_subnet.vcn-public-subnet.id]
    }  
}

Save the cluster.tf file.

Explanation

Go to Oracle Cloud Infrastructure Provider.
In the left navigation Filter, enter container engine.
Under Container Engine, go to Resources and click oci_containerengine_cluster.
Find the Resource Type from the title of the page:
- Type: oci_containerengine_cluster
In the Argument Reference section, find all arguments (inputs) labeled as (Required)
- compartment_id
- kubernetes_version
- name
- vcn_id
Construct a resource block:
- Declare a resource block with the keyword: resource
- Add a label for resource type: "oci_containerengine_cluster"
- Add a label for a local name (your choice):
  - The label can contain letters, digits, underscores (_), and hyphens (-). The first character must not be a digit.
  - Example: "oke-cluster"
- Inside the code block, provide a value for the required arguments. They don't have a default value:
  - compartment_id: Point to the compartment declared in compartment.tf: oci_identity_compartment.tf-compartment.id
  - kubernetes_version: This tutorial uses version v1.21.5. You can check the Quick Create option in the Console for the latest version.
  - name: Assign a name of your choice.
  - vcn_id: Point to the compartment declared in vcn-module.tf: module.vcn.vcn_id
- Provide values for the following optional arguments to override their default values.
  - add_ons: Assign true or false for the following arguments:
    - is_kubernetes_dashboard_enabled
    - is_tiller_enabled
  - kubernetes_network_config: Assign a CIDR block as a string for the following arguments:
    - pods_cidr
    - services_cidr
    Note
    
    The CIDR block for the pods must not overlap with the worker node and load balancer subnet CIDR blocks.
    
    The CIDR block for the Kubernetes service must not overlap with the VCN CIDR block.
    
    The example code in this tutorial uses the same CIDR blocks as the Quick Create option in the Console.
    
    For more explanation, see CIDR Blocks and Container Engine for Kubernetes.
  - service_lb_subnet_ids: Assign the public subnet you declared in public-subnet.tf
    Note
    
    The argument, service_lb_subnet_ids accepts a list of subnet ids:
    - Even if you have one subnet, use square brackets to denote a list.
    - Example: [oci_core_subnet.vcn-public-subnet.id]

Declare a Node Pool

Create a file called node-pool.tf.

Add the following code to node-pool.tf.

Replace the following fields with the information you gathered in section one:
- <node-shape> with VM.Standard2.1
- <image-ocid>
Replace the following field with the name you chose when you declared a cluster:
- <your-cluster-name>

# Source from https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/containerengine_node_pool

resource "oci_containerengine_node_pool" "oke-node-pool" {
    # Required
    cluster_id = oci_containerengine_cluster.oke-cluster.id
    compartment_id = oci_identity_compartment.tf-compartment.id
    kubernetes_version = "v1.21.5"
    name = "pool1"
    node_config_details{
        placement_configs{
            availability_domain = data.oci_identity_availability_domains.ads.availability_domains[0].name
            subnet_id = oci_core_subnet.vcn-private-subnet.id
        } 
        placement_configs{
            availability_domain = data.oci_identity_availability_domains.ads.availability_domains[1].name
            subnet_id = oci_core_subnet.vcn-private-subnet.id
        }
         placement_configs{
            availability_domain = data.oci_identity_availability_domains.ads.availability_domains[2].name
            subnet_id = oci_core_subnet.vcn-private-subnet.id
        }
        size = 3
    }
    node_shape = "<node-shape>"

    # Using image Oracle-Linux-7.x-<date>
    # Find image OCID for your region from https://docs.oracle.com/iaas/images/ 
    node_source_details {
         image_id = "<image-ocid>"
         source_type = "image"
    }
 
    # Optional
    initial_node_labels {
        key = "name"
        value = "<your-cluster-name>"
    }    
}

Save the node-pool.tf file.

Explanation

Go to Oracle Cloud Infrastructure Provider.
In the left navigation Filter, enter container engine.
Under Container Engine, go to Resources and click oci_containerengine_node_pool.
Find the Resource Type from the title of the page:
- Type: oci_containerengine_node_pool
In the Argument Reference section, find all arguments (inputs) labeled as (Required)
- cluster_id
- compartment_id
- kubernetes_version
- name
- node_config_details
  - placement_configs
    - availability_domain
    - subnet_id
- node_shape
- node_source_details
  - image_id
  - source_type
Construct a resource block:
- Declare a resource block with the keyword: resource
- Add a label for resource type:
```
"oci_containerengine_node_pool"
```
- Add a label for a local name (your choice):
  - The label can contain letters, digits, underscores (_), and hyphens (-). The first character must not be a digit. Example:
```
"oke-node-pool"
```
- Inside the code block, provide a value for the required arguments:
  - cluster_id: Point to the cluster declared in cluster.tf:
```
oci_containerengine_cluster.oke-cluster.id
```
  - compartment_id Point to the compartment declared in compartment.tf:
```
oci_identity_compartment.tf-compartment.id
```
  - kubernetes_version: This tutorial uses the same version as the Create Cluster wizard in the Console.
  - name: Assign a name of your choice. The Create Cluster wizard, uses the name pool1.
  - node_shape: Enter information you gathered in section one.
  - node_source_details:
    - image_id: Enter information you gathered in section one.
    - source_type: Set to "image".
- Provide values for the following optional arguments to override their default values.
  - initial_node_labels: Assign key/value pairs for the nodes.
    - key: Assign a key of your choice. The Console's Quick Create option creates the key "name".
    - value: Assign a value for the key. The Console's Quick Create option assigns "<your-cluster-name>" to the "name" key.

Add Outputs

In this section, you declare outputs for the cluster and the node pool.

Add the following code to outputs.tf.

# Outputs for k8s cluster

output "cluster-name" {
  value = oci_containerengine_cluster.oke-cluster.name
}
output "cluster-OCID" {
  value = oci_containerengine_cluster.oke-cluster.id
}
output "cluster-kubernetes-version" {
  value = oci_containerengine_cluster.oke-cluster.kubernetes_version
}
output "cluster-state" {
  value = oci_containerengine_cluster.oke-cluster.state
}

# Outputs for k8s node pool

output "node-pool-name" {
  value = oci_containerengine_node_pool.oke-node-pool.name
}
output "node-pool-OCID" {
  value = oci_containerengine_node_pool.oke-node-pool.id
}
output "node-pool-kubernetes-version" {
  value = oci_containerengine_node_pool.oke-node-pool.kubernetes_version
}
output "node-size" {
  value = oci_containerengine_node_pool.oke-node-pool.node_config_details[0].size
}
output "node-shape" {
  value = oci_containerengine_node_pool.oke-node-pool.node_shape
}

Save the outputs.tf file.

Explanation

For cluster outputs, refer to the Attribute Reference of the Resource: oci_containerengine_cluster page.
For node pool outputs, refer to the Attribute Reference of the Resource: oci_containerengine_node_pool page.

Outputs for List Items

Usually list attributes are plural: (end in s).
List attribute example for Node Pool:
- node_config_details
To output all the attributes in a list, use the list attribute by itself, without any brackets.

Example:

output "node-configuration-details"{
  value = oci_containerengine_node_pool.oke-node-pool.node_config_details
}

Sample output:

node-configuration-details = [
  {
    "placement_configs" = [
      {
        "availability_domain" = "QnsC:US-ASHBURN-AD-1"
        "subnet_id" = "ocid1.subnet.xxx"
      },
      {
        "availability_domain" = "QnsC:US-ASHBURN-AD-2"
        "subnet_id" = "ocid1.subnet.xxx"
      },
      {
        "availability_domain" = "QnsC:US-ASHBURN-AD-3"
        "subnet_id" = "ocid1.subnet.xxx"
      },
    ]
    "size" = 3
  },
]

To output or call an item from a list:
- Use the following format:
  <list-attribute-name>[index].<attribute-from-list>
- Replace [index] with:
  - [0] for the first item.
  - [1] for the second item.
  - ...
  - [n] for the (n+1)^th item.

Example:

Value for the size attribute:

output "node-size" {
  value = oci_containerengine_node_pool.oke-node-pool.node_config_details[0].size
}

4. Run Scripts

Run your Terraform scripts to create a compartment, a virtual cloud network, a Kubernetes cluster, and a node pool.

Initialize

Initialize a working directory in the tf-cluster directory.

terraform init

Example output:

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/oci versions matching ">= 4.41.0"...
- Installing hashicorp/oci v4.59.0...
- Installed hashicorp/oci v4.59.0 (signed by HashiCorp)

Terraform has been successfully initialized!

Check the contents of the tf-cluster directory.
```
ls -a
```
You now have a folder called .terraform that includes the plugins for the oci provider.
Note

Troubleshooting:
- After running terraform init
- error message: Failed to query available provider packages:
  - If you are on a VPN, check your proxy settings.

Plan

Create an execution plan:
```
terraform plan
```

Review the changes that Terraform plans to make to your account.

Example output:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

Plan: 14 to add, 0 to change, 0 to destroy.

Apply

Run your Terraform scripts:
```
terraform apply
```
When prompted for confirmation, enter yes, for your resources to be created.

It might take 15 minutes or more for the cluster to be created. After Terraform creates the resources, review the output in the terminal.

Apply complete! Resources: 14 added, 0 changed, 0 destroyed.

Outputs:
...
cluster-OCID = ocid1.cluster.xxx
cluster-kubernetes-version = "v1.21.5"
cluster-name = <your-cluster-name>
cluster-state = ACTIVE
...
node-pool-OCID = ocid1.nodepool.xxx
node-pool-kubernetes-version = "v1.21.5"
node-pool-name = "pool1"
node-shape = "VM.Standard2.1"
node-size = 3
...

Troubleshooting

401 errors - (Service error:NotAuthenticated):
- You have an incorrect value for one of the following:
  - tenancy OCID
  - user OCID
  - fingerprint
  - RSA private key (the path or the key)
no such host:
- You have an incorrect value for the following:
  - region identifier

References:

What's Next

Congratulations! You have created a Kubernetes cluster using Terraform, in your Oracle Cloud Infrastructure account.

Now that you have a Kubernetes cluster, try the Kubernetes tutorials.

To explore more information about development with Oracle products, check out these sites:

Oracle Cloud Infrastructure Documentation

Terraform: Create a Kubernetes Cluster

Before You Begin

1. Gather Required Information

2. Copy Existing Scripts

3. Create Scripts

Outputs for List Items

4. Run Scripts

What's Next