Configure the Terraform Modules

The resources necessary for this solution are defined in Terraform modules.

Before You Begin

Before you start configuring the Terraform modules, complete the following steps:

  1. Learn the basics of Terraform.

    At a minimum, read the introduction in the Terrraform documentation.

  2. Keep the following information ready:
    • The OCID of your tenancy.

      You can find your tenancy's OCID in the Oracle Cloud Infrastructure web console. Select Administration from the services menu, and then click Tenancy Details.

    • The OCID of the user that you want Terraform to use to authenticate with Oracle Cloud Infrastructure.

      To find the user's OCID, select Identity from the services menu, and then select Users. Locate your user name in the list, and copy its OCID.

    • The OCID of the compartment in which you want to create the resources.

      To find a compartment's OCID, select Identity from the services menu, and then select Compartments. Locate the compartment that you need in the list, and copy its OCID.

    • The ID of region where you want to create the resources.

      For example, the ID of the US East (Ashburn) region is us-ashburn-1.

      See Regions and Availability Domains.

  3. Decide the following:
    • The OCIDs of the images that you want to use for the bastion and admin hosts.
      The default image defined in the Terraform configuration for the bastion host is an Oracle Autonomous Linux image. If you want to use a different image, then identify the OCID of the image you need.
      • To find the OCID of a custom image, sign in to the Oracle Cloud Infrastructure web console, select Compute from the service menu, and then select Custom Images.
      • To find the OCID of an Oracle-provided image, complete the following steps:
        1. Go to Oracle Cloud Images.
        2. In the navigation pane on the left, select an image family (for example, Oracle Linux 7.x).
        3. On the resulting page, scroll down to the image version that you want to use, and click it (for example, Oracle-Linux-7.7-2019.09.25-0).
        4. On the resulting page, scroll down to the Image OCIDs section.
        5. Copy the OCID corresponding to the region where you want to create the bastion host.

          The image OCID contains the ID of the region that the image can be used in. For example, the OCIDs of the images in the Germany Central (Frankfurt) region would be in the format ocid1.image.oc1.eu-frankfurt-1.aaaaaaaaxxx… Be sure to copy the OCID of the image for the region where you want to create your resources.

    • The time zone for the bastion and admin hosts.

      On UNIX-like systems, you can get a list of the time zones by running the command: timedatectl list-timezones

    • The compute shape to be used for the bastion host, the admin host, and the Kubernetes worker nodes.

      See Compute Shapes.

  4. Complete the prerequisites for creating Kubernetes clusters in Oracle Cloud Infrastructure. See Preparing for Container Engine for Kubernetes.
  5. (Optional) This step is required if you intend to pull images of containerized applications from a private repository in Oracle Cloud Infrastructure Registry.
    1. Generate an authentication token for the user name that should be used to pull images from Oracle Cloud Infrastructure Registry. See Getting an Auth Token.
    2. Store the authentication token that you generated as a secret in Oracle Cloud Infrastructure Vault. See Managing Secrets.

Download the Terraform Code

The Terraform code for this solution is available on GitHub.

  1. In the navigation pane on the left, click Download Code.
  2. Click Git Repo.
  3. Clone or download the repository to your local computer.

About the Terraform Code

The Terraform code for this solution is organized into reusable modules, each containing the resources for a specific component of the target topology.

Coding your cloud resources in Terraform configuration files enables you to provision the topology quickly and manage the resources efficiently.

The Terraform code contains the following directories and files at the top level:
  • docs directory and *.adoc: Documentation for the code. All the information and instructions you need is included in the documentation that you're reading now. You won't need to refer to the documentation that's included in the code.
  • *.tf: The Terraform configuration files that the solution uses. Do not edit these files.
  • terraform.tfvars.example: A template that you'll use to create the Terraform variables file. Don't edit or remove the template. Copy it to terraform.tfvars
  • modules: Directories containing the core Terraform configurations for the resources that you create by using this solution. Do not edit them.
  • .github directory and .gitignore: Internal Github configuration files. Do not edit them.

Set the Terraform Variables

Specify the parameters required for Terraform to connect to the Oracle Cloud Infrastructure tenancy. Also specify the networking parameters, attributes of the bastion host and admin host, and the Kubernetes settings.

  1. In the top-level directory of the code that you downloaded or cloned, create a plain-text file named provider.tf, containing the following code:
    provider "oci" {
      tenancy_ocid         = var.tenancy_id
      user_ocid            = var.user_id
      fingerprint          = var.api_fingerprint
      private_key_path     = var.api_private_key_path
      region               = var.region
      disable_auto_retries = var.disable_auto_retries
    }
  2. Locate the terraform.tfvars.example file in the top-level directory of the code that you downloaded or cloned, and copy it to terraform.tfvars

    Note:

    To manage resources in multiple tenancies, maintain a separate terraform.tfvars file for each tenancy.
  3. Make sure that you've completed the prerequisites described earlier. See Before You Begin.
  4. Open terraform.tfvars in a plain-text editor, and set values for the variables in it as follows:
    Variable Description
    api_fingerprint (required) The fingerprint of the API signing key that you uploaded.
    api_private_key_path (required) The full path and name of the file that contains your private API signing key.
    compartment_id (required) The OCID of the compartment in which you want to create the resources.
    tenancy_id (required) The OCID of your tenancy.
    user_id (required) The OCID of the user that you want Terraform to use to authenticate with Oracle Cloud Infrastructure.
    ssh_private_key_path The full path and name of the file that contains the private SSH key corresponding to the public key that you want to provide for the bastion host.

    This value is used to construct the ssh command that you can use to access the bastion host. The ssh command is displayed in the output when you apply the Terraform configuration. Note that Terraform does not read or copy the private key.

    ssh_public_key_path The full path and name of the file that contains the public SSH key that you want to provide for the bastion host.
    label_prefix A short identifier, which you want to be used as a prefix in the names of the resources.

    Use a string that will help you identify the purpose or nature of the resources by looking at their names. For example, if you intend to use the Terraform configuration to set up a test or staging environment, then consider using the prefix test or staging.

    region The ID of region where you want to create the resources.

    For example, the ID of the US East (Ashburn) region is us-ashburn-1.

    nat_gateway_enabled Specify true to create a NAT gateway for the VCN.

    A NAT gateway is required if any of the private compute instances (such as the admin host or the Kubernetes worker nodes) need to access hosts on the public internet.

    newbits and netnum When you apply the configuration, Terraform passes the values of newbits and netnum as arguments to the Terraform function cidrsubnet(). This function calculates the CIDR prefixes of the subnets for the bastion host, admin host, load balancer nodes, and Kubernetes worker nodes.
    • newbits is used to determine the size of the subnet. It is the difference between the netmask of the VCN and the netmask that you need for the bastion subnet.

      For example, to create a subnet with the netmask /29 in a /16 VCN, specify 13 as the value of newbits (that is, 29 minus 16).

      A lower newbits value results in a subnet with a larger address space.

    • netnum is used to determine the boundaries of the subnet. It is the zero-based index of the subnet when the network is masked with newbits.

      Continuing with the previous example, if you specify newbits=13 and netnum=0, then the cidrsubnet() function returns the subnet CIDR prefix 10.0.0.0/29, which is the first /29 address space within the 10.0.0.0/16 VCN.

    Default values:
    netnum = {
      admin   = 33
      bastion = 32
      int_lb  = 16
      pub_lb  = 17
      workers = 1
    }
    
    newbits = {
      admin   = 13
      bastion = 13
      lb      = 11
      workers = 2
    }
    If you leave these variables at the default values and specify 10.0.0.0/16 as the CIDR range for the VCN, then the Terraform function cidrsubnet() calculates the following CIDR prefixes for the subnets. The available addresses are shown in parentheses. Note that the first two addresses and the last address in a subnet are reserved by the networking service.
    • Bastion subnet: 10.0.1.0/29 (available addresses: 10.0.1.2 to 10.0.1.6; that is, 5 hosts)
    • Admin subnet: 10.0.1.8/29 (10.0.1.10 to 10.0.1.14; 5 hosts)
    • Internal load balancer subnet: 10.0.2.0/27 (10.0.2.2 to 10.0.2.30; 29 nodes)
    • Public load balancer subnet: 10.0.2.32/27 (10.0.2.34 to 10.0.2.62; 29 nodes)
    • Kubernetes worker nodes subnet: 10.0.64.0/18 (10.0.64.2 to 10.0.127.254; 16381 nodes)

    If you need subnets that have different addresses or sizes than the default settings, then you should determine the appropriate values for newbits and netnum. To do this, you must have basic knowledge about classless IP addresses. Also see the Terraform documentation for the cidrsubnet() function.

    Make sure that the CIDR blocks that you specify here don't overlap with the CIDR block that you specify for the Kubernetes pods (pods_cidr).

    service_gateway_enabled Specify true to create a service gateway for the VCN.

    A service gateway is required if the compute instances in the VCN need to access other Oracle services such as Oracle Cloud Infrastructure Object Storage.

    vcn_cidr An IPv4 CIDR block of your choice for the VCN.

    The default is 10.0.0.0/16. The allowed range is /16 to /30

    Make sure that the CIDR block that you specify here doesn't overlap with the CIDR block that you specify for the Kubernetes services (services_cidr).

    vcn_dns_label The name prefix for the internal DNS name of the VCN.

    The name that you specify here is prefixed to oraclevcn.com to form the DNS domain name of the VCN. For example, if you specify oke as the prefix, then the VCN's DNS domain name would be oke.oraclevcn.com

    vcn_name The name of the VCN resource.
    bastion_access The range of IP addresses (in CIDR notation) from which SSH access to the bastion must be allowed.

    To allow SSH access from any host (that is, 0.0.0.0/0), leave the variable at its default value, ANYWHERE.

    bastion_enabled Specify true to create a bastion host.
    bastion_image_id The OCID of the image to be used to create the bastion host.

    If you leave this variable at the default value, NONE, then an Oracle Autonomous Linux image is used.

    bastion_notification_enabled You can use the Oracle Cloud Infrastructure Notification service to receive status messages from the bastion host when updates are applied or when a known exploit attempt has been detected by Oracle Ksplice.
    Specify true to enable sending notifications for the bastion host.

    Note:

    The Terraform code in this solution configures notifications for the bastion host only when you use the default Oracle Autonomous Linux image.
    bastion_notification_endpoint The email address to which the notifications should be sent. This variable is required if you set bastion_notification_enabled to true.
    bastion_notification_protocol Set this variable to EMAIL.
    bastion_notification_topic A name for the notification topic to be created. This variable is required if you set bastion_notification_enabled to true.
    bastion_package_upgrade Specify true if you want the security packages of the bastion host to be upgraded the first time the host boots.

    Note that when this variable is set to true, after the bastion host is provisioned, it will be unavailable for a short period while the security packages are upgraded. But enabling this upgrade minimizes the vulnerabilities of the bastion host.

    bastion_shape The compute shape that you want to use for the bastion host.
    bastion_timezone The time zone to configure for the bastion host, in the IANA time zone format (for example, America/Los_Angeles).
    admin_enabled Specify true to create a admin host.
    admin_image_id The OCID of the image to be used to create the bastion host.

    If you leave this variable at the default value, NONE, then an Oracle-provided Linux image is used.

    admin_instance_principal Specify true if you want to enable the admin host to manage all the resources in the compartment that you specify.
    Use this feature if you intend to run CLI commands or make API calls from the admin host to manage resources in the topology.

    Note:

    Any user who can connect to a compute instance using SSH inherits the instance-principal privileges granted to the instance. Consider this when deciding whether to designate the admin host as an instance principal. You can turn this feature off or on at any time without any impact on the admin host.

    If you set this variable to true, then the admin host is made a member of a dynamic group, and a policy statement is created to allow the dynamic group to manage all the resources in the compartment.

    admin_notification_enabled

    admin_notification_endpoint

    admin_notification_protocol

    admin_notification_topic

    Leave these variables at the default values. Enabling notifications for the admin host is not supported currently in this Terraform code.
    admin_package_upgrade Specify true if you want the security packages of the admin host to be upgraded the first time the host boots.

    Note that when this variable is set to true, after the admin host is provisioned, it will be unavailable for a short period while the security packages are upgraded. But enabling this upgrade minimizes the vulnerabilities of the admin host.

    admin_shape The compute shape that you want to use for the admin host.
    admin_timezone The time zone to configure for the admin host, in the IANA time zone format (for example, America/Los_Angeles).
    availability_domains The availability domain where you want to provision the admin and bastion hosts.

    For example, to provision the bastion host in the second availability domain, set bastion = 2.

    If the region that you specified contains only one availability domain, then leave this variable at its default value, 1.

    tagging Specify the tags that you want to assign to the compute and networking resources.
    allow_node_port_access Specify true if you want to allow TCP traffic to the Kubernetes worker nodes when they are deployed in the public mode.
    allow_worker_ssh_access Specify true if you want to allow SSH connections to the Kubernetes worker nodes through the bastion host.

    Note that even if the worker nodes are deployed in the public mode, SSH connections must go through the bastion host.

    If you set this variable to true, then you must also set bastion_enabled = true.

    cluster_name A name for the Kubernetes cluster.
    dashboard_enabled Specify true if you want the default Kubernetes dashboard to be created.
    kubernetes_version The version of Kubernetes to be used for the worker nodes.

    If you leave this variable at its default value, LATEST, then the latest supported version is selected. To use a specific version, specify that version.

    node_pools The number of node pools to be created, the size of each pool, and the compute shape to be used for the worker nodes, in the following format:
    node_pools = {
      "np1" = ["computeShape", numberOfNodes]
      "np2" = ["computeShape", numberOfNodes]
      "np3" = ["computeShape", numberOfNodes]
      ...
    }
    • np1, np2, and np3 are arbitrary names representing individual node pools.
    • computeShape is the compute shape to be used for the worker nodes in the pool.
    • numberOfNodes is the number of Kubernetes worker nodes to be created in the pool. A minimum of three nodes are created in each pool, even if you specify a lower value.
    The following example is for a cluster consisting of two pools, each using a different compute shape and containing a different number of Kubernetes worker nodes:
    node_pools = {
      "np1" = ["VM.Standard2.1", 3]
      "np2" = ["VM.Standard2.2", 5]
    }
    node_pool_name_prefix The name prefix for the node pools.

    The names of the node pools are generated by concatenating the values of label_prefix, node_pool_name_prefix, and the node pool number. For example, if you specify label_prefix = "prod" and node_pool_name_prefix = "np", then the generated names of the node pools would be prod-np-1, prod-np-2, prod-np-3, and so on.

    node_pool_image_id The OCID of the image to be used for the Kubernetes worker nodes.

    If you leave this variable at the default value, NONE, then an image that matches the values you specify for node_pool_os and node_pool_os_version is used.

    node_pool_os The operating system that should be used for the Kubernetes worker nodes (for example, "Oracle Linux").

    This setting is considered only if you set node_pool_image_id = "NONE"

    node_pool_os_version The version of the operating system that should be used for the Kubernetes worker nodes (for example, "7.7").

    This setting is considered only if you set node_pool_image_id = "NONE"

    pods_cidr An IPv4 CIDR block of your choice for the Kubernetes pods.

    Make sure that the CIDR block that you specify here doesn't overlap with the CIDR block that you specify for the VCN (vcn_cidr).

    services_cidr An IPv4 CIDR block of your choice for the Kubernetes pods.

    Make sure that the CIDR block that you specify here doesn't overlap with the CIDR block that you specify for the VCN (vcn_cidr).

    worker_mode Specify public if the worker nodes must be accessible from the public internet. Otherwise, set this variable to private.

    If you set worker_mode = "private", then set nat_gateway_enabled = true

    lb_subnet_type and preferred_lb_subnets The values that you specify for lb_subnet_type and preferred_lb_subnets determine the type of subnets that must be used for any load balancer nodes that you deploy by using the Kubernetes service of type LoadBalancer.

    Public load balancers have public IP addresses. Internal load balancers have only private IP addresses and are not accessible from the public internet.

    • If you intend to use public load balancers, then set preferred_lb_subnet = "public" and subnet_type to either "both" or "public"
    • If you intend to use internal load balancers, then set preferred_lb_subnet = "internal" and subnet_type to either "both" or "internal"

      Even if you set the load balancer subnets to be internal, you must set the appropriate annotations (such as service.beta.kubernetes.io/oci-load-balancer-internal: "true") when you create internal load balancer services. Merely setting the subnets to be private is not sufficient.

      For information about creating internal load balancers, see the Oracle Cloud Infrastructure documentation.

    secret_id The ID of the secret in the Oracle Cloud Infrastructure Vault service, where the authentication token to be used for pulling application images from Oracle Cloud Infrastructure Registry is stored.
    You must set the following as well:
    bastion_enabled = true
    admin_enabled = true
    admin_instance_principal = true
    email_address The email address to be used when generating the Docker secret. An email address is required, but it doesn't matter what you specify.

    This variable is required if you specify secret_id.

    tenancy_name The Oracle Cloud Infrastructure Object Storage namespace of the tenancy containing the registry from which images should be pulled for deployments to your Kubernetes cluster.

    This variable is required if you specify secret_id.

    username The user name for which you generated the authentication token stored in the secret_id.

    This variable is required if you specify secret_id.

    install_helm Specify true if you want Helm to be installed.

    Helm is a package manager for Kubernetes.

    To install Helm, you must set admin_instance_principal = true as well.

    helm_version The version of the Helm client to be installed.

    Tiller (the server-side counterpart of Helm) is upgraded automatically.

    install_calico Specify true if you want Calico to be installed.

    You can use Calico to implement network policies for containers workloads deployed to Kubernetes clusters.

    If you set install_calico = true, then you must also set the following:
    bastion_enabled = true
    admin_enabled = true
    admin_instance_principal = true
    calico_version The version of Calico to be installed.
    install_metricserver Specify true if you want Kubernetes Metrics Server to be installed.

    By default, the latest version is installed in the kube-system namespace. Kubernetes Metrics Server aggregates resource usage data across a cluster.

    If you set install_metricserver = true, then you must also set the following:
    bastion_enabled = true
    admin_enabled = true
    admin_instance_principal = true
    use_encryption If you want to use the Oracle Cloud Infrastructure Vault service to encrypt Kubernetes secrets, then set this variable to true.
    If you set use_encryption = true, then you must also set the following:
    bastion_enabled = true
    admin_enabled = true
    admin_instance_principal = true
    existing_key_id The OCID of an existing key created in the Oracle Cloud Infrastructure Vault service.

    This variable is required if you set use_encryption to true.

    create_service_account If you want external processes and tools (such as a CI/CD pipeline) to access the cluster, then set this variable to true. A service account is created with its own authentication token.
    If you set create_service_account = true, then you must also set the following:
    bastion_enabled = true
    admin_enabled = true
    admin_instance_principal = true
    service_account_name The name of the service account to be created.
    service_account_namespace The Kubernetes namespace in which the account should be created.
    service_account_cluster_role_binding The name of the cluster role binding for the service account.

    The following is an example of a completed terraform.tfvars file.

    # Identity and access parameters
    
    api_fingerprint = "d4:dc:...(truncated)"
    
    api_private_key_path = "/home/joe/.oci/oci_api_key.pem"
    
    compartment_id = "ocid1.compartment.oc1..aaaaaaaaxxx... (truncated)"
    
    tenancy_id = "ocid1.tenancy.oc1..aaaaaaaaxxx... (truncated)"
    
    user_id = "ocid1.user.oc1..aaaaaaaaxxx... (truncated)"
    
    ssh_private_key_path = "/home/joe/.ssh/id_rsa"
    
    ssh_public_key_path = "/home/joe/.ssh/id_rsa.pub"
    
    # general oci parameters
    label_prefix = "prod"
    
    region = "us-phoenix-1"
    
    # networking
    
    nat_gateway_enabled = true
    
    netnum = {
      admin   = 33
      bastion = 32
      int_lb  = 16
      pub_lb  = 17
      workers = 1
    }
    
    newbits = {
      admin   = 13
      bastion = 13
      lb      = 11
      workers = 2
    }
    
    service_gateway_enabled = true
    
    vcn_cidr = "10.0.0.0/16"
    
    vcn_dns_label = "oke"
    
    vcn_name = "oke vcn"
    
    # bastion
    
    bastion_access = "ANYWHERE"
    
    bastion_enabled = true
    
    bastion_image_id = "NONE"
    
    bastion_notification_enabled = true
    
    bastion_notification_endpoint = "joe@example.com"
    
    bastion_notification_protocol = "EMAIL"
    
    bastion_notification_topic = "bastion_server_notification"
    
    bastion_package_upgrade = true
    
    bastion_shape = "VM.Standard.E2.1"
    
    bastion_timezone = "America/Los_Angeles"
    
    admin_enabled = true
    
    admin_image_id = "NONE"
    
    admin_instance_principal = true
    
    admin_notification_enabled = false
    
    admin_notification_endpoint = "joe@example.com"
    
    admin_notification_protocol = "EMAIL"
    
    admin_notification_topic = "admin_server_notification"
    
    admin_package_upgrade = true
    
    admin_shape = "VM.Standard.E2.1"
    
    admin_timezone = "America/Los_Angeles"
    
    # availability_domains
    
    availability_domains = {
      bastion = 1
      admin   = 1
    }
    
    tagging = {
      computetag = {"Environment" = "dev" }
      networktag = { "Name" = "network" }
    }
    
    # oke
    
    allow_node_port_access = false
    
    allow_worker_ssh_access = false
    
    cluster_name = "oke"
    
    dashboard_enabled = true
    
    kubernetes_version = "LATEST"
    
    node_pools = {
      np1 = ["VM.Standard2.1", 3]
      #np2 = ["VM.Standard2.8", 4]
      #np3 = ["VM.Standard1.4", 5]
    }
    
    node_pool_name_prefix = "np"
    
    node_pool_image_id = "NONE"
    
    node_pool_os = "Oracle Linux"
    
    node_pool_os_version = "7.7"
    
    pods_cidr = "10.244.0.0/16"
    
    services_cidr = "10.96.0.0/16"
    
    worker_mode = "private"
    
    # oke load balancers
    
    lb_subnet_type = "public"
    
    preferred_lb_subnets = "public"
    
    # ocir
    
    secret_ocid = "ocid1.key.oc1..aaaaaaaaxxx... (truncated)"
    
    email_address = "joe@example.com"
    
    tenancy_name = "mytenancy"
    
    username = "joe_k8s_admin"
    
    # helm
    
    helm_version = "3.0.0"
    
    install_helm = false
    
    # calico
    
    calico_version = "3.9"
    
    install_calico = false
    
    # metrics server
    
    install_metricserver = false
    
    use_encryption = false
    
    existing_key_id = ""
    
    # service accountcreate_service_account = true
    service_account_name = "kubeconfigsa"
    service_account_namespace = "kube-system"
    service_account_cluster_role_binding = "myapps-manage-binding"
  5. Save and close terraform.tfvars.