Terraform in Complex Environments
Securing your state file and Terraform scripts
As you will have seen from the previous section, the Terraform state file is a critical component of your deployment, and losing or corrupting it means that you will no longer be able to manage your cloud with Terraform as there is currently no way to reverse engineer from a deployment to a set of Terraform scripts.
Clearly, storing your Terraform state files on your local PC, or even on a cloud server is not suitable for any live system, and so the first step is to use a backend that stores the state in a secure and resilient manner. Just using a “standard” backend (in Terraform-speak this means any backend that is not your local file system, nor Terraform’s Enterprise service) is not enough - you need to be sure that the backend is fully protected from data loss.
Object storage is a good option, or any database service which is regularly backed-up and version controlled. Oracle supports using OCI object storage as a Terraform back-end, and full details on how to set up and use can be found in Using Object Storage for State Files.
Managing your Terraform scripts
Your Terraform scripts are just as critical as your state files, as these also cannot be regenerated in any really practical way. These are somewhat easier to manage - they can be stored on the local file system and configure managed with any configuration management system such as “git”. We will cover this in a little more detail later.
If you are using a remote git (or similar) service, then great care must be taken to ensure that the git repository reflects the current state file. As there is no explicit link between the scripts and the state, it’s very easy to run an “apply” and forget to “commit” or to “push” with the result that if anyone else “pulls” the config, then they will pick up an old version of the terraform scripts - and these will be out of date when compared to the state file.
Just a reminder, if your Terraform state file is local, then it should never be put into git. Make sure you add it to your .gitignore file.
Multi-user configurations
The next step on from securing your Terraform scripts and state is to allow others to use them to manage the deployment. As we mentioned at the end of the previous section, Terraform does not attempt to manage multi-user environments - it’s up to you as a Terraform architect or user to put the necessary governance and control in place to ensure that scripts and state are managed across users.
In most cases, the process of changing a deployment will be embedded in a tool chain managed by a CI/CD tool and the governance of keeping the Terraform scripts, the git repository and the state file all exactly can be automated. We strongly recommend this approach for every environment except early development of the Terraform scripts where the flexibility of working directly with Terraform commands takes precedence.
In multi-user environments, securing the Terraform state against simultaneous update by different users becomes essential. Terraform implements a degree of locking for some backends, but this is limited to the duration of an apply command. In a true multi-user environment, what is potentially much more risky is the time gap between changing Terraform scripts, running the plan command to check those scripts and then running the apply command. If two users have made independent copies of the Terraform scripts and made incompatible changes, then the outcome of subsequent apply commands would certainly cause confusion at the very least.
So, while Terraform supports multi-user environments with shared back-ends, strong CI/CD and DevOps governance is still required. The governance of CI/CD is beyond the scope of this guide, however, in very complex environments, we would advise that a check-out/check-in process is implemented so that at any one time, only one user has permission to change the Terraform scripts for a specific state file.
Multi-environments
This brings us nicely to how Terraform is used to manage more than just a single deployment in a single environment. The vast majority of systems and applications have more than environment to support the application lifecycle. Regardless of whether the choice is a modern agile development process or a more traditional waterfall method, the vast majority of applications will have more than a single environment.
At this point we need to recognize that a simplistic use of Terraform really starts to hit a major barrier. If every resource that is deployed and managed by Terraform has “hard-coded” environment specifics (e.g. IP addresses, routing) then those scripts will not be usable across multiple environments.
Declaring variables in Terraform in a variables.tf file :
variable "compartment_ocid" {
description = "(Updatable) The OCID of the compartment where to create all resources"
type = string
}
variable "instance_timeout" {
description = "Timeout setting for creating instance."
type = string
default = "25m"
}
Fortunately, Terraform supports the use of variables that will allow us to remove hard-coded values from our Terraform scripts. These variables can then be held in a .tfvars file and so the same scripts can be used across multiple environments, and only the .tfvars file needs to be changed between environments. For exact copies of environments, this is a reasonable approach, but for a more flexible approach to re-use, we will cover modules later.
Flexible Environments
In a typical multi-environment application deployment, each environment fulfils a different set of end-user and business requirements. For example, across dev, integration test, system test, user acceptance test, pre-prod, prod and DR, systems will likely vastly differ in size, user governance, availability, resilience. Yet we would prefer them to be as similar as possible in core functional capability.
For example, while we may need a cluster of eight web servers in production, we may want only one in development and two in system test. Terraform can accommodate these requirements with the use of a set of conditional and iterative constructs. It is not the intent of this guide to be a Terraform tutorial (and there are many good books and tutorials out there) and so we will not cover how to write good Terraform.
However, we will mention that although these constructs look similar to procedural language iterators, loops and conditional statements, they are better thought of as a shorthand for describing a meta-data object. Terraform uses those constructs for building its internal graph data model - Terraform does not implement them as simple procedural statements.
At this point, we are well passed the point of a casual use of Terraform. A Terraform programmer will need both good training and some months of practical experience to get to the maturity of writing conditionals, iterators, for-loops and similar constructs. In most organisations, there are likely to be only a few dedicated Terraform people with this level of expertise.
Summary
So in this section we have covered some advanced Terraform features that allow some re-use of code across environments that are fundamentally similar. However, to manage this approach becomes tricky and error-prone very quickly. Although we are re-using code across environments, we are, in fact just cut-and-pasting code across multiple directories and manyually managing the .tfvars file - this is a recipe for disaster.
In the next section we will cover to Terraform modules - a much better solution for code re-use.