Using Object Storage for State Files

Store Terraform state files in Oracle Cloud Infrastructure Object Storage by configuring a backend.

When you update the terraform block within the Terraform configuration to define a backend type, run terraform init, and then run terraform apply, the generated state file is uploaded to Object Storage.

This page addresses the following backend approaches:

Note

Because variables and locals aren't accepted in the terraform block, you must hard-code backend configuration values.

Using an HTTP Backend

Using the HTTP backend type allows you to store state using a simple REST client. With the HTTP backend type, you can easily fetch, update, and purge state using the HTTP GET, POST, and DELETE methods.

To configure the HTTP backend to store your OCI Terraform state files, do the following:

Create a Pre-Authenticated Request

Creating a pre-authenticated request in Oracle Object Storage enables accessing a bucket or object in the OCI without needing to provide credentials. To do so, you must create a pre-authenticated request that has read/write permissions to the object store where you intend to save the Terraform state file. You can do so in any of three ways: by using the Console UI, by using the command line interface (CLI), or by using the REST APIs.

Note

A state file must exist in the bucket before you create the pre-authenticated request. This file can be an existing state file, or an empty file for the initial state.

For guidance, see Using Pre-Authenticated Requests.

Upload Existing State

If you have an existing state file, you can upload it using Curl to make an HTTP PUT request to the object store URL, as shown here:

curl -X PUT -H "Content-Type: text/plain" --data-binary "@path/to/local/tfstate" http://<prefix>/<my-access-uri>

Configure HTTP as a Terraform Backend

The HTTP backend type stores state using a simple REST client and allows you to easily fetch, update, and purge state using the HTTP GET, POST, and DELETE methods.

The access URI for addressing OCI Terraform configurations must be of the form : https://objectstorage.us-phoenix-1.oraclecloud.com/my-access-uri (where region and access URI are specific to you).

For more example configuration and state files that reference code, and a summary of configuration variables, see Standard Backends: HTTP.

Following is an example Terraform configuration. The region in the URL can be something other than the Phoenix region.

terraform {
  backend "http" {
    address = "https://objectstorage.us-phoenix-1.oraclecloud.com/<my-access-uri>"
    update_method = "PUT"
  }
}

Reinitialize Terraform

Finally, you must reinitialize Terraform and then run the apply command:

terraform init
terraform apply

After completing these steps, you are able to use Oracle Cloud Infrastructure as the backend for storing Terraform state files.

Using an S3-Compatible Backend

To configure the S3-compatible backend, complete the steps in the following sections.

For more information about S3 compatibility, see Amazon S3 Compatibility API Prerequisites. For information about permissions for Object Lifecycle Management, see Required IAM Policies.

Setting Up Access to Oracle Cloud Infrastructure

  1. Sign Up for Oracle Cloud Infrastructure and obtain a unique namespace.
  2. Any user of the Amazon S3 Compatibility API with Object Storage needs permission to work with the service. If you're not sure if you have permission, contact your administrator. For basic information about policies, see How Policies Work. For policies that enable use of Object Storage, see Common Policies and the Policy Reference.
  3. Use an existing or create a Customer Secret Key. A Customer Secret Key consists of an Access Key/Secret Key pair. See Working with Customer Secret Keys for details. To use or create the key pair:
    • To use an existing Customer Secret Key, you must already know the Secret Key. For security reasons, you can't retrieve a Secret Key after generation. To show or copy the Access Key, open the Profile menu and click User Settings. On the left side of the page, click Customer Secret Keys. Hover over the Access Key associated with the Name of a particular Customer Secret key, then click Copy.
    • To create a Customer Secret Key using the Console, see To create a Customer Secret key.
    • To create a Customer Secret Key using the Command Line Interface (CLI), see oci iam customer-secret-key create.

Configuring Authentication

  1. Set the location for the credentials file.

    The default location is ~/.aws/credentials. You can set an alternate location by using the S3 backend shared_credentials_file option.

    Caution

    Never set the access_key and the secret_key attributes in the same Terraform backend configuration. Storing these attributes in the same configuration creates a security risk.
  2. Configure the [default] entry in the credentials file with the appropriate Object Storage credentials.

    The file can contain any number of credential profiles. If you provide a different profile name, you must also update the backend profile option in the Terraform configuration file.

    Following is an example of Object Storage credentials:

    [default]
    aws_access_key_id=ae37c0b4ffc488c7d4e6b360a21312244330718f
    aws_secret_access_key=mSTdaWhlbWj3ty4JZXlm0NUZV52xlImWjayJLJ6OH9A=
    Note

    The key values provided in the example aren't valid. Valid aws_access_key_id and aws_secret_access_key are user-specific values generated using the previous steps.

Configuring the Terraform Backend

  1. Set the Object Storage endpoint value in the following format:
    https://<namespace>.compat.objectstorage.<region>.oraclecloud.com
  2. Add the appropriate block to the Terraform configuration.

    Following are blocks organized by Terraform version. Select the block that maches the version of the Terraform configuration.

    Terraform version 1.6.4 or later

    # (Terraform version >= 1.6.4)
    terraform {
      backend "s3" {
        bucket                    = "terraform-states"
        region                    = "us-phoenix-1"
        key                       = "tf.tfstate"
        skip_region_validation      = true
        skip_credentials_validation = true
        skip_requesting_account_id  = true
        use_path_style              = true
        skip_s3_checksum            = true
        skip_metadata_api_check = true
        endpoints = {
          s3 = "https://<namespace>.compat.objectstorage.<region>.oraclecloud.com"
        }
      }
    }

    Terraform versions before 1.64

    # (Terraform version < 1.6.4)
    terraform {
      backend "s3" {
        bucket   = "terraform-states"
        key      = "networking/terraform.tfstate"
        region   = "us-phoenix-1"
        endpoint = "https://<namespace>.compat.objectstorage.<region>.oraclecloud.com"
       skip_region_validation      = true
        skip_credentials_validation = true
        skip_metadata_api_check     = true
        force_path_style            = true
      }
    }
    Caution

    If the same bucket is used across many Terraform configurations, the key must be unique to avoid overwriting the state file. This example uses a single bucket (terraform-states) to store all Terraform state files, but uses a unique prefix for the object name based on the resource (networking).
  3. To finish the setup, run terraform init.

    If you already have an existing terraform.tfstate file, then Terraform prompts you to confirm that the current state file is the one to upload to the remote state.

    Note

    The S3 backend configuration can also be used for the terraform_remote_state data source to enable sharing state across Terraform projects. For more information, see Accessing Remote States.

Accessing Remote States

You can use the terraform_remote_state data source to access properties of objects in one Terraform configuration from another configuration.

For example, you might use one configuration to define compartments and another to define VCNs. If your resources were in the same Terraform configuration folder, you could refer to a compartment OCID from the VCN configuration by using something like this: module.iam_compartment_SANDBOX.compartment_id.

But assume that our definitions do not share a state file and we have a file structure similar to the following:

.
├── governance
│   ├── compartments.tf
│   ├── outputs.tf
│   ├── remote-backend.tf
│   └── variables.tf
├── networking
│   ├── outputs.tf
│   ├── remote-backend.tf
│   ├── remote-state-data_governance.tf
│   ├── variables.tf
│   └── vcns.tf
└── terraform-states_bucket_credentials

Both governance and networking configurations store their respective state files on an OCI Object Storage bucket using the remote-backend.tf and terraform-states_bucket_credentials files.

In this example, the compartments.tf file creates a compartment at the root level using the iam-compartment module from the Terraform Registry as follows:

module "iam_compartment_SANDBOX" {
  source = "oracle-terraform-modules/iam/oci//modules/iam-compartment"
  version = "2.0.0"
  tenancy_ocid = var.tenancy_ocid
  compartment_id = var.tenancy_ocid # define the parent compartment. Creation at tenancy root if omitted
  compartment_name = "SANDBOX"
  compartment_description = "Test and evaluate OCI features here"
  compartment_create = true # if false, a data source with a matching name is created instead
  enable_delete = true # if false, on `terraform destroy`, compartment is deleted from the terraform state but not from oci
}

Defining Outputs

The terraform_remote_state data source can access output values from another Terraform configuration using the latest state file with a remote backend. For your networking configuration to access the governance configuration and dynamically retrieve Terraform resources properties, you must define outputs for the governance Terraform configuration. Without a defined output, the values cannot be used from outside of its configuration.

Your governance/outputs.tf file would look like the following:

output "iam_compartment_SANDBOX" {
  description = "compartment ocid, parent ocid, name, description"
  value = {
    id = module.iam_compartment_SANDBOX.compartment_id
    parent_id = module.iam_compartment_SANDBOX.parent_compartment_id
    name = module.iam_compartment_SANDBOX.compartment_name
    description = module.iam_compartment_SANDBOX.compartment_description
  }
}

Referring to a Remote State

In this example, we are using the vcn module from Terraform Registry to define a new VCN. The networking configuration refers to the governance configuration to define the VCN's compartment OCID:

module "vcn_hub1iad" {
  source = "oracle-terraform-modules/vcn/oci"
  version = "2.2.0"

  # general oci parameters
  compartment_id = data.terraform_remote_state.governance.outputs.iam_compartment_SANDBOX["id"]
  tags = var.tags

  # vcn parameters
  create_drg = false
  internet_gateway_enabled = true
  lockdown_default_seclist = true
  nat_gateway_enabled = false
  service_gateway_enabled = false
  vcn_cidr = "10.0.0.0/16"
  vcn_dns_label = "hub1iad"
  vcn_name = "hub1"
}

But, for the compartment_id = data.terraform_remote_state.governance.outputs.iam_compartment_SANDBOX["id"] line to be correctly interpreted, you must define a data.terraform_remote_state object.

Defining the Remote State Data Source

After the following terraform_remote_state data source is added to the networking configuration, you can access the governance Terraform outputs from configurations within the networking folder:

data "terraform_remote_state" "governance" {
  backend = "s3"
  config = {
    bucket = "terraform-states"
    key = "governance/terraform.tfstate"
    region = "us-phoenix-1"
    endpoint = "https://acme.compat.objectstorage.us-phoenix-1.oraclecloud.com"
    shared_credentials_file = "../terraform-states_bucket_credentials"
    skip_region_validation = true
    skip_credentials_validation = true
    skip_metadata_api_check = true
    force_path_style = true
  }
}

If you define your remote state data source in a separate file, such as remote-state-data_governance.tf, you can copy and paste the file as needed. Each new configuration can then refer to your compartment in the same way.