Oracle Cloud Infrastructure Documentation

Using Object Storage for State Files

Store Terraform state files in OCI Object Storage.

Note

Of the backend types covered on this page, we recommend using an Using the OCI native backend.

Using an S3-Compatible Backend (Deprecated)

Note

The S3-compatible backend method is deprecated. Only use this method if you're not able to upgrade to Terraform version v1.12.0 or greater.

To configure the S3-compatible backend, complete the steps in the following sections.

Task 1: Set Up Access to OCI

For more information about S3 compatibility, see Amazon S3 Compatibility API Prerequisites. For information about permissions for Object Lifecycle Management, see Required IAM Policies.
  1. Sign Up for Oracle Cloud Infrastructure and obtain a unique namespace.
  2. Any user of the Amazon S3 Compatibility API with Object Storage needs permission to work with the service. If you're not sure if you have permission, contact your administrator. For basic information about policies, see How Policies Work. For policies that enable use of Object Storage, see Common Policies and the Policy Reference.
  3. Use an existing or create a Customer Secret Key. A Customer Secret Key consists of an Access Key/Secret Key pair. See Working with Customer Secret Keys for details. To use or create the key pair:
    • To use an existing Customer Secret Key, you must already know the Secret Key. For security reasons, you can't retrieve a Secret Key after generation. To show or copy the Access Key: In the navigation menu , select the Profile menu Profile menu icon and then select User settings or My profile, depending on the option that you see. On the left side of the page, select Customer Secret Keys. Hover over the Access Key associated with the Name of a particular Customer Secret key, then select Copy.
    • To create a Customer Secret Key using the Console, see To create a Customer Secret key.
    • To create a Customer Secret Key using the Command Line Interface (CLI), see oci iam customer-secret-key create.

Task 2: Configure Authentication

  1. Set the location for the credentials file.
    Caution

    Never set the access_key and the secret_key attributes in the same Terraform backend configuration. Storing these attributes in the same configuration creates a security risk.

    The default location is ~/.aws/credentials. You can set an alternate location by using the S3 backend shared_credentials_file option.

  2. Configure the [default] entry in the credentials file with the appropriate Object Storage credentials.

    The file can contain any number of credential profiles. If you provide a different profile name, you must also update the backend profile option in the Terraform configuration file.

    Following is an example of Object Storage credentials:

    [default]
aws_access_key_id=ae37c0b4ffc488c7d4e6b360a21312244330718f
aws_secret_access_key=mSTdaWhlbWj3ty4JZXlm0NUZV52xlImWjayJLJ6OH9A=
    Note

    The key values provided in the example aren't valid. Valid aws_access_key_id and aws_secret_access_key are user-specific values generated using the previous steps.

Task 3: Configure the S3 Backend in Terraform

  1. Set the Object Storage endpoint value in the following format:
    https://<namespace>.compat.objectstorage.<region>.oraclecloud.com
  2. Update the terraform block within the Terraform configuration to define the backend type.
    Note

    Because variables and locals aren't accepted in the terraform block, you must hard-code backend configuration values.

    Following are blocks organized by Terraform version. Select the block that maches the version of the Terraform configuration.

    Terraform version 1.6.4 or later

    # (Terraform version >= 1.6.4)
terraform {
  backend "s3" {
    bucket                    = "terraform-states"
    region                    = "us-phoenix-1"
    key                       = "tf.tfstate"
    skip_region_validation      = true
    skip_credentials_validation = true
    skip_requesting_account_id  = true
    use_path_style              = true
    skip_s3_checksum            = true
    skip_metadata_api_check = true
    endpoints = {
      s3 = "https://<namespace>.compat.objectstorage.<region>.oraclecloud.com"
    }
  }
}

    Terraform versions before 1.64

    # (Terraform version < 1.6.4)
terraform {
  backend "s3" {
    bucket   = "terraform-states"
    key      = "networking/terraform.tfstate"
    region   = "us-phoenix-1"
    endpoint = "https://<namespace>.compat.objectstorage.<region>.oraclecloud.com"
   skip_region_validation      = true
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    force_path_style            = true
  }
}
    Caution

    If the same bucket is used across many Terraform configurations, the key must be unique to avoid overwriting the state file. This example uses a single bucket (terraform-states) to store all Terraform state files, but uses a unique prefix for the object name based on the resource (networking).
  3. Run terraform init.

    If you already have an existing terraform.tfstate file, then Terraform prompts you to confirm that the current state file is the one to upload to the remote state.

  4. Run terraform apply.
The generated state file is uploaded to Object Storage.
To share state across Terraform projects, use the backend configuration in terraform_remote_state. For more information, see Accessing Remote States.

Using an HTTP Backend

Note

Because the HTTP backend requires a pre-authenticated request (PAR) for each state file, the preferred method for storing state files in Object Storage is Using the OCI native backend.

Use the HTTP backend type to store state using a REST client, and to fetch, update, and purge state using the HTTP GET, POST, and DELETE methods.

To configure the HTTP backend, complete the steps in the following sections.

Task 1: Upload Existing State

A state file must exist in the bucket before you create the pre-authenticated request (PAR). This file can be an existing state file, or an empty file for the initial state.
To upload an existing state file it using Curl, make an HTTP PUT request to the object store URL: 
curl -X PUT -H "Content-Type: text/plain" --data-binary "@path/to/local/tfstate" http://<prefix>/<my-access-uri>
For instructions to upload a file to a bucket using the Console, CLI, or API, see Uploading an Object Storage Object to a Bucket.

Task 2: Create a Read/Write Pre-Authenticated Request

With a pre-authenticated request (PAR) in Object Storage that specifies read/write permissions, you can access the Terraform state file without providing credentials.

For instructions to create a PAR using the Console, CLI, or API, see Creating a Pre-Authenticated Request in Object Storage.

Task 3: Configure the HTTP Backend in Terraform

  1. Set the address value in the following format, where the region and access URI are specific to you: 
    https://objectstorage.<region>.oraclecloud.com/<my-access-uri>

    Example:

    https://objectstorage.us-phoenix-1.oraclecloud.com/p/example-guid/n/example/b/terraform-state/o/terraform.tfstate
  2. Update the terraform block within the Terraform configuration to define the backend type.
    Note

    Because variables and locals aren't accepted in the terraform block, you must hard-code backend configuration values.

    Following is an example Terraform configuration using the region us-phoenix-1.

    terraform {
  backend "http" {
    update_method = "PUT"
    address       = "https://objectstorage.us-phoenix-1.oraclecloud.com/p/example-guid/n/example/b/terraform-state/o/terraform.tfstate"
  }
}

    For more example configuration and state files that reference code, and a summary of configuration variables, see HTTP.

  3. Run terraform init.
  4. Run terraform apply.
The generated state file is uploaded to Object Storage.
To share state across Terraform projects, use the backend configuration in terraform_remote_state. For more information, see Accessing Remote States.

Accessing Remote States

Use terraform_remote_state to access properties of objects in one Terraform configuration from another configuration.

For example, you might use one configuration to define compartments and another to define VCNs. If resources are in the same Terraform configuration folder, you can refer to a compartment OCID from the VCN configuration by using something such as this: module.iam_compartment_SANDBOX.compartment_id.

But assume that our definitions don't share a state file and we have a file structure similar to the following:

.
├── governance
│   ├── compartments.tf
│   ├── outputs.tf
│   ├── remote-backend.tf
│   └── variables.tf
├── networking
│   ├── outputs.tf
│   ├── remote-backend.tf
│   ├── remote-state-data_governance.tf
│   ├── variables.tf
│   └── vcns.tf
└── terraform-states_bucket_credentials

In this example:

  • Both governance and networking configurations store their respective state files on an OCI Object Storage bucket using the remote-backend.tf and terraform-states_bucket_credentials files.
  • The compartments.tf file creates a compartment at the root level using the iam-compartment module from the Terraform Registry as follows:
    module "iam_compartment_SANDBOX" {
  source = "oracle-terraform-modules/iam/oci//modules/iam-compartment"
  version = "2.0.0"
  tenancy_ocid = var.tenancy_ocid
  compartment_id = var.tenancy_ocid # define the parent compartment. Creation at tenancy root if omitted
  compartment_name = "SANDBOX"
  compartment_description = "Test and evaluate OCI features here"
  compartment_create = true # if false, a data source with a matching name is created instead
  enable_delete = true # if false, on `terraform destroy`, compartment is deleted from the terraform state but not from oci
}

Defining Outputs

The terraform_remote_state data source can access output values from another Terraform configuration using the latest state file with a remote backend. For the networking configuration to access the governance configuration and dynamically retrieve Terraform resources properties, you must define outputs for the governance Terraform configuration. Without a defined output, the values can't be used from outside of its configuration.

The governance/outputs.tf file would look similar to the following:

output "iam_compartment_SANDBOX" {
  description = "compartment ocid, parent ocid, name, description"
  value = {
    id = module.iam_compartment_SANDBOX.compartment_id
    parent_id = module.iam_compartment_SANDBOX.parent_compartment_id
    name = module.iam_compartment_SANDBOX.compartment_name
    description = module.iam_compartment_SANDBOX.compartment_description
  }
}

Referring to a Remote State

In this example, we're using the vcn module from Terraform Registry to define a new VCN. The networking configuration refers to the governance configuration to define the VCN's compartment OCID:

module "vcn_hub1iad" {
  source = "oracle-terraform-modules/vcn/oci"
  version = "2.2.0"

  # general oci parameters
  compartment_id = data.terraform_remote_state.governance.outputs.iam_compartment_SANDBOX["id"]
  tags = var.tags

  # vcn parameters
  create_drg = false
  internet_gateway_enabled = true
  lockdown_default_seclist = true
  nat_gateway_enabled = false
  service_gateway_enabled = false
  vcn_cidr = "10.0.0.0/16"
  vcn_dns_label = "hub1iad"
  vcn_name = "hub1"
}

But, for the compartment_id = data.terraform_remote_state.governance.outputs.iam_compartment_SANDBOX["id"] line to be correctly interpreted, you must define a data.terraform_remote_state object.

Defining the Remote State Data Source

After the following terraform_remote_state data source is added to the networking configuration, you can access the governance Terraform outputs from configurations within the networking folder:

data "terraform_remote_state" "governance" {
  backend = "oci"
  config = {
    bucket = "terraform-states"
    key = "governance/terraform.tfstate"
    region = "us-phoenix-1"
    endpoint = "https://acme.compat.objectstorage.us-phoenix-1.oraclecloud.com"
    shared_credentials_file = "../terraform-states_bucket_credentials"
    skip_region_validation = true
    skip_credentials_validation = true
    skip_metadata_api_check = true
    force_path_style = true
  }
}

If you define the remote state data source in a separate file, such as remote-state-data_governance.tf, you can copy and paste the file as needed. Each new configuration can then refer to the compartment in the same way.