Troubleshooting the Terraform Provider

This topic describes how to troubleshoot common Oracle Cloud Infrastructure (OCI) Terraform provider issues.

Start with Troubleshooting Basics and then refer to the following sections:

Troubleshooting Basics

When troubleshooting or getting support for the Oracle Cloud Infrastructure (OCI) Terraform provider, it is often useful to first check the status of the OCI services, the version of Terraform and the provider, and enable and collect verbose logging.

Tip

Checking service status and verbose log output can help you determine whether an issue is related to the Terraform provider or the OCI service the provider is using.

Refer to the list of common issues after you start with the basics.

Checking OCI Service Status and Outages

To check on the latest status and whether there are any outages in OCI, see OCI Status.

Checking the Terraform and OCI Terraform Provider Versions

To verify the version of Terraform and the OCI Terraform provider, initialize Terraform from a directory with your configurations and then run the -version command. For example:

terraform init
terraform -version

The versions are displayed:

Terraform v0.12.20
+ provider registry.terraform.io/hashicorp/oci v3.95.0
Tip

Newer versions of the OCI Terraform provider include the version of the provider in error messages.

The OCI Terraform provider documentation reflects the latest version. You can view documentation for earlier provider versions by visiting the HashiCorp reference and selecting a specific version. You can also download and install a specific version of the provider.

Verbose Logging for OCI Terraform Provider

To get verbose console output when the provider is running, precede your Terraform command with the TF_LOG and OCI_GO_SDK_DEBUG flags. For example:

TF_LOG=DEBUG OCI_GO_SDK_DEBUG=v terraform plan

The TF_LOG level and OCI_GO_SDK_DEBUG flags can also be set as environment variables.

Automatic Retries

While applying, refreshing, or destroying a plan, Terraform may encounter some intermittent OCI errors (such as 429 or 500 errors) that could succeed on retry. By default, the OCI Terraform provider automatically retries such operations for up to 10 minutes. The following fields can be specified in the provider block to further configure the retry behavior:

  • disable_auto_retries - Disable automatic retries for retriable errors.
  • retry_duration_seconds - The minimum duration (in seconds) to retry a resource operation in response to HTTP 429 and HTTP 500 errors. The actual retry duration may be slightly longer due to jittering of retry operations. This value is ignored if the disable_auto_retries field is set to true.

Concurrency Control Using Retry Backoff and Jitter

To alleviate contention between parallel operations against OCI services; the OCI Terraform provider schedules retry attempts using quadratic backoff and full jitter. Quadratic backoff increases the maximum interval between subsequent retry attempts, while full jitter randomly selects a retry interval within the backoff range.

For example, the wait time between the first and second retry attempts is chosen randomly between 1 and 8 seconds. The wait time between the second and third retry attempts is chosen randomly between 1 and 18 seconds. Regardless of the number of retry attempts, the retry interval time is capped after the 12th attempt at 288 seconds.

Note that the retry_duration_seconds field only affects retry duration in response to HTTP 429 and 500 errors; as these errors are more likely to result in success after a long retry duration. Other HTTP errors (such as 400, 401, 403, 404, and 409) are unlikely to succeed on retry. The retry_duration_seconds field does not affect the retry behavior for such errors.

Common Issues

Note

See Terraform state drift with tag defaults and tags for secondary resources for a known issue with tags related to Terraform.

Resources Are Destroyed When Applying Changes

Existing OCI resources might be destroyed and re-created when Terraform configurations attempt to update a resource property that is not updatable. Terraform warns you when changes will destroy a resource. Always run terraform plan before applying changes to see what resources will be affected. See Destructive Changes for more information.

Dependent Resource Cannot Be Updated

Sometimes, an update to an OCI resource should force the resource to be destroyed and re-created, but another resource's dependency on the resource disallows the operation.

For example, you might want to make an update to an oci_core_instance_configuration resource, but an instance pool uses that instance configuration. The instance configuration cannot be deleted because the instance pool references it in a required argument.

To work around this type of behavior, you can use the lifecycle and create_before_destroy meta-arguments in the resource block.

In this example, Terraform creates a second oci_core_instance_configuration resource that includes your updates, then assigns the new instance configuration to the related instance pool. Finally, Terraform destroys the original instance configuration. For example:

resource "oci_core_instance_configuration" "test_instance_configuration" {
...

 lifecycle {
    create_before_destroy = true
  }
}

See The lifecycle Meta-Argument and Destructive Changes for more information.

Resource Cannot Be Destroyed or Updated

You can prevent an OCI resource from being destroyed by including the lifecycle and prevent_destroy = true meta-arguments in the resource block of your Terraform configuration file. The following configuration, for example, results in an Object Storage bucket that cannot be destroyed:

resource "oci_objectstorage_bucket" "test_bucket" {
  #Required
  compartment_id = var.tenancy
  name = "test"
  namespace = "exampleNamespace"

  lifecycle {
    prevent_destroy = true
  }
} 

This meta-argument prevents the use of terraform destroy. Because certain configuration updates require resources to be destroyed before they can be applied, this setting can make some updates impossible to apply as well. In this example, name is a property that cannot be updated without destroying and re-creating the resource. Therefore, you cannot update the name of the bucket without removing or changing the lifecycle meta-argument.

See The lifecycle Meta-Argument for more information.

Resource Argument Cannot Be Unset

Many Oracle Cloud Infrastructure resources managed by the OCI Terraform provider accept configuration arguments that are optional. Once set, whether during resource creation or a subsequent update, these arguments cannot be unset by passing an empty string or removing the argument from the configuration. Attempts to unset these arguments are ignored by Terraform.

The full reference of the OCI Terraform provider's supported resources and data sources contains usage, argument, and attribute details. The full reference is available at docs.oracle.com and HashiCorp.

Data sources and resources are grouped by service within the reference.

Referencing Triggers in Lifecycle Meta-arguments

Terraform v0.14 and later might require that you replace global variables in your configuration files with a combination of local variables and triggers. To reference a trigger in lifecycle and ignore_changes meta-arguments and avoid executing the configuration on subsequent Terraform apply operations, reference the trigger as follows:

resource "null_resource" "exampleB" {
  depends_on = [null_resource.exampleA]
  triggers = {
    os_user = var.os_user
  }
  lifecycle {
    ignore_changes = [
      triggers["os_user"]
    ]
  }

Cannot Delete Compartment

By default, the Terraform provider does not delete a compartment when using the destroy command.

You must set the enable_delete argument to true for the provider to attempt to delete the compartment. For example:

resource "oci_identity_compartment" "test_compartment" {
    compartment_id = var.compartment_id
    description = var.compartment_description
    name = var.compartment_name

    enable_delete = true
}
Note

To destroy a compartment, the compartment must also be empty. Use the depends_on argument to ensure that any hidden dependencies are defined. See Resources for more information.

"Operation Timeout" Error

If the Terraform CLI returns an error message like the following:

Error: Operation Timeout
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current. 
Service: <service>
Error Message: timeout while waiting for state to become 'SUCCEEDED, FAILED, CANCELED' (last state: 'IN_PROGRESS', timeout: 15m)

Then the specified OCI service is indicating that the resource has not yet reached the expected state after polling for some time.

You may need to increase the operation timeout for your resource to continue polling for longer. See Operation Timeouts for details on how to do this.

"Unexpected LifeCycle state" Error

If the Terraform CLI returns an error message like the following:

Error: Unexpected LifeCycle state
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current. 
Service: <service>
Error Message: During deletion, Terraform expected the resource to reach state: TERMINATED, but the service reported unexpected state: RUNNING.
Resource OCID: exampleuniqueID
Suggestion: Please retry or contact support for help with service: <service>

Then the specified OCI service encountered an unknown error. Retry, or contact support regarding that service.

Terraform CLI Issues

This section contains information dealing with the installation and configuration of the Terraform CLI.

"No such file" Error After Upgrading the OCI Terraform Provider

If the Terraform CLI returns an error message like the following:

Error asking for user input: 1 error(s) occurred:

* provider.oci: dial unix /var/folders/6r/8fk5dmbj4_z3sl0mc_y_fhjw0000gn/T/plugin811254328|netrpc: connect: no such file or directory

You are likely using a version of the OCI Terraform provider that is not compatible with the Terraform binary you have installed. For OCI Provider versions v3.x.x and above, a minimum Terraform version of v.0.10.1 is required.

"TCP...i/o timeout" Message When Connecting Via Proxy

If the Terraform CLI returns an error message like the following:

* provider.oci: ... dial tcp 134.70.16.0:443: i/o timeout

Then you may not have properly configured your proxy settings. The OCI Terraform provider supports http_proxy, https_proxy and no_proxy variables where the inclusion or exclusion lists can be defined as follows:

export http_proxy=http://www.your-proxy.com:80/
export https_proxy=http://www.your-proxy.com:80/
export no_proxy=localhost,127.0.0.1

"x509: certificate signed by unknown authority" Error Message

If the Terraform CLI returns an error message like the following:

Error: Get https://iaas.<region>.oraclecloud.com/20160918/services: x509: certificate signed by unknown authority
  on ../modules/network/modules/main/main.tf line 3...

Ensure that Terraform is using trusted TLS certificates and the certificate chain is valid. See Terraform runs failing with "x509: certificate signed by unknown authority" error for more information. If using MacOS Catalina, refer to the MacOS section of the document for more specifics on resolving certificate issues.

"Outdated GPG key...unable to verify new provider releases" Error Message

If the Terraform CLI returns an error message like the following:

Warning: registry.terraform.io: 
This version of Terraform has an outdated GPG key and is unable to verify new provider releases.
Please upgrade Terraform to at least <version> to receive new provider updates.
For details see: https://discuss.hashicorp.com/t/hcsec-2021-12-codecov-security-event-and-hashicorp-gpg-key-exposure/23512

This message means that the Terraform registry is omitting the Terraform provider versions signed by a new GPG key. The Terraform CLI will install the last version of the OCI Terraform provider that it can successfully verify, which might not be the latest version.

To remove this message and ensure you can use the latest version of the OCI Terraform provider, upgrade the Terraform CLI to the latest maintenance release available for the major Terraform version you are using. For example, if you are using Terraform v0.12.21, upgrade to the lastest available version of v0.12.

Terraform Provider Issues

This section contains information related to the installation and configuration of the OCI Terraform provider.

"NotAuthenticated" Error When Using Terraform

If the Terraform CLI returns an error message like the following:

Error: 401-NotAuthenticated
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current.
Service: <service>
Error Message: The required information to complete authentication was not provided or was incorrect.
OPC request ID: exampleuniqueID
Suggestion: Please retry or contact support for help with service: <service>
  • Verify you have properly set user_ocid, tenancy_ocid, fingerprint and private_key_path.
  • Verify your private_key_path is pointing to your private key and not the corresponding public key.
  • Verify you have added the corresponding public key to the user account you have specified with user_ocid.
  • Verify the public/private key pairs you are using are of the correct format. See Required Keys for details on the correct format and how to generate keys.
  • Verify the user account is part of a group with the appropriate permissions to perform the actions in the plan you are executing.
  • Verify your Tenancy has been subscribed to the Region you are targeting in your plan.

"Can not create client, bad configuration: did not find a proper configuration for tenancy" Message When Using Aliases

If the Terraform CLI returns a message like this, it might indicate an issue with your environment:

Error: can not create client, bad configuration: did not find a proper configuration for tenancy

If your provider configuration includes an alias, your resources should explicitly specify the provider alias using provider = "oci.alias_name". If a resource does not use the alias to specify the provider, Terraform creates a default provider to use with such resources. The default provider loads configuration values from environment variables or the ~/.oci/config file. These values may differ from those used by your aliased provider and cause the configuration error.

Either remove the alias in your provider configuration, or ensure that every resource specifies the provider by the proper alias. Read more about using alias in the official Terraform documentation, and see Configuring the Provider for more information about how Terraform uses environment variables and the OCI config file.

"Field cannot be set" Error Message

If the Terraform CLI returns an error message like the following:

* Error: "field_name": this field cannot be set

You are likely using an older version of the OCI Terraform provider and the field you are trying to set was released in a later version. Use the following command to check your Terraform provider version.

terraform -version

The OCI Terraform provider documentation reflects the latest version.

"Could not get info about the first DbHome in the dbSystem" Error Message When Importing db_home

If the oci_database_db_system being imported is missing a primary db_home, an empty placeholder for db_home is set in the Terraform state file. To keep configurations consistent with the imported state, add an empty placeholder for db_home to your configuration. For example:

# Add this placeholder into your oci_database_db_system configuration to indicate that the primary db home is empty. 
db_home {
 database { 
  admin_password = "" 
  } 
}

"Failed to query available provider packages" Error Message When Running Resource Discovery

If the Terraform CLI returns an error message like the following when using resource discovery:

Failed to query available provider packages
 
Could not retrieve the list of available versions for provider hashicorp/oci:
the previously-selected version is no longer available

Then you can ensure that Terraform uses an existing local provider binary by specifying its location using the provider_bin_path environment variable. For example:

export provider_bin_path=/Users/user/go/bin/

Terraform attempts to download the latest version of the OCI Terraform provider when you use resource discovery.

Default Tags Deleted on Apply

Sometimes, the OCI Terraform provider can unexpectedly delete existing tag defaults from a resource when running terraform apply. This issue affects the Oracle-Tags automatic tag defaults in particular.

To work around this issue, you can add the ignore_defined_tags attribute to your provider block.

The ignore_defined_tags attribute allows you to list out the keys of the defined tags that Terraform will ignore as part of plan or apply. The ignore_defined_tags attribute can only be specified at the provider level, and has a maximum allowed size of 100. The tags provided in this attribute are ignored for all the resources in that Terraform file.

In the following example, "Oracle-Tags.CreatedOn" and "Oracle-Tags.CreatedBy" are the keys in the defined_tags map associated with a remote resource:

provider "oci" {
    ignore_defined_tags = ["Oracle-Tags.CreatedBy", "Oracle-Tags.CreatedOn"]
}

For more information, see Provider Definitions and refer to the related GitHub issue.

Service API Errors

Because the Terraform provider interacts with OCI services on your behalf, many error messages surfaced by the Terraform provider come directly from OCI services. The API Errors reference lists common errors returned by all services.

Service error messages returned by the OCI Terraform provider include the following information:

  • Error - the HTTP status and API error codes
  • Provider version - the version of the OCI Terraform provider used to make the request
  • Service - the OCI service responding with the error
  • Error message - details regarding the error returned by the service
  • OPC request ID - the request ID
  • Suggestion - suggested next steps

For example:

Error: <http_code>-<api_error_code>
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current. 
Service: <service>
Error Message: <error_message>
OPC request ID: exampleuniqueID
Suggestion: <next_steps>

This section details a few of the more commonly returned service errors.

"400-InvalidParameter" Error

If the Terraform CLI returns an error message like the following:

Error: 400-InvalidParameter
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current. 
Service: <service>
Error Message: <error_message>
OPC request ID: exampleuniqueID
Suggestion: Please update the parameter(s) in the Terraform config as per error message: <error_message>

Update the parameter specified in the error message in the Terrform configuration for the resource.

"400-LimitExceeded" Error

While using Terraform, you might encounter errors indicating that you have reached or exceeded the service limits for a resource. For example:

Error: 400-LimitExceeded
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current. 
Service: <service>
Error Message: Fulfilling this request exceeds the Oracle-defined limit for this tenancy for this resource type.
OPC request ID: exampleuniqueID
Suggestion: Request a service limit increase for this resource <service>

To understand more about your OCI service limits and how to request a limit increase, see Service Limits

"404-NotAuthorized" Error

If the Terraform CLI returns an error message like the following:

Error: 404-NotAuthorizedOrNotFound
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current.
Service: <service>
Error Message: Authorization failed or requested resource not found.
OPC request ID: exampleuniqueID
Suggestion: Either the resource has been deleted or service <service> need policy to access this resource. Policy reference: <link>

Verify the user account is part of a group with the appropriate permissions to perform the actions in the plan you are executing. Refer to the Policy Reference for your service for more information.

"500-InternalError" Error

If the Terraform CLI returns an error message like the following:

Error: 500-InternalError
Provider version: <provider_version>, released on <release_date>. This provider is <n> updates behind to current. 
Service: <service>
Error Message: Internal error occurred
OPC request ID: exampleuniqueID
Suggestion: The service for this resource encountered an error. Please contact support for help with service <service>

The service responded to the request from the Terraform provider with an internal error. If you contact support for this issue, reference the information in the message.