Stack Creation Failed

Troubleshoot a failed Oracle WebLogic Server domain that you attempted to create with Oracle WebLogic Server for OCI.

View the stack log files

Use the Terraform job logs in Resource Manager to identify the cause of the failure.

Click the navigation menu , select Developer Services. Under the Resource Manager group, click Jobs.
Identify and click the job for your stack.
- The Type is Apply.
- The State is Failed.
- The Stack is the name of your Oracle WebLogic Server for OCI stack.
From the Logs section, search the log for error messages.
You can optionally Download the log files and search the files offline.
See below for details about specific error messages.

Modify the stack configuration

If necessary, delete the current stack resources, modify your stack configuration, and then apply the changes.

Click the navigation menu , select Developer Services. Under the Resource Manager group, click Stacks.
Click the name of your stack.
Click Terraform Actions and select Destroy.
Wait for the destroy job to complete.
Click Edit Stack.
When done, click Save Changes.
Click Terraform Actions and select Apply.

Cannot launch a stack in Marketplace

Example message: Unable to accept Terms of Use

In Marketplace, you might see the message when you click Launch Stack, after you've selected a stack version and compartment, and checked the Oracle Standard Terms and Restrictions box.

You likely don't have permission to:

Create Marketplace applications in the selected compartment. Verify that this policy exists in the compartment where you want to create the stack.
Allow group Your_Group to manage app-catalog-listing in compartment Your_Compartment
Access the selected compartment. Choose another compartment or ask your administrator.

Cannot determine home region

Example message:

data.oci_core_app_catalog_subscriptions.mp_image_subscription[0]: Refreshing state...
Error: Null value found in list ... "oci_identity_regions" "home-region"

If you are not an administrator, ask them to verify that the following root-level policy exists in your tenancy:

Allow group Your_Group to inspect tenancies in tenancy

Cannot find dynamic group and secrets policy

Example messages:

Error: Service error:NotAuthorizedOrNotFound. Authorization failed or requested resource not found. http status code: 404.
 Opc request id: request_id on modules/policies/groups.tf line 8, in resource...
 "oci_identity_dynamic_group" "wlsc_instance_principal_group" {

Error: Service error:NotAuthorizedOrNotFound. Authorization failed or requested resource not found. http status code: 404.
 Opc request id: request_id on wlsc-policies.tf line 10, in resource...
 "oci_identity_policy" "wlsc_secret-service-policy" {

When the OCI Policies check box is selected (by default), Oracle WebLogic Server for OCI creates a dynamic group and one or more root-level policies in your tenancy.

You must be an Oracle Cloud Infrastructure administrator, or be granted root-level permissions to create domains. If you are not an administrator, ask them to verify that root-level policies exist in your tenancy. For example:

Allow group Your_Group to manage dynamic-groups in tenancy
Allow group Your_Group to manage policies in tenancy
Allow group Your_Group to use secret-family in tenancy

See:

Maximum number of dynamic groups has exceeded

Example message:

<WLSC-VM-ERROR-0119> : Failed to get secret content for [ocid1.vaultsecret.oc1.iad.alongstring123]: [{'status': 400, 'message': "This instance principal matches more than '5' dynamic groups, update your dynamic groups' matching rules"...'}]>

When the OCI Policies check box is selected (by default), Oracle WebLogic Server for OCI creates a dynamic group and one or more root-level policies in your tenancy. The maximum number of dynamic groups allowed is 5.

Solution:

Add the following policy that uses the existing dynamic group to access the new secrets for the new stack:

Allow dynamic-group <existing-dyanmic-group-name> to read secret-bundles in tenancy where target.secret.id = '<OCID_of_the_secret>'

Deselect the OCI Policies check box and try to create the stack again.

Unable to get secret content or decrypted credential

Example messages:

Failed to get secret content for Your_vault_secret_OCID
Authorization failed or requested resource not found
Error retrieving %s password from Secret Vault
Failed in create domain due to exception [Failed to retrieve WebLogic Password from Secrets Vault]
Failed to retrieve IDCS Client Secret from Secrets Vault
Unable to get decrypt credential
Key or Vault does not exist or you are not authorized to access them.

When you create a domain with Oracle WebLogic Server for OCI, you provide the OCID values of the secrets that contain the passwords to use for the domain and during provisioning. The compute instances use this information to decrypt the passwords. The compute instances are granted access to vault secrets using policies.

You must be an Oracle Cloud Infrastructure administrator, or be granted root-level permissions to create domains. If you are not an administrator, ask them to verify that relevant vault secret policies exist in your tenancy and compartment. For example:

Allow group Your_Group to use secret-family in tenancy
Allow dynamic-group Your_DynamicGroup to use secret-family in compartment MyCompartment
Allow dynamic-group Your_DynamicGroup to use keys in compartment MyCompartment
Allow dynamic-group Your_DynamicGroup to use vaults in compartment MyCompartment

If the policies exist, check that the OCID of the compartment in listed in dynamic group.

See:

Unable to get decrypted credential when creating a stack in a private subnet

Example message: <WLSC-VM-ERROR-001> Unable to get decrypt credential [HTTPSConnectionPool(host='auth.us-phoenix-1.oraclecloud.com', port=443): Max retries exceeded with url: /v1/x509 (Caused by ConnectTimeoutError(<oci._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x1e5110>, 'Connection to auth.us-phoenix-1.oraclecloud.com timed out. (connect timeout=10)'))]>

When you create a domain with Oracle WebLogic Server for OCI in an existing private subnet, provisioning fails if the WebLogic Server subnet is using a route table that does not include a service gateway or a Network Address Translation (NAT) gateway.

Modify the private subnet, and select a route table that uses a service gateway or NAT gateway. Or select a virtual cloud network (VCN) whose default route table uses a service gateway or NAT gateway. Refer to these topics:

Failed to download Oracle Autonomous Database wallet

Example message: module.provisioners.null_resource.status_check[0] (remote-exec): <Nov 23, 2019 09:37:17 PM GMT> <ERROR> <oci_api_utils> <(host:stackname-wls-0.subnetxxx.stacknamevcn.oraclevcn.com) - <WLSC-VM-ERROR-0052> : Unable to download atp wallet. [{'status': 403, 'message': u'Forbidden', 'code': u'Forbidden', 'opc-request-id': 'FA6C16D8B'}]

You must be an Oracle Cloud Infrastructure administrator, or be granted root-level and compartment-level permissions to create domains. Access to the database wallet is needed when you create a JRF-enabled domain that uses an autonomous database. If you are not an administrator, ask them to verify that relevant policies for autonomous databases exist in your tenancy and compartment. For example:

Allow group Your_Group to inspect autonomous-transaction-processing-family in compartment Your_ATP_Compartment
Allow dynamic-group Your_DynamicGroup to inspect autonomous-transaction-processing-family in compartment Your_ATP_Compartment

See:

Failed to validate DB connectivity

When you create a domain that includes the Java Required Files (JRF) components, you must select an existing database and provide connection details. The compute instances use this information to connect to the database and provision the JRF database schemas.

Possible causes for this error include:

You entered the wrong database password or a plain text password.
The database does not allow the compute instances to access its listen port (1521 by default).
- Oracle Autonomous Database - Check your access control list (ACL).
- Oracle Cloud Infrastructure Database - Check the network security group that was assigned to the database, and the security lists for the subnet on which the database was created.
You selected an Oracle Cloud Infrastructure Database running Oracle Database 12c or later, and you did not provide the name of a pluggable database (PDB).

Invalid or overlapping network CIDR

Stack provisioning fails if you specify subnets with overlapping CIDRs or use the same subnet for WebLogic Server and the load balancer.

Example messages:

Error: module.network-wls-public-subnet.oci_core_subnet.wls-subnet: 1 error(s) occurred: oci_core_subnet.wls-subnet: Service error:InvalidParameter. The requested CIDR 10.0.3.0/24 is invalid: subnet ocid1.subnet.oc1.iad.aaan4a with CIDR 10.0.3.0/24 overlaps with this CIDR.. http status code: 400.

Error: module.validators.null_resource.duplicate_lb2_subnet_cidr: : invalid or unknown key: WLSC-ERROR: Load balancer subnet 2 CIDR has to be unique value.

Error: module.validators.null_resource.duplicate_wls_subnet_cidr: : invalid or unknown key: WLSC-ERROR: Weblogic subnet CIDR has to be unique value.

Possible causes for these errors include:

You chose to create new subnets for WebLogic Server, the load balancer, or the bastion, and the CIDR you specified for these subnets overlaps with the CIDRs for existing subnets in the same virtual cloud network (VCN).
You chose to use an existing subnet when provisioning a stack with a load balancer, and you specified the same subnet for WebLogic Server and the load balancer.
You created a JRF-enabled domain, your Oracle Cloud Infrastructure Database and WebLogic domain are in different VCNs, and the VCNs have overlapping CIDRs. For example, you cannot create a WebLogic domain on VCN 10.0.0.0/16 that uses a database on VCN 10.0.0.1/24.

Job is still running or has timed out

Most stack creation jobs for Oracle WebLogic Server for OCI should complete within an hour. Some internal provisioning problems might cause the job to run indefinitely until it eventually times out after 24 hours.

After the current Apply job times out, run a new Apply job on the same stack. This will destroy any resources that were created, and then attempt to create the resources again. If the problem occurs again, contact support.

Failed to check database port is open for Exadata DB system

When you create a domain that includes Java Required Files (JRF) components, for Exadata DB systems, the database port open check does not work if the Create DB Security List checkbox is selected. In this case, the provisioning fails if the database subnet has more than five security lists.

So, when provisioning, deselect the Create DB Security List check box to avoid creating an additional security list for the database port in the VCN, and manually open the database port (1521 by default).

View the Stack Log Files

Use the Terraform job logs in Resource Manager to identify the cause of the failure.

Click the navigation menu , select Developer Services. Under the Resource Manager group, click Jobs.
Identify and click the job for your stack.
- The Type is Apply.
- The State is Failed.
- The Stack is the name of your Oracle WebLogic Server for OCI stack.
From the Logs section, search the log for error messages.
You can optionally Download the log files and search the files offline.
See the following topics for details about specific error messages.

Modify the Stack Configuration

If necessary, delete the current stack resources, modify your stack configuration, and then apply the changes.

Click the navigation menu , select Developer Services. Under the Resource Manager group, click Stacks.
Click the name of your stack.
Click Terraform Actions and select Destroy.
Wait for the destroy job to complete.
Click Edit Stack.
When done, click Save Changes.
Click Terraform Actions and select Apply.

Cannot Launch a Stack in Marketplace

Example message: Unable to accept Terms of Use

In Marketplace, you might see the message when you click Launch Stack, after you've selected a stack version and compartment, and checked the Oracle Standard Terms and Restrictions box.

You likely don't have permission to:

Create Marketplace applications in the selected compartment. Verify that this policy exists in the compartment where you want to create the stack.
Allow group Your_Group to manage app-catalog-listing in compartment Your_Compartment
Access the selected compartment. Choose another compartment or ask your administrator.

Cannot Determine Home Region

Example message:

data.oci_core_app_catalog_subscriptions.mp_image_subscription[0]: Refreshing state...
Error: Null value found in list ... "oci_identity_regions" "home-region"

If you are not an administrator, ask them to verify that the following root-level policy exists in your tenancy:

Allow group Your_Group to inspect tenancies in tenancy

Cannot Find Dynamic Group and Secrets Policy

Example messages:

Error: Service error:NotAuthorizedOrNotFound. Authorization failed or requested resource not found. http status code: 404.
 Opc request id: request_id on modules/policies/groups.tf line 8, in resource...
 "oci_identity_dynamic_group" "wlsc_instance_principal_group" {

Error: Service error:NotAuthorizedOrNotFound. Authorization failed or requested resource not found. http status code: 404.
 Opc request id: request_id on wlsc-policies.tf line 10, in resource...
 "oci_identity_policy" "wlsc_secret-service-policy" {

When the OCI Policies check box is selected (by default), Oracle WebLogic Server for OCI creates a dynamic group and one or more root-level policies in your tenancy.

Allow group Your_Group to manage dynamic-groups in tenancy
Allow group Your_Group to manage policies in tenancy
Allow group Your_Group to use secret-family in tenancy

See:

Maximum Number of Dynamic Groups Has Exceeded

Example message:

<WLSC-VM-ERROR-0119> : Failed to get secret content for [ocid1.vaultsecret.oc1.iad.alongstring123]: [{'status': 400, 'message': "This instance principal matches more than '5' dynamic groups, update your dynamic groups' matching rules"...'}]>

Solution:

Add the following policy that uses the existing dynamic group to access the new secrets for the new stack:

Allow dynamic-group <existing-dyanmic-group-name> to read secret-bundles in tenancy where target.secret.id = '<OCID_of_the_secret>'

Deselect the OCI Policies check box and try to create the stack again.

Unable to Get Secret Content or Decrypted Credential

Example messages:

Failed to get secret content for Your_vault_secret_OCID
Authorization failed or requested resource not found
Error retrieving %s password from Secret Vault
Failed in create domain due to exception [Failed to retrieve WebLogic Password from Secrets Vault]
Failed to retrieve IDCS Client Secret from Secrets Vault
Unable to get decrypt credential
Key or Vault does not exist or you are not authorized to access them.

Allow group Your_Group to use secret-family in tenancy
Allow dynamic-group Your_DynamicGroup to use secret-family in compartment MyCompartment
Allow dynamic-group Your_DynamicGroup to use keys in compartment MyCompartment
Allow dynamic-group Your_DynamicGroup to use vaults in compartment MyCompartment

If the policies exist, check that the OCID of the compartment in listed in dynamic group.

See:

Unable to Get Decrypted Credential When Creating a Stack in a Private Subnet

Failed to Download Oracle Autonomous Database Wallet

Allow group Your_Group to inspect autonomous-transaction-processing-family in compartment Your_ATP_Compartment
Allow dynamic-group Your_DynamicGroup to inspect autonomous-transaction-processing-family in compartment Your_ATP_Compartment

See:

Failed to Validate DB Connectivity

Possible causes for this error include:

You entered the wrong database password or a plain text password.
The database does not allow the compute instances to access its listen port (1521 by default).
- Oracle Autonomous Database - Check your access control list (ACL).
- Oracle Cloud Infrastructure Database - Check the network security group that was assigned to the database, and the security lists for the subnet on which the database was created.
You selected an Oracle Cloud Infrastructure Database running Oracle Database 12c or later, and you did not provide the name of a pluggable database (PDB).

Invalid or Overlapping Network CIDR

Stack provisioning fails if you specify subnets with overlapping CIDRs or use the same subnet for WebLogic Server and the load balancer.

Example messages:

Error: module.validators.null_resource.duplicate_lb2_subnet_cidr: : invalid or unknown key: WLSC-ERROR: Load balancer subnet 2 CIDR has to be unique value.

Error: module.validators.null_resource.duplicate_wls_subnet_cidr: : invalid or unknown key: WLSC-ERROR: Weblogic subnet CIDR has to be unique value.

Possible causes for these errors include:

You chose to create new subnets for WebLogic Server, the load balancer, or the bastion, and the CIDR you specified for these subnets overlaps with the CIDRs for existing subnets in the same virtual cloud network (VCN).
You chose to use an existing subnet when provisioning a stack with a load balancer, and you specified the same subnet for WebLogic Server and the load balancer.
You created a JRF-enabled domain, your Oracle Cloud Infrastructure Database and WebLogic domain are in different VCNs, and the VCNs have overlapping CIDRs. For example, you cannot create a WebLogic domain on VCN 10.0.0.0/16 that uses a database on VCN 10.0.0.1/24.