4 Troubleshoot Oracle WebLogic Server for Oracle Cloud Infrastructure

Identify common problems in Oracle WebLogic Server for Oracle Cloud Infrastructure and learn how to diagnose and solve them.

Check Known Issues

Learn about known problems in Oracle WebLogic Server for Oracle Cloud Infrastructure and how to work around them.

Stack Creation Failed

Troubleshoot a failed Oracle WebLogic Server domain that you attempted to create with Oracle WebLogic Server for Oracle Cloud Infrastructure.

View the stack log files

Use the Terraform job logs in Resource Manager to identify the cause of the failure.

  1. From the navigation menu, select Resource Manager, and then click Jobs.
  2. Identify and click the job for your stack.
    • The Type is Apply.
    • The State is Failed.
    • The Stack is the name of your Oracle WebLogic Server for Oracle Cloud Infrastructure stack.
  3. From the Logs section, search the log for error messages.

    You can optionally Download the log files and search the files offline.

  4. See below for details about specific error messages.

Modify the stack configuration

If necessary, delete the current stack resources, modify your stack configuration, and then apply the changes.

  1. From the navigation menu, select Resource Manager, and then click Stacks.
  2. Click the name of your stack.
  3. Click Terraform Actions and select Destroy.

    Wait for the destroy job to complete.

  4. Click Edit Stack.
  5. When done, click Save Changes.
  6. Click Terraform Actions and select Apply.

Cannot launch a stack in Marketplace

Example message: Unable to accept Terms of Use

In Marketplace, you might see the message when you click Launch Stack, after you've selected a stack version and compartment, and checked the Oracle Standard Terms and Restrictions box.

You likely don't have permission to:

  • Create Marketplace applications in the selected compartment. Verify that this policy exists in the compartment where you want to create the stack.

    Allow group Your_Group to manage app-catalog-listing in compartment Your_Compartment

  • Access the selected compartment. Choose another compartment or ask your administrator.

Cannot determine home region

Example message:

data.oci_core_app_catalog_subscriptions.mp_image_subscription[0]: Refreshing state...
Error: Null value found in list ... "oci_identity_regions" "home-region"

If you are not an administrator, ask them to verify that the following root-level policy exists in your tenancy:

Allow group Your_Group to inspect tenancies in tenancy

Cannot find dynamic group and secrets policy

Example messages:

Error: Service error:NotAuthorizedOrNotFound. Authorization failed or requested resource not found. http status code: 404.
 Opc request id: request_id on modules/policies/groups.tf line 8, in resource...
 "oci_identity_dynamic_group" "wlsc_instance_principal_group" {
Error: Service error:NotAuthorizedOrNotFound. Authorization failed or requested resource not found. http status code: 404.
 Opc request id: request_id on wlsc-policies.tf line 10, in resource...
 "oci_identity_policy" "wlsc_secret-service-policy" {

When the OCI Policies check box is selected (by default), Oracle WebLogic Server for Oracle Cloud Infrastructure creates a dynamic group and one or more root-level policies in your tenancy.

You must be an Oracle Cloud Infrastructure administrator, or be granted root-level permissions to create domains. If you are not an administrator, ask them to verify that root-level policies exist in your tenancy. For example:

Allow group Your_Group to manage dynamic-groups in tenancy
Allow group Your_Group to manage policies in tenancy
Allow group Your_Group to use secret-family in tenancy

See:

Maximum number of dynamic groups has exceeded

Example message:

<WLSC-VM-ERROR-0119> : Failed to get secret content for [ocid1.vaultsecret.oc1.iad.alongstring123]: [{'status': 400, 'message': "This instance principal matches more than '5' dynamic groups, update your dynamic groups' matching rules"...'}]>

When the OCI Policies check box is selected (by default), Oracle WebLogic Server for Oracle Cloud Infrastructure creates a dynamic group and one or more root-level policies in your tenancy. The maximum number of dynamic groups allowed is 5.

Deselect the OCI Policies check box and try to create the stack again. Alternatively, delete the dynamic groups that are no longer needed.

Unable to get secret content or decrypted credential

Example messages:

  • Failed to get secret content for Your_vault_secret_OCID
  • Authorization failed or requested resource not found
  • Error retrieving %s password from Secret Vault
  • Failed in create domain due to exception [Failed to retrieve WebLogic Password from Secrets Vault]
  • Failed to retrieve IDCS Client Secret from Secrets Vault
  • Unable to get decrypt credential
  • Key or Vault does not exist or you are not authorized to access them.

When you create a domain with Oracle WebLogic Server for Oracle Cloud Infrastructure, you provide the OCID values of the secrets that contain the passwords to use for the domain and during provisioning. The compute instances use this information to decrypt the passwords. The compute instances are granted access to vault secrets using policies.

You must be an Oracle Cloud Infrastructure administrator, or be granted root-level permissions to create domains. If you are not an administrator, ask them to verify that relevant vault secret policies exist in your tenancy and compartment. For example:

Allow group Your_Group to use secret-family in tenancy
Allow dynamic-group Your_DynamicGroup to use secret-family in compartment MyCompartment
Allow dynamic-group Your_DynamicGroup to use keys in compartment MyCompartment
Allow dynamic-group Your_DynamicGroup to use vaults in compartment MyCompartment

If the policies exist, check that the OCID of the compartment in listed in dynamic group.

See:

Unable to get decrypted credential when creating a stack in a private subnet

Example message: <WLSC-VM-ERROR-001> Unable to get decrypt credential [HTTPSConnectionPool(host='auth.us-phoenix-1.oraclecloud.com', port=443): Max retries exceeded with url: /v1/x509 (Caused by ConnectTimeoutError(<oci._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x1e5110>, 'Connection to auth.us-phoenix-1.oraclecloud.com timed out. (connect timeout=10)'))]>

When you create a domain with Oracle WebLogic Server for Oracle Cloud Infrastructure in an existing private subnet, provisioning fails if the WebLogic Server subnet is using a route table that does not include a service gateway or a Network Address Translation (NAT) gateway.

Modify the private subnet, and select a route table that uses a service gateway or NAT gateway. Or select a virtual cloud network (VCN) whose default route table uses a service gateway or NAT gateway. Refer to these topics:

Failed to download ATP wallet

Example message: module.provisioners.null_resource.status_check[0] (remote-exec): <Nov 23, 2019 09:37:17 PM GMT> <ERROR> <oci_api_utils> <(host:stackname-wls-0.subnetxxx.stacknamevcn.oraclevcn.com) - <WLSC-VM-ERROR-0052> : Unable to download atp wallet. [{'status': 403, 'message': u'Forbidden', 'code': u'Forbidden', 'opc-request-id': 'FA6C16D8B'}]

You must be an Oracle Cloud Infrastructure administrator, or be granted root-level and compartment-level permissions to create domains. Access to the database wallet is needed when you create a JRF-enabled domain that uses an autonomous database. If you are not an administrator, ask them to verify that relevant policies for autonomous databases exist in your tenancy and compartment. For example:

Allow group Your_Group to inspect autonomous-transaction-processing-family in compartment Your_ATP_Compartment
Allow dynamic-group Your_DynamicGroup to inspect autonomous-transaction-processing-family in compartment Your_ATP_Compartment

See:

Failed to validate DB connectivity

When you create a domain that includes the Java Required Files (JRF) components, you must select an existing database and provide connection details. The compute instances use this information to connect to the database and provision the JRF database schemas.

Possible causes for this error include:

  • You entered the wrong database password or a plain text password.
  • The database does not allow the compute instances to access its listen port (1521 by default).
    • Oracle Autonomous Transaction Processing - Check your access control list (ACL).
    • Oracle Cloud Infrastructure Database - Check the network security group that was assigned to the database, and the security lists for the subnet on which the database was created.
  • You selected an Oracle Cloud Infrastructure Database running Oracle Database 12c or later, and you did not provide the name of a pluggable database (PDB).

Invalid or overlapping network CIDR

Stack provisioning fails if you specify subnets with overlapping CIDRs or use the same subnet for WebLogic Server and the load balancer.

Example messages:

Error: module.network-wls-public-subnet.oci_core_subnet.wls-subnet: 1 error(s) occurred: oci_core_subnet.wls-subnet: Service error:InvalidParameter. The requested CIDR 10.0.3.0/24 is invalid: subnet ocid1.subnet.oc1.iad.aaan4a with CIDR 10.0.3.0/24 overlaps with this CIDR.. http status code: 400.

Error: module.validators.null_resource.duplicate_lb2_subnet_cidr: : invalid or unknown key: WLSC-ERROR: Load balancer subnet 2 CIDR has to be unique value.

Error: module.validators.null_resource.duplicate_wls_subnet_cidr: : invalid or unknown key: WLSC-ERROR: Weblogic subnet CIDR has to be unique value.

Possible causes for these errors include:

  • You chose to create new subnets for WebLogic Server, the load balancer, or the bastion, and the CIDR you specified for these subnets overlaps with the CIDRs for existing subnets in the same virtual cloud network (VCN).
  • You chose to use an existing subnet when provisioning a stack with a load balancer, and you specified the same subnet for WebLogic Server and the load balancer.
  • You created a JRF-enabled domain, your Oracle Cloud Infrastructure Database and WebLogic domain are in different VCNs, and the VCNs have overlapping CIDRs. For example, you cannot create a WebLogic domain on VCN 10.0.0.0/16 that uses a database on VCN 10.0.0.1/24.

Cannot create a Standard Edition 11g stack with more than 4 nodes

Stack provisioning fails if you select Oracle WebLogic Server Standard Edition and WebLogic Server 11g, and set WebLogic Server Node Count to a value greater than 4.

Example message: WLSC-ERROR: The value for wls_node_count=[6] is not valid for Weblogic 11g Standard Edition. The permissible values are [1-4].

If you require more than 4 nodes, you must select WebLogic Server 12c. Destroy the stack, delete the stack, and then create a new stack with 12c. If you must use WebLogic Server 11g, set WebLogic Server Node Count to a value of 4 or less.

Job is still running or has timed out

Most stack creation jobs for Oracle WebLogic Server for Oracle Cloud Infrastructure should complete within an hour. Some internal provisioning problems might cause the job to run indefinitely until it eventually times out after 24 hours.

After the current Apply job times out, run a new Apply job on the same stack. This will destroy any resources that were created, and then attempt to create the resources again. If the problem occurs again, contact support.

Failed to check database port is open for Exadata DB system

When you create a domain that includes Java Required Files (JRF) components, for Exadata DB systems, the database port open check does not work if the Create DB Security List checkbox is selected. In this case, the provisioning fails if the database subnet has more than five security lists.

So, when provisioning, deselect the Create DB Security List check box to avoid creating an additional security list for the database port in the VCN, and manually open the database port (1521 by default).

Unable to Access the Domain

Troubleshoot problems accessing an Oracle WebLogic Server domain after it's successfully created.

Cannot access the WebLogic Console from the Internet

By default the WebLogic Server Administration Console is accessed through port 7001 or 7002.

To check port access:

  1. Access the Oracle Cloud Infrastructure console.
  2. From the navigation menu, select Networking, and then click Virtual Cloud Networks.
  3. Select the compartment in which you created the domain.
  4. Select the virtual cloud network in which the domain was created.
  5. Select the subnet where the WebLogic Server compute instance is provisioned.
  6. Select the security list assigned to this subnet.
  7. For a domain that's not on a private subnet, make sure the following ingress rules exist:
    Source: 0.0.0.0/0
    IP Protocol: TCP
    Source Port Range: All
    Destination Port Range: 7002
    Source: 0.0.0.0/0
    IP Protocol: TCP
    Source Port Range: All
    Destination Port Range: 7001

    For a domain on a private subnet, set the Source to the CIDR of the bastion instance subnet.

Cannot access the sample application using the load balancer: Not Found

On a domain running Oracle WebLogic Server Standard Edition, the sample application is deployed only to the first Managed Server. If your Standard Edition domain has multiple Managed Servers and you access the sample application using a load balancer, the Managed Servers that aren't hosting the sample application will respond with the code 404 (Not Found).

You can use the WebLogic Server Administration Console to update the targets for the sample application, and add the remaining Managed Servers.

Cannot access applications using the load balancer: Bad Gateway

If you restart the compute instances running your Managed Servers, or you restart the compute instances running the App Gateway, the backend set of the load balancer will temporarily be in an unhealthy state. By default, a load balancer in this state will respond with the code 502 (Bad Gateway). After the WebLogic Server and App Gateway processes are running, the load balancer should return to the OK state.

To check the status of the load balancer and backend servers:

  1. Access the Oracle Cloud Infrastructure console.
  2. From the navigation menu, select Networking, and then click Load Balancers.
  3. Click the load balancer that was created for your domain, prefix-lb.
  4. Click Backend Sets, and then click prefix-lb-backendset.
  5. Click Backends, and then check the state of each backend.
  6. Access the WebLogic compute instances using a secure shell (SSH) client. Check that the Managed Server process is listening on its assigned port (the default is 7003).
    curl -s -o /dev/null -w "%{http_code}\n" http://private_ip:7003

    A 404 response indicates that the Managed Server is running.

  7. If you enabled authentication with Oracle Identity Cloud Service, then access the App Gateway compute instances using an SSH client. Check that the App Gateway process is listening on its assigned port (the default is 9999).
    curl -s -o /dev/null -w "%{http_code}\n" http://private_ip:9999

    A 404 response indicates that the App Gateway is running.

See Managing Backend Servers in the Oracle Cloud Infrastructure documentation.

Cannot access the Fusion Middleware Control Console from the Internet

If you enabled authentication with Oracle Identity Cloud Service on a WebLogic Server 12.2.1.4 domain, you might be redirected to an error page when you try to log in to the Fusion Middleware Control Console.

Example message:

<Error> <oracle.help.web.rich.OHWFilter>
<BEA-000000> <ADFSHARE-00120: Error encountered while creating the MDS
Session. Application state will be reset. Please logout and log back in if
problem persists.
oracle.adf.share.ADFShareException: ADFSHARE-00120: Error encountered while
creating the MDS Session. Application state will be reset. Please logout and
log back in if problem persists.

Add the Cloud Gate App Role to your confidential application. Then restart your WebLogic Server domain and try logging in to the Fusion Middleware Control Console again.

See Create a Confidential Application.

Get Additional Help

Use online help, email, customer support, and other tools if you have questions or problems with Oracle WebLogic Server for Oracle Cloud Infrastructure.

For general help with Oracle Cloud Marketplace, see How Do I Get Support in Using Oracle Cloud Marketplace.