1. Administering Oracle Advanced Authentication and Oracle Adaptive Risk Management
  2. Installing Oracle Advanced Authentication, Oracle Adaptive Risk Management, and Oracle Universal Authenticator
  3. Procedure for Installing OAA, OARM, and OUA
  4. Troubleshooting the Installation

4.9 Troubleshooting the Installation

This section provides troubleshooting tips for installing OAA, OARM, and OUA.

Podman issues during OAA Management Container installation

  • Podman fails to load the OAA images in the tar file due to image or file format errors. For example:
    Storing signatures
Getting image source signatures
Copying blob 01092b6ac97d skipped: already exists
Copying blob dba9a6800748 skipped: already exists
Copying blob bae273a35c58 skipped: already exists
Copying blob 7f4b55b885b0 skipped: already exists
Copying blob 93e8a0807a49 skipped: already exists
Copying blob fa5885774604 skipped: already exists
Copying blob 3b8528487f10 skipped: already exists
Copying blob 3a1c2e3e35f4 [==========================>-----------] 213.8MiB / 298.1MiB
Copying blob 6d31843e131e [=================================>----] 210.5MiB / 236.5MiB
Copying blob f35b9630ef38 [===========>--------------------------] 213.8MiB / 672.2MiB
Copying blob ef894c2768e3 done
Copying blob 846fd069f886 [==========>---------------------------] 197.7MiB / 672.2MiB
Copying blob 257c48b76c82 done
Error: payload does not match any of the supported image formats (oci, oci-archive, dir, docker-archive)
    This may happen because of lack of free space in the root partition of the installation host (podman stores temporary files under /var/tmp), or because the podman version is not 3.3.0 or later. If this error occurs, remove all files under /var/tmp before retrying the installation once the issues have been addressed.
  • Podman fails to load the OAA images in the tar file due to permissions issues. For example:
    Using image release files ./releaseimages.txt and ./nonreleaseimages.txt...
tee: ./oaainstall-tmp/run.log: Permission denied
Using install settings from ./installOAA.properties.
tee: ./oaainstall-tmp/run.log: Permission denied
Checking kubectl client version...
WARNING: version difference between client (1.23) and server (1.21) exceeds
the supported minor version skew of +/-1
tee: ./oaainstall-tmp/run.log: Permission denied
kubectl version required major:1 minor:18, version detected major:1 minor:23
tee: ./oaainstall-tmp/run.log: Permission denied

    This may happen if you extract the zip file as one user and run installManagementContainer.sh as a different user who doesn't have permissions. In this situation remove the $WORKDIR/oaaimages/oaa-install/oaainstall-tmp directory and retry the install with the same user who extracted the zip file.

  • Podman failed to load the OAA images in the previous attempt to install and now it won't pull/tag/push of all required images. In this situation remove the $WORKDIR/oaaimages/oaa-install/oaainstall-tmp directory and retry.

OAA Management chart installation failure

If the OAA management chart installation fails with the following:
Executing 'helm install ...  oaamgmt charts/oaa-mgmt'.
Continue? [Y/N]:
y
Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Deployment.spec.template.spec.containers[0]): unknown field "volumMounts" in io.k8s.api.core.v1.Container
it is likely that the manifest files for the OAA management chart got corrupted. Copy installOAA.properties, cert.p12, and trust.p12 to a safe location, remove the install directory $WORKDIR/oaaimages/oaa-install, extract the <OAA_Image>.zip and restart the installation.

Installation script times out waiting for OAA Management Container pod to start

If you see the following error:
NAME                                     READY   STATUS              RESTARTS   AGE
oaamgmt-oaa-mgmt-74c9ff789d-wq82h   0/1     ContainerCreating   0          2m3s
Waiting 15 secs for OAA mgmt deployment to run...
Executing 'kubectl get pods oaamgmt-oaa-mgmt-74c9ff789d-wq82h -n oaans'...
NAME                                     READY   STATUS              RESTARTS   AGE
oaamgmt-oaa-mgmt-74c9ff789d-wq82h   0/1     ContainerCreating   0          2m18s
Waiting 15 secs for OAA mgmt deployment to run...
...
OAA mgmt pod is not running after 450 secs, cannot proceed with install.
Critical error, exiting. Check ./oaainstall-tmp/run.log for additional information.
then run the following commands to get additional information:
$ kubectl get pods -n oaans
$ kubectl describe pod oaamgmt-<pod> -n oaans
  • In case of NFS errors, verify that the NFS volume information in installOAA.properties is correct. In this situation kubectl describe will show the following:
    Output: mount.nfs: mounting <ipaddress>:/scratch/oaa/scripts-creds failed, reason given by server: No such file or directory
  Warning  FailedMount  15s  kubelet, <ipaddress>  Unable to attach or mount volumes: unmounted volumes=[oaamgmt-oaa-mgmt-configpv oaamgmt-oaa-mgmt-credpv oaamgmt-oaa-mgmt-logpv], unattached volumes=[oaamgmt-oaa-mgmt-configpv oaamgmt-oaa-mgmt-credpv oaamgmt-oaa-mgmt-logpv oaamgmt-oaa-mgmt-vaultpv default-token-rsh62]: timed out waiting for the condition
  • In case of image pull errors verify that the image pull secret (dockersecret) was created correctly, and that the properties install.global.repo, install.global.image.tag, and install.global.imagePullSecrets\[0\].name in installOAA.properties are correct. In this situation kubectl describe pod will show the following:
    Warning  Failed     21s (x3 over 61s)  kubelet, <ipaddress>  Error: ErrImagePull
Normal   BackOff    7s (x3 over 60s)   kubelet, <ipaddress>  Back-off pulling image "container-registry.example.com/oracle/shared/oaa-mgmt:<tag>"
Warning  Failed     7s (x3 over 60s)   kubelet, <ipaddress>  Error: ImagePullBackOff
  • In case of timeouts with no apparent error it may be possible that the cluster took too long to download the OAA management image. In this case the management pod will eventually start but the installation will abort. If this happens, delete the OAA management helm release using helm delete oaamgmt -n oaans and rerun the installation script.

General failures during OAA.sh

If the OAA.sh deployment fails at any stage during the install you can generally fix the issue and rerun OAA.sh. The install performs a number of checks against the Database, OAuth, and Vault. If re-running the OAA.sh fails at these checks because the Database schema, OAuth configuration, or Vault already exists, then set these properties in the installOAA.properties before trying the OAA.sh again:
  • If Database schema is already present:
    • database.createschema=false
  • If OAuth configuration is already present:
    • oauth.createdomain=false
    • oauth.createresource=false
    • oauth.createclient=false
  • If Vault configuration is present:
    • vault.create.deploy=false

OAuth creation fails during OAA.sh

During the installation, the OAuth domain, client, and resource server are created. If they fail, check if the parameters for OAuth are correct. See Configuring Oracle Access Management OAuth.

OAuth check fails during OAA.sh

This occurs if the httpd.conf and mod_wl_ohs.conf files are not updated. To update the values, see Configuring Oracle Access Management OAuth.

During OAA.sh installation fails because of pods in Container Creating status

Run the following command to check the logs. For example:
kubectl logs oaainstall-email-6fd7c9b9dd-lr5lm -n oaans
If the logs do not provide the required details about the error, run the describe pod command. For example:
kubectl describe pod oaainstall-email-6fd7c9b9dd-lr5lm -n oaans

During OAA.sh pods fail to start and show CrashLoopBackOff

Run the kubectl logs <pod> -n <namespace> command against the pods showing the error. The following may be one of the reasons for the error:

Pods were not able to connect to http://www.example.oracle.com:7791/.well-known/openid-configuration because the PathTrim and PathPrepend in the mod_wl_ohs.conf for that entry were not updated. See Configuring Oracle Access Management OAuth.

OAA.sh installation timed out but pods show as running

If the OAA installation timed out but the OAA pods show no errors and eventually end up in running state, it is possible that the cluster took too long to download the OAA images. In this case the OAA pods will eventually start but the installation will not complete. If this happens, clean up the installation and rerun the installation script.

kubectl reports "Unable to connect to the server: net/http: TLS handshake timeout"

Possible causes are:
  • Proxies are defined in the environment and the no_proxy" environment variable does not include the cluster nodes. To resolve the issue the cluster node IPs or hostnames must be added to the no_proxy environment variable.
  • The kube config file ~/.kube/config or /etc/kubernetes/admin.conf is not valid.

Unable to delete the OAA domain from OAuth during cleanup

List all clients and resources within the domain and delete each one of them before deleting the domain:
  1. Encode the OAM administrator user and its password by using the command:
    echo -n <username>:<password> | base64
    For example: 
    echo -n weblogic:<password> | base64
    This value should be used for <ENCODED_OAMADMIN> in the example below.
  2. Run the following:
    $ curl --location --request DELETE 'http://<OAuth_Host>:<OAuth_port>/oam/services/rest/ssa/api/v1/oauthpolicyadmin/oauthidentitydomain?name=OAADomain' \
--header 'Authorization: Basic <ENCODED_OAMADMIN>'
OAuth Identity Domain is not empty. Kindly remove (resource/client) entities from identity domain
$ curl --location --request GET 'http://<OAuth_Host>:<OAuth_port>/oam/services/rest/ssa/api/v1/oauthpolicyadmin/client?identityDomainName=OAADomain' --header 'Content-Type: application/json' --header 'Authorization: Basic <ENCODED_OAMADMIN>'
$ curl --location --request GET 'http://<OAuth_Host>:<OAuth_port>/oam/services/rest/ssa/api/v1/oauthpolicyadmin/application?identityDomainName=OAADomain' --header 'Content-Type: application/json' --header 'Authorization: Basic <ENCODED_OAMADMIN>'

Error 'jq: error: Invalid escape at line 1, column 6` Creating tap partner in OAA

If you see the following error running OAA.sh then the oua.tapAgentFilePass value was not was not set in base64:
jq: error: Invalid escape at line 1, column 6 (while parsing '"\�"') at <top-level>, line 1:
.agentName |= if . == "" then "MFAOAAPartner17ohsapr9" else . end |             .privateKey |= if . == "" then "CECECECE0000000200000001..etc..  
jq: 1 compile error
Creating tap partner in OAA

To solve this problem, set the value to the base64 version of the password and run the OAA.sh again. See, Oracle Universal Authenticator Configuration.

Bad Oracle Access Manager Request in DRSS Logs

If you see the following error in the DRSS pod logs:
<DATE> Thread[http-thread-34,5,server]: INFO oracle.security.am.drss.handler.oam.OAMHandler parseOAMResponse Exception during parseOAMResponse Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (String)"<html><head><title>Bad Oracle Access Manager Request</title><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8"></head><body><h1>Bad Oracle Access Manager Request</h1><p>Unable to process the request due to unexpected error.</p></body></html>
Then the oua.oamRuntimeEndpoint was either set incorrectly in the installOAA.properties, not set to the fully qualified hostname of the OAM server, or the OAM server is not functioning correctly.