C Troubleshooting Oracle Key Vault

Oracle provides checklists and tips for commonly encountered errors that will help you install and deploy Oracle Key Vault.

C.1 Oracle Key Vault Pre-Installation Checklist

The pre-installation checklist covers all the requirements to successfully install Key Vault.

Table C-1 Oracle Key Vault Pre-Installation Checklist

Item# Check Task

1. [ x ]

System requirements

Confirm that you have enough CPU, memory, and disk as described in System Requirements.

2. [ x ]

Open all the required network ports in your firewall

For details on network ports, see Network Port Requirements.

3. [ x ]

Supported endpoint platforms

See Supported Endpoint Platforms.

4. [ x ]

Set the COMPATIBLE initialization parameter for online master key (previously TDE direct connect).

Guidance for setting this parameter for Oracle Database 11.2.0.0 and later is in Supported Endpoint Platforms.

5. [ x ]

Get a fixed IP address, network mask, and gateway address from your network administrator.

You will need this information for Step 9 in Installing the Oracle Key Vault Appliance Software

C.2 Integrating Oracle Key Vault with Oracle Audit Vault and Database Firewall

You can consolidate audits between Oracle Audit Vault and Database Firewall (AVDF) with Oracle Key Vault.

To do this, you must integrate Audit Vault and Database Firewall with Oracle Key Vault.

C.2.1 Step 1: Check the Environment

Before you begin the integration, you should ensure that the required components are all in place.

  1. Ensure that Oracle Audit Vault and Database Firewall is properly installed and configured, and that it has the Audit Vault administrator and auditor user accounts.
  2. In Oracle Key Vault, enable SSH access.

    Log in to the Oracle Key Vault management console as a user who has the System Administrator role. Select the System Settings tab, then under Network Services, select SSH Access. Select IP address(es) and then enter only the IP addresses that you need. Click Save.

C.2.2 Step 2: Register Oracle Key Vault as a Secured Target with AVDF

You must register the Oracle Key Vault server as a secured target on the Oracle Audit Vault and Database Firewall server.

  1. Log in to the Oracle Audit Vault Server console as an administrator (avadmin).
  2. Click the Secured Targets tab, and then select Register.
  3. In the New Secured Target Name field, enter the name of the Oracle Key Vault server.
  4. For Secured Target Type, enter Oracle Key Vault.
  5. In the Add Secured Target Location section, click Advanced, and then add the following for the connect string:
    jdbc:oracle:thin:@//127.0.0.1:1521/dbfwdb
  6. For the user name and password, enter the following: avcollector/integration_password
  7. Add the following collection attribute:
    av.collector.securedTargetVersion 12.2.0.0.0
    av.collector.TimeZoneOffset +5:30 
    

    You can use any TimeZoneOffset setting that applies to your situation.

C.2.3 Step 3: Register Oracle Key Vault as a Host with AVDF

Next, you must register the Oracle Key Vault server as a host on the Audit Vault and Database Firewall server.

  1. Log in to the Oracle Audit Vault Server console as an administrator (avadmin).
  2. Click the Hosts tab, then select Register.
  3. In the Host Name field, enter the name of the Oracle Key Vault server.
  4. In the Host IP field, enter the IP address of the Oracle Key Vault server.
  5. Click Save.
  6. When the new entry appears with an agent activation key, copy this value and store it in a safe place.
    You will need this agent activation key value later on in these steps.

C.2.4 Step 4: Download the AVDF Agent and Upload it to Oracle Key Vault

You must next download the Oracle Audit Vault and Database Firewall agent and then upload it to the Oracle Key Vault server.

  1. Log in to the Oracle Audit Vault Server console as an administrator (avadmin).
  2. Select the Agent tab, and then select Agent Release.
  3. Select the agent, which should be the first item in the list of agents.
  4. Save this item as agent.jar.
  5. Use scp to upload agent.jar to the Oracle Key Vault server.

C.2.5 Step 5: Install the AVDF agent.jar File on the Oracle Key Vault Server

At this stage, you are ready to install the agent.jar file on the Oracle Key Vault server.

  1. Log on to the Oracle Key Vault command line as user support and then su to root to create the directory, get the agent, and set permissions:
    cd /usr/local/okv
    mkdir avdf
    cp /home/support/agent.jar /usr/local/okv/avdf
    chown oracle:oinstall /usr/local/okv/avdf /usr/local/okv/avdf/*
  2. su to oracle and extract the agent:
    su - oracle
    cd /usr/local/okv/avdf
    java -jar agent.jar -d /usr/local/okv/avdf
  3. As oracle, start the agent and enter the Oracle Audit Vault and Database Firewall agent activation key that you generated earlier when you registered the Oracle Key Vault server as a host on the Audit Vault and Database Firewall server.
    cd /usr/local/okv/avdf/bin
    ./agentctl start -k
  4. In the Oracle Key Vault management console, enable the database user.
    1. Select System, then System Settings.
    2. In the Oracle Audit Vault Integration section, check Enable.
    3. In the Password and Reenter Password fields, enter the integration password.
      This is the password of the user in the database that Audit Vault and Database Firewall will use to extract audit records.
    4. Click Save.
  5. Because you no longer need to copy files from one server to another, disable SSH access.

    Log in to the Oracle Key Vault management console as a user who has the System Administrator role. Select the System Settings tab, then under Network Services, select Disabled. Click Save. Restart the Oracle Key Vault server by clicking Reboot on the top right.

C.2.6 Step 6: Add the Oracle Key Vault Audit Trail to AVDF

Next, you can add the audit trail to Oracle Audit Vault and Database Firewall.

  1. Log in to the Oracle Audit Vault Server console as an administrator (avadmin).
  2. Select Secured Targets, and then under Monitoring, select Audit Trails.
  3. To add the new audit trail, click Add.
  4. In the Collection Host field, specify the host computer where you installed the agent.jar agent file. (You can use the Search icon to find this host computer.)
  5. In the Secured Target Name field, enter the name of the secured target.
  6. From the Audit Trail Type drop-down list, select TABLE. Then choose the secured target and host that you created.
  7. In the Trail Location field, enter keyvault.audit_trail.
  8. Click Save.
  9. Select this audit trail and then start the audit trail collection.
    In the Audit Trails page, select the audit trail and then click Start.

C.2.7 Step 7: View Oracle Key Vault Audit Data Collected by AVDF

After you have completed the integration and are collecting data, you can view data collected by Oracle Audit Vault and Database Firewall.

  1. Log in to the Oracle Audit Vault Server console as an auditor (avauditor).
  2. Select the Reports tab.
  3. Select All Activity.
  4. Select All Activity Report.

C.3 RESTful Services Troubleshooting Help

The Oracle Key Vault log files capture all the error messages sent by the server.

The error messages are written to the /var/log/messages file. The first debugging step is to read the messages file.
  1. Log in to the Oracle Key Vault server.
  2. As the root user, check for log file errors as follows:
    root# vi /var/log/messages

C.4 Error: Cannot Open Keystore Message

The Cannot Open Keystore error can appear when you try to upload a Java keystore to the Oracle Key Vault server.

To remedy this problem, try the following solutions:

  • Ensure that the PATH environment variable has been correctly set.
  • Check where the keytool and Java are pointing to, by entering the following commands in a shell:
    which keytool
    which java
    
  • Ensure that you are using Oracle Java.

C.5 KMIP Error: Invalid Field

The Invalid Field KMIP error can occur when you are trying to upload Oracle wallets to virtual wallets on multiple endpoints.

The KMIP error can occur for other scenarios, but this scenario is the most common.

The steps that result in this error are as follows:

  1. You configure two or more endpoints (for example, Endpoint A and Endpoint B) to share a wallet (Oracle Wallet C), and hence also share the wallet keys.
  2. You register Endpoints A and B with Oracle Key Vault.
  3. You create a default wallet (Virtual Wallet A) for Endpoint A and then a default wallet (Virtual Wallet B) for Endpoint B. Each virtual wallet is accessible only to the corresponding endpoint. For example, Endpoint B has no access to Virtual Wallet A.
  4. You upload Oracle Wallet C into Virtual Wallet A on Endpoint A.
  5. You attempt to upload Oracle Wallet C from Endpoint B into Virtual Wallet B Endpoint B.

The KMIP error occurs because there are two copies of the same key being created and Endpoint B does not have visibility for both. If Endpoint A tries to upload the first key again, Oracle Key Vault detects this action and accounts for it. But because in Step 5, Endpoint B is not allowed to see the first key, Oracle Key Vault is unable to perform the necessary harmonization for the two Oracle wallets.

This is expected behavior. To avoid the Invalid Field, create an endpoint group so that you can share the wallet with multiple endpoints.

C.6 WARNING: Could Not Store Private Key Errors

If you upload two keystores with the same file name but different contents, a WARNING: Could not store private key error appears.

This error occurs if you use the same alias (-alias slserver) in each keytool command. When you download two such keystores that have the same alias, the okvutil download process ignores the second one because the JKS aliases must be unique.
  • To remedy this problem, download the second keystore using a unique alias.

C.7 Errors After Upgrading Oracle Key Vault

Some errors that appear after the upgrade can be ignored.

After you perform an upgrade of Oracle Key Vault on an standalone server, ORA-1109: database does not open, ORA-00313: open failed for members, and ORA-00312: online log 3 thread 1 error messages may appear in the /var/log/messages log file.

You can safely ignore these messages. Error messages also appear in the /var/log/debug file.

C.8 Error: Failed to Open Wallet

An Failed to Open Wallet error can appear if you attempt to use an online master key.

If you are attempting to use an online master key (previously called TDE direct connect) and encounter this error, then first check your environment variable ORACLE_BASE. In an Oracle Real Application Clusters environment, you must perform this step on all database instances.

You must also set the ORACLE_SID, ORACLE_HOME, and OKV_HOME environment variables needed by the PKCS #11 library as follows:

  1. Log in to the server where the PKCS #11 library resides and then set the ORACLE_SID, ORACLE_HOME, and OKV_HOME environment variables.
  2. Log in to the database instance using SQL*Plus as a user with the SYSDBA administrative privilege.
    sqlplus sys / as sysdba
    Enter password: password
  3. Shut down the database.
    For example:
    SHUTDOWN IMMEDIATE
  4. From the command line, restart the database service.
    su - oracle
    lsnrctl start
  5. In SQL*Plus, restart the database.
    STARTUP

C.9 Transaction Check Error: Diagnostics Generation Utility

If you are trying to perform an upgrade of Oracle Key Vault, a transaction check error may appear.

For example:

file /usr/local/dbfw/etc/dbfw-diagnostics-package.yml from install of
appliance-18.3.0.0.0-52_190425.2253.d.x86_64 conflicts with file from
package okv-diagnostic-12.2.0.8.0-40_181013.1730.x86_64 

The problem is that the diagnostic generation utility interferes with the upgrade process. You must remove the diagnostic generation utility before you can perform the upgrade.

C.10 Fast-Start Failover (FSFO) Suspended (ORA-16818)

An ORA-16818: Fast-Start Failover suspended error can appear as a result of a fast start failover operation failing.

If the primary server was shut down gracefully in a controlled way, such as by clicking a Power Off button instead of manually turning off the computer, then a fast start failover cannot be performed and the ORA-16818: Fast-Start Failover suspended error appears. In a graceful shutdown operation, the primary server's failover status goes into a suspended state with the standby waiting indefinitely for the primary server to be available. This is the expected behavior for a fast-start failover (FSFO) operation, as defined by Oracle Data Guard avoid a split brain scenario. By design, a fast-start failover operation error occurs only when the primary server shuts down unexpectedly. If you perform a SHUTDOWN IMMEDIATE or SHUTDOWN NORMAL command in SQL*Plus, then the FSFO does not occur because the database shuts down gracefully.

C.11 SSH Tunnel Add Failure

While you are configuring the SSH tunnel, a Failed to establish SSH tunnel. Refer to Oracle documentation error may appear.

You might get the following error message while trying to set up the SSH tunnel:

The failure may be due to one or more of the following problems:

  • The following settings may be invalid:

    • Invalid IP address
    • Invalid port
    • Invalid user name
  • The public SSH Oracle Key Vault key was not copied to the authorized_keys file of the okv user on the Database as a Service instance.

  • The Database as a Service instance is not reachable because of network overload.

To remedy this problem, check your input values and connection and retry.

C.12 Error: Provision Command Fails if /usr/bin/java Does Not Exist

The RESTful service command to provision an endpoint fails if the soft link /usr/bin/java does not exist or points to an incorrect Java directory.

To remedy this problem, ensure that the Java version is 1.7.21 or later. You can create a soft link to the Java home directory as follows:
ln -s Java_home_directory/bin/java /usr/bin/java

C.13 TDE Endpoint Integration Issues

Several issues related to Transparent Data Encryption (TDE) endpoint integration problems can arise.

Common Transparent Data Encryption (TDE) endpoint integration problems caused by installation errors, svrctl misuse, and the mismanagement of security objects in a primary-standby environment can arise.

  • Installing the Oracle Key Vault library: You must run root.sh to install the Oracle Key Vault library only once on a computer that has multiple Oracle databases. During an upgrade you must upgrade the library only after you have shut down all the associated endpoints. Oracle Key Vault servers are backward compatible with endpoint libraries.
  • Using svrctl to manage the database: If you use the svrctl utility to manage the database, remember that svrctl can set the ORACLE_BASE environment variable to NULL. Oracle recommends that you set ORACLE_BASE to ORACLE_HOME if ORACLE_BASE is not used in your environment.
  • Managing security objects the same way in a primary-standby environment: In a primary-standby configuration, ensure that both the primary and standby servers use the same mechanism to manage security objects. These servers should either both use a wallet or both use Oracle Key Vault.

C.14 Failover Situations in Primary-Standby Mode

Failover situations can occur with or without read-only restricted mode or during a planned shutdown operation for both primary and standby servers.

C.14.1 About Failover Situations in Primary-Standby Mode

Failover situations in primary-standby node can occur with read-only restricted mode disabled, and with read-only restricted mode enabled.

The types of failover situations are as follows:

  • Planned shutdown of the primary server: A system administrator shuts down the primary server during an upgrade or maintenance window.

  • Planned shutdown of the standby server: A system administrator shuts down the standby server during an upgrade or maintenance window.

  • Unplanned shutdown of the primary server: The primary server is offline due to unforeseen circumstances such as power loss or network failure.

  • Unplanned shutdown of the standby server: The standby server is offline due to unforeseen circumstances such as power loss or network failure.

C.14.2 Failover Situations Without Read-Only Restricted Mode

If read-only restricted mode is not used, then various failover situations may occur in Oracle Key Vault.

C.14.2.1 Primary Server: Planned Shutdown During an Upgrade

In a failover, the use of read-only restricted mode affects a planned shutdown of a primary server during an upgrade.

If read-only restricted mode is not used, when the primary server goes offline during an upgrade, then the standby server waits in read-only mode for the primary server to return online. During the upgrade, you cannot access the Oracle Key Vault management console.

  • Recovery process: When the primary server is back online after the upgrade the standby server will automatically synchronize with the primary server. The primary and standby servers retain their earlier roles, and both servers will continue to operate in primary-standby mode.
  • Primary server state: Down
  • Standby server state: Up
  • Does failover occur? No
C.14.2.2 Primary Server: Planned Shutdown During Maintenance

In a failover, not using read-only restricted mode affects the planned shutdown of a primary server during maintenance.

If read-only restricted mode is not used, when the primary server is powered off or restarted during maintenance, then the standby server takes over from the primary server.

  • Recovery process: The standby server is now the new primary server. When the old primary server is back online after maintenance, it will automatically synchronize with the new primary server and take over the role of standby server. Both servers will continue to operate in primary-standby mode. Be aware that when the primary server is offline, data replication is disabled. If the new primary server goes offline before synchronizing with the new standby server, it may cause a loss of critical data.
  • Primary server state: Down
  • Standby server state: Up
  • Does failover occur? Yes
C.14.2.3 Standby Server: Planned Shutdown

In a failover, not using read-only restricted mode affects a planned shutdown in a standby server.

If read-only restricted mode is not used, when the standby server is powered off during upgrade or maintenance, the primary server continues operating as the primary server. Read and write operations are allowed.

  • Recovery process: When the standby server is back online post-upgrade or post-maintenance, the primary server will automatically synchronize with the standby server. The primary and standby servers retain their earlier roles, and both servers will continue to operate in primary-standby mode. Be aware that when the standby server is offline, data replication is disabled. If the primary server goes offline before synchronizing with the standby server, then it may cause a loss of critical data.
  • Primary server state: Up
  • Standby server state: Down
  • Does failover occur? No
C.14.2.4 Primary Server: Unplanned Shutdown

In a failover, not using read-only restricted mode affects an unplanned shutdown in a primary server.

If read-only restricted mode is not used, when the primary server goes offline because of power loss, network failure, or hardware failure, the standby server waits for the duration specified in the Fast Start Failover Threshold field on the Configure High Availability page in the Oracle Key Vault management console. If the primary server cannot be reached after the specified duration has elapsed, then the standby server takes over from the primary server.

  • Recovery process: The standby server is now the new primary server. Rectify the failure that affected the primary server by restarting the server or restoring network connectivity. When the primary server is back online, it will automatically synchronize with the new primary server and take over the role of standby server.
  • Primary server state: Down
  • Standby server state: Up
  • Does failover occur? Yes
C.14.2.5 Standby Server: Unplanned Shutdown

In a failover, not using read-only restricted mode affects a standby server during an unplanned shutdown.

If read-only restricted mode is not used, when the standby server goes offline because of power loss, network failure, or hardware failure, then the primary server becomes unavailable. All operations are disabled.

  • Recovery process: Rectify the failure that affected the standby server by restarting the server or restoring network connectivity. When the standby server is back online, it will automatically synchronize with the primary server. You cannot re-establish synchronization or network connectivity between the primary and standby servers, contact Oracle Support.
  • Primary server state: Up
  • Standby server state: Down
  • Does failover occur? No

C.14.3 Failover Situations with Read-Only Restricted Mode

The use read-only restricted mode affects failover operations in Oracle Key Vault.

C.14.3.1 Primary Server: Planned Shutdown During an Upgrade

In a failover, not using read-only restricted mode affects a planned shutdown of a primary server during an upgrade

When the primary server goes offline during an upgrade, the standby server enters read-only restricted mode and waits for the primary server to come back online. During the upgrade, you cannot access the Oracle Key Vault management console.

  • Recovery process: When the primary server is back online post-upgrade, the standby server will automatically synchronize with the primary server. The primary and standby servers retain their earlier roles, and both servers will continue to operate in primary-standby mode.
  • Primary server state: Down
  • Standby server state: Up
  • Does failover occur? No
C.14.3.2 Primary Server: Planned Shutdown During Maintenance

In a failover, using read-only restricted mode affects the planned shutdown of a primary server during maintenance.

When the primary server is powered off or restarted during maintenance, the standby server enters read-only restricted mode, and takes over from the primary server. The Oracle Key Vault management console displays a warning.

  • Recovery process: The standby server is now the new primary server. When the old primary server is back online after maintenance, it will automatically synchronize with the new primary server and take over the role of standby server. Both servers will continue to operate in primary-standby mode.
  • Primary server state: Down
  • Standby server state: Up
  • Does failover occur? Yes
C.14.3.3 Standby Server: Planned Shutdown

In a failover, using read-only restricted mode affects a planned shutdown in a standby server.

When the standby server is powered off during upgrade or maintenance, the primary server enters read-only restricted mode, and continues operating as the primary server. The Oracle Key Vault management console displays a warning.

  • Recovery process: When the standby server is back online post-upgrade or after maintenance, the primary server will automatically synchronize with the standby server. The primary and standby servers retain their earlier roles, and both servers will continue to operate in primary-standby mode.
  • Primary server state: Up
  • Standby server state: Down
  • Does failover occur? No
C.14.3.4 Primary Server: Unplanned Shutdown

In a failover, using read-only restricted mode affects an unplanned shutdown in a primary server.

When the primary server goes offline because of power loss, network failure, or hardware failure, then the standby server waits for the duration specified in the Fast Start Failover Threshold field on the Configure Primary-Standby page in the Oracle Key Vault management console. If the primary server is not reachable after the specified duration has elapsed, then the standby server enters read-only restricted mode, and takes over from the primary server.

  • Recovery process: The standby server is now the new primary server. Rectify the failure that affected the primary server by restarting the server or restoring network connectivity. When the primary server is back online, it will automatically synchronize with the new primary server and take over the role of standby server.
  • Primary server state: Down
  • Standby server state: Up
  • Does failover occur? Yes
C.14.3.5 Standby Server: Unplanned Shutdown

In a failover, using read-only restricted mode affects a standby server during an unplanned shutdown.

When the standby server goes offline because of power loss, network failure, or hardware failure, the primary server waits for the duration specified in the Fast Start Failover Threshold field on the Configure Primary-Standby page of the Oracle Key Vault management console. If the standby server is not reachable after the specified duration has elapsed, the primary server enters read-only restricted mode, and continues operating as the primary server.

  • Recovery process: Rectify the failure that affected the standby server by restarting the server or restoring network connectivity. When the standby server is back online, it will automatically synchronize with the primary server. If you cannot re-establish synchronization or network connectivity between the primary and standby servers, then contact Oracle Support.
  • Primary server state: Up
  • Standby server state: Down
  • Does failover occur? No

C.15 Performing a Planned Shutdown

A user who has the System Administrator role can perform planned shutdowns during an upgrade or a maintenance window.

C.15.1 Primary Server Planned Shutdown

A user who has the System Administrator role can plan a shutdown of the primary server during an upgrade or a maintenance window.

C.15.1.1 Performing a Primary Server Planned Shutdown During an Upgrade

You must upgrade both servers in a pair when you perform a primary server shutdown.

During an upgrade, failover does not occur. The primary and standby servers retain their earlier roles after the primary server is back online post-upgrade. After an upgrade, the primary and standby server retain their old roles.
  • To perform a primary upgrade, upgrade both of the Oracle Key Vault server pair used in the primary-standby configuration.
C.15.1.2 Performing a Primary Server Planned Shutdown During Maintenance

To perform a primary server planned shutdown during maintenance, power off or restart Oracle Key Vault.

  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
  2. Select the System tab, and then select System Settings.
  3. Do one of the following:
    • Click the Power Off button.
    • Click the Reboot button.
    When the primary server is shut down, the standby server waits for the duration specified in the Fast Start Failover Threshold field on the Configure Primary-Standby page. When the duration has elapsed, the standby server will failover and take over from the primary server. The standby server is now the new primary server.
After a maintenance window, the primary and standby server switch roles. When the old primary server is back online after maintenance, it will take over the role of standby server.

C.15.2 Standby Server Planned Shutdown

A user who has the System Administrator role can plan a shutdown of the primary server during an upgrade or a maintenance window.

During upgrade, the standby server shuts down automatically, and no manual steps are necessary.

C.15.2.1 Performing a Standby Server Planned Shutdown During an Upgrade

You must upgrade both servers in a pair when you perform a standby server shutdown.

When you restart the standby server during the upgrade, the upgrade script initiates an automatic shutdown. There are no manual steps to be performed after the standby server is restarted.
  • To perform a standby upgrade, upgrade both of the Oracle Key Vault server pair used in the primary-standby configuration.
C.15.2.2 Performing a Standby Server Planned Shutdown During Maintenance

You can perform a standby server planned shutdown during maintenance from the standby server using SSH.

  1. Log in to the standby server terminal using ssh as user support, then switch user (su) to root.
  2. Switch user (su) to oracle.
  3. Log in to the standby database instance as a user who as the ALTER DATABASE system privilege.
    For example:
    sqlplus sec_admin
    Enter password: password
  4. Execute the ALTER DATABASE statement as follows:
    ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
  5. Shut down the database.
    SHUTDOWN IMMEDIATE
  6. Power off the standby server.
    1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
    2. Select the System tab, and then select System Settings.
    3. Click the Power Off button.