1. Administrator's Guide
  2. Troubleshooting Oracle Key Vault

C Troubleshooting Oracle Key Vault

Oracle provides checklists, tips, instructions, and how-tos for common errors to help you smoothly install and deploy Oracle Key Vault.

C.1 Finding Log File Errors

The Oracle Key Vault log files capture all the error messages sent by the server.

The error messages are written to the /var/log/messages file. The first debugging step is to read the messages file.
  1. Log in to the Oracle Key Vault server.
  2. As the root user, check for log file errors as follows: 
    root# vi /var/log/messages

Parent topic: Troubleshooting Oracle Key Vault

C.2 Directory Permissions for Oracle Key Vault Migration

The migration of a TDE-enabled database from local wallet to Oracle Key Vault may fail even if the permissions for the /opt/oracle directory are set to 700.

Running root.sh creates the directory structure for Oracle Database to get the PKCS#11 library that is under the /opt/oracle directory. There is a chance that the directory structure may already exist and could have permissions different from the one set by root.sh.

To remedy this problem, check if the /opt/oracle directory already exists and then ensure that the permissions for every directory in the /opt/oracle/extapi/64/hsm/oracle/1.0.0/liborapkcs.so path are set to 755.

Parent topic: Troubleshooting Oracle Key Vault

C.3 Error: Cannot Open Keystore Message

The Cannot Open Keystore error can appear when you try to upload a Java keystore to the Oracle Key Vault server.

To remedy this problem, try the following solutions:

  • Ensure that the PATH environment variable has been correctly set.
  • Check where the keytool and Java are pointing to, by entering the following commands in a shell: 
    which keytool
which java
  • Ensure that you are using Oracle Java.

Parent topic: Troubleshooting Oracle Key Vault

C.4 KMIP Error: Invalid Field

The Invalid Field KMIP error can occur when you are trying to upload Oracle wallets to virtual wallets on multiple endpoints.

The KMIP error can occur for other scenarios, but this scenario is the most common.

The steps that result in this error are as follows:

  1. You configure two or more endpoints (for example, Endpoint A and Endpoint B) to share a wallet (Oracle Wallet C), and hence also share the wallet keys.
  2. You register Endpoints A and B with Oracle Key Vault.
  3. You create a default wallet (Virtual Wallet A) for Endpoint A and then a default wallet (Virtual Wallet B) for Endpoint B. Each virtual wallet is accessible only to the corresponding endpoint. For example, Endpoint B has no access to Virtual Wallet A.
  4. You upload Oracle Wallet C into Virtual Wallet A on Endpoint A.
  5. You attempt to upload Oracle Wallet C from Endpoint B into Virtual Wallet B Endpoint B.

The KMIP error occurs because there are two copies of the same key being created and Endpoint B does not have visibility for both. If Endpoint A tries to upload the first key again, Oracle Key Vault detects this action and accounts for it. But because in Step 5, Endpoint B is not allowed to see the first key, Oracle Key Vault is unable to perform the necessary harmonization for the two Oracle wallets.

This is expected behavior. To avoid the Invalid Field, create an endpoint group so that you can share the wallet with multiple endpoints.

Parent topic: Troubleshooting Oracle Key Vault

C.5 WARNING: Could Not Store Private Key Errors

If you upload two keystores with the same file name but different contents, a WARNING: Could not store private key error appears.

This error occurs if you use the same alias (-alias slserver) in each keytool command. When you download two such keystores that have the same alias, the okvutil download process ignores the second one because the JKS aliases must be unique.
  • To remedy this problem, download the second keystore using a unique alias.

Related Topics

Parent topic: Troubleshooting Oracle Key Vault

C.6 Errors After Upgrading Oracle Key Vault

Some errors that appear after the upgrade can be ignored.

After you perform an upgrade of Oracle Key Vault on an standalone server, ORA-1109: database does not open, ORA-00313: open failed for members, and ORA-00312: online log 3 thread 1 error messages may appear in the /var/log/messages log file.

You can safely ignore these messages. Error messages also appear in the /var/log/debug file.

Parent topic: Troubleshooting Oracle Key Vault

C.7 Error: Failed to Open Wallet

An Failed to Open Wallet error can appear if you attempt to use an online master encryption key.

If you are attempting to use an online master encryption key (previously called TDE direct connect) and encounter this error, then first check your environment variable ORACLE_BASE. In an Oracle Real Application Clusters environment, you must perform this step on all database instances.

You must also set the ORACLE_SID, ORACLE_HOME, and OKV_HOME environment variables needed by the PKCS #11 library as follows:

  1. Log in to the server where the PKCS #11 library resides and then set the ORACLE_SID, ORACLE_HOME, and OKV_HOME environment variables.
  2. Log in to the database instance using SQL*Plus as a user with the SYSDBA administrative privilege. 
    sqlplus sys / as sysdba
Enter password: password
  3. Shut down the database.
    For example:
    SHUTDOWN IMMEDIATE
  4. From the command line, restart the database service.
    su - oracle
lsnrctl start
  5. In SQL*Plus, restart the database.
    STARTUP

Related Topics

Parent topic: Troubleshooting Oracle Key Vault

C.8 Transaction Check Error: Diagnostics Generation Utility

If you are trying to perform an upgrade of Oracle Key Vault, a transaction check error may appear.

For example:

file /usr/local/dbfw/etc/dbfw-diagnostics-package.yml from install of
appliance-18.3.0.0.0-52_190425.2253.d.x86_64 conflicts with file from
package okv-diagnostic-12.2.0.8.0-40_181013.1730.x86_64

The problem is that the diagnostic generation utility interferes with the upgrade process. You must remove the diagnostic generation utility before you can perform the upgrade.

Parent topic: Troubleshooting Oracle Key Vault

C.9 Fast-Start Failover (FSFO) Suspended (ORA-16818)

An ORA-16818: Fast-Start Failover suspended error can appear as a result of a fast start failover operation failing.

If the primary server was shut down gracefully in a controlled way, such as by clicking a Power Off button instead of manually turning off the computer, then a fast start failover cannot be performed and the ORA-16818: Fast-Start Failover suspended error appears. In a graceful shutdown operation, the primary server's failover status goes into a suspended state with the standby waiting indefinitely for the primary server to be available. This is the expected behavior for a fast-start failover (FSFO) operation, as defined by Oracle Data Guard avoid a split brain scenario. By design, a fast-start failover operation error occurs only when the primary server shuts down unexpectedly. If you perform a SHUTDOWN IMMEDIATE or SHUTDOWN NORMAL command in SQL*Plus, then the FSFO does not occur because the database shuts down gracefully.

Parent topic: Troubleshooting Oracle Key Vault

C.10 SSH Tunnel Add Failure

While you are configuring the SSH tunnel, a Failed to establish SSH tunnel. Refer to Oracle documentation error may appear.

You might get the following error message while trying to set up the SSH tunnel:

The failure may be due to one or more of the following problems:

  • The following settings may be invalid:

    • Invalid IP address
    • Invalid port
    • Invalid user name

  • The public SSH Oracle Key Vault key was not copied to the authorized_keys file of the okv user on the Database as a Service instance.

  • The Database as a Service instance is not reachable because of network overload.

To remedy this problem, check your input values and connection and retry.

Parent topic: Troubleshooting Oracle Key Vault

C.11 Error: Provision Command Fails if /usr/bin/java Does Not Exist

The RESTful services utility command to provision an endpoint fails if the soft link /usr/bin/java does not exist or points to an incorrect Java directory.

To remedy this problem, ensure that the Java version is 1.7.21 or later. You can create a soft link to the Java home directory as follows: 
ln -s Java_home_directory/bin/java /usr/bin/java

Parent topic: Troubleshooting Oracle Key Vault

C.12 TDE Endpoint Integration Issues

Several issues related to Transparent Data Encryption (TDE) endpoint integration problems can arise.

Common Transparent Data Encryption (TDE) endpoint integration problems caused by installation errors, svrctl misuse, and the mismanagement of security objects in a primary-standby environment can arise.

  • Installing the Oracle Key Vault library: You must run root.sh to install the Oracle Key Vault library only once on a computer that has multiple Oracle databases. During an upgrade you must upgrade the library only after you have shut down all the associated endpoints. Oracle Key Vault servers are backward compatible with endpoint libraries.
  • Using svrctl to manage the database: If you use the svrctl utility to manage the database, remember to set ORACLE_BASE and ORACLE_UNQNAME with srvctl setenv.
  • Managing security objects in an Oracle Data Guard environment:In a Data Guard environment, ensure that primary and standby servers use the same mechanism to manage security objects. These servers should either both use a wallet or both use Oracle Key Vault.

Parent topic: Troubleshooting Oracle Key Vault

C.13 Failover Situations in Primary-Standby Mode

Failover situations can occur with or without read-only restricted mode or during a planned shutdown operation for both primary and standby servers.

Parent topic: Troubleshooting Oracle Key Vault

C.13.1 About Failover Situations in Primary-Standby Mode

Failover situations in primary-standby node can occur with read-only restricted mode disabled, and with read-only restricted mode enabled.

The types of failover situations are as follows:

  • Planned shutdown of the primary server: A system administrator shuts down the primary server during an upgrade or maintenance window.

  • Planned shutdown of the standby server: A system administrator shuts down the standby server during an upgrade or maintenance window.

  • Unplanned shutdown of the primary server: The primary server is offline due to unforeseen circumstances such as power loss or network failure.

  • Unplanned shutdown of the standby server: The standby server is offline due to unforeseen circumstances such as power loss or network failure.

Parent topic: Failover Situations in Primary-Standby Mode

C.13.2 Failover Situations Without Read-Only Restricted Mode

If read-only restricted mode is not used, then various failover situations may occur in Oracle Key Vault.

Parent topic: Failover Situations in Primary-Standby Mode

C.13.2.1 Primary Server: Planned Shutdown During an Upgrade

In a failover, the use of read-only restricted mode affects a planned shutdown of a primary server during an upgrade.

If read-only restricted mode is not used, when the primary server goes offline during an upgrade, then the standby server waits in read-only mode for the primary server to return online. During the upgrade, you cannot access the Oracle Key Vault management console.

  • Recovery process: When the primary server is back online after the upgrade the standby server will automatically synchronize with the primary server. The primary and standby servers retain their earlier roles, and both servers will continue to operate in primary-standby mode.
  • Primary server state: Down
  • Standby server state: Up
  • Does failover occur? No
C.13.2.2 Primary Server: Planned Shutdown During Maintenance

In a failover, not using read-only restricted mode affects the planned shutdown of a primary server during maintenance.

If read-only restricted mode is not used, when the primary server is powered off or restarted during maintenance, then the standby server takes over from the primary server.

  • Recovery process: The standby server is now the new primary server. When the old primary server is back online after maintenance, it will automatically synchronize with the new primary server and take over the role of standby server. Both servers will continue to operate in primary-standby mode. Be aware that when the primary server is offline, data replication is disabled. If the new primary server goes offline before synchronizing with the new standby server, it may cause a loss of critical data.
  • Primary server state: Down
  • Standby server state: Up
  • Does failover occur? Yes
C.13.2.3 Standby Server: Planned Shutdown

In a failover, not using read-only restricted mode affects a planned shutdown in a standby server.

If read-only restricted mode is not used, when the standby server is powered off during upgrade or maintenance, the primary server continues operating as the primary server. Read and write operations are allowed.

  • Recovery process: When the standby server is back online post-upgrade or post-maintenance, the primary server will automatically synchronize with the standby server. The primary and standby servers retain their earlier roles, and both servers will continue to operate in primary-standby mode. Be aware that when the standby server is offline, data replication is disabled. If the primary server goes offline before synchronizing with the standby server, then it may cause a loss of critical data.
  • Primary server state: Up
  • Standby server state: Down
  • Does failover occur? No
C.13.2.4 Primary Server: Unplanned Shutdown

In a failover, not using read-only restricted mode affects an unplanned shutdown in a primary server.

If read-only restricted mode is not used, when the primary server goes offline because of power loss, network failure, or hardware failure, the standby server waits for the duration specified in the Fast Start Failover Threshold field on the Configure High Availability page in the Oracle Key Vault management console. If the primary server cannot be reached after the specified duration has elapsed, then the standby server takes over from the primary server.

  • Recovery process: The standby server is now the new primary server. Rectify the failure that affected the primary server by restarting the server or restoring network connectivity. When the primary server is back online, it will automatically synchronize with the new primary server and take over the role of standby server.
  • Primary server state: Down
  • Standby server state: Up
  • Does failover occur? Yes
C.13.2.5 Standby Server: Unplanned Shutdown

In a failover, not using read-only restricted mode affects a standby server during an unplanned shutdown.

If read-only restricted mode is not used, when the standby server goes offline because of power loss, network failure, or hardware failure, then the primary server becomes unavailable. All operations are disabled.

  • Recovery process: Rectify the failure that affected the standby server by restarting the server or restoring network connectivity. When the standby server is back online, it will automatically synchronize with the primary server. You cannot re-establish synchronization or network connectivity between the primary and standby servers, contact Oracle Support.
  • Primary server state: Up
  • Standby server state: Down
  • Does failover occur? No

C.13.3 Failover Situations with Read-Only Restricted Mode

The use read-only restricted mode affects failover operations in Oracle Key Vault.

Parent topic: Failover Situations in Primary-Standby Mode

C.13.3.1 Primary Server: Planned Shutdown During an Upgrade

In a failover, not using read-only restricted mode affects a planned shutdown of a primary server during an upgrade

When the primary server goes offline during an upgrade, the standby server enters read-only restricted mode and waits for the primary server to come back online. During the upgrade, you cannot access the Oracle Key Vault management console.

  • Recovery process: When the primary server is back online post-upgrade, the standby server will automatically synchronize with the primary server. The primary and standby servers retain their earlier roles, and both servers will continue to operate in primary-standby mode.
  • Primary server state: Down
  • Standby server state: Up
  • Does failover occur? No
C.13.3.2 Primary Server: Planned Shutdown During Maintenance

In a failover, using read-only restricted mode affects the planned shutdown of a primary server during maintenance.

When the primary server is powered off or restarted during maintenance, the standby server enters read-only restricted mode, and takes over from the primary server. The Oracle Key Vault management console displays a warning.

  • Recovery process: The standby server is now the new primary server. When the old primary server is back online after maintenance, it will automatically synchronize with the new primary server and take over the role of standby server. Both servers will continue to operate in primary-standby mode.
  • Primary server state: Down
  • Standby server state: Up
  • Does failover occur? Yes
C.13.3.3 Standby Server: Planned Shutdown

In a failover, using read-only restricted mode affects a planned shutdown in a standby server.

When the standby server is powered off during upgrade or maintenance, the primary server enters read-only restricted mode, and continues operating as the primary server. The Oracle Key Vault management console displays a warning.

  • Recovery process: When the standby server is back online post-upgrade or after maintenance, the primary server will automatically synchronize with the standby server. The primary and standby servers retain their earlier roles, and both servers will continue to operate in primary-standby mode.
  • Primary server state: Up
  • Standby server state: Down
  • Does failover occur? No
C.13.3.4 Primary Server: Unplanned Shutdown

In a failover, using read-only restricted mode affects an unplanned shutdown in a primary server.

When the primary server goes offline because of power loss, network failure, or hardware failure, then the standby server waits for the duration specified in the Fast Start Failover Threshold field on the Configure Primary-Standby page in the Oracle Key Vault management console. If the primary server is not reachable after the specified duration has elapsed, then the standby server enters read-only restricted mode, and takes over from the primary server.

  • Recovery process: The standby server is now the new primary server. Rectify the failure that affected the primary server by restarting the server or restoring network connectivity. When the primary server is back online, it will automatically synchronize with the new primary server and take over the role of standby server.
  • Primary server state: Down
  • Standby server state: Up
  • Does failover occur? Yes
C.13.3.5 Standby Server: Unplanned Shutdown

In a failover, using read-only restricted mode affects a standby server during an unplanned shutdown.

When the standby server goes offline because of power loss, network failure, or hardware failure, the primary server waits for the duration specified in the Fast Start Failover Threshold field on the Configure Primary-Standby page of the Oracle Key Vault management console. If the standby server is not reachable after the specified duration has elapsed, the primary server enters read-only restricted mode, and continues operating as the primary server.

  • Recovery process: Rectify the failure that affected the standby server by restarting the server or restoring network connectivity. When the standby server is back online, it will automatically synchronize with the primary server. If you cannot re-establish synchronization or network connectivity between the primary and standby servers, then contact Oracle Support.
  • Primary server state: Up
  • Standby server state: Down
  • Does failover occur? No

C.14 Performing a Planned Shutdown

A user who has the System Administrator role can perform planned shutdowns during an upgrade or a maintenance window.

  • Primary Server Planned Shutdown
    A user who has the System Administrator role can plan a shutdown of the primary server during an upgrade or a maintenance window.
  • Standby Server Planned Shutdown
    A user who has the System Administrator role can plan a shutdown of the primary server during an upgrade or a maintenance window.

Parent topic: Troubleshooting Oracle Key Vault

C.14.1 Primary Server Planned Shutdown

A user who has the System Administrator role can plan a shutdown of the primary server during an upgrade or a maintenance window.

Parent topic: Performing a Planned Shutdown

C.14.1.1 Performing a Primary Server Planned Shutdown During an Upgrade

You must upgrade both servers in a pair when you perform a primary server shutdown.

During an upgrade, failover does not occur. The primary and standby servers retain their earlier roles after the primary server is back online post-upgrade. After an upgrade, the primary and standby server retain their old roles.
  • To perform a primary upgrade, upgrade both of the Oracle Key Vault server pair used in the primary-standby configuration.
C.14.1.2 Performing a Primary Server Planned Shutdown During Maintenance

To perform a primary server planned shutdown during maintenance, power off or restart Oracle Key Vault.

  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
  2. Select the System tab, and then select Status in the left navigation bar.
  3. On the Status page, do one of the following:
    • Click the Power Off button.
    • Click the Reboot button.
    When the primary server is shut down, the standby server waits for the duration specified in the Fast Start Failover Threshold field on the Configure Primary-Standby page. When the duration has elapsed, the standby server will failover and take over from the primary server. The standby server is now the new primary server.
After a maintenance window, the primary and standby server switch roles. When the old primary server is back online after maintenance, it will take over the role of standby server.

C.14.2 Standby Server Planned Shutdown

A user who has the System Administrator role can plan a shutdown of the primary server during an upgrade or a maintenance window.

During upgrade, the standby server shuts down automatically, and no manual steps are necessary.

Parent topic: Performing a Planned Shutdown

C.14.2.1 Performing a Standby Server Planned Shutdown During an Upgrade

You must upgrade both servers in a pair when you perform a standby server shutdown.

When you restart the standby server during the upgrade, the upgrade script initiates an automatic shutdown. There are no manual steps to be performed after the standby server is restarted.
  • To perform a standby upgrade, upgrade both of the Oracle Key Vault server pair used in the primary-standby configuration.
C.14.2.2 Performing a Standby Server Planned Shutdown During Maintenance

You can perform a standby server planned shutdown during maintenance from the standby server using SSH.

  1. Log in to the standby server terminal using ssh as user support, then switch user (su) to root.
  2. Switch user (su) to oracle.
  3. Log in to the standby database instance as a user who as the ALTER DATABASE system privilege.
    For example:
    sqlplus sec_admin
Enter password: password
  4. Execute the ALTER DATABASE statement as follows:
    ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
  5. Shut down the database.
    SHUTDOWN IMMEDIATE
  6. Power off the standby server.
    1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
    2. Select the System tab, and then select Status from the left navigation bar.
    3. In the Status page, click the Power Off button.