B Troubleshooting Oracle Key Vault

This section contains checklists and tips for commonly encountered errors that will help you install and deploy Key Vault quickly.

B.1 Key Vault Pre-Installation Checklist

The pre-installation checklist covers all the requirements to successfully install Key Vault.

Table B-1 Oracle Key Vault Pre-Installation Checklist

Item#	Check	Task
1. [ x ]	System Requirements	Confirm that you have enough CPU, Memory, and Disk as described here, in "System Requirements"
2. [ x ]	Open all the required network ports in your firewall	For details on network ports, see "Network Ports"
3. [ x ]	Supported endpoint platforms	We have expanded platform support as listed here, "Supported Endpoint Platforms"
4. [ x ]	Set the COMPATIBLE initialization parameter for Online Master Key (previously TDE direct connect)	How to set this parameter for Oracle Database 11.2.0.0 and higher is described here, "Supported Endpoint Platforms"
5. [ x ]	Get a fixed IP address, Network Mask, and Gateway Address from your network administrator	You will need this for Step 9 of Task 1 of the installation process described here, Installing the Oracle Key Vault Appliance Software

Parent topic: Troubleshooting Oracle Key Vault

B.2 Audit Vault Database Firewall Integration Instructions

To consolidate audits between Audit Vault and Database Firewall (AVDF) with Oracle Key Vault, you must integrate AVDF with Key Vault as follows:

Install an AVDF 12.2.server and set up the AV administrator and auditor users.
Install an OKV 12.2 server. Then, log on as the system administrator and enable ssh access.
Register the OKV server as a secured target on the AVDF server. Log in as AV administrator (avadmin), and go to Secured Targets > Register. Enter the OKV server name as the name. Set type to the Oracle Key Vault plugin and use the connect string: jdbc:oracle:thin:@//127.0.0.1:1521/dbfwdb. For username and password, use avcollector/<integration_password>. Set collection attributes on the secured target:
```
av.collector.securedTargetVersion 12.2.0.0.0
av.collector.TimeZoneOffset +5:30 
```
Note: You can use any offset that applies to your situation.
Register the OKV server as a host on the AVDF machine. Go to Hosts > Register and add it. You will need the OKV server IP. Note that after the host is added, a new entry will appear with an Agent Activation Key. Copy this value and save it somewhere.
Download the AVDF agent. Go to Hosts > Agent > Agent Release and get the agent (first item in the list). Save as agent.jar and upload to the OKV server using scp.
Installing agent.jar on the OKV server:
1. Log on to the OKV command line as support and su to root to create the directory, get the agent, and set permissions:
```
cd /usr/local/okv
mkdir avdf
cp /home/support/agent.jar /usr/local/okv/avdf
chown oracle:oinstall /usr/local/okv/avdf /usr/local/okv/avdf/*
```
2. su to oracle and extract the agent:
```
su - oracle
cd /usr/local/okv/avdf
java -jar agent.jar -d /usr/local/okv/avdf
```
3. As oracle, start the agent and enter the ADVF Agent Activation Key:
```
cd /usr/local/okv/avdf/bin
./agentctl start -k
```
4. Using the OKV browser-based console, enable the database user. To enable the database user, go to System > System Settings. In the Oracle Audit Vault Integration section, check Enable. Enter the integration password.
Add audit trail to AVDF. Log on as the AV administrator (avadmin) user. Go to Secured Targets > Audit Trails > Add to add a new audit trail. Select type as TABLE, and choose the secured target and host that you created. Set the table name to keyvault.audit_trail. After saving, select the audit trail and start collection.
View data collected by AVDF: Log into AVDF as avauditor and go to Reports > All Activity > All Activity Report.

Parent topic: Troubleshooting Oracle Key Vault

B.3 RESTful Services Troubleshooting Help

The Key Vault log files capture all the error messages sent by the server. The error messages are written to the /var/log/messages file. The first debugging step is to read the messages file.

To check for log file errors, do the following as a root user:

root# vi /var/log/messages

Parent topic: Troubleshooting Oracle Key Vault

B.4 Error: Cannot Open Keystore Message

The Cannot Open Keystore error can appear when you try to upload a Java keystore to the Oracle Key Vault server.

You can try the following solutions:

Ensure that the PATH environment variable has been correctly set.
Check where the keytool and Java are pointing to, by entering the following commands in a shell:
```
which keytool
which java
```
Ensure that you are using Oracle Java.

Parent topic: Troubleshooting Oracle Key Vault

B.5 KMIP Error: Invalid Field

The Invalid Field KMIP error can occur when you are trying to upload Oracle wallets to virtual wallets on multiple endpoints as follows:

You configure two or more endpoints (for example, Endpoint A and Endpoint B) to share a wallet (Oracle Wallet C), and hence also share the wallet keys.
You register Endpoints A and B with Oracle Key Vault.
You create a default wallet (Virtual Wallet A) for Endpoint A and then a default wallet (Virtual Wallet B) for Endpoint B. Each virtual wallet is accessible only to the corresponding endpoint. For example, Endpoint B has no access to Virtual Wallet A.
You upload Oracle Wallet C into Virtual Wallet A on Endpoint A.
You attempt to upload Oracle Wallet C from Endpoint B into Virtual Wallet B Endpoint B.

The KMIP error occurs because there are two copies of the same key being created and Endpoint B does not have visibility for both. If Endpoint A tries to upload the first key again, Oracle Key Vault detects this action and accounts for it. But because in Step 5, Endpoint B is not allowed to see the first key, Oracle Key Vault is unable to perform the necessary harmonization for the two Oracle wallets.

This is expected behavior. Instead, create an endpoint group so that you can share the wallet with multiple endpoints. See "Manage Endpoint Groups" for more information.

Note:

The KMIP error can occur for other scenarios, but this scenario is the most common.

Parent topic: Troubleshooting Oracle Key Vault

B.6 WARNING: Could Not Store Private Key Errors

If you upload two keystores with the same file name but with different contents, a WARNING: Could not store private key error is generated.

This occurs if you use the same alias (-alias slserver) in each keytool command. When you download two such keystores that have the same alias, the okvutil download process ignores the second one because the JKS aliases must be unique. Download the second keystore using a unique alias.

B.7 Errors After Upgrading Oracle Key Vault

After you perform an upgrade of Oracle Key Vault on an standalone server, ORA-1109, ORA-00313, and ORA-00312 error messages may appear in the /var/log/messages log file.

You can safely ignore these messages. Error messages also appear in the /var/log/debug file.

Parent topic: Troubleshooting Oracle Key Vault

B.8 Error: Failed to Open Wallet

If you are attempting Online Master Key (previously TDE direct connect) and encounter the error Failed to Open Wallet, you must first check your environment variable ORACLE_BASE. See Step 1 of "Configuring a Connection Between Oracle Key Vault and a New TDE-Enabled Database"

You must also set the following environment variables: ORACLE_SID, ORACLE_HOME, and OKV_HOME needed by the PKCS #11 library as follows:

Set system ENV variables for the USER.
Shutdown DB.
Restart Service.
Restart DB.

Parent topic: Troubleshooting Oracle Key Vault

B.9 Error: Provision Command Fails if /usr/bin/java does not Exist

The RESTful Service command to provision an endpoint fails if the soft link /usr/bin/java does not exist or point to the correct Java directory. The Java version should be 1.4 or greater.

To create the soft link to your Java home directory:

ln -s <your_Java_home_directory>/bin/java /usr/bin/java

Parent topic: Troubleshooting Oracle Key Vault

B.10 Transaction Check Error: Diagnostics Generation Utility

If you are trying to perform an upgrade of Oracle Key Vault, a transaction check error may appear.

For example:

file /usr/local/dbfw/etc/dbfw-diagnostics-package.yml from install of
appliance-18.1.0.0.0-52_190425.2253.d.x86_64 conflicts with file from
package okv-diagnostic-12.2.0.8.0-40_181013.1730.x86_64

The problem is that the diagnostic generation utility interferes with the upgrade process. You must remove the diagnostic generation utility before you can perform the upgrade.

Related Topics

Download System Diagnostics

Parent topic: Troubleshooting Oracle Key Vault

B.11 Fast-Start Failover (FSFO) Suspended (ORA-16818)

If the primary was shutdown gracefully in a controlled way like clicking on Power Off instead of 'pulling the plug', a fast start failover is not going to be performed. In a graceful shutdown the primary server's failover status goes into a suspended state with the standby waiting indefinitely for the primary to come back up. This is the expected behavior for FSFO as defined by Oracle DataGuard avoid a split brain scenario. By design, an FSFO will occur only when the primary goes down unexpectedly. If you do a shutdown immediate (or shutdown normal), FSFO will not occur.

Parent topic: Troubleshooting Oracle Key Vault

B.12 SSH Tunnel Add Failure

You might get the following error message while trying to set up the SSH tunnel:

Failed to establish SSH tunnel. Refer to Oracle documentation.

Figure B-1 SSH Tunnel Add Failure

Description of "Figure B-1 SSH Tunnel Add Failure"

The failure may be due to one or more of the following:

Invalid Tunnel Name
Invalid IP Address
Invalid Port
Invalid Username
The public SSH Key Vault key is not copied to the authorized_keys file of the okv user on the Database as a Service instance
The Database as a Service instance is not reachable because of network overload

Check your input values and connection and retry.

Parent topic: Troubleshooting Oracle Key Vault

B.13 TDE Endpoint Integration Issues

This section addressed the common problems with TDE endpoint integration and how to overcome them.

Notes on Installing Oracle Key Vault Library

You must run root.sh to install the Oracle Key Vault library only ONCE on a machine with multiple Oracle Databases
During an upgrade you must upgrade the library only after all the end points are shut down. Oracle Key Vault servers are backwards compatible with endpoint libraries at present.

If Using SVRCTL Tool to Manage Database

If you use the svrctl tool to manage the database, note that the tool may set ORACLE_BASE to NULL. A good practice is to set ORACLE_BASE to ORACLE_HOME, if ORACLE_BASE is not used in your environment.

Primary and Standby Should Manage Security Objects in the Same Way

In a high availability configuration both the primary and standby servers should use the same mechanism to manage security objects. They should either both use a wallet or both use Oracle Key Vault.

Parent topic: Troubleshooting Oracle Key Vault

B.14 Failover Situations in High Availability Mode

The following sections list common failover scenarios that are encountered in High Availability mode deployments, for various versions of Oracle Key Vault, and for Oracle Key Vault 12.2.0.5.0 with Read-Only Restricted mode disabled, and with Read-Only Restricted mode enabled.

Parent topic: Troubleshooting Oracle Key Vault

B.14.1 Types of Failover Situations

The types of failover situations are:

Planned shutdown of the primary server: The primary server is shut down by a system administrator during an upgrade or maintenance window.
Planned shutdown of the standby server: The standby server is shut down by a system administrator during an upgrade or maintenance window.
Unplanned shutdown of the primary server: The primary server is offline due to unforeseen circumstances such as power loss or network failure.
Unplanned shutdown of the standby server: The standby server is offline due to unforeseen circumstances such as power loss or network failure.

Parent topic: Failover Situations in High Availability Mode

B.14.2 Failover Situations without Read-Only Restricted Mode

The following table describes the various failover situations that may be encountered in the following versions of Oracle Key Vault:

12.2.0.0.0
12.2.0.1.0
12.2.0.2.0
12.2.0.3.0
12.2.0.4.0
12.2.0.5.0 (without Read-Only Restricted Mode)

Table B-2 Failover Situations without Read-Only Restricted Mode

Number	Failover Situation	Primary Server State	Standby Server State	Does failover occur?	Details	Recovery Steps
1	Primary Server: Planned Shutdown during upgrade	Down	Up	No	When the primary server goes offline during an upgrade, the standby server waits in read-only mode for the primary server to come back online. During the upgrade, you cannot access the Oracle Key Vault management console. A failover does not occur. For more information about performing a primary server planned shutdown during upgrade, see Performing a Primary Server Planned Shutdown during Upgrade.	When the primary server is back online post-upgrade, the standby server will automatically synchronize with the primary server. The primary and standby servers retain their earlier roles, and both servers will continue to operate in High Availability mode.
2	Primary Server: Planned Shutdown during maintenance	Down	Up	Yes	When the primary server is powered off or rebooted during maintenance, the standby server takes over from the primary server. Note: For more information about performing a primary server planned shutdown during maintenance, see Performing a Primary Server Planned Shutdown during Maintenance.	The standby server is now the new primary server. When the old primary server is back online post-maintenance, it will automatically synchronize with the new primary server, and takes over the role of standby server. Both servers will continue to operate in High Availability mode. Note: When the primary server is offline, data replication is disabled. If the new primary server goes offline before synchronizing with the new standby server, it may cause a loss of critical data.
3	Standby Server: Planned Shutdown	Up and functional	Down	No	When the standby server is powered off during upgrade or maintenance, the primary server continues operating as the primary server. Read/Write operations are allowed. For more information about performing a standby server planned shutdown during upgrade, see Performing a Standby Server Planned Shutdown during Upgrade. For more information about performing a standby server planned shutdown during maintenance, see Performing a Standby Server Planned Shutdown during Maintenance.	When the standby server is back online post-upgrade or post-maintenance, the primary server will automatically synchronize with the standby server. The primary and standby servers retain their earlier roles, and both servers will continue to operate in High Availability mode. Note: When the standby server is offline, data replication is disabled. If the primary server goes offline before synchronizing with the standby server, it may cause a loss of critical data.
4	Primary Server: Unplanned Shutdown	Down	Up	Yes	When the primary server goes offline because of power loss, network failure, or hardware failure, the standby server waits for the duration specified in the Fast Start Failover Threshold field on the Configure High Availability page. If the primary server is not reachable after the specified duration has elapsed, the standby server takes over from the primary server.	The standby server is now the new primary server. Rectify the failure that affected the primary server by rebooting the server or restoring network connectivity. When the primary server is back online, it will automatically synchronize with the new primary server, and takes over the role of standby server.
5	Standby Server: Unplanned Shutdown	Up	Down	No	When the standby server goes offline because of power loss, network failure, or hardware failure, the primary server becomes unavailable. All operations are disabled.	Rectify the failure that affected the standby server by rebooting the server or restoring network connectivity. When the standby server is back online, it will automatically synchronize with the primary server. If it is not possible to re-establish synchronization or network connectivity between the primary and standby servers, contact Oracle Support.

Parent topic: Failover Situations in High Availability Mode

B.14.3 Failover Situations with Read-Only Restricted Mode

The following table describes the various failover situations that may be encountered in Oracle Key Vault 12.2.0.5.0 and later (with Read-Only Restricted mode enabled):

Table B-3 Failover Situations in 12.2.0.5.0 and later (with Read-Only Restricted Mode enabled)

Number	Failover Situation	Primary Server State	Standby Server State	Does failover occur?	Details	Recovery Steps
1	Primary Server: Planned Shutdown during upgrade	Down	Up	No	When the primary server goes offline during an upgrade, the standby server enters Read-Only Restricted mode and waits for the primary server to come back online. During the upgrade, you cannot access the Oracle Key Vault management console. A failover does not occur. For more information about performing a primary server planned shutdown during upgrade, see Performing a Primary Server Planned Shutdown during Upgrade.	When the primary server is back online post-upgrade, the standby server will automatically synchronize with the primary server. The primary and standby servers retain their earlier roles, and both servers will continue to operate in High Availability mode.
2	Primary Server: Planned Shutdown during maintenance	Down	Up	Yes	When the primary server is powered off or rebooted during maintenance, the standby server enters Read-Only Restricted mode, and takes over from the primary server. The Oracle Key Vault management console displays a warning. For more information about performing a primary server planned shutdown during maintenance, see Performing a Primary Server Planned Shutdown during Maintenance.	The standby server is now the new primary server. When the old primary server is back online post-maintenance, it will automatically synchronize with the new primary server, and takes over the role of standby server. Both servers will continue to operate in High Availability mode.
3	Standby Server: Planned Shutdown	Up and functional	Down	No	When the standby server is powered off during upgrade or maintenance, the primary server enters Read-Only Restricted mode, and continues operating as the primary server. The Oracle Key Vault management console displays a warning. For more information about performing a standby server planned shutdown during upgrade, see Performing a Standby Server Planned Shutdown during Upgrade. For more information about performing a standby server planned shutdown during maintenance, see Performing a Standby Server Planned Shutdown during Maintenance.	When the standby server is back online post-upgrade or post-maintenance, the primary server will automatically synchronize with the standby server. The primary and standby servers retain their earlier roles, and both servers will continue to operate in High Availability mode.
4	Primary Server: Unplanned Shutdown	Down	Up	Yes	When the primary server goes offline because of power loss, network failure, or hardware failure, the standby server waits for the duration specified in the Fast Start Failover Threshold field on the Configure High Availability page. If the primary server is not reachable after the specified duration has elapsed, the standby server enters Read-Only Restricted mode, and takes over from the primary server.	The standby server is now the new primary server. Rectify the failure that affected the primary server by rebooting the server or restoring network connectivity. When the primary server is back online, it will automatically synchronize with the new primary server, and takes over the role of standby server.
5	Standby Server: Unplanned Shutdown	Up	Down	No	When the standby server goes offline because of power loss, network failure, or hardware failure, the primary server waits for the duration specified in the Fast Start Failover Threshold field on the Configure High Availability page. If the standby server is not reachable after the specified duration has elapsed, the primary server enters Read-Only Restricted mode, and continues operating as the primary server.	Rectify the failure that affected the standby server by rebooting the server or restoring network connectivity. When the standby server is back online, it will automatically synchronize with the primary server. If it is not possible to re-establish synchronization or network connectivity between the primary and standby servers, contact Oracle Support.

Parent topic: Failover Situations in High Availability Mode

B.14.4 Performing a Planned Shutdown

A planned shutdown is performed by a system administrator during an upgrade or maintenance window.

Parent topic: Failover Situations in High Availability Mode

B.14.4.1 Primary Server Planned Shutdown

A planned shutdown of the primary server is performed by a system administrator during an upgrade or a maintenance window.

Note:

After an upgrade, the primary and standby server retain their old roles. After a maintenance window, the primary and standby server switch roles.

Parent topic: Performing a Planned Shutdown

B.14.4.1.1 Performing a Primary Server Planned Shutdown during Upgrade

To perform a primary server planned shutdown during upgrade, follow the procedure Upgrade a Pair of Key Vault Servers in a High Availability Deployment.

During an upgrade, failover does not occur. The primary and standby servers retain their earlier roles after the primary server is back online post-upgrade.

Parent topic: Primary Server Planned Shutdown

B.14.4.1.2 Performing a Primary Server Planned Shutdown during Maintenance

To perform a primary server planned shutdown during maintenance, power off or reboot Oracle Key Vault.

Power off Oracle Key Vault by clicking the Power Off button on the Settings page. You can also power off Oracle Key Vault by using the terminal.

Reboot Oracle Key Vault by clicking the Reboot button on the Settings page. You can also reboot Oracle Key Vault by using the terminal.

When the primary server is shut down, the standby server waits for the duration specified in the Fast Start Failover Threshold field on the Configure High Availability page. When the duration has elapsed, the standby server will failover and take over from the primary server. The standby server is now the new primary server.

When the old primary server is back online after maintenance, it will take over the role of standby server.

Parent topic: Primary Server Planned Shutdown

B.14.4.2 Standby Server Planned Shutdown

A planned shutdown of the standby server is performed by a system administrator during a maintenance window. During upgrade, the standby server shuts down automatically, and no manual steps are necessary.

Parent topic: Performing a Planned Shutdown

B.14.4.2.1 Performing a Standby Server Planned Shutdown during Upgrade

To perform a standby server planned shutdown during upgrade, follow the procedure Upgrade a Pair of Key Vault Servers in a High Availability Deployment.

Note:

When you reboot the standby server during upgrade, the upgrade script initiates an automatic shutdown. There are no manual steps to be performed after the standby server is rebooted.

Parent topic: Standby Server Planned Shutdown

B.14.4.2.2 Performing a Standby Server Planned Shutdown during Maintenance

To perform a standby server planned shutdown during maintenance:

Log in to the standby server terminal using SSH as user support, then switch user (su) to root.
Switch user (su) to oracle.

Open SQL *Plus and run the following commands on the database:

alter database recover managed standby database cancel
shutdown immediate

Power off the standby server.

Parent topic: Standby Server Planned Shutdown