C.3 okvutil and Endpoint Issues

Learn how to run the endpoint health check utility to triage endpoint related issues

• Database Wallet Status Not Open or Not Found, TDE HEARTBEAT Check Failed
  Database wallet status for Oracle Key Vault is closed. When trying to open the Database wallet it fails with error ORA-28365, ORA-28407, or HSM heartbeat check failed errors.
• Oracle Key Vault Server Communication or Connection Failed Error
  Oracle Key Vault displays a server communication or connection failed error when you try to fetch data using okvutil or when installing okvclient.jar.
• Could Not Store Private Key Errors on Wallet Upload
  When uploading Java keystores, using okvutil, Oracle Key Vault displays could not store private key errors.
• RESTful Services Endpoint Provisioning Command Failure
  The endpoint provisioning using RESTful services utility fails due to incorrect path or directory.
• Uploading Certificate File Failure
  When uploading a console certificate an error is displayed .
• Error in Uploading the Java Keystore
  Oracle Key Vault displays an error when uploading the Java keystore.
• SSL layer Error while migrating MYSQL Database Keys to Oracle Key Vault
  An SSL layer error displays when the MySQL database is migrated to Oracle Key Vault.

C.3.1 Database Wallet Status Not Open or Not Found, TDE HEARTBEAT Check Failed

Database wallet status for Oracle Key Vault is closed. When trying to open the Database wallet it fails with error ORA-28365, ORA-28407, or HSM heartbeat check failed errors.

Example

1. Failed to open keystore on endpoint DB

SQL> ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY
"*";
 ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY "*"
*
ERROR at line 1:
ORA-28365: wallet is not open

2. Failed to fetch the keys from v$encryption keys on endpoint DB.

SQL> SELECT KEY_ID value FROM V$ENCRYPTION_KEYS;
*
ERROR at line 1:
ORA-28407: Hardware Security Module failed with PKCS#11 error
CKR_GENERAL_ERROR(5)

3. Failed to open key store on endpoint DB.

administer key management set keystore open identified by “*”;
*
ERROR at line 1:
ORA-28353: failed to open wallet

4. Endpoint DB alert log shows the below error messages.

HSM connection lost, closing wallet
kzthsmcc1: HSM heartbeat check failed 

Probable Cause 1

When the endpoint DB does not use the WALLET_ROOT configuration. This can happen if the environment variables ORACLE_BASE, ORACLE_HOME, ORACLE_SID,OKV_HOME, and JAVA_HOME are not set.

Solution

1. For Oracle RAC environments, check if the environment variables ORACLE_BASE, ORACLE_HOME, ORACLE_SID,OKV_HOME, and JAVA_HOME are set in the configuration.
   1. If the variables are not set, then set them using srvctl environment.

      srvctl setenv database -db db_unique_name -t "ORACLE_UNQNAME=db_unique_name, ORACLE_BASE=/u01/opt/oracle"

   2. You can restart the database instance using the srvctl command.

      srvctl getenv database -db db_unique_name

2. If you get the error when running a third-party script for RMAN backup, then check if the ORACLE_BASE, ORACLE_HOME, ORACLE_SID,OKV_HOME, and JAVA_HOME environment variables are exported in the script. Also check if the user running the script has set the environment variables in the profile.
3. If the environment variables are not set, then set the variables in the user profile and script, and then run the backup.
4. If the endpoint health check utility reports that the environment variables are not set in the gen0 environment, then set the variables using the command line and restart the database.
   1. Log in to SQLPLUS as the SYSDBA user and shutdown the database.

      ./sqlplus sys / as sysdba 
      
      shutdown immediate

   2. Exit from SQLPLUS and restart the database service.

      su - oracle
      snrctl start

   3. Log in to SQLPLUS and start the database.

      startup

5. Check if the issue is resolved.

Probable Cause 2

Oracle Key Vault server KMIP service is not available.

Solution

1. Check if the Oracle Key Vault server can successfully list the keys:

   $OKV_HOME/bin/okvutil list -v 4

2. If the keys do not list, then check the availability of the Oracle Key Vault server by accessing the Oracle Key Vault management console.
3. If the Oracle Key Vault server is up and running, then verify the KMIP service status using one of these methods:
   • If the REST services utility is configured on the endpoint DB server then use the REST services command okv status get to get the server information and verify the status of the KMIP service.
   • If the SNMP monitoring is configured, then use SNMP monitoring to find the status of the KMIP service.
   • Log in to the Oracle Key Vault management console as the System Administrator. On the Oracle Key Vault home page, check if there is an alert stop responding process alert.
   • Log in to the Oracle Key Vault management console as the System Administrator. Navigate to the System, Status page and check the KMIP service status.
4. If the KMIP service status is down, then follow restart the KMIP service:
   1. Log in to the Oracle Key Vault server using SSH as the support user and then switch to the root user.

      $ ssh support@okv_server_IP_address 
      $ su - root

   2. Run the following command to verify the status of the KMIP process.

      ps -eaf | grep kmip

   3. If the KMIP process is not running, then restart the KMIP service:

      systemctl restart kmip

   4. Verify the status of the KMIP process:

      ps -eaf | grep kmip 

   5. On the endpoint DB server, check if okvutil lists the keys:

      OKV_HOME/bin/okvutil list -v 4

   6. Check if the issue is resolved.
5. If the status of the KMIP service is up and running, then from the endpoint DB server run the following command:

   curl -v telnet://OKV_SERVER_IP:5696

6. If this command does not connect to the Oracle Key Vault server, then contact your network administrator to verify that port 5696 is open. If not, ensure that port 5696 is open.
7. Check if the issue is resolved.

Probable Cause 3

Symbolic link to okvclient.ora does not exist or points to an incorrect configuration.

Solution

Perform the following steps:

1. On the endpoint DB server, verify whether the symbolic link for okvclient.ora exists.

   cd $ORACLE_BASE/okv/$ORACLE_SID 
   ls -ltr okvclient.ora

   OR

   cd $ORACLE_HOME/okv/$ORACLE_SID 
   ls -ltr okvclient.ora

   Command output should include the okvclient.ora symbolic link information, if it exists. For example:

   lrwxrwxrwx 1 oracle oinstall 64 Aug 3 2020 okvclient.ora -> /u02/app/oracle/okv/conf/okvclient.ora.

2. If the okvclient.ora symbolic link does not exist or is pointing to a path other than $OKV_HOME/conf, then either correct the symbolic link, or reinstall the Oracle Key Vault client. See, How to Re-Enroll an Endpoint on an Endpoint Database.
3. Verify that okvclient.ora exists and points to $OKV_HOME/conf.
4. Check if the issue is resolved.

Probable Cause 4

The auto-login wallet is not configured properly.

Solution

Perform the following steps:

1. Verify the wallet status by connecting to the endpoint DB and executing below SQL query.

   select * from v$encryption_wallet

2. For Oracle Databases 12c and earlier:

   If the auto-login is configured and wallet for OKV/HSM is not open and for FILE is open, then verify whether the auto-login wallet location is set in sqlnet.ora file. If not, then set the location, and restart the database (downtime required). Verify whether the issue is resolved

   For Oracle database 12.1.0.2 or earlier the sqlnet.ora entry looks like,

   ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=HSM)(METHOD_DATA=(DIRECTORY=/home/oracle/wallet_okv)))

   For 12.2.0.1 and above the sqlnet.ora entry looks like,

   ENCRYPTION_WALLET_LOCATION=(SOURCE=(METHOD=OKV)(METHOD_DATA=(DIRECTORY=/home/oracle/wallet_okv)))

3. For the Oracle databases 18c or later:

   If the WALLET_ROOT is configured, then the auto-login wallet should exist under the WALLET_ROOT_directory/tde directory and Oracle Key Vault endpoint software installation is under the WALLET_ROOT_directory/okv directory. If any of them do not exist, apply the necessary configuration changes to address this.

4. Check if the issue is resolved.

Probable Cause 5

The default wallet is not configured properly for the endpoint database.

Solution

Verify whether the default wallet is assigned to the endpoint:

1. Log in to the Oracle Key Vault management console as the Key Administrator user.
2. Verify whether the latest keys are added to the wallet. Go to the Keys & Secrets page, and sort the keys in descending order by Creation Date.
   
   
   Description of the illustration 21.5_keys_and_secrets.png
   
   
   1. Check whether the latest keys created from that particular endpoint, Creating Endpoint are assigned to the wallet.
   2. If the key is not assigned to any wallet then, edit the key by clicking on the pencil icon.
      
      
      Description of the illustration 21.5_creating_endpoint_pencil_icon.png
      
      
   3. On the edit page, click on Add Wallet Membership and select the wallet assigned to the endpoint and save it.
      
      
      Description of the illustration 21.5_wallet_membership.png
      
      
   4. Verify whether the key has wallet membership by revisiting the Keys & Secrets page.
3. Go to the endpoints page.
4. Search for the endpoint then click on the endpoint name.
5. Check the default wallet setting. If no default wallet is assigned, then click Choose Wallet to select the desired wallet.
6. Click Save.
   
   
   Description of the illustration 21.5_default_wallet.png
   
   
7. Revisit the endpoint edit page to verify whether the default wallet is assigned correctly.
8. Verify if the endpoint has READ/MODIFY and MANAGE permissions to the wallet from the Access to Wallets section.
9. Once verified, re-enroll the endpoint and install the new okvclient.jar on the endpoint DB server at the OKV_HOME path. See, How to Re-Enroll an Endpoint on an Endpoint Database.
10. Check if the issue is resolved.

C.3.2 Oracle Key Vault Server Communication or Connection Failed Error

Oracle Key Vault displays a server communication or connection failed error when you try to fetch data using okvutil or when installing okvclient.jar.

Example

Running okvutil list on endpoint server returns the below errors

1.
bash$ ./okvutil list
Enter Oracle Key Vault endpoint password:
Error: Server Communication Error

2.
bash$ ./okvutil list
Enter Oracle Key Vault endpoint password:
Error: Server Connect Failed

Probable Cause 1

Port 5696 is blocked between the endpoint DB and the Oracle Key Vault server.

Solution

1. Check the endpoint connectivity to the Oracle Key Vault server:

   $OKV_HOME/bin/okvutil list -v 4

2. If the endpoint connection to the Oracle Key Vault server fails, then check if port 5696 is open:

   curl -v telnet://OKV_SERVER_IP:5696

3. If the preceding command fails to connect to the Oracle Key Vault server, then contact your network administrator to verify that port 5696 is open. If not, ensure that port 5696 is open.
4. Check if the issue is resolved.

Probable Cause 2

File ownership of okvclient.ora is incorrect.

Solution

1. Check the file ownership of the okvclient.ora file.

   ls -l $OKV_HOME/conf/okvclient.ora

2. If okvclient.ora ownership is set to a different user than expected, reset the file ownership to the correct user. The okvclient.ora file ownership may have changed if the okvutil or other endpoint software was run with an unintended user (say, root).
3. Check if the issue is resolved.

Probable Cause 3

Oracle Key Vault server, CA, or endpoints certificates have expired.

Solution

1. Check if the endpoint certificates have expired.
   1. Log in to the Oracle Key Vault management console as a user with System Administrator role or privilege.
   2. Navigate to the Endpoints tab, and search for the endpoint. Check the expiration date under the Endpoint Certificate Expiration Date column. You can also check the Oracle Key Vault Home page for the endpoint certificate expiration alerts.
   3. If the endpoint certificate has expired, select and re-enroll the endpoint. See, How to Re-Enroll an Endpoint on an Endpoint Database.
   4. Check if the issue is resolved.
2. Check if the Oracle Key Vault server or node certificates have expired.
   1. Log in to the Oracle Key Vault management console as a user with the System Administrator role.
   2. To check the server or node certificate expiration status, navigate to System, Status and check the Server or Node Certificate Expiration Date. You can also check the Oracle Key Vault Home page for server or node certificate expiration alerts.
   3. Regenerate the server certificate if it has expired. Go to System, click Settings.
   4. Go to Service Certificates and select Manage Server or Node Certificate to regenerate the new server certificate.
3. Check if the Oracle Key Vault CA certificate has expired.
   1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
   2. To check the CA certificate expiration status, navigate to System, Status and check the CA Certificate Expiration Date. You can also check the Oracle Key Vault Home page for any alerts for the CA certificate expiration.
   3. If the Oracle Key Vault CA certificate has expired, you must manually recover and regenerate the CA certificate, For 21.5 and later See section 17.6, Managing the Oracle Key Vault CA Certificate after it has expired, for details on how to do so. For Oracle Key Vault 21.4 and earlier, contact Oracle Support.
4. Check if the issue is resolved.

Probable Cause 4

The SSL_WALLET_LOC in okvclient.ora is set to an incorrect location.

Solution

1. Endpoint certificate rotation places the new endpoint certificates in a new directory under $OKV_HOME/ssl and SSL_WALLET_LOC is modified to point to this new directory.

   Check if the SSL_WALLET_LOC is correctly pointing to the most recently created directory under $OKV_HOME/ssl.

2. Go to the $OKV_HOME/conf directory, open okvclient.ora and verify if SSL_WALLET_LOC is pointing to the most recently created directory under $OKV_HOME/ssl.
3. If not, then modify theSSL_WALLET_LOC setting appropriately and save okvclient.ora.
4. Check if the issue is resolved.

Probable Cause 5

During an endpoint reenrollment, Oracle Key Vault endpoint software was upgraded in-place instead of installing the endpoint software in an empty directory.

Solution

1. During endpoint re-enrollment, if the existing $OKV_HOME directory is not removed before installing new okvclient.jar, then the installation process only upgrades the endpoint software. The endpoint certificates and okvclient.ora file are not updated in such a case. See, How to Re-Enroll an Endpoint on an Endpoint Database.
2. Check if the issue is resolved.

Probable Cause 6

Oracle Key Vault server KMIP service is not available.

Solution

1. Run the following command to check if it can list the keys successfully:

   $OKV_HOME/bin/okvutil list -v4

2. If the preceding command fails to list the keys then check the availability of the Oracle Key Vault server by accessing the Oracle Key Vault management console.
3. If the Oracle Key Vault server is up, then verify the status of the KMIP service using one of these methods:
   1. If the REST services utility is configured on the endpoint DB server, then run the REST services command okv server status get to get the server information and verify the status of the KMIP service.
   2. If the SNMP monitoring is configured, then use the SNMP monitoring to determine the status of the KMIP service.
   3. Log in to the Oracle Key Vault management console as a user who has the System Administrator role. In the Oracle Key Vault home page, check if there is an alert reporting the KMIP process as stop responding.
   4. Log in to the Oracle Key Vault management console as a user who has the System Administrator role. Navigate to the System, Status page and check the status of the KMIP service.
4. If the status of the KMIP service is down, then restart the KMIP service.
   • Log in to the Oracle Key Vault server through SSH as the support user and then switch user su to root.

     ssh support@okv_server_IP_address
     su - root

5. Run the following command to verify the status of the KMIP process.

   ps -eaf | grep kmip

6. If the preceding command shows that the KMIP process is not running, then restart the KMIP service:

   systemctl stop kmip
   systemctl start kmip

7. Verify the status of the KMIP process:

   ps -eaf | grep kmip

8. On the endpoint DB server, check if okvutil lists the keys:

    $OKV_HOME/bin/okvutil list -v4

9. Check if the issue is resolved.

Probable Cause 7

A strict IP check is enabled for the endpoint. The endpoint IP address is different from the IP address that was saved during endpoint enrollment.

Solution

1. Check IP address of the endpoint that is currently saved in the Oracle Key Vault server.

   Log in to the Oracle Key Vault management console as a user who has the System Administrator role or the privilege to manage this endpoint.

   Navigate to the Endpoints page, search for the endpoint and verify the IP address saved for the endpoint.

2. Click on the endpoint name to open the Endpoint Details page.
3. Deselect the Strict IP option.
4. In the endpoint DB server, check if okvutil lists the keys.

   $OKV_HOME/bin/okvutil list -v4

5. If deselecting the strict IP check resolves the issue, then:
   1. Check if more than one IP address is configured for the endpoint DB server. In such a case, StrictIPCheck cannot be enabled.
   2. If the endpoint DB server is configured with only one IP address, then re-enroll the endpoint. See, How to Re-Enroll an Endpoint on an Endpoint Database and re-enable the Strict IP Check for the endpoint.
6. The Strict IP check should be re-enabled if the endpoint DB server is reconfigured and its connecting from one IP address only.
7. Check if the issue is resolved.

Probable Cause 8

There is a time lag between the endpoint DB server and the Oracle Key Vault server.

Solution

1. Check if there is a time drift between the endpoint DB server and the Oracle Key Vault server.
2. If yes, resolve the time drift. Consider using the NTP configuration to synchronize the system clock of the Oracle Key Vault server.
3. Check if the issue is resolved.

C.3.3 Could Not Store Private Key Errors on Wallet Upload

When uploading Java keystores, using okvutil, Oracle Key Vault displays could not store private key errors.

Example

okvutil upload command returns the below error:

$ okvutil upload -l ./fin_jceks.jck -t JCEKS -g fin_wal -v2
okvutil version 21.5.0.0.0
Configuration file: /tmp/fin_okv/conf/okvclient.ora
Server: 192.0.2.254:5696
Uploading from /tmp/fin_okv/keystores/jks/keystore.jks
Enter source Java keystore password:
Uploading private key
Uploading private key

Upload Failed
WARNING: Could not store private key error

Probable Cause

This error may occur when you upload two Java keystores using okvutil with the same file name but different contents, or if you use the same alias, like -alias slserver, in each keytool command.

Solution

1. When you download two keystores with the same alias, the okvutil download process ignores the second one because the JKS aliases must be unique.
2. To fix this error, upload the second keystore using a unique alias.

C.3.4 RESTful Services Endpoint Provisioning Command Failure

The endpoint provisioning using RESTful services utility fails due to incorrect path or directory.

Example

$ okv admin endpoint provision --generate-json-input
Error:/usr/bin/java does not exist.

Probable Cause

The RESTful service command to provision an endpoint fails if the soft link /usr/bin/java does not exist or points to an incorrect Java directory.

Solution

Ensure that you use Java version 1.7.21 or later. Create a soft link to the Java home directory:

ln -s Java_home_directory/bin/java /usr/bin/java

C.3.5 Uploading Certificate File Failure

When uploading a console certificate an error is displayed .

Example

Uploading a console certificate from Oracle Key Vault Management console returns the error.

ORA-20101: Failed to upload certificate file

Probable Cause

Mandatory certificates have expired.

Solution

1. Log in to the Oracle Key Vault server as a support user.
2. Switch to the root user.
3. Regenerate the certificates:

   # /usr/local/bin/gensslcert create-certs

4. Log in to the Oracle Key Vault management console as SYSADMIN.
5. Regenerate the console certificate request from the Console Certificate page.
6. Download and sign the console certificate.
7. Upload the signed console certificate to the Console Certificate page.
8. Verify if the issue is resolved.

C.3.6 Error in Uploading the Java Keystore

Oracle Key Vault displays an error when uploading the Java keystore.

Example

Uploading Java Keystore using okvutil fails with below error,

SEVERE: Error occured while determining entry type.
java.security.UnrecoverableKeyException: Cannot recover key

Probable Cause

The Java keystore (JKS) has multiple passwords. One password protects the complete keystore. Each private key also gets an additional password that can be different from the other keystore password. To upload a JKS from okvutil, all the key passwords should be the same as the keystore password.

Solution

1. Set both the keystore and the key passwords to the same value.
2. Upload JKS to Oracle Key Vault.
3. Reset the passwords again if you want them to be different.
   To change the keystore password:

   $ keytool -storepasswd -keystore keystorename
   Enter keystore password: <old password>
   New keystore password: <new password>
   Re-enter new keystore password: <new password>

   To change an individual key password:

   $keytool -keypasswd -keystore keystorename -alias <alias>
   Enter keystore password: <keystore password>
   Enter key password for <alias> <old key password>
   New key password for <alias>: <new key password>
   Re-enter new key password for <alias>: <new key password>

4. Upload the entire keystore as an Other type instead of JKS, if the passwords cannot be changed. The password is stored as a binary file and you must track the passwords elsewhere.

   Note:

   You get a centralized repository for backup and distribution, but cannot manage keys and certifications at an object-level granularity.

C.3.7 SSL layer Error while migrating MYSQL Database Keys to Oracle Key Vault

An SSL layer error displays when the MySQL database is migrated to Oracle Key Vault.

Example

Below errors are seen in the mysql database trace or log file.

reported: 'Error setting the certificate file.'
reported: 'Could not initialize ssl layer'
reported: 'keyring_okv initialization failure. Please check that the
keyring_okv_conf_dir points to a readable directory and that the directory
contains Oracle Key Vault configuration file and ssl materials. 

Probable Cause

Corrupted certificate file.

Solution

1. Check if Oracle Key Vault is operational.
2. Edit the certificate files copied from the SSL folder extracted from okvclient.jar.
3. Delete any blank line at the end of the certificate file and click Save.
4. Verify if the issue is resolved.