Server and Node Issues

Review these troubleshooting tips for common server and node related errors when working with Oracle Key Vault.

Enabling a Read/Write Node Hangs After Upgrade

When upgrading a read/write node from a release earlier than Oracle Key Vault 21.9 to a release later than Oracle Key Vault 21.9, enabling the upgraded node may not complete. The following error is recorded in the alert log of the node being enabled:

Fatal NI connect error 29024, connecting to ...

Probable Cause

The platform certificate used for redo shipping, stored in the avwallet wallet, and the orapwdbfwdb database password file are not synchronized between the read/write nodes.

Solution

This procedure compares the platform certificate used for redo shipping, stored in the avwallet wallet, and the orapwdbfwdb database password file on the node being enabled and the other read/write peer node. If either differs, copy the avwallet contents or database password file from the node being enabled to the other read/write peer node. Then restart the required services and enable the cluster nodes from the management console.

  1. On both Oracle Key Vault servers, SSH into the system as user support, then switch to user root:

    ssh support@<OKV_IP_address>
su - root

  2. On both servers, generate the Oracle Key Vault health check reports:

    /usr/local/okv/bin/okv_healthcheck

    Note: For Oracle Key Vault 21.10 and earlier versions, follow the MOS note “OKV Server Healthcheck Utility Version 2.0 (KB164555)” to run okv_healthcheck.

  3. Open the generated health check reports from both nodes and verify the expiry dates of the PLATFORM_AVWALLET_CERT certificates.

    If the expiry dates differ, perform the following steps to copy the avwallet certificate files from the node being enabled to the other read/write peer node.

    1. On the node being enabled, as the root user, run:

      cd /var/lib/oracle/dbfw/network/admin/avwallet
zip -r avwallet_regenerated.zip *
scp avwallet_regenerated.zip support@<OKV_PEER_NODE_IP_ADDRESS>:/tmp

    2. On the other read/write peer node, as the root user, run:

      cd /tmp
unzip avwallet_regenerated.zip -d avwallet_regenerated
chmod -R 775 avwallet_regenerated
cd /var/lib/oracle/dbfw/network/admin
cp -Rp avwallet avwallet_expired
cp /tmp/avwallet_regenerated/* avwallet/

  4. On both read/write nodes, as the root user, verify the checksum of the /var/lib/oracle/dbfw/dbs/orapwdbfwdb file:

    cksum /var/lib/oracle/dbfw/dbs/orapwdbfwdb

    If the checksums differ, perform the following steps:

    1. On the node being enabled, as the root user, run:

      scp /var/lib/oracle/dbfw/dbs/orapwdbfwdb support@<OKV_PEER_NODE_IP_ADDRESS>:/tmp

    2. On the other read/write peer node, as the root user, run:

      cp /var/lib/oracle/dbfw/dbs/orapwdbfwdb /var/lib/oracle/dbfw/dbs/orapwdbfwdb_bak
cp /tmp/orapwdbfwdb /var/lib/oracle/dbfw/dbs/orapwdbfwdb
chown oracle:oinstall /var/lib/oracle/dbfw/dbs/orapwdbfwdb
chmod u+rw /var/lib/oracle/dbfw/dbs/orapwdbfwdb

    3. Verify the ownership and permissions of the copied file:

      ls -l /var/lib/oracle/dbfw/dbs/orapwdbfwdb

      Confirm that the file owner is oracle, the group is oinstall, and the oracle user has read and write permissions.

  5. On the other read/write peer node, not the node being enabled, restart the following services as the root user:

    service dbfwlistener restart
service dbfwdb restart

  6. Restart cluster services and enable node using Oracle Key Vault management console:

    1. Log in to the Oracle Key Vault management console as a System Administrator.
    2. Navigate to the Cluster Monitoring page.
    3. Click Restart Cluster Services. Wait a few minutes, then refresh the page.
    4. Select all cluster nodes, then click Enable.
    5. Monitor the cluster status. After a few minutes, the upgraded node should appear as Active on all cluster nodes.

SSL Client Error Message

The alert and trace log display the SSL client error message when the Server Domain Name does not contain the expected Security Identifier (SID) name.

Alert log from Oracle Key Vault Server shows the following error message.

SSL Client: Server DN does not contain expected SID name

Probable Cause

These messages are from earlier SSL configurations.

Solution

Ignore these messages.

Incorrect Value Returned for Custom Attributes of Integer Type

In certain scenarios, an invalid value is returned for an integer type custom attribute.

This problem scenario applies to values that were created from an Oracle Key Vault version 21.2.0.0.0 or earlier. Retrieving an integer type custom attribute created by RESTful services utility using C or Java SDK may return an invalid value. It is also possible that a custom attribute results in an invalid value.

Probable Cause

In Oracle Key Vault versions 21.2.0.0.0 or earlier, when an integer type custom attribute is added or modified using RESTful services utility, the attribute value is stored in a representation that is different from the representation that is used by other interfaces, for example, C or JAVA SDK, and KMIP.

This means that a value created or modified using the RESTful service utility cannot be retrieved correctly with C or JAVA SDK and KMIP interfaces. Likewise, a value created or modified using C or JAVA SDK and KMIP interfaces could not be retrieved correctly using RESTful service utility and the Oracle Key Vault management console.

Note: The values that are created or modified using Oracle Key Vault 21.3.0.0.0 or later are always returned correctly.

The invalid values for a custom attribute of integer types are returned in these cases:

The correct value is returned when the value is created and retrieved using the same interface.

In a multi-master cluster, Oracle Key Vault version in this section refers to the cluster version of the deployment.

Solution

To identify the values that are suspect and establish a correct value for them use the following procedure after the upgrade to Oracle Key Vault version 21.3.0.0.0 or later. A value for the custom attribute of integer type is considered suspect for cross utility use if it was created prior to Oracle Key Vault 21.3.0.0.0. This includes the values that may already be stored using the correct representation. Because from the value itself, it is not feasible to determine the representation, all such values are considered suspect and must be corrected.

  1. Identify suspect values for the custom attributes of integer type:

    1. Log in to the Oracle Key Vault server through SSH as user support, then switch user su to root.

      ssh support@okv_server_IP_address
su - root

    2. Run the script to generate a report with the suspected values:

      /usr/bin/su - okv -c
      /usr/local/okv/bin/gen_custom_attr_suspect_values

      A report with the list of suspected values is generated:

      /tmp/suspect_values_for_custom_attribute_integer_type.txt

      For each entry in the report, following values are shown:

      • Creating Endpoint: Endpoint that created the value.

      • Unique ID of Object: Unique ID (UUID) of the object.

      • Custom Attribute Name: Name of the custom attribute.

      • Index: Index of the value.

      • SDK Value: Value as retrieved by the C/JAVA SDK interfaces.

      • REST Value: Value as retrieved by the RESTful services utility.

      Between SDK and REST values, one of the values will be the correct value.

  2. Update the suspect value with the chosen correct value.

    For each suspect entry:

    1. Review the SDK and REST values.

    2. Determine the correct value of the custom attribute from the two possible values. One of the values will be the correct value of the custom attribute.

      In some cases, a ‘-‘ may be shown for one of the values. In such cases, the correct value would be the one that is shown as the integer value.

    3. Update the custom attribute with the chosen correct value. You can use any interface to update this value.

      You must update the custom attribute value even when one of the value is shown as ‘-‘.

      This step may require the coordination between the root user and the Oracle Key Vault users who can update the suspect values. A user who has the Key Administrator role can update all suspect values. In addition, a user or an endpoint who has read/write access on an object can modify the suspect custom attribute value for that object. You can determine such users using the endpoint information shown under ‘Creating Endpoint’.

It is recommended to complete this procedure by establishing the correct value for all suspect values in one iteration. However, in case if it becomes necessary, above procedure could be repeated and then it will include only the remaining suspect values in the report.

It is recommended to verify the completion of this procedure by executing Step 1 again and ensuring that the generated report does not contain any suspect value entries.

Not Receiving Email Alerts

Even after configuring SMTP successfully, administrators are not receiving email alerts from the Oracle Key Vault server.

Probable Cause 1

Requires restart of tomcat service

Solution

  1. Login to the Oracle Key Vault server or node through ssh and switch user to root.

  2. Restart tomcat service.

    $service tomcat status
$service tomcat stop
$service tomcat start

  3. Verify if the tomcat service is up and running.

    $service tomcat status
ps -eaf | grep tomcat

  4. Send a test email and see if the email is received.

Probable Cause 2

Email ID is not configured at the user level.

Solution

  1. Log in to the Oracle Key Vault Management console as a system administrator.

  2. Select the Users tab, and then Manage Users from the left navigation bar.

    The Manage Users page is displayed with the list of users.

  3. Click the user’s name in the User Name column.

    The User Details page is displayed.

  4. Type the email address of the user in the Email field, and then click Save.

Oracle Key Vault Server and NTP Server Date and Time Not Synchronized

Learn two methods to resolve Oracle Key Vault date and time unsynchronized issue.

Probable Cause

The time on the Oracle Key Vault server does not match with the time on the NTP server.

Solution

Perform the following steps to synchronize the time on the NTP and Oracle Key Vault server:

  1. Log in to the Oracle Key Vault management console as a system administrator role.

  2. Select the System tab, then Settings from the left navigation side bar.

  3. In the Network Services area, Select NTP to display the System Time page.

  4. Click Apply Server to perform a re-synchronization of the clock on the Oracle Key Vault server with the NTP server.

Alternatively, you can also perform the following steps to synchronize the time on the NTP and the Oracle Key Vault server:

  1. Log in to the Oracle Key Vault server through SSH as user support, then switch user su to root.

    ssh support@okv_server_IP_address
su - root

  2. Run the following command to perform a re-synchronization of the Oracle Key Vault server clock:

    /bin/chronyc makestep

Failed To Enable FIPS mode

Enabling FIPS fails with an error on the Oracle Key Vault management console

Example

Failed to enable FIPS mode error seen on the Oracle key vault management console while enabling FIPS.

Probable Cause

The Oracle Key Vault server was upgraded from release 21.5 or previous releases and the FIPS was disabled before the upgrade.

Solution

  1. As the support user, log in to the Oracle Key Vault server using SSH, and then switch user su to root.

    ssh support@okv_server_IP_address
su - root

  2. Run the following command:

    /usr/local/okv/bin/okv_fps_disable

  3. Restart the Oracle Key Vault server using the reboot command.

  4. As the sysadmin user, log in to the Oracle Key Vault management console to re-enable FIPS.

Non-Oracle Database Endpoints Fails to Connect to Oracle Key Vault

Non-Oracle Database endpoints such as MySQL, Mongo DB, fails to connect to Oracle Key Vault server during CA certificate rotation.

Example

Server communication or SSL peer certificate validation failed error seen on the Oracle Key Vault during CA certificate rotation.

Probable Cause

CA certificate rotation is not complete.

Solution

  1. Log in to the Oracle Key Vault Management console, verify if the CA certificate rotation is still in progress.

  2. Select and re-enroll all non-oracle endpoints from the Endpoint page if the certificate rotation is still in progress.

  3. If certificate rotation is still in progress, then on endpoints page re-enroll all non-oracle endpoints selecting all of them.

  4. When the CA certificate rotation is complete and the new CA certificate is active perform the following steps:

    1. Go to the Endpoint page and copy the token id of the desired non-oracle endpoint.

    2. Download the okvclient.jar file.

    3. Copy the okvclient.jar file to the endpoint system.

    4. Extract the ssl certificates and use it for the respective Database configuration.

  5. After copying the ssl certificates, reboot the Database.

  6. Verify if the issue is fixed.