5 Vendor Instructions for Integrating an HSM as the Root of Trust for Oracle Key Vault

HSM vendors are responsible for documenting and testing instructions to integrate their HSM with Oracle Key Vault.

5.1 About Integrating an HSM as a Root of Trust for Oracle Key Vault

This type of integration enables Oracle Key Vault to use HSMs other than Thales Luna Network HSM version 7000, Entrust nShield Connect + and XC models, or Utimaco’s SecurityServer 4.31.1.

You, the vendor, need to work with the Oracle Key Vault administrator to complete the integration. Oracle Key Vault provides files to enable the setting of parameters and environment variables, as well as scripts to help with the upgrade process. The Oracle Key Vault administrator will need to configure these scripts. You will need to provide documentation on how to use and configure them, and be responsible for updating this documentation for subsequent releases.

In the general, you will need to perform the following steps:

  1. Ensure that your HSM meets the requirements for performing the integration, which include the ability to install the HSM client software, verifying the required functionality of the PKCS#11 library, and the ability to enroll the Oracle Key Vault as a client of the HSM.
  2. Provide documentation for managing the configuration and environment variable files, and the upgrade scripts. Ensure that you update this documentation as the HSM's needs change. Oracle Key Vault provides the following files: okv_hsm.conf, okv_hsm_env, and okv_hsm_mid_upgrade.
  3. After you document the necessary steps to integrate Oracle Key Vault with the HSM, confirm the success of the integration. This includes enabling the HSM, and performing tasks such as upgrades, backups, and restore operations.
  4. Validate the HSM as a Root of Trust (RoT) in standalone, multi-master cluster, and primary-standby environments.

5.2 Requirements Before Starting the Integration

The vendor of the HSM to be integrated with Oracle Key Vault must meet certain requirements, such as providing instructions on how to install the client software and ensuring the PKCS#11 library provides the required functionality.

5.2.1 Requirements for the Vendor's HSM Client Installation Software

The HSM client software installation process must fulfill requirements such as installing the HSM client software into its own directory

As the vendor, you must do the following:

  • Install the HSM client software into its own directory on the same server where Oracle Key Vault is installed, in a new subdirectory under the /opt directory.
  • Set file permissions for the client software, which uses the PKCS#11 library, by running as the oracle user.
  • Only allow the root user to modify any configuration files that are provided by the HSM client software.
  • Ensure any new services introduced by the HSM client software are started before the Oracle Key Vault dbfwdb.service service. Do not edit the dbfwdb.service unit configuration file.

5.2.2 Requirements for the PKCS#11 Library

As the vendor, ensure that your library supports all of the necessary PKCS#11 functionality.

The PKCS#11 library must supply the following functions:

  • C_CloseSession
  • C_CreateObject
  • C_Decrypt
  • C_DecryptInit
  • C_Encrypt
  • C_EncryptInit
  • C_Finalize
  • C_FindObjects
  • C_FindObjectsFinal
  • C_FindObjectsInit
  • C_GenerateKey
  • C_GenerateRandom
  • C_GetAttributeValue
  • C_GetFunctionList
  • C_GetSlotList
  • C_GetTokenInfo
  • C_Initialize
  • C_Login
  • C_OpenSession
  • C_SetAttributeValue

5.2.3 Configuration Requirements for Enrolling Oracle Key Vault as the HSM Client

HSMs typically require additional configuration for enrolling Oracle Key Vault as a client of the HSM.

This enrollment enables the communication between Oracle Key Vault and the HSM.

As the vendor, you must document and maintain these detailed instructions. The requirements are as follows:

  • The instructions must ensure that the HSM configuration does not affect other Oracle Key Vault files or components.
  • The communication between the client and the HSM server must be encrypted.

5.3 Integrating the HSM

The integration process includes setting up configuration files and scripts.

5.3.1 okv_hsm.conf Parameters

Oracle Key Vault provides a configuration file to enable Oracle Key Vault administrators to define information such as the vendor name.

The okv_hsm.conf file is located in the /usr/local/okv/hsm/generic/ directory. As the vendor, ensure that you document the correct way to complete the okv_hsm.conf configuration file, including updating this documentation for subsequent releases.

The okv_hsm.conf file contains the following settings:

  • VENDOR_NAME="name_of_vendor"
    • This parameter is required.
    • This setting enables the Oracle Key Vault management console to display the name_of_vendor value in the vendor drop-down lists when users perform the HSM initialize and set credential operations.
  • PKCS11_LIB_LOC="pkcs11_library_path"
    • This parameter is required.
    • The Oracle Key Vault administrator must specify the full path to the location of the installed HSM PKCS#11 library, and include this file in the list of files that are defined in the PRESERVED_FILES parameter.
  • PRESERVED_FILES="path_to_directory:path_to_file:pkcs11_library_path"
    • The parameter is required, but it will not be used until the next major version upgrade.
    • Oracle Key Vault upgrades are to the next major Oracle Key Vault version, or to a newer upgrade of the same major Oracle Key Vault version. Upgrading to a newer major Oracle Key Vault version is called a release upgrade or major version upgrade. Upgrading to a newer minor version is termed as release update upgrade or minor version upgrade. Upgrading to Oracle Key Vault release 21.x from Oracle Key Vault release 18.x is an example of a major version upgrade. Similarly, upgrading from Oracle Key Vault release 21.1 to Oracle Key Vault release 21.2 is an example a minor version upgrade. Major version upgrades can also refresh the major version of OL of the Oracle Key Vault appliance. For example, an upgrade from Oracle Key Vault release 18 to Oracle Key Vault release 21 refreshes the OS version from OL 6 to OL 7.
    • Oracle Key Vault uses this parameter for major version upgrades only, when the OS version is refreshed to a newer major version. This parameter enables vendor-specific files and directories to survive these upgrades. Files and directories that are not specified using this parameter will likely be deleted. Therefore, all files and directories that were created while you set up the HSM client software or are required for Oracle Key Vault to be able to communicate with HSM using the PKCS#11 library should be included. Do not specify files and directories that existed on Oracle Key Vault by default.
    • The Oracle Key Vault administrator will need to specify files and directories using the following format, including separating each of the paths by colons (:), as shown here:
      PRESERVED_FILES="path_to_directory:path_to_file:pkcs11_library_path"

5.3.2 okv_hsm_env Environment File

Oracle Key Vault provides an environment file so that Oracle Key Vault administrators can specify environment variables that are needed when Oracle Key Vault uses the PKCS#11 library.

If the HSM's PKCS#11 library requires special environment variables, then the Oracle Key Vault administrator can include these variables in the okv_hsm_env file. Ensure that you document the correct way to complete the okv_hsm_env file, including updating this documentation for subsequent releases.

5.3.3 okv_hsm_mid_upgrade Script

Oracle Key Vault provides a script that can help to reapply configurations that may be lost during an upgrade.

As the vendor, ensure that you document the correct way to complete the okv_hsm_mid_upgrade script, including updating this documentation for subsequent releases.

This script is necessary during upgrades, particularly a major version that upgrades the operating system version, to ensure that information such as user accounts are recreated in the newer Oracle Key Vault version. Because Oracle Key Vault will need access to the Root of Trust (RoT) key in the HSM after the appliance restarts with the newer operating system version, these changes must be executed mid-upgrade.

As the vendor, you must address any configurations that need to be modified during the upgrade. The script must perform the following actions:

  • Check the installed HSM software, users, groups, services, or any other configuration that is needed for Oracle Key Vault to successfully connect to the HSM.
  • Correct any configuration (users, groups, configuration, services) that was changed during upgrade or missing after upgrade.

For example, if your HSM installation includes a service as a part of the HSM client software setup, and the service is needed for Oracle Key Vault to connect to the HSM, then in the mid-upgrade script, you should check and enforce that both the service exists and is running. If you created a user, in the mid-upgrade script, then you should check both that the user exists and that it continues to own any HSM-related files that it owned before the upgrade was started (or recreate the user and chown any files as appropriate). Anything that is not a preserved file or directory as specified in the parameter file is liable to be removed, so ideally this script would check and fix all vendor-owned files and directories (and their permissions) on the Oracle Key Vault server itself in detail.

You can rely on host name resolution to work during this step, provided that DNS servers were saved in Oracle Key Vault's settings before upgrading.

5.3.4 Pre- or Post-Upgrade Validation Scripts

You can provide pre- or post-upgrade scripts that the Oracle Key Vault administrator can run before and after the upgrade process to ensure that the HSM configuration is correct.

These scripts are useful for Oracle Key Vault administrators to ensure that all HSM-related configurations are as expected either before the upgrade begins or after the upgrade completes.

As the vendor, ensure that you document the correct way to configure these scripts, including updating this documentation for subsequent releases.

5.4 Confirming the Success of the HSM Configuration

Testing operations such as HSM-enabling, upgrading, backing up, and restoring Oracle Key Vault while integrated with an HSM is essential to confirming a successful integration.

5.4.1 Enabling the HSM in a Standalone Environment

The HSM should be enabled as a Root of Trust (RoT) for a standalone Oracle Key Vault environment in much the same way that any of the standard HSMs, such as Thales Luna Network HSM version 7000, Entrust nShield Connect + and XC models, or Utimaco’s SecurityServer 4.31.1, can be enabled.

The HSM vendor must provide instructions on how to specify the token label. Oracle recommends that a token label is specified.

The following behavior indicates that the HSM-enable operation is successful:

  • There should be a success message appear on the Hardware Security Module page of the Oracle Key Vault management console after the operation completes.
  • There should be a green, upward status arrow on the Hardware Security Module page on the Oracle Key Vault management console, with the correct HSM information beneath it.
  • After restarting Oracle Key Vault, the Oracle Key Vault management console should appear and a user should be able to log in. This ensures that the server can use the new HSM in unattended mode when the server is restarted. You can restart Oracle Key Vault from the Oracle Key Vault management console, as described in Oracle Key Vault Administrator's Guide.

Check the log files in the /var/okv/log/hsm directory.

This operation creates a RoT key and creates or updates a key number object in the HSM. The key number object is used to associate a unique key number with each RoT key that is created in the HSM. The key number starts at 1 and is incremented by 1 with each successive HSM-enable operation. To increment the number, the current value is fetched and updated. It is possible the key number object’s value could change before the update operation completes. The HSM vendor must take special care to document any additional steps that may be necessary to prevent conflicts (for example, two RoT keys created with the same key number). Failure to do so may render an Oracle Key Vault unable to create the auto-login wallet when the Oracle Key Vault server is restarted, and will require manual intervention. You can find the key number in the generated log files in the /var/okv/log/hsm directory.

5.4.2 Performing an Upgrade of an HSM-Enabled Oracle Key Vault Server

The vendor should be able to upgrade an Oracle Key Vault server that is integrated with other HSMs in much the same way that Oracle Key Vault is normally upgraded.

Perform the Oracle Key Vault upgrade, and include any additional steps that the HSM vendor's documentation requires.

When an Oracle Key Vault server is configured with an HSM, the following activities indicate that the upgrade was successful:

  • For all servers, there should be a message saying Oracle Key Vault Server version_number upgrade has completed on the console (with a blue background).
  • For a standalone, multi-master cluster, or primary-standalone configuration, log in to the Oracle Key Vault management console as a user who has the System Administrator role. Select the System tab. Verify that the version displayed is the latest release number. The release number is also at the bottom of each page, to the right of the copyright information.

5.4.3 Performing a Backup of an HSM-Enabled Oracle Key Vault Server

The vendor should be able to perform a backup of an HSM-enabled Oracle Key Vault server in the Oracle Key Vault management console.

Perform the Oracle Key Vault backup operation.

The following behavior indicates that the backup operation is successful:

  • In the Completed Backups table on the System Backup page in the Oracle Key Vault management console, the most recent backup should show DONE in the Status column and there should be nothing in the Run Error column.
  • The backup should be able to be successfully restored.

Check the log files in the /var/okv/log/hsm directory.

5.4.4 Performing a Restore of an HSM-Enabled Oracle Key Vault Server

The vendor should be able to perform a restore of an HSM-enabled Oracle Key Vault server from the Oracle Key Vault management console.

Perform the Oracle Key Vault restore operation.

The following behavior indicates that the restore operation is successful:

  • On the Restore page, under Last Restore Details in the Oracle Key Vault management console, the most recent backup should show DONE in the Status column and there should be nothing in the Run Error column.
  • Oracle Key Vault should automatically restart during the restore operation. If it did not restart, then it is likely that the restore operation did not complete successfully. Check /var/log/messages on the server for details. Also check the log files in the /var/okv/log/hsm directory.

5.4.5 Confirming the Success of the HSM Configuration in a Multi-Master Cluster

To confirm the HSM configuration in a multi-master cluster environment, the vendor must be able to HSM-enable a multi-master cluster starting with a single node or an entire cluster.

5.4.5.1 Starting with an HSM-Enabled Single Node

The vendor should be able to build a multi-master cluster, starting with a single HSM-enabled node, and then add other HSM-enabled nodes to the cluster.

  1. HSM-enable one node, and then add a second HSM-enabled node as a read-write peer of the first node.
    After the pairing completes, the Oracle Key Vault management console Cluster Management page on both nodes must show both nodes in the ACTIVE state.
  2. Restart both nodes, as described in Oracle Key Vault Administrator's Guide.
  3. After Oracle Key Vault restarts, ensure that the Oracle Key Vault management console is accessible to both of the nodes.
  4. Add a third HSM-enabled node to the cluster, using either the first or the second node as the controller node.
    After the pairing completes, the Oracle Key Vault management console Cluster Management page on all three nodes must show that these nodes are in the ACTIVE state.
  5. Restart the third node, and then ensure that the Oracle Key Vault management console is accessible to this node.
  6. Create a wallet on both of the read-write nodes.
  7. Ensure that the status for both of the wallets goes from PENDING to ACTIVE.
    To do so, check the wallets' name status on all three nodes.
5.4.5.2 HSM-Enabling a Cluster with Multiple Nodes

The vendor must first build the multi-master cluster with no HSM-enabled nodes. From there, the vendor HSM-enables each node one at a time.

  1. Create or use an existing multi-master cluster that has at least three nodes, in which two are read-write peer nodes and one is a read-only node.
  2. HSM-enable one node in this cluster.
  3. Create the HSM bundle on the HSM-enabled node and apply the HSM bundle to the other two nodes in the cluster.
  4. Restart all nodes, as described in Oracle Key Vault Administrator's Guide.
  5. After Oracle Key Vault restarts, confirm that the Oracle Key Vault management console is accessible to these nodes and that replication works between all three nodes.
  6. HSM-enable the other two nodes.
  7. Restart the three nodes.
  8. After Oracle Key Vault restarts, ensure that the Oracle Key Vault management console is accessible to all three nodes.
  9. Create a wallet on both of the read-write nodes.
  10. Ensure that the status for both of the wallets goes from PENDING to ACTIVE.
    To do so, check the wallets' name status on all three nodes.

5.4.6 Confirming the Success of the HSM Configuration in a Primary-Standby Environment

To confirm the HSM configuration in a primary-standby environment, the vendor should perform switchover and reverse migration operations.

5.4.6.1 Creating an HSM-Enabled Primary Standby Configuration

The vendor should be able to configure both the primary and standby Oracle Key Vault servers to use an HSM as a Root of Trust (RoT).

After you complete the configuration, in the Oracle Key Vault management console of the primary server Primary-Standby Status page, you should see Primary-Standby mode is enabled, and the Switchover Status should be TO STANDBY.

Check the log files in the /var/okv/log/hsm directories of both the primary and standby servers.

5.4.6.2 Performing an HSM-Enabled Primary-Standby Switchover Operation

The vendor should be able to execute a switchover operation between a pair of HSM-enabled primary and standby Oracle Key Vault servers, as described in Oracle Key Vault Administrator's Guide.

After you complete the switchover operation, the Oracle Key Vault management console Primary-Standby Status page for the new primary should show Primary-Standby mode is enabled, and the Switchover Status should be TO STANDBY.

Check the log files in the /var/okv/log/hsm directories of both the primary and standby servers.

5.4.6.3 Performing a Reverse Migration of an HSM-Enabled Primary-Standby Configuration

The vendor should be able to perform a reverse migration of both the primary and standby HSM-enabled Oracle Key Vault servers.

After you complete the reverse migration operation on the primary Oracle Key Vault server, there should be a green success message, and there should be a red downward status arrow.

Next, do the following:

  1. Reverse migrate the standby Oracle Key Vault server.
  2. Perform a switchover operation as described in Oracle Key Vault Administrator's Guide, and then verify the operation’s success.
  3. Restart both the primary and standby Oracle Key Vault servers, as described in as described in Oracle Key Vault Administrator's Guide.
  4. Verify that the primary server can access the Oracle Key Vault management console.

5.4.7 Performing a Reverse Migration Operation

In this operation, the vendor removes the HSM as a Root of Trust (RoT) configuration by reverse migrating an Oracle Key Vault server.

Perform the reverse migration operation as is normally done in Oracle Key Vault.

The following behavior indicates that the reverse migration operation is successful:

  • There should be a success message appear on the Oracle Key Vault management console after the operation completes.
  • There should be a red, downward status arrow on the Hardware Security Module page on the Oracle Key Vault management console.
  • Restart the Oracle Key Vault server, as described in Oracle Key Vault Administrator's Guide. After doing so, the Oracle Key Vault management console should appear and a user should be able to log in. This ensures that the server no longer has to use the HSM to recreate the auto-login wallet when the server is restarted.

Check the log files in the /var/okv/log/hsm directory.

5.5 Validating the HSM as a Root of Trust for Oracle Key Vault

After the vendor creates instructions on how to integrate their HSM with Oracle Key Vault, they should also validate the integration.

5.5.1 Test Cases for Standalone Configurations

For a standalone Oracle Key Vault server, the vendor should be able to HSM-enable a server (initialize) and perform backup and restore operations.

  • Test 1: Perform three successive HSM-enable operations. Perform the following operations on the Oracle Key Vault standalone server in this order:

    1. HSM-enable the server.

    2. Perform a reverse migration operation on the server.

    3. HSM-enable the server a second time.

    4. Perform a reverse migration operation on the server.

    5. HSM-enable the server a third time.

    Independently verify each operation to ensure that the operation is successful. This is important because the first three HSM-enable operations on the HSM do different things: the first HSM-enable operation creates the key number object that is used to track the current Root of Trust (RoT) key. This first HSM-enable operation also creates the RoT key. The second HSM-enable operation updates the key number object and creates a new RoT key. The third HSM-enable operation is the same as the second, but is used for verifying that the second HSM-enable operation successfully updated the key number object.

  • Test 2: Perform at least three successive HSM-enable operations on different Oracle Key Vault servers. Perform at least three initialize operations on different Oracle Key Vault servers, sequentially. Independently verify each operation to ensure that it is successful.

  • Test 3: Perform a backup and restore operation on the standalone Oracle Key Vault server. Create a remote and a local backup on an Oracle Key Vault server that is HSM-enabled. Verify each for success. Afterward, restore the local backup to the Oracle Key Vault server on which it was taken. Restore the remote backup to a different Oracle Key Vault server. Verify each for success.

5.5.2 Test Cases for Multi-Master Cluster Configurations

For Oracle Key Vault servers in a multi-master cluster environment, the vendor should be able to HSM-enable nodes, perform backup and restore operations, reverse migrate a cluster, and upgrade a cluster.

As the vendor, perform the following tests:

  • Test 1: Create an HSM-enabled cluster configuration, starting with a single HSM-enabled node. Afterward, verify that this operation was successful.
  • Test 2: Create an HSM-enabled cluster configuration, HSM-enabling a cluster with multiple nodes. Afterward, verify that this operation was successful.
  • Test 3: Perform a backup and restore operation on the HSM-enabled nodes. Create a remote backup on more than one node and restore these nodes to freshly installed, separate Oracle Key Vault servers. Verify that the operations were successful.
  • Test 4: Reverse migrate the cluster that has HSM-enabled nodes. Reverse migrate each HSM-enabled node in the cluster. Verify that each operation was successful.
  • Test 5: Perform an upgrade on the cluster that has HSM-enabled nodes. Perform cluster upgrades from all previous versions of Oracle Key Vault on which your vendor is certified. Verify that each upgrade is successful on all nodes of the cluster.

5.5.3 Test Cases for Primary-Standby Configurations

For Oracle Key Vault servers in a primary-standby configuration, the vendor should be able to create the HSM-enabled primary-standby, and perform standard primary-standby operations.

These operations include switchovers, reverse migrations, backups and restores, and upgrades.

  • Test 1: Create an HSM-enabled primary-standby configuration. Verify that the operation was successful.
  • Test 2: Perform a primary-standby switchover operation. Verify that the operation was successful.
  • Test 3: Reverse migrate an HSM-enabled primary-standby configuration. Verify that the operation was successful.
  • Test 4: Perform a backup and restore operation on the HSM-enabled configuration. Create a remote backup on the primary server and then restore it to a freshly installed, separate Oracle Key Vault server. Verify that the operations were successful.
  • Test 5: Upgrade the HSM-enabled configuration. Perform primary-standby upgrades from all previous versions of Oracle Key Vault on which your vendor is certified. Verify that each upgrade is successful on both the primary and standby servers.