1. Root of Trust HSM Configuration Guide
  2. Support Guidance

4 Support Guidance

The support guidance provides information about troubleshooting and vendor specific notes.

  • General Troubleshooting
    Oracle Key Vault provides general troubleshooting help. Vendor-specific notes cover vendor-specific troubleshooting.
  • Vendor Specific Notes for SafeNet
    Oracle Key Vault supports Oracle Key Vault integration with SafeNet Luna SA Hardware Security Modules from Thales version 7000, but does not support Host Trust Link (HTL) for SafeNet Luna HSM.
  • Vendor Specific Notes for nCipher
    You can integrate Oracle Key Vault release 12.2 BP 3 and later with the HSM from nCipher nShield Connect 6000+.

4.1 General Troubleshooting

Oracle Key Vault provides general troubleshooting help. Vendor-specific notes cover vendor-specific troubleshooting.

Parent topic: Support Guidance

4.1.1 Trace Files for Diagnosing Issues

Oracle Key Vault provides trace files so that you can better diagnose issues that may arise.

Use these trace files to more finely diagnose issues when you attempt hardware security module operations. These trace files are located in the /var/okv/log/hsm/ directory on the Oracle Key Vault server. To see the most recently failed operation, you can sort the trace files by their last modified time. For example, ls -ltr /var/okv/log/hsm lists the most recently modified trace files at the bottom of the list.

Parent topic: General Troubleshooting

4.1.2 HSM Alert

Oracle Key Vault provides an alert mechanism that periodically monitors the HSM configuration to check for Root of Trust key availability and file health.

When an Oracle Key Vault server is HSM-enabled, Oracle Key Vault contacts the HSM every five minutes (or whatever you have set the monitoring interval to) to ensure that the Root of Trust key is available and the TDE wallet password can be decrypted. When a problem in the HSM configuration arises (for example, the HSM cannot be reached or if there are conflicting keys in the HSM with the same ID), then the up arrow on the Hardware Security Module tab switches to a down arrow and an alert is raised. The down arrow signifies that the HSM is not configured or the HSM configuration has a problem. When the HSM configuration encounters a problem (for example, the HSM cannot be reached or if there are conflicting keys in the HSM with the same ID), the up arrow on the Hardware Security Module tab switches to a down arrow and an alert is raised. The down arrow signifies that the HSM is not configured or the HSM configuration has a problem. When an alert has been raised, an error saying HSM configuration error. Please refer to the HSM Alert section in the Oracle Key Vault HSM Integration Guide. appears.

If this alert appears, then follow these steps:

  1. Log in as root as follows: 
    ssh support@okv_instance_IP_address
Enter support user password when prompted.
su - root
Enter root user password when prompted.
  2. Back up the SSO wallet. For example:
    cp /mnt/okvram/cwallet.sso /var/lib/oracle/cwallet_hsm_backup.sso
  3. Diagnose the source of the alert.

    The following verify command should show why the alert was raised. The ls -ltrh command shows the most recent log file at the bottom of the output. 

    su - oracle
/usr/local/okv/hsm/bin/hsmclient verify
cd /var/okv/log/hsm
ls -ltrh
  4. If you cannot resolve this problem, then contact Oracle Support.

Parent topic: General Troubleshooting

4.1.3 hsm_initialize: Could Not Get Slot for HSM

The hsm_initialize: Could Not Get Slot for HSM error indicates that Oracle Key Vault is not properly enrolled as a client of the HSM.

Check the vendor-specific instructions for more information.

Parent topic: General Troubleshooting

4.1.4 hsm_initialize: Could Not Load PKCS#11 Library

The hsm_initialize: Could Not Load PKCS#11 Library error indicates that Oracle Key Vault is not properly enrolled as a client of the HSM.

Check the vendor-specific instructions for more information.

Parent topic: General Troubleshooting

4.1.5 Oracle Key Vault Management Console Does Not Start After Restarting HSM-Enabled Oracle Key Vault Server

You can remedy this problem by using the support account.

If the Oracle Key Vault management console does not appear after you restart the HSM-enabled Oracle Key Vault server, then log into the Oracle Key Vault server using SSH as user support and try manually opening the wallet as follows: 

$ ssh support@okv_instance_IP_address
$ su root 
root# su oracle
$ cd /usr/local/okv/hsm/bin
$ ./hsmclient open_wallet

If the open_wallet command succeeds, the database will open and the management console will appear, unless there is another non-HSM problem. If the command does not succeed, then check for vendor-specific instructions. Otherwise, provide a copy of the output to Oracle Support.

Parent topic: General Troubleshooting

4.1.6 Primary-Standby Errors

The okv_security.conf file contains settings that can help you diagnose primary-standby errors.

  1. Check that the files have been transported to the standby server.

    Execute the command ls -l as root on the standby server: 

    $ ls -l /usr/local/okv/hsm/wallet
-rw------- 1 oracle oinstall 324 May 16 22:57 cwallet.sso
-rw------- 1 oracle oinstall 176 May 16 22:57 enctdepwd
$ ls -l /usr/local/okv/hsm/restore
-rw------- 1 oracle oinstall 320 May 16 22:57 ewallet.p12

    You should see cwallet.sso and enctdepwd in the /usr/local/okv/hsm/wallet directory and ewallet.p12 in the /usr/local/okv/hsm/restore directory.

  2. Check that the mode is set to HSM on the standby server:

    Open the file okv_security.conf as root on the standby server: 

    $ cat /usr/local/okv/etc/okv_security.conf
 Look for the line:
 HSM_ENABLED="1"

    You should see the number within double quotes.

  3. Check the vendor-specific instructions.

Parent topic: General Troubleshooting

4.1.7 Errors from HSM-Enabled Oracle Key Vault Backups

You can use the cwallet.sso file to diagnose HSM-enabled Oracle Key Vault backup errors.

You should check that the pre_restore command has been run on the target as follows:

Execute the command ls -l as root on the Oracle Key Vault server to which you are restoring the backup: 

$ ls -l /usr/local/okv/hsm/wallet
-rw------- 1 oracle oinstall 324 May 16 22:57 cwallet.sso

You should see the wallet file cwallet.sso .

You should also check that you have followed the instructions from the HSM vendor.

Parent topic: General Troubleshooting

4.1.8 Restoring an HSM-Enabled Backup

Before you restore an HSM-enabled backup, ensure that you have the correct HSM credential.

Consult the HSM documentation for this credential. For SafeNet, the credential is the SafeNet partition password. For nCipher, the credential is the Operator Card Set password.

This procedure must only be used in a restore operation and you must enter the HSM credential correctly. If you enter an incorrect credential or if Oracle Key Vault is unable to connect to the HSM, then the credential will not be stored. Ensure that Oracle Key Vault is enrolled as a client of the HSM and then ensure that the correct credential has been entered.

For more information about enrolling Oracle Key Vault as a client of the HSM, see Enrolling Oracle Key Vault as a Client of the HSM.

Parent topic: General Troubleshooting

4.2 Vendor Specific Notes for SafeNet

Oracle Key Vault supports Oracle Key Vault integration with SafeNet Luna SA Hardware Security Modules from Thales version 7000, but does not support Host Trust Link (HTL) for SafeNet Luna HSM.

Parent topic: Support Guidance

4.2.1 Installing the HSM Client Software on the Oracle Key Vault Server for SafeNet

You must use the SafeNet client version 6.2 for Linux x64 for the installation.

  1. Obtain the SafeNet client software package, version 6.2 for Linux x64.

  2. Transport the SafeNet client software package to the Oracle Key Vault machine. Oracle recommends using SCP. For example, assuming the SafeNet client software packages is called safenet.tar:

    scp safenet.tar support@okv_instance_ip_address:/tmp

  3. Install the SafeNet client software on Oracle Key Vault.

  4. Log in to the Oracle Key Vault Server through SSH as user support, and switch user (su) to root: 

    $ ssh support@okv_instance_IP_address
$ su root 
$ cd /usr/local/okv/hsm 
$ cp /tmp/safenet.tar /usr/local/okv/hsm 
$ tar -xvf safenet.tar 
$ cd 64 
$ ./install.sh

  5. Accept the SafeNet license by typing y at the prompt.

  6. Install the Luna SA by entering 1, n, i at the successive prompts:

    This installs the SafeNet software in the directory /usr/safenet/lunaclient .

  7. Delete the safenet.tar file from /tmp directory. 

     $ rm -f /tmp/safenet.tar

Parent topic: Vendor Specific Notes for SafeNet

4.2.2 HSM Credential for SafeNet

The HSM credential is the SafeNet partition password.

If you are using SafeNet as your HSM, then you can use the SafeNet assignPassword command to assign an HSM SafeNet partition password as the credential for a partition. Oracle Key Vault then uses this password when it needs to connect to the partition.

Parent topic: Vendor Specific Notes for SafeNet

4.2.3 Enrolling Oracle Key Vault as a Client of a SafeNet HSM

To perform the enrollment, you must use the client commands at the command line.

  1. Set the DNS servers for Oracle Key Vault via the management console. Log in to the Oracle Key Vault management console as a user who has the System Administrator role, and then to access the DNS settings area, from the System tab, select System Settings.

    This step is required for the SafeNet Luna SA HSM to communicate with Oracle Key Vault.

    You must configure the DNS servers on each Oracle Key Vault server that you plan to register as a client of the HSM. In a primary-standby environment, configure the DNS servers on both primary and standby server before pairing. For a multi-master cluster, configure DNS on each node in the cluster that will be registered as a client of the HSM.

    If you are using DNS with the HSM configuration, then due to the known issue, Bug 24478865 (fixed in Oracle Key Vault release 18c), ensure that DNS entries are both in the following locations:

    • Oracle Key Vault management console: from the System tab, select System Settings, and then check the DNS setting on that page.
    • The /etc/resolv.conf file

  2. Exchange certificates between Oracle Key Vault and the SafeNet Luna SA HSM.

    Log in to the Oracle Key Vault Server through SSH as user support, and switch user (su) to root: 

    $ ssh support@okv_instance_IP_address
$ su root 
$ cd /usr/safenet/lunaclient/bin 
$ scp admin@hsm_hostname:server.pem . 
$ ./vtl addServer -n hsm_hostname -c server.pem 
$ ./vtl createCert -n okv_hostname 
$ scp /usr/safenet/lunaclient/cert/client/okv_hostname.pem admin@hsm_hostname:

    You must enter the HSM administrative password when using SCP with the HSM.

  3. Register Oracle Key Vault as a client of the Luna SA.

    This assumes that you have a partition set up on the SafeNet Luna SA HSM. You can use any client name that is not yet taken. Oracle recommends using a descriptive name that will identify the Oracle Key Vault instance.

    Access the HSM administrative console by using SSH to admin@hsm_hostname and providing the administrative password: 

    $ client register -client client_name -hostname okv_hostname
$ client hostip map -c client_name -i okv_IP_address
$ client assignPartition -client client_name -partition partition_name

  4. Verify the enrollment as follows:

    Log in to Oracle Key Vault as the support user using SSH:

    $ su root 
$ cd /usr/safenet/lunaclient/bin 
$ ./vtl verify

    The following output appears:

    The following Luna SA Slots/Partitions were found:

Slot    Serial #        Label
====    ==========      =====
 1      serial_number   partition_name

Parent topic: Vendor Specific Notes for SafeNet

4.2.4 HSM Provider Value for SafeNet

For SafeNet, the provider value is 1.

If you are setting this value manually for primary-standby, set HSM_PROVIDER="1" in the okv_security.conf file. For more information about enabling HSM in a primary-standby deployment, see Enabling HSM in a High Availability Deployment.

Parent topic: Vendor Specific Notes for SafeNet

4.2.5 HSM Vendor Specific Checks for SafeNet

You should check the SafeNet vendor-specific settings.

You can verify the connection to the HSM for every Oracle Key Vault server as follows:

Log in to the Oracle Key Vault server as user support using SSH: 

$ ssh support@okv_instance_IP_address
$ su root
$ cd /usr/safenet/lunaclient/bin
$ ./vtl verify

The following output appears when the HSM is set up properly:

The following Luna SA Slots/Partitions were found:

Slot    Serial #        Label
====    ========        =====
 1      [serial #]      [partition name]

If you do not see this output, then it means that the HSM is not set up properly. You can diagnose further as follows:

  1. Log into the Luna SA administrative console.

  2. Type the command: client show -client client_name

  3. Verify that the expected client exists and is assigned a partition.

  4. If it does not exist, register the client with the command:

    client register -client client_name-hostname host_name

  5. If no partition is assigned, assign a partition with the command:

    client assignPartition -client client_name -partition partition_name

  6. Verify that all client IPs are mapped correctly. If entries are missing, run the command:

    client hostip map -c client_name -i ip_address

Parent topic: Vendor Specific Notes for SafeNet

4.2.6 Verifying the Connection Between Oracle Key Vault and SafeNet

You can verify that Oracle Key Vault can reach the HSM using the vtl verify command.

Execute the following commands;
$ su root
root# cd /usr/safenet/lunaclient/bin
root# ./vtl verify
The following output appears:
The following Luna SA Slots/Partitions were found:

Slot    Serial #        Label
====    ========        =====
 1      [serial #]      [partition name]
If the command fails, then it means that the Oracle Key Vault server is unable to contact the HSM. Check the vendor’s other troubleshooting sections for instructions to restore vtl verify functionality. Contact your HSM administrator and confirm that Oracle Key Vault's access to the HSM has not been revoked. If you are unable to resolve the problem, then contact Oracle Support.

Parent topic: Vendor Specific Notes for SafeNet

4.3 Vendor Specific Notes for nCipher

You can integrate Oracle Key Vault release 12.2 BP 3 and later with the HSM from nCipher nShield Connect 6000+.

Parent topic: Support Guidance

4.3.1 Installing the HSM Client Software on the Oracle Key Vault Server for nCipher

The nCipher HSM requires a separate non-HSM computer on the network to use as the remote file system.

You must set up this computer and copy the nCipher software files to it before you start.

  1. Log in to the Oracle Key Vault server as support user using SSH:
    $ ssh support@okv_instance_IP_address
  2. Switch to root: 
    $ su root
  3. Go to the root directory and create the directories ctls, hwsp, and pkcs11:
    root# cd /root
root# mkdir ctls
root# mkdir hwsp
root# mkdir pkcs11

  4. Transfer the nCipher software installation files using the Secure Copy (SCP) protocol as follows:

    For example:
    root# scp user@remote_file_system_computer:/source_directory/ncipher/nfast/ctls/agg.tar ctls
root# scp user@remote_file_system_computer:/source_directory/ncipher/nfast/hwsp/agg.tar hwsp
root# scp user@remote_file_system_computer:/source_directory/ncipher/nfast/pkcs11/user.tar pkcs11
  5. Install these files as follows: 
    root# cd /
root# tar xvf /root/ctls/agg.tar
root# tar xvf /root/hwsp/agg.tar
root# tar xvf /root/pkcs11/user.tar
root# /opt/nfast/sbin/install
  6. As root, perform additional edits on the Oracle Key Vault server: 
    root# usermod -a -G nfast oracle
root# cd /etc/rc.d/rc5.d
root# mv S50nc_hardserver S40nc_hardserver
root# cd /etc/rc.d/rc3.d
root# mv S50nc_hardserver S41nc_hardserver
  7. Switch to user oracle and verify the installation: 
    root# su oracle
oracle$ PATH=/opt/nfast/bin:$PATH
oracle$ export PATH      
oracle$ enquiry
    The state should say operational in the output.

  8. Restart Oracle Key Vault for the group change to take effect.

    In the Oracle Key Vault management console, log in as a user with the System Administrator role. Select the System tab, and then select System Settings. Then click the Reboot button.

Parent topic: Vendor Specific Notes for nCipher

4.3.2 HSM Credential for nCipher

The HSM credential is the Operator Card Set password.

Parent topic: Vendor Specific Notes for nCipher

4.3.3 Enrolling Oracle Key Vault as a Client of an nCipher HSM

You use both the nCipher user interface and the command line to enroll Oracle Key Vault as a client of an nCipher HSM.

  1. Log in to the nCipher user interface as an HSM administrator.

  2. Add the Oracle Key Vault server IP address to the client list on the HSM using the front panel. Select privileged on any port.

    • In a primary-standby environment, register both the primary server and the standby server to use the nCipher HSM.
    • In a multi-master cluster environment, register each Oracle Key Vault node that will use the nCipher HSM.
  3. Switch to user oracle:
    root# su oracle
oracle$ PATH=/opt/nfast/bin:$PATH
oracle$ export PATH
  4. On the Oracle Key Vault server, enroll with the HSM :
    oracle$ nethsmenroll hsm_ip_address hsm_esn hsm_keyhash
  5. Configure the TCP sockets: 
    oracle$ config-serverstartup --enable-tcp --enable-privileged-tcp
  6. Switch to root and restart the hardserver (nCipher client process that communicates with the HSM): 
    oracle$ su root
root# /opt/nfast/sbin/init.d-ncipher restart
  7. On the remote file system computer, run the following command:
    rfs-setup --gang-client --write-noauth okv_server_ip_address
  8. On the Oracle Key Vault server as user oracle, run the following commands: 
    oracle$ rfs-sync --setup --no-authenticate remote_file_system_ip_address 
oracle$ rfs-sync --update
  9. Test PKCS#11 access as follows: 
    root# /opt/nfast/bin/ckcheckinst
    A prompt appears listing the module. You can confirm or exit.
  10. Create the config file /opt/nfast/cknfastrc as user root. Write the following lines to the file:
    CKNFAST_NO_ACCELERATOR_SLOTS=1
CKNFAST_OVERRIDE_SECURITY_ASSURANCES=explicitness;tokenkeys;longterm

  11. Perform the steps described in Protecting the Oracle Key Vault TDE Master Key with the HSM.

  12. On the Oracle Key Vault server as user oracle run the command: 
    oracle$ /opt/nfast/bin/rfs-sync --commit

Parent topic: Vendor Specific Notes for nCipher

4.3.4 HSM Provider Value for nCipher

For nCipher, the provider value is 2.

If you are setting this value manually for the primary-standby, then set HSM_PROVIDER="2". For more information about enabling HSM in a primary-standby deployment, see Enabling HSM in a High Availability Deployment.

Parent topic: Vendor Specific Notes for nCipher

4.3.5 Enabling HSM Instances for nCipher

After installing HSM software and enrolling Oracle Key Vault as an HSM client, you can enable an nCipher HSM for an Oracle Key Vault instance.

Use the Oracle Key Vault management console to enable the nCipher HSM.

  1. Log into the Oracle Key Vault management console as a user with the System Administrator role.

    For a multi-master cluster environment, log in to each Oracle Key Vault node where you want to enable the HSM.

  2. Click the System tab.
  3. Click Hardware Security Module in the left sidebar.

    The Hardware Security Module page appears. The red downward arrow shows the non-initialized Status. The Type field displays None

  4. Click Initialize.
  5. In the Initialize HSM page, do the following:
    • From the Vendor menu, select nCipher.
    • Enter and then reenter the HSM credential for the nCipher HSM, which is the Operator Card Set password.
    • Enter the Oracle Key Vault recovery passphrase.
  6. Click Initialize.

Parent topic: Vendor Specific Notes for nCipher

4.3.6 Backup Operations for nCipher

You can back up the Oracle Key Vault server in HSM mode.

  1. Install a new Oracle Key Vault server.

  2. Install the nCipher HSM software.

  3. From the Oracle Key Vault user interface add the backup destination on the System Backup page, just as you would in for a node that is not HSM enabled.

  4. Perform a backup as usual from the user interface on the management console.

Related Topics

Parent topic: Vendor Specific Notes for nCipher

4.3.7 Restore Operations for nCipher

You can restore an Oracle Key Vault server from a backup for nCipher.

  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
  2. Click the System tab.
  3. Click Hardware Security Module in the left sidebar.
  4. Click Set Credential to display the Prepare for HSM Restore page.

  5. Select nCipher from the Vendor drop down list and enter the HSM credential twice as requested.

    The HSM credential for nCipher is the Operator Card Set password.

  6. Click Set Credential.

  7. Log in to the Oracle Key Vault Server through SSH as user support, switch user (su) to root, then switch user (su) to oracle. 

    $ ssh support@okv_instance_IP_address
<Enter password when prompted> 
$ su root
root# su oracle
  8. Run the following command, which retrieves information from the remote file system:
    oracle$ /opt/nfast/bin/rfs-sync --update

  9. Restore using the Oracle Key Vault management console as you would for a node that is not HSM enabled.

Related Topics

Parent topic: Vendor Specific Notes for nCipher

4.3.8 HSM in a Primary-Standby Oracle Key Vault Installation for nCipher

You can pair two Oracle Key Vault servers in that have HSM-enabled nodes in a primary-standby configuration for nCipher.

You must HSM-enable the nodes in both primary and standby Oracle Key Vault servers before pairing them. To configure the HSM for primary-standby, please see the vendor documentation.

  1. Install Oracle Key Vault on two servers that you mean to designate as primary and standby.

  2. Install the nCipher HSM software on each Oracle Key Vault server.

  3. On the server to be used as the primary server, do the following:

    • Log in to the designated Oracle Key Vault primary server through SSH as user support, switch user (su) to root, then switch user (su) to oracle:
      $ ssh support@okv_primary_instance_IP_address
<Enter password when prompted>
$ su root
root# su oracle
    • Run the following command:
      oracle$ /opt/nfast/bin/rfs-sync --update

  4. From the user interface on the Oracle Key Vault management console initialize the intended primary server for the nCipher HSM-enabled node.

  5. On the server to be used as the primary server, do the following:

    • Log in to the designated Oracle Key Vault Primary Server through SSH as user support, switch user (su) to root, then switch user (su) to oracle:
      $ ssh support@okv_primary_instance_IP_address
$ su root
root# su oracle
    • Run the following command:
      oracle$ /opt/nfast/bin/rfs-sync --commit

  6. Repeat Step 3 on the intended standby server.

  7. Perform the following manual steps on the intended primary as user oracle:
    $ ssh support@okv_primary_instance_IP_address
$ su root
root# su oracle
oracle$ cd /usr/local/okv/hsm/wallet
oracle$ scp cwallet.sso support@standby:/tmp
oracle$ scp enctdepwd support@standby:/tmp
oracle$ cd /usr/local/okv/hsm/restore
oracle$ scp ewallet.p12 support@standby:/tmp
  8. Perform the following manual steps on the intended standby as user root:
    $ ssh support@okv_standby_instance_IP_address
$ su root
root# cd /usr/local/okv/hsm/wallet
root# mv /tmp/enctdepwd .
root# mv /tmp/cwallet.sso .
root# chown oracle *
root# chgrp oinstall *
root# cd /usr/local/okv/hsm/restore
root# mv /tmp/ewallet.p12 .
root# chown oracle *
root# chgrp oinstall *

    Note:

    While performing this procedure on Oracle Key Vault 12.2.0.5.0 and earlier, use the commands in Enabling an HSM in a Primary-Standby Pre-Release 12.2 Oracle Key Vault Installation.

  9. Continuing as user root, open the file okv_security.conf:
    root# vi /usr/local/okv/etc/okv_security.conf

  10. Make two updates to the file as follows:

    1. Set the variable HSM_ENABLED to 1. If the variable does not exist, add it and set its value to 1. 
      HSM_ENABLED="1"
    2. Add the following line:
      HSM_PROVIDER="2"

  11. Proceed to configure the primary-standby environment by using the Oracle Key Vault management console.

Related Topics

Parent topic: Vendor Specific Notes for nCipher

4.3.9 HSM Upgrades Using nCipher

You can only upgrade while HSM-enabled when you upgrade from Oracle Key Vault release 18.1.0.0.0 and later to a higher version.

Because HSM configurations can vary, it is your responsibility to run test upgrades on non-production environments to ensure that the upgrade will work with your HSM configuration.
  1. Connect as the root user. 
    $ ssh support@okv_primary_instance_IP_address
$ su root
  2. Execute the steps to upgrade as described in Oracle Key Vault Administrator's Guide, up to and including the command where you run the following command: 
    /usr/bin/ruby/images/upgrade.rb --confirm
  3. After running /usr/bin/ruby/images/upgrade.rb --confirm but before you execute the reboot operation, execute the following commands:
    usermod -a -G nfast oracle
cd /etc/rc.d/rc5.d
mv S50nc_hardserver S40nc_hardserver
cd /etc/rc.d/rc3.d
mv S50nc_hardserver S41nc_hardserver
  4. Continue with the upgrade process as described in Oracle Key Vault Administrator's Guide.

Related Topics

Parent topic: Vendor Specific Notes for nCipher