4 Support Guidance

The support guidance provides information about troubleshooting, vendor specific notes, and the CNSA Suite support.

4.1 General Troubleshooting

This section covers general troubleshooting help. Vendor specific troubleshooting is covered in the vendor specific notes.

4.1.1 hsm_initialize: Could Not Get Slot for HSM

This error indicates that Oracle Key Vault is not properly enrolled as a client of the HSM. Check vendor-specific instructions for more information.

4.1.2 hsm_initialize: Could Not Load PKCS#11 Library

This error indicates that Oracle Key Vault is not properly enrolled as a client of the HSM. Check vendor-specific instructions for more information.

4.1.3 Oracle Key Vault Management Console Does Not Start After Restarting HSM-Enabled Oracle Key Vault Server

If the management console does not appear after restarting the HSM-enabled Oracle Key Vault server, log into the Oracle Key Vault server using SSH as user support and verify the following:

  • Try to manually open the wallet:
    $ ssh support@okv_instance
    <Enter password when prompted>
    $ su root 
    root# su oracle
    $ cd /usr/local/okv/hsm/bin
    $ ./hsmclient open_wallet

    If the open_wallet command succeeds, the database will open and the management console will appear, unless there is another non-HSM problem. If the command does not succeed, check for vendor-specific instructions. Otherwise, copy the output and contact Oracle Support.

  • If using DNS with the HSM configuration, due to the known issue, Bug 24478865, ensure that DNS entries are both in the management console (System tab > System Settings page > DNS section) and in /etc/resolv.conf.

    An example configuration of /etc/resolv.conf:

    search <default search domains>
    nameserver <dns ip 1>
    nameserver <dns ip 2>
    nameserver <dns ip 3>

4.1.4 Primary-Standby Errors

  1. Check that the files have been transported to the standby server:

    Execute the command ls -l as root on the standby server:

    $ ls -l /usr/local/okv/hsm/wallet
    -rw------- 1 oracle oinstall 324 May 16 22:57 cwallet.sso
    -rw------- 1 oracle oinstall 176 May 16 22:57 enctdepwd
    $ ls -l /usr/local/okv/hsm/restore
    -rw------- 1 oracle oinstall 320 May 16 22:57 ewallet.p12
    

    You must see cwallet.sso and enctdepwd in the /usr/local/okv/hsm/wallet directory and ewallet.p12 in the /usr/local/okv/hsm/restore directory.

  2. Check that the mode is set to HSM on the standby server:

    Open the file okv_security.conf as root on the standby server:

    $ cat /usr/local/okv/etc/okv_security.conf
     Look for the line:
     HSM_ENABLED="1"
    

    You must see the number within double quotes.

  3. Check the vendor-specific instructions.

4.1.5 Backup

You must check that the pre_restore command has been run on the target as follows:

Execute the command ls -l as root on the standby server:

$ ls -l /usr/local/okv/hsm/wallet
-rw------- 1 oracle oinstall 324 May 16 22:57 cwallet.sso

You must see the wallet file cwallet.sso .

You must also check that you have followed the instructions from the HSM vendor.

4.1.6 Restoring an HSM-Enabled Backup

This procedure must only be used in a restore operation and you must enter the HSM credential correctly. If you enter an incorrect credential or if Oracle Key Vault is unable to connect to the HSM, the credential will not be stored. Ensure that Oracle Key Vault is enrolled as a client of the HSM and then ensure that the correct credential has been entered.

For more information about enrolling Oracle Key Vault as a client of the HSM, see Enroll Oracle Key Vault as a Client of HSM.

4.2 Vendor Specific Notes - SafeNet

Release 12.2 BP 1 and higher support Oracle Key Vault integration with SafeNet (Gemalto) Luna SA 7000. The use of a Host Trust Link (HTL) for SafeNet Luna HSM is unsupported at this time.

The following installation and enrollment instructions apply to the SafeNet Luna SA 7000 HSM.

4.2.1 Install the HSM Client Software on the Oracle Key Vault Server

To install the HSM client on Oracle Key Vault:

  1. Obtain the SafeNet client software package, version 6.2 for Linux x64. For the purposes of this document, we will refer to this as "safenet.tar".

  2. Transport the SafeNet client software package to the Oracle Key Vault machine. Oracle recommends using scp, for example:

    scp safenet.tar support@[okv hostname]:/tmp

  3. Install the SafeNet client software on Oracle Key Vault.

  4. Log in to the Oracle Key Vault Server through SSH as user support, and switch user (su) to root:

    $ ssh support@okv_instance
    $ su root 
    $ cd /usr/local/okv/hsm 
    $ cp /tmp/safenet.tar /usr/local/okv/hsm 
    $ tar -xvf safenet.tar 
    $ cd 64 
    $ ./install.sh
  5. Accept the SafeNet license by typing ‘y’ at the prompt.

  6. Install the Luna SA by entering ‘1’, ‘n’, ‘i’ at the successive prompts:

    This installs the SafeNet software in the directory /usr/safenet/lunaclient .

  7. Delete the safenet.tar file from /tmp directory.

     $ rm -f /tmp/safenet.tar

4.2.2 HSM Credential

The HSM credential is the SafeNet partition password. You choose a partition with the client assignPartition command.

4.2.3 Enroll Oracle Key Vault as a Client of HSM

To enroll Oracle Key Vault as an HSM client:

  1. Set the DNS servers for Oracle Key Vault via the management console from System -> System Settings. This step is required for the Luna SA to communicate with Oracle Key Vault.

    If using DNS with the HSM configuration, due to the known issue, Bug 24478865, ensure that DNS entries are both in the management console (System tab > System Settings page > DNS section) and in /etc/resolv.conf.

  2. Exchange certificates between Oracle Key Vault and the Luna SA:

    Log in to the Oracle Key Vault Server through SSH as user support, and switch user (su) to root:

    $ ssh support@okv_instance
    $ su root 
    $ cd /usr/safenet/lunaclient/bin 
    $ scp admin@[hsm hostname]:server.pem . 
    $ ./vtl addServer -n [hsm hostname] -c server.pem 
    $ ./vtl createCert -n [okv hostname] 
    $ scp /usr/safenet/lunaclient/cert/client/[okv hostname].pem admin@[hsm hostname]:

    You will need to enter the HSM admin password when using scp with the HSM.

  3. Register Oracle Key Vault as a client of the Luna SA. This assumes you have a partition set up on the Luna SA. You can use any client name that is not yet taken. Oracle recommends using a descriptive name that will identify the Oracle Key Vault instance.

    Access the HSM administrative console by using SSH to admin@[hsm hostname] and providing the admin password:

    $ client register -client [client name] -hostname [okv hostname]
    $ client hostip map -c [client name] -i [okv IP]
    $ client assignPartition -client [client name] -partition [partition name] 
  4. Verify enrollment:

    Login to Oracle Key Vault as the support user using SSH:

    $ su root 
    $ cd /usr/safenet/lunaclient/bin 
    $ ./vtl verify

    The following output appears:

    The following Luna SA Slots/Partitions were found:
    
    Slot    Serial #        Label
    ====    ========        =====
     1      [serial #]      [partition name]
     

4.2.4 HSM Provider Value

For Safenet, the provider value is 1. If setting manually for primary-standby, set HSM_PROVIDER="1". For more information about enabling HSM in a primary-standby deployment, see Enabling HSM in a High Availability Deployment.

4.2.5 HSM Vendor Specific Checks

The instructions in this section apply to the SafeNet (Gemalto) Luna SA 7000 only.

You can verify the connection to the HSM for every Oracle Key Vault server as follows:

Login to the Oracle Key Vault server as user support using SSH:

$ ssh support@okv_instance
$ su root
$ cd /usr/safenet/lunaclient/bin
$ ./vtl verify

The following output appears when the HSM is set up properly:

The following Luna SA Slots/Partitions were found:

Slot    Serial #        Label
====    ========        =====
 1      [serial #]      [partition name]

If you do not see this output, it means that the HSM is not set up properly. You may diagnose further as follows:

  1. Log into the Luna SA administrative console.

  2. Type the command: client show -client [client name]

  3. Verify that the expected client exists and is assigned a partition.

  4. If it does not exist, register the client with the command:

    client register -client [client name]-hostname [hostname]

  5. If no partition is assigned, assign a partition with the command:

    client assignPartition -client [client name] -partition [partition name]

  6. Verify that all client IPs are mapped correctly. If entries are missing, run the command:

    client hostip map -c [client name] -i [ip]

4.2.6 Verify Connection between Oracle Key Vault and SafeNet

You can verify that Oracle Key Vault can reach the HSM using the vtl verify command as follows:
$ su root
root# cd /usr/safenet/lunaclient/bin
root# ./vtl verify
The following output appears:
The following Luna SA Slots/Partitions were found:

Slot    Serial #        Label
====    ========        =====
 1      [serial #]      [partition name]
If the command fails, it means that the Oracle Key Vault server is unable to contact the HSM. Check the vendor’s other troubleshooting sections for instructions to restore vtl verify functionality. Contact your HSM administrator and confirm that Oracle Key Vault's access to the HSM has not been revoked. If you are unable to resolve the problem, contact Oracle Support.

4.3 Vendor Specific Notes - nCipher

Release 12.2 BP 3 and higher support Oracle Key Vault integration with the nCipher HSM. At this time, only the nCipher nShield Connect 6000+ is supported.

The following installation and enrollment instructions apply to the nCipher HSM.

4.3.1 Install the HSM Client Software on the Oracle Key Vault Server

The nCipher HSM requires a separate non-HSM computer on the network to use as the Remote File System. You must set up this computer and copy the nCipher software files to it before you start.

To install the nCipher software on the Oracle Key Vault server do:

  1. Log in to the OKV server as support user using SSH:
    $ ssh support$okv_instance
    <Enter the support user password when prompted>
  2. Switch to root:
    $ su root
  3. Go to the root directory and create the directories ctls, hwsp, and pkcs11:
    root# cd /root
    root# mkdir ctls
    root# mkdir hwsp
    root# mkdir pkcs11
  4. Transfer the nCipher software installation files using the Secure Copy (SCP) protocol as follows:

    For example:
    root# scp <user@remote_file_system_machine>:/<your_source_directory>/ncipher/nfast/ctls/agg.tar ctls
    root# scp <user@remote_file_system_machine>:/<your_source_directory>/ncipher/nfast/hwsp/agg.tar hwsp
    root# scp <user@remote_file_system_machine>:/<your_source_directory>/ncipher/nfast/pkcs11/user.tar pkcs11
    
  5. Install these files as follows:
    root# cd /
    root# tar xvf /root/ctls/agg.tar
    root# tar xvf /root/hwsp/agg.tar
    root# tar xvf /root/pkcs11/user.tar
    root# /opt/nfast/sbin/install
    
  6. As root perform additional edits on the Oracle Key Vault server:
    root# usermod -a -G nfast oracle
    root# cd /etc/rc.d/rc5.d
    root# mv S50nc_hardserver S40nc_hardserver
    root# cd /etc/rc.d/rc3.d
    root# mv S50nc_hardserver S41nc_hardserver
  7. Switch to user oracle and verify the installation:
    root# su oracle
    oracle$ PATH=/opt/nfast/bin:$PATH
    oracle$ export PATH      
    oracle$ enquiry
    The state should say “operational” in the output.
  8. Reboot the system for the group change to take effect.

4.3.2 HSM Credential

The HSM credential is the Operator Card Set password.

4.3.3 Enroll Oracle Key Vault as a Client of HSM

Enroll Oracle Key Vault as an HSM client as follows:

  1. Add the Oracle Key Vault server IP address to the client list on the HSM using the front panel. Select privileged on any port.

  2. Switch to user oracle :
    root# su oracle
    oracle$ PATH=/opt/nfast/bin:$PATH
    oracle$ export PATH
  3. On the Oracle Key Vault server, enroll with the HSM :
    oracle$ nethsmenroll --privileged <HSM IP address> <HSM ESN> <HSM keyhash> 
  4. Configure TCP sockets:
    oracle$ config-serverstartup --enable-tcp --enable-privileged-tcp
  5. Switch to root and restart the hardserver (nCipher client process that communicates with the HSM):
    oracle$ su root
    root# /opt/nfast/sbin/init.d-ncipher restart
  6. On the Remote File System machine run the following command:
    rfs-setup --gang-client --write-noauth <IP address of your Oracle Key Vault server>
  7. On the Oracle Key Vault server as user oracle run the commands:
    oracle$ rfs-sync --setup --no-authenticate <IP address of Remote File System machine> 
    oracle$ rfs-sync --update
  8. Test PKCS#11 access as follows:
    root# /opt/nfast/bin/ckcheckinst
    A prompt appears listing the module. You can confirm or exit.
  9. Create the config file /opt/nfast/cknfastrc as user root. Write the following lines to the file:
    CKNFAST_NO_ACCELERATOR_SLOTS=1
    CKNFAST_OVERRIDE_SECURITY_ASSURANCES=explicitness;tokenkeys;longterm
  10. Perform the steps described in Protect the Oracle Key Vault TDE Master Key with the HSM.

  11. On the Oracle Key Vault server as user oracle run the command:
    oracle$ /opt/nfast/bin/rfs-sync --commit 
    

4.3.4 HSM Provider Value

For nCipher, the provider value is 2. If setting manually for primary-standby, set HSM_PROVIDER="2". For more information about enabling HSM in a primary-standby deployment, see Enabling HSM in a High Availability Deployment.

4.3.5 Enable HSM Mode

After installing HSM software and enrolling Oracle Key Vault as an HSM client, you can enable HSM mode with nCipher from the Oracle Key Vault user interface on the management console. Just select nCipher from the vendor drop-down list.

4.3.6 Backup

To take a backup of the Oracle Key Vault server in HSM mode, do the following:

  1. Install a new Oracle Key Vault server.

  2. Install the nCipher HSM software as described in a previous section.

  3. From the Oracle Key Vault user interface add the backup destination on the System Backup page, just as you would in non-HSM mode.

  4. Perform a backup as usual from the user interface on the management console.

4.3.7 Restore

To restore an Oracle Key Vault server from a backup, do the following:

  1. Go to the Prepare for HSM Restore page from the user interface.

  2. Select nCipher from the Vendor drop down list and enter the HSM credential twice as requested.

  3. Click Set Credential .

  4. Log in to the Oracle Key Vault Server through SSH as user support, switch user (su) to root, then switch user (su) to oracle.

    $ ssh support@okv_instance
    <Enter password when prompted> 
    $ su root
    root# su oracle
  5. Run the following command, which retrieves information from the RFS:
    oracle$ /opt/nfast/bin/rfs-sync --update
  6. Restore via the user interface as usual, as in non-HSM mode.

4.3.8 HSM in a Primary-Standby Oracle Key Vault Installation

This procedure shows you how to pair two Oracle Key Vault servers in HSM mode in a primary-standby configuration. You must enable HSM mode in both primary and standby Oracle Key Vault servers before pairing them. To configure the HSM for primary-standby, please see the vendor documentation.

To configure Oracle Key Vault with nCipher HSM in a primary-standby installation, do the following:

  1. Install Oracle Key Vault on two servers that you mean to designate as primary and standby.

  2. Install the nCipher HSM software on each Oracle Key Vault server.

  3. On the server you mean to designate as primary server do the following:

    • Log in to the designated Oracle Key Vault Primary Server through SSH as user support, switch user (su) to root, then switch user (su) to oracle:
      $ ssh support@okv_primary_instance
      <Enter password when prompted>
      $ su root
      root# su oracle
    • Run the following command:
      oracle$ /opt/nfast/bin/rfs-sync --update
  4. From the user interface on the Oracle Key Vault management console initialize the intended primary server for HSM mode with nCipher.

  5. On the server you mean to designate as primary server do the following:

    • Log in to the designated Oracle Key Vault Primary Server through SSH as user support, switch user (su) to root, then switch user (su) to oracle:
      $ ssh support@okv_primary_instance
      <Enter password when prompted>
      $ su root
      root# su oracle
    • Run the following command:
      oracle$ /opt/nfast/bin/rfs-sync --commit
  6. Repeat Step 3 on the intended standby server.

  7. Perform the following manual steps on the intended primary as user oracle:
    $ ssh support@okv_primary_instance
    <Enter password when prompted>
    $ su root
    root# su oracle
    oracle$ cd /usr/local/okv/hsm/wallet
    oracle$ scp cwallet.sso support@standby:/tmp
    oracle$ scp enctdepwd support@standby:/tmp
    oracle$ cd /usr/local/okv/hsm/restore
    oracle$ scp ewallet.p12 support@standby:/tmp

    Note:

    While performing this procedure on Oracle Key Vault 12.2.0.5.0 and earlier, use the commands in HSM in a Primary-Standby Oracle Key Vault Installation.

  8. Perform the following manual steps on the intended standby as user root:
    $ ssh support@okv_standby_instance
    <Enter password when prompted>
    $ su root
    root# cd /usr/local/okv/hsm/wallet
    root# mv /tmp/enctdepwd .
    root# mv /tmp/cwallet.sso .
    root# chown oracle *
    root# chgrp oinstall *
    root# cd /usr/local/okv/hsm/restore
    root# mv /tmp/ewallet.p12 .
    root# chown oracle *
    root# chgrp oinstall *
    

    Note:

    While performing this procedure on Oracle Key Vault 12.2.0.5.0 and earlier, use the commands in HSM in a Primary-Standby Oracle Key Vault Installation.

  9. Continuing as user root open the file okv_security.conf for writing:
    root# vi /usr/local/okv/etc/okv_security.conf
    
  10. Make two updates to the file as follows:

    1. Set the variable HSM_ENABLED to 1. If the variable does not exist, add it and set its value to 1.
      HSM_ENABLED="1"
    2. Add the following line:
      HSM_PROVIDER="2"	
  11. Then proceed to set up primary-standby as usual via the user interface on the Oracle Key Vault management console.

4.4 CNSA Suite Support

Oracle Key Vault 12.2 BP3 and higher offer compliance with the Commercial National Security Algorithm (CNSA) for TLS connections to and from the appliance.

The CNSA suite is a list of strong encryption algorithms and key lengths, that offer greater security and relevance into the future. A link to the full CNSA specification is in the Related Links section that follows this section.

Note that 12.2 BP3 and higher do not provide complete compliance across every component in the system. You will be able to switch to the CNSA algorithms, where available by means of two scripts that are packaged with the ISO:

  1. The first script /usr/local/okv/bin/okv_cnsa makes configuration file changes to update as many components as possible to use the enhanced algorithms. It is reversible and will not interfere with existing operations.

  2. The second script /usr/local/okv/bin/okv_cnsa_cert regenerates CNSA compliant public key pairs and certificates.

    Note:

    The second script /usr/local/okv/bin/okv_cnsa_cert is disruptive because it replaces the old key pairs with new ones. This has consequences for the following operations:
    • Endpoint Enrollment: Enroll endpoints after running this script when possible. If you had endpoints enrolled before running the CNSA script, you must re-enroll them so that fresh CNSA compliant keys are generated using CNSA algorithms.

    • Primary-Standby: Run the CNSA scripts on both Oracle Key Vault instances before pairing them in a primary-standby configuration when possible. If you had primary-standby set up prior to running the CNSA scripts, you must re-configure primary-standby as follows: unpair the primary and standby servers, run the CNSA scripts individually on each server, then pair them again.

4.4.1 Running the CNSA Scripts

To run the CNSA scripts, do the following:

  1. Install Oracle Key Vault and complete the post-installation tasks. The last post-installation task is to set the support user password, which is needed now.

  2. Log into the Oracle Key Vault browser-based management console and enable SSH access to the server.

  3. SSH into the Oracle Key Vault server as the support user. Enter the support user password created during post-installation, when prompted.
     $ ssh support@okv_instance
     <Enter support user password created during post-installation>
  4. Change to root user:
    $  su root
  5. Run the scripts as root user:
    root#  /usr/local/okv/bin/okv_cnsa
    root#  /usr/local/okv/bin/okv_cnsa_cert
  6. The scripts put data into /usr/local/okv/etc/okv_security.conf.

    The line USE_ENHANCED_ALGORITHMS_ONLY="1" will be added if the scripts are run.

4.4.2 Backup

After restoring a backup, re-run the first script:/usr/local/okv/bin/okv_cnsa to update the configuration to use the enhanced CNSA algorithms as follows:

  1. Wait for the system to reboot after the restore operation initiated via the user interface of the Oracle Key Vault management console.

  2. SSH into the Oracle Key Vault server as the support user:
     $ ssh support@okv_instance
     <Enter support user password created during post-installation>
  3. Switch to root user:
    $   su root
  4. Run the first CNSA script :
     root#  /usr/local/okv/bin/okv_cnsa
    

4.4.3 Upgrade

You must re-run the first script during the upgrade to ensure CSNA compliance as follows:

  1. Execute Step 8 of the upgrade procedure which is to run the ruby script as root:
    root# /usr/bin/ruby/images/upgrade.rb --format
  2. Run the first CNSA script :
     root#  /usr/local/okv/bin/okv_cnsa
    
  3. Continue with Step 9 of upgrade procedure:
    root# /sbin/reboot

Limitations:

  • CNSA compliance is not supported for some components in the Oracle Key Vault infrastructure, for example SSH, or for the database encryption via TDE.

  • The Firefox browser is not supported for use with the Oracle Key Vault management console when CNSA is enabled. This is because the Firefox browser does not support CNSA-approved cipher suites.