4 Support Guidance

The support guidance provides information about troubleshooting and vendor specific notes.

4.1 General Troubleshooting

Oracle Key Vault provides general troubleshooting help. Vendor-specific notes cover vendor-specific troubleshooting.

4.1.1 Trace Files for Diagnosing Issues

Oracle Key Vault provides trace files so that you can better diagnose issues that may arise.

Use these trace files to more finely diagnose issues when you attempt hardware security module operations. These trace files are located in the /var/okv/log/hsm/ directory on the Oracle Key Vault server. To see the most recently failed operation, you can sort the trace files by their last modified time. For example, ls -ltr /var/okv/log/hsm lists the most recently modified trace files at the bottom of the list.

4.1.2 HSM Alert

Oracle Key Vault provides an alert mechanism that periodically monitors the HSM configuration to check for Root of Trust key availability and file health.

When an Oracle Key Vault server is HSM-enabled, Oracle Key Vault contacts the HSM every five minutes (or whatever you have set the monitoring interval to on the Configure Alerts page) to ensure that the Root of Trust key is available and the TDE wallet password can be decrypted. When a problem in the HSM configuration arises (for example, the HSM cannot be reached or if there are conflicting keys in the HSM with the same ID), then the up arrow on the Hardware Security Module tab switches to a down arrow and an alert is raised. The down arrow signifies that the HSM is not configured or the HSM configuration has a problem. When an alert has been raised, the following error message appears: HSM configuration error. Please refer to the HSM Alert section in the Oracle Key Vault Root of Trust HSM Configuration Guide.

If this alert appears, then follow these steps:

  1. Log in as root as follows:
    $ ssh support@okv_instance_ip_address
    support$ su - root
  2. Back up the SSO wallet. For example:
    root# cp /mnt/okvram/cwallet.sso /var/lib/oracle/cwallet_hsm_backup.sso
  3. Diagnose the source of the alert.

    The following verify command should show why the alert was raised. The ls -ltrh command shows the most recent log file at the bottom of the output.

    root# su - oracle
    oracle$ /usr/local/okv/hsm/bin/hsmclient verify
    oracle$ cd /var/okv/log/hsm
    oracle$ ls -ltrh
  4. If you cannot resolve this problem, then contact Oracle Support.

4.1.3 Could Not Get Slot for HSM Error

The Could Not Get Slot for HSM error indicates that Oracle Key Vault could not get a slot from the HSM.

Consult the most recent trace files for more details. Possible causes include providing an invalid or nonexistent token label and the HSM failing to return a list of slots.

4.1.4 Could Not Load PKCS#11 Library Error

The Could Not Load PKCS#11 Library error indicates that Oracle Key Vault could not load the PKCS#11 library.

Possible reasons for this error could be due to file permission issues or failing to properly deploy the HSM client software on Oracle Key Vault. More details can be found in recent trace files. Oracle looks for the PKCS#11 library at the following locations, depending on the vendor:

  • For Thales Luna, /usr/safenet/lunaclient/lib/libCryptoki2_64.so
  • For nCipher, /opt/nfast/toolkits/pkcs11/libcknfast.so
  • For Utimaco, /opt/utimaco/lib/libcs_pkcs11_R2.so

4.1.5 Oracle Key Vault Management Console Does Not Start After Restarting HSM-Enabled Oracle Key Vault Server

The Oracle Key Vault management console may not appear after you restart the HSM-enabled Oracle Key Vault server.

If this happens, then log into the Oracle Key Vault server using SSH as user support and try manually opening the wallet as follows:

$ ssh support@okv_instance_ip_address
support$ su root 
root# su oracle
oracle$ cd /usr/local/okv/hsm/bin
oracle$ ./hsmclient open_wallet

If the open_wallet command succeeds, the database will open and the management console will appear, unless there is another non-HSM problem. If the command does not succeed, then check the recent log files under /var/okv/log/hsm and check for vendor-specific instructions.

4.1.6 Primary-Standby Errors

The okv_security.conf file contains settings that can help you diagnose primary-standby errors.

  1. Check that the files have been transported to the standby server.

    Execute the command ls -l as root on the standby server:

    root# ls -l /usr/local/okv/hsm/wallet
    -rw------- 1 oracle oinstall 324 May 16 22:57 cwallet.sso
    -rw------- 1 oracle oinstall 176 May 16 22:57 enctdepwd
    root# ls -l /usr/local/okv/hsm/restore
    -rw------- 1 oracle oinstall 320 May 16 22:57 ewallet.p12
    

    You should see cwallet.sso and enctdepwd in the /usr/local/okv/hsm/wallet directory and ewallet.p12 in the /usr/local/okv/hsm/restore directory.

  2. Check that the mode is set to HSM on the standby server:

    Open the file okv_security.conf as root on the standby server:

    root# cat /usr/local/okv/etc/okv_security.conf
     Look for the line:
     HSM_ENABLED="1"
    

    You should see the number within double quotes.

  3. Check the vendor-specific instructions.

4.1.7 Errors from HSM-Enabled Oracle Key Vault Backups

You can use the cwallet.sso file to diagnose HSM-enabled Oracle Key Vault backup errors.

You should check that the pre_restore command has been run on the target as follows:

Execute the command ls -l as root on the Oracle Key Vault server to which you are restoring the backup:

root# ls -l /usr/local/okv/hsm/wallet
-rw------- 1 oracle oinstall 324 May 16 22:57 cwallet.sso

You should see the wallet file cwallet.sso, which indicates that the credential has successfully been set and stored on Oracle Key Vault.

You should also check that you have followed the instructions from the HSM vendor. In addition, check the most recent log files generated by the recent backup restore, which are in the /var/okv/log/db directory.

4.1.8 Restoring an HSM-Enabled Backup

Before you restore a backup that was taken on an HSM-enabled Oracle Key Vault, ensure that you have set the same HSM credential and token label that were used when the backup was taken.

The HSM credential for Thales Luna is the Thales Luna partition password. For nCipher, the credential is the password that is associated with the Operator Card Set or Softcard. For Utimaco, the credential is the PIN that was initialized when the token was configured.

When using the Set Credential operation, if you enter an incorrect credential or token label, or if Oracle Key Vault is unable to connect to the HSM, then the operation will not succeed and the credential, token label, and vendor provided will not be stored. Ensure that Oracle Key Vault has been enrolled as a client of the HSM and then ensure that the correct credential and token label are entered such that Oracle Key Vault will be able to access the same Root of Trust key that was in use when the backup was taken.

For more information about enrolling Oracle Key Vault as a client of the HSM, see Enrolling Oracle Key Vault as a Client of the HSM.

4.2 Vendor Specific Notes for Thales Luna

Oracle Key Vault supports Oracle Key Vault integration with Thales Luna (formerly Safenet Luna) SA Hardware Security Modules from Thales Luna version 7000, but does not support Host Trust Link (HTL) for Thales Luna HSM.

4.2.1 Installing the HSM Client Software on the Oracle Key Vault Server for Thales Luna

You must use the Thales Luna (formerly Safenet Luna) client version 6.2 for Linux x64 for the installation.

  1. Obtain the Thales Luna client software package, version 6.2 for Linux x64.

  2. Transport the Thales Luna client software package to the Oracle Key Vault machine. Oracle recommends using SCP. For example, assuming the Thales Luna client software packages is called safenet.tar:
    $ scp safenet.tar support@okv_instance_ip_address:/tmp
  3. Install the Thales Luna client software on Oracle Key Vault.

  4. Log in to the Oracle Key Vault Server through SSH as user support, and switch user (su) to root:

    $ ssh support@okv_instance_ip_address
    support$ su root 
    root# cd /usr/local/okv/hsm 
    root# cp /tmp/safenet.tar /usr/local/okv/hsm 
    root# tar -xvf safenet.tar 
    root# cd 64 
    root# ./install.sh
  5. Accept the Thales Luna license by typing y at the prompt.

  6. Install the Luna SA by entering 1, n, i at the successive prompts.

    This installs the Thales Luna software in the directory /usr/safenet/lunaclient .

  7. Delete the safenet.tar file from /tmp directory.

    root# rm -f /tmp/safenet.tar

4.2.2 HSM Credential for Thales Luna

The HSM credential is the Thales Luna partition password.

If you are using Thales Luna as your HSM, then you can use the Thales Luna assignPassword command to assign a password for a partition. However, do not do this when a partition is currently in use by an Oracle Key Vault server, because Oracle Key Vault will no longer be able to access the Root of Trust key as its stored credential will no longer be correct.

4.2.3 Token Label for Thales Luna

The token label for Thales Luna is the name of the partition.

4.2.4 Enrolling Oracle Key Vault as a Client of a Thales Luna HSM

To perform the enrollment, you use the Oracle Key Vault management console and the command-line interface.

  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
  2. Set up the DNS servers.

    When enrolling Oracle Key Vault as a client of a Thales Luna HSM, if the HSM will be registered using a host name, you should first set up DNS using the Oracle Key Vault the management console. To access the DNS settings, select the System tab, and then from the left menu, select System Settings.

    You must configure the DNS servers on each Oracle Key Vault server that you plan to register as a client of the HSM. In a primary-standby environment, configure the DNS servers on both primary and standby server before pairing. For a multi-master cluster, configure DNS on each node in the cluster that will be registered as a client of the HSM.

  3. Exchange certificates between Oracle Key Vault and the Thales Luna SA HSM.

    Log in to the Oracle Key Vault Server through SSH as user support, and switch user (su) to root:

    $ ssh support@okv_instance_ip_address
    support$ su root 
    root# cd /usr/safenet/lunaclient/bin 
    root# scp admin@hsm_hostname:server.pem . 
    root# ./vtl addServer -n hsm_hostname -c server.pem 
    root# ./vtl createCert -n okv_hostname 
    root# scp /usr/safenet/lunaclient/cert/client/okv_hostname.pem admin@hsm_hostname:

    You must enter the HSM administrative password when using SCP with the HSM.

  4. Register Oracle Key Vault as a client of the Thales Luna SA.

    This assumes that you have a partition set up on the Thales Luna SA HSM. You can use any client name that is not yet taken. Oracle recommends using a descriptive name that will identify the Oracle Key Vault instance.

    Access the HSM administrative console by using SSH to admin@hsm_hostname and providing the administrative password:

    $ client register -client client_name -hostname okv_hostname
    $ client hostip map -c client_name -i okv_ip_address
    $ client assignPartition -client client_name -partition partition_name
  5. Verify the enrollment as follows:

    Log in to Oracle Key Vault as the support user using SSH:

    $ ssh support@okv_instance_ip_address
    support$ su root 
    root# cd /usr/safenet/lunaclient/bin 
    root# ./vtl verify

    The following output appears:

    The following Luna SA Slots/Partitions were found:
    
    Slot    Serial #        Label
    ====    ==========      =====
     1      serial_number   partition_name
     

4.2.5 HSM Provider Value for Thales Luna

For Thales Luna, the provider value is 1.

If you are setting this value manually for a primary-standby configuration, then set HSM_PROVIDER="1" in the okv_security.conf file. For more information about enabling HSM in a primary-standby deployment, see Enabling HSM in a High Availability Deployment.

4.2.6 HSM Vendor Specific Checks for Thales Luna

You should check the Thales Luna vendor-specific settings.

You can verify the connection to the HSM for every Oracle Key Vault server as follows:

Log in to the Oracle Key Vault server as user support using SSH:

$ ssh support@okv_instance_ip_address
support$ su root
root# cd /usr/safenet/lunaclient/bin
root# ./vtl verify

The following output appears when the HSM is set up properly:

The following Luna SA Slots/Partitions were found:

Slot    Serial #        Label
====    ========        =====
 1      [serial #]      [partition name]

If you do not see this output, then it means that the HSM is not set up properly. You can diagnose further as follows:

  1. Log into the Thales Luna SA administrative console.

  2. Type the command: client show -client client_name

  3. Verify that the expected client exists and is assigned a partition.

  4. If it does not exist, register the client with the command:

    client register -client client_name-hostname host_name

  5. If no partition is assigned, assign a partition with the command:

    client assignPartition -client client_name -partition partition_name

  6. Verify that all client IP addresses are mapped correctly. If entries are missing, run the command:

    client hostip map -c client_name -i ip_address

  7. Verify that Oracle Key Vault can reach the HSM using the vtl verify command:
    $ su root
    root# cd /usr/safenet/lunaclient/bin
    root# ./vtl verify
    The output should look similar to the following output:
    The following Luna SA Slots/Partitions were found:
    
    Slot    Serial #        Label
    ====    ========        =====
     1      [serial #]      [partition name]

    If the command fails, then it means that the Oracle Key Vault server is unable to contact the HSM. Check the vendor’s other troubleshooting sections for instructions to restore vtl verify functionality. Contact your HSM administrator and confirm that Oracle Key Vault's access to the HSM has not been revoked. If you are unable to resolve the problem, then contact Oracle Support.

4.3 Vendor Specific Notes for nCipher

You can integrate Oracle Key Vault release 12.2 BP 3 and later with the HSM from nCipher nShield Connect 6000+.

4.3.1 Installing the HSM Client Software on the Oracle Key Vault Server for nCipher

The nCipher HSM requires a separate non-HSM computer on the network to use as the remote file system.

After setting up the remote file system, you can proceed with the following steps:

  1. Log in to the Oracle Key Vault server as support user using SSH:
    $ ssh support@okv_instance_ip_address
  2. Switch to root:
    support$ su root
  3. Go to the root directory and create the directories ctls, hwsp, and pkcs11:
    root# cd /root
    root# mkdir ctls
    root# mkdir hwsp
    root# mkdir pkcs11
  4. Transfer the nCipher software installation files using the Secure Copy (SCP) protocol as follows:

    For example:
    root# scp user@remote_file_system_computer:/source_directory/ncipher/nfast/ctls/agg.tar ctls
    root# scp user@remote_file_system_computer:/source_directory/ncipher/nfast/hwsp/agg.tar hwsp
    root# scp user@remote_file_system_computer:/source_directory/ncipher/nfast/pkcs11/user.tar pkcs11
    
  5. Install these files as follows:
    root# cd /
    root# tar xvf /root/ctls/agg.tar
    root# tar xvf /root/hwsp/agg.tar
    root# tar xvf /root/pkcs11/user.tar
    root# /opt/nfast/sbin/install
    
  6. As root, perform additional edits on the Oracle Key Vault server:
    root# usermod -a -G nfast oracle
    root# cd /etc/rc.d/rc5.d
    root# mv S50nc_hardserver S40nc_hardserver
    root# cd /etc/rc.d/rc3.d
    root# mv S50nc_hardserver S41nc_hardserver
  7. Switch to user oracle and verify the installation:
    root# su oracle
    oracle$ PATH=/opt/nfast/bin:$PATH
    oracle$ export PATH      
    oracle$ enquiry
    The state should say operational in the output.
  8. Restart Oracle Key Vault for the group change to take effect.

    In the Oracle Key Vault management console, log in as a user with the System Administrator role. Select the System tab, and then select System Settings. Then click the Reboot button.

4.3.2 HSM Credential for nCipher

The HSM credential for nCipher is the password that is associated with the Operator Card Set or Softcard.

The HSM-credential if they use an Operator Card Set is the Operator Card Set password. If they use a Softcard, then the password is the Softcard password.

4.3.3 Token Label for nCipher

The token label for nCipher is the name of the Operator Card Set or Softcard.

4.3.4 Enrolling Oracle Key Vault as a Client of an nCipher HSM

You use both the nCipher user interface and the command line to enroll Oracle Key Vault as a client of an nCipher HSM.

  1. Add the Oracle Key Vault server IP address to the client list on the HSM using the front panel. Select privileged on any port.

    • In a primary-standby environment, register both the primary server and the standby server to use the nCipher HSM.
    • In a multi-master cluster environment, register each Oracle Key Vault node that will use the nCipher HSM.
  2. Switch to user oracle:
    root# su oracle
    oracle$ PATH=/opt/nfast/bin:$PATH
    oracle$ export PATH
  3. On the Oracle Key Vault server, enroll with the HSM :
    oracle$ nethsmenroll hsm_ip_address hsm_esn hsm_keyhash 
  4. Configure the TCP sockets:
    oracle$ config-serverstartup --enable-tcp --enable-privileged-tcp
  5. Switch to root and restart the hardserver (nCipher client process that communicates with the HSM):
    oracle$ su root
    root# /opt/nfast/sbin/init.d-ncipher restart
  6. On the remote file system computer, run the following command:
    $ rfs-setup --gang-client --write-noauth okv_server_ip_address
  7. On the Oracle Key Vault server as user oracle, run the following commands:
    oracle$ rfs-sync --setup --no-authenticate remote_file_system_ip_address 
    oracle$ rfs-sync --update
  8. Test PKCS#11 access as follows:
    root# /opt/nfast/bin/ckcheckinst
    A prompt appears listing the module. You can confirm or exit.
  9. Create the config file /opt/nfast/cknfastrc as user root. Write the following lines to the file:
    CKNFAST_NO_ACCELERATOR_SLOTS=1
    CKNFAST_OVERRIDE_SECURITY_ASSURANCES=none
  10. Perform the steps described in HSM-Enabling in a Standalone Oracle Key Vault Deployment.

  11. On the Oracle Key Vault server as user oracle run the command:
    oracle$ /opt/nfast/bin/rfs-sync --commit 
    

    If you do not run this command after each HSM initialize operation, then the Root of Trust key may not be available for other operations such as restoring backups and setting up a primary-standby configuration.

4.3.5 HSM Provider Value for nCipher

For nCipher, the provider value is 2.

If you are setting this value manually for the primary-standby, then set HSM_PROVIDER="2". For more information about enabling HSM in a primary-standby deployment, see Enabling HSM in a High Availability Deployment.

4.4 Vendor Specific Notes for Utimaco

Oracle Key Vault supports Oracle Key Vault integration with Utimaco SecurityServer 4.31.1.

4.4.1 Installing the HSM Client Software on the Oracle Key Vault Server for Utimaco

The setup files for Utimaco are provided in the SecurityServerEvaluation-V4.31.1.0.zip file from Utimaco.

  1. Locate the necessary setup files provided in the SecurityServerEvaluation-V4.31.1.0.zip file from Utimaco.

  2. After unzipping the Utimaco zip file, transport the necessary files to the Oracle Key Vault machine. Oracle recommends using SCP. For example:
    $ scp <unzip directory>/Software/Linux/x86-64/Crypto_APIs/PKCS11_R2/sample/cs_pkcs11_R2.cfg support@okv_instance_ip_address:/tmp
    $ scp <unzip directory>/Software/Linux/x86-64/Crypto_APIs/PKCS11_R2/lib/libcs_pkcs11_R2.so support@okv_instance_ip_address:/tmp
    $ scp <unzip directory>/Software/Linux/x86-64/Crypto_APIs/PKCS11_R2/bin/p11tool2 support@okv_instance_ip_address:/tmp
    $ scp <unzip directory>/Software/Linux/x86-64/Crypto_APIs/CXI/bin/cxitool support@okv_instance_ip_address:/tmp
    $ scp <unzip directory>/Software/Linux/x86-64/Administration/csadm support@okv_instance_ip_address:/tmp
  3. Login to the Oracle Key Vault server as user support, and switch user (su) to root:
    $ ssh support@okv_instance_ip_address
    support$ su - root
  4. Create the appropriate directories for the Utimaco files:
    root# mkdir -p /opt/utimaco/lib
    root# mkdir /opt/utimaco/bin
    root# mkdir /etc/utimaco
  5. Move the Utimaco files to the correct directories:
    root# mv /tmp/cs_pkcs11_R2.cfg /etc/utimaco
    root# mv /tmp/p11tool2 /opt/utimaco/bin
    root# mv /tmp/cxitool /opt/utimaco/bin
    root# mv /tmp/csadm /opt/utimaco/bin
    root# mv /tmp/libcs_pkcs11_R2.so /opt/utimaco/lib
  6. Change the configuration file permissions:
    root# /bin/chmod 640 /etc/utimaco/cs_pkcs11_R2.cfg
    root# /bin/chown oracle:oinstall /etc/utimaco/cs_pkcs11_R2.cfg
  7. Change the executable file permissions:
    root# /bin/chmod 550 /opt/utimaco/bin/*
    root# /bin/chown oracle:oinstall /opt/utimaco/bin/*
  8. Change the library file permissions:
    root# /bin/chmod 440 /opt/utimaco/lib/libcs_pkcs11_R2.so
    root# /bin/chown oracle:oinstall /opt/utimaco/lib/libcs_pkcs11_R2.so
  9. Modify the configuration file /etc/utimaco/cs_pkcs11_R2.so so that Device is set to the Utimaco HSM's IP address.
    Device = utimaco_ip_address
    If you are testing with an Utimaco HSM simulator, the line should be in the format:
    Device = 3001@utimaco_ip_address
    It is not recommended that you use the simulator in a production environment.
  10. To verify that you have set up your Utimaco HSM and the client files correctly, you can use p11tool2. The p11tool2 command call can be used to verify that the PKCS11 token has been configured:
    root# /opt/utimaco/bin/p11tool2 GetSlotInfo
    The output should look similar to the following output:
    CK_SLOT_INFO (slot ID: 0x00000000):
    
      slotDescription     33303031 4031302e  3234302e 3131382e |3001@10.240.118.|
                          32333120 2d20534c  4f545f30 30303020 |231 - SLOT_0000 |
                          20202020 20202020  20202020 20202020 |                |
                          20202020 20202020  20202020 20202020 |                |
    
      manufacturerID      5574696d 61636f20  49532047 6d624820 |Utimaco IS GmbH |
                          20202020 20202020  20202020 20202020 |                |
    
      flags: 0x00000005
        CKF_TOKEN_PRESENT    : CK_TRUE
        CKF_REMOVABLE_DEVICE : CK_FALSE
        CKF_HW_SLOT          : CK_TRUE
    
      hardwareVersion        : 5.01
      firmwareVersion        : 2.03
  11. The csadm command call can be used to confirm that the PKCS11 users have been defined:

    root# /opt/utimaco/bin/csadm Dev=utimaco_ip_address ListUsers
    The output should look similar to the following output:
    Name      Permission   Mechanism      Attributes
    ADMIN      22000000    RSA sign       Z[0]
    SO_0000    00000200    HMAC passwd    Z[0]A[CXI_GROUP=SLOT_0000]
    USR_0000   00000002    HMAC passwd    Z[0]A[CXI_GROUP=SLOT_0000]
    

4.4.2 HSM Credential for Utimaco

The HSM credential for Utimaco is the PIN that was initialized when the token was configured.

See the Utimaco documentation for more details.

4.4.3 Token Label for Utimaco

The token label for Utimaco is the name of the token that was set up for the HSM.

4.4.4 HSM Provider Value for Utimaco

For Utimaco, the provider value is 3.

If you are setting this value manually for primary-standby, set HSM_PROVIDER="3" in the okv_security.conf file. For more information about enabling HSM in a primary-standby deployment, see Enabling HSM in a High Availability Deployment.

4.4.5 HSM Vendor Specific Checks for Utimaco

You should check the Utimaco vendor-specific settings.

In addition to the p11tool2 GetSlotInfo and csadm ListUsers commands, you can also check to see that a key was created after completing the HSM Initialize operation. Note that more keys may be created after subsequent HSM initialize commands.
root# /opt/utimaco/bin/p11tool2 LoginUser=<HSM Credential> ListObjects

The output should look similar to the following output:
CKO_DATA:

+ 1.1
  CKA_LABEL                      = OKV 18.1 HSM Key Number

CKO_SECRET_KEY:

+ 2.1
  CKA_KEY_TYPE                   = CKK_AES
  CKA_SENSITIVE                  = CK_TRUE
  CKA_EXTRACTABLE                = CK_TRUE
  CKA_LABEL                      = OKV 18.1 HSM Root Key
  CKA_ID                         = 0x00000001 ()