9 Upgrading Oracle Key Vault Endpoints

This chapter enables you to plan an endpoint upgrade, and then describes how you can upgrade endpoints in different scenarios.

9.1 About Upgrading Oracle Key Vault Endpoints

This topic introduces the process of upgrading Oracle Key Vault Endpoints.

When you upgrade the Oracle Key Vault server software appliance, also upgrade the endpoint software to get access to the latest enhancements.

Oracle Key Vault client software is backward-compatible. While older versions of Oracle Key Vault client software are fully functional with an upgraded Oracle Key Vault server, some new Oracle Key Vault features are only available with the current client software. For example, to use non-extractable TDE master key, the endpoint software must be upgraded to a release that supports the feature.

9.2 Planning for Endpoints Upgrade

Planning endpoint upgrades helps you minimize downtime and facilitates a smooth upgrade process.

Careful planning helps you minimize downtime and facilitates a smooth upgrade process for your Oracle Key Vault endpoints. Key considerations for planning include:
  • Choose Upgrade Approach: For minor version upgrades, updating the endpoint software is typically sufficient. Major version upgrades may require re-enrollment.
  • Schedule and Coordinate: Plan upgrade activities during non-peak usage periods to minimize business impact, especially when database downtime is necessary to update libraries used by TDE-enabled databases.
  • Leverage Near Zero Downtime Options: For Oracle Database 23ai and later, use the near-zero downtime upgrade scheme to further reduce or eliminate downtime.

9.3 Endpoint Upgrade Considerations

The following topics describe how you can upgrade an endpoint by upgrading the endpoint software or by re-enrolling the endpoint. Additionally, information about the options that must be set to install endpoint software for use with Oracle Database 23ai or later, is included.

9.3.1 Upgrading Endpoint Software or Re-enrolling an Endpoint

You can upgrade an endpoint by upgrading the endpoint software or re-enrolling the endpoint.

Upgrading the endpoint software does not affect the existing endpoint certificate or okvclient.ora, the endpoint configuration file. Re-enrolling an endpoint invalidates an existing endpoint certificate, and a new endpoint certificate as well as okvclient.ora are installed. Oracle recommends that you upgrade the endpoint software for minor version upgrades (for example, from 21.x to 21.y) and consider re-enrolling the endpoint when upgrading across major versions (for example, from 18.x to 21.y).

Before you re-enroll an endpoint for TDE-enabled database, you must shut down the database.

When you update the software for the TDE-enable database endpoint, you may either eliminate or minimize database downtime.

9.3.2 Endpoint Software for Oracle Database 23ai or Later

Oracle Key Vault ships with newer endpoint software to use with Oracle Database 23ai or later.

The new endpoint software for Oracle Database 23ai or later is required to support the database running in FIPS mode using OpenSSL libraries and the improved version of local auto login wallets. The new endpoint software for Oracle Database 23ai or later includes additional libraries and is supported on the Linux-x64 platform only.

To install the endpoint software for use with Oracle Database 23ai or later, specify the following options during installation:

  • Use the -arch db23ai option when installing with the okvclient.jar file.
  • Use the --arch db23ai option when installing with the okv admin endpoint provision command.

9.4 Understanding Endpoint Upgrade for TDE-enabled Databases

This topic introduces the upgrade process for TED-enabled endpoints.

When you upgrade a TDE-enabled database endpoint, complete the following steps:

  • Updated the endpoint software or re-enroll the endpoint.
  • Install the PKCS#11 library by running the root.sh or root.bat scripts.

Oracle Database does not use the PKCS#11 library from the $OKV_HOME/lib directory. You can install the PKCS#11 library to a standard location, or to an Oracle Key Vault release-specific location (for Oracle Database 23ai or later).

9.4.1 Using the PKCS#11 Library from a Standard Location

This topic describes how to use the PKCS#11 library from a standard location.

By default, Oracle Database uses the PKCS#11 library from a standard location:
/opt/oracle/extapi/64/hsm/oracle/1.0.0/liborapkcs.so

To update the PKCS#11 library at the standard location, run the root.sh (or root.bat) script without any arguments.

This script also copies additional libraries included in the endpoint software for Oracle Database 23ai to the Oracle Key Vault version-specific location:
/opt/oracle/extapi/64/pkcs11/okv/<okv_version>/lib
The following limitations apply for databases using the PKCS#11 library from the standard location:
  • All such databases on the same host must use the same version of the endpoint software.
  • All such databases on the host must be shut down before installing the PKCS#11 library at the standard location.
  • Upgrade the endpoint software for all such databases on a host before running the root.sh or root.bat scripts. While you may upgrade the endpoints one by one, do not run the root.sh or root.bat scripts until all endpoints are upgraded. Database shutdown is not required during endpoint software upgrades. A shutdown is only required before running the root.sh or root.bat scripts, which must be run only once per host.
  • Use this approach for databases that do not support the database parameter PKCS11_LIBRARY_LOCATION.

9.4.2 Using the PKCS#11 Library from an Oracle Key Vault Release-specific Location

This topic describes steps to use the PKCS#11 library from an Oracle Key Vault Release-specific location.

For Oracle Database 23ai and later, Oracle recommends that you install the liborapkcs.so library in an Oracle Key Vault release-specific location.

To install the library in the recommended location, run the root.sh script with the --okv_pkcs11_library_location option:
root.sh --okv_pkcs11_library_location
This installs the liborapkcs.so library and other libraries at the Oracle Key Vault release-specific location:
/opt/oracle/extapi/64/pkcs11/okv/<okv-version>/lib
For example, the Oracle Key Vault release-specific location for release 21.12 is:
/opt/oracle/extapi/64/pkcs11/okv/21.12.0.0.0/lib

To use the PKCS#11 library from the Oracle Key Vault release-specific path, set the PKCS11_LIBRARY_LOCATION database parameter to the new library location. Perform this step, which requires the database to be restarted, only once when switching from the standard location to the Oracle Key Vault release-specific path for the first time.

For subsequent endpoint upgrades, switching to the newer PKCS#11 library from its release-specific path does not require a database shutdown. Run the ADMINISTER KEY MANAGEMENT SWITCHOVER command to begin using the updated library.

The following benefits apply when you use the PKCS#11 library from an Oracle Key Vault release-specific location:
  • You can implement near-zero downtime upgrades for endpoints.
  • When upgrading multiple endpoints on the same host, you can upgrade each endpoint independently without affecting other endpoints. For each release, install the PKCS#11 library in its release-specific path only once.
  • TDE-enabled databases on the same host can use different versions of Oracle Key Vault endpoint software.

9.5 Near-Zero Downtime Endpoint Upgrade

For endpoints used with Oracle Database 23ai and later, you can upgrade them with near-zero downtime of the database.

The endpoint upgrade is termed near-zero downtime because you need to shut down the database once during initial setup. For subsequent endpoint upgrades, you do not need to shut down databases, effectively making them zero downtime endpoint upgrades.

To implement near-zero downtime endpoint upgrade, you must:
  1. Install the PKCS#11 library in an Oracle Key Vault release-specific directory.
  2. Configure the database to use the PKCS#11 library from the release-specific location.

9.5.1 Installing the PKCS#11 Library in an Oracle Key Vault Release-specific Location

This topic describes how you can install the liborapkcs.so library to implement a near-zero downtime endpoint upgrade.

To take advantage of a near-zero downtime endpoint upgrade, install the liborapkcs.so library in the Oracle Key Vault release-specific location.

To do so, run:
root.sh --okv_pkcs11_library_location
This installs the liborapkcs.so library and other libraries at the OKV release-specific location:
/opt/oracle/extapi/64/pkcs11/okv/<okv-version>/lib

When you install the latest liborapkcs.so library in the Oracle Key Vault release-specific location, it does not overwrite the liborapkcs.so library from a previous Oracle Key Vault version. Databases that are currently using the previous version of the library remain unaffected by the endpoint software upgrade.

9.5.2 Configuring the Database to use the PKCS#11 Library from a Custom Location

This topic describes how to enable near-zero downtime endpoint upgrade for Oracle Database 23ai and later.

To enable near-zero downtime endpoint upgrade for Oracle Database 23ai and later, you must configure the database to use the PKCS#11 library from the specified location by setting the PKCS11_LIBRARY_LOCATION database parameter.
  1. Log in to the CDB Root as a user with the ALTER SYSTEM privilege.
  2. Set the static initialization parameter PKCS11_LIBRARY_LOCATION to point to the liborapkcs.so library from the Oracle Key Vault release-specific library location. For example:
    ALTER SYSTEM SET
    PKCS11_LIBRARY_LOCATION=’/opt/oracle/extapi/64/pkcs11/okv/21.12.0.0.0/lib/liborapkcs.so’ SCOPE=SPFILE SID=’*’;
  3. Restart the database for the parameter to take effect.

    After the restart, the database starts using the PKCS#11 library from the specified Oracle Key Vault release-specific path.

    The database is now set to switch to a newer PKCS#11 library without requiring a shutdown.

9.5.3 Library Switchover for Subsequent Endpoint Upgrades

This topic describes how you can complete subsequent endpoint upgrades without any database downtime.

You can complete subsequent endpoint upgrades without any database downtime if the database is already configured to use the PKCS#11 library from the specified location by using the PKCS11_LIBRARY_LOCATION parameter.
  1. Install the new endpoint software.
  2. Install the new PKCS#11 library in the Oracle Key Vault release-specific location using the root.sh file of the upgraded endpoint:
    root.sh -–okv_pkcs11_library_location

    Install the new PKCS#11 library only once for each version of Oracle Key Vault.

  3. Initiate the change in the database to switch to the new library:
    ADMINISTER KEY MANAGEMENT SWITCHOVER TO LIBRARY 
    ‘/opt/oracle/extapi/64/pkcs11/okv/<new-okv-version>/lib/liborapkcs.so’ 
    FOR ALL CONTAINERS;
The SQL command ADMINISTER KEY MANAGEMENT SWITCHOVER TO LIBRARY instructs the database to switch over from the current PKCS#11 library to a new PKCS#11 library without undergoing system downtime. After running this command, database foreground and background processes are gradually switched over to the new library.

You can use the database view V$PKCS11_PATH to monitor the PKCS#11 library that is used by each database process.

9.6 Upgrade the Endpoint

You can upgrade an endpoint by updating the endpoint software or by re-enrolling the endpoint.

The following steps describe how to upgrade an endpoint.
  1. For TDE-enabled databases that use the PKCS#11 library from the standard location, all endpoints on the same host must be upgraded together.

    Review the instructions in step 6 before proceeding with the upgrade of these endpoints.

    You can upgrade an endpoint by updating the endpoint software or by re-enrolling the endpoint. Perform steps 2 - 4 to update the endpoint software.

    Alternatively, perform step 5 to re-enroll the endpoint.

  2. Download the endpoint software (okvclient.jar) and install it in your existing endpoint directory path as follows:
    1. Go to the Oracle Key Vault management console login screen.
    2. Click the Endpoint Enrollment and Software Download link.
    3. In the Download Endpoint Software Only section, select the appropriate platform from the drop-down list.
    4. Click the Download button to download the okvclient.jar file.
  3. Identify the path to your existing endpoint installation that you are about to upgrade. For example, /etc/ORACLE/KEYSTORES/okv (where /etc/ORACLE/KEYSTORES is WALLET_ROOT of your database, or the soft link in $ORACLE_BASE/okv/$ORACLE_SID points to).
  4. For endpoints used with Oracle Database 21c and earlier, or other non-TDE endpoints, install the endpoint software by running the following command:
    java -jar okvclient.jar -d existing_endpoint_directory_path
    For example:
    java -jar okvclient.jar -d /etc/ORACLE/KEYSTORES/okv
    For endpoints used with Oracle Database 23ai and later, include the -arch db23ai option during the installation:
    java -jar okvclient.jar -d existing_endpoint_directory_path -arch db23ai
    For example:
    java -jar okvclient.jar -d /etc/ORACLE/KEYSTORES/okv -arch db23ai
    The new endpoint software for Oracle Database 23ai includes additional libraries, and is supported on the Linux-x64 platform only.
  5. Perform the following steps to re-enroll the endpoint software, which also generates a new endpoint certificate. The easiest way to re-enroll an endpoint is by using the following commands of the RESTful services utility:
    1. Re-enroll the endpoint by using the following RESTful services utility command:
      okv admin endpoint re-enroll
    2. Back up the OKV_HOME directory and delete the files under OKV_HOME:
      cp -R ${OKV_HOME}_bkp_`date +%Y%m%d`
    3. Go to the $OKV_HOME directory and remove all the files.
    4. For endpoints used with Oracle Database 21c and earlier, or other non-TDE endpoints:
      Download and install the endpoint software by using the following RESTful services utility command:
      okv admin endpoint provision

      For endpoints used with Oracle Database 23ai and later:

      Download and install the endpoint software by using the following RESTful services utility command with the --arch db23ai option:
      okv admin endpoint provision --arch db23ai

    Re-enrolling an endpoint generates a new okvclient.jar file and installs the file in the OKV_HOME directory but maintains the relationship between the endpoint and its default wallet.

    Note:

    To re-enroll an endpoint without using the RESTful services utility, follow the steps described in How to Re-enroll an Endpoint.
  6. Install the updated PKCS#11 library file.

    This step is needed only for online TDE master encryption key management by Oracle Key Vault. If an endpoint uses online TDE master encryption key management by Oracle Key Vault, then you must upgrade the PKCS#11 library while upgrading the endpoint software.

    You can install the PKCS#11 library either at the standard location or Oracle Key Vault release-specific location depending upon the Oracle Database version.

    • For Oracle Database 21c and Earlier:
      Install the PKCS#11 library at the standard location:
      /opt/oracle/extapi/64/hsm/oracle/1.0.0/liborapkcs.so
      Before installing the PKCS#11 library at the standard location, you must:
      1. Complete the endpoint software upgrade for all databases on that host that uses the PKCS#11 library from the standard location.
      2. Shut down all such databases on that host.
        • Database shutdown is required only before running the root.sh or root.bat scripts, which must be run only once per host to install the PKCS#11 library at the standard location.
        • You do not need to shut down the databases during the upgrade of the endpoint software.
      On UNIX/Linux platforms: Run root.sh from the bin directory of the endpoint installation directory to copy the latest liborapkcs.so file for Oracle Database endpoints:
      sudo /etc/ORACLE/KEYSTORES/okv/bin/root.sh

      Or

      su - root
      /etc/ORACLE/KEYSTORES/okv/bin/root.sh
      On Windows platforms: Run root.bat from the bin directory of the endpoint installation directory to copy the latest liborapkcs.dll file for Oracle Database endpoints. You will be prompted for the version of the database in use.
      bin\root.bat
    • For Oracle Database 23ai and Later:
      For Oracle Database 23ai and later, Oracle recommends that you install the latest PKCS#11 library liborapkcs.so in an Oracle Key Vault release-specific location:
      /opt/oracle/extapi/64/pkcs11/okv/<okv_version>/lib
      For example, for the Oracle Key Vault 21.12 release, the location is as follows:
      /opt/oracle/extapi/64/pkcs11/okv/21.12.0.0.0/lib

      If you are upgrading multiple endpoints for TDE-enabled databases on a host, you must install the latest Oracle Key Vault PKCS#11 library only once for each endpoint software version.

      On UNIX/Linux platforms: Run root.sh from the bin directory of the endpoint installation directory to copy the latest liborapkcs.so file for Oracle Database endpoints. Specify the --okv_pkcs11_library_location option to install the library at the Oracle Key Vault release-specific location:
      sudo /etc/ORACLE/KEYSTORES/okv/bin/root.sh –-okv_pkcs11_library_location
      su - root
      /etc/ORACLE/KEYSTORES/okv/bin/root.sh --okv_pkcs11_library_location
      

      Before installing the latest liborapkcs.so library in the Oracle Key Vault release-specific location, you do not need to shut down databases on that host.

      When there are multiple TDE-enabled databases (23ai and later) on a host, you can upgrade their endpoints independently at different times, as long as you configure them to use the PKCS#11 library from the release-specific location.

      Note:

      Installing the liborapkcs.so library in the standard location is also supported with Oracle Database 23ai. However, Oracle does not recommend it.
  7. For Oracle Database 23ai and later, configure the database to use the new PKCS#11 library from the Oracle Key Vault release-specific location.

    Configuration steps differ depending upon whether the database is being configured to use the PKCS#11 library from the release-specific location for the first time. A database start is required only during this initial one-time configuration. For subsequent endpoint upgrades, the database can switch to the newer PKCS#11 library without requiring a shutdown.

    First-time PKCS#11 Library Switchover from the Standard Location to the Release-specific Location

    For databases that are currently using the PKCS#11 library from the standard location, set the PKCS11_LIBRARY_LOCATION database parameter to switch them over to use the library from the release-specific location.
    1. Log in to the CDB Root as a user with the ALTER SYSTEM privilege.
    2. Set the static initialization parameter PKCS11_LIBRARY_LOCATION to point to the liborapkcs.so library from the Oracle Key Vault release-specific library location.
      For example:
      ALTER SYSTEM SET PKCS11_LIBRARY_LOCATION=’/opt/oracle/extapi/64/pkcs11/okv/21.12.0.0.0/lib/liborapkcs.so’ SCOPE=SPFILE SID=’*’;
    3. Restart the database for the parameter to take effect.

      After the restart, the database starts using the PKCS#11 library from the specified Oracle Key Vault release-specific location.

    PKCS#11 Library Switchover from one Release-specific Version to Another

    For databases that are already using the PKCS#11 library from the Oracle Key Vault release-specific location, a database restart is not needed to switch them to use the new library. You can complete endpoint upgrades for such databases with zero downtime.

    To initiate the change to switch the databases to use the new library, run:
    ADMINISTER KEY MANAGEMENT SWITCHOVER TO LIBRARY ‘/opt/oracle/extapi/64/pkcs11/okv/<new-okv-version>/lib/liborapkcs.so’ FOR ALL CONTAINERS

    The SQL command ADMINISTER KEY MANAGEMENT SWITCHOVER TO LIBRARY instructs the database to switch from the current PKCS#11 library to a new PKCS#11 library, without incurring system downtime. After you issue this command, database foreground and background processes are gradually switched over to the new library.

    You can use the database view V$PKCS11_PATH to monitor the PKCS#11 library that is used by each database process.

    Note:

    After upgrading all endpoints on a host and confirming all TDE-enabled databases have switched to the new PKCS#11 library, remove the older version of the PKCS#11 library from the host. Ensure that library switchover, which happens in the background, is complete for all the TDE-enabled databases before you delete the old library. You can use the database view V$PKCS11_PATH to monitor the PKCS#11 library that is used by each database process.
  8. Update the SDK software.
    If you have already deployed the SDK software, Oracle recommends that you redeploy the software in the same location after you complete the upgrade to Oracle Key Vault release 21.12. This enables you to have access to new SDK APIs that were introduced since the Oracle Key Vault version that you are upgrading from.
    1. Go to the Oracle Key Vault management console login screen.
    2. Click the Endpoint Enrollment and Software Download link.
    3. In the Download Software Development Kit section, select the appropriate language and platform for your site.
    4. Click the Download button to get the SDK zip file.
    5. Identify the existing location where SDK software was already deployed.
    6. Navigate to the directory in which you saved the SDK zip file.
    7. Unzip the SDK zip file.
      For example, on Linux, to unzip the Java SDK zip file, use the following command:
      unzip -o okv_jsdk.zip -d existing_endpoint_sdk_directory_path

      For the C SDK zip file, use the following command:

      unzip -o okv_csdk.zip -d existing_endpoint_sdk_directory_path
    8. Do not exit this page.
  9. If you had deployed the RESTful services utility in the previous release, then re-deploy the latest okvrestclipackage.zip file.

    The latest okvrestclipackage.zip file enables you to have access to the new RESTful services utility commands that were introduced since the Oracle Key Vault version that you are upgrading from.

    You can use wget or curl to download the okvrestclipackage.zip file.
    wget --no-check-
    certificate https://Oracle_Key_Vault_IP_address:5695/ okvrestclipackage.zip 
    curl -O -
    k https://Oracle_Key_Vault_IP_address:5695/okvrestclipackage.zip
  10. For the TDE-enabled databases that are using the PKCS#11 library from the standard location, start the Oracle Databases if the upgrade of Oracle Key Vault endpoints for all of the TDE enabled databases on this host machine is complete.

    For databases using the PKCS#11 library from the Oracle Key Vault release-specific location, each endpoint can be upgraded individually, with any required database restart performed as part of that upgrade process in step 7 above. No additional database restart is needed for such databases.

    At this stage, the endpoint is fully upgraded.

  11. If your site requires that you restrict TDE master encryption keys from leaving Oracle Key Vault and if you are using an Oracle Real Application Clusters (Oracle RAC) environment, then perform the following steps on each Oracle RAC node:
    1. Perform the endpoint upgrade on each Oracle RAC node.
    2. Set the extractable attribute value for symmetric keys.
      By default, the extractable attribute value is true, which means that the key material of symmetric keys can be extracted from Oracle Key Vault during certain operations. If you want to prevent symmetric keys from being extracted, then set this value to false. You can set an extractable attribute value as follows:
      • Set the default value for the extractable attribute of new symmetric keys in the endpoint settings. Endpoint-specific settings override global endpoint settings.
      • Explicitly specify the value of the extractable attribute when creating or registering a new symmetric key.
      • Modify the extractable attribute of an existing symmetric key.

      See the Oracle Key Vault Administrator's Guide.

    3. As a user who has the SYSDBA or SYSKM administrative privilege, perform a rekey operation in the Oracle RAC node. Use the following syntax:
      ADMINISTER KEY MANAGEMENT SET [ENCRYPTION] KEY 
      [FORCE KEYSTORE][USING TAG 'tag_name'] 
      IDENTIFIED BY [EXTERNAL STORE | keystore_password]
      [WITH BACKUP [USING backup_identifier']];
      

      See the Oracle Database Advanced Security Guide for more information about rekeying a TDE master encryption key.

  12. If your site requires that you restrict TDE master encryption keys from leaving Oracle Key Vault and if you are using an Oracle Data Guard environment, then do the following on the primary and standby databases:
    1. Perform the endpoint upgrade on the primary and standby databases.
    2. Set the extractable attribute value for symmetric keys.
      By default, the extractable attribute value is true, which means that the key material of symmetric keys can be extracted from Oracle Key Vault during certain operations. If you want to prevent symmetric keys from being extracted, then you must set this value to false. You can set an extractable attribute value as follows:
      • Set the default value for the extractable attribute of new symmetric keys in the endpoint settings. Endpoint-specific setting overrides the global endpoint settings.
      • Explicitly specify the value of the extractable attribute when creating or registering a new symmetric key.
      • Modify the extractable attribute of an existing symmetric key.
    3. As a user who has the SYSDBA or SYSKM administrative privilege, perform a rekey operation in the primary and standby databases. Use the following syntax:
      ADMINISTER KEY MANAGEMENT SET [ENCRYPTION] KEY 
      [FORCE KEYSTORE]
      [USING TAG 'tag_name'] 
      IDENTIFIED BY [EXTERNAL STORE | keystore_password]
      [WITH BACKUP [USING 'backup_identifier'']];

      See Oracle Database Advanced Security Guide for more information about rekeying a TDE master encryption key.