17 Managing Service Certificates

This chapter explains about Oracle Key Vault-generated certificates, you can learn how to manage self-signed and third-party certificates.

17.1 Overview of Oracle Key Vault Certificates

Oracle Key Vault uses certificates for various purposes including endpoint authentication, server authentication, and securing the communication channels using the TLS protocol.

The TLS protocol protects communication between the Oracle Key Vault server or node, and the endpoints. The TLS protocol also protects the back channel communication between the Oracle Key Vault nodes in the cluster deployment or Oracle Key Vault servers in the primary-standby deployment. The TLS certificates used by endpoints and the Oracle Key Vault servers or cluster-nodes are issued by the Oracle Key Vault itself using its CA certificate. The Oracle Key Vault’s CA certificate may be a self-signed Root CA or an intermediate CA.

Oracle Key Vault generates the TLS certificates with the exception of the intermediate CA certificate.

CA Certificate

The CA certificate is a self-signed Root CA or an intermediate CA certificate that the Oracle Key Vault uses to issue endpoint certificates, as well as server or node certificates. The self-signed Root CA certificate is generated at the time of Oracle Key Vault installation. Customers can choose to replace it with an intermediate CA certificate that is signed by the organization's own internal CA or a third-party CA post-installation or post-upgrade. The CA certificate is the same for all nodes in a multi-master cluster deployment and for both the primary and standby servers of a primary-standby deployment. The CA certificate is different from the console certificates.

If you do not rotate the CA certificate before it expires, none of the endpoints can communicate with the Oracle Key Vault server or any node of the Oracle Key Vault cluster and all the endpoints will face a downtime. In the cluster deployment none of the Oracle Key Vault nodes will be able to communicate with each other and in case of primary-standby, the communication between primary and standby servers will breakdown.

Note:

The CA Certificate must be rotated before it expires to prevent outage to endpoints. Start the CA certificate rotation several weeks in advance of CA certificate expiry to prevent outage to the Oracle Key Vault deployment and endpoints.

Rotating CA certificates also rotates the server or node certificates and endpoint certificates.

Server and Node Certificate

Server or Node Certificate is the TLS certificate of the Oracle Key Vault server or a cluster node. While in a standalone or primary-standby deployment, Oracle Key Vault uses server certificates to communicate with its endpoints. In a multi-master deployment of Oracle Key Vault, each cluster node has its own node certificate. Oracle Key Vault cluster nodes use node certificates to communicate with each other and with the endpoints.

These certificates are referred to as server certificates for standalone and primary-standby systems and as node certificates in multi-master cluster configurations. The Oracle Key Vault CA certificate is used to issue these certificates.

Rotate the server or node certificate before they expire as described in section Managing Server Certificates and Node Certificates Rotation. The CA and endpoint certificates are not rotated when server or node certificates are rotated.

Note:

Rotating the node certificate in a multi-master cluster deployment is a per-node operation.

If the server certificate is not rotated in a standalone deployment before it expires, none of the endpoints can communicate to the Oracle Key Vault server and all the endpoints will face a downtime. If you do not rotate the server certificate in a primary-standby deployment before it expires, then none of the endpoints can communicate to the primary server and all the endpoints face a downtime.

If you do not rotate the node certificate in cluster deployment before it expires, the endpoints use the other nodes for the endpoint operations like fetching a key. However, the inter-node communication will be impacted and operations like creating a new endpoint or creating a new wallet will be impacted.

Note:

If all of the node certificates in the cluster deployment have expired, endpoints cannot communicate with any node in the multi-master cluster.

Endpoint Certificate

Each endpoint is issued a unique endpoint TLS certificate that is used to authenticate the endpoint with the Oracle Key Vault. The Oracle Key Vault Certificate Authority (CA) certificate is used to issue the endpoint certificates. To rotate an endpoint certificate, you can re-enroll the endpoint. The CA or server or node certificates are not rotated when endpoint certificates are rotated.

If the endpoint certificate is not rotated before expiry, the endpoint using the endpoint certificate faces a downtime.

To rotate an endpoint certificate, re-enroll the endpoint.

17.2 Certificates Validity Period

You can set the validity periods for Oracle Key Vault certificates to meet the security, compliance, and operational requirements.

17.2.1 About Certificates Validity Period

Compliance and best security practices have different requirements for certificate validity depending upon the purpose and use of the certificate.

For simplicity, all the three certificates, including the self-signed Root CA certificate, are rotated together up until Oracle Key Vault release version 21.3. However, the server or node, endpoint, and CA certificates can have different validity periods. Generally, the validity period requirements for the endpoint and server or node certificates are different than that of the CA certificate. You can configure the validity periods of the self-signed Root CA, the server or node certificates and the endpoint certificates independently with different values. You can rotate the server or node certificates independent of the CA certificate rotation.

The default and the range of the validity periods of the TLS certificates in Oracle Key Vault are described below.

Table 17-1

Certificate Default Validity (out of the box) Minimum Validity Maximum Validity

Self-Signed Root CA

1095 days or 3 years

365 days or 1 year

3650 days or 10 years

Intermediate CA Defined by signing CA Defined by signing CA Defined by signing CA

Server/Node Certificate

365 days or 1 year

365 days or 1 year

1095 days or 3 years

Endpoint Certificate

365 days or 1 year

365 days or 1 year

1095 days or 3 years

The certificate validity period automatically determines the certificate expiry. Rotate the certificates before they expire.

You can set different validity periods for each type of certificate to meet your requirements.

Setting the validity period of certificates does not affect the validity period of existing certificates. The configured validity periods take effect when a new certificate is generated either during the certificate rotation or when you set up a new endpoint or cluster node.

The CA signing authority sets the validity period of the intermediate CA certificate.

Note:

For simplicity, till Oracle Key Vault release 21.3, all three types of certificates - self-signed Root CA certificate, server or node certificate, and the endpoint certificate have the same certificate validity period. The server or node certificates cannot be rotated independently of the CA certificate rotation.

17.2.2 Setting Validity Period of Self-Signed Root CA Certificate

You can configure the validity period for the self-signed Root certificate authority (CA) certificate from the Oracle Key Vault management console.

The CA certificate validity period governs the end date of the CA certificate. The end date of the CA certificate acts as an upper bound on the validity period of the server or node, and the endpoint certificates, when they are issued.
Setting the validity of self-signed Root CA certificate does not enable it, that is, switch it into use. You have to rotate the CA certificate to generate and enable a new self-signed Root CA with the set validity period as described in Rotating CA Certificate.

To set the validity period for the self-signed Root CA:
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
    In a primary-standby environment, log in to the primary Oracle Key Vault server. In a multi-master cluster environment, log in to the node selected for CA certificate rotation in the cluster.
  2. Select the System tab, then Settings from the left navigation side bar.
  3. In the Certificates area, click Service Certificates.
  4. In the Service Certificates page, select Manage CA Certificate.
  5. In the CA Certificate Details page, select the Self-Signed Root CA option. The Self-Signed Root CA option is selected by default.
  6. Set the validity value in the Self-Signed Root CA Certificate Validity (in days) field. The default is 1095 days (3 years). You can set a maximum validity period of 3650 days (10 years) and a minimum validity period of 365 days (1 year).
  7. Click Save.

17.2.3 Configuring Certificate Validity Period for Server and Node Certificates

You can configure the validity period for server or node certificates in the Oracle Key Vault management console.

The certificate validity period takes effect the next time you rotate the server or node certificates. It will also be taken into account when you generate the server or node certificates as part of a CA certificate rotation, or when you add a new node to the cluster, to the node certificates for that new node. Irrespective of the value that the server or node certificate validity is set to, when the certificates are eventually generated, Oracle Key Vault ensures that their expiry date is less than that of the CA certificate.

  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
    In a multi-master cluster environment, you can log in to any node in the cluster.
  2. Select the System tab, then Settings from the left navigation side bar.
  3. In the Certificates area, select Service Certificates.
  4. Depending on your environment, perform the following:
    • In a standalone or primary-standby environment: In the Current Server Certificate area, select Manage Server Certificate.
    • In a multi-master cluster environment: In the Current Node Certificate area, select Manage Node Certificate.
  5. In the Server Certificate Validity (in days) or Node Certificate Validity (in days) field, enter a value between 365 days (the minimum and the default) and 1095 days for this setting.
  6. Click Save.

17.2.4 About Configuring Certificate Validity Period for Endpoint Certificates

You can set the validity period for the endpoint certificates in the Global Endpoint Configuration parameters.

The default value is 365 days (1 year). You can set a maximum validity period of 1095 days (3 years) and a minimum validity period of 365 days (1 year).

The certificate validity period takes effect the next time you re-enroll the endpoint, or when a new endpoint is added. Irrespective of the value endpoint certificate validity is set to, when the endpoint is eventually rotated, Oracle Key Vault ensures that the endpoint certificate expiry date is less than that of the CA certificate.

17.3 Monitoring Certificates Expiry

Proactively set alerts and monitor the expiry dates of the Oracle Key Vault certificates and rotate them before they expire.

17.3.1 Monitoring Certificates Expiry Using Certificate Expiration Alerts

Set expiration alerts as reminders to rotate the certificates before their expiration date.

Expiration of a certificate, especially the CA certificate, breaks the endpoint and Oracle Key Vault communication, and impacts the operations of one or more endpoints to the extent of stopping of endpoint operations completely. In addition, upgrades and communication between the Oracle Key Vault multi-master cluster nodes may also fail. Ensure that you rotate certificates much before their expiration date.

To avoid this scenario, Oracle recommends that you configure an alert as a reminder to rotate the certificate before the certificate validity period expires. There is one alert for endpoint certificates and another alert for the CA and server or node certificates.

You can find the certificate expiration date of the CA certificate, server or node certificates, and endpoint certificates using the Oracle Key Vault management console.

Note:

You must promptly address the certification expiration alerts by rotating the certificates indicated by the alert. Depending upon your deployment, the rotation of the Oracle Key Vault CA certificate, in particular, may take a very long time (in the order of several days). Begin the CA certificate rotation process well before CA certificate expiry to avoid outages.

17.3.2 Server Certificate Expiration Date on Status Page

You can check the Server Certificate Expiration Date, which is the expiration date of CA or node or server certificate that is expiring first, from the Status page.

The Server Certificate Expiration Date field in the System Status page reflects the expiration date of the CA certificate, or the server/node certificate, whichever is expiring first. The Server Certificate Expiring In on the System Status page shows how many days are left to expire for any one of the CA or the server or node certificate that is expiring first.

To navigate to the System Status page, select the System tab.

  1. Log in to the Oracle Key Vault management console as a System Administrator.

    In a multi-master cluster environment, you can log in to any node in the cluster.

  2. Select the System tab and then Status from the left navigation side bar.
  3. Check the Server Certificate Expiration Date field.
  4. Check the Server Certificate Expiring in field.

Description of 214_ca_certificate_details_status_page.png follows
Description of the illustration 214_ca_certificate_details_status_page.png

Oracle Key Vault raises an alert for the Server Certificate Expiration Date when either the CA, or node, or server certificate falls within the alert threshold period. You can also monitor the Server Certificate Expiration Date over SNMP.

17.3.3 Finding the Expiration Date of the CA Certificate

You can find how much time the Oracle Key Vault CA certificate has before it expires by navigating to the Service Certificates page.

  1. Log in to the Oracle Key Vault management console as the System Administrator.
    In a multi-master cluster environment, you can log in to any node in the cluster.
  2. Select the System tab, then Settings from the left navigation side bar.
  3. In the Certificates area, select Service Certificates.
  4. In the Current CA Certificate area, check the End Date setting to know when the CA certificate is expiring.
    The Expiring In setting also shows the number of days left for the CA certificate to expire.


    Description of 214_ca_certificate_and_node_certificate.png follows
    Description of the illustration 214_ca_certificate_and_node_certificate.png

17.3.4 Finding the Expiration Date of Server Certificates and Node Certificates

You can find the expiration date of server certificates and node certificates in the Oracle Key Vault management console.

Perform the following steps to review the end dates and time to expire of all the node certificates in the cluster:
  1. Log in to Oracle Key Vault management console as a user who has the System Administrator role.
    In a multi-master cluster environment, you can log in to any node in the cluster.
  2. Select the System tab, then Settings from the left navigation side bar.
  3. In the Certificates area, select Service Certificates.
  4. In a standalone or primary-standby environment:
    • Under Current Server Certificate, check the End Date setting to determine when the server certificate is expiring. The Expiring In setting also shows the number of days left for the server certificate expiry.
  5. In a multi-master cluster environment:
    • In the Current Node Certificate area, select Manage Node Certificate. Under Current Node Certificate, check the End Date setting. The Expiring In setting also shows the number of days left for the node certificate expiry.
    • You can view the end dates and time to expire of all the node certificates in the cluster in the Cluster Node Certificate Details area.

If a server or node certificate is expiring soon, then Oracle recommends that you rotate the certificate at the earliest.

17.3.5 Finding the Expiration Date of Endpoint Certificates

You can find the expiration date of endpoint certificates in the Oracle Key Vault management console.

To find the expiration date of the endpoint certificates, navigate to the Endpoints page and check the Endpoint Certification Expiration field.
  1. Log in to the Oracle Key Vault management console.
    In a multi-master cluster environment, you can log in to any node in the cluster.
  2. Select the Endpoints tab.
  3. In the Endpoints table, check Endpoint Certification Expiration.

17.4 Managing CA Certificate Rotation

You can use the Oracle Key Vault management console to rotate the CA certificate. The new CA certificate can be a self-signed Root CA certificate or an intermediate CA certificate.

17.4.1 Steps for Managing CA Certificate Rotation

A user with the System Administrator role can perform CA certificate rotation. The user can set up a new self-signed Root CA or an intermediate certificate and put the new certificate into use. The server or node certificates, and the endpoint certificates are also rotated as part of this process.

A user with System Administrator role can perform CA certificate rotation. The CA certificate rotation process involves the following steps:
  1. Set the validity of the self-signed Root CA certificate or set up an intermediate CA certificate.
  2. Chose the endpoint certificate rotation controls.
  3. Start CA certificate rotation.
    • Oracle Key Vault generates a new self-signed Root CA certificate.
    • In case of the intermediate CA certificate, the new intermediate CA certificate was uploaded in an earlier step. No changes are done.
  4. Activate the new CA certificate.
  5. Monitor the progress of the endpoint certificate rotation issued with the new CA certificate.
  6. Oracle Key Vault issues server or node certificate with the new CA certificate after all the endpoint certificates rotation.
  7. Complete the post CA certificate rotation tasks.
CA certificate rotation process is the same for standalone, primary-standby, and cluster environments. In the multi-master environment, Oracle recommends that you select one of the cluster nodes to drive the CA certificate rotation. Oracle Key Vault automatically synchronizes the certificates in both systems in a primary-standby configuration, and in all nodes in a multi-master cluster configuration. You do not have to perform any extra configuration.

17.4.2 Checking for Self-Signed Root CA or Intermediate CA Certificate

Oracle Key Vault uses either a self-signed root CA certificate or an intermediate CA certificate.

To check if the current Oracle Key Vault CA certificate is a self-signed root CA or an intermediate CA, check the Common Name and Certificate Issuer fields in the Service Certificates page. If they are similar, for example, both are CA, or start with OKV_CA_, then the current CA certificate is a self-signed root CA. Otherwise, the current CA certificate is an intermediate CA. Additionally, in the intermediate CA certificate, the Certificate Issuer field displays the common name of the trusted third party.
Check the Common Name and Certificate Issuer of the current CA certificate in the Oracle Key Vault management console.
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
    In a multi-master cluster environment, you can log in to any node in the cluster.
  2. Select the System tab, then Settings from the left navigation side bar.
  3. In the Certificates area, select Service Certificates.
  4. In the Current CA Certificate area, check and compare the Common Name and Certificate Issuerfields.

17.4.3 Setting the Validity of Self-Signed Root CA Certificate

You can set the number of days for the validity of a self-signed Root certificate authority (CA) certificate.

The CA certificate validity period acts as an upper limit on the validity period of the server certificates, node certificates and endpoint certificates.
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
    In a primary-standby environment, log in to the primary Oracle Key Vault server. In a multi-master cluster environment, you can log in to any node in the cluster.
  2. Select the System tab, then Settings from the left navigation side bar.
  3. In the Certificates area, click Service Certificates.
  4. In the Service Certificates page, select Manage CA Certificate.
  5. In the CA Certificate Details page, select the Self-Signed Root CA option (this should be selected by default).
  6. Set the validity value in the Self-Signed Root CA Certificate Validity (in days) field.
    The default is 1095 days (3 years). You can set a maximum of 3650 days (10 years).
  7. Click Save.

    Go to section Rotating CA Certificate to generate and enable the self-signed root CA certificate.

17.4.4 Setting Up the Intermediate CA Certificate

Use the Oracle Key Vault management console to generate the certificate signing request for the intermediate CA certificate, and upload the intermediate CA certificate signed by a trusted third party.

Uploading the intermediate CA certificate does not enable it, that is, switch it into use. Perform the following steps to rotate the CA certificate and enable the uploaded intermediate CA certificate:
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
    In a primary-standby environment, log in to the primary Oracle Key Vault server. In a multi-master cluster environment, log in to the node selected for CA certificate rotation in the cluster.
  2. Select the System tab, then Settings from the left navigation side bar.
  3. In the Certificates area, click Service Certificates.
  4. In the Service Certificates page, select Manage CA Certificate.
  5. Under CA Certificate Details, select the Intermediate CA option.
  6. Enter information about your organization in the following fields:
    • Organization Name
    • Country/Region
    • Organization Unit
    • City
    • State/Province

    Note:

    These fields do not allow blank spaces.
  7. Select Generate Certificate Request.
  8. In the dialog box that lets you know that the generation will take a few minutes, click OK.
  9. After Oracle Key Vault generates the certificate request, in the CA Certificate Details area, select Download Certificate Request to download the certificate request file.

    The certificate signing request file is named as follows:

    OKV_Intermediate_CA_Certificate.csr

    The Intermediate CA Certificate Signing Request Details area shows the details of certificate signing request.

    At this stage, the current CA certificate is still enabled in Oracle Key Vault. The Current Certificate area displays the details of the currently active CA. If you want to cancel the setup of the intermediate CA certificate, then click Abort.

    Description of 214_ca_download_certificate_request.png follows
    Description of the illustration 214_ca_download_certificate_request.png
  10. Have a trusted third party issue the intermediate CA certificate using the downloaded certificate signing request.
  11. Upload the intermediate CA certificate. In the CA Certificate Details area, select Choose File for Intermediate CA Certificate to find and select the intermediate CA certificate file, and then click Upload. In a multi-master cluster environment, you must upload the intermediate CA certificate on the same node where the certificate signing request was downloaded.
  12. Upload the chain of trust for the intermediate CA certificate. In the CA Certificate Details area, select Choose File for Certificate Chain of Trust to find and select the chain of trust file, and then click Upload.
    Description of 214_upload_ca_intermediate_and_trust_chain_certificate.png follows
    Description of the illustration 214_upload_ca_intermediate_and_trust_chain_certificate.png

    The chain of trust file is a PEM bundle that consists of the CA certificate used by the external signing authority to sign the intermediate certificate signing request, OKV_Intermediate_CA_Certificate.csr file, as well as all of the certificates in that CA's trust chain, in reverse order.

    For example, the CA certificate that you use to sign OKV_Intermediate_CA_certificate.csr is CACertA. CACertA was, in turn, issued by CACertB. CACertB was issued by CACertC. The certificate trust chain file that you must upload must consist of CACertA, CACertB, CACertC, in that order, in the PEM bundle format.

    In a multi-master cluster environment, you must upload the certificate chain of trust on the same node where you uploaded the intermediate CA certificate.

    As part of the upload, Oracle Key Vault performs the following validations:
    1. The uploaded intermediate CA is verified using the uploaded certificate chain of trust.
    2. The certificate chain of trust has a depth of less than or equal to 8.

    After the uploads are successful, the Rotate CA Certificate button is displayed.

    Go to section Rotating CA Certificate to enable the self-signed root CA certificate.

17.4.5 Rotating CA Certificate

Use the Oracle Key Vault management console to rotate CA certificate and enable either a self-signed root CA certificate or an intermediate CA certificate.

Back up Oracle Key Vault before you start the certificate rotation process.
CA certificate rotation issues new certificates for the Oracle Key Vault servers, nodes, and endpoints.

Perform these steps to complete the CA certificate rotation process throughout the Oracle Key Vault environment.

  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.

    In a primary-standby environment, log in to the primary Oracle Key Vault server.

    In a multi-master cluster environment, log in to the node selected for CA certificate rotation in the cluster. If you want to enable an intermediate CA certificate, then ensure that you initiate the CA certificate rotation from the same node where the intermediate certificate was uploaded.

  2. Select the System tab, then Settings from the left navigation side bar.
  3. In the Certificates area, click Service Certificates.
  4. In the Service Certificates page, select Manage CA Certificate.
  5. If you want to enable a self-signed root CA certificate, then in CA Certificate Details Area, select the Self-Signed Root CA option.
    If necessary, set the Self-Signed Root CA Certificate Validity value as described in Setting the Validity of Self-Signed Root CA Certificate
  6. If you want to enable an intermediate CA certificate, then upload the intermediate CA certificate and the certificate chain of trust successfully. The Rotate CA Certificate button will now be visible.

    If you do not see the Rotate CA Certification button, then set up the intermediate CA certificate as described in sectionSetting Up the Intermediate CA Certificate.

  7. Click the Rotate CA certificate.
  8. In the Manage CA Certificate page, you may set endpoint certificate rotation controls - batch size and in multi-master cluster deployments, sequence.

    In a multi-master cluster environment, if necessary, choose the sequence in which the endpoint certificates should be rotated as described in sectionSetting the Endpoint Certificate Rotation Sequence

    Description of 214_endpoint_certiface_rotation_and_ca_certificate_controls.png follows
    Description of the illustration 214_endpoint_certiface_rotation_and_ca_certificate_controls.png

  9. In the Manage CA Certificate page, in the Current CA Certificate area, select Start CA Certificate Rotation.
  10. In the confirmation dialog box, click OK.

    If you enable a self-signed root CA certificate, a new self-signed root CA certificate is created. In a multi-master cluster environment, Oracle Key Vault distributes and installs the newly created self-signed root CA certificate or uploaded intermediate CA certificate to all nodes of the cluster. In a primary-standby environment Oracle Key Vault distributes and installs these certificates to the standby. In case of a standalone environment, Oracle Key Vault simply installs the certificate that you enable.

    At this stage, the endpoints continue to use the certificates issued using the previous CA certificate. The Old CA Certificate area displays the details of the currently active CA. The New CA Certificate area displays the certificate you have rotated along with its common name.

    If you want to cancel the rotation process, click Abort Rotation.

    In a multi-master cluster environment, note the following:

    • After the start of the certification rotation process, the details of the new certificate that was generated is displayed on the node on which you started the CA rotation. If you refresh the Manage CA Certificate page on all of the other nodes, this page displays a message that the new certificate is propagated to that node.
    • To access this page, select the System tab, select Settings in the left navigation bar, select Service Certificates,and then select Manage CA Certificate in the Certificates area.
    • The certificate is now distributed to all the nodes. The propagation process takes a few minutes to complete.
    • You can abort the certificate rotation before the point where:
      • All nodes in the cluster have received the new CA certificates.
      • Each node has notified the other nodes that it has received the certificate.

        The Abort button disappears and only the Activate Certificate is displayed.

    Periodically refresh the Manage CA Certificate page, in case there are any changes to the rotation status. For example, refresh the page to determine if the Abort button is no longer displayed and the Activate Certificate button is displayed.

  11. In the Manage CA Certificate window, click the Activate Certificate button when it is displayed and enabled.
  12. In the confirmation dialog box, click OK.

    Click and confirm the Activate Certificate button to begin the process of enabling the new Oracle Key Vault CA certificate. This process may take a few minutes to complete and displays the following message when it completes:

    Automatic certificate update of the endpoints is in progress.

    You cannot cancel the rotation process after you click Activate Certificate.

    In a multi-master cluster environment, Activate Certificate enables the certificate on all nodes in the cluster. As listed in step 11, the Activate Certificate button is displayed only when all nodes in the cluster have installed the new CA certificate, and the Manage CA Certificate page of all nodes does not display the Abort button.

    Ensure that you click Activate Certificate on only one node.



    The Activate Certificate takes a few minutes to propagate to all the nodes and the Manage Server Certificate page on other nodes may show no change in status. Refresh the Manage CA Certificate page on the other nodes till the following message is displayed:

    Automatic certificate update of the endpoints is in progress.

    The new CA certificate is now activated and the Oracle Key Vault servers or nodes begin issuing new endpoint certificates signed by the new CA certificate. The endpoints can now connect to the Oracle Key Vault server or nodes using the endpoint certificate issued by either the new or the old Oracle Key Vault CA. In the background, Oracle Key Vault starts issuing certificates for its endpoints, a few endpoints at a time.

    After Oracle Key Vault generates the new endpoint certificates for a given endpoint, the endpoint receives the certificate when it next makes a connection to the Oracle Key Vault server or node that generated the new certificates. After an endpoint receives its updated certificates from the Oracle Key Vault server, it must connect to the Oracle Key Vault server a second time to inform the server that it has successfully received the endpoint certificates. When the endpoint uses the certificates issued by the new CA, the value in the Common Name of Certificate Issuer field for that endpoint on the Endpoints page should reflect the common name of the new Oracle Key Vault CA certificate.

    Note:

    Periodically check the status of replication across the cluster by viewing either the Cluster Monitoring page or the Cluster Management page. To access either of these pages, click the Cluster tab, and then select either Management or Monitoring in the left navigation bar.
  13. To check if the credentials for an endpoint are updated, click the Check Endpoint Progress button.

    Click the Check Endpoint Progress button to display the Endpoints page.

    For more information, see, Checking Certificate Rotation Status for Endpoints



  14. Complete the CA certificate rotation.

    After Oracle Key Vault issues certificates to all the endpoints using the new CA certificate, the Oracle Key Vault server rotates the server certificates for standalone and primary-standby environments and the node certificates for the cluster environment.

    CA certificate rotation process is complete when the Manage CA Certificate page does not list the certificates but only lists the new CA certificate. In a multi-master cluster environment, to check if rotation is complete, go to each node and check the Manage CA Certificate page for that node. The CA certificate rotation process is complete when the Start CA Certificate Rotation button is available on the Manage CA Certificate page, along with the Current CA Certificate and Current Server Certificate.

    The CA certificate rotation process is complete when clicking the Manage CA certificate button on the Service Certificates page takes you to the CA Certificate Details page and you can make a choice between the Self-Signed Root CA and Intermediate CA. In a multi-master cluster environment CA certificate rotation process is complete when certificate rotation is complete on every node of the cluster.

    You can initiate another certificate rotation only after all the servers or nodes have completed their certification rotation process. After you complete the rotation, configure an alert for when the new certificate should be rotated next.

    Note:

    The CA Certificate rotation process can take several days to complete. Oracle recommends that you start the process ahead of the CA certificate expiration to avoid Oracle Key Vault and endpoint downtime.

17.4.6 Setting the Endpoint Certificate Rotation Batch Size

The endpoint certificate rotation batch size value represents the number of endpoints that can be in the ROTATED state on a given Oracle Key Vault server or node during the CA certification rotation process.

During the CA certificate rotation process, an endpoint is considered to be in a ROTATED state when Oracle Key Vault server or node has issued the endpoint certificate using the new CA certificate but the new endpoint certificate is either not yet received or acknowledged by the endpoint.
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
    In a primary-standby environment, log in to the primary Oracle Key Vault server.

    In a multi-master cluster environment, log in to the node selected for initiating the CA certificate rotation in the cluster.

  2. Select the System tab, then Settings from the left navigation side bar.
  3. In the Certificates area, click Service Certificates.
  4. In the Service Certificates page, select Manage CA Certificate.
  5. In the CA Certificate Details page, select the Self-Signed Root CA option or the Intermediate CA option (this should be selected by default). Click Rotate CA Certificate. For the Intermediate CA option, Rotate CA Certificate button is displayed only after the intermediate CA and its trust chain has been uploaded.
  6. Scroll to the Endpoint Certificate Rotation Controls area.
  7. Enter a value in the Endpoint Certificate Rotation Batch Size field.
    Enter a value from 5 through 50. The default is 15.
  8. Click Save.

17.4.7 Setting the Endpoint Certificate Rotation Sequence

In a multi-master cluster environment, when you rotate certificate authority (CA) certificate, you broadly set the order in which endpoints can be rotated by ordering cluster subgroups.

  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
    In a primary-standby environment, log in to the primary Oracle Key Vault server.

    In a multi-master cluster environment, log in to the node selected for initiating the CA certificate rotation in the cluster.

  2. Select the System tab, then Settings from the left navigation side bar.
  3. In the Certificates area, click Service Certificates.
  4. In the Service Certificates page, select Manage CA Certificate.
  5. In the CA Certificate Details page, select the Self-Signed Root CA option or the intermediate CA option (this should be selected by default).Then click Rotate CA Certificate. For the Intermediate CA option, Rotate CA Certificate button is displayed only after the intermediate CA and its trust chain has been uploaded.
  6. Scroll to the Endpoint Certificate Rotation Sequence area.
  7. Click Select Cluster Subgroup.
  8. In the Select Cluster Subgroup Order dialog box, move the cluster subgroups that contain the endpoints to rotate to the right, and then use the arrow keys to set their order.
    For example, if this is your priority list:
    1. ClusterSubgroupA (EP1, EP4)
    2. ClusterSubgroupB (EP2, EP3, EP5)
    3. ClusterSubgroupC (EP6, EP7)

    Endpoints EP1 and EP4, which belong to ClusterSubgroupA, will be rotated first. After EP1 and EP4 receive and acknowledge their updated endpoint certificates, the rotation process will move to the next set of endpoints, ClusterSubgroupB (EP2, EP3, EP5).

    You can check if an endpoint has received and acknowledged its new certifications by navigating to the Endpoints page. The endpoint's Certificate Issuer field will change from Updating to Current Certificate Issuer to DN_of_new_OKV_CA.

    Note:

    If you specify the cluster subgroup priority order, then the number of endpoints that are processed at a time may be less than the Endpoint Certificate Rotation Batch Size parameter. For instance, if a given cluster subgroup has far fewer endpoints associated with it than the Endpoint Certificate Rotation Batch Size parameter, then only endpoints from the chosen cluster subgroup will be processed. Oracle Key Vault server or node does not begin processing of endpoints from other cluster subgroups with the lower priority order until certificate rotation is complete for all of the endpoints from the current cluster subgroup.

  9. Click Apply.

Cluster subgroups are usually used to group endpoints in a region or data center. Since the re-issue of endpoint certificates during CA certificate rotation could be a time consuming process, it is convenient to process endpoints per cluster subgroup for operations simplicity.

17.4.8 Checking Overall Certificate Rotation Status

Use the Oracle Key Vault management console to check the overall status of a certificate rotation.

After all the endpoints have been updated to using the new certificate, the Oracle Key Vault server begins the process of fully rotating its own server certificates in the background.
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
  2. Select the System tab, then Settings in the left navigation side bar.
  3. In the Certificates area, select Service Certificate.
    By default, Service Certificate is selected.
  4. Check the Manage CA Certificate page.
  5. Check the certificate rotation status.

    After clicking the Manage CA Certificate and if you are directed to the CA Certificate Details page, you can make a choice between the Self-Signed Root CA and Intermediate CA, the certificate rotation is complete. Otherwise it is still in progress.

    The End Date field in the Service Certificates page should reflect the expiration time of the new CA certificate.

    In a multi-master cluster environment CA certificate rotation process is complete when certificate rotation is complete on every node of the cluster.

    you can initiate another certificate rotation only after all the nodes have completed their certification rotation process.

17.4.9 Checking Certificate Rotation Status for Endpoints

Use the Oracle Key Vault management console to check the status of a certificate rotation for endpoints.

  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
  2. Select the Endpoints tab.
  3. Select Endpoints.

    In the Endpoints page, Common Name of Certificate Issuer field tracks the progress of how many endpoints have been issued certificates using the new CA certificate.

    The Common Name of Certificate Issuer field shows if the endpoint certificate is issued by the Old CA, the new CA, or if the endpoint is in the process of updating its endpoint certificate.

    For an endpoint, whose endpoint certificate:
    • Has been issued using the new CA certificate, the Common Name of Certificate Issuer field shows the common name of the new CA.
    • Is in the process of being issued using the new CA certificate, the Common Name of Certificate Issuer field shows Updating to Current Certificate Issuer.
    • Has not been issued using the new CA certificate, Common Name of Certificate Issuer field shows the common name of the old CA.

    Note:

    If there are errors with the certificate rotation of an endpoint, then Oracle recommends that you re-enroll the endpoint.

17.4.10 Post-CA Certificate Rotation Tasks

After you complete the CA certificate rotation, perform the post-rotation tasks.

  • If you had previously downloaded the Oracle Key Vault RESTful services software utility (okvrestclipackage.zip), then download it again to continue to use the RESTful services utility.

    Ensure that you have fully rotated the certificate, across all the nodes in a multi-master cluster environment and in the servers of a primary-standby environment, before you download okvrestclipackage.zip.

    To do this, select the Endpoint Enrollment and Software Download link on the Oracle Key Vault management console login page. Select the Download RESTful Service Utility tab, and then click Download to download the okvrestclipackage.zip file to a secure location.

  • Update the backup destinations

    After the CA certificate rotation, each server or node will have been issued a new certificate. The public key of the Oracle Key Vault node or server will also have changed. You need to copy the public key that appears in the Public Key field on the Backup Destination Details page and then paste it in the appropriate configuration file, such as authorized_keys, on the backup destination server.

    To do so, navigate to the System tab, then Settings in the left navigation side bar. In the System Configuration area, select Backup and Restore. Click on the Manage Backup Destination to view all backup destinations. Click on the Createbutton. The Public Keyfield will have the new public key.

  • Back up all Oracle Key Vault nodes and servers.

    It is important to perform this backup operation after the certificate rotation is complete. Later, if you have to restore a backup, the backup to restore must have been initiated after the CA certificate rotation. Restoring the backup before the CA certificate rotation can make the Oracle Key Vault server available but then the endpoints will not be able to connect to the restored Oracle Key Vault server. The CA certificate of the restored system may have expired and the endpoints would be using the endpoint certificates issued by the new CA not present in the backup done before CA certificate rotation.

17.4.11 Factors Affecting CA Certificate Rotation Process

Consider these factors that affect the certificate authority (CA) certificate rotation process in cluster environments.

The duration of CA certificate rotation is determined by how quickly the CA, node, and endpoint certificates are rotated. The endpoint certificate rotation takes the most time.

During the CA certificate rotation process, Oracle Key Vault rotates certificates for endpoints in batches on each node of the cluster, with an upper limit on the number of endpoints that are allowed to be in the ROTATED state at any one time. The number of endpoints that can be in a ROTATED state at any given time on an Oracle Key Vault node is defined by the endpoint certificate rotation batch size. The endpoint must receive its new certificate from the issuing node and acknowledge the receipt of the certificate back to the issuing node. An endpoint must have created at least one object for it to receive the certificate.

Note:

Generally, the node that issues an endpoint's certificate is one of those in the endpoint's affiliated cluster subgroup.

The following factors affect the endpoint certificate rotation process:

  • In order to receive the new certificates, the endpoint must reach out to the issuing node on which its certificates have been generated. Since the endpoint can communicate with any node in the endpoint node scan list, the endpoint may run many operations before it reaches the creator node and receives its certificate. The endpoint also has to acknowledge the receipt of the new certificates by reaching out to a node in the cluster.
  • The endpoint certificate rotation times increases with the number of nodes in the cluster. The endpoints prioritize the nodes in the local subgroup, hence consider setting a different subgroup for each node during the CA certificate rotation.
  • The endpoint certificate rotation batch size applies to each node of the cluster. So, if the endpoints are created on each node evenly, each node will rotate the number of endpoints equal to the batch size simultaneously. However, if all the endpoints are created on a single node, then the certificate rotation burden for all the endpoints will fall on that one node instead of being distributed across other nodes.
  • For faster endpoint certification rotation and general load balancing in the cluster, consider distributing the endpoint creation among all nodes of the cluster.
  • If the endpoints were created before an upgrade from Oracle Key Vault release 12.2, then the endpoints may all be associated with one single node. This can make the rotation process slower than if the endpoints had been created on different cluster nodes.
  • An endpoint can only successfully receive an update if it has at least one object uploaded to the Oracle Key Vault server. You can check if the endpoint has objects by executing the okvutil list command.

    For any endpoint stalling the endpoint certificate rotation, consider endpoint re-enroll or running the okvutil list command.

17.4.12 Guidelines for Managing CA Certificate Rotations

Consider these Oracle Key Vault guidelines for managing certificate authority (CA) certificate.

Guidelines for Endpoint Software Versions

  • For self-signed root CA certificate rotation, ensure that all the endpoints software are at version 18.2.0.0.0 or later.
  • For intermediate CA certificate rotation, ensure that all the endpoints software are at version 21.4.0.0.0 or later.
  • Upgrade the endpoint software to the same version as Oracle Key Vault before initiating a CA certificate rotation to ensure that the latest fixes to certificate rotation are also available on the endpoint software.

Recommendations for CA Certificate Rotation

  • In a multi-master cluster environment, Oracle recommends that you initiate the rotation from one node only. Use this node to complete the CA certificate rotation process. In case a node is made unavailable during certificate rotation, pick another node and use that node to complete the rest of the CA certificate rotation process. Do not switch nodes while performing certificate rotation.
  • Before performing a CA certificate rotation, back up the Oracle Key Vault system.
  • If a given endpoint does not receive its re-issued endpoint certificate due to network or other issues, or is in the SUSPENDED state, Oracle recommends that you re-enroll the endpoint during the endpoint certificate rotation, or delete the endpoint. If an endpoint uses the persistent master encryption key cache, it is recommended that the PKCS11 Persistent Cache Refresh Window parameter should be set to a large value before initiating a CA certificate rotation process. This will allow the certificate rotation process to continue to completion. You can find the current certificate rotation status by going to the Endpoints page and looking for Common Name of Certificate Issuer.

Checks Before Initiating CA Certificate Rotation

  • Before beginning certificate rotation, ensure that the recovery pass phrase is the same across all multi-master cluster nodes.
  • You cannot perform a CA certificate rotation when a backup operation or a restore operation is in progress.
  • Depending on the deployment, the CA certificate rotation process can take several days to complete, begin the CA certificate rotation well in advance of the CA certificate expiry.
  • In order for the CA certificate rotation process to reach completion, you must delete or re-enroll all endpoints that are not in the Enrolled state. If you no longer need the endpoint, then delete it.
  • Ensure node addition is not in progress. Do not initiate a CA certificate rotation while a node addition is in progress.
  • Ensure any node operation is not in progress. Do not try node operations (such as adding or disabling nodes) when a CA certificate rotation is in process.
  • In the multi-master cluster environment, ensure all the nodes are active. Do not initiate CA certificate rotation till all nodes in the cluster are active. You can check if a node is active by checking the Cluster Monitoring page. Click the Cluster tab, and then select Monitoring from the left navigation bar.
  • In a primary-standby environment, ensure the primary server is active. Do not perform CA certificate rotation if the primary server is in read-only restricted mode. Only initiate a CA certificate rotation when both servers in the configuration are active and synchronized with each other.

Expired CA Certificate

  • Do not initiate the CA certificate rotation process if the CA certificate has expired. Contact Oracle Support.
  • Do not attempt to upgrade the systems if CA certificate have already expired. This will lead to failed upgrades.

Note:

Do not start CA certificate rotation or attempt to upgrade Oracle Key Vault if the CA certificates have already expired and contact Oracle support.

17.5 Managing Server Certificates and Node Certificates Rotation

Use the Oracle Key Vault management console to rotate server or node certificates.

17.5.1 About Server Certificates and Node Certificates Rotation

Oracle Key Vault uses server certificates to communicate with its endpoints. Oracle Key Vault cluster nodes use node certificates to communicate with each other and with the endpoints.

These certificates are referred to as server certificates for standalone and primary-standby configurations and as node certificates in multi-master cluster configurations. The Oracle Key Vault certificate authority (CA) certificate issues these certificates.

You can rotate just these certificates, independently of the CA certificate rotation process. Doing so has no impact on the certificate expiry dates of the Oracle Key Vault CA or on any endpoints.

It is useful to rotate just the server and node certificates in situations where the Oracle Key Vault CA is still valid for much longer, but the server node certification will expire soon. This can happen because the CA validity is usually longer than the server or node certification validity.

The server or node certificate rotation process is described below:
  • Set the validity of the server or node certificate
  • Rotate server or node certificate

17.5.2 Configuring Certificate Validity Period for Server and Node Certificates

You can configure the validity period for server or node certificates in the Oracle Key Vault management console.

The certificate validity period takes effect the next time you rotate the server or node certificates. It will also be taken into account when you generate the server or node certificates as part of a CA certificate rotation, or when you add a new node to the cluster, to the node certificates for that new node. Irrespective of the value that the server or node certificate validity is set to, when the certificates are eventually generated, Oracle Key Vault ensures that their expiry date is less than that of the CA certificate.

  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
    In a multi-master cluster environment, you can log in to any node in the cluster.
  2. Select the System tab, then Settings from the left navigation side bar.
  3. In the Certificates area, select Service Certificates.
  4. Depending on your environment, perform the following:
    • In a standalone or primary-standby environment: In the Current Server Certificate area, select Manage Server Certificate.
    • In a multi-master cluster environment: In the Current Node Certificate area, select Manage Node Certificate.
  5. In the Server Certificate Validity (in days) or Node Certificate Validity (in days) field, enter a value between 365 days (the minimum and the default) and 1095 days for this setting.
  6. Click Save.

17.5.3 Rotating Server Certificates and Node Certificates

You can rotate server certificates and node certificates in the Oracle Key Vault management console.

Before you perform the rotation, ensure that you read the guidelines for rotating server certificates and node certificates.
  1. Log in to the Oracle Key Vault management console as a user who has the System Administrator role.
    In a multi-master cluster environment, you can log in to any node in the cluster.
  2. Select the System tab, then Settings from the left navigation side bar.
  3. In the Certificates area, select Service Certificates.
  4. Depending on your environment, perform the following:
    • In a standalone or primary-standby environment: In the Current Server Certificate area, select Manage Server Certificate.
    • In a multi-master cluster environment: In the Current Node Certificate area, select Manage Node Certificate.
  5. If required, in the Server Certificate Validity (in days) field (for standalone or primary-standby environments) or Node Certificate Validity (in days) field (for multi-master cluster environments), enter a value between 365 days (the minimum and the default) and 1095 days for this setting.
    Wait several minutes to make sure that this setting takes effect, particularly in a multi-master cluster environment. When the change is visible across all cluster nodes (navigate to the same page on each node to verify), you are ready to initiate a server or node certificate rotation.
  6. Depending on your environment, do the following:
    • In a standalone or primary-standby environment: Select Generate Server Certificate.
    • In a multi-master cluster environment: Select Generate Node Certificate.
  7. In the confirmation window, click OK.
    This process can take several minutes to complete. It may also result in a momentary disruption of endpoint servicing.
If the process successfully completes, then the Current Server Certificate (for standalone or primary-standby environments) and the Current Node Certificate (for multi-master cluster environments) sections display new values for the End Date and Expiring in settings. In a multi-master cluster environment, you can view the expiry dates of all the node certificates in the cluster in the Cluster Node Certificate Details area.

17.5.4 Guidelines for Rotating Server Certificates and Node Certificates

Review these guidelines before you perform a rotation of server certificates or node certificates.

  • Do not perform a certificate authority (CA) certificate rotation while a server or node certificate rotation is in progress.
  • Do not perform a server or node certificate rotation while a CA certificate rotation is in progress.
  • Do not perform a node certificate rotation on one node while another is in progress on a different node.
  • Do not alter the CA certificate validity period while a CA certificate rotation is in progress.
  • Do not attempt to rotate the server certificates if the CA certificate is already expired.
  • Do not alter the Server Certificate Validity (in days) field (for standalone or primary-standby environments) or Node Certificate Validity (in days) field while either a CA certificate rotation or a server or node certificate rotation is in progress.