E Managing Oracle Key Vault Platform Certificates

This chapter describes how to manage Oracle Key Vault Platform certificates by monitoring and rotating platform certificates before they expire.

• Overview of Oracle Key Vault Platform Certificates
  Oracle Key Vault platform certificates are used when adding a new node to an Oracle Key Vault multi-master cluster, or when adding systems to a primary-standby deployment.
• Monitoring Oracle Key Vault Platform Certificate Expiration
  You can proactively set alerts and monitor the expiration dates of the Oracle Key Vault platform certificates and rotate them before they expire.
• Rotating Platform Certificates
  You must rotate the platform certificates by logging in to the Oracle Key Vault system and running a series of commands.

E.1 Overview of Oracle Key Vault Platform Certificates

Oracle Key Vault platform certificates are used when adding a new node to an Oracle Key Vault multi-master cluster, or when adding systems to a primary-standby deployment.

They are also used when shipping redo between read/write nodes in the cluster. These certificates are different from the Oracle Key Vault service certificates and have different expiration dates. They are also managed using a different process than Oracle Key Vault service certificates. If you do not rotate the Oracle Key Vault platform certificates before they expire, you cannot add a new node to the Oracle Key Vault multi-master cluster. The redo shipping between Oracle Key Vault read/write nodes may also be impacted, causing each node of the read/write pair to go into read-only restricted mode. You cannot upgrade an Oracle Key Vault system with expired platform certificates. You must rotate the Oracle Key Vault platform certificates before they expire.

Rotating the Oracle Key Vault platform certificates does not rotate the Oracle Key Vault service certificates and does not impact endpoint communication with Oracle Key Vault. Similarly, rotating Oracle Key Vault service certificates does not rotate the platform certificates.

E.2 Monitoring Oracle Key Vault Platform Certificate Expiration

You can proactively set alerts and monitor the expiration dates of the Oracle Key Vault platform certificates and rotate them before they expire.

• Finding the Expiration Date of Platform Certificates
  You can find the expiration date of platform certificates on the Oracle Key Vault management console.
• Monitoring Platform Certificates Expiration Using Platform Certificate Expiration Alerts
  You can set the expiration alerts as reminders to rotate the platform certificates before they expire.

E.2.1 Finding the Expiration Date of Platform Certificates

You can find the expiration date of platform certificates on the Oracle Key Vault management console.

1. Log in to the Oracle Key Vault management console as a System Administrator. In a multi-master cluster environment, you can log in to any node of the cluster.
2. Select the System tab, then select Status from the navigation side bar.
3. Check the Platform Certificates Expiration Date field to determine when the platform certificates will expire.
4. Check the Platform Certificates Expiring In field to determine how many days are left for the certificates to expire.
   Oracle Key Vault raises an alert for the platform certificates expiration when the platform certificates expiration date falls within the alert threshold period.

   Note:

   The expiration date shown is the minimum of the expiration dates of the platform certificate used when adding new multi-master cluster nodes or configuring a primary-standby deployment and the one that is used for redo shipping.

E.2.2 Monitoring Platform Certificates Expiration Using Platform Certificate Expiration Alerts

You can set the expiration alerts as reminders to rotate the platform certificates before they expire.

Expiration of the platform certificates could result in redo shipping failures between read/write nodes in a multi-master cluster or between the primary and standby of a primary-standby deployment, resulting in the systems going into read-only restricted mode. It may also prevent the addition of a new node to the Oracle Key Vault cluster and prevent upgrade of the Oracle Key Vault system. Ensure that you rotate the Oracle Key Vault platform certificates before their expiration date. To avoid this scenario, Oracle recommends that you configure the Platform Certificate Expiration alert as a reminder to rotate the Oracle Key Vault platform certificates before they expire. This alert is separate from those monitoring expiration of the Oracle Key Vault service certificates (CA, server/node, and endpoint certificates).

E.3 Rotating Platform Certificates

You must rotate the platform certificates by logging in to the Oracle Key Vault system and running a series of commands.

In the case of a multi-master cluster, different steps may need to be run on different nodes of the cluster. In the case of a primary-standby environment, the steps may need to be run on both primary and standby.

• Rotating Platform Certificates on a Standalone Oracle Key Vault Server
  Rotate the platform certificates on a standalone Oracle Key Vault server to replace the existing certificates near their expiration date with the new ones.
• Rotating Platform Certificates in a Multi-Master Cluster Environment
  Learn how to rotate platform certificates in a multi-master cluster environment.

E.3.1 Rotating Platform Certificates on a Standalone Oracle Key Vault Server

Rotate the platform certificates on a standalone Oracle Key Vault server to replace the existing certificates near their expiration date with the new ones.

1. SSH into the Oracle Key Vault system as the support user.

   ssh support@OKV_SERVER_IP_ADDRESS

2. Switch to user root:

   su- root

3. Check the validity of the platform certificate used when adding new nodes to a multi-master cluster or configuring a primary-standby deployment as follows:

   openssl x509 -noout -enddate -in /usr/local/dbfw/etc/ca.crt
   openssl x509 -noout -enddate -in /etc/pki/tls/certs/localhost_internal.crt
   openssl x509 -noout -enddate -in /usr/local/dbfw/etc/cert.crt
   openssl x509 -noout -enddate -in /usr/local/dbfw/etc/avs/avs_apex_client.crt
   openssl x509 -noout -enddate -in /etc/pki/tls/certs/localhost.crt

   If the dates shown indicate that any of the certificates have expired or will soon expire (as indicated by the platform certificate alerts), you must rotate the certificate by performing the following steps:

   • For Oracle Key Vault systems at release 21.1, 21.2, 21.3, 21.4, and 21.5, download the Oracle Audit Vault and Database Firewall patch for bug 34378212. SCP the downloaded zip file to the Oracle Key Vault server and unzip the file:

      scp <downloaded_patch_file> support@OKV_NODEA_IP
      ssh support@OKV_NODEA_IP
      su - root
      cd /tmp
      unzip <downloaded_zip_file_for_patch_34378212> -d /tmp
      mkdir /root/gensslcert
      cp /tmp/gensslcert.avs.tar.gz /root/gensslcert
      cd /root/gensslcert
      tar xvfz gensslcert.avs.tar.gz
      export GENSSLCERT_HOME="/root/gensslcert"
      echo $GENSSLCERT_HOME

   • If the Oracle Key Vault system is at release 21.6 or higher, then run the following command:

     export GENSSLCERT_HOME= " /usr/local/bin/"
     echo $GENSSLCERT_HOME

   • Run the following command to regenerate the platform certificate used when adding new nodes to a multi-master cluster or configuring a primary-standby deployment:

     $GENSSLCERT_HOME/gensslcert destroy-certs create-ca

4. Restart the necessary services by running the following commands:

     systemctl reload httpd
     systemctl restart controller

5. Update the platform CA certificate bundle, using the following command:

   cat /usr/local/dbfw/etc/ha_partner.crt /usr/local/dbfw/etc/ca.crt > /etc/pki/tls/certs/dbfw-ca.crt
   systemctl reload httpd
   systemctl restart controller
   systemctl start monitor

6. Check the validity of the platform certificate used for redo shipping by performing the following steps. It is sufficient to check the validity on one multi-master cluster node only.

   su oracle
   cd /var/lib/oracle/dbfw/network/admin/avwallet
   orapki wallet display -wallet  /var/lib/oracle/dbfw/network/admin/avwallet -complete
   orapki wallet export -wallet . -dn "DC=com,CN=avserver,OU=db,O=oracle" -cert /tmp/avserver_check_cert_validity.cert
   openssl x509 -in /tmp/avserver_check_cert_validity.cert -subject -enddate -noout

   If the certificate has expired or is expiring soon, then you must rotate it using the steps starting with step 8.

7. Switch back to the root user using the command:

   exit

8. Run the following commands to regenerate the platform certificate used for redo shipping between multi-master cluster read/write nodes or between the systems in a primary-standby deployment:

   cd /opt/avdf/lib/ruby/avdf
   ruby update_agent_cert_task.rb

9. Run the following commands to verify the validity of the regenerated certificates:

   openssl x509 -startdate -enddate -noout -in /usr/local/dbfw/etc/ca.crt
   openssl x509 -startdate -enddate -noout -in /etc/pki/tls/certs/localhost_internal.crt
   openssl x509 -startdate -enddate -noout -in /usr/local/dbfw/etc/cert.crt
   openssl x509 -startdate -enddate -noout -in /usr/local/dbfw/etc/avs/avs_apex_client.crt
   openssl x509 -startdate -enddate -noout -in /etc/pki/tls/certs/localhost.crt

10. Switch to user oracle to verify the validity of the last certificate, using the following commands:

    su oracle
    cd /var/lib/oracle/dbfw/network/admin/avwallet
    orapki wallet display -wallet  /var/lib/oracle/dbfw/network/admin/avwallet -complete
    orapki wallet export -wallet . -dn "DC=com,CN=avserver,OU=db,O=oracle" -cert /tmp/avserver_check_cert_validity.cert
    openssl x509 -in /tmp/avserver_check_cert_validity.cert -subject -startdate -enddate -noout

11. Switch back to user root using the following command:

    exit

12. Restart the database service on the system using the following command:

    systemctl restart dbfwdb

13. Verify the validity of the platform certificates from the Oracle Key Vault management console.

E.3.2 Rotating Platform Certificates in a Multi-Master Cluster Environment

Learn how to rotate platform certificates in a multi-master cluster environment.

• Rotate Platform CA Certificate on Read/Write Multi-Master Cluster Nodes
  Rotate the platform certificates on Read/Write Multi-Master Cluster Nodes to replace the existing certificates near their expiration date with the new ones.
• Rotate Platform CA Certificate on Read-Only Multi-Master Cluster Nodes
  Learn how to rotate platform certificates on each read-only Oracle Key Vault multi-master cluster node.
• Rotate Platform Certificate Used For Redo Shipping On Any One Multi-Master Cluster Node
  In a multi-master cluster environment, you must rotate the platform certificate used for redo shipping on one node of the multi-master cluster, and then transfer the certificate to all of the other multi-master cluster nodes.
• Transfer the Rotated Redo Shipping Platform Certificate to Other Multi-Master Cluster Nodes
  Learn how to transfer the redo shipping platform certificate to other multi-master cluster node after rotating it on one multi-master cluster node.

E.3.2.1 Rotate Platform CA Certificate on Read/Write Multi-Master Cluster Nodes

Rotate the platform certificates on Read/Write Multi-Master Cluster Nodes to replace the existing certificates near their expiration date with the new ones.

In this section, Node A and Node B refer to the two nodes of a given read/write pair. Implement these steps on each set of read/write pairs in turn.

1. (Node A) SSH into Node A as user support:

   ssh support@OKV_NODEA_IP

2. (Node A) Switch to user root on Node A :

   su - root

3. Check the validity of the platform certificate used when adding new nodes to a multi-master cluster or configuring a primary-standby deployment as shown below. This step is necessary only on any one node of the multi-master cluster.

   openssl x509 -noout -enddate -in /usr/local/dbfw/etc/ca.crt
   openssl x509 -noout -enddate -in /etc/pki/tls/certs/localhost_internal.crt
   openssl x509 -noout -enddate -in /usr/local/dbfw/etc/cert.crt
   openssl x509 -noout -enddate -in /usr/local/dbfw/etc/avs/avs_apex_client.crt
   openssl x509 -noout -enddate -in /etc/pki/tls/certs/localhost.crt

   If the dates indicate that any of the certificates have expired or will soon expire (as indicated by the platform certificate alerts), you must rotate the certificates using the following steps:

   • (Node A)
     • For Oracle Key Vault systems at release 21.1, 21.2, 21.3, 21.4, and 21.5, download the Oracle Audit Vault and Database Firewall patch for bug 34378212. SCP the downloaded zip file to the Oracle Key Vault server and unzip the file using the following commands:

        scp <downloaded_patch_file> support@OKV_NODEA_IP
        ssh support@OKV_NODEA_IP
        su - root
        cd /tmp
        unzip <downloaded_zip_file_for_patch_34378212> -d /tmp
        mkdir /root/gensslcert
        cp /tmp/gensslcert.avs.tar.gz /root/gensslcert
        cd /root/gensslcert
        tar xvfz gensslcert.avs.tar.gz
        export GENSSLCERT_HOME="/root/gensslcert"
        echo $GENSSLCERT_HOME

     • If the Oracle Key Vault system is at release 21.6 or higher, then run the following command:

       export GENSSLCERT_HOME= " /usr/local/bin/"
       echo $GENSSLCERT_HOME

     • Run the following command on Node A (at release 21.1 or higher) to regenerate the platform certificate used when adding new nodes to a multi-master cluster or configuring a primary-standby deployment:

       $GENSSLCERT_HOME"/gensslcert destroy-certs create-ca
       systemctl reload httpd
       systemctl restart controller

4. (Node A) Transfer a copy of the regenerated certificate to Node B using SCP:

   scp /usr/local/dbfw/etc/ca.crt support@OKV_NODEB_IP:/tmp/ha_partner.crt

5. (Node B)
   • SSH into Node B as the support user, then switch to user root:

     ssh support@OKV_NODEB_IP
     su - root

   • For Oracle Key Vault systems at release 21.1, 21.2, 21.3, 21.4, and 21.5, download the Oracle Audit Vault and Database Firewall patch for bug 34378212. SCP the downloaded zip file to the Oracle Key Vault server and unzip the file using the commands below:

      scp <downloaded_patch_file> support@OKV_NODEB_IP
      ssh support@OKV_NODEB_IP
      su - root
      cd /tmp
      unzip <downloaded_zip_file_for_patch_34378212> -d /tmp
      mkdir /root/gensslcert
      cp /tmp/gensslcert.avs.tar.gz /root/gensslcert
      cd /root/gensslcert
      tar xvfz gensslcert.avs.tar.gz
      export GENSSLCERT_HOME="/root/gensslcert"
      echo $GENSSLCERT_HOME

   • If the Oracle Key Vault system is at release 21.6 or higher, then run the following command:

     export GENSSLCERT_HOME= " /usr/local/bin/"
     echo $GENSSLCERT_HOME

6. (Node B) Run the following commands on Node B:

   cp /tmp/ha_partner.crt /usr/local/dbfw/etc/ha_partner.crt
   systemctl reload httpd
   systemctl restart controller
   "$GENSSLCERT_HOME"/gensslcert destroy-certs create-ca
    systemctl reload httpd
    systemctl restart controller      

7. (Node B) Transfer a copy of the regenerated certificate to Node A using SCP:

   scp /usr/local/dbfw/etc/ca.crt support@OKV_NODEA_IP:/tmp/ha_partner.crt

8. (Node A) SSH into Node A as the support user, then switch to user root:

   ssh support@OKV_NODEA_IP
   su - root

9. (Node A) Run the following commands on Node A:

   cp /tmp/ha_partner.crt /usr/local/dbfw/etc/ha_partner.crt
   systemctl reload httpd
   systemctl restart controller
   cat /usr/local/dbfw/etc/ha_partner.crt /usr/local/dbfw/etc/ca.crt > /etc/pki/tls/certs/dbfw-ca.crt
   systemctl reload httpd
   systemctl restart controller
   systemctl restart monitor

10. (Node B) SSH into Node B as the support user, then switch to user root:

    ssh support@OKV_NODEB_IP
    su - root

11. (Node B) Run the following commands on Node B:

    cat /usr/local/dbfw/etc/ha_partner.crt /usr/local/dbfw/etc/ca.crt > /etc/pki/tls/certs/dbfw-ca.crt
    systemctl reload httpd
    systemctl restart controller
    Systemctl restart monitor

    Repeat steps 1 to 11 on each pair of read/write nodes in the multi-master cluster.

E.3.2.2 Rotate Platform CA Certificate on Read-Only Multi-Master Cluster Nodes

Learn how to rotate platform certificates on each read-only Oracle Key Vault multi-master cluster node.

1. For Oracle Key Vault systems at release 21.1, 21.2, 21.3, 21.4, and 21.5, download the Oracle Audit Vault and Database Firewall patch for bug 34378212.
   • SCP the downloaded zip file to the Oracle Key Vault server and unzip the file using the following commands:

     scp <downloaded_patch_file> support@OKV_SERVER_IP_ADDRESS
     ssh support@OKV_SERVER_IP_ADDRESS
     su - root
     cd /tmp
     unzip <downloaded_zip_file_for_patch_34378212> -d /tmp
     mkdir /root/gensslcert
     cp /tmp/gensslcert.avs.tar.gz /root/gensslcert
     cd /root/gensslcert
     tar xvfz gensslcert.avs.tar.gz
     export GENSSLCERT_HOME="/root/gensslcert"
     echo $GENSSLCERT_HOME

   • If the Oracle Key Vault system is at release 21.6 or higher, then run the following commands:

     export GENSSLCERT_HOME= " /usr/local/bin/"
     echo $GENSSLCERT_HOME

2. Run the following command to regenerate the platform certificate used when adding new nodes to a multi-master cluster or configuring a primary-standby deployment:

   $GENSSLCERT_HOME/gensslcert destroy-certs create-ca

3. Restart the necessary services by running the following commands:

   systemctl reload httpd
   systemctl restart controller

4. Update the platform CA certificate bundle:

   cat /usr/local/dbfw/etc/ha_partner.crt /usr/local/dbfw/etc/ca.crt > /etc/pki/tls/certs/dbfw-ca.crt
   systemctl reload httpd
   systemctl restart controller
   systemctl start monitor

   Repeat steps 1 to 4 on each read-only node in the multi-master cluster.

E.3.2.3 Rotate Platform Certificate Used For Redo Shipping On Any One Multi-Master Cluster Node

In a multi-master cluster environment, you must rotate the platform certificate used for redo shipping on one node of the multi-master cluster, and then transfer the certificate to all of the other multi-master cluster nodes.

Perform these steps to rotate the redo shipping platform certificates on one node of the multi-master cluster.

1. Select one node of the multi-master cluster on which to rotate the redo shipping platform certificate.
2. SSH into the Oracle Key Vault node as the support user.

   ssh support@OKV_SERVER_IP_ADDRESS

3. Switch to user root.

   su -root

4. Check the validity of the platform certificate used for redo shipping using the following commands. It is sufficient to check this on one multi-master cluster node only.

   su oracle
   cd /var/lib/oracle/dbfw/network/admin/avwallet
   orapki wallet display -wallet  /var/lib/oracle/dbfw/network/admin/avwallet -complete
   orapki wallet export -wallet . -dn "DC=com,CN=avserver,OU=db,O=oracle" -cert /tmp/avserver_check_cert_validity.cert
   openssl x509 -in /tmp/avserver_check_cert_validity.cert -subject -enddate -noout

   If the certificate has expired or will soon expire, then you must rotate it using the subsequent steps.

5. Switch back to the user root using the following command:

   exit

6. Rotate the platform certificate used for redo shipping using the commands below, still as user root.

   cd /opt/avdf/lib/ruby/avdf
   ruby update_agent_cert_task.rb

7. Switch to user oracle to verify the validity of the certificate, using the following commands:

   su oracle
   cd /var/lib/oracle/dbfw/network/admin/avwallet
   orapki wallet display -wallet  /var/lib/oracle/dbfw/network/admin/avwallet -complete
   orapki wallet export -wallet . -dn "DC=com,CN=avserver,OU=db,O=oracle" -cert /tmp/avserver_check_cert_validity.cert
   openssl x509 -in /tmp/avserver_check_cert_validity.cert -subject -startdate -enddate -noout

8. Switch back to user root using the following command:

   exit

9. Restart the database service on the system using the following command:

   systemctl restart dbfwdb

10. Verify the validity of the platform certificates from the Oracle Key Vault management console.
11. Transfer the rotated redo shipping platform certificate to all other multi-master cluster nodes using the steps detailed in the section Transfer the Rotated Redo Shipping Platform Certificate to Other Multi-Master Cluster Nodes.

E.3.2.4 Transfer the Rotated Redo Shipping Platform Certificate to Other Multi-Master Cluster Nodes

Learn how to transfer the redo shipping platform certificate to other multi-master cluster node after rotating it on one multi-master cluster node.

1. SSH into the Oracle Key Vault node on which the redo shipping platform certificate was rotated, as the support user.

   ssh support@OKV_SERVER_IP_ADDRESS 

2. Switch to user root.

   su - root

3. Create a bundle containing the rotated platform certificate using the following steps:

    cd /var/lib/oracle/dbfw/network/admin/avwallet
          zip -r avwallet_regenerated.zip *

4. Use SCP to copy the bundle to all other multi-master cluster nodes.

     scp avwallet_regenerated.zip support@<OKV_cluster_node_IP>:/tmp

5. Repeat the following set of steps on each of the other multi-master cluster nodes:
   1. SSH into the Oracle Key Vault node on which the redo shipping platform certificate was rotated, as the support user.

      ssh support@OKV_SERVER_IP_ADDRESS 

   2. Switch to user root.

      su - root

   3. Unzip the avwallet_regenerated zip file that was copied using SCP using the following command:

      cd /tmp
      unzip avwallet_regenerated.zip -d avwallet_regenerated
      chmod -R 775 avwallet_regenerated

   4. Take a backup of the old platform certificate using the following command:

      cd /var/lib/oracle/dbfw/network/admin
      cp -Rp avwallet avwallet_expired

   5. Copy the new wallet, ensuring to preserve permissions :

      cp /tmp/avwallet_regenerated/* avwallet/

   6. Restart services using the following commands:

      service dbfwlistener restart
      service dbfwdb restart 

   7. Switch to user oracle to verify the validity of the certificate, using the following commands:

      su oracle
      cd /var/lib/oracle/dbfw/network/admin/avwallet
      orapki wallet display -wallet  /var/lib/oracle/dbfw/network/admin/avwallet -complete
      orapki wallet export -wallet . -dn "DC=com,CN=avserver,OU=db,O=oracle" -cert /tmp/avserver_check_cert_validity.cert
      openssl x509 -in /tmp/avserver_check_cert_validity.cert -subject -startdate -enddate -noout

   Verify the validity of the platform certificates from the Oracle Key Vault management console.