Rotating Expired Platform Certificates

This chapter describes how you can rotate Oracle Key Vault platform certificates after they have expired or are in a deprecated primary-standby deployment. Oracle recommends proactively monitoring and rotating platform certificates before they expire to avoid cluster downtime.

If the certificates have not expired, follow the steps from Rotating Unexpired Platform Certificates.

You must rotate expired platform certificates by logging in to the Oracle Key Vault system and running a series of commands.

In a multi-master cluster, different steps may need to be run on different nodes of the cluster. In a primary-standby environment, the steps may need to be run on the primary and standby nodes.

Rotating Expired Platform Certificates on a Standalone Oracle Key Vault Server

Rotate platform certificates on a standalone Oracle Key Vault server to replace expired certificates with new certificates.

  1. SSH into the Oracle Key Vault system as the support user.

    ssh support@OKV_SERVER_IP_ADDRESS

  2. Switch to user root:

    su - root

  3. Check the validity of the platform certificates that are used when adding new nodes to a multi-master cluster or configuring a primary-standby deployment as follows:

    openssl x509 -noout -enddate -in /usr/local/dbfw/etc/ca.crt
openssl x509 -noout -enddate -in /etc/pki/tls/certs/localhost_internal.crt
openssl x509 -noout -enddate -in /usr/local/dbfw/etc/cert.crt
openssl x509 -noout -enddate -in /usr/local/dbfw/etc/avs/avs_apex_client.crt
openssl x509 -noout -enddate -in /etc/pki/tls/certs/localhost.crt

    If the dates indicate that any of the certificates have expired (as indicated by the platform certificate alerts), you must rotate the certificate by performing the following steps:

    • For Oracle Key Vault systems at release 21.1, 21.2, 21.3, 21.4, and 21.5, download the Oracle Audit Vault and Database Firewall patch for bug 34378212. SCP the downloaded zip file to the Oracle Key Vault server and unzip the file:

      scp <downloaded_patch_file> support@OKV_SERVER_IP_ADDRESS:/tmp
ssh support@OKV_SERVER_IP_ADDRESS
su - root
cd /tmp
unzip <zip_file_for_patch_34378212> -d /tmp
mkdir /root/gensslcert
cp /tmp/gensslcert.avs.tar.gz /root/gensslcert
cd /root/gensslcert
tar xvfz gensslcert.avs.tar.gz
export GENSSLCERT_HOME="/root/gensslcert"
env | grep GENSSLCERT_HOME

    • If the Oracle Key Vault system is at release 21.6 or higher, then run the following command:

      export GENSSLCERT_HOME="/usr/local/bin/"
env | grep GENSSLCERT_HOME

    • Run the following command to regenerate the platform certificate used when adding new nodes to a multi-master cluster or configuring a primary-standby deployment:

      "$GENSSLCERT_HOME"/gensslcert destroy-certs create-ca

  4. Restart the necessary services by running the following commands:

    systemctl reload httpd
systemctl restart controller

  5. Update the platform CA certificate bundle, using the following command:

    cat /usr/local/dbfw/etc/ha_partner.crt /usr/local/dbfw/etc/ca.crt > /etc/pki/tls/certs/dbfw-ca.crt
systemctl reload httpd
systemctl restart controller
systemctl start monitor

  6. Check the validity of the platform certificate used for redo shipping by performing the following steps. It is sufficient to check the validity on one multi-master cluster node only.

    su oracle
cd /var/lib/oracle/dbfw/network/admin/avwallet
orapki wallet display -wallet  /var/lib/oracle/dbfw/network/admin/avwallet -complete
orapki wallet export -wallet . -dn "DC=com,CN=avserver,OU=db,O=oracle -cert /tmp/avserver_check_cert_validity.cert
openssl x509 -in /tmp/avserver_check_cert_validity.cert -subject -enddate -noout

    If the certificate has expired, then you must rotate it using the steps starting with Step 8.

  7. Switch back to the root user using the command:

    exit

  8. For Oracle Key Vault releases 21.1 to 21.4, contact Oracle Support for help with rotating the platform redo shipping certificate. For Oracle Key Vault release 21.5 and later, run the following commands to regenerate the platform redo shipping certificate:

    cd /opt/avdf/lib/ruby/avdf
ruby update_agent_cert_task.rb

  9. Run the following commands to verify the validity of the regenerated certificates:

    openssl x509 -startdate -enddate -noout -in /usr/local/dbfw/etc/ca.crt
openssl x509 -startdate -enddate -noout -in /etc/pki/tls/certs/localhost_internal.crt
openssl x509 -startdate -enddate -noout -in /usr/local/dbfw/etc/cert.crt
openssl x509 -startdate -enddate -noout -in /usr/local/dbfw/etc/avs/avs_apex_client.crt
openssl x509 -startdate -enddate -noout -in /etc/pki/tls/certs/localhost.crt

  10. Switch to user oracle to verify the validity of the redo shipping certificate, using the following commands:

    su oracle
cd /var/lib/oracle/dbfw/network/admin/avwallet
orapki wallet display -wallet  /var/lib/oracle/dbfw/network/admin/avwallet -complete
orapki wallet export -wallet . -dn "DC=com,CN=avserver,OU=db,O=oracle -cert /tmp/avserver_check_cert_validity.cert
openssl x509 -in /tmp/avserver_check_cert_validity.cert -subject -startdate -enddate -noout

  11. Switch back to user root using the following command:

    exit

  12. Restart the database service on the system using the following command:

    systemctl restart dbfwdb

  13. Verify the validity of the platform certificates from the Oracle Key Vault management console.

Rotating Expired Platform Certificates in a Multi-Master Cluster Environment

Learn how to rotate platform certificates in a multi-master cluster environment.

Rotating Expired Platform CA Certificate on Read/Write Multi-Master Cluster Nodes

Rotate platform certificates on read/write multi-master cluster nodes to replace expired certificates with new ones.

In this section, Node A and Node B refer to the two nodes of a given read/write pair. Implement these steps on each set of read/write pairs in turn.

  1. (Node A) SSH into Node A as user support:

    ssh support@OKV_NODEA_IP

  2. (Node A) Switch to user root on Node A :

    su - root

  3. Check the validity of the platform certificates that are used when adding new nodes to a multi-master cluster or configuring a primary-standby deployment using the following commands:

    openssl x509 -noout -enddate -in /usr/local/dbfw/etc/ca.crt
openssl x509 -noout -enddate -in /etc/pki/tls/certs/localhost_internal.crt
openssl x509 -noout -enddate -in /usr/local/dbfw/etc/cert.crt
openssl x509 -noout -enddate -in /usr/local/dbfw/etc/avs/avs_apex_client.crt
openssl x509 -noout -enddate -in /etc/pki/tls/certs/localhost.crt

    If the dates indicate that any of the certificates have expired (as indicated by the platform certificate alerts), you must rotate the certificates using the following steps:

    • (Node A)

      • For Oracle Key Vault systems at release 21.1, 21.2, 21.3, 21.4, and 21.5, download the Oracle Audit Vault and Database Firewall patch for bug 34378212. SCP the downloaded zip file to the Oracle Key Vault system and unzip the file using the following commands:

        scp <zip_file_for_patch_34378212> support@OKV_NODEA_IP:/tmp
ssh support@OKV_NODEA_IP
su - root
cd /tmp
unzip <zip_file_for_patch_34378212> -d /tmp
mkdir /root/gensslcert
cp /tmp/gensslcert.avs.tar.gz /root/gensslcert
cd /root/gensslcert
tar xvfz gensslcert.avs.tar.gz
export GENSSLCERT_HOME=/root/gensslcert
env | grep GENSSLCERT_HOME

      • If the Oracle Key Vault system is at release 21.6 or higher, then run the following command:

        export GENSSLCERT_HOME="/usr/local/bin/"
env | grep GENSSLCERT_HOME

      • Run the following command on Node A (at release 21.1 or higher) to regenerate the platform certificate used when adding new nodes to a multi-master cluster or configuring a primary-standby deployment:

        "$GENSSLCERT_HOME"/gensslcert destroy-certs create-ca
systemctl reload httpd
systemctl restart controller

  4. (Node A) Transfer a copy of the regenerated certificate to Node B using SCP:

    scp /usr/local/dbfw/etc/ca.crt support@OKV_NODEB_IP:/tmp/ha_partner.crt

  5. (Node B)

    • For Oracle Key Vault systems at release 21.1, 21.2, 21.3, 21.4, and 21.5, download the Oracle Audit Vault and Database Firewall patch for bug 34378212. SCP the downloaded zip file to the Oracle Key Vault system and unzip the file using the following commands:

      scp <zip_file_for_patch_34378212> support@OKV_NODEB_IP:/tmp
ssh support@OKV_NODEB_IP
su - root
cd /tmp
unzip <zip_file_for_patch_34378212> -d /tmp
mkdir /root/gensslcert
cp /tmp/gensslcert.avs.tar.gz /root/gensslcert
cd /root/gensslcert
tar xvfz gensslcert.avs.tar.gz
export GENSSLCERT_HOME=/root/gensslcert
env | grep GENSSLCERT_HOME

    • If the Oracle Key Vault system is at release 21.6 or higher, then run the following command:

      export GENSSLCERT_HOME="/usr/local/bin/"
env | grep GENSSLCERT_HOME

  6. (Node B) Run the following commands on Node B:

    cp /tmp/ha_partner.crt /usr/local/dbfw/etc/ha_partner.crt
systemctl reload httpd
systemctl restart controller
"$GENSSLCERT_HOME"/gensslcert destroy-certs create-ca
systemctl reload httpd
systemctl restart controller

  7. (Node B) Transfer a copy of the regenerated certificate to Node A using SCP:

    scp /usr/local/dbfw/etc/ca.crt support@OKV_NODEA_IP:/tmp/ha_partner.crt

  8. (Node A) SSH into Node A as the support user, then switch to user root:

    ssh support@OKV_NODEA_IP
su - root

  9. (Node A) Run the following commands on Node A:

    cp /tmp/ha_partner.crt /usr/local/dbfw/etc/ha_partner.crt
systemctl reload httpd
systemctl restart controller
cat /usr/local/dbfw/etc/ha_partner.crt /usr/local/dbfw/etc/ca.crt > /etc/pki/tls/certs/dbfw-ca.crt
systemctl reload httpd
systemctl restart controller
systemctl restart monitor

  10. (Node B) SSH into Node B as the support user, then switch to user root:

    ssh support@OKV_NODEB_IP
su - root

  11. (Node B) Run the following commands on Node B:

    cat /usr/local/dbfw/etc/ha_partner.crt /usr/local/dbfw/etc/ca.crt > /etc/pki/tls/certs/dbfw-ca.crt
systemctl reload httpd
systemctl restart controller
systemctl restart monitor

    Repeat steps 1 to 11 on each pair of read/write nodes in the multi-master cluster.

Rotating Expired Platform CA Certificate on Read-Only Multi-Master Cluster Nodes

Learn how to rotate platform certificates on each read-only Oracle Key Vault multi-master cluster node using the steps described in this topic.

  1. SSH in to the node as user support.

    ssh support@OKV_NODE_IP

  2. Switch to user root.

    su - root

  3. Check the validity of the platform certificates that are used when adding new nodes to a multi-master cluster or configuring a primary-standby deployment using the following commands:

    openssl x509 -noout -enddate -in /usr/local/dbfw/etc/ca.crt
openssl x509 -noout -enddate -in /etc/pki/tls/certs/localhost_internal.crt
openssl x509 -noout -enddate -in /usr/local/dbfw/etc/cert.crt
openssl x509 -noout -enddate -in /usr/local/dbfw/etc/avs/avs_apex_client.crt
openssl x509 -noout -enddate -in /etc/pki/tls/certs/localhost.crt

  4. For Oracle Key Vault release 21.1 to release 21.5, download the Oracle Audit Vault and Database Firewall patch for bug 34378212. SCP the downloaded zip file to the Oracle Key Vault server and unzip the file using the following commands:

    scp <zip_file_for_patch_34378212> support@OKV_NODE_IP_ADDRESS:/tmp
ssh support@OKV_NODE_IP_ADDRESS
su - root
cd /tmp
unzip <zip_file_for_patch_34378212> -d /tmp
mkdir /root/gensslcert
cp /tmp/gensslcert.avs.tar.gz /root/gensslcert
cd /root/gensslcert
tar xvfz gensslcert.avs.tar.gz
    export GENSSLCERT_HOME="/root/gensslcert"
env | grep GENSSLCERT_HOME

    • If the Oracle Key Vault system is at release 21.6 or later, then run the following commands:

      export GENSSLCERT_HOME="/usr/local/bin/"
env | grep GENSSLCERT_HOME

  5. Run the following command to regenerate the platform certificate used when adding new nodes to a multi-master cluster or configuring a primary-standby deployment:

    "$GENSSLCERT_HOME"/gensslcert destroy-certs create-ca

  6. Reload and restart the httpd and controller services:

    systemctl reload httpd
systemctl restart controller

  7. Update the platform CA certificate bundle:

    cat /usr/local/dbfw/etc/ha_partner.crt /usr/local/dbfw/etc/ca.crt > /etc/pki/tls/certs/dbfw-ca.crt
systemctl reload httpd
systemctl restart controller
systemctl start monitor

    Repeat these steps on each read-only node in the multi-master cluster.

Rotating Platform Certificates Used For Redo Shipping on a Node in a Multi-Master Cluster

In a multi-master cluster environment, if the platform certificate for redo shipping has expired, you must rotate it on one node of the cluster, and then transfer the certificate to all nodes in the cluster.

Follow these steps to check the expiration date of the redo shipping platform certificate and to rotate the certificate if required.

  1. Choose one node of the multi-master cluster on which to rotate the redo shipping platform certificate.

  2. SSH into the Oracle Key Vault node as the support user.

    ssh support@OKV_NODE_IP_ADDRESS

  3. Switch to user root.

    su - root

  4. Check the validity of the platform certificate used for redo shipping using the following commands. It is sufficient to check this on one multi-master cluster node only.

    su oracle
cd /var/lib/oracle/dbfw/network/admin/avwallet
orapki wallet display -wallet  /var/lib/oracle/dbfw/network/admin/avwallet -complete
orapki wallet export -wallet . -dn "DC=com,CN=avserver,OU=db,O=oracle" -cert /tmp/avserver_check_cert_validity.cert
openssl x509 -in /tmp/avserver_check_cert_validity.cert -subject -enddate -noout

    If the certificate has expired, then you must rotate it using the subsequent steps.

  5. Switch back to the user root using the following command:

    exit

  6. Rotate the platform certificate used for redo shipping using the following commands, still as user root:

    cd /opt/avdf/lib/ruby/avdf
ruby update_agent_cert_task.rb

  7. Switch to user oracle to verify the validity of the certificate, using the following commands:

    su oracle
cd /var/lib/oracle/dbfw/network/admin/avwallet
orapki wallet display -wallet  /var/lib/oracle/dbfw/network/admin/avwallet -complete
orapki wallet export -wallet . -dn "DC=com,CN=avserver,OU=db,O=oracle" -cert /tmp/avserver_check_cert_validity.cert
openssl x509 -in /tmp/avserver_check_cert_validity.cert -subject -startdate -enddate -noout

  8. For Oracle Key Vault releases 21.1 to 21.4, contact Oracle Support for help with rotating the platform redo shipping certificate. For Oracle Key Vault release 21.5 and later, run the following commands to regenerate the platform redo shipping certificate:

    cd /opt/avdf/lib/ruby/avdf
ruby update_agent_cert_task.rb

  9. Restart the database service on the system using the following command:

    systemctl restart dbfwdb

  10. Verify the validity of the platform certificates from the Oracle Key Vault management console.

  11. Transfer the rotated redo shipping platform certificate to all other multi-master cluster nodes using the steps detailed in the section Transfer the Rotated Redo Shipping Platform Certificate to Other Multi-Master Cluster Nodes.

Transfer the Rotated Redo Shipping Platform Certificate to Other Multi-Master Cluster Nodes

Learn how to transfer the redo shipping platform certificate to other multi-master cluster node after rotating it on one multi-master cluster node.

  1. As the support user, SSH into the Oracle Key Vault node on which the redo shipping platform certificate was rotated.

    ssh support@OKV_NODE_IP_ADDRESS

  2. Switch to user root.

    su - root

  3. Create a bundle containing the rotated platform certificate using the following steps:

    cd /var/lib/oracle/dbfw/network/admin/avwallet
zip -r avwallet_regenerated.zip *

  4. Use SCP to copy the bundle to all other multi-master cluster nodes.

    scp avwallet_regenerated.zip support@<OKV_OTHER_CLUSTER_NODE_IP_ADDRESS>:/tmp

  5. Repeat the following set of steps on each of the other multi-master cluster nodes:

    1. SSH (as the support user) in to the Oracle Key Vault node to which the regenerated redo shipping certificate was transferred.

      ssh support@OKV_OTHER_CLUSTER_NODE_IP_ADDRESS

    2. Switch to user root.

      su - root

    3. Unzip the avwallet_regenerated zip file that was copied using SCP using the following command:

      cd /tmp
unzip avwallet_regenerated.zip -d avwallet_regenerated
chmod -R 775 avwallet_regenerated

    4. Take a backup of the old platform certificate using the following command:

      cd /var/lib/oracle/dbfw/network/admin
cp -Rp avwallet avwallet_expired

    5. Copy the new wallet, ensuring to preserve permissions :

      cp /tmp/avwallet_regenerated/* avwallet/

    6. Take a backup of the /var/lib/oracle/dbfw/dbs/orapwdbfwdb file, ensuring that you preserve permissions:

      /bin/cp -p /var/lib/oracle/dbfw/dbs/orapwdbfwdb /var/lib/oracle/okv_orapwd_backup_dir/orapwdbfwdb_$RANDOM

    7. Replace /var/lib/oracle/dbfw/dbs/orapwdbfwdb and the backup file with the new file that is just copied from the other node: /bin/cp /tmp/orapwdbfwdb /var/lib/oracle/dbfw/dbs/orapwdbfwdb.

      /bin/cp -p /var/lib/oracle/dbfw/dbs/orapwdbfwdb /var/lib/oracle/okv_orapwd_backup_dir/orapwdbfwdb_$RANDOM

    8. Restart services using the following commands:

      service dbfwlistener restart
service dbfwdb restart

    9. Switch to user oracle to verify the validity of the certificate, using the following commands:

      su oracle
cd /var/lib/oracle/dbfw/network/admin/avwallet
orapki wallet display -wallet  /var/lib/oracle/dbfw/network/admin/avwallet -complete
orapki wallet export -wallet . -dn "DC=com,CN=avserver,OU=db,O=oracle" -cert /tmp/avserver_check_cert_validity.cert
openssl x509 -in /tmp/avserver_check_cert_validity.cert -subject -startdate -enddate -noout

    Verify the validity of the platform certificates from the Oracle Key Vault management console.