Deploying Oracle DBSecCentral in High Availability Configuration Using Oracle Real Application Clusters (Oracle RAC)

Oracle Real Application Clusters (Oracle RAC) is a “shared-everything” database architecture in which multiple instances run on different servers (nodes) while accessing a single set of database files on shared storage. This design provides high availability, horizontal scalability, and fault tolerance for mission-critical applications. Oracle DBSecCentral uses Oracle RAC to support high availability.

Oracle DBSecCentral uses Oracle RAC to support high availability by ensuring that DBSecCentral services and the database remain continuously accessible, even if one node fails. With a two-node RAC deployment, users benefit from automatic failover. In the event that one node becomes unavailable due to hardware, software, or maintenance activities, the other node can seamlessly take over without disrupting service. In addition to high availability, deploying Oracle DBSecCentral on RAC provides performance improvements by allowing workloads to be distributed across both nodes. This balanced processing can reduce bottlenecks during periods of heavy database activity, improving response times for audit collection and reporting operations. Together, these features help organizations achieve greater reliability, data protection, and enhanced performance in their DBSecCentral environments.

You must reconfigure the regular standalone version of Audit Vault Server to enable Oracle RAC on Oracle DBSecCentral.

Deploying Oracle DBSecCentral on Oracle RAC is limited to a two-node deployment. Deployments using only one RAC-enabled node should be used for testing purposes only and are not recommended for production environments.

When Oracle DBSecCentral is deployed on RAC in high availability, certain setting and configuration options are not available in the administrator’s Audit Vault Server. See UI Restrictions and Workarounds to learn more.

Tip: You should be familiar with general Oracle RAC deployments before continuing with an Oracle DBSecCentral RAC deployment. To learn more about Oracle RAC see Overview of Oracle RAC and Clusterware Best Practices guide.

RAC Networking Preparation

When deploying Oracle DBSecCentral on Oracle RAC, it is recommended to plan for and note the following networking and architecture information.

Related Topics

RAC Prerequisites

Ensure you set up and complete the following prerequisites before deploying Oracle DBSecCentral on Oracle RAC.

Related Topics

RAC Deployment Steps

Complete the following steps to deploy Oracle DBSecCentral in a high availability configuration on Oracle RAC. Before converting to a RAC deployment, complete all prerequisites and complete a standalone installation of Oracle DBSecCentral.

Ensure that you have reviewed the following sections before you begin:

To deploy Oracle DBSecCentral in a high availability configuration on Oracle RAC:

  1. On both hosts, follow the standalone AVS installation process.
  1. Complete this step only if the appliance requires migration to suitable shared storage.
2. Following the successful installation of a standalone instance of Oracle DBSecCentral, if you need to rebalance the existing storage onto new shared storage, move the 2 appliance to the SAN by running the following command as the `root` user on the first node:

    <pre class="text copy"><code>/opt/avdf/config-utils/bin/config-asm \ rebalance \ eventdata=/dev/mapper/eventdata_SAN_path \ systemdata=/dev/mapper/systemdata_SAN_path \ recovery=/dev/mapper/recovery_SAN_path \ confirm \ wait</code></pre>

    The arguments of ` /opt/avdf/config-utils/bin/config-asm ` are:

    - rebalance - Instruct the CLI to rebalance the ASM, moving the disk groups from their current location, to the specified location.

    - eventdata - A comma-separated list (no spaces or use quotes) of block devices to use for the disk group.

    - systemdata - A comma-separated list (no spaces or use quotes) of block devices to use for the disk group.

    - recovery - A comma-separated list (no spaces or use quotes) of block devices to use for the disk group.

    - confirm - The CLI will dry run the options if this flag is not present.

    - wait - The CLI will not exit until the (ASM controlled) estimation of the rebalancing runtime is complete. Note that this is an ETA, not factual, be careful about performing additional disk operations before confirmation.

      <div class="infoboxtip" markdown="1">

      **Tip:**

      ASM can be extended by running the command again, specifying the block devices that are currently being used as well as new block devices.

      Devices can be removed and extended by specifying devices to add, leaving out devices to remove.

      </div>

  1. If you are enabling FIPS, see Enabling FIPS 140-2 in Oracle DBSecCentral.

  2. Test and then move the DBSecCentral appliance to RAC by running the following as the root user on the first node:

    For testing, run the command with the test_run argument while keeping all other arguments the same. This allows you to validate the process without making significant runtime changes.

    /opt/avdf/config-utils/bin/config-rac_cluster first_node\
test_run \
rac_cluster_name=AVDFCLUSTER \
rac_instance_name=AVDFRAC1 \
rac_device=eth0 \
rac_scan_device=bond0 \
rac_scan_name=avdfcluster.db.com \
rac_scan_port=1522 \
rac_vip_address=avdfrac1-vip.db.com \
racdata_dg=/dev/mapper/racdata_SAN_path

    Once testing is complete and error-free, run the command to move the first RAC node.

    /opt/avdf/config-utils/bin/config-rac_cluster first_node\
rac_cluster_name=AVDFCLUSTER \
rac_instance_name=AVDFRAC1 \
rac_device=eth0 \
rac_scan_device=bond0 \
rac_scan_name=avdfcluster.db.com \
rac_scan_port=1522 \
rac_vip_address=avdfrac1-vip.db.com \
racdata_dg=/dev/mapper/racdata_SAN_path

    The arguments of /opt/avdf/config-utils/bin/config-rac_cluster are:

    • rac_cluster_name - The name this cluster will use, this is a stand-alone cluster.

    • rac_instance_name - The name this node will assume in the cluster, this must be unique for each cluster node.

    • rac_device - The name of the device that will be used for the RAC interconnect.

    • rac_scan_device - The name of the network device used for SCAN traffic, usually the same as the default network device.

    • rac_scan_name - The DNS name for the SCAN.

    • rac_scan_port - The port number used by the SCAN listener.

    • rac_vip_address - The VIP address for this node.

    • racdata_dg - A comma-separated list of disks that will be used to create the RACDATA disk group.

  3. Register the second node with the cluster by running the following command as the root user on the first node:

    /opt/avdf/config-utils/bin/config-rac_cluster register_node \
peer_ip=IP address of the new node to register \
peer_ca=CA certificate of new node

    For example,

    /opt/avdf/config-utils/bin/config-rac_cluster register_node \
peer_ip=192.168.62.15 \
peer_ca=- <<EOF-----BEGIN CERTIFICATE-----.........-----END CERTIFICATE-----EOF

    The arguments of /opt/avdf/config-utils/bin/config-rac_cluster register_node are:

    • peer_ip - The address of the new node to register.

    • peer_ca - The CA certificate of the new node (/usr/local/dbfw/etc/ca.crt).

  4. Test and then add the second node to the cluster by running the following command as the root user on the new node:

    For testing, run the command with the test_run argument while keeping all other arguments the same. This allows you to validate the process without making significant runtime changes.

    /opt/avdf/config-utils/bin/config-rac_cluster new_node \
test_run \
rac_instance_name=AVDFRAC2 \
rac_device=eth0 \
rac_scan_device=bond0 \
rac_vip_address=avdfrac2-vip.db.com \
peer_ip=IP address of the new node to register \
peer_ca=CA certificate of new node

    Once testing is complete and error-free, run the command to add the second node to the cluster.

    /opt/avdf/config-utils/bin/config-rac_cluster new_node \
rac_instance_name=AVDFRAC2 \
rac_device=eth0 \
rac_scan_device=bond0 \
rac_vip_address=avdfrac2-vip.db.com \
peer_ip=IP address of the new node to register \
peer_ca=CA certificate of new node

    For example,

    /opt/avdf/config-utils/bin/config-rac_cluster new_node \
rac_instance_name=rac2 \
rac_device=enp0s9 \
rac_scan_device=enp0s8 \
rac_vip_address=rac-startup-2-vip.vboxlocal.net \
peer_ip=192.168.62.10 \
peer_ca=- <<EOF-----BEGIN CERTIFICATE-----.........-----END CERTIFICATE-----EOF

    The arguments of /opt/avdf/config-utils/bin/config-rac_cluster new_node are:

    • rac_instance_name - The name this node will assume in the cluster, this must be unique for each cluster node.

    • rac_device - The name of the device that will be used for the RAC interconnect.

    • rac_scan_device - The name of the device that will be used to host the SCAN, usually the same as the default device.

    • rac_vip_address - The VIP address for this node.

    • racdata_dg - A comma-separated list of disks that will be used to create the RACDATA disk group.

    • peer_ip - The address of the new node to register.

    • peer_ca - The CA certificate of the new node (/usr/local/dbfw/etc/ca.crt).

  5. On the first RAC node, remove the old logical volumes holding the ASM disks that are used during the default installation.

    Run the following commands:

    oracleasm deletedisk /dev/mapper/vg_root-lv_systemdata1 ;\
oracleasm deletedisk /dev/mapper/vg_root-lv_eventdata1 ;\
oracleasm deletedisk /dev/mapper/vg_root-lv_recovery1
lvremove -v -y /dev/mapper/vg_root-lv_systemdata1
lvremove -v -y /dev/mapper/vg_root-lv_eventdata1
lvremove -v -y /dev/mapper/vg_root-lv_recovery1

    Once the two node RAC deployment is complete, you can use either web UI address to access the cluster.

Related Topics

UI Restrictions and Workarounds

When deployed in high availability on Oracle RAC, certain settings and configurations are not accessible in the administrator’s Oracle Database Security Central console. You can use the corresponding AVCLI commands when they’re not available in the console.

DNS Settings

To configure DNS for RAC on AVS, run the following commands on each AVS:

/opt/avdf/config-utils/bin/config-dns set servers="a.b.c.d,a.b.c.d,a.b.c.d"

All DNS servers listed must contain the same information. If applying the RAC configuration to the available DNS server is not possible, you can set local overrides to the hosts file on each appliance. When updating the hosts file, ensure the changes are made to both the hosts file and the hosts file template:

You may also need to replicate changes made to the hosts file on other systems as well (agent hosts) which may generate significant administrative overhead. It is advisable to ensure that the new records can be added to the appropriate DNS servers.

NTP Settings

To configure NTP settings for RAC on AVS, run the following command on each AVS:

/opt/avdf/config-utils/bin/config-ntp set \
  servers="a.b.c.d,a.b.c.d,a.b.c.d" \
  enabled=true \
  sync_on_save=true \

Syslog Settings

To configure Syslog settings for RAC on AVS, run the following command on each AVS:

/opt/avdf/config-utils/bin/config-syslog set \
  udp_destinations="a.b.c.d,a.b.c.d,a.b.c.d" \
  tcp_destinations="a.b.c.d,a.b.c.d,a.b.c.d" \
  categories="system,alerts,info,debug,heartbeat" \
  max_message_length=1024 \

Enable or Disable FIPS

To enable FIPS for RAC on AVS, run the following commands:

  1. To stop the cluster on all nodes, run the following command on one node:

    /u01/app/23.26.0.0.0/grid/bin/crsctl stop cluster -all

  2. To enable FIPS, run the following command on all nodes:

    /opt/avdf/config-utils/bin/config-fips set\
  enabled=true \
  reboot_required=true

  3. Reboot servers after gracefully shutting down CRS:

    systemctl isolate multi-user
/u01/app/26.3.0.0.0/grid/bin/crsctl stop crs
reboot now

To disable FIPS, repeat the steps above, ensuring you set enabled=false in Step 2.

Configure SSO

To configure Single sign on (SSO) for RAC on AVS:

  1. Identify the node whose /usr/local/dbfw/etc/dbfw.conf shows the appropriate IdP domain. This is your source.

    grep '^SSO_DOMAIN' /usr/local/dbfw/etc/dbfw.conf

  2. On the non-source server, update dbfw.conf to use that same domain (replace IDP_DOMAIN_FROM_SOURCE).

    sudo -u dbfw /usr/local/dbfw/bin/dbfwconf_editor.rb /usr/local/dbfw/etc/dbfw.conf SSO_DOMAIN='IDP_DOMAIN_FROM_SOURCE'

  3. On each server (source first), back up the local SAML files.

    cp /usr/local/dbfw/etc/apexsaml.crt /usr/local/dbfw/etc/apexsaml.crt.bak
    cp /usr/local/dbfw/etc/apexsaml.key /usr/local/dbfw/etc/apexsaml.key.bak

  4. From the source server, copy the known-good files to the other node.

    scp /usr/local/dbfw/etc/apexsaml.crt support@OTHER_HOST:/tmp
    scp /usr/local/dbfw/etc/apexsaml.key support@OTHER_HOST:/tmp

  5. On the non-source server, copy the temp files to the appropriate destination.

    cp /tmp/apexsaml.crt /usr/local/dbfw/etc/
    cp /tmp/apexsaml.key /usr/local/dbfw/etc/

  6. On each server, normalize ownership and permissions.

    chown root:dbfwcerts /usr/local/dbfw/etc/apexsaml.crt /usr/local/dbfw/etc/apexsaml.key
    chmod 460 /usr/local/dbfw/etc/apexsaml.crt /usr/local/dbfw/etc/apexsaml.key

  7. On the non-source server only, refresh Apache to reload the SSO configuration.

    /opt/avdf/bin/configure-networking run-only=apache

Configure LDAP

To configure Lightweight Directory Access Protocol (LDAP) for RAC on AVS:

  1. Identify the source node that already talks to LDAP (wallet populated):

    ls -l /usr/local/dbfw/etc/wallets/ldap_wallet

  2. On that source node, archive the working wallet directory:

    tar czf /tmp/ldap_wallet.tar.gz -C /usr/local/dbfw/etc/wallets ldap_wallet

  3. Copy the archive to the other node:

    scp /tmp/ldap_wallet.tar.gz support@OTHER_HOST:/tmp/

  4. On the other node, back up any existing wallet:

    mv /usr/local/dbfw/etc/wallets/ldap_wallet /usr/local/dbfw/etc/wallets/ldap_wallet.bak.$(date +%Y%m%d-%H%M%S)

  5. Still on the other node, unpack the known-good wallet:

    tar xzf /tmp/ldap_wallet.tar.gz -C /usr/local/dbfw/etc/wallets

    Verify that cwallet.sso and ewallet.p12 are located in /usr/local/dbfw/etc/wallets/ldap_wallet.

  6. Enforce the expected ownership and permissions:

    chown -R oracle:oinstall /usr/local/dbfw/etc/wallets/ldap_wallet
    chmod 700 /usr/local/dbfw/etc/wallets/ldap_wallet
    chmod 600 /usr/local/dbfw/etc/wallets/ldap_wallet/*

  7. (Optional) Clean up the transfer archive on both nodes:

    rm -f /tmp/ldap_wallet.tar.gz

Update the UI Console Certificate

To update the UI console certificate for RAC on AVS:

  1. Create the UI certificate in the UI.

  2. Identify the node whose UI presents the correct certificate. Run the following command on each node to locate the new UI cert and key:

    openssl x509 -noout -subject -issuer -dates -in /etc/pki/tls/certs/localhost.crt

  3. On that source node, package the live cert/key and their saved copies:

    tar czf /tmp/ui_console_cert.tgz -C /etc/pki/tls certs/user_uploaded_ui.crt certs/localhost.crt private/user_uploaded_ui.key private/localhost.key

  4. Copy the archive to the other server:

    scp /tmp/ui_console_cert.tgz support@OTHER_HOST:/tmp/

  5. On the destination, back up the current HTTPS cert and key:

    cp /etc/pki/tls/certs/localhost.crt /etc/pki/tls/certs/localhost.crt.bak.$(date +%Y%m%d-%H%M%S)
    cp /etc/pki/tls/private/localhost.key /etc/pki/tls/private/localhost.key.bak.$(date +%Y%m%d-%H%M%S)

  6. Extract the transferred files:

    tar xzf /tmp/ui_console_cert.tgz -C /etc/pki/tls

  7. Set the appropriate permissions:

    chown root:root /etc/pki/tls/private/localhost.key /etc/pki/tls/private/user_uploaded_ui.key
    chmod 600 /etc/pki/tls/private/localhost.key /etc/pki/tls/private/user_uploaded_ui.key
    chown root:root /etc/pki/tls/certs/localhost.crt /etc/pki/tls/certs/user_uploaded_ui.crt
    chmod 644 /etc/pki/tls/certs/localhost.crt /etc/pki/tls/certs/user_uploaded_ui.crt

  8. Reload Apache on the destination so the UI picks up the new certificate:

    systmectl restart httpd

  9. Verify both servers present identical details.

    openssl x509 -noout -subject -issuer -dates -in /etc/pki/tls/certs/localhost.crt

Rotate Certificates

Follow the appropriate instructions to rotate certificates for RAC on AVS.

Start certificate rotation on AVS appliances

To start the certificate rotation process on on AVS appliances:

  1. Define shell vars for each session:

    AVS1=<AVS1-ip-address>
AVS2=<AVS2-ip-address>
DBFW1=<DBFW1-ip-address>

    Ensure that you replace the IP address placeholders with the IP addresses of each AVS and DBFW.

  2. Rotate on each AVS and DBFW (do not restart any services yet):

    root $> /usr/local/bin/gensslcert --create-ca --no-service-restart > /tmp/gensslcert.log 2>&1

  3. Share CA between AVS and run configure-networking:

    1. As the root user on AVS1, run the following commands:

      cp /usr/local/dbfw/etc/ca.crt /opt/avdf/grid/cluster-ca/avs-$AVS1-ca.crt
      sudo -u grid scp /opt/avdf/grid/cluster-ca/avs-$AVS1-ca.crt $AVS2:/opt/avdf/grid/cluster-ca/avs-$AVS1-ca.crt.tmp
      sudo -u grid ssh $AVS2 mv /opt/avdf/grid/cluster-ca/avs-$AVS1-ca.crt.tmp /opt/avdf/grid/cluster-ca/avs-$AVS1-ca.crt

    2. As the root user on AVS2, run the following commands:

      cp /usr/local/dbfw/etc/ca.crt /opt/avdf/grid/cluster-ca/avs-$AVS2-ca.crt
      sudo -u grid scp /opt/avdf/grid/cluster-ca/avs-$AVS2-ca.crt $AVS1:/opt/avdf/grid/cluster-ca/avs-$AVS2-ca.crt.tmp
      sudo -u grid ssh $AVS1 mv /opt/avdf/grid/cluster-ca/avs-$AVS2-ca.crt.tmp /opt/avdf/grid/cluster-ca/avs-$AVS2-ca.crt

  4. As the root user, run configure-networking and restart Apache on each AVS:

    /opt/avdf/bin/configure-networking
    systemctl restart httpd

Start certificate rotatation on DBFW

To rotate cerificates on AVS appliances for RAC:

  1. As the root user, run the following command to disable the DBFW certificate synchronization on each AVS:

    systemctl stop dbfw-certs.path

  2. Share the CA from the cluster to DBFW.

    1. As the root user on AVS1, run the following command:

      /opt/avdf/config-utils/bin/config-rac_cluster register_dbfw

    2. Run the output from command run in the previous step once on each DBFW.

  3. As the root user on DBFW, run configure-networking and restart Apache:

    /opt/avdf/bin/configure-networking
    systemctl restart httpd

  4. Share each DBFW CA to the cluster.

    1. On DBFW, copy the CA to each AVS:
    scp /usr/local/dbfw/etc/ca.crt support@$AVS1:/tmp/fw-$DBFW1.crt

    1. On AVS1, prepare the fwcerts directory:

      mkdir /usr/local/dbfw/etc/avs/fwcerts.old/
      mv /usr/local/dbfw/etc/avs/fwcerts/* /usr/local/dbfw/etc/avs/fwcerts.old/

    2. Update the AVS1 fwcerts for each DBFW CA:

      cp /tmp/fw-$DBFW1.crt /usr/local/dbfw/etc/avs/fwcerts/
      ln -s /usr/local/dbfw/etc/avs/fwcerts/fw-$DBFW1.crt \
      /usr/local/dbfw/etc/avs/fwcerts/$(openssl x509 -hash -noout -in /usr/local/dbfw/etc/avs/fwcerts/fw-$DBFW1.crt).0

    3. Update the AVS1 wallet with each DBFW CA:

      su - oracle -c \
    "/var/lib/oracle/dbfw/bin/avca import_cert -wrl \
    /usr/local/dbfw/etc/avs/avswallet/ \
    -cert /usr/local/dbfw/etc/avs/fwcerts/fw-$DBFW1.crt -trusted"
  5. On AVS2, clear the fwcerts:</code></pre>text mkdir /usr/local/dbfw/etc/avs/fwcerts.old/ <pre class="copy"></pre>text mv /usr/local/dbfw/etc/avs/fwcerts/* /usr/local/dbfw/etc/avs/fwcerts.old/ <pre class="copy">
  6. Run the following commands to enable DBFW certificate synchronization on each AVS:</code></pre>text systemctl start dbfw-certs.path
  7. As the root user, transfer the DBFW CAs to AVS2:</code></pre>text /opt/avdf/bin/rac-dbfw-certs <pre class="copy">

    Start certificate rotation for agents

    The following commands update the Agent wallet on each AVS.

  8. Run the following commands to create the new wallet:</code></pre>text rm -Rf /var/lib/oracle/dbfw/network/admin/avwallet_agent/network/admin/avwallet/

    text su - oracle -c
    “/var/lib/oracle/dbfw/bin/avca create_wallet
    -wrl /var/lib/oracle/dbfw/network/admin/avwallet_agent/network/admin/avwallet”

    text openssl x509 -noout -subject -in /usr/local/dbfw/etc/ca.crt

    text su - oracle -c ‘orapki wallet display
    -wallet /var/lib/oracle/dbfw/network/admin/avwallet_agent/network/admin/avwallet/’
    | grep Subject
  9. Run the following commands to add the CA for both the AVS to the wallet:</code></pre>text rm -f /tmp/avs-$AVS1-ca.crt rm -f /tmp/avs-$AVS2-ca.crt

    text sudo -u oracle cp /opt/avdf/grid/cluster-ca/avs-$AVS1-ca.crt /tmp sudo -u oracle cp /opt/avdf/grid/cluster-ca/avs-$AVS2-ca.crt /tmp

    text su - oracle -c
    “/var/lib/oracle/dbfw/bin/avca import_cert -wrl
    /var/lib/oracle/dbfw/network/admin/avwallet_agent/network/admin/avwallet/
    -cert /tmp/avs-$AVS1-ca.crt -trusted”

    text su - oracle -c
    “/var/lib/oracle/dbfw/bin/avca import_cert -wrl
    /var/lib/oracle/dbfw/network/admin/avwallet_agent/network/admin/avwallet/
    -cert /tmp/avs-$AVS2-ca.crt -trusted”

    text openssl x509 -noout -subject -in /usr/local/dbfw/etc/ca.crt

    text su - oracle -c
    “orapki wallet display
    -wallet /var/lib/oracle/dbfw/network/admin/avwallet_agent/network/admin/avwallet/”
    | grep Subject

    text su - oracle -c “/var/lib/oracle/dbfw/bin/avca configure_bootstrap”

    
#### Restart services {#GUID-78328400-D1DC-4EF4-B980-49A6F9EFEB3E}

To finalize the rotation process, restart services:
  10. On each AVS, run the following commands:</code></pre>text systemctl stop javafwk controller

    text systemctl restart httpd rac-controller rac-jfwk
  11. On each DBFW, run the following command:</code></pre>text systemctl restart stund httpd ```

SMTP setup

To set up and manage SMTP email notifications for RAC on AVS, use the SMTP Connection AVCLI commands.