Deploying Oracle DBSecCentral in High Availability Configuration Using Oracle Real Application Clusters (Oracle RAC)

Oracle Real Application Clusters (Oracle RAC) is a “shared-everything” database architecture in which multiple instances run on different servers (nodes) while accessing a single set of database files on shared storage. This design provides high availability, horizontal scalability, and fault tolerance for mission-critical applications. Oracle DBSecCentral uses Oracle RAC to support high availability.

Oracle DBSecCentral uses Oracle RAC to support high availability by ensuring that DBSecCentral services and the database remain continuously accessible, even if one node fails. With a two-node RAC deployment, users benefit from automatic failover. In the event that one node becomes unavailable due to hardware, software, or maintenance activities, the other node can seamlessly take over without disrupting service. In addition to high availability, deploying Oracle DBSecCentral on RAC provides performance improvements by allowing workloads to be distributed across both nodes. This balanced processing can reduce bottlenecks during periods of heavy database activity, improving response times for audit collection and reporting operations. Together, these features help organizations achieve greater reliability, data protection, and enhanced performance in their DBSecCentral environments.

You must reconfigure the regular standalone version of Audit Vault Server to enable Oracle RAC on Oracle DBSecCentral.

Deploying Oracle DBSecCentral on Oracle RAC is limited to a two-node deployment. Deployments using only one RAC-enabled node should be used for testing purposes only and are not recommended for production environments.

When Oracle DBSecCentral is deployed on RAC in high availability, certain setting and configuration options are not available in the administrator’s Audit Vault Server. See UI Restrictions and Workarounds to learn more.

Tip: You should be familiar with general Oracle RAC deployments before continuing with an Oracle DBSecCentral RAC deployment. To learn more about Oracle RAC see Overview of Oracle RAC and Clusterware Best Practices guide.

RAC Networking Preparation

When deploying Oracle DBSecCentral on Oracle RAC, it is recommended to plan for and note the following networking and architecture information.

Create a network diagram of the services you want to provide.
- Where are the Audit Vault Server nodes located?
- Where are the Database Firewall nodes located?
- What hosts will the agents/host monitor be running on?
- Where is the shared storage located?
List the external devices you want to connect to the RAC.
- What gateways or static routes will I use to establish connections between the network endpoints?
  - RAC Nodes
    - Single Client Access Name (SCAN)
    - Management
    - Storage
  - Database Firewalls
    - Proxy
    - Management
    - Mirroring
  - Agents
  - Host Monitor
List the RAC node virtual/hardware you are using. Both nodes must have exactly the same specifications.
- CPUs per Node
- RAM per Node
- Connection to shared storage
- Interconnect network device
- SCAN network device
- Management network device
If the deployment is virtual, is the server dedicated, or does it have multiple workloads running on it?
- What actual spare capacity does the server have?
- What is the expected network performance for the virtual guests?
- What is the expected disk performance for the virtual guests?

Related Topics

Overview of Oracle RAC and Clusterware Best Practices

RAC Prerequisites

Ensure you set up and complete the following prerequisites before deploying Oracle DBSecCentral on Oracle RAC.

Configure a storage area network (SAN) or shared storage for four ASM disk groups.
- Each group can be made from a collection of disks that must all be available to every node that will be included in the cluster.
- You do not need to preconfigure the disk groups with ASM headers.
- You do not need to partition the disks. A single partition per disk is recommended.
  
  Refer to the sizing guide for more information on disk capacity.
  
  The four disk groups are:
  - SYSTEMDATA
  - EVENTDATA
  - RECOVERY
  - RACDATA
  The collection of disks can be presented as a single disk (hardware permitting) or connected individually.
  
  Critical for production deployments: Because DBSecCentral uses EXTERNAL redundancy, ASM does not provide data redundancy. You must configure disk redundancy at the storage layer to ensure data protection and availability.
Two hosts with at least the following configurations:

Note: These specifications are the minimum required for testing and are not recommended for production environments.
- 4 CPUs
- RAM: 32GB
- Hard disk: 400GB
- Network devices for which you can use either of the following recommended setups
  - 2 Network Devices
    - Management and SCAN (database traffic) share the same network device
    - RAC interconnect
  - 3 Network Devices (network separation)
    - Management
    - SCAN (database traffic)
    - RAC interconnect
- Other important recommendations include:
  - The RAC interconnect
    - The network must be a dedicated, high speed network
    - The interconnect network must be fully closed to external traffic
    - A low performance network connection between the nodes will adversely affect the cluster performance
  - The Storage Network
    - A high speed network device for SAN is recommended
    - High data throughput is required for some workloads (may vary by deployment)
    - Hosting the shared storage on networks used for other types of traffic is likely to be detrimental to the system performance
Network time protocol (NTP) must be configured for all hosts
DNS must be accessible from both RAC hosts. The SCAN name must be configured in DNS to resolve to three IP addresses.

DNS is required for proper RAC configuration. If you need to set up RAC without DNS servers available, see Audit Vault Agent Fails To Start in RAC Deployment.
Ensure you meet the general RAC requirements, Server Hardware and Software Review Checklist for Oracle RAC Installation, in the Oracle Real Application Clusters Installation Guide for Linux and UNIX guide.

Related Topics

Overview of Oracle RAC and Clusterware Best Practices

RAC Deployment Steps

Complete the following steps to deploy Oracle DBSecCentral in a high availability configuration on Oracle RAC. Before converting to a RAC deployment, complete all prerequisites and complete a standalone installation of Oracle DBSecCentral.

Ensure that you have reviewed the following sections before you begin:

RAC Networking Preparation
RAC Prerequisites

To deploy Oracle DBSecCentral in a high availability configuration on Oracle RAC:

On both hosts, follow the standalone AVS installation process.
- Fresh install: Install two standalone Audit Vault Servers by following the installation procedure described in Downloading and Installing Oracle Database Security Central. Ensure that both servers meet the minimum hardware requirements described in RAC Prerequisites.
- Upgrade: If you are upgrading to RAC from 20.x, do the following:
  - If the existing deployment is an HA pair, you must first unpair the primary AVS.
  - Upgrade the stand-alone server to the new version following the upgrade guide. See Upgrading from Oracle Audit Vault and Database Firewall to Oracle Database Security Central.
  - Follow the RAC deployment steps that follow.
Tip: It is recommended to migrate to RAC on a fresh system.
Complete this step only if the appliance requires migration to suitable shared storage.

Before continuing, verify that the storage size and performance meets the necessary requirements for your workload and that the shared storage will be accessible to all RAC nodes. Consider that the disk array may be required to write up to the maximum throughput of every data collection network adapter connected to the cluster according to the traffic profile that your collectors generate. For 2 node RAC, this means at least 2 adapters will be collecting data (1 per node).

The instructions in this step are primarily intended to migrate the configured database from the default install to shared storage, but you can also them to migrate older storage to more appropriate hardware, if necessary.
1. You will need to ensure that disks are named appropriately so as to be differentiated from the stand-alone installation naming convention. Specifically, the config-asm command below attempts to give disks unique names to avoid collisions or confusion when attaching new nodes to shared storage.
  
  Ensure that all pre-existing disks allocated to shared storage are named appropriately with:
```
LABEL(SYSTEMDATA\|RECOVERY\|EVENTDATA)UNIQUE-ALPHANUMERIC-STRING
```
  The label must not exceed 24 characters.
  
  Alternatively, you can generate names with the following command that generates a label for a SYSTEMDATA group disk:
```
ruby -I /usr/local/dbfw/lib/ruby \ -r dbfw/grid/asmcmd \ -e "p(DBFW::Grid::Asmcmd.new.create_label(group: 'SYSTEMDATA'))"
```
  If you encounter any of the following issues, you need to fix them before proceeding to the next step:
  - Issue: You have installed shared storage, but have added the storage to the existing disk groups and that now includes the local disks from the initial install and the new disks from the SAN.
    
    Fix: Remove the local disks from the disk group before continuing otherwise parts of the disk group will not be accessible to the second node.
  - Issue: You have installed the SAN correctly and removed the local disks but the names are not unique. For example, using a name such as “EVENTDATA1”, which now matches the default install disks.
    
    Fix: Rename the new disks according to the convention described in this step so that they have unique names otherwise the migration may remove them at the end of the process.
2. Following the successful installation of a standalone instance of Oracle DBSecCentral, if you need to rebalance the existing storage onto new shared storage, move the 2 appliance to the SAN by running the following command as the root user on the first node:
```
/opt/avdf/config-utils/bin/config-asm \
rebalance \
eventdata=/dev/mapper/eventdata_SAN_path \
systemdata=/dev/mapper/systemdata_SAN_path \
recovery=/dev/mapper/recovery_SAN_path \
confirm \
wait
```
  The arguments of ` /opt/avdf/config-utils/bin/config-asm ` are:
  - rebalance - Instruct the CLI to rebalance the ASM, moving the disk groups from their current location, to the specified location.
  - eventdata - A comma-separated list (no spaces or use quotes) of block devices to use for the disk group.
  - systemdata - A comma-separated list (no spaces or use quotes) of block devices to use for the disk group.
  - recovery - A comma-separated list (no spaces or use quotes) of block devices to use for the disk group.
  - confirm - The CLI will dry run the options if this flag is not present.
  - wait - The CLI will not exit until the (ASM controlled) estimation of the rebalancing runtime is complete. Note that this is an ETA, not factual, be careful about performing additional disk operations before confirmation.
    
    Tip:
    
    ASM can be extended by running the command again, specifying the block devices that are currently being used as well as new block devices.
    
    Devices can be removed and extended by specifying devices to add, leaving out devices to remove.
If you are enabling FIPS, see Enabling FIPS 140-2 in Oracle DBSecCentral.
Test and then move the DBSecCentral appliance to RAC by running the following as the root user on the first node:

For testing, run the command with the test_run argument while keeping all other arguments the same. This allows you to validate the process without making significant runtime changes.
```
/opt/avdf/config-utils/bin/config-rac_cluster first_node\
test_run \
rac_cluster_name=AVDFCLUSTER \
rac_instance_name=AVDFRAC1 \
rac_device=eth0 \
rac_scan_device=bond0 \
rac_scan_name=avdfcluster.db.com \
rac_scan_port=1522 \
rac_vip_address=avdfrac1-vip.db.com \
racdata_dg=/dev/mapper/racdata_SAN_path
```
Once testing is complete and error-free, run the command to move the first RAC node.
```
/opt/avdf/config-utils/bin/config-rac_cluster first_node\
rac_cluster_name=AVDFCLUSTER \
rac_instance_name=AVDFRAC1 \
rac_device=eth0 \
rac_scan_device=bond0 \
rac_scan_name=avdfcluster.db.com \
rac_scan_port=1522 \
rac_vip_address=avdfrac1-vip.db.com \
racdata_dg=/dev/mapper/racdata_SAN_path
```
The arguments of /opt/avdf/config-utils/bin/config-rac_cluster are:
- rac_cluster_name - The name this cluster will use, this is a stand-alone cluster.
- rac_instance_name - The name this node will assume in the cluster, this must be unique for each cluster node.
- rac_device - The name of the device that will be used for the RAC interconnect.
- rac_scan_device - The name of the network device used for SCAN traffic, usually the same as the default network device.
- rac_scan_name - The DNS name for the SCAN.
- rac_scan_port - The port number used by the SCAN listener.
- rac_vip_address - The VIP address for this node.
- racdata_dg - A comma-separated list of disks that will be used to create the RACDATA disk group.
Register the second node with the cluster by running the following command as the root user on the first node:
```
/opt/avdf/config-utils/bin/config-rac_cluster register_node \
peer_ip=IP address of the new node to register \
peer_ca=CA certificate of new node
```
For example,
```
/opt/avdf/config-utils/bin/config-rac_cluster register_node \
peer_ip=192.168.62.15 \
peer_ca=- <<EOF-----BEGIN CERTIFICATE-----.........-----END CERTIFICATE-----EOF
```
The arguments of /opt/avdf/config-utils/bin/config-rac_cluster register_node are:
- peer_ip - The address of the new node to register.
- peer_ca - The CA certificate of the new node (/usr/local/dbfw/etc/ca.crt).
Test and then add the second node to the cluster by running the following command as the root user on the new node:

For testing, run the command with the test_run argument while keeping all other arguments the same. This allows you to validate the process without making significant runtime changes.
```
/opt/avdf/config-utils/bin/config-rac_cluster new_node \
test_run \
rac_instance_name=AVDFRAC2 \
rac_device=eth0 \
rac_scan_device=bond0 \
rac_vip_address=avdfrac2-vip.db.com \
peer_ip=IP address of the new node to register \
peer_ca=CA certificate of new node
```
Once testing is complete and error-free, run the command to add the second node to the cluster.
```
/opt/avdf/config-utils/bin/config-rac_cluster new_node \
rac_instance_name=AVDFRAC2 \
rac_device=eth0 \
rac_scan_device=bond0 \
rac_vip_address=avdfrac2-vip.db.com \
peer_ip=IP address of the new node to register \
peer_ca=CA certificate of new node
```
For example,
```
/opt/avdf/config-utils/bin/config-rac_cluster new_node \
rac_instance_name=rac2 \
rac_device=enp0s9 \
rac_scan_device=enp0s8 \
rac_vip_address=rac-startup-2-vip.vboxlocal.net \
peer_ip=192.168.62.10 \
peer_ca=- <<EOF-----BEGIN CERTIFICATE-----.........-----END CERTIFICATE-----EOF
```
The arguments of /opt/avdf/config-utils/bin/config-rac_cluster new_node are:
- rac_instance_name - The name this node will assume in the cluster, this must be unique for each cluster node.
- rac_device - The name of the device that will be used for the RAC interconnect.
- rac_scan_device - The name of the device that will be used to host the SCAN, usually the same as the default device.
- rac_vip_address - The VIP address for this node.
- racdata_dg - A comma-separated list of disks that will be used to create the RACDATA disk group.
- peer_ip - The address of the new node to register.
- peer_ca - The CA certificate of the new node (/usr/local/dbfw/etc/ca.crt).

On the first RAC node, remove the old logical volumes holding the ASM disks that are used during the default installation.

Run the following commands:

oracleasm deletedisk /dev/mapper/vg_root-lv_systemdata1 ;\
oracleasm deletedisk /dev/mapper/vg_root-lv_eventdata1 ;\
oracleasm deletedisk /dev/mapper/vg_root-lv_recovery1
lvremove -v -y /dev/mapper/vg_root-lv_systemdata1
lvremove -v -y /dev/mapper/vg_root-lv_eventdata1
lvremove -v -y /dev/mapper/vg_root-lv_recovery1

Once the two node RAC deployment is complete, you can use either web UI address to access the cluster.

Related Topics

Overview of Oracle RAC and Clusterware Best Practices

UI Restrictions and Workarounds

When deployed in high availability on Oracle RAC, certain settings and configurations are not accessible in the administrator’s Oracle Database Security Central console. You can use the corresponding AVCLI commands when they’re not available in the console.

DNS Settings

To configure DNS for RAC on AVS, run the following commands on each AVS:

/opt/avdf/config-utils/bin/config-dns set servers="a.b.c.d,a.b.c.d,a.b.c.d"

All DNS servers listed must contain the same information. If applying the RAC configuration to the available DNS server is not possible, you can set local overrides to the hosts file on each appliance. When updating the hosts file, ensure the changes are made to both the hosts file and the hosts file template:

/etc/hosts
/usr/local/dbfw/templates/template-iptables

You may also need to replicate changes made to the hosts file on other systems as well (agent hosts) which may generate significant administrative overhead. It is advisable to ensure that the new records can be added to the appropriate DNS servers.

NTP Settings

To configure NTP settings for RAC on AVS, run the following command on each AVS:

/opt/avdf/config-utils/bin/config-ntp set \
  servers="a.b.c.d,a.b.c.d,a.b.c.d" \
  enabled=true \
  sync_on_save=true \

Syslog Settings

To configure Syslog settings for RAC on AVS, run the following command on each AVS:

/opt/avdf/config-utils/bin/config-syslog set \
  udp_destinations="a.b.c.d,a.b.c.d,a.b.c.d" \
  tcp_destinations="a.b.c.d,a.b.c.d,a.b.c.d" \
  categories="system,alerts,info,debug,heartbeat" \
  max_message_length=1024 \

Enable or Disable FIPS

To enable FIPS for RAC on AVS, run the following commands:

To stop the cluster on all nodes, run the following command on one node:
```
/u01/app/23.26.0.0.0/grid/bin/crsctl stop cluster -all
```

To enable FIPS, run the following command on all nodes:

/opt/avdf/config-utils/bin/config-fips set\
  enabled=true \
  reboot_required=true

Reboot servers after gracefully shutting down CRS:

systemctl isolate multi-user
/u01/app/26.3.0.0.0/grid/bin/crsctl stop crs
reboot now

To disable FIPS, repeat the steps above, ensuring you set enabled=false in Step 2.

Configure SSO

To configure Single sign on (SSO) for RAC on AVS:

Identify the node whose /usr/local/dbfw/etc/dbfw.conf shows the appropriate IdP domain. This is your source.
```
grep '^SSO_DOMAIN' /usr/local/dbfw/etc/dbfw.conf
```

On the non-source server, update dbfw.conf to use that same domain (replace IDP_DOMAIN_FROM_SOURCE).

sudo -u dbfw /usr/local/dbfw/bin/dbfwconf_editor.rb /usr/local/dbfw/etc/dbfw.conf SSO_DOMAIN='IDP_DOMAIN_FROM_SOURCE'

On each server (source first), back up the local SAML files.

cp /usr/local/dbfw/etc/apexsaml.crt /usr/local/dbfw/etc/apexsaml.crt.bak

cp /usr/local/dbfw/etc/apexsaml.key /usr/local/dbfw/etc/apexsaml.key.bak

From the source server, copy the known-good files to the other node.

scp /usr/local/dbfw/etc/apexsaml.crt support@OTHER_HOST:/tmp

scp /usr/local/dbfw/etc/apexsaml.key support@OTHER_HOST:/tmp

On the non-source server, copy the temp files to the appropriate destination.

cp /tmp/apexsaml.crt /usr/local/dbfw/etc/

cp /tmp/apexsaml.key /usr/local/dbfw/etc/

On each server, normalize ownership and permissions.

chown root:dbfwcerts /usr/local/dbfw/etc/apexsaml.crt /usr/local/dbfw/etc/apexsaml.key

chmod 460 /usr/local/dbfw/etc/apexsaml.crt /usr/local/dbfw/etc/apexsaml.key

On the non-source server only, refresh Apache to reload the SSO configuration.
```
/opt/avdf/bin/configure-networking run-only=apache
```

Configure LDAP

To configure Lightweight Directory Access Protocol (LDAP) for RAC on AVS:

Identify the source node that already talks to LDAP (wallet populated):
```
ls -l /usr/local/dbfw/etc/wallets/ldap_wallet
```

On that source node, archive the working wallet directory:

tar czf /tmp/ldap_wallet.tar.gz -C /usr/local/dbfw/etc/wallets ldap_wallet

Copy the archive to the other node:

scp /tmp/ldap_wallet.tar.gz support@OTHER_HOST:/tmp/

On the other node, back up any existing wallet:

mv /usr/local/dbfw/etc/wallets/ldap_wallet /usr/local/dbfw/etc/wallets/ldap_wallet.bak.$(date +%Y%m%d-%H%M%S)

Still on the other node, unpack the known-good wallet:
```
tar xzf /tmp/ldap_wallet.tar.gz -C /usr/local/dbfw/etc/wallets
```
Verify that cwallet.sso and ewallet.p12 are located in /usr/local/dbfw/etc/wallets/ldap_wallet.

Enforce the expected ownership and permissions:

chown -R oracle:oinstall /usr/local/dbfw/etc/wallets/ldap_wallet

chmod 700 /usr/local/dbfw/etc/wallets/ldap_wallet

chmod 600 /usr/local/dbfw/etc/wallets/ldap_wallet/*

(Optional) Clean up the transfer archive on both nodes:
```
rm -f /tmp/ldap_wallet.tar.gz
```

Update the UI Console Certificate

To update the UI console certificate for RAC on AVS:

Create the UI certificate in the UI.
Identify the node whose UI presents the correct certificate. Run the following command on each node to locate the new UI cert and key:
```
openssl x509 -noout -subject -issuer -dates -in /etc/pki/tls/certs/localhost.crt
```

On that source node, package the live cert/key and their saved copies:

tar czf /tmp/ui_console_cert.tgz -C /etc/pki/tls certs/user_uploaded_ui.crt certs/localhost.crt private/user_uploaded_ui.key private/localhost.key

Copy the archive to the other server:

scp /tmp/ui_console_cert.tgz support@OTHER_HOST:/tmp/

On the destination, back up the current HTTPS cert and key:

cp /etc/pki/tls/certs/localhost.crt /etc/pki/tls/certs/localhost.crt.bak.$(date +%Y%m%d-%H%M%S)

cp /etc/pki/tls/private/localhost.key /etc/pki/tls/private/localhost.key.bak.$(date +%Y%m%d-%H%M%S)

Extract the transferred files:

tar xzf /tmp/ui_console_cert.tgz -C /etc/pki/tls

Set the appropriate permissions:

chown root:root /etc/pki/tls/private/localhost.key /etc/pki/tls/private/user_uploaded_ui.key

chmod 600 /etc/pki/tls/private/localhost.key /etc/pki/tls/private/user_uploaded_ui.key

chown root:root /etc/pki/tls/certs/localhost.crt /etc/pki/tls/certs/user_uploaded_ui.crt

chmod 644 /etc/pki/tls/certs/localhost.crt /etc/pki/tls/certs/user_uploaded_ui.crt

Reload Apache on the destination so the UI picks up the new certificate:
```
systmectl restart httpd
```

Verify both servers present identical details.

openssl x509 -noout -subject -issuer -dates -in /etc/pki/tls/certs/localhost.crt

Rotate Certificates

Follow the appropriate instructions to rotate certificates for RAC on AVS.

Start certificate rotation on AVS appliances

To start the certificate rotation process on on AVS appliances:

Define shell vars for each session:
```
AVS1=<AVS1-ip-address>
AVS2=<AVS2-ip-address>
DBFW1=<DBFW1-ip-address>
```
Ensure that you replace the IP address placeholders with the IP addresses of each AVS and DBFW.

Rotate on each AVS and DBFW (do not restart any services yet):

root $> /usr/local/bin/gensslcert --create-ca --no-service-restart > /tmp/gensslcert.log 2>&1

Share CA between AVS and run configure-networking:

As the root user on AVS1, run the following commands:

cp /usr/local/dbfw/etc/ca.crt /opt/avdf/grid/cluster-ca/avs-$AVS1-ca.crt

sudo -u grid scp /opt/avdf/grid/cluster-ca/avs-$AVS1-ca.crt $AVS2:/opt/avdf/grid/cluster-ca/avs-$AVS1-ca.crt.tmp

sudo -u grid ssh $AVS2 mv /opt/avdf/grid/cluster-ca/avs-$AVS1-ca.crt.tmp /opt/avdf/grid/cluster-ca/avs-$AVS1-ca.crt

As the root user on AVS2, run the following commands:

cp /usr/local/dbfw/etc/ca.crt /opt/avdf/grid/cluster-ca/avs-$AVS2-ca.crt

sudo -u grid scp /opt/avdf/grid/cluster-ca/avs-$AVS2-ca.crt $AVS1:/opt/avdf/grid/cluster-ca/avs-$AVS2-ca.crt.tmp

sudo -u grid ssh $AVS1 mv /opt/avdf/grid/cluster-ca/avs-$AVS2-ca.crt.tmp /opt/avdf/grid/cluster-ca/avs-$AVS2-ca.crt

As the root user, run configure-networking and restart Apache on each AVS:
```
/opt/avdf/bin/configure-networking
```
```
systemctl restart httpd
```

Start certificate rotatation on DBFW

To rotate cerificates on AVS appliances for RAC:

As the root user, run the following command to disable the DBFW certificate synchronization on each AVS:
```
systemctl stop dbfw-certs.path
```
Share the CA from the cluster to DBFW.
1. As the root user on AVS1, run the following command:
```
/opt/avdf/config-utils/bin/config-rac_cluster register_dbfw
```
2. Run the output from command run in the previous step once on each DBFW.
As the root user on DBFW, run configure-networking and restart Apache:
```
/opt/avdf/bin/configure-networking
```
```
systemctl restart httpd
```

Share each DBFW CA to the cluster.

On DBFW, copy the CA to each AVS:

scp /usr/local/dbfw/etc/ca.crt support@$AVS1:/tmp/fw-$DBFW1.crt

On AVS1, prepare the fwcerts directory:

mkdir /usr/local/dbfw/etc/avs/fwcerts.old/

mv /usr/local/dbfw/etc/avs/fwcerts/* /usr/local/dbfw/etc/avs/fwcerts.old/

Update the AVS1 fwcerts for each DBFW CA:

cp /tmp/fw-$DBFW1.crt /usr/local/dbfw/etc/avs/fwcerts/

ln -s /usr/local/dbfw/etc/avs/fwcerts/fw-$DBFW1.crt \
      /usr/local/dbfw/etc/avs/fwcerts/$(openssl x509 -hash -noout -in /usr/local/dbfw/etc/avs/fwcerts/fw-$DBFW1.crt).0

Update the AVS1 wallet with each DBFW CA:

su - oracle -c \
    "/var/lib/oracle/dbfw/bin/avca import_cert -wrl \
    /usr/local/dbfw/etc/avs/avswallet/ \
    -cert /usr/local/dbfw/etc/avs/fwcerts/fw-$DBFW1.crt -trusted"

On AVS2, clear the fwcerts:

mkdir /usr/local/dbfw/etc/avs/fwcerts.old/

mv /usr/local/dbfw/etc/avs/fwcerts/* /usr/local/dbfw/etc/avs/fwcerts.old/

Run the following command to enable DBFW certificate synchronization on each AVS:
```
systemctl start dbfw-certs.path
```
As the root user, transfer the DBFW CAs to AVS2:
```
/opt/avdf/bin/rac-dbfw-certs
```

Start certificate rotation for agents

The following commands update the Agent wallet on each AVS.

Run the following commands to create the new wallet:

rm -Rf /var/lib/oracle/dbfw/network/admin/avwallet_agent/network/admin/avwallet/

su - oracle -c \
      "/var/lib/oracle/dbfw/bin/avca create_wallet \
      -wrl /var/lib/oracle/dbfw/network/admin/avwallet_agent/network/admin/avwallet"

openssl x509 -noout -subject -in /usr/local/dbfw/etc/ca.crt

su - oracle -c 'orapki wallet display \
      -wallet /var/lib/oracle/dbfw/network/admin/avwallet_agent/network/admin/avwallet/' \
      | grep Subject

Run the following commands to add the CA for both the AVS to the wallet:

rm -f /tmp/avs-$AVS1-ca.crt
rm -f /tmp/avs-$AVS2-ca.crt

sudo -u oracle cp /opt/avdf/grid/cluster-ca/avs-$AVS1-ca.crt /tmp
sudo -u oracle cp /opt/avdf/grid/cluster-ca/avs-$AVS2-ca.crt /tmp

su - oracle -c \
      "/var/lib/oracle/dbfw/bin/avca import_cert -wrl \
      /var/lib/oracle/dbfw/network/admin/avwallet_agent/network/admin/avwallet/ \
      -cert /tmp/avs-$AVS1-ca.crt -trusted"

su - oracle -c \
      "/var/lib/oracle/dbfw/bin/avca import_cert -wrl \
      /var/lib/oracle/dbfw/network/admin/avwallet_agent/network/admin/avwallet/ \
      -cert /tmp/avs-$AVS2-ca.crt -trusted"

openssl x509 -noout -subject -in /usr/local/dbfw/etc/ca.crt

su - oracle -c \
      "orapki wallet display \
      -wallet /var/lib/oracle/dbfw/network/admin/avwallet_agent/network/admin/avwallet/" \
      | grep Subject

su - oracle -c "/var/lib/oracle/dbfw/bin/avca configure_bootstrap"

Restart services

To finalize the rotation process, restart services:

On each AVS, run the following commands:

systemctl stop javafwk controller

systemctl restart httpd rac-controller rac-jfwk

On each DBFW, run the following command:
```
systemctl restart stund httpd
```

SMTP setup

To set up and manage SMTP email notifications for RAC on AVS, use the SMTP Connection AVCLI commands.