Disaster Recovery Solution

A Data Center (DC) and Disaster Recovery (DR) solution helps ensure that critical business systems remain available and recoverable during planned maintenance, hardware failures, cyber incidents, or site-level disruptions. The DC site is the primary location where applications and services normally run and where production data is processed. The DR site is a separate, secondary location that maintains synchronized copies of critical systems and data, allowing operations to be restored or switched over if the primary DC site becomes unavailable. Together, DC and DR capabilities minimize downtime, protect data, and support business continuity.

Oracle Database Security Central provides disaster recovery through an Audit Vault Server High Availability deployment. The solution uses two appliances configured in primary and standby roles, primary in the Data Center (DC) and standby in Disaster Recovery.

About High Availability in Audit Vault Servers

In this configuration, you designate one Audit Vault Server instance as the primary in the DC site and the other as the standby in the DR site. The primary Audit Vault Server is the active server that provides the Audit Vault Server functionality.

If the primary Audit Vault Server becomes unavailable because of an unplanned outage for a period of 10 minutes, the configuration automatically fails over (failover) to the standby server. The earlier standby becomes the new primary.

In high availability, configuration data pertaining to target registration, Audit Vault Agent machine registration, and Database Firewall configuration is automatically synchronized with the standby.

The high availability in Audit Vault Server is internally managed by using Oracle Data Guard. You deploy the pair of Audit Vault Servers in maximum performance mode. This ensures the highest level of data protection without affecting the performance of the primary Audit Vault Server instance.

Note: The archivelog mode is enabled after you set up high availability. High availability requires archivelog mode, so don’t disable it after you set up high availability.

Tip: Oracle recommends that you configure high availability for the Audit Vault Servers before deploying Audit Vault Agents and Database Firewalls.

The Audit Vault Servers in high availability communicate through HTTPS and Oracle Net. There are no restrictions on where the Audit Vault Servers are located, as long as they can communicate with each other.

Important Points to Consider Before Configuring High Availability

Because existing data on the designated standby Audit Vault Server is purged during high availability configuration, consider the following points:

Impact on existing Database Firewalls: All Database Firewalls that are registered with the designated standby Audit Vault Server must be registered again after high availability is configured.
Impact on existing Audit Vault Agents: There is no impact on the Audit Vault Agents that are registered on the designated primary Audit Vault Server.

However, all the registered Audit Vault Agents on the designated standby Audit Vault Server must be redeployed on the primary after high availability is configured. See Post High Availability Pairing Steps.
Impact on existing audit trails and Database Firewall monitoring points: Because audit and network event data that is collected from targets by the designated standby Audit Vault Server is purged during high availability configuration, you must reconfigure these audit trails and Database Firewall monitoring points on the Audit Vault Server after high availability is configured to ensure that these targets continue to be protected.
Agentless collection is supported on both standalone and high availability AVS.

Audit Vault Server High Availability Configuration Process

To configure Audit Vault Servers for high availability, follow this high-level process:

Install two standalone Audit Vault Servers to use as the primary and standby servers.

Tip: Place the two Audit Vault Servers in two different data centers.
Configure the designated standby Audit Vault Server.
Configure the designated primary Audit Vault Server.

Prerequisites for Configuring High Availability for Audit Vault Servers

Ensure that you meet these prerequisites before configuring high availability for Audit Vault Servers.

Install two standalone Audit Vault Servers to use as the primary and standby servers.
Ensure that the designated primary and standby Audit Vault Servers have identical configurations so that they can stand in for each other. All of the following configurations should be the same:
- Oracle Database Security Central (Oracle DBSecCentral) version
- Total system memory
- Total repository storage size
- Number of NFS archive locations
- Repository encryption status
Ensure that the system time difference between the two Audit Vault Servers is less than 60 seconds.

Related Topics

Specifying Initial System Settings and Options on Audit Vault Server (Required)

Configure the Designated Standby Audit Vault Server

Learn how to configure the designated standby Audit Vault Server.

Make a note of the IP address of the designated primary Audit Vault Server.
Copy the server certificate of the designated primary Audit Vault Server:
1. Log in to primary Oracle Database Security Central console as super administrator.
2. Select Settings. The Security in the left navigation menu is selected by default.
3. Now, select Certificate sub in the main page.
4. Select Server Certificate sub.
5. Select Copy Certificate .
In the left navigation menu of the designated standby Audit Vault Server, perform these steps:
1. Select System.
2. Select High Availability link under the Configuration section on the main page.
3. In the Configure High Availability dialog, the Current status field indicates the status of the current Audit Vault Server, which is Standalone.
4. In the Configure this server as field, select the Standby server option.
5. In the expanded Configure High Availability dialog, enter the following settings:
  - Primary server IP address: Enter the IP address of the designated primary Audit Vault Server.
  - Primary server certificate: Paste the certificate that you copied from the designated primary Audit Vault Server.
6. Select Save. The designated primary Audit Vault Server’s IP address and certificate is now saved on the standby Audit Vault Server, and is now ready to be paired.

Configure the Designated Primary Audit Vault Server

Learn how to configure the designated primary Audit Vault Server.

Make a note of the IP address of the designated standby Audit Vault Server.
Copy the server certificate from the designated standby Audit Vault Server:
1. Log in to the designated standby Oracle Database Security Central console as super administrator.
2. Select Settings. The Security in the left navigation menu is selected by default.
3. Now select Certificate sub in the main page.
4. Select Server Certificate sub.
5. Select Copy Certificate .
Log in to the designated primary Oracle Database Security Central console as super administrator and perform these steps:
1. In the left navigation menu, select System.
2. Select High Availability link under the Configuration section in the main page.
3. In the Configure High Availability dialog, the Current status field indicates the status of the current Audit Vault Server, which is Standalone.
4. In the Configure this server as field, select the Primary server option.
5. In the expanded Configure High Availability window, enter the following settings:
  - Standby server IP address: Enter the IP address of the standby Audit Vault Server.
  - Standby server certificate: Paste the certificate that you copied from the standby Audit Vault Server.
6. Select Initiate Pairing at the bottom of the dialog, to initiate high availability pairing. To get the updated status, refresh the Oracle Database Security Central console periodically, as the process can take at least 15 minutes. This process can take longer depending on the amount of data in the repository. When the high availability pairing is complete, the High Availability Status field in the main page displays the current status.
Note:
- After high availability pairing is successfully completed, perform all the configuration tasks on the primary Audit Vault Server only. This includes tasks such as downloading the Audit Vault Agent, registering targets and hosts, adding Database Firewalls and monitoring points. To perform tasks like setting system time or changing IP address for the standby Audit Vault Server refer to section Specifying the Server Date, Time, and Keyboard Settings.
- During high availability pairing, the NFS archive locations pertaining to the primary and standby Audit Vault Servers are mapped. The mapping of these locations is displayed in the primary Oracle Database Security Central console after high availability pairing is successful.

Checking the High Availability Status of an Audit Vault Server

Learn how to check the high availability status of an Audit Vault Server.

After high availability pairing is successfully completed, the standby Oracle Database Security Central console is not accessible. Perform all tasks on the primary Oracle Database Security Central console. If you attempt to access the standby Oracle Database Security Central console, it redirects to the primary Oracle Database Security Central console.

Check High Availability Status Through the Console

In the Oracle Database Security Central console, select Settings.
In the left navigation menu, select System.
Under the Status section, check the High Availability Status field.

The possible values are:
- Standalone - This server is not configured for high availability and is a standalone instance.
- Primary - This server is currently the primary Audit Vault Server.
- Disconnected - This primary Audit Vault Server switches to this mode if it detects that the standby Audit Vault Server changed its role to standalone or primary. This indicates that the high availability pairing is broken. Contact Oracle Support for further assistance.

Check High Availability Status Through Commands

Log in to the Audit Vault Server through SSH as the support user.

Note: If you’re using the Oracle Cloud Infrastructure (OCI) marketplace image, connect through SSH as the OPC user.
```
ssh support@<audit_vault_server_ip_address>
```
Switch to the root user.
```
su - root
```
Note: If you’re using the OCI marketplace image, use the sudo su - command.
Switch to the oracle user.
```
su - oracle
```
Run the following command
```
/usr/local/dbfw/bin/setup_ha.rb --status
```
The output of above command will tell the current high availability (HA) status and different properties such as Data Guard broker status, fast recovery area usage, and apply lag of HA system.

Post High Availability Pairing Steps

Learn post high availability pairing steps for Audit Vault Agents and Host Monitor Agents.

Audit Vault Agents and Host Monitor Agents deployed on the designated primary Audit Vault Server require no further action. The information of Audit Vault Agents is replicated to the standby Audit Vault Server during high availability pairing.

Audit Vault Agents and Host Monitor Agents deployed on the designated standby Audit Vault Server will be unable to communicate with the designated primary Audit Vault Server after high availability pairing. To redeploy the Agents on the specific Agent machines, follow these steps:

Clean up the Agent_Home folder on the Agent machine.

Agent installed on Windows host machine	Agent installed on Linux (or other Unix) machine
Run the command `agentctl.bat unregistersvc` from the `Agent_Home` directory. Remove the contents in the `Agent_Home` directory on the Agent host machine.	Remove the contents in the `Agent_Home` directory on the Agent host machine.

Register the Agent on the Audit Vault Server and activate.
Download the agent.jar file from the Oracle Database Security Central console.
Copy the agent.jar file to the Agent_Home directory on the Agent host machine.
In the Agent_Home directory, run the following command:
```
java -jar agent.jar
```
Run the following command and provide the Agent activation key when prompted. The key is available on the Oracle Database Security Central console.
```
agentctl start -k
```

Note: This key is not displayed as you type.

Audit Vault Agent Communication with Audit Vault Server in High Availability

Learn how Audit Vault Agent communicates with Audit Vault Server.

Audit Vault Agent software is packaged with the connection details pertaining to Audit Vault Server. In case of high availability environment, the Audit Vault Agent software is packaged with the connection details pertaining to both the primary and standby Audit Vault Servers.

Existing Audit Vault Agents on the designated primary Audit Vault Server receive the connection details of both the primary and standby Audit Vault Servers during high availability configuration. New Audit Vault Agents that are deployed after high availability configuration are also packaged with the connection details pertaining to both the primary and standby Audit Vault Servers.

In the event of Audit Vault Server failover, the Audit Vault Agents reconnect to the new primary Audit Vault Server (previous standby).

Swapping Roles Between a Primary and Standby Audit Vault Server

Learn how to swap the roles of the primary and standby Audit Vault Servers.

If automatic failover is disabled, enable it. See Disabling or Enabling Failover of the Audit Vault Server.
Ensure that the status of the Oracle Data Guard observer is YES. To check the status, run the following commands on each Audit Vault Server:
1. Using the ssh utility, run the following command:
```
ssh support@<IP address of Audit Vault Server>
```
2. Log in as the root user.
```
su root
```
3. Switch to the oracle user.
```
su oracle
```
4. Run the following command:
```
/usr/local/dbfw/bin/setup_ha.rb --status
```
```
 The `Data guard observer` field in the output should say `YES`.
```
Log in to the Oracle Database Security Central console as a super administrator.
Select Settings tab.
In the left navigation menu, select System.
In the Configuration section, select High Availability. The Configure High Availability dialog appears.
Select Switch Roles.
In the confirmation window, select OK.

A message shows the progress of the high availability configuration. During this process, which takes at least 10 minutes, the console is unavailable. Refresh the browser periodically. When the configuration is complete, it redirects to the new primary Audit Vault Server.

Related Topics

Using Oracle Database Security Central Console

Initiating a Switchover Between Primary and Standby Audit Vault Servers

You can initiate a switchover if you know that your primary Audit Vault Server is going to be offline for an extended period of time (more than 10 minutes) and you wish to maintain the high availability configuration. You can also initiate a switchover if you wish to promote the standby Audit Vault Server to primary because the designation of primary data center has changed.

Log in to the Audit Vault Server through SSH as the support user.

Note: If you’re using the Oracle Cloud Infrastructure (OCI) marketplace image, connect through SSH as the OPC user.
```
ssh support@<audit_vault_server_ip_address>
```
Switch to the root user.
```
su - root
```
Note: If you’re using the OCI marketplace image, use the sudo su - command.
Switch to the oracle user.
```
su - oracle
```
Run the switchover command on the existing primary Audit Vault Server:
```
/usr/local/dbfw/bin/setup_ha.rb --switchover
```

Handling a Failover Scenario

In a high availability environment, automatic failover mechanism is enabled by default. You can disable it manually through the Oracle Database Security Central console.

When automatic failover is in effect, the system periodically monitors the availability of the primary Audit Vault Server. If the primary becomes unavailable for more than 10 minutes, then the failover to the standby Audit Vault Server is automatically triggered. However, if the primary Audit Vault Server has been gracefully shut down by the user, then no failover is automatically triggered. In this case, to manually initiate the failover, carefully examine the situation as required, and run the following command as the oracle user on the standby Audit Vault Server:

/usr/local/dbfw/bin/setup_ha.rb --failover

In a failover, the standby Audit Vault Server becomes the new primary. If the previous primary comes back within 20 minutes, it is reinstated as the new standby and both systems will be in a high availability configuration.

If the previous primary does not come back within 20 minutes, then it becomes unusable. The new primary unpairs and becomes a standalone instance. Perform the following procedure to bring the system back into high availability configuration:

Install a new Audit Vault Server for the new designated standby.
Follow the configuration steps again to configure the Audit Vault Servers for high availability. See Configuring High Availability for Audit Vault Servers.

Related Topics

Specifying Audit Vault Server System Settings

Unpair Primary and Standby Audit Vault Servers

Learn how to unpair primary and standby Audit Vault Servers in high availability environment.

Log in to the Oracle Database Security Central console as a super administrator.
Select Settings tab.
In the left navigation menu, select System.
In the Status page, select High Availability link under the Configuration section.
To unpair Audit Vault Servers in high availability mode, select Unpair.

After unpairing, the Audit Vault Servers are not synchronized. Make a note of the following details:
- The primary Audit Vault Server goes into Standalone mode and the standby Audit Vault Server stays in Standby mode. However, there is no communication between these two Audit Vault Servers.
- In case you attempt to connect to the standby Oracle Database Security Central console, it directs you to the primary Oracle Database Security Central console, which is the Standalone.
- The Audit Vault Agents communicate only with the standalone Audit Vault Server (previous primary).
- Do not try to pair the standby server with primary server; it will not work, as standby server is unusable after unpair. If you want to use the standby server to do the pairing, reinstall the standby server, and do the pairing.
  Note:
  - You can continue to perform backup operation on the standalone (previous primary) Audit Vault Server.
  - You can restore high availability after unpairing. See Handling a Failover Scenario for complete information.

Disabling or Enabling Failover of the Audit Vault Server

Learn how to enable or disable failover for Audit Vault Servers.

When you configure high availability, the system is configured for automatic failover. However, in some cases, you may want to disable automatic failover. For example, you may need to disconnect the Audit Vault Servers for maintenance or you may be in an environment with an unstable network that may cause frequent failover. In these cases, you may choose to disable automatic failover, and trigger the failover manually by following the steps mentioned below.

To enable or disable automatic failover using the Oracle Database Security Central console:

Log in to the primary Audit Vault Server as a super administrator.
Select Settings tab, and then in the left navigation menu, select System tab.
Select High Availability link under the Configuration section.
Select Enable Failover or Disable Failover as needed.

Alternately, you can run the following commands to disable or enable the failover as oracle user:

/usr/local/dbfw/bin/setup_ha.rb --disable_failover

/usr/local/dbfw/bin/setup_ha.rb --enable_failover

Note:

You can run the following command to determine if failover is currently disabled or enabled.

sudo -u oracle /usr/local/dbfw/bin/setup_ha.rb --status

Archiving and Retrieving in High Availability

Learn about archiving and retrieving audit and network event data in a high availability scenario.

Archive and retrieve functionality in high availability automatically handles the necessary steps to process the datafiles on both the primary and standby Audit Vault Server instances. In order to archive, you must provide an NFS archive location. An NFS archive location in a high availability environment contains separate NFS details for primary and standby Audit Vault Servers.

In case there is no NFS archive location, then follow these steps to create a new NFS archive location:

Log in to the Oracle Database Security Central console as super administrator.
Select Settings tab, and then select Archiving tab in the left navigation menu.
Select Manage Archive Locations sub tab in the main page.
Select Create, to create a new archive location using NFS.

Network File System (NFS) option is selected by default. Enter the following details to create a new NFS archive location:

Field	Description
Location Name	The name of the NFS archive location.
Remote Filesystem	Select an existing filesystem, or one will be created automatically based on the details of this archive location.
NFS Server for Primary	NFS Server IP address or host name for mounting the remote filesystem on primary Audit Vault Server.
NFS Server for Standby	NFS Server IP address or host name for mounting the remote filesystem on standby Audit Vault Server.
NFS Export Directory for Primary	Export directory on the NFS server for primary Audit Vault Server.
NFS Export Directory for Standby	Export directory on the NFS server for standby Audit Vault Server.
NFS Server Path for Primary	The destination path relative to NFS Export Directory for Primary.
NFS Server Path for Standby	The destination path relative to NFS Export Directory for Standby.

Note: The combination of NFS server, export directory, and the path specified for primary and standby Audit Vault Servers must be unique.

Select Save.

Note: Each Audit Vault Server instance has its own copy of the datafiles. When you archive or retrieve, the datafiles associated with each instance are automatically archived to, or retrieved from the associated archive location.

Tip: Place the NFS servers for primary and standby Audit Vault Servers in separate data centers.

Backup and Restore of Audit Vault Server in High Availability

Learn about backup and restore of Audit Vault Server in high availability.

In a high availability configuration, you must perform the backup operation on the primary Audit Vault Server and not on the standby. To recover from a disaster, you can restore from the backup taken earlier. However, the restored system is not automatically configured for high availability. You need to once again configure for high availability after completing the restore from backup.

See Also:

Removing High Availability Configuration

You may wish to remove the high availability configuration from the primary Audit Vault Server if the secondary host has failed and you need to re-create the high availability pair with a new standby host.

Log in to the Audit Vault Server through SSH as the support user.

Note: If you’re using the Oracle Cloud Infrastructure (OCI) marketplace image, connect through SSH as the OPC user.
```
ssh support@<audit_vault_server_ip_address>
```
Switch to the root user.
```
su - root
```
Note: If you’re using the OCI marketplace image, use the sudo su - command.
Switch to the oracle user.
```
su - oracle
```
Ensure the standby host is offline and removed from the network. Its IP address must not be accessible from the existing primary.
Run the setup_ha.rb script on the primary Audit Vault Server to remove the high availability configuration:
```
/usr/local/dbfw/bin/setup_ha.rb -v

--<password> --unconfigure
```

Managing Archival and Retrieval in High Availability Environments

Learn how to manage archival and retrieval in high availability environments.

Oracle Database Security Central supports archiving. Archiving was configured only on the primary Audit Vault Server and there was no ability to configure archiving on the standby server. After a failover, archive locations had to be manually set on the former standby (new primary). You can now configure NFS archive locations on both the primary and standby Audit Vault Servers, reducing the amount of manual work that needs to be performed following a failover.

Oracle Database Security Central ensures that the primary and secondary Audit Vault Servers have the same number of NFS archive locations. This is crucial for archiving and file management functionality to work effectively in a high availability environment.

Note:

Any user with admin privileges can perform archival and retrieval tasks.
It is recommended that NFS archive locations for primary and secondary Audit Vault Servers are on separate NFS servers.
It is recommended to have these NFS servers within the same Data Center as the Audit Vault Server. As in the NFS server for primary Audit Vault Server should be in the same data center and NFS server for secondary Audit Vault Server should be in the same data center.
NFS is a mount point on the Audit Vault Server. If you want to replace NFS server, then make sure the Audit Vault Server does not access the mount point.

Prerequisite

Ensure that all of the Prerequisites for Configuring High Availability for Audit Vault Servers are satisfied before configuring high availability.

After you complete the high availability pairing, the NFS locations pertaining to both the primary and secondary Audit Vault Servers are displayed under Manage Archive Locations of the primary Oracle Database Security Central console. These NFS locations include those created on both the primary and secondary Audit Vault Servers before and after configuring high availability. The names of these NFS locations have the primary location name or the name defined while creating the location once high availability is configured. The Oracle Database Security Central console provides details of the host, export directory, and destination path for both the primary and secondary Audit Vault Servers.

Note: Oracle Database Security Central supports automatic archival on both primary and secondary Audit Vault Servers. If automatic archival is enabled on the primary Audit Vault Server, it is enabled on the corresponding secondary Audit Vault Server as well. The Oracle Database Security Central console displays the archive locations of the primary host with their mapped corresponding secondary locations.

Upgrade and archiving functionality in high availability environment

Archiving functionality is disabled during the upgrade process only when there are datafiles archived to the NFS locations. Upon completion of the upgrade process the admin user must enable the archive functionality to start archiving.

Updating or Deleting NFS locations

The super admin can update or delete the NFS locations after high availability pairing of primary and secondary Audit Vault Servers. The NFS locations on both the primary and secondary Audit Vault Servers can be updated or deleted. In case the datafiles are archived, the location cannot be updated or deleted. The Location Name and the Primary Server Path or the Secondary Server Path can be updated in case high availability is enabled. However, the NFS mount point is internal and cannot be changed.

See Also: