8 Configuring High Availability

Topics

8.1 About High Availability Configurations in Oracle Audit Vault and Database Firewall

You can configure Database Firewalls pairs or Audit Vault Server pairs, or both, to provide a high-availability system architecture. These are known as resilient pairs. For the Database Firewall, the resilient pair configuration described in this chapter applies to Database Activity Monitoring (DAM) mode only.

In a resilient Audit Vault Server pair, the primary Audit Vault Server performs all server functions. Audit and configuration data are copied from the primary to the secondary Audit Vault Server. The Audit Vault Server console is not available on the secondary Audit Vault Server, so if you attempt to access the console on the secondary server, you will be redirected to the Audit Vault Server console on the primary server.

In a high availability Audit Vault Server pair, when failover is enabled, the secondary server becomes the primary in the event of a failover.

In a resilient Database Firewall pair, both primary and secondary Database Firewall:

Receive the same span traffic
Have the same configuration that is synchronized by the Audit Vault Server. This is the configuration of secured targets, enforcement points, policies, and other monitoring settings. This excludes the system configuration like the IP address, network mask, DNS of the primary and secondary Database Firewall instances on the system page of the Database Firewall console. These settings are not synchronized.
Create log files according to the policy applied
Send out alerts to the Audit Vault Server. The Audit Vault Server then sends only the alerts from the primary Database Firewall.

The Audit Vault Server collects tra ffic logs from the primary and secondary instances of Database Firewall. The Audit Vault Server controls the state of the resilient pair. There is no communication between the primary and secondary instances of Database Firewall. If the Audit Vault Server is unable to contact the primary Database Firewall for an extended period, or if there is a time gap in the traffic logs collected from the primary Database Firewall due to non-availability of the specific Database Firewall instance, then the Audit Vault Server uses the traffic log files collected from the secondary Database Firewall. The Audit Vault Server deletes the traffic log files from both instances of Database Firewall after the data is stored for the specified time range in the Audit Vault Server database.

Figure 8-1 illustrates a pair of Audit Vault Servers and a pair of Database Firewalls in high availability mode.

Figure 8-1 Pairs of Audit Vault Servers and Database Firewalls in High Availability Mode

Description of "Figure 8-1 Pairs of Audit Vault Servers and Database Firewalls in High Availability Mode"

Note:

High availability pairing or unpairing must only be executed once in a period of one hour.

See Also:

Oracle Audit Vault and Database Firewall Concepts Guide for more information on DAM and DPE (Database Policy Enforcement) modes.

8.2 Managing A Resilient Audit Vault Server Pair

These topics provide an introduction to pairing Audit Vault Servers.

Specifically, these topics describe configuring the primary and secondary Audit Vault Servers, checking the high availability status of the Audit Vault Servers, upating the Audit Vault Agent after pairing the Audit Vault Servers, swapping roles between the primary and secondary Audit Vault Servers, and handling failover of the Audit Vault Server pair.

8.2.1 About Pairing Audit Vault Servers

When you pair two Audit Vault Servers, designating one as the primary and the other as the secondary server, all data and configuration in the primary server (with the exception of network settings) is automatically copied to, and thereafter synchronized with the secondary server.

After configuring the resilient pair of Audit Vault Servers, do all configuration tasks on the primary server only. This includes tasks such as deploying the Audit Vault Agent, setting up secured targets and hosts, and adding Database Firewalls and enforcement points.

Remember that if you are deploying Database Firewalls, and you configure a resilient pair of Audit Vault Servers, you must provide the server certificate and IP address of both the primary and secondary Audit Vault Server to each Database Firewall.

If you have deployed Audit Vault Agents of the secondary server before pairing Audit Vault Servers, then you should manually update the previously deployed Audit Vault Agents of the secondary server once pairing is complete.

This is not required, if you deployed Audit Vault Agents of primary server before performing high availability pairing of the Audit Vault Servers.

See Also:

8.2.2 Prerequisites for Configuring a Resilient Pair of Audit Vault Servers

This topic describes the prerequisites for configuring a pair of Audit Vault Servers.

Ensure that both the primary and secondary Audit Vault Servers have the same specification. The specification includes:
- System version
- System time
- Encryption status (enabled on both or disabled on both)
- Shared memory size
- RAM size
- ASM disk space
Ensure that the clock is synchronized on both systems. (An incorrect time setting can cause certificate validation errors.)
Do the initial system configuration tasks for both primary and secondary Audit Vault Servers.
The high availability functionality does not work if the number of NFS locations are not the same. The admin user must verify and create the same number of archive locations before enabling high availability. When creating an NFS location, select Create New Filesystem in the field Remote Filesystem to ensure there is a different filesystem for every location added on the primary and secondary Audit Vault Servers. These locations are mapped during high availability pairing. The mapping of these locations is displayed in the primary Audit Vault Server console once high availability pairing is successful.
The host:export and directory:destination path combination of NFS archive locations for primary and secondary Audit Vault Servers must be unique.
The admin user must ensure there is sufficient space on the NFS archive location configured for the secondary Audit Vault Server. Before pairing for high availability, the admin user must check all of the NFS archive locations where archived datafiles exist on the primary Audit Vault Server. Then, compute the sum of all datafile size in those locations and add the same size or more to the NFS archive locations on secondary Audit Vault Server. The minimum size of the filesystem for a location on secondary Audit Vault Server should be the maximum size of the filesystems created on the primary Audit Vault Server.
In a high availability environment if the Audit Vault Agents are deployed on the secondary Audit Vault Server before pairing, then manually update the previously deployed Audit Vault Agents pertaining to the secondary Audit Vault Server after pairing is complete.

Related Topics

Specifying Initial System Settings and Options (Required)

8.2.3 Configure the Secondary Audit Vault Server

To configure Server2, the secondary server:

Copy the server certificate from Server1 (the primary):
1. Log in to Server1 as an administrator.
2. In the Settings tab of Server1, from the Security menu, click Server Certificate.
3. Copy the certificate.
In another browser window, log in to Server2 as a super administrator.
In the Server2 console, click the Settings tab.
From the System menu, select High Availability.
In the Configure this server as field, select Secondary server.
Enter the Primary server IP address in the field provided.
Paste the certificate you copied from Server1 in the Primary server certificate field.
Click Save.

After validation, the primary server's IP address and certificate are saved. If you wish to initiate pairing at this point, you can click the primary server's URL at the top of the page.

A Reset button appears, allowing you to cancel the settings configured in this procedure, thereby resetting the system to its original state.

8.2.4 Configure the Primary Audit Vault Server

In this procedure, the primary server is called Server1, and the secondary or standby server is called Server2.

To configure Server1, the primary server:

Copy the server certificate from Server2 (the secondary):
1. Log in to Server2 as an administrator.
2. In the Settings tab of Server1, from the Security menu, click Server Certificate.
3. Copy the certificate.
In another browser window, log in to Server1 as a super administrator.
In the Server1 console, click the Settings tab.
From the System menu, select High Availability.
In the Configure this server as field, select Primary server.

An Initiate Pairing button appears.
Enter the Secondary server IP address in the field provided.
Paste the certificate you copied from Server2 in the Secondary server certificate field.

When you are ready to start the pairing of Server1 and Server2, go to the next step.
Initiate high availability pairing at the primary server (Server1). This will take a few minutes, and once it is complete, the secondary server will no longer have a console UI.
Note:
- The user must ensure to execute the high availability pairing procedure prior to archiving of ILM. Else, it may result in an error.
- After completing this procedure, do all configuration tasks on the primary server only. This includes tasks such as deploying the Audit Vault Agent, setting up secured targets and hosts, and adding Database Firewalls and enforcement points. The console UI of Server2 (the standby) will be unavailable and you will be redirected to Server1.
Click Initiate Pairing.

A message is displayed indicating the progress of the high availability configuration. During this process, which may take at least 10 minutes, the console may be unavailable.
Refresh the browser periodically.

When the configuration is complete, the High Availability Status is displayed.

8.2.5 Checking the High Availability Status of an Audit Vault Server

To check the high av ailability status of an Audit Vault Server:

In the Audit Vault Server console, click the Settings tab.
From the System menu, click Status.
Check the High Availability Status. The values are:
- Standalone - This server has no partner server.
- Primary - This server is currently the primary server.
- Disconnected - This primary server switches to this mode if it detects that the standby Audit Vault Server changed its role to Standalone or Primary. In Disconnected mode, the Audit Vault Server stops downloading the traffic log files from the Database Firewall and then blocks Audit Vault agents by blocking external access to the Audit Vault Server database. However, the user interface (Audit Vault console) is accessible.
To see the IP address and certificate of the other (pe er) server in a paired system, in the System menu, click High Availability.

8.2.6 Updating Audit Vault Agents and Host Monitor Agents After Pairing Audit Vault Servers

This topic describes how to update Audit Vault Agents after pairing Audit Vault Servers.

In a high availability pair of Audit Vault Servers, the secondary server becomes the primary in the event of a failover. If you have deployed Audit Vault Agents of secondary server before performing the high availability pairing of the Audit Vault Servers, after failover, then the status of the Agent in the new primary server becomes UNREACHABLE. To avoid this scenario, manually update previously deployed Audit Vault Agents of the secondary server once pairing is complete.

This is not required, if you have deployed Audit Vault Agents of primary server before performing high availability pairing of the Audit Vault Servers.

Note:

Audit Vault Agents and and Host Monitor Agents are automatically updated after pairing or unpairing Audit Vault Servers. This is applicable for releases 12.2.0.5.0 and onwards.
In case the version of Oracle Audit Vault and Database Firewall is below 12.2.0.4.0, then use this procedure to update Audit Vault Agents or Host Monitor Agents manually.

To manually update an Audit Vault Agent after pairing Audit Vault Servers:

Remove complete Agent_Home folder.
Deactivate the host and activate the host again using the Audit Vault Server GUI.
Download the new agent.jar from the new primary Audit Vault Server GUI, and copy it to the new Agent_Home directory on the host machine.
In the Agent_Home directory execute the following command:

java -jar agent.jar
Execute the following command and provide the Agent activation key:

agentctl start -k

Enter Activation Key:

Note:

Enter the activation key when prompted. This key is not displayed as you type.
Restart audit trails.

Note:

On Windows, execute the following command, from the Agent_Home folder before removing the complete Agent_Home folder.

agentctl.bat unregistersvc

8.2.7 Swapping Roles Between a Primary and Standby Audit Vault Server

Follow this procedure if you want to swap the roles of the primary and standby Audit Vault Servers.

Log in to the Audit Vault Server console as a super administrator.
Click the Settings tab.
In the SYSTEM menu, select High Availability.
Click the Switch Roles button.
In the Confirmation window, click OK.

A message is displayed indicating the progress of the high availability configuration. Be aware that during this process, which can take at least 10 minutes, the console may be unavailable. Click the Refresh button periodically. When the configuration is complete, it will redirect to the new primary Audit Vault Server.

See Also:

Logging in to the Audit Vault Server Console UI

8.2.8 Handling a Failover of the Audit Vault Server Pair

When failover is enabled, during normal operation, the system periodically checks the availability of the primary Audit Vault Server in the resilient pair.

Note the following scenarios:

If the primary Audit Vault Server becomes unavailable, the system automatically fails over to the secondary Audit Vault Server after a 10 minute delay. The delay prevents a failover due to a reboot of the primary server.
If the database is shut down properly, there is no failover. If the primary Audit Vault Server is shut down (for example powered off) and it does not come up within 10 minutes, then it results in failover.
If the primary Audit Vault Server is manually shut down and reinstalled or replaced with another server, then you must perform the following procedure:
1. Manually failover the current standby server by issuing the following command as the oracle user:
```
/usr/local/dbfw/bin/setup_ha.rb --failover
```
2. Then log in to the Audit Vault console as the super administrative user so that you can unpair the two servers.
3. Select Settings, and then select High Availability.
4. In the High Availability status page, click the Unpair button.
5. Copy the new certificates between the two Audit Vault servers.
6. Initiate the high availability setup again by clicking the Initiate Pairing button.

In the event of a failover, the secondary server becomes the new primary Audit Vault Server. You must do the following to configure this primary server, and repeat the high availability pairing:

Log in to the Audit Vault Server console as a super administrator.
Click on the Settings tab.
Select Settings, and then select High Availability.
In the High Availability Status page, unpair the new primary server to convert it to a standalone server by clicking on the Unpair button.
On the standalone server, configure the network and services settings (for example DNS settings).
On the standalone server, manually mount any remote filesystems (NFS shares) defined as archive locations, using this AVCLI command:

ALTER REMOTE FILESYSTEM filesystem_name MOUNT
Disconnect the failed server and replace it. The replacement server can now be configured as the new secondary server.
Follow the configuration steps again to pair the two Audit Vault Servers.

See Also:

8.2.9 Disabling or Enabling Failover of the Audit Vault Server

Under some conditions you may want to disable automatic failover of a high availability pair of Audit Vault Servers. For example, you may need to disconnect the Audit Vault Server for maintenance without triggering the automatic failover, or you may be in an environment with an unstable network that causes frequent failovers. In these cases, you may choose to disable automatic failover, and to do it manually if needed.

To disable or enable failover for a high availability pair of Audit Vault Servers:

Log in to the primary Audit Vault Server as a super administrator.
Click the Settings tab, and then in the System menu, click High Availability.
Click the Enable/Disable Failover button.

Note:

Alternately, you can execute the following commands to disable or enable the failover through the Audit Vault and Database Firewall server as root user:

sudo -u oracle /usr/local/dbfw/bin/setup_ha.rb --disable_failover

sudo -u oracle /usr/local/dbfw/bin/setup_ha.rb --enable_failover

8.2.10 Performing a Manual Failover of the Audit Vault Server

If you have disabled automatic failover, you can perform a manual failover by running this command as oracle on the Audit Vault Server:

/usr/local/dbfw/bin/setup_ha.rb --failover

8.3 Managing A Resilient Database Firewall Pair

These topics describe how to manage, configure, switch roles, and break a resilient Database Firewall pair.

8.3.1 About Managing A Resilient Database Firewall Pair

The procedure described here applies to a Database Firewall in DAM mode only.

Prerequisites

Before you designate two Database Firewalls as a resilient pair, do the initial configuration tasks for each of them. See Configuring the Database Firewall for more information.
There must be no enforcement points configured on either of the Database Firewalls that you plan to pair. Be sure to delete all enforcement points on both Database Firewalls before creating a resilient pair.

If You Configure a Resilient Pair of Audit Vault Servers

If you have also configured a resilient pair of Audit Vault Servers, remember you must provide each Audit Vault Server's IP address and certificate to each Database Firewall in your system.

Note:

Specifying the Audit Vault Server Certificate and IP Address

8.3.2 Configuring A Resilient Database Firewall Pair

To configure a resilient Database Firewall pair:

Log in to the Audit Vault Server console as an administrator.
Note:
- If you have defined a resilient Audit Vault Server pair, then use the primary server's console.
- In case the Audit Vault Server is configured with resilient firewall pair without configuring enforcement points, then failover may not occur.
Click the Database Firewalls tab.
In the Database Firewalls menu, select Resilient Pair.
In the Primary and Secondary fields, select the primary and secondary firewalls you want to use in this pair.

See Also:

Logging in to the Audit Vault Server Console UI

8.3.3 Switching Roles in a Resilient Pair of Database Firewalls

Follow this procedure if you want to switch the roles of the primary and secondary Database Firewalls in a resilient pair.

Log in to the Audit Vault Server console as an administrator.

If you have defined a resilient pair of Audit Vault Servers, then use the primary server's console.
Click the Database Firewalls tab.
In the Database Firewalls menu, select Resilient Pair.
Click the Swap.
In the confirmation dialog box, click OK.

Note:

In case of Database Firewall configured for high availability, the settings must be the same for all the Database Firewall instances. In the event of a failover, the standby Database Firewall instance becomes the primary. The SYSLOG settings on the standby Database Firewall instance is in effect. In this due course, some SYSLOG settings and logging is turned off. This is done to avoid duplicate logs sent by both the instances.

When the previous primary becomes active again, there is no transfer or sharing of settings between the Database Firewall instances. Manual modification of the rsyslog.conf must be avoided as any changes result in erasing the settings during the following failover. The actual saved values in the SYSLOG settings should not be changed on failover.

See Also:

Logging in to the Audit Vault Server Console UI

8.3.4 Breaking (Un-pairing) a Resilient Pair of Database Firewalls

Use this procedure if you want to break (or un-pair) a resilient pair of Database Firewalls.

Log in to the Audit Vault Server console as an administrator.

If you have defined a resilient pair of Audit Vault Servers, use the primary server's console.
Click the Database Firewalls tab.
In the Database Firewalls menu, select Resilient Pair.
Select the resilient pair you want, and then click Break.

See Also:

Logging in to the Audit Vault Server Console UI

8.4 High Availability For Database Firewall In Proxy Mode

The high availability configuration of a resilient pair of Database Firewall instances is usually set up in Monitoring Only modes. Oracle Audit Vault and Database Firewall provides an option to set up the high availability configuration for Database Firewall that you have deployed in Monitoring / Blocking (Proxy) mode.

Two or more individual Database Firewall instances deployed in proxy mode can be configured to achieve high availability.

8.4.1 Configuring High Availability For Database Firewall In Proxy Mode Through Client Configuration

This topic contains the necessary information about configuring high availability for two or more individual Database Firewall instances in proxy mode. This can be achieved by making changes to the client configuration.

Oracle SQL OCI (Oracle Call Interface) based clients use tnsnames.ora file to connect to the database. The parameters in this file need to be modified as part of this configuration.

Note:

This feature is available in Oracle Audit Vault and Database Firewall release 12.2.0.11.0 and later.

Prerequisite

Complete the basic set up that is required before executing the configuration procedure:

Audit Vault Server instance is up and running.
Two or more Database Firewall instances are up and running.
Register the Database Firewall instances in the Audit Vault Server.
Configure the target database.
Configure one monitoring point for each Database Firewall instance to the same database target.

The client can be configured using the following parameters:

ADDRESS_LIST
CONNECT_TIMEOUT
LOAD_BALANCE

ADDRESS_LIST

Include multiple Database Firewall IP addresses in the ADDRESS_LIST to connect to the Database target.

Note:

Avoid using port numbers that are configured for Audit Vault Server or Database Firewall internally.

For example:

dbfw1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.1)(PORT=1111))(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.2)(PORT=2222)))(CONNECT_DATA=(SERVICE_NAME=dbfwdb)))

Connect to the database using a client application.

For example: If you are using SQL*Plus client, then use the following command:

sqlplus <username>/<password>@<net_service_name>
The client attempts to connect to the first Database Firewall instance. In case the first instance is down or not reachable, then the client attempts to connect to the second instance. It works similar to active and passive deployment.

CONNECT_TIMEOUT

Note:

The admin user must use caution to set this parameter.

Use CONNECT_TIMEOUT parameter to quickly detect if the node is down. It also reduces the connection time in case of failover.

For example:

dbfw1=(DESCRIPTION=(CONNECT_TIMEOUT=10)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.1)(PORT=1111))(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.2)(PORT=2222)))(CONNECT_DATA=(SERVICE_NAME=dbfwdb)))

The client attempts to connect to the first Database Firewall instance. In case the first instance is down or not reachable, then the client waits for the duration (seconds) mentioned in the CONNECT_TIMEOUT parameter. In the above example it is 10 seconds. Later the client attempts to connect to the second instance. This behavior can be modified accordingly.

Note:

By default the wait time is 60 seconds.

LOAD_BALANCE

Use LOAD_BALANCE parameter to load balance client connections across deployed instances of Database Firewall.

For example:

dbfw1=(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.1)(PORT=1111))(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.2)(PORT=2222)))(CONNECT_DATA=(SERVICE_NAME=dbfwdb)))

Once this parameter is defined, the clients connect to the list of addresses in a random sequence and balance the load on various Database Firewall instances. It works similar to load balancing deployment.

Related Topics

See Also:

Creating Database Firewall Policies

8.4.2 Configuring High Availability For Database Firewall In Proxy Mode Through DNS Setup

This topic contains the necessary information about configuring high availability of two or more individual Database Firewall instances when monitoring points are in proxy mode. This can be achieved by making configuration changes to the local DNS.

Note:

This feature is available in Oracle Audit Vault and Database Firewall release 12.2.0.11.0 and later.

Prerequisite

Complete the basic set up that is required before executing the configuration procedure:

Audit Vault Server instance is up and running.
Two or more Database Firewall instances are up and running.
Register the Database Firewall instances in the Audit Vault Server.
Configure the target database.
Configure one monitoring point for each Database Firewall instance to the same database target.
A server with DNS setup is up and running.

Setup a domain name in DNS:

In the server where the DNS is configured, create a domain name to represent IP addresses for multiple Database Firewall instances. This is the local DNS that is used by the client for DNS or host name resolution.
Make this change in the client that is used to connect to the target database. Define DNS server as name server for the existing domain. This configuration should be done on the client hosts.
Clients should use the domain name as hostname while connecting to the database through the Database Firewall.
Specify other details like the port number and service name.
In case you are using SQL*Plus client, use the following string to connect:

sqlplus username/password@<domain name>:port/servicename
In this setup the Domain Name Server can be configured in one of the following ways:
1. Configure DNS to always connect to an ordered list of Database Firewall instances (for example DBFW1, DBFW2, etc). In this case if a client is not able to connect to the first instance (DBFW1), then it attempts to connect to the second instance (DBFW2).
2. Configure DNS to use round-robin algorithm while connecting to Database Firewall instances. This is for load balancing and works like active-active setup.

Related Topics

See Also:

Creating Database Firewall Policies