9 Configuring High Availability

You can configure Database Firewalls pairs or Audit Vault Server pairs, or both, to provide a high-availability system architecture. These are known as resilient pairs.

For Oracle Database Firewall, resilient pair configurations apply to monitoring mode only.

9.1 About High Availability Configurations in Oracle Audit Vault and Database Firewall

This topic describes high availability configuration in Oracle Audit Vault and Database Firewall.

In a resilient Audit Vault Server pair, the primary Audit Vault Server performs all server functions. Audit and configuration data are copied from the primary to the secondary Audit Vault Server. The Audit Vault Server console is not available on the secondary Audit Vault Server, so if you attempt to access the console on the secondary server, you will be redirected to the Audit Vault Server console on the primary server.

In a high availability Audit Vault Server pair, when failover is enabled, the secondary server becomes the primary in the event of a failover.

In a resilient Database Firewall pair, both primary and secondary Database Firewall:

  • Receive the same span traffic

  • Have the same configuration that is synchronized by the Audit Vault Server. This is the configuration of targets, Database Firewall monitoring points, policies, and other settings. This excludes the system configuration like the IP address, network mask, DNS of the primary and secondary Database Firewall instances in the Audit Vault Server console. These settings are not synchronized.

  • Create log files according to the policy applied

  • Send out alerts to the Audit Vault Server. The Audit Vault Server then sends only the alerts from the primary Database Firewall.

The Audit Vault Server collects traffic logs from the primary and secondary instances of Database Firewall. The Audit Vault Server controls the state of the resilient pair. There is no communication between the primary and secondary instances of Database Firewall. If the Audit Vault Server is unable to contact the primary Database Firewall for an extended period, or if there is a time gap in the traffic logs collected from the primary Database Firewall due to non-availability of the specific Database Firewall instance, then the Audit Vault Server uses the traffic log files collected from the secondary Database Firewall. The Audit Vault Server deletes the traffic log files from both instances of Database Firewall after the data is stored for the specified time range in the Audit Vault Server database.

Note:

High availability pairing or unpairing must only be executed once in a period of one hour.

9.2 Managing A Resilient Audit Vault Server Pair

These topics provide an introduction to pairing Audit Vault Servers.

Specifically, these topics describe configuring the primary and secondary Audit Vault Servers, checking the high availability status of the Audit Vault Servers, upating the Audit Vault Agent after pairing the Audit Vault Servers, swapping roles between the primary and secondary Audit Vault Servers, and handling failover of the Audit Vault Server pair.

9.2.1 About Pairing Audit Vault Servers

When you pair two Oracle Audit Vault Servers, designating one as the primary and the other as the secondary server, all of the data and configuration information in the primary server, with the exception of network settings, is automatically copied to, and thereafter synchronized with, the secondary server.

After configuring the resilient pair of Audit Vault Servers, do all configuration tasks on the primary server only. This includes tasks such as deploying the Audit Vault Agent, setting up targets and hosts, and adding Database Firewalls and monitoring points.

If you deploy database firewalls and you configure a resilient pair of Oracle Audit Vault Servers, then you must provide the server certificate and IP address of both the primary and secondary Oracle Audit Vault Servers to each Oracle Database Firewall.

If you have deployed Audit Vault Agents of the secondary server before pairing Audit Vault Servers, then you should manually update the previously deployed Audit Vault Agents of the secondary server once pairing is complete.

This is not required, if you deployed Audit Vault Agents of primary server before performing high availability pairing of the Audit Vault Servers.

9.2.2 Prerequisites for Configuring a Resilient Pair of Audit Vault Servers

This topic describes the prerequisites for configuring a pair of Audit Vault Servers.

  • Ensure that both the primary and secondary Audit Vault Servers have the same specification. The specification includes:

    • System version
    • System time
    • Encryption status (enabled on both or disabled on both)
    • Shared memory size
    • RAM size
    • ASM disk space
  • Ensure that the clock is synchronized on both systems. (An incorrect time setting can cause certificate validation errors.)
  • Do the initial system configuration tasks for both primary and secondary Audit Vault Servers.
  • The high availability functionality does not work if the number of NFS locations are not the same. The admin user must verify and create the same number of archive locations before enabling high availability. When creating an NFS location, select Create New Filesystem in the field Remote Filesystem to ensure there is a different filesystem for every location added on the primary and secondary Audit Vault Servers. These locations are mapped during high availability pairing. The mapping of these locations is displayed in the primary Audit Vault Server console once high availability pairing is successful.
  • The host:export and directory:destination path combination of NFS archive locations for primary and secondary Audit Vault Servers must be unique.
  • The admin user must ensure there is sufficient space on the NFS archive location configured for the secondary Audit Vault Server. Before pairing for high availability, the admin user must check all of the NFS archive locations where archived datafiles exist on the primary Audit Vault Server. Then, compute the sum of all datafile size in those locations and add the same size or more to the NFS archive locations on secondary Audit Vault Server. The minimum size of the filesystem for a location on secondary Audit Vault Server should be the maximum size of the filesystems created on the primary Audit Vault Server.
  • In a high availability environment if the Audit Vault Agents are deployed on the secondary Audit Vault Server before pairing, then manually update the previously deployed Audit Vault Agents pertaining to the secondary Audit Vault Server after pairing is complete.

9.2.3 Configure the Secondary Audit Vault Server

Learn how to configure the secondary Audit Vault Server.

To configure the secondary server:

  1. Make a note of the IP addresses for Server 1 and Server 2.
  2. Copy the server certificate from Server 1 (the primary):
    1. Log in to Server1 as an administrator.
    2. Click the Settings tab.
    3. In the left navigation menu, click Security.
    4. In the page that appears, click the Certificate sub tab.
    5. Click Server Certificate sub tab.
    6. Click the Copy Certificate button.
  3. In the left navigation menu, select System.
  4. Click High Availability link under the Configuration section on the main page.
  5. In the Configure High Availability dialog, the Current Status field indicates the current Audit Vault Server which is standalone.
  6. To configure high availability, configure the Secondary Audit Vault Server first, then the Primary Audit Vault Server.
  7. In the Configure this server as field, select the Secondary server option.
  8. In the expanded Configure High Availability window, enter the following settings:
    • Primary server IP address: Enter the IP address of the primary server.
    • Primary server certificate: Paste the certificate that you copied from the primary server.
  9. Click Save.

    After validation, the primary server's IP address and certificate are saved.

9.2.4 Configure the Primary Audit Vault Server

Learn how to configure the primary Audit Vault Server.

In this procedure, the primary server is called Server 1, and the secondary or standby server is called Server 2.

  1. Make a note of the IP address for Server 1.
  2. Copy the server certificate from Server 2 (the secondary):
    1. Log in to Server 2 as an administrator.
    2. Click the Settings tab.
    3. In the left navigation menu, click Security.
    4. In the page that appears, click the Certificate sub tab.
    5. Click Server Certificate sub tab.
    6. Click the Copy Certificate button.
  3. In the left navigation menu, select System.
  4. Click High Availability link under the Configuration section on the main page.
  5. In the Configure High Availability dialog, the Current Status field indicates the current Audit Vault Server which is standalone.
  6. In the Configure this server as field, select the Primary server option.
  7. In the expanded Configure High Availability window, enter the following settings:
    • Secondary server IP address: Enter the IP address of the secondary server.
    • Secondary server certificate: Paste the certificate that you copied from the secondary server.

      When you are ready to start the pairing of Server 1 and Server 2, go to the next step.

  8. Click Initiate Pairing button at the bottom of the dialog, to initiate high availability pairing at the primary Audit Vault Server. This will take a few minutes, and when it is complete, the secondary server will no longer have a console user interface.

    Note:

    • The user must ensure to execute the high availability pairing procedure prior to archiving of ILM. Else, it may result in an error.
    • After completing this procedure, do all configuration tasks on the primary server only. This includes tasks such as deploying the Audit Vault Agent, setting up targets and hosts, and adding Database Firewalls and monitoring points. The console UI of Server 2 (the standby) will be unavailable and you will be redirected to Server 1.
  9. A message is displayed indicating the progress of the high availability configuration. During this process, which may take at least 10 minutes, the console may be unavailable.
  10. Refresh the browser periodically. When the configuration is complete, the High Availability Status field displays the updated status.

9.2.5 Checking the High Availability Status of an Audit Vault Server

Learn how to check the high availability status of an Audit Vault Server.

  1. In the Audit Vault Server console, click the Settings tab.
  2. In the left navigation menu, select System.
  3. Under the Status section, check the High Availability Status field.

    Possible values are:

    • Standalone - This server has no partner server.

    • Primary - This server is currently the primary server.

    • Disconnected - This primary server switches to this mode if it detects that the standby Audit Vault Server changed its role to Standalone or Primary. In Disconnected mode, the Audit Vault Server stops downloading the traffic log files from the Database Firewall and then blocks Audit Vault Agents by blocking external access to the Audit Vault Server database. However, the Audit Vault Server console is accessible.

9.2.6 Updating Audit Vault Agents and Host Monitor Agents After Pairing Audit Vault Servers

Learn how to update Audit Vault Agents after pairing Audit Vault Servers.

In a high availability pair of Audit Vault Servers, the secondary server becomes the primary in the event of a failover. If you have deployed Audit Vault Agents of a secondary server before performing the high availability pairing of the Audit Vault Servers, then after failover the status of the Agent in the new primary server becomes UNREACHABLE. To avoid this scenario, manually update previously deployed Audit Vault Agents of the secondary server once pairing is complete.

Audit Vault Agents and Host Monitor Agents are automatically updated after pairing or unpairing of the Audit Vault Servers.

This is not required if you have deployed Audit Vault Agents of a primary server before performing high availability pairing of the Audit Vault Servers.

To manually update an Audit Vault Agent after pairing Audit Vault Servers:

  1. Remove the entire Agent_Home folder.
  2. Deactivate the host and activate the host again using the Audit Vault Server GUI.
  3. Download the new agent.jar file from the new primary Audit Vault Server GUI, and copy it to the new Agent_Home directory on the host machine.
  4. In the Agent_Home directory run the following command:

    java -jar agent.jar

  5. Run the following command and provide the Agent activation key:

    agentctl start -k

    Enter Activation Key:

    Note:

    Enter the activation key when prompted. This key is not displayed as you type.

  6. Restart audit trails.

Note:

On Windows, run the following command from the Agent_Home folder before removing the Agent_Home folder.

agentctl.bat unregistersvc

9.2.7 Swapping Roles Between a Primary and Standby Audit Vault Server

This topic describes the procedure if you want to swap the roles of the primary and standby Audit Vault Servers.

  1. Log in to the Audit Vault Server console as a super administrator.
  2. Click the Settings tab.
  3. In the left navigation menu, select System.
  4. In the Configuration section click High Availability link.
  5. Click the Switch Roles button.
  6. In the Confirmation window, click OK.

    A message is displayed indicating the progress of the high availability configuration. Be aware that during this process, which can take at least 10 minutes, the console may be unavailable. Click the Refresh button periodically. When the configuration is complete, it will redirect to the new primary Audit Vault Server.

9.2.8 Handling a Failover of the Audit Vault Server Pair

When failover is enabled, during normal operations, the system periodically checks the availability of the primary Audit Vault Server in the resilient pair. This topic describes how to handle a failover of the Audit Vault Server pair.

Note the following scenarios:
  • If the primary Audit Vault Server becomes unavailable, the system automatically fails over to the secondary Audit Vault Server after a 10 minute delay. The delay prevents a failover due to a reboot of the primary server.

  • If the primary Audit Vault Server is manually shut down, the failover process is not triggered. If you bring the primary Audit Vault Server back online, then it continues in high availability mode.

  • If the primary Audit Vault Server becomes unavailable, the system automatically fails over to the secondary Audit Vault Server after a 10 minute delay. The delay prevents a failover due to a reboot of the primary server.

  • If the primary Audit Vault Server is manually shut down and reinstalled or replaced with another server, then you must perform the following procedure:

    1. Manually failover the current standby server by issuing the following command as the oracle user:

      /usr/local/dbfw/bin/setup_ha.rb --failover
      
    2. Log in to the Audit Vault console as the super administrative user so that you can unpair the two servers.

    3. Select Settings tab, and then in the left navigation menu, select the System tab.

    4. In the Status page, click High Availability under the Configuration section.
    5. In the Configure High Availability dialog, click Primary server. The dialog expands.
    6. Click the Unpair button.

    7. Copy the new certificates between the two Audit Vault servers.

    8. Initiate the high availability setup again by clicking the Initiate Pairing button in the expanded dialog.

In the event of a failover, the secondary server becomes the new primary Audit Vault Server. You must do the following to configure this primary server, and repeat the high availability pairing:

  1. Log in to the Audit Vault Server console as a super administrator.

  2. Click on the Settings tab.

  3. In the left navigation menu, select System.
  4. In the Status page, click High Availability link under the Configuration section.

  5. Click Primary server option. In the expanded dialog, unpair the new primary server to convert it to a standalone server by clicking on the Unpair button.

  6. On the standalone server, configure the network and services settings (for example DNS settings).

  7. On the standalone server, manually mount any remote filesystems (NFS shares) defined as archive locations, using this SQL*Plus statement:

    ALTER REMOTE FILESYSTEM filesystem_name MOUNT

  8. Disconnect the failed server and replace it. The replacement server can now be configured as the new secondary server.

  9. Follow the configuration steps again to pair the two Audit Vault Servers.

9.2.9 Disabling or Enabling Failover of the Audit Vault Server

Under some conditions, you may want to disable automatic failover of a high availability pair of Audit Vault Servers.

For example, you may need to disconnect the Audit Vault Server for maintenance without triggering the automatic failover, or you may be in an environment with an unstable network that causes frequent failovers. In these cases, you may choose to disable automatic failover, and to do it manually if needed.
To disable or enable failover for a high availability pair of Audit Vault Servers:
  1. Log in to the primary Audit Vault Server as a super administrator.
  2. Click the Settings tab, and then in the left navigation menu, select System tab.
  3. In the Status page, click High Availability link under the Configuration section.
  4. Click the Enable/Disable Failover button.

    Note:

    Alternately, you can execute the following commands to disable or enable the failover through the Audit Vault and Database Firewall server as root user:

    sudo -u oracle /usr/local/dbfw/bin/setup_ha.rb --disable_failover

    sudo -u oracle /usr/local/dbfw/bin/setup_ha.rb --enable_failover

9.2.10 Performing a Manual Failover of the Audit Vault Server

If you have disabled automatic failover, you can perform a manual failover of the Audit Vault Server.

To perform a manual failover, run this command as oracle on the Audit Vault Server:

/usr/local/dbfw/bin/setup_ha.rb --failover

9.3 Managing A Resilient Database Firewall Pair

These topics describe how to manage, configure, switch roles, and break a resilient Database Firewall pair.

9.3.1 About Managing A Resilient Database Firewall Pair

This topic describes how to manage a resilient Database Firewall pair. The procedure described in this topic applies to a Database Firewall in monitoring mode only.

Prerequisites
  • Before you designate two Database Firewalls as a resilient pair, do the initial configuration tasks for each of them.

  • There must be no monitoring points configured on either of the Database Firewalls that you plan to pair. Be sure to delete all monitoring points on both Database Firewalls before creating a resilient pair.

If You Configure a Resilient Pair of Audit Vault Servers

If you have also configured a resilient pair of Audit Vault Servers, remember you must provide each Audit Vault Server's IP address and certificate to each Database Firewall in your system.

9.3.2 Configuring A Resilient Database Firewall Pair

This topic describes how to configure a resilient Database Firewall pair.

  1. Log in to the Audit Vault Server console as an administrator.
    If you have defined a resilient Audit Vault Server pair, then use the primary server's console. In case the Audit Vault Server is configured with resilient firewall pair without configuring monitoring points, then failover may not occur.
  2. Click the Database Firewalls tab.
  3. In the left navigation menu, select High Availability to display the available resilient pairs.
  4. In the Database Firewalls menu, select Resilient Pair.
  5. Click Create.
  6. In the Create Resilient Pair window, from the Primary and Secondary menus, select the primary and secondary firewalls that you want to use in this pair.
  7. Click Save.

9.3.3 Switching Roles in a Resilient Pair of Database Firewalls

Follow this procedure if you want to switch the roles of the primary and secondary Database Firewalls in a resilient pair.

  1. Log in to the Audit Vault Server console as an administrator.

    If you have defined a resilient pair of Audit Vault Servers, then use the primary server's console.

  2. Click the Database Firewalls tab.
  3. In the left navigation menu, select High Availability.
  4. Click the Swap button.
  5. In the confirmation dialog box, click OK.

9.3.4 Breaking (Un-pairing) a Resilient Pair of Database Firewalls

Use this procedure if you want to break (or un-pair) a resilient pair of Database Firewalls.

  1. Log in to the Audit Vault Server console as an administrator.

    If you have defined a resilient pair of Audit Vault Servers, use the primary server's console.

  2. Click the Database Firewalls tab.
  3. In the left navigation menu, select High Availability.
  4. Select the resilient pair you want, and then click Break.

9.4 High Availability For Database Firewall In Proxy Mode

The high availability configuration of a resilient pair of Database Firewall instances is usually set up in Monitoring Only modes. Oracle Audit Vault and Database Firewall provides an option to set up the high availability configuration for Database Firewall that you have deployed in Monitoring / Blocking (Proxy) mode.

Two or more individual Database Firewall instances deployed in proxy mode can be configured to achieve high availability.

9.4.1 Configuring High Availability For Database Firewall In Proxy Mode Through Client Configuration

This topic contains the necessary information about configuring high availability for two or more individual Database Firewall instances in proxy mode. This can be achieved by making changes to the client configuration.

Oracle SQL OCI (Oracle Call Interface) based clients use tnsnames.ora file to connect to the database. The parameters in this file need to be modified as part of this configuration.

Note:

This feature is available in Oracle Audit Vault and Database Firewall release 12.2.0.11.0 and later.

Prerequisite

Complete the basic set up that is required before executing the configuration procedure:

  1. Audit Vault Server instance is up and running.
  2. Two or more Database Firewall instances are up and running.
  3. Register the Database Firewall instances in the Audit Vault Server.
  4. Configure the target database.
  5. Configure one monitoring point for each Database Firewall instance to the same database target.

The client can be configured using the following parameters:

  1. ADDRESS_LIST

  2. CONNECT_TIMEOUT

  3. LOAD_BALANCE

ADDRESS_LIST

Include multiple Database Firewall IP addresses in the ADDRESS_LIST to connect to the Database target.

Note:

Avoid using port numbers that are configured for Audit Vault Server or Database Firewall internally.

For example:

dbfw1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.1)(PORT=1111))(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.2)(PORT=2222)))(CONNECT_DATA=(SERVICE_NAME=dbfwdb)))

  1. Connect to the database using a client application.

    For example: If you are using SQL*Plus client, then use the following command:

    sqlplus <username>/<password>@<net_service_name>

  2. The client attempts to connect to the first Database Firewall instance. In case the first instance is down or not reachable, then the client attempts to connect to the second instance. It works similar to active and passive deployment.

CONNECT_TIMEOUT

Note:

The admin user must use caution to set this parameter.

Use CONNECT_TIMEOUT parameter to quickly detect if the node is down. It also reduces the connection time in case of failover.

For example:

dbfw1=(DESCRIPTION=(CONNECT_TIMEOUT=10)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.1)(PORT=1111))(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.2)(PORT=2222)))(CONNECT_DATA=(SERVICE_NAME=dbfwdb)))

The client attempts to connect to the first Database Firewall instance. In case the first instance is down or not reachable, then the client waits for the duration (seconds) mentioned in the CONNECT_TIMEOUT parameter. In the above example it is 10 seconds. Later the client attempts to connect to the second instance. This behavior can be modified accordingly.

Note:

By default the wait time is 60 seconds.
LOAD_BALANCE

Use LOAD_BALANCE parameter to load balance client connections across deployed instances of Database Firewall.

For example:

dbfw1=(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.1)(PORT=1111))(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.2)(PORT=2222)))(CONNECT_DATA=(SERVICE_NAME=dbfwdb)))

Once this parameter is defined, the clients connect to the list of addresses in a random sequence and balance the load on various Database Firewall instances. It works similar to load balancing deployment.

9.4.2 Configuring High Availability For Database Firewall In Proxy Mode Through DNS Setup

This topic contains the necessary information about configuring high availability of two or more individual Database Firewall instances when monitoring points are in proxy mode. This can be achieved by making configuration changes to the local DNS.

Note:

This feature is available in Oracle Audit Vault and Database Firewall release 12.2.0.11.0 and later.

Prerequisite

Complete the basic set up that is required before executing the configuration procedure:

  1. Audit Vault Server instance is up and running.
  2. Two or more Database Firewall instances are up and running.
  3. Register the Database Firewall instances in the Audit Vault Server.
  4. Configure the target database.
  5. Configure one monitoring point for each Database Firewall instance to the same database target.
  6. A server with DNS setup is up and running.
Setup a domain name in DNS:
  1. In the server where the DNS is configured, create a domain name to represent IP addresses for multiple Database Firewall instances. This is the local DNS that is used by the client for DNS or host name resolution.

  2. Make this change in the client that is used to connect to the target database. Define DNS server as name server for the existing domain. This configuration should be done on the client hosts.

  3. Clients should use the domain name as hostname while connecting to the database through the Database Firewall.

  4. Specify other details like the port number and service name.

  5. In case you are using SQL*Plus client, use the following string to connect:

    sqlplus username/password@<domain name>:port/servicename

  6. In this setup the Domain Name Server can be configured in one of the following ways:

    1. Configure DNS to always connect to an ordered list of Database Firewall instances (for example DBFW1, DBFW2, etc). In this case if a client is not able to connect to the first instance (DBFW1), then it attempts to connect to the second instance (DBFW2).
    2. Configure DNS to use round-robin algorithm while connecting to Database Firewall instances. This is for load balancing and works like active-active setup.