9 High Availability in Oracle AVDF

Oracle AVDF supports high availability for Audit Vault Server and Database Firewall.

9.1 About High Availability in Oracle AVDF

Learn about high availability in Oracle AVDF.

High availability in Oracle AVDF increases reliability by ensuring continuity in Database Activity Monitoring services like audit data collection, network event data collection, analysis, reporting, etc. High availability requires a pair of Audit Vault Server instances or a pair of Database Firewall instances. One instance works as the primary and another instance works as the standby.

9.2 Managing High Availability in Audit Vault Server

Learn about configuring, monitoring, or updating high availability of Audit Vault Servers.

9.2.1 About High Availability in Audit Vault Servers

Learn about high availability in Audit Vault Servers.

High availability in Audit Vault Server involves two Audit Vault Server instances that are paired for business continuity. Start with the primary Audit Vault Server and then install another Audit Vault Server which can be the standby. In this configuration, one Audit Vault Server instance is designated as the primary and the other as the standby. The primary Audit Vault Server is the active server that provides the Audit Vault Server functionality. The standby is automatically synchronized (audit data and network event data) and has a consistent copy of the primary.

If the primary Audit Vault Server becomes unavailable because of unplanned outage for a period of 10 minutes, the configuration automatically fails over (failover) to the standby server. The earlier standby becomes the new primary.

In high availability, configuration data pertaining to target registration, Audit Vault Agent machine registration, Database Firewall configuration is automatically synchronized with the standby.

The high availability in Audit Vault Server is internally managed using Oracle Data Guard - Physical standby deployed in maximum protection mode. This ensures highest level of data protection without affecting the performance of the primary Audit Vault Server instance.

Best Practice:

Oracle recommends high availability to be configured before deploying Audit Vault Agents and Database Firewalls.

The Audit Vault Servers in high availability communicate through HTTPS and Oracle Net. There are no restrictions on where the Audit Vault Servers are located as long as they can communicate with each other.

Important Aspects to Consider Before Configuring High Availability

Since existing data on the designated standby Audit Vault Server is purged during high availability configuration, you must consider these aspects:

  • Impact on existing Database Firewalls: All Database Firewalls registered with designated standby Audit Vault Server must be registered again after high availability is configured.

  • Impact on existing Audit Vault Agents: There is no impact on the Audit Vault Agents registered on the designated primary Audit Vault Server.

    However, all the registered Audit Vault Agents on the designated standby Audit Vault Server must be redeployed on the primary after high availability configuration. See Post High Availability Pairing Steps for more information.

  • Impact on existing Audit Trails and Database Firewall monitoring points: Since audit and network event data collected from targets by the designated standby Audit Vault Server is purged during high availability configuration, you must reconfigure these audit trails and Database Firewall monitoring points on the Audit Vault Server after high availability is configured to ensure these targets continue to be protected.

Setting up Audit Vault Servers in high availability configuration involves these steps:

  1. Have two standalone Audit Vault Servers or install them as required.

    Best Practice:

    Place the two Audit Vault Servers in two different data centers.
  2. Configuring the designated standby Audit Vault Server.

  3. Configuring the designated primary Audit Vault Server.

9.2.2 Prerequisites for Configuring High Availability in Audit Vault Servers

Learn about the prerequisites to be performed before configuring high availability in Audit Vault Servers.

Prerequisites

  • The designated primary and standby Audit Vault Servers must have identical configuration so they can stand in for each other. They should have the same:

    • Oracle AVDF version
    • Total system memory
    • Total repository storage size
    • Number of NFS archive locations
    • Repository encryption status
  • System time difference between the two Audit Vault Servers must be less than 60 seconds.

9.2.3 Configure the Designated Standby Audit Vault Server

Learn how to configure the designated standby Audit Vault Server.

  1. Make a note of the IP address of the designated primary Audit Vault Server.
  2. Copy the server certificate of the designated primary Audit Vault Server:

    1. Log in to primary Audit Vault Server console as super administrator.
    2. Click the Settings tab. The Security tab in the left navigation menu is selected by default.
    3. Now, click the Certificate sub tab in the main page.
    4. Click the Server Certificate sub tab.
    5. Click the Copy Certificate button.
  3. In the left navigation menu of the designated standby Audit Vault Server, perform these steps:

    1. Select System tab.
    2. Click High Availability link under the Configuration section on the main page.
    3. In the Configure High Availability dialog, the Current status field indicates the status of the current Audit Vault Server which is Standalone.
    4. In the Configure this server as field, select the Standby server option.
    5. In the expanded Configure High Availability dialog, enter the following settings:

      • Primary server IP address: Enter the IP address of the designated primary Audit Vault Server.
      • Primary server certificate: Paste the certificate that you copied from the designated primary Audit Vault Server.
    6. Click Save. The designated primary Audit Vault Server's IP address and certificate is now saved on the standby Audit Vault Server, and is now ready to be paired.

9.2.4 Configure the Designated Primary Audit Vault Server

Learn how to configure the designated primary Audit Vault Server.

  1. Make a note of the IP address of the designated standby Audit Vault Server.
  2. Copy the server certificate from the designated standby Audit Vault Server:

    1. Log in to the designated standby Audit Vault Server console as super administrator.
    2. Click the Settings tab. The Security tab in the left navigation menu is selected by default.
    3. Now click the Certificate sub tab in the main page.
    4. Click Server Certificate sub tab.
    5. Click the Copy Certificate button.
  3. Log in to the designated primary Audit Vault Server console as super administrator and perform these steps:

    1. In the left navigation menu, select System tab.
    2. Click High Availability link under the Configuration section in the main page.
    3. In the Configure High Availability dialog, the Current status field indicates the status of the current Audit Vault Server which is Standalone.
    4. In the Configure this server as field, select the Primary server option.
    5. In the expanded Configure High Availability window, enter the following settings:

      • Standby server IP address: Enter the IP address of the standby Audit Vault Server.
      • Standby server certificate: Paste the certificate that you copied from the standby Audit Vault Server.
    6. Click Initiate Pairing button at the bottom of the dialog, to initiate high availability pairing. To get the updated status, refresh the Audit Vault Server console periodically as the process can take at least 15 minutes. This process can take longer depending on the amount of data in the repository. When the high availability pairing is complete, the High Availability Status field in the main page displays the current status.

      Note:

      • After high availability pairing is successfully completed, perform all the configuration tasks on the primary Audit Vault Server only. This includes tasks such as downloading the Audit Vault Agent, registering targets and hosts, adding Database Firewalls and monitoring points. To perform tasks like setting system time or changing IP address for the standby Audit Vault Server refer to section Specifying the Server Date, Time, and Keyboard Settings.
      • During high availability pairing, the NFS archive locations pertaining to the primary and standby Audit Vault Servers are mapped. The mapping of these locations is displayed in the primary Audit Vault Server console after high availability pairing is successful.

9.2.5 Checking the High Availability Status of an Audit Vault Server

Learn how to check the high availability status of an Audit Vault Server.

After high availability pairing is successfully completed, the standby Audit Vault Server console is not accessible. Perform all tasks on the primary Audit Vault Server console. If you attempt to access the standby Audit Vault Server console, it redirects to the primary Audit Vault Server console.

  1. In the Audit Vault Server console, click the Settings tab.
  2. In the left navigation menu, select System.
  3. Under the Status section, check the High Availability Status field.

    The possible values are:

    • Standalone - This server is not configured for high availability and is a standalone instance.

    • Primary - This server is currently the primary Audit Vault Server.

    • Disconnected - This primary Audit Vault Server switches to this mode if it detects that the standby Audit Vault Server changed its role to standalone or primary. This indicates that the high availability pairing is broken. Contact Oracle Support for further assistance.

9.2.6 Post High Availability Pairing Steps

Learn post high availability pairing steps for Audit Vault Agents and Host Monitor Agents.

Audit Vault Agents and Host Monitor Agents deployed on the designated primary Audit Vault Server, require no further action. The information of Audit Vault Agents is replicated to the standby Audit Vault Server during high availability pairing.

Audit Vault Agents and Host Monitor Agents deployed on the designated standby Audit Vault Server, will be unable to communicate with the designated primary Audit Vault Server after high availability pairing. To redeploy the Agents on the specific Agent machines, follow these steps:

  1. Clean up the Agent_Home folder on the Agent machine.
  2. Register the Agent on the Audit Vault Server and activate.
  3. Download the agent.jar file from the Audit Vault Server console.
  4. Copy the agent.jar file to the Agent_Home directory on the Agent host machine.
  5. In the Agent_Home directory, run the following command:
    java -jar agent.jar
  6. Run the following command and provide the Agent activation key when prompted. The key is available on the Audit Vault Server console.
    agentctl start -k

    Note:

    This key is not displayed as you type.

9.2.7 Audit Vault Agent Communication with Audit Vault Server in High Availability

Learn how Audit Vault Agent communicates with Audit Vault Server.

Audit Vault Agent software is packaged with the connection details pertaining to Audit Vault Server. In case of high availability environment, the Audit Vault Agent software is packaged with the connection details pertaining to both the primary and standby Audit Vault Servers.

Existing Audit Vault Agents on the designated primary Audit Vault Server receive the connection details of both the primary and standby Audit Vault Servers during high availability configuration. New Audit Vault Agents that are deployed after high availability configuration are also packaged with the connection details pertaining to both the primary and standby Audit Vault Servers.

In the event of Audit Vault Server failover, the Audit Vault Agents reconnect to the new primary Audit Vault Server (previous standby).

9.2.8 Swapping Roles Between a Primary and Standby Audit Vault Server

Learn how to swap the roles of the primary and standby Audit Vault Servers.

  1. Log in to the Audit Vault Server console as a super administrator.
  2. Click the Settings tab.
  3. In the left navigation menu, select System.
  4. In the Configuration section click High Availability link. The Configure High Availability dialog is displayed.
  5. Click the Switch Roles button.
  6. In the confirmation window, click OK.

    A message is displayed indicating the progress of the high availability configuration. During this process, which takes at least 10 minutes, the console is unavailable. Click the browser refresh button periodically. When the configuration is complete, it will redirect to the new primary Audit Vault Server.

9.2.9 Handling a Failover Scenario

Learn how to handle Audit Vault Server failover scenario.

In a high availability environment, automatic failover mechanism is enabled by default. It can be disabled manually through the Audit Vault Server console.

When automatic failover is in effect, the system periodically monitors the availability of primary Audit Vault Server. If the primary becomes unavailable for more than 10 minutes, then the failover to the standby Audit Vault Server is automatically triggered. However, if the primary Audit Vault Server has been gracefully shutdown by the user, then no failover is automatically triggered. In this case to manually initiate the failover, carefully examine the situation as required, and run the following command as oracle user on any of the Audit Vault Servers:

/usr/local/dbfw/bin/setup_ha.rb --failover

In the event of a failover, the standby Audit Vault Server becomes the new primary. In case the previous primary comes back up, it will be re-instated as the new standby. In case the previous primary is lost completely, then you must perform the following procedure to bring the system back into high availability configuration.

  1. Unpair the Audit Vault Servers. Access the Audit Vault Server console of the new primary and then unpair them.
  2. Install a new Audit Vault Server for the new designated standby.
  3. Follow the configuration steps again, to configure Audit Vault Servers for high availability. See Managing High Availability in Audit Vault Server for more information.

9.2.10 Unpair Primary and Standby Audit Vault Servers

Learn how to unpair primary and standby Audit Vault Servers in high availability environment.

  1. Log in to the Audit Vault Server console as a super administrator.
  2. Click Settings tab.
  3. In the left navigation menu, select System.
  4. In the Status page, click High Availability link under the Configuration section.
  5. To unpair Audit Vault Servers in high availability mode, click Unpair.

    After unpairing, the Audit Vault Servers are not synchronized. Make a note of the following details:

    • The primary Audit Vault Server goes into Standalone mode and the standby Audit Vault Server stays in Standby mode. However, there is no communication between these two Audit Vault Servers.
    • In case you attempt to connect to the standby Audit Vault Server console, it directs you to the primary Audit Vault Server console which is the Standalone.
    • The Audit Vault Agents communicate only with the standalone Audit Vault Server (previous primary).

    Note:

    • You can continue to perform backup operation on the standalone (previous primary) Audit Vault Server.
    • You can restore high availability after unpairing. See Handling a Failover Scenario for complete information.

9.2.11 Disabling or Enabling Failover of the Audit Vault Server

Learn how to enable or disable failover for Audit Vault Servers.

When you configure high availability, the system is configured for automatic failover. However, in some cases, you may want to disable automatic failover. For example, you may need to disconnect the Audit Vault Servers for maintenance or you may be in an environment with unstable network that may cause frequent failover. In these cases, you may choose to disable automatic failover, and trigger the failover manually by following the steps mentioned below.

To enable or disable automatic failover using the Audit Vault Server console:

  1. Log in to the primary Audit Vault Server as a super administrator.
  2. Click the Settings tab, and then in the left navigation menu, select System tab.
  3. Click High Availability link under the Configuration section.
  4. Click the Enable Failover or Disable Failover button as needed.

    Note:

    Alternately, you can execute the following commands to disable or enable the failover as oracle user:

    /usr/local/dbfw/bin/setup_ha.rb --disable_failover
    /usr/local/dbfw/bin/setup_ha.rb --enable_failover

9.2.12 Archiving and Retrieving in High Availability

Learn about archiving and retrieving audit and network event data in a high availability.

Archive and retrieve functionality in high availability automatically handles the necessary steps to process the datafiles on both the primary and standby Audit Vault Server instances. In order to archive, you must provide an NFS archive location. An NFS archive location in high availability environment contains separate NFS details for primary and standby Audit Vault Servers.

In case there is no NFS archive location, then follow these steps to create a new NFS archive location:

  1. Log in to the Audit Vault Server console as super administrator.
  2. Click Settings tab, and then click Archiving tab in the left navigation menu.
  3. Click Manage Archive Locations sub tab in the main page.
  4. Click Create, to create a new archive location using NFS.
  5. Network File System (NFS) option is selected by default. Enter the following details to create a new NFS archive location:

    Note:

    The combination of NFS server, export directory, and the path specified for primary and standby Audit Vault Servers must be unique.
  6. Click Save.

    Note:

    Each Audit Vault Server instance has its own copy of the datafiles. When you archive or retrieve, the datafiles associated with each instance is automatically archived to, or retrieved from the associated archive location.

    Best Practice:

    Place the NFS servers for primary and standby Audit Vault Servers in separate data centers.

9.2.13 Backup and Restore of Audit Vault Server in High Availability

Learn about backup and restore of Audit Vault Server in high availability.

In a high availability configuration, you must perform the backup operation on the primary Audit Vault Server and not on the standby. To recover from a disaster, you can restore from the backup taken earlier. However, the restored system is not automatically configured for high availability. You need to once again configure for high availability after completing the restore from backup.

9.3 High Availability for Database Firewall

Learn how to manage, configure, switch roles, and unpair a Database Firewall pair.

9.3.1 High Availability for Database Firewall

Learn about high availability in Database Firewall.

High availability in Database Firewall ensures uninterrupted network event monitoring in the event of network or Database Firewall failure. It also ensures that the corporate security policies for monitoring the database targets is enforced at all times.

High availability for Database Firewall can be accomplished in the following two ways:

  1. A pair of Database Firewall instances in Monitoring (Host Monitor) or Monitoring (Out of Band) modes.
  2. Multiple Database Firewall instances operating in Monitoring/Blocking (Proxy) mode.

Prerequisite

Ensure to create the Database Firewall instances, register them in the Audit Vault Server console, and configure them for high availability first. Later create monitoring points, register targets, and define policies later for the Database Firewall instances configured for high availability.

High Availability in Monitoring (Host Monitor) Or Monitoring (Out of Band) Modes

In this configuration:

  • High availability (primary and standby) is configured through Audit Vault Server.
  • In case of Monitoring (Host Monitor) mode, the Host Monitor Agent is configured to capture and forward the traffic to the primary and standby Database Firewall instances.
  • In case of Monitoring (Out of Band) deployment mode, the network switch is configured to mirror and forward the traffic to both the primary and standby Database Firewall instances.
  • The configuration of targets, monitoring points, and policies is automatically applied to the primary and standby Database Firewall instances by Audit Vault Server.

The Audit Vault Server collects network events from the primary or standby Database Firewall instance. If the Audit Vault Server is unable to contact the primary Database Firewall for a specified period of time (default of 10 minutes), then the Audit Vault Server collects the network events from the standby Database Firewall. The Audit Vault Server deletes the network events from both instances of Database Firewall after storing the data in the Audit Vault Server repository.

High Availability in Monitoring/Blocking (Proxy) Mode

Database Firewall instances deployed in Monitoring/Blocking (Proxy) mode, can be configured for high availability in the following ways:

  1. Active (primary) and passive (standby)
  2. Active and active

In active and passive configuration:

  • Client programs are configured to connect to the primary Database Firewall instance. If the primary Database Firewall instance is not reachable or is down, then they connect to the standby.
  • Audit Vault Server collects the network events from the Database Firewall instance (either active or passive) that receives the traffic.

In active and active configuration:

  • Multiple Database Firewall instances can be part of this configuration.
  • Client programs can connect to any of the active Database Firewall instances that are part of this configuration.
  • Once a client establishes a session with an active Database Firewall instance, it communicates with the same instance throughout the session.
  • Audit Vault Server collects the network events from all the active Database Firewall instances that are part of this configuration.

9.3.2 High Availability for Database Firewall in Host Monitor or Out of Band Modes

Learn how to configure a Database Firewall high availability pair in Host Monitor or Out of Band modes.

Prerequisites

  • Register both the Database Firewall instances in the Audit Vault Server console.
  • If you have Audit Vault Servers in high availability mode, then you must provide the primary and standby Audit Vault Server's IP address and certificate to each Database Firewall instance during registration.
  • Ensure there are no monitoring points configured on either of the Database Firewalls. In case there are any existing monitoring points, then they must be deleted.
  1. Log in to the Audit Vault Server console as an administrator.
  2. Click Database Firewalls tab.
  3. In the left navigation menu, select High Availability.
  4. Click Create.
  5. In the Create Resilient Pair dialog, select the Database Firewall instances for Primary and Standby fields from the drop down list.
  6. Click Save.

9.3.3 Swapping Roles Between Primary and Standby Database Firewalls

Learn to swap the roles of primary and standby Database Firewall instances in a high availability.

  1. Log in to the Audit Vault Server console as an administrator.
  2. Click the Database Firewalls tab.
  3. In the left navigation menu, select High Availability.
  4. Select the specific pair for which you want to swap roles.
  5. Click the Swap button.
  6. In the confirmation dialog, click OK.

9.3.4 Unpair Primary and Standby Database Firewalls

Learn to unpair Database Firewall instances in high availability.

  1. Log in to the Audit Vault Server console as an administrator.
  2. Click the Database Firewalls tab.
  3. In the left navigation menu, select High Availability.
  4. Select the specific pair of Database Firewalls that you want to unpair.
  5. Click the Unpair button.

    Note:

    For releases Oracle AVDF 20.4 and prior, click Break button.

9.4 High Availability for Database Firewall in Proxy Mode

Learn how to configure Database Firewall instances for high availability Monitoring / Blocking (Proxy) mode.

Oracle AVDF provides an option to set up the high availability configuration for multiple Database Firewall instances deployed in Monitoring / Blocking (Proxy) mode. These multiple instances are installed and configured independently.

Prerequisites

  • Install and register all Database Firewall instances that will be part of the high availability.

  • For each Database Firewall instance:

    • The configuration of the monitoring points must be same. For example Database Firewall instances DBFW1 and DBFW2 should have the same number of monitoring points and the configuration of these monitoring points should also be the same.

    • Deploy the same Database Firewall policy for a specific target. For example, deploy Database Firewall policy P1 (for target T1) on instances DBFW1 and DBFW2.

High availability configuration in proxy mode can be achieved in the following ways:

  • Through Client Configuration for Oracle Databases
  • Using DNS for Oracle and Other Database Types

9.4.1 Configuring High Availability for Database Firewall in Proxy Mode through Client Configuration

Learn how to configure high availability for two or more Database Firewall instances in proxy mode using tnsnames.ora for Oracle databases.

OCI (Oracle Call Interface) based clients use tnsnames.ora file to connect to Oracle database. The following parameters in this file should be modified as part of this configuration:

  1. ADDRESS_LIST

  2. CONNECT_TIMEOUT

  3. LOAD_BALANCE

ADDRESS_LIST

Include addresses of all the Database Firewall instances in the ADDRESS_LIST. The client programs connect to the first Database Firewall instance. In case of a failed attempt, the client connects to the next instance in the order.

For example:


dbfw1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.1)(PORT=1111))

                                 (ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.2)(PORT=2222)))(CONNECT_DATA=(SERVICE_NAME=dbfwdb)))

where:

dbfw1 is referred to as net_service_name.

Host = 192.0.2.1 and Host = 192.0.2.2 are the IP addresses of Database Firewall instances configured for high availability.

If you are using SQL*Plus client, then use the following command:

sqlplus <username>/<password>@<net_service_name>

The SQL*Plus client attempts to connect to the first Database Firewall instance with IP 192.0.2.1. In case the first instance is down or not reachable, then the client attempts to connect to the second Database Firewall instance with IP address 192.0.2.2.

CONNECT_TIMEOUT

Use CONNECT_TIMEOUT (seconds) parameter to quickly detect if the Database Firewall instance is down.

For example:

dbfw1=(DESCRIPTION=(CONNECT_TIMEOUT=10)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.1)(PORT=1111))(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.2)(PORT=2222)))(CONNECT_DATA=(SERVICE_NAME=dbfwdb)))

The client attempts to connect to the first Database Firewall instance with IP 192.0.2.1. In case the first instance is down or not reachable, then the client waits for the duration (seconds) mentioned in the CONNECT_TIMEOUT parameter. In the above example it is 10 seconds. Next the client attempts to connect to the second Database Firewall instance with IP address 192.0.2.2.

Note:

  • By default the value of CONNECT_TIMEOUT is 60 seconds.
  • Refer to Oracle Database Net Services Administrator's Guide for more details.

LOAD_BALANCE

Use LOAD_BALANCE parameter for client connections to connect to Database Firewall instances in a random sequence.

For example:

dbfw1=(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.1)(PORT=1111))(ADDRESS=(PROTOCOL=TCP)(HOST=192.0.2.2)(PORT=2222)))(CONNECT_DATA=(SERVICE_NAME=dbfwdb)))

Here clients will connect to either 192.0.2.1 or 192.0.2.2 in a random sequence.

Note:

  • When set to on, the LOAD_BALANCE parameter instructs clients to progress through the list of Database Firewall addresses in a random sequence. When set to off, instructs clients to try the addresses sequentially until one succeeds.
  • Refer to Oracle Database Net Services Administrator's Guide for more details.

9.4.2 Configuring High Availability for Database Firewall in Proxy Mode using DNS

Learn how to configure high availability for multiple Database Firewall instances in Monitoring / Blocking (Proxy) mode using DNS for Oracle and other database types.

Prerequisites

  1. Install and register Database Firewall instances.

  2. For each Database Firewall instance:

    • The configuration of the monitoring points must be same. For example Database Firewall instances DBFW1 and DBFW2 should have the same number of monitoring points and the configuration of these monitoring points should also be the same.

    • Deploy the same Database Firewall policy for a specific target. For example, deploy Database Firewall policy P1 (for target T1) on instances DBFW1 and DBFW2.

  3. Client programs should be able to connect to the configured DNS server.

Setup a fully qualified Domain Name in DNS

  1. Create a fully qualified domain name to represent IP addresses of the Database Firewall instances.

  2. Configure the selected DNS server as the name resolution server on the client hosts.

  3. Start the SQL*Plus connection as sqlplus /nolog without the username or password.

  4. In SQL*Plus run the command: connect <username>. Enter the password when prompted.

  5. Clients should use the fully qualified domain name in the connection string to connect to the Database Firewall instance. Use the following connection string:

    sqlplus <username/password>@<fully qualified domain name>:port/servicename
  6. DNS can be configured in one of the following ways:

    1. Configure DNS to always connect to an ordered list of Database Firewall instances (for example DBFW1, DBFW2, etc). If a client is not able to connect to the first instance (DBFW1), then it attempts to connect to the second instance (DBFW2).
    2. Configure DNS to use round-robin algorithm for connecting to Database Firewall instances.