J Troubleshooting Oracle Audit Vault and Database Firewall

Oracle Audit Vault and Database Firewall provides troubleshooting advice for a range of scenarios.

J.1 Audit Vault Agent or Host Monitor is not Upgraded to the New Release

Learn how to upgrade the Audit Vault Agent or Host Monitor Agent manually.

Problem

After upgrading to Oracle AVDF 20.1 or later, some of the Audit Vault Agents or Host Monitor Agents are not upgraded.

Symptom - 1

Audit Vault Agent is in STOPPED state after Audit Vault Server upgrade.

Symptom - 2

Host Monitor Agent is in NEEDS UPGRADE or UPDATE FAILED state after Audit Vault Server upgrade.

Solution - 1

The symptom indicates that the Audit Vault Agent has failed to auto upgrade during the Audit Vault Server upgrade. Execute the following steps as the user who installed Agent previously:

  1. Check for any Agent processes on the host machine. Ensure there are no Agent related processes currently running.
  2. Remove the existing agent.jar file and the Agent folder from the host machine.
  3. Download the new agent.jar file from the upgraded Audit Vault Server.
  4. Execute the following command:

    java -jar agent.jar [-d <AgentHome>]
  5. Verify the Agent is in RUNNING state.

Solution - 2

The symptom indicates that the Host Monitor Agent has failed to auto upgrade during the Audit Vault Server upgrade. Execute the following steps as root user:

  1. Check for any Host Monitor related processes on the host machine. Ensure there are no hostmonitor, hmdeployer, or hostmonmanager processes currently running.
  2. Navigate to the directory outside of hm where the Host Monitor is installed.
  3. Execute the following command to uninstall the Host Monitor:

    ./hm/hostmonsetup uninstall
  4. Download the new Host Monitor installable bundle from the Audit Vault Server console, for the specific platform on which it will be reinstalled.
  5. Extract the Host Monitor bundle inside the hm directory.
  6. Execute the following command to reinstall the Host Monitor in a root owned location:

    ./hostmonsetup install

J.2 Partial or No Traffic Seen for an Oracle Database Monitored by Oracle Database Firewall

Review the troubleshooting advice for when you see limited or no traffic for an Oracle Database that is monitored by Oracle Database Firewall.

Problem

I see no traffic, or only partial traffic, captured in reports for an Oracle Database monitored by the Database Firewall.

Solutions

Go through the following checks to find the trouble:

  1. In the Audit Vault Server, check that the report filters are set correctly, including the time slot.

  2. Check that the system time on the Database Firewall is synchronized with the time on the Audit Vault Server and the target system.

  3. Check that the target's network traffic is visible to the Database Firewall using the Live Capture utility on the firewall.

  4. Check that the Oracle Database service name or SID is used correctly. If you specified an Oracle Database service name in the monitoring point settings for this target, you will only see traffic for that service name. To see all traffic, remove the service name from the monitoring point settings.

    If you have entered a service name in the monitoring point, and see no traffic, check to see that the service name is entered correctly in the monitoring point settings.

    For monitoring points set to use monitoring only mode, the Database Firewall may be monitoring traffic for existing client connections to the database. Since these connections were in place before you deployed the Database Firewall, it will not be able to detect the service name you specify in the monitoring point. In this case, restart the client connections to the database.

  5. Check that the correct Database Firewall policy is deployed.

J.3 Agent Activation Request Returns 'host is not registered' Error

Read the troubleshooting advice if you receive a 'host is not registered' error.

Problem

I used the following two commands to register the Oracle Audit Vault Agent's host computer (where the agent is deployed), and to request Audit Vault Agent activation:

From the Audit Vault Server:

avcli> register host 'host_name'

From the host computer:

agentctl activate

But the agentctl activate command returns: Agent host is not registered

Solution

Your agent host may be multi homed. In this case, the agent hostname to IP address resolution may resolve to the NIC/IP that is not used by the agent while connecting to the AV server. To resolve this issue, try to register the agent host using the with ip option and then try activating the agent again.

From the Audit Vault Server, use the following command:

avcli> register host 'host_name' with ip 'host_ip_address'

If you still have issues, try finding the IP address used in the database session when you connect to the Audit Vault server from the agent host, using these commands:

Start SQL*Plus connection as sqlplus /nolog without the username or password.

In SQL*Plus execute the command: connect <user>. Enter the password when prompted.

sqlplus username/password@"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=Audit_Vault_Server_IP)(PORT=1521))(CONNECT_DATA= (SERVICE_NAME=dbfwdb)))"
sqlplus> select SYS_CONTEXT('USERENV','IP_ADDRESS') from dual;

Use the IP address from the above query to register your host.

J.4 Unable to Deploy Agent on the Secondary Audit Vault Server

Learn the resolution if you are unable to deploy an agent on a secondary Oracle Audit Vault server.

Problem

When I try to deploy the Audit Vault Agent on the secondary Audit Vault Server in a high availability pair, I get an error that the host is not registered.

Cause

After you pair two Audit Vault Servers for high availability, you do all configuration on the primary server in the pair only, including Audit Vault Agent deployment.

J.5 Failure While Building a Host Monitor or Collecting Oracle Database Trails

Learn what to do when you experience a failure while building host monitors or collecting Oracle Database trails.

Problem

This problem may manifest with various symptoms:

  • When I try to build a host monitor, the operation fails or the operation cannot locate the correct binaries.

  • When I try to collect audit data from an Oracle Database target, the operation fails.

  • The Audit Vault Agent cannot connect to the Audit Vault Server.

  • Audit trail does not start.

Solution

  1. Unset all environment variables except the following:

    • PATH

    • TERM

    • PS1

    • LANG

    • LC_*

    • JAVA_HOME

    Then run the java -jar agent.jar command again on the host machine.

  2. If you deployed the Audit Vault Agent in a Linux environment, then ensure that the host machine name appears in the /etc/hosts file.

J.6 'java -jar agent.jar' Failed on Windows Machine

Review the resolution procedures when the java -jar agent.jar command fails on Windows machines.

Problem

The command java -jar agent.jar failed on my Windows target machine, and I noticed in the log files that the Audit Vault Agent services installation/un-installation failed.

Solution

  1. Follow the instructions for unregistering the agent in Registering and Unregistering the Audit Vault Agent as a Windows Service.

    If Method 1 fails, then try Method 2.

  2. Run the java -jar agent.jar command again.

J.7 Unable to Install the Agent or Generate the agent.jar File

Determine the steps to perform if you are unable to install the agent or generate the agent.jar file.

Problem

Unable to install the Audit Vault Agent. Attempts to regenerate the agent.jar file are also unsuccessful.

Solution

Follow these steps to regenerate the agent.jar file:

  1. Log in to the Audit Vault Server through SSH as user oracle.

  2. Go to the directory /var/lib/oracle/dbfw/av/conf/ location.

  3. Delete the bootstrap.prop file.

  4. Execute the following command:

    /var/lib/oracle/dbfw/bin/avca configure_bootstrap

  5. Check the avca.log file that is available at /var/lib/oracle/dbfw/av/log/ to check if the above command was executed successfully.

  6. Switch the user (su) to avsys.

  7. Run the following query:

    select agent_gen_ts from file_repos where file_name='agent.jar';

  8. The above query displays the current time in case the agent.jar file is generated successfully.

J.8 Unable to Un-install the Oracle Audit Vault Agent Windows Service

Review the troubleshooting advice if you are unable to un-install the Oracle Audit Vault Agent Windows Service.

Follow the instructions for unregistering the Agent inRegistering and Unregistering the Audit Vault Agent as a Windows Service.

If Method 1 fails, then try Method 2.

J.9 Access Denied Error While Installing Agent as a Windows Service

Learn how to resolve access denied errors when installing Oracle Audit Vault agent as a Windows service.

Problem

I got an error during installation of Oracle Audit Vault Agent on Windows, and I noticed the following error in the AGENT_HOME\av\log\av.agent.prunsvr log file:

[2013-05-02 11:55:53] [info] Commons Daemon procrun (1.0.6.0 32-bit) started
[2013-05-02 11:55:53] [error] Unable to open the Service Manager
[2013-05-02 11:55:53] [error] Access is denied.
[2013-05-02 11:55:53] [error] Commons Daemon procrun failed with exit value:
7 (Failed to )
[2013-05-02 11:55:53] [error] Access is denied. 

Solution

The above message means that the logged in user does not have privileges to install the Audit Vault Agent as a Windows Service. If you get the above message, try launching the command shell with the Run As Administrator option, and then execute java -jar agent.jar in that command shell.

J.10 Unable to Start the Agent Through the Services Applet on the Control Panel

Review how to resolve being unable to start the agent through the services applet on the control panel.

Problem

I did the following:

  1. Installed the Audit Vault Agent using the java -jar agent.jar command.

  2. Activated the Audit Vault Agent.

  3. Started the Audit Vault Agent using the agentctl start -k key command.

    The agent started up and is in RUNNING state.

  4. Stopped the Audit Vault Agent.

  5. Tried to start the Audit Vault Agent using the Services Applet on the Windows Control Panel.

    The Audit Vault Agent errored out immediately.

Solution

This means that the Audit Vault Agent is configured to use a Windows account that does not have privileges to connect to the Audit Vault Server.

Take the following steps:

  1. Go to Control Panel, then to Services Applet.

  2. Select the Oracle Audit Vault Agent service.

  3. Right click and select the Properties menu.

  4. Click the Log on tab.

  5. Select This account: and then enter a valid account name and password.

  6. Save and exit.

  7. Start the Audit Vault Agent through the Services Applet.

J.11 Error When Starting the Agent

Resolved errors that occur when starting the agent.

Problem

After I installed the Audit Vault Agent, I set the username and password in the OracleAVAgent Windows Service Properties Log On tab. However, when I try to start the OracleAVAgent service, I see the following error in the Agent_Home\av\log\av.agent.prunsvr.date.log file:

[info]  Commons Daemon procrun (1.0.6.0 32-bit) started
[info]  Running 'OracleAVAgent' Service...
[info]  Starting service...
[error] Failed creating java 
[error] ServiceStart returned 1
[info]  Run service finished.
[info]  Commons Daemon procrun finished

Solution

This means that the OracleAVAgent service is not able to launch the Java process. Try the following:

  1. Uninstall all JDKs and/or JREs in the system.

  2. Reinstall JDK SE or JRE and then start the OracleAVAgent service.

  3. If this doesn't help, you can install 32 bit JDK SE or JRE and then start the OracleAVAgent service.

J.12 Error When Running Host Monitor Setup

Review the resolutions for errors that occur when running host monitor setup.

Problem

I am setting up a Host Monitor. When I run the command bin/hostmonsetup install, the following error is displayed:

Failed to generate executables for Host monitor

Solution

This means the host computer does not have the required libraries for the Host Monitor. Install the required libraries mentioned in Host Monitor Requirements.

Even after installing the required libraries, if the Host Monitor installation fails with above error message, then examine the makelogerror file which is available in the Host Monitor installation directory. The following errors, may appear in the file:

/bin/ld: cannot find -laio

/bin/ld: cannot find -lssl

/bin/ld: cannot find -lcrypto

/bin/ld: cannot find -lnsl

To resolve the issue on Linux (64 bit) systems, create a symbolic link (or symlink) in /lib64/ location. For error /bin/ld: cannot find -lssl as an example, follow these steps:

  1. Search for libssl in /lib64/ location, which is the actual SSL library and not a symbolic link.
  2. Create a symbolic link in /lib64/ location with the name libssl.so.1 pointing to the actual SSL library.
  3. Create another symbolic link in /lib64/ location with the name libssl.so pointing to libssl.so.1.
  4. Repeat these steps for all such errors in the makelogerror file.

J.13 Alerts on Oracle Database Targets Are Not Triggered for Extended Periods of Time

Learn what to do when alerts on targets are not triggered for a long time.

Problem

I configured an Oracle Database target to audit to XML files, configured an audit trail in Oracle AVDF of type DIRECTORY, and then configured an alert to trigger on certain events. My alert did not get triggered for a long time.

Solution

This issue can occur if the Oracle Database target is not flushing the audit records to the file immediately. Contact Oracle Support in order to access support note 1358183.1 Audit Files Are Not Immediately Flushed To Disk.

J.14 Error When Creating an Audit Policy

Resolve errors that can occur when you create an audit policy.

Problem

I received this error message when I tried to create a new audit policy setting for Oracle Database:

-ORA-01400: cannot insert NULL into ("AVSYS"."AUDIT_SETTING_ARCHIVE_MAP"."ARCHIVE_ID")

Cause

The Oracle Database must have at least one audit policy setting before you can create and provision new audit settings using Oracle Audit Vault and Database Firewall. Oracle Database comes with a predefined set of audit policy settings. You must not manually remove these settings. If the audit settings have been removed, then you can manually create at least one audit setting in the Oracle Database. Then try again to create new audit settings using Oracle Audit Vault and Database Firewall.

See Also:

Oracle Database Security Guide for detailed information on Oracle Database auditing.

J.15 Connection Problems When Using Oracle Database Firewall Monitoring and Blocking

Resolve the connection problems that might occur when using Oracle Database Firewall monitoring and blocking.

Problem

In monitoring and blocking mode, my client application cannot connect to the target database.

Solution 1

  1. Log in as root on the Database Firewall server.

  2. Run this command using the target database IP address or host name:

    ping -I secured_target_ip_address_or_hostname

    If you do not receive a response, then ensure that the DNS is configured on Oracle Database Firewall.

    If a response is received, check:

    • The firewall policy to ensure that it is not blocking the connection attempt.

    • The client connection settings to ensure that the client is attempting to connect to the correct target database.

Solution 2

If your client application computer is on a different subnet than the target database, then see document number 1566766.1 on My Oracle Support https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1566766.1.

J.16 Audit Trail Does Not Start

Learn the resolution to use when the audit trail does not start.

Problem

An audit trail does not start. For example, in the Audit Vault Server console, in the Audit Trails page, the Collection Status column indicates that the trail is Stopped or Unreachable.

Solution

When a trail does not start, you can show the associated error in two ways:

  • In the Audit Vault Server console:

    1. Click the Targets tab, and then from the Monitoring menu, click Audit Trails.

    2. Click the Actions button, and then click Select Columns.

    3. From the left-hand box, double-click Error Message so that it moves into the Display in Report box on the right.

    4. Click Apply.

    The Error Message column is displayed on the Audit Trails page and contains the error message for the stopped trail.

  • On the Audit Vault Agent host computer:

    1. Go to the logs directory:

      cd %agenthome%/av/logs

    2. Run the following:

      grep -i 'error|warning|fail' *

    The error messages should indicate the cause of the problem.

If the cause is still unclear, or the grep command returns no results, raise an SR with Oracle Support and include Audit Vault Agent log files.

See also document number 1566766.1 on My Oracle Support https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1566766.1.

J.17 Cannot See Data for Targets

Learn what to do when you cannot see the data for a target.

Problem

Data for my Target does not appear on reports.

Solution

If you cannot see the data you expect to see in the Audit Vault Server, you can troubleshoot by trying one or more of the following:

  • Confirm that Audit Vault Agent hosts are up and that the Audit Vault Agents are running.

  • Confirm that audit trails are running and that the audit trail settings match the audit configuration of the Target database

    For example, the audit trail configuration in Oracle Audit Vault and Database Firewall should have the correct trail type and location.

  • Check the audit policy on the target to ensure you are auditing the activity that you are expecting to see in the reports.

  • Check the firewall policy to ensure you are logging the activity you are expecting to see in reports.

  • Clear any time filters on reports, and then check time settings on the target and on the AVS. If the time is incorrect, the time recorded against audit events will not be accurate. As a result, the audit events may not be displayed in the time window you expect.

  • Check the /var/log/messages file on Audit Vault Server and on the Database Firewall for errors.

  • Check that the Database Firewall monitoring point is created and running.

  • Check that the Database Firewall monitoring point traffic source is correct.

  • If the Database Firewall is in monitoring only mode, use the Database Firewall Live Capture utility to verify that traffic is being seen on the relevant traffic source. If necessary, use the File Capture utility to capture traffic to a file and verify (using Wireshark or a similar product) that the traffic being captured is consistent with the settings in the Target Addresses section of your Target configuration.

  • Check that you have used the correct Oracle Database service name when configuring the Target Address in your Target configuration.

    Also, have you included all available Oracle Service names in the Target Addresses section of the Target configuration? Unless you intend to define a different firewall policy for each service name, Oracle recommends you omit service name and use only IP address and TCP ports in Target Addresses.

  • On the Database Firewall, check the /var/log/httpd/ssl_access_log file to confirm that the Audit Vault Server is collecting logs.

  • On the Audit Vault Server, check the /var/dbfw/tmp/processing* directories and make sure kernel*.dat files are arriving in the directory, and then being deleted once the Audit Vault Server has processed them.

  • On the Audit Vault Server, check that the mwecsvc process is running. For example, run the command:

    ps -ef | grep mwecsvc

    If the process is not running, use this command to restart it:

    service controller start

J.18 Problems Pairing Oracle Database Firewall and Oracle Audit Vault Server

Review the procedure to follow when you have problems pairing Oracle Database Firewall with Oracle Audit Vault Server.

Problem

I encounter errors when I try to associate a Database Firewall with the Audit Vault Server.

Solution

Check the following:

  • Ensure that you have entered the correct Audit Vault Server IP address in the Database Firewall Certificate page.

    Log in to the Audit Vault Server console, and click the Settings tab. Then click the Certificate tab on the main page.

  • Ensure that both the Database Firewall server and the Audit Vault Server are configured to use NTP and that each machine is synced to the NTP time server.

J.19 User Names Do Not Appear on Database Firewall Reports

Learn what to do when names do not appear on Database Firewall reports.

Problem

When I generate a Database Firewall report, I do not see user names.

Solution

Check the following possibilities:

  • If this is occurring for a Microsoft SQL Server database target, check to make sure that retrieve session information is turned on.

  • This problem may be caused by bad network traffic arriving at the Database Firewall. Check for duplicate or missing network packets. You can use the Database Firewall's Live Capture utility to capture network traffic to a file and analyze it.

Note:

Sometimes unknown_username is displayed in the User field of Database Firewall reports for SQL server. This can be resolved by enabling Retrieve session information from target DB option under the Advanced tab for the Database Firewall monitoring point. The report may also display unknown_osusername in the OS User field of Database Firewall reports for SQL server. This information is available to Database Firewall only if the client uses Windows authentication or a trusted connection.

J.20 Alerts Are Not Generated

Review the resolution to use when alerts that you created are not generated.

Problem

Alerts I have created are not being generated.

Solution

Try the following:

J.21 Problems Retrieving or Provisioning Audit Settings on Oracle Target

Learn what to do when you encounter problems while retrieving or provisioning Oracle target audit settings.

Problem

I have a problem either retrieving audit settings form an Oracle Database target, or provisioning audit settings to an Oracle Database target.

Solution

If you have problems retrieving audit settings, try the following:

  • Check the job status of the retrieval job for errors:

    Log in to the Audit Vault Server console as an auditor, click Settings, and then click Jobs in the System menu.

  • Ensure you have entered the correct connect string in the Oracle Database's target configuration:

    Log in to the Audit Vault Server as an administrator, click the Targets tab, and then click the name of this Oracle target. Check the Target Location field for the connect string.

If you have problems provisioning audit settings, and the Oracle Database target has Database Vault enabled, confirm that the Oracle Audit Vault and Database Firewall user you created on this database has the AUDIT SYSTEM and AUDIT ANY privileges.

J.22 Operation Failed Message Appears When Attempting to Enable Oracle Audit Vault and Database Firewall Policies

Learn how to resolve operation failures when you try to enable Oracle Audit Vault and Database Firewall policies.

Problem

I configured Oracle Audit Vault and Database Firewall for a backup and restore operation. After I completed the procedure, I could not enable an Oracle Audit Vault and Database Firewall policy. The error message Operation failed. Please contact Oracle Support appeared.

Solution

During the backup and restore process, Oracle Audit Vault and Database Firewall must perform a restart of the Oracle Audit Vault Server database. The internal tool Java Framework may need to be restarted. To remedy this problem:

  1. Log in to Oracle Audit Vault Server.

  2. At the command line, run the following command to check the status of the Java Framework:

    /usr/local/dbfw/bin/javafwk status
    
  3. If the output says Java framework process is stopped, then restart it as follows:

    /usr/local/dbfw/bin/javafwk start 

J.23 Out of Memory Error Message During Restore

Learn the resolution when you receive an out of memory error message during a restore.

Problem

Encounter out of memory error while performing restore task.

Solution

Prior to initiating the restore task, ensure that the RAM size and Disk size in the new system is equal or bigger than the original system. This ensures that the out of memory error is not encountered while performing the restore task.

J.24 JAVA.IO.IOEXCEPTION Error

Learn how to resolve a JAVA.IO.IOEXCEPTION error.

Problem

SSL peer shuts down incorrectly with the following error:

JAVA.IO.IOEXCEPTION: IO ERROR:SSL PEER SHUT DOWN INCORRECTLY

Solution

  1. Access the target through SSH.

  2. Change to the following location using the command:

    cd $ORACLE_HOME/network/admin

  3. Edit the sqlnet.ora file. Add parameter sqlnet.recv_timeout=100000 in the file.

  4. Restart the target listener.

  5. Once the target listener is started, start the agent, and the audit trail.

J.25 Failed to Start ASM Instance Error

Learn what to do when you receive a Failed to start ASM instance error.

Problem

The avdf-upgrade --confirm command stops and results in an error. The command may fail for many reasons. The error mainly occurs due to failure in starting or stopping of a service.

The following is an example of Failed to start ASM instance error:

{{{ 
[support@avs00161e637973 ~]$ su - root 
Password: 
[root@avs00161e637973 ~]# /usr/bin/avdf-upgrade --confirm 
Please wait while validating SHA256 checksum for 
/var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso 
Checksum validation successfull for 
/var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso 
Mounting /var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso on /images 
Successfuly mounted /var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso on /images 
Starting Oracle High Availability Service 
2016-08-05 15:32:09.097: 
CLSD: Failed to generate a fullname. Additional diagnostics: ftype: 2 
(:CLSD00167:) 
CRS-4639: Could not contact Oracle High Availability Services 
CRS-4000: Command Start failed, or completed with errors. 
Starting ASM instance 
Error: Failed to start ASM Instance 
Unmounted /var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso on /images 
Failed to start ASM Instance 
}}}

Solution

Rerun the command avdf-upgrade --confirm

Executing this command again will get past the Failed to start ASM instance error.

J.26 Internal Capacity Exceeded Messages Seen in the /var/log/messages file

Learn how to resolve Internal capacity exceeded messages that appear in the /var/log/messages file.

Problem

Not all the expected traffic is being captured or logged by the Database Firewall, and error messages are present in the /var/log/messages file containing the text Internal capacity exceeded.

Solution - 1

Increase the processing resources available for the target on which the issue is observed through the setting of the MAXIMUM_ENFORCEMENT_POINT_THREADS collection attribute.

Solution - 2

The size of the buffer used for inter-process communication on the Database Firewall can be increased to improve throughput, though at the cost of more memory being allocated by the relevant processes. Please note that this setting is in units of Megabytes, and has a default value of 16. To change the configuration for this value execute the following procedure:

  1. Log in to the Audit Vault Server console as the root user.

  2. Edit the file /usr/local/dbfw/etc/dbfw.conf. Look for an entry with the key IPC_PRIMARY_BUF_SIZE_MB. If it exists, this is the line to change. If it does not exist, add a new line beginning with IPC_PRIMARY_BUF_SIZE_MB.

  3. Change the IPC_PRIMARY_BUF_SIZE_MB line to reflect the required buffer size. For example, if you wished to change the buffer size to 24 megabytes, the configuration line should be IPC_PRIMARY_BUF_SIZE_MB="24". Save the changes.

  4. From the command line restart the Database Firewall processes so that the new setting is used with the command line /etc/init.d/dbfw restart.

There is also a second setting available to alter the maximum size that the inter-process communication buffer can grow to. It's units are in megabytes, and has a default value of 64 megabytes. To change the configuration for this value execute the following procedure:

  1. Log in to the Audit Vault Server console as the root user.

  2. Edit the file /var/dbfw/va/N/etc/appliance.conf, where N is the number of the Database Firewall monitoring points in question. Look for an entry with the key IPC_BUF_SIZ_MB. If it exists, this is the line to change. If it does not exist, add a new line beginning with IPC_BUF_SIZ_MB.

  3. Change the IPC_BUF_SIZ_MB to reflect the desired maximum buffer size. For example, if you wished to change the buffer size to 80 megabytes, the configuration line should be IPC_BUF_SIZ_MB="80". Save the changes.

  4. From the command line restart the Database Firewall processes so that the new setting is used with the command line /etc/init.d/dbfw restart.

If the problem persists and after altering the above settings the Internal capacity exceeded error is still encountered, then further investigation by support is required.

Perform the following:

  1. Log in to the Audit Vault Server console as the root user.

  2. Edit the file /usr/local/dbfw/etc/logging.conf

  3. Find the line log4j.logger.com.oracle.dbfw.Metrics=ERROR

  4. Comment out this line by placing a # character at the beginning of the line log4j.logger.com.oracle.dbfw.Metrics=ERROR. Save the changes.

  5. From the command line restart the Database Firewall processes so that the new setting is used with the command line /etc/init.d/dbfw restart

  6. Leave the Database Firewall running for several hours under load even while the Internal capacity exceeded error is still encountered.

  7. After this period, get the diagnostics output from the Database Firewall as detailed in MOS note How to Collect Diagnostic Logs From Audit Vault Server (Doc ID 2144813.1). Provide the diagnostics output to support for further analysis.

J.27 First Archive Or Retrieve Job After Upgrade

Learn what to do if after an upgrade, the first archive or rervireve job submission displays the status of Starting.

Problem

After upgrade the first archive or retrieve job submission may display the status as Starting.

Solution

Submit the job again. This is a known issue and subsequent submission of job succeeds.

J.28 Audit Vault Agent Installation Fails After HA Pairing Or Separation

Learn what to do after the Oracle Audit Vault installation fails after an HA pairing or separation.

Problem

Installation of Audit Vault agent fails after performing pairing or separation (un-pairing) of Oracle Audit Vault server.

The following command generates agent debug logs during agent installations.

java -jar agent.jar -v

Symptoms

The following errors may be found during agent installation in the agent log file:

PKIX path validation failed

signature check failed

Solution

After the pairing or separating of Oracle Audit Vault servers, you must download the Audit Vault agent from the GUI and install the agent again after removing the existing Audit Vault Agent.

If the Audit Vault agent fails to install after pairing or separating of Audit Vault server, then install the Audit Vault agent using -v option.

To resolve the above errors, follow the steps mentioned below:

  1. Log in to the Audit Vault server as user root.

  2. Run the following script to generate a new agent.jar file.

    /usr/local/dbfw/bin/priv/update_connect_string_ip.sh

  3. Download the new agent.jar file from the GUI.

  4. Install the newly downloaded agent.jar file.

J.29 Error in Restoring Files

Learn what to do when you encounter errors while restoring files.

Problem

An attempt to restore the data files results in a failure. The restore job completes successfully, however the data files are not restored. There is no information in the restore job log file.

Solution

Check for the following to troubleshoot the issue:

  • The restore policy must follow the guidelines listed under the section Configuring Archive Locations and Retention Policies.

  • Check the tablespace that needs to be archived and the corresponding tablespace that needs to be purged as per the policy defined.

  • Restoring data into empty tablespaces is not possible. Check accordingly.

  • In case the tablespace enters the delete period, it is deleted automatically from Oracle Audit Vault Server.

  • Every tablespace is uniquely identified by the month it moves offline and the month during which it is purged. They are created automatically based on the policies that you create.

  • When the retention policy is changed, the new policy is applied to the incoming data immediately. It does not affect existing tablespaces that adhere to the old policy.

  • You can archive the tablespace when it enters the offline period.

  • After restoring the tablespace, it is online. Once it is released, it goes offline. The tablespace must be rearchived once released.

J.30 DB2 Collector Fails Due to Source Version NULL Errors

If the DB2 collector fails due to source version NULL errors, then follow these steps.

Problem

The following error or trace is displayed in the collector log file.

Caused by: java.lang.ClassNotFoundException:

sun.io.MalformedInputException

at java.net.URLClassLoader.findClass(Unknown Source)

at java.lang.ClassLoader.loadClass(Unknown Source)

Solution

Check the Java version on the host system This failure is due to Java SE version 8. Attempt to use Java SE 7.

Note:

This issue may be encountered in releases prior to 12.2.0.11.0.

J.31 DB2 Collector Fails Due to Database Connection or Permission Issues

If the DB2 collector fails due to database connection or persmission issues, then follow these steps.

Problem

The following error or trace is displayed in the collector log file.

Caused by: oracle.ucp.UniversalConnectionPoolException: Cannot get Connection from Datasource: java.sql.SQLSyntaxErrorException: [Audit Vault][DB2 JDBC Driver][DB2]<User> DOES NOT HAVE PRIVILEGE TO PERFORM OPERATION EXECUTE ON THIS OBJECT NULLID.DDJC360B

Solution

Run the following command for successful execution of DB2 collector:

grant execute on package NULLID.DDJC360B to <User> (user while registering the target)

J.32 ORA-12660 Error While Registering Target

Learn how to resolve the ORA-12660 error.

Problem

Audit Vault agent fails with ORA-12660 error.

Solution

The server encryption is set to REQUIRED in on-premises by default. Set the server encryption to ACCEPTED or REQUESTED or REJECTED.

Note:

REJECTED is not a recommended option. The following table describes these options in detail.

Table J-1 Server Encryption Types

Option Description

ACCEPTED

The server does not enable both encrypted and non-encrypted connections. This is the default value in case the parameter is not set.

REJECTED

The server does not enable encrypted traffic.

REQUESTED

The server requests encrypted traffic if it is possible, but accepts non-encrypted traffic if encryption is not possible.

REQUIRED

The server accepts only encrypted traffic.

J.33 Audit Trail Performance Issues Occur After Audit Vault Server Upgrade

Learn what to do when audit trail performance issues occur after upgrading Oracle Audit Vault Server.

Problem

You might experience audit trail performance issues after upgrading Oracle Audit Vault Server.

Solution

The audit_trail_id_idx index that is created resolves the performance issues encountered. However, you must retain sufficient disk space if there is large amount of event data for the period prior to upgrading Oracle Audit Vault Server. The amount of disk space required is about 5% of the total event log data size.

J.34 Failures Due to Dropping Users

Learn how to resolve failures that occur when dropping users.

Problem

Failed to drop the user with an error message and the user was not listed in the Audit Vault Server GUI.

Solution

Contact Oracle Support for the best workaround and to drop the user manually using SQL*Plus.

J.35 Failure of Agent Automatic Upgrades

Learn what to do when agent automatic upgrades fail.

Problem

The automatic upgrade of the Agent fails with the following error. This is because the Agent is unable to connect to the Audit Vault Database.

Message: Exception occurred while updating Agent.
Cause: Unable to connect to AV Server.
Note: Agent will try to re-connect automatically in 10 seconds.

Solution

The Agent attempts to connect to the Audit Vault Database and auto upgrade after 10 seconds. Check the Oracle Audit Vault Database connection or contact Oracle Support.

J.36 Some Services May Not Start After Backup

Learn what to do when services fail to start after a backup.

Problem

The system may not be stable after a cold backup operation failed to complete.

Solution

Oracle recommends that you reboot the system if there is a failure while performing a cold backup operation.

J.37 Data Overflow Issues in the Oracle Audit Vault UI

Learn how to resolve data overflow issues in the Oracle Audit Vault UI.

Problem

The Recently Raised Alerts Report region appears on your dashboard and displays the list of alerts with data overflowing in the Audit Vault GUI. This may occur when you launch the GUI using Internet Explorer and the Microsoft Windows Server operating system.

Solution

To fix this issue and to display the data properly on the Audit Vault GUI, you should make minor changes to the Internet Explorer browser settings. Press F12 and click the Emulation tab.

Change the Document mode and Browser profile fields from the default settings. For example, change the Document mode value to 10 from the drop down menu and change the Browser profile field to Desktop.

J.38 Oracle Audit Vault Agent is Unreachable and the Transaction Log Audit Trail is Frozen in Starting Status

Learn what to do when the Oracle Audit Vault Agent is unreachable and the transaction log audit trail is frozen in Starting status.

Problem

The status of Oracle Audit Vault Agent is unreachable from the AV GUI. The status of the Transaction Log audit trail persistently remains in the Starting status.

This may be due to a user application that is blocking the creation of streams by ORAAUDIT user.

Symptom

The Transaction Log audit trail does not start. The following information may be found in the thread dump that is taken using jstack tool:

oracle.av.platform.agent.collfwk.impl.redo.RedoCollector.sourceSetup(RedoCollector.java:634) 

Solution

Terminate the user application that is blocking the creation of streams. Restart the Transaction Log audit trail.

J.39 Scheduled PDF or XLS Reports Result in a Hung State

To resolve a hung state that occurs for scheduled PDF or XLS reports, follow these recommendations.

Problem

Scheduled PDF or XLS reports remain incomplete for an extended period of time or ramin in q RUNNING state.

Solution

You can schedule reports to be sent to other users in PDF or XLS formats. Avoid triggering or scheduling concurrent long-running reports at the same time. Producing PDF and XLS reports occupies a lot of system resources because there is a significant amount of data involved. Scheduled concurrent long-running reports can remain in a hung state indefinitely. The reports must be scheduled with staggered intervals in between. For example, run the reports at intervals of 5, 10, or 20 minutes.

J.40 Pending Reports Remain in Scheduled Status

To resolve pending reports that remain in scheduled status, follow these steps.

Problem

Many reports are stuck in scheduled or pending status. These reports may never be completed and may be stopped.

Solution

This may be due to an issue with the Java Framework process in the background. Use these steps to check and resolve this issue:

  1. Log in to the CLI as support user.

  2. Switch to root user using the command:

    su root

  3. Run the following command to check the status of the Java Framework:

    systemctl status javafwk

  4. Stop the Java Framework even if it is running. Run the following command:

    systemctl stop javafwk

  5. Run the following command to start the Java Framework:

    systemctl start javafwk

  6. Run the following command to restart the Java Framework:

    systemctl restart javafwk

Use the following procedure to check the status of the reports from the operating system logs after running one of the procedures mentioned above and restarting the Java Framework:

  1. Log in to AVCLI as admin user.

  2. Run the following command to enable diagnostics for the reports:

    ALTER SYSTEM SET loglevel=ReportLog:DEBUG|JfwkLog:DEBUG;

  3. The diagnostics can also be enabled using the Oracle Audit Vault Server console by following these steps:

    1. Log in to the console as admin user.
    2. Click Settings tab.
    3. Click on Diagnostics on the left navigation menu.
    4. Select Debug against Report Generation.
    5. Click Save.
  4. Run a PDF report. For example, Activity Overview.

    1. Log in to the Oracle Audit Vault Server console as auditor.
    2. Click Reports tab.
    3. Click Activity Reports under Built-in Reports.
    4. In the Activity Reports tab on the screen, you can schedule a report and view the generated report.
  5. After a while, check on the /var/lib/oracle/dbfw/av/log file. For example, av.report* file. It contains the PDF/XLS report generation debug logs.

J.41 Audit Vault Log Displays a Message to Install WinPcap and OpenSSL

To resolve the Audit Vault log message to install WinPcap and OpenSSL, follow these steps.

Problem

The Host Monitor can collect audit data from Windows 2016 servers. A message displays alerting you to install WinPcap and OpenSSL.

Solution

A set of DLL files may be causing issues. Run the following procedure to resolve this problem:

  1. Search for the following files in the system:

    • ssleay32.dll
    • libeay32.dll
    • wpcap.dll
    • packet.dll
  2. Append the file names with the .bk format notation.

  3. Go to Control Panel then to Uninstall Programs and uninstall OpenSSL and WinPcap.

  4. Reinstall WinPcap and OpenSSL 1.0.2.q (64-bit). The DLL files are restored to Windows system folder.

  5. Check the Control Panel to verify that these two programs are installed.

  6. Go to C:\Windows\System32 or C:\Windows\SysWOW64 folders and search for the above four DLL files. At least one file for each DLL must be present without the .bk extension.

  7. Go to the OpenSSL installation location and search for libssl-1_1-x64.dll and libcrypto-1_1-x64.dll files. One for each type is available.

  8. Upon confirmation, add the C:\Windows\System32 or C:\Windows\SysWOW64 to the path variable.

  9. Restart the trail.

  10. If the network audit does not start, then check the collfwk logs present at <AgentHome>\av\log location. If the following message is available in the collfwk log, then check the Host Monitor logs present at <AgentHome>\hm\log location.

    <AgentHome> refers to the Audit Vault Agent installation directory.

    Note:

    Continue with the remaining steps if your installation is 12.2.0.10.0 or before. The steps are not required for release 12.2.0.11.0 and later.
  11. If the following message is available in the Host Monitor log, then execute the remaining procedure:

    Invalid AVS Credentials provided
  12. Open the av/conf/bootstrap.prop file.

  13. Copy the following line:

    CONNECT_STRING_PARAM_POSTFIX=9999
  14. Paste this line in the hm/bootstrap.prop file.

  15. Restart the trail.

  16. In case the network audit trail starts without any errors, then the collection status on the Audit Vault Server console confirms the same.

  17. Navigate to AVAUDIT then to Target then Firewall Policies and, finally, Log All.

  18. Connect to the target database instance using SQL Developer, or any other tool.

  19. Generate the traffic for collecting data.

  20. It must be recorded in the reports of the event_log table.

J.42 Host Monitor Agent Fails to Start

Learn what to do when the Host Monitor Agent fails to start.

Problem

The Host Monitor network trail does not start after installation. The collection framework (collfwk) log file contains one of the following errors:

  • java.io.IOException: Cannot run program "<AgentHome>/hm/hostmonmanager" (in directory "<AgentHome>/hm"): error=13, The file access permissions do not allow the specified action.
  • HMCommandExecutor : startTrail :  binary is not found here: <AgentHome>/hm/hostmonmanager

Solution

This issue may arise due to insufficient privileges while starting Host Monitor. Ensure the Audit Vault Agent user belongs to the group that owns hm (Host Monitor installation) directory. Also ensure that the group that owns Host Monitor installation (hm) directory has read and execute permission on the hm directory and execute permission on hostmonmanager binary.

Note:

  • AgentHome is the Audit Vault Agent installation directory.

  • hm is the Host Monitor installation directory.

J.43 Error OAV-47409 While Managing Archive Locations

Learn what to do when you receive the OAV-47409 error while managing archive locations.

Problem

The following error message displays in the Auto Archive Message column under Manage NFS Locations tab:


OAV-47409: Absolute path does not exist on remote filesystem
ORA-06510: PL/SQL: unhandled user-defined exception

The configured path of the archive location is either missing or outside of the remote filesystem.

Solution

The value under Auto Archive Order column is set to 0 [zero]. The system has set this value as the archive location is problematic. You must ensure that the NFS location issue is resolved to a valid directory on the remote filesystem. Upon resolving this issue, set the value under Auto Archive Order column to 1 or higher. This sets the appropriate priority for the auto archive order.

J.44 Error OAV-47402 While Defining Archive Locations Using NFS Mount Point

Learn what to do when you receive the OAV-47402 error while defining archive locations.

Problem

An error is observed after registering the archive location using NFS mount point through AVCLI. The created remote file system shows inaccessible when running the SHOW STATUS command. The following error is observed when running ALTER REMOTE FILESYSTEM <file system name> MOUNT command. However, the process of defining or creating the archive location is successful.

OAV-47402: Unable to mount export /exabackup from host <host Ip address>

Solution

This issue is observed when using NFS version v3 only. Reach out to the NAS storage support or NFS administrator support team to verify if the mount point in the NFS server is properly configured. It must support both v3 and v4 to integrate with Oracle AVDF.

Note:

NFS version v3 only is not supported for Oracle AVDF releases 20.3 and prior. It is supported starting Oracle AVDF release 20.4.

Follow the steps documented in MOS (Doc ID 2466520.1) to verify if the mount point in the NFS server is properly configured.

See Defining Archive Locations for complete information.

J.45 Audit Trail Stopped After Relocating Windows Event Log Files

Use this procedure when the audit trail stops after you relocate the Windows event log files.

Problem

Windows event log relocation causes audit trail to be stopped.

Solution

Follow this procedure to resolve this problem:

  1. Stop the audit trail.
  2. Drop the audit trail.
  3. Restart the audit trail. The new trail recognizes the new location for event logs.

J.46 Missing or Incomplete Client Information in Oracle Database Firewall Logs

Learn how to resolve missing or incomplete client information in Oracle Database Firewall logs.

Problem

Empty client information in the Oracle Database Firewall logs after upgrading Oracle Audit Vault and Database Firewall. The logs that are generated are missing some of the client information such as the user name.

Note:

This issue occurs only when you are in DAM mode deployment of Oracle Database Firewall. You will not experience this issue in the Proxy mode deployment.

Cause

Oracle Database Firewall records information that is related to the TCP sessions during inspection and it saves this data to disk. This recorded information includes client user names and other metadata about the connection. When Oracle Database Firewall processes are restarted after a configuration change or an upgrade, Oracle Database Firewall continues to generate logs accurately by re-reading this cached information.

The format of the cache file has changed in the recent releases. Oracle Database Firewall may not be able to read the file in the old format. Therefore, existing client connections to the database that were established before performing the upgrade may not retain certain information such as client user names. This can lead to logs missing information such as the client username.

Solution

Restart the database clients.

J.47 Network Audit Trail Does Not Start on Unix Platforms

Learn the resolution when the network audit trail failst to start on Unix platforms.

Problem

Network audit trail does not start on Unix platforms.

Symptoms

  • The Oracle Audit Vault Server console displays the following error:

    Unable to start Host Monitor process

  • The collection framework log displays the following error:

    <Host Monitor home>/hostmonmanager binary is not found here

Solution

  1. Connect to the host machine on which the Audit Vault Agent and Host Monitor are installed.
  2. In the Agent Home location there is an hm symlink pointing to Host Monitor installation location.
  3. Run the following command from the Agent Home as the user who installed Oracle Audit Vault Agent:

    ls -lrt hm

  4. Check if it is possible to list the contents of Host Monitor install directory.
  5. Check the permission of all directories in the hierarchy of the path under which Host Monitor is installed.

    Note:

    The entire directory hierarchy must be owned by the root user. All of the directories in this hierarchy must have read and execute permission for other users or groups, but not write permission.
  6. Grant the necessary permissions as stated above.
  7. Restart the network audit trail.

J.48 Issues with Retrieving Session Information Through Clients Connecting to Microsoft SQL Server

Learn what to do when you have issues retrieving session information through clients that connect through Microsoft SQL Server.

Problem

Database Firewall is unable to retrieve session information through some clients (for example, MS SQL Server Management Studio) as the information is encrypted. You can retrieve session information for non Oracle databases to obtain the name of the database user, operating system, and client program that originated a SQL statement.

Symptom

Audit Reports show unknown user names and unknown program names where the target is Microsoft SQL Server.

Solution

Ensure the following steps are accurate while registering Microsoft SQL Server as a target.

  1. In the User Name field, enter the user name of the system administrator.
  2. In the Password field, enter the password of the system administrator.
  3. In the Host Name / IP Address field, enter the IP address of the SQL Server.
  4. In the Port field, enter the port of the SQL server listening port.
  5. In the Service Name field, enter a valid database service name on SQL Server. In case the database service name is not correct, then SQL server DDI requests fail on the SQL Server with invalid request error.

J.49 Performance Issues Due to High Memory Usage

Learn how to address performance issues in Oracle AVDF with very large deployments.

Problem

Audit Vault Server in large deployments may have performance issues due to increased memory usage.

Solution

  • Ensure the Audit Vault Server is sized as per the sizing guidelines documented in Audit Vault and Database Firewall Best Practices and Sizing Calculator for AVDF 12.2 and AVDF 20.1 (Doc ID 2092683.1).
  • Audit Vault Server has Transparent Huge Pages set by default which should work in most cases. However, in some cases it has to be disabled by setting transparent_hugepages to never. This helps in improving the performance. For detailed the steps, refer to Oracle Linux 7 - How to disable Transparent HugePages for RHCK kernel? (Doc ID 2066217.1).
  • If you still face performance issues after applying the above mentioned solution, contact Oracle Support.

J.50 httpd Crash Issue on Database Firewall

Learn how to fix httpd crash issue in Database Firewall.

Problem

The httpd process in Database Firewall may crash under some circumstances.

Symptom

The status of the Database Firewall instance appears Down in the Audit Vault Server console. The Database Firewall logs are not transferred to the Audit Vault Server.

The following is observed in the log files of the impacted Database Firewall instance. The httpd.service file in /etc is symlinked to the file in /usr path.

# ls -l /etc/systemd/system/multi-user.target.wants/httpd.service

lrwxrwxrwx. 1 root root 37 Nov 27 09:26 /etc/systemd/system/multi-user.target.wants/httpd.service -> /usr/lib/systemd/system/httpd.service

# ls -lL /etc/systemd/system/multi-user.target.wants/httpd.service

-rw-r--r--. 1 root root 752 Nov 10 20:33 /etc/systemd/system/multi-user.target.wants/httpd.service

#

Solution

Follow these steps to change the configuration of the system and restart the httpd process:

  1. Log in to the Database Firewall instance as root user.

  2. Check and confirm that the above mentioned symptom exists.

  3. Copy the base file from /usr to /etc by running the following command:

    # install -m 0644 -o root -g root /usr/lib/systemd/system/httpd.service /etc/systemd/system/httpd.service

  4. Edit the file in /etc and find the below mentioned Service block:

    # vi /etc/systemd/system/httpd.service

    ...

    [Service]

    Type=notify

    EnvironmentFile=/etc/sysconfig/httpd

    ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND

    ...

  5. Modify the file and add the following code to include the restart failure directive. The file looks like the folllowing:

    
    ...
    [Service]
    Restart=on-failure
    Type=notify
    EnvironmentFile=/etc/sysconfig/httpd
    ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
    ...
    
  6. Save the file.

  7. Disable and re-enable the service to fully apply the following changes:

    
    # systemctl disable httpd
    Removed symlink /etc/systemd/system/multi-user.target.wants/httpd.service.
    # systemctl enable httpd
    Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /etc/systemd/system/httpd.service.
    #
    
  8. Verify the following changes:

    
    # sha256sum -c - <<EOF
    eac607c17f2c122619b3e1459eafdfef6bde003d24964891aa506735df4f55c2  /etc/systemd/system/multi-user.target.wants/httpd.service
    EOF
    /etc/systemd/system/multi-user.target.wants/httpd.service: OK
    #
    
  9. Reload the systemd configuration and restart httpd by running the following commands:

    # systemctl daemon-reload
    # systemctl restart httpd
  10. Verifying the service is enabled by running the following command:

    # systemctl list-unit-files | grep http
  11. Observe the following output:

    httpd.service enabled

    #

  12. If the daemon subsequently fails, the systemd will restart it, and write the following example audit trail to the system log:

    Nov 27 08:38:09 example systemd: httpd.service: main process exited, code=killed, status=11/SEGV

    Nov 27 08:39:40 example systemd: httpd.service stop-sigterm timed out. Killing.

    Nov 27 08:39:40 example systemd: Unit httpd.service entered failed state.

    Nov 27 08:39:40 example systemd: httpd.service failed.

    Nov 27 08:39:40 example systemd: httpd.service holdoff time over, scheduling restart.

    Nov 27 08:39:40 example systemd: Stopped The Apache HTTP Server.

    Nov 27 08:39:40 example systemd: Starting The Apache HTTP Server...

    Nov 27 08:39:40 example systemd: Started The Apache HTTP Server.

J.51 Issue with Retrieval of Return Row Count

Learn how to fix the issue related to retrieval of return row count.

Problem

Database Firewall captures the number of rows returned by a SELECT query and display them in reports under the column Row Count.

If the database takes a while to generate response result set, then return row count may not be extracted due to timeout configuration.

Workaround

Follow these steps to adjust the timeout interval:

  1. Connect to the Database Firewall appliance as support user.

  2. Switch to root user.

  3. Change to /var/dbfw/va directory.

  4. Identify the Database Firewall monitoring point by searching for the target name configured in the Audit Vault Server. Run the following command:

    grep -lr <TARGET NAME> *
  5. The output contains the name and path of the configuration file. For example: 1/etc/appliance.conf. In this example, 1 is the monitoring point number.

  6. Change the directory to the identified monitoring point and open configuration file of the appliance.

  7. Search for the following entry in the file:

    MAX_LOG_FILE_TIMERANGE

  8. Modify the MAX_LOG_FILE_TIMERANGE line to reflect the required time range in seconds. For example, if you wish to change the time range to 5 minutes, then the configuration line should be MAX_LOG_FILE_TIMERANGE=="300".

  9. Save the changes.

  10. Run the following command to restart the Database Firewall processes so that the new setting takes effect:

    /usr/local/dbfw/bin/dbfwctl restart <monitoring point number>

    In this case the monitoring point number was 1.

    Hence, the command should be:

    /usr/local/dbfw/bin/dbfwctl restart 1

Note:

Increasing the timeout configuration delays the availability of captured SQL statements in the reports and any alerts configured for the same. Use your discretion while configuring the above value close to the actual query completion time.

J.52 Unable to Log in to the Oracle AVDF Appliance through SSH

Learn how to fix log in issue to Oracle AVDF appliance.

Problem

The user is unable to log in to the Oracle AVDF appliance through SSH. This may be because of using old SSH clients to log in to the Oracle AVDF appliance.

Workaround

Log in to ARU (Automated Release Updates). Apply the patch number 32287150 that solves the problem.

Note:

This patch must be applied on Oracle AVDF 20.3 and later only.

J.53 Error When Changing IP Address of Management Interface

Learn how to resolve the error encountered when changing the IP address of the Management Interface.

Problem

The Management Interface IP address is the IP address of the Database Firewall which was used to register the Database Firewall in the Audit Vault Server console.

In Oracle AVDF 20.1, the following error may be encountered while attempting to change the IP address of the Management Interface:

Operation failed OAV-46981: Unable to connect to Database Firewall with IP <ipaddress>

Solution

This error may come up because the IP Address of the Database Firewall is changed successfully. However, there may be a delay in the response from Database Firewall. It may take a few seconds for the network update on the Database Firewall and for the system to settle.

Click Save and Close buttons to exit the dialog. Do not click on the cross (X) mark in the top right corner of the dialog.

J.54 Unable to Configure Microsoft SQL Server XEL Audit Trail After Upgrade

Problem

The following error is observed while configuring Microsoft SQL Server XEL audit trail on Audit Vault Server after upgrading to Oracle AVDF 20.3:

[oracle][SQLServer JDBC Driver][SQLServer]VIEW SERVER STATE permission was denied on object 'server', database 'master'

Solution

Follow these steps to resolve this issue in Oracle AVDF 20.3:

  1. Create a new user on Microsoft SQL Server target database.
  2. Grant the necessary privileges. See Oracle AVDF Administrators Guide for complete information.
  3. Modify the registered target with the newly created user credentials.
  4. Configure the Microsoft SQL Server XEL audit trail.

This issue is resolved in Oracle AVDF 20.4. Follow these steps after upgrading to Oracle AVDF 20.4 (or later):

  1. Revoke audit data collection privileges by running the mssql_drop_db_permissions.sql script as follows:

    sqlcmd -S server_name -U sa -i mssql_drop_db_permissions.sql -v username="username" mode="AUDIT_COLL" all_databases="NA" database="NA"
  2. Run the mssql_user_setup.sql script as follows:

    sqlcmd -S server_name -U sa -i mssql_user_setup.sql -v username="username" mode="AUDIT_COLL" all_databases="NA" database="NA"
  3. Configure the Microsoft SQL Server XEL audit trail.

J.55 Transaction Log Audit Trail Stops Due to an Error While Parsing XML File Containing Emoji

Problem

Transaction Log audit trail stops while parsing a file that contains emoji. The following error is observed in the Agent logs:

javax.xml.stream.XMLStreamException: ParseError at [row,col]

Solution

Follow these steps to resolve this error:

  1. Run the following command to stop the Audit Vault Agent:
    AGENT_HOME/bin/agentctl stop
    .
  2. Delete the sjsxp.jar file present in the AGENT_HOME/av/jlib directory.
  3. Run the following command to start the Audit Vault Agent:
    AGENT_HOME/bin/agentctl start
    .

J.56 Unable to Find the FIPS Status for Database Firewall Instance

Learn how to fix the error when the FIPS status for a Database Firewall instance is not displayed in the Audit Vault Server console.

Problem

The FIPS status for the Database Firewall instance could not be determined from the Audit Vault Server console.

Solution

Perform the following checks to determine the root cause of the problem:

  • The Database Firewall version is 20.4 or later.
  • Check the network connectivity between the Audit Vault Server and the two Database Firewall instances.
  • Ensure the Audit Vault Server's certificate is correctly copied or installed on the Database Firewall instance.
  • Check if the Audit Vault Server can connect to the Database Firewall by confirming that the status of the Database Firewall instance is online.

If none of the above points are helpful in identifying the cause of the problem, then contact Oracle Support.

J.57 Unable to Modify the Database Firewall FIPS Mode Through Audit Vault Server Console

Learn how to fix the error when the FIPS mode cannot be modified through the Audit Vault Server console.

Problem

This could be caused due to a communication issue between the Audit Vault Server and the Database Firewall instances.

Solution

Perform the following checks to determine the root cause of the problem:

  • The Database Firewall version is 20.4 or later.
  • Check the network connectivity between the Audit Vault Server and the two Database Firewall instances.
  • Ensure the Audit Vault Server's certificate is correctly copied or installed on the Database Firewall instance.
  • Check if the Audit Vault Server can connect to the Database Firewall by confirming that the status of the Database Firewall instance is online.

If none of the above points are helpful in identifying the cause of the problem, then contact Oracle Support.

J.58 The FIPS Status on Both the Database Firewall Instances is Different

Learn how to fix the error when the FIPS mode is different on both the Database Firewall instances.

Problem

The FIPS mode is different on both the Database Firewall instances. This could be caused when FIPS mode is manually changed on one of the Database Firewall instances. It can also be caused when such an attempt to manually change the FIPS mode failed.

Solution

All the Database Firewall instances that are part of high availability must have the same FIPS 140-2 mode. The FIPS 140-2 status of the Database Firewall instances must either be Off or On.

FIPS 140-2 mode can be disabled or enabled on both the Database Firewall instances. In case, these two instances have different FIPS mode, then an error message is displayed on the screen.

Verify the high availability status of the Database Firewall instances, and change the FIPS mode again.

J.59 After Restarting Secondary Audit Vault Server, the Primary Instance Fails to Switchover

Learn how to fix a switchover issue on the primary Audit Vault Server, after the secondary instance is restarted.

Problem

After restarting the secondary Audit Vault Server, the switchover status of the primary Audit Vault Server shows NOT ALLOWED state.

This status of the primary Audit Vault Server is not recoverable and the following error messages appear and are repeated every 50 seconds on the secondary Audit Vault Server:


<Date> <avs-instance-name> observerctl: com.oracle.avs.observerctl DEBUG - DGMGRL:[W000 <date and timestamp>] The primary database has requested a transition to the UNSYNC/LAGGING state with the standby database DBFWDB_HA2.
<Date> <avs-instance-name> observerctl: com.oracle.avs.observerctl DEBUG - DGMGRL:[W000 <date and timestamp>] Permission granted to the primary database to transition to LAGGING state with the standby database DBFWDB_HA2.
<Date> <avs-instance-name> observerctl: com.oracle.avs.observerctl DEBUG - DGMGRL:[W000 <date and timestamp>] Reconnect interval expired, create new connection to primary database.
<Date> <avs-instance-name> observerctl: com.oracle.avs.observerctl DEBUG - DGMGRL:[W000 <date and timestamp>] The primary database has been in LAGGING state for 7138 seconds.

Solution

In case the primary Audit Vault Server's switchover status goes into NOT ALLOWED status after restarting the secondary instance, then follow the steps mentioned in MOS Note (Doc ID 1258074.1) to restart the standby Audit Vault Server.

J.60 Incorrect Syntax Near Connectivity Entry in Audit Logs

Learn how to fix incorrect syntax error entry in audit logs.

Problem

When attempting to add an audit trail for Microsoft SQL Server, the Audit Vault Agent attempts to acquire a target connection using JDBC driver. After the connection is established, a test query is sent to validate the connection by the JDBC driver.

This test query may generate the following error:

Incorrect syntax near ‘Connectivity’

This error is visible in the database audit records.

Solution

Starting Oracle AVDF release 20.6, to avoid unnecessary logging of records or events due test queries in the target database, define the collection attribute as follows:

av.collector.validateConnectionOnBorrow = false

J.61 Certificate Regenerate Failure Error

Learn how to fix a certificate regenerate failure error.

Problem

In case the certificate regenerate operation fails, then one of the possible reasons can be the incorrect date and time of the appliance (Audit Vault Server or Database Firewall).

Solution

Specify the correct time, and then run the following command to regenerate the certificate:

/usr/local/bin/gensslcert create-certs

To retrieve the details about certificate expiry date, run the following command:

openssl x509 -enddate  -startdate -noout -in {certificate path}

For example:


openssl x509 -enddate  -startdate -noout -in /usr/local/dbfw/etc/cert.crt
notAfter=Oct 17 17:44:53 2022 GMT
notBefore=Sep 14 17:44:53 2021 GMT

Note:

The audit trails go to UNREACHABLE state for about 45 minutes after the certificates are rotated and all the relevant services are restarted. The trails continue to work normally after that. This behavior is observed in Oracle AVDF release 20.6 only.

J.62 User Entitlement or Audit Policy Job Stuck in Running State

Learn how to manage the user entitlement or audit policy job stuck in RUNNING state.

Problem

The user entitlement job or audit policy job is stuck in RUNNING state for a long time. This job is stuck and has to be manually stopped.

Workaround

This issue may be due to an issue with the Java Framework process in the background. Follow these steps and submit the job again:

  1. Log in to the Audit Vault Server as support user through SSH.

  2. Switch to root user by running the following command:

    su root
  3. Restart the Java Framework by running the following command:

    systemctl restart javafwk

J.63 Audit Trails are Toggling Between COLLECTING and UNREACHABLE Status

Learn how to fix the incorrect audit trail status issue.

Problem

The Audit Trails tab in the Audit Vault Server console displays the status of all the audit trails. Some audit trails are continuously toggling between the status COLLECTING and UNREACHABLE.

The trails go to UNREACHABLE state if they take more than 120 seconds (2 heartbeat intervals) to update the trail status. This can happen if either the target or Audit Vault Server is temporarily loaded, causing the trails to take more time to update the trail status.

Solution

Consider increasing the heartbeat interval to 120 seconds. Currently, the default value is 60 seconds. Run the following command as avsys user:

exec avsys.adm.add_config_param('SYS.HEARTBEAT_INTERVAL', 120);

Note:

This scenario is applicable for Oracle AVDF releases 20.5 and earlier, where the default value is 60 seconds. Starting with Oracle AVDF 20.6, the default value is 120 seconds.

J.64 Displaying Job Status Takes Lot of Time in the Audit Vault Server Console

Learn how to resolve the Jobs dialog issue.

Problem

The Jobs dialog in the System tab takes lot of time to load and to display the jobs and their current status.

Solution

Delete unwanted or old data from the Status column. This resolves the issue and the Jobs dialog displays the required information.

For example: Delete unwanted or old data from the avsys.job_status table that is more than 30 days old using the following SQL query:


Delete from job_status
where status = 'Completed'
and status_time < sysdate - 30;

J.65 Microsoft SQL Server Database Audit Trails are in Stopped State After Upgrading Java

Learn how to fix issue when audit trails belonging to Microsoft SQL Server database go to stopped state after upgrading Java version u291 or greater.

Problem

Audit trails that belong to Microsoft SQL Server database are not collecting audit data. This issue is observed after upgrading the Java version to u291 or greater and when Microsoft SQL Server target’s connect string is one of the following:

  • jdbc:av:sqlserver://<MSSQL Host name>:<Port number>;encryptionMethod=SSL; validateServerCertificate=false;
  • jdbc:av:sqlserver://<MSSQL Host name>:<Port number>;encryptionMethod=SSL;validateServerCertificate=true; trustStore=<key store jks path>;trustStorePassword=<keystore password>;extendedOptions=enableCipherSuites=SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA

Solution

Modify the connect string for Microsoft SQL Server database (in Audit Vault Server console or AVCLI) to one of the following:

  • jdbc:av:sqlserver://<MSSQL Host name>:<Port number>;encryptionMethod=SSL;validateServerCertificate=false;CryptoProtocolVersion=TLSv1.2;
  • jdbc:av:sqlserver://<MSSQL Host name>:<Port number>;encryptionMethod=SSL;validateServerCertificate=true;CryptoProtocolVersion=TLSv1.2;trustStore=<key store jks path>;trustStorePassword=<keystore password>;extendedOptions=enableCipherSuites=SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA

J.66 Unable to Delete Database Firewall

Learn how to fix an issue observed when attempting to delete Database Firewall.

Problem

An error OAV-47704 is observed when attempting to delete Database Firewall. This issue is observed in the following scenario:

  • Oracle AVDF releases 20.1 to 20.5
  • Audit Vault Server is upgraded to Oracle AVDF 20, but Database Firewall is not upgraded to Oracle AVDF 20
  • Error observed in the Audit Vault Server console or in AVCLI

Solution

This issue is fixed in Oracle AVDF release 20.6. In case the installed version is Oracle AVDF releases 20.5 and earlier, then follow these steps:

  1. Log in to the Audit Vault Server through SSH.

  2. Switch user to root:
    su root
  3. Switch user to dvaccountmgr:
    su dvaccountmgr
  4. Start SQL*Plus connection without the username or password:
    sqlplus /nolog
  5. Unlock avsys user and assign a password by running the command:
    alter user avsys identified by <pwd> profile default account unlock;
  6. Run the command:
    exit
  7. Start SQL*Plus connection without the username or password:
    sqlplus /nolog
  8. In SQL*Plus run the command:
    connect avsys
  9. Enter the password when prompted. Alternatively, run the command:
    connect <avsys/password>
  10. Run the following SQL query:
    select id from avsys.firewall where name= '<firewall_name> ' and deleted_at is null;
  11. Make a note of the Database Firewall ID.

  12. Run the command:
    update avsys.firewall set software_version=’<avs_version>’ where id=<firewall_id>;

    For example: update avsys.firewall set software_version=’20.5.0.0.0’ where id=<firewall_id>;

  13. Run the command:
    commit;
  14. Repeat the process for any other Database Firewall instance that needs to be deleted.

  15. Run the command:
    exit
  16. Attempt to delete the Database Firewall instance from the Audit Vault Server console or through AVCLI.

J.67 Issue in Language Setting of the Audit Vault Agent

Learn how to fix the language setting in Audit Vault Agent.

Problem

Unable to change or set the language in Audit Vault Agent. Audit Vault Agent supports languages other than English.

Audit Vault Agent uses the language specified in the locale settings of the host machine (Agent machine), provided the language is supported. In case the specific language is already set on the system, then there is no need to change the settings for the Agent to use the specific language.

Solution

The locale settings for the Windows platform can be changed through the Control Panel on the Windows host machine.

To change the locale settings on Linux/Unix/AIX/Solaris platform, set the LC_ALL and LANG environment variables.

For example:

export LC_ALL=fr_FR.iso88591
export LANG=fr_FR.iso88591

J.68 Unable to Create a Database Firewall Monitoring Point

Learn how to fix an error while creating a Database Firewall monitoring point.

Problem

An attempt to create a Database Firewall monitoring point using the target host name does not succeed.

Symptom

  • Failure to create a Database Firewall monitoring point using the target host name displays the status as Starting. The status changes to Unreachable after a while.

  • The /var/log/messages file in Database Firewall contains an error similar to the following:

    
    May 10 11:06:02 dbfw08002718dd46 hostname_lookup.rb[19691]:
    foobar.example.com.oracle.dbfw.hostname-lookup WARN - ODF-10505: Failed to resolve hostname:
    Unable to resolve the hostname ["hostname1.foobar.example.com"].
    Verify DNS settings. Hostname resolution will be tried every minute.
    

Solution

DNS is not configured and hence the above error is observed. Configure the DNS and attempt to create the Database Firewall monitoring point again.

In case DNS is configured, verify the DNS settings. Attempt to resolve the host name is made once every minute.

J.69 Issue with Configuring or Managing Oracle AVDF through Oracle Enterprise Manager Cloud Control

Learn how to solve an issue with configuring or managing Oracle AVDF through Oracle Enterprise Manager Cloud Control.

Problem

Unable to configure or manage Oracle AVDF through Oracle Enterprise Manager Cloud Control.

Solution

Oracle AVDF plug-in is an interface within Oracle Enterprise Manager Cloud Control for administrators to manage and monitor Oracle AVDF components. Refer to System Monitoring Plug-in User's Guide for Audit Vault and Database Firewall in case of any issues when configuring the Oracle EM plug-in.

Refer to Compatibility with Oracle Enterprise Manager to check the supported versions of Oracle Enterprise Manager with Oracle AVDF 20.