L Troubleshooting Oracle Audit Vault and Database Firewall

Oracle Audit Vault and Database Firewall provides troubleshooting advice for a range of scenarios.

L.1 Information to Provide Support When Filing a Service Request

Review this list of information to provide support when filing a service request.

Note:

Diagnostics data, especially trace files, often contains sensitive information. Protect it accordingly and only gather and send the information that's required.

• Oracle AVDF version, including any installed bundle patches
• If virtualization is being used? If so, which one?
• How much physical memory is available to Audit Vault Server and Database Firewall appliances?
• How much disk space was available with the initial installation?
• Did you add any SAN storage and in that case how much disk space?
• Provide any relevant details about the brand and model of the hardware being used. This is relevant if you have specific issues relating to booting from the installation media.
• Host OS for the secured target database and version, this is relevant for checking agent compatibility issues.
• Brand of the secured target database, such as Oracle, MySQL, SQL Server, etc.
• Version of the secured target database, including PSU and other one-off patches.
• Upload the alert.log file of the secured target database.
• From any Oracle secured target database provide the output of:
  • show parameter audit
  • opatch lsinventory -patch -detail
  • If unified auditing was configured (for some versions of Oracle database only)
  • Audit Trail type that is being configured and all relevant attributes
• Detailed diagnostic information for Audit Vault Server, see Downloading Detailed Diagnostics Reports for Oracle Audit Vault Server
• If requested by Oracle Support, diagnostic information from Oracle Trace File Analyzer. See Using Oracle Trace File Analyzer (TFA).
• Information about Database Firewall:
  • Detailed diagnostic info for Database Firewall, see Viewing the Status and Diagnostics Report for Database Firewall
  • How many Network Interface Cards are installed in the database firewall appliance?
  • Is the enforcement point using default password enumeration (DPE) or database activity monitoring (DAM)? If so is it bridge, span, or proxy?
  • Do you use VLAN tagging? There are restrictions for support of VLANs.
• For installation issues, diagnostic files related to the installation. See Collecting Logs to Debug Installation Failures.

Before contacting support, the Audit Trail Transaction Log should follow these guidelines:

• The user setup script must be run with the argument REDO_COLL
• The secured target database must be configured with ARCHIVELOG
• The streams recommended patches must be applied to the secured target db: Streams Recommended Patches (Doc ID 437838.1)
• global_name must be fully qualified (select global_name from global_name;)
• Parameter global_names = true is recommended
• If errors happen on capture or apply side please check respective alert.logfiles as you would do with any Streams related issue (av log will show only limited information for this audit trail type)

L.2 Using Oracle Trace File Analyzer (Oracle AVDF 20.1 - 20.11)

If you request support from Oracle Support, they may ask you to install and run Oracle Trace File Analyzer on the Audit Vault Server to collect diagnostic information.

Note:

Install Oracle Trace File Analyzer only when requested by Oracle Support, and uninstall it when you're done to maintain a high level of security. Make sure that it's uninstalled before patching or upgrading to the latest version of Oracle AVDF.

1. Log in to the Audit Vault Server through SSH and switch to the root user.

   See Logging In to Oracle AVDF Appliances Through SSH.

2. Enter the following command to install Oracle Trace File Analyzer:

   /usr/local/dbfw/bin/setup_TraceFileAnalyzer.py --install

3. Run any tfactl command to collect diagnostics, as needed. For example:

   tfactl diagcollect <options>

4. Securely copy the collected diagnostic file to a location from which you can upload the file to the service request. For example:

   scp /opt/ahf_installation/oracle.ahf/data/repository/<diagnostic_zip_file> <new_location>

5. Run the following command to uninstall Oracle Trace File Analyzer:

   /usr/local/dbfw/bin/setup_TraceFileAnalyzer.py --uninstall

If you have modified the IP address of the Audit Vault Server and are encountering the TFA-00104 Cannot establish connection with TFA Server. Please check TFA Certificates error when running TFA commands, follow these steps to resolve the error:

1. Log in to the Audit Vault Server through SSH and switch to the root user.

   See Logging In to Oracle AVDF Appliances Through SSH.

2. tfactl syncnodes -regenerate

L.3 Using Oracle Trace File Analyzer (Oracle AVDF 20.12 and later)

If you request support from Oracle Support, they may ask you to run Oracle Trace File Analyzer on the Audit Vault Server to collect diagnostic information. Oracle Trace File Analyzer is already installed on the Audit Vault Server starting with Oracle AVDF 20.12.

1. Log in to the Audit Vault Server through SSH and switch to the root user.

   See Logging In to Oracle AVDF Appliances Through SSH.

2. Run tfactl command to collect diagnostics, as needed. For example:

   tfactl diagcollect -avs -noclassify  -noinsight

   The avs parameter should be used to ensure the Audit Vault Server application layer logs will also get collected.

3. Securely copy the collected diagnostic file to a location from which you can upload the file to the service request. For example:

   scp /var/opt/oracle/ahf/oracle.ahf/data/repository/<diagnostic_zip_file> <new_location>

   Oracle Trace File Analyzer on the Audit Vault Server will automatically collect logs in /var/opt/oracle/ahf/oracle.ahf/data.

If you have modified the IP address of the Audit Vault Server and are encountering the TFA-00104 Cannot establish connection with TFA Server. Please check TFA Certificates error when running TFA commands, follow these steps to resolve the error:

1. Log in to the Audit Vault Server through SSH and switch to the root user.

   See Logging In to Oracle AVDF Appliances Through SSH.

2. tfactl syncnodes -regenerate

L.4 Ability to Boot Into Rescue Mode When Troubleshooting

Starting in Oracle AVDF 20.10 you can boot directly into rescue mode from the grub menu on both the Audit Vault Server and the Database Firewall. Booting into the rescue mode does not run AVDF processes and allows for easier troubleshooting. Running rescue mode is intended for use by and under the direction of Oracle Support.

Once the system has booted you can switch to rescue mode by running the following command:

- systemctl isolate avdf-minimal.target

You can switch back to usual runtime by running the following command:

- systemctl isolate avdf-runtime.target

Note:

Switching from rescue mode on the Audit Vault Server to the usual runtime mode can take a long time, around 15 minutes.

L.5 Audit Vault Agent or Host Monitor Agent Is Not Upgraded to the New Release

Learn how to upgrade the Audit Vault Agent or Host Monitor Agent manually.

Problem

After upgrading to Oracle AVDF 20.1 or later, some of the Audit Vault Agents or Host Monitor Agents are not upgraded.

Symptom - 1

Audit Vault Agent is in STOPPED state after Audit Vault Server upgrade.

Symptom - 2

Host Monitor Agent is in NEEDS UPGRADE or UPDATE FAILED state after Audit Vault Server upgrade.

Solution - 1

The symptom indicates that the Audit Vault Agent has failed to auto upgrade during the Audit Vault Server upgrade. Execute the following steps as the user who installed Agent previously:

1. Check for any Agent processes on the host machine. Ensure there are no Agent related processes currently running.
2. Remove the existing agent.jar file and the Agent folder from the host machine.
3. Download the new agent.jar file from the upgraded Audit Vault Server.

4. Execute the following command:

   java -jar agent.jar [-d <AgentHome>]

5. Verify the Agent is in RUNNING state.

Solution - 2

The symptom indicates that the Host Monitor Agent has failed to auto upgrade during the Audit Vault Server upgrade. Execute the following steps as root user:

1. Check for any Host Monitor Agent related processes on the host machine. Ensure there are no hostmonitor, hmdeployer, or hostmonmanager processes currently running.
2. Navigate to the directory outside of hm where the Host Monitor Agent is installed.

3. Execute the following command to uninstall the Host Monitor Agent:

   ./hm/hostmonsetup uninstall

4. Download the new Host Monitor Agent installable bundle from the Audit Vault Server console, for the specific platform on which it will be reinstalled.
5. Extract the Host Monitor Agent bundle inside the hm directory.

6. Execute the following command to reinstall the Host Monitor Agent in a root owned location:

   ./hostmonsetup install

L.6 Failure While Building a Host Monitor Agent or Collecting Oracle Database Trails

Learn what to do when you experience a failure while building Host Monitor Agents or collecting Oracle Database trails.

Problem

This problem may manifest with various symptoms:

• When I try to build a Host Monitor Agent, the operation fails or the operation cannot locate the correct binaries.

• When I try to collect audit data from an Oracle Database target, the operation fails.

• The Audit Vault Agent cannot connect to the Audit Vault Server.

• Audit trail does not start.

Solution

1. Unset all environment variables except the following:

   • PATH

   • TERM

   • PS1

   • LANG

   • LC_*

   • JAVA_HOME

   Then run the java -jar agent.jar command again on the host machine.

   See Also:

   Deploying the Audit Vault Agent

2. If you deployed the Audit Vault Agent in a Linux environment, then ensure that the host machine name appears in the /etc/hosts file.

L.7 Error When Running Host Monitor Agent Setup

Review the resolutions for errors that occur when running Host Monitor Agent setup.

Problem

I am setting up a Host Monitor Agent. When I run the command $HOSTMON_HOME/hm/hostmonsetup install, the following error is displayed:

Failed to generate executables for Host monitor

This means the host computer does not have the required libraries for the Host Monitor Agent. Install the required libraries mentioned in Host Monitor Agent Requirements.

Symptom 1

Even after installing the required libraries, if the Host Monitor Agent installation fails with above error message, then examine the makelogerror file which is available in the Host Monitor Agent installation directory. The following errors, may appear in the file:

/bin/ld: cannot find -laio

/bin/ld: cannot find -lssl

/bin/ld: cannot find -lcrypto

/bin/ld: cannot find -lnsl

/bin/ld: cannot find -lpcap

/bin/ld: cannot find -lcap

Solution 1

To resolve the issue on Linux (64 bit) systems, follow these steps:

1. Search where the actual binaries (not symlinks) are present – libssl, libnsl,libaio, libpcap, libcap. In most scenarios it should be present either in /lib64 or /usr/lib
2. Create below symlinks if not already present in /lib64 or /usr/lib
   1. libcap binary
      • ln -s <location from step 1>/libcap.so.<version> /lib64/libcap.so.1
      • ln -s <location from step 1>/libcap.so.1 /lib64/libcap.so
   2. libaio binary
      • ln -s <location from step 1>/libaio.so.<version> /lib64/libaio.so.1
      • ln -s /lib64/libaio.so.1 /lib64/libaio.so
   3. libnsl binary
      • ln -s <location from step 1>/libnsl.so.<version> /lib64/libnsl.so.1
      • ln -s /lib64/libnsl.so.1 /lib64/libnsl.so
   4. libpcap binary
      • ln -s <location from step 1>/libpcap.so.<version> /lib64/libpcap.so.1
      • ln -s /lib64/libpcap.so.1 /lib64/libpcap.so
   5. libssl binary
      • ln -s <location from step 1>/libssl.so.<version> /lib64/libssl.so.1
      • ln -s /lib64/libssl.so.1 /lib64/libssl.so
   6. libcrypto binary
      • ln -s <location from step 1>/libcrypto.so.<version> /lib64/libcrypto.so.1
      • ln -s /lib64/libcrypto.so.1 /lib64/libcrypto.so

Symptom 2

The following error is observed in the HOSTMON_HOME/makelogerror file:

Undefined first referenced symbol in file 
__1cG__CrunKex_dealloc6Fpv_v_       ./libhostmon19.a(Class.o) 
ld: fatal: symbol referencing errors 
make: Fatal error: Command failed for target `hostmonitor'

Solution 2

This error is observed when attempting to install Host Monitor Agent on Solaris 11.3 host machine. Some of the Solaris OS libraries are corrupt. Upgrade the operating system to Solaris 11.4 or contact the Solaris team for further assistance.

L.8 Host Monitor Agent Fails to Start

Learn what to do when the Host Monitor Agent fails to start.

Problem

The Host Monitor Agent network trail does not start after installation. The collection framework (collfwk) log file contains one of the following errors:

• java.io.IOException: Cannot run program "<AgentHome>/hm/hostmonmanager" (in directory "<AgentHome>/hm"): error=13, The file access permissions do not allow the specified action.

• HMCommandExecutor : startTrail :  binary is not found here: <AgentHome>/hm/hostmonmanager

Solution

This issue may arise due to insufficient privileges while starting Host Monitor Agent. Ensure the Audit Vault Agent user belongs to the group that owns hm (Host Monitor Agent installation) directory. Ensure that the following permissions are given:

• The group that owns the Host Monitor Agent installation (hm) directory has read and execute permission on the hm directory.
• The group that owns the Host Monitor Agent installation (hm) directory has execute permission on hostmonmanager binary.

In the event that assigning the above permissions to the group did not work, use Access Control Lists (ACL) to ensure that the following permissions are given:

• The Audit Vault Agent user has read and execute permissions on the hm directory.
• The Audit Vault Agent user has execute permissions on hmdeployer, hostmonitor and hostmonmanager binaries.
• The Audit Vault Agent user has read permissions on libnnz*.so and libociicus.so libraries.

Note:

• AgentHome is the Audit Vault Agent installation directory.

• hm is the Host Monitor Agent installation directory.

L.9 Host Monitor Agent Network Trail is in STOPPED State

Learn how to fix the issue when Host Monitor Agent network trail is in STOPPED state.

Problem

After starting the Host Monitor Agent network trail it goes into a STOPPED state.

Symptom

The following error is observed in the HOSTMON_HOME/log/av.hostmonitor*.log file:

[2022-01-27 13:40:57,061] [PID: <ID>, TName: main] [WARNING] 
- Failed to perform SSL handshake using TLS protocol TLS 1.2 Error Msg: SSL error:
Error in system call. Details: error:00000000:lib(0):func(0):reason(0)
Retrying with lower protocol

Solution

The Host Monitor Agent certificate is corrupt. Follow the steps in the topic Using Mutual Authentication for Communication Between the Database Firewall and the Host Monitor Agent to regenerate the certificate.

L.10 Network Audit Trail Does Not Start on Unix Platforms

Learn the resolution when the network audit trail fails to start on Unix platforms.

Problem

The network audit trail does not start on Unix platforms.

Symptoms

• The Oracle Audit Vault Server console displays the following error:

  Unable to start Host Monitor process

• The collection framework log displays the following error:

  <Host Monitor home>/hostmonmanager binary is not found here

Solution

1. Connect to the host machine on which the Audit Vault Agent and Host Monitor Agent are installed.
2. In the Agent Home location there is an hm symlink pointing to Host Monitor Agent installation location.
3. Run the following command from the Agent Home as the user who installed Audit Vault Agent:

   ls -lrt hm

4. Verify that it's possible to list the contents of the Host Monitor Agent installation directory.

5. Check the permissions of all directories in the hierarchy of the path under which the Host Monitor Agent is installed.

   Note:

   The entire directory hierarchy must be owned by the root user. All of the directories in this hierarchy must have read and execute permission for other users or groups, but not write permission.

   In addition, the hostmonitor and hostmonmanager binaries should have execute access for the user who owns the Host Monitor Agent. These permissions should be granted by using an access control list (ACL).

6. Grant the necessary permissions according to the preceding note.
7. Restart the network audit trail.

L.11 Partial or No Traffic Seen for an Oracle Database Monitored by Oracle Database Firewall

Review the troubleshooting advice for when you see limited or no traffic for an Oracle Database that is monitored by Oracle Database Firewall.

Problem

I see no traffic, or only partial traffic, captured in reports for an Oracle Database monitored by the Database Firewall.

Solutions

Go through the following checks to find the trouble:

1. In the Audit Vault Server, check that the report filters are set correctly, including the time slot.

2. Check that the system time on the Database Firewall is synchronized with the time on the Audit Vault Server and the target system.

3. Check that the target's network traffic is visible to the Database Firewall using the Live Capture utility on the firewall.

4. Check that the Oracle Database service name or SID is used correctly. If you specified an Oracle Database service name in the monitoring point settings for this target, you will only see traffic for that service name. To see all traffic, remove the service name from the monitoring point settings.

   If you have entered a service name in the monitoring point, and see no traffic, check to see that the service name is entered correctly in the monitoring point settings.

   For monitoring points set to use monitoring only mode, the Database Firewall may be monitoring traffic for existing client connections to the database. Since these connections were in place before you deployed the Database Firewall, it will not be able to detect the service name you specify in the monitoring point. In this case, restart the client connections to the database.

5. Check that the correct Database Firewall policy is deployed.

See Also:

• Oracle Audit Vault and Database Firewall Auditor's Guide for information on editing and deploying firewall policies.

• Configuring Database Firewall Monitoring Points

• Viewing Network Traffic for a Database Firewall

L.12 Incomplete or Missing SQL Statements or Network Traffic in Oracle AVDF Reports

Learn about the probable causes that may result in missing SQL statements in Oracle AVDF Reports.

Problem

Sometime there may be SQL statements missing or incomplete network traffic information in Oracle AVDF Reports. This topic contains the probable causes and some tips to troubleshoot.

Symptoms

Although there may be multiple reasons that may cause this issue, the following are some of the probable causes:

• The database client is unable to connect to the Database Firewall instance
• The Database Firewall is unable to connect to the target database
• The Audit Vault Server may be down
• The Database Firewall is unable to connect to the Audit Vault Server
• The Audit Vault Server may not be collecting data

Solution

Take necessary steps to resolve depending on the issue and diagnostic information. The following table contains some of the probable issues and some measures for the resolution:

Issue	Resolution
The database client is unable to connect to the Database Firewall proxy port.	Check the IP address and TCP port used by the database client to connect to the Database Firewall. Does that belong to the specific Database Firewall instance or monitoring point? Ping the Database Firewall server from the database client’s host machine. In case there is a failed response, then contact the network administrator. Is there any other firewall between the database client and Oracle Database Firewall which may block the connection? Is the Database Firewall listening to the expected or configured port? Log in to the Database Firewall instance through SSH as support user and then switch to root user. Run the command netstat -alnpt | grep LISTEN | grep nnnnn where nnnnn is the port number the database client is attempting to connect. If there is no port displayed in the output, then check the proxy port settings for the configured target.
The Database Firewall deployed in Monitoring / Blocking (Proxy) mode is unable to connect to the target database.	Check the IP address and TCP port of the target database. Verify if this information is properly entered when registering the target in the Audit Vault Server console. Is there any other firewall between the database client and Oracle Database Firewall which may block the connection? Ping the target database from the Database Firewall server. In case there is a failed response, then contact the network administrator. Log in to the Database Firewall instance through SSH as support user and then switch to root user. Access and examine the contents of the /var/log/messages file. Check if the string com.oracle.dbfw.fw ERROR - ODF-10501: Failed connecting to the Target exists. See Database Firewall Messages for more information on the error message. The message also contains additional information for the failure. For example: No route to host.
Missing SQL statements that were expected to be captured by the Database Firewall.	Check if the Database Firewall monitoring point is running. In the Audit Vault Server console, check the status of the specific Database Firewall monitoring point. Check if the Database Firewall instance is generating the required log files. There are two ways of doing this: Check the Database Firewall policy applied to the specific target. See if the option Pass-All is selected. This does not log any traffic. In case of a custom policy, analyze and revert to Log-All policy option and check if the expected traffic data is reflecting in the Oracle AVDF reports. If records are present, then examine the details of the custom policy and redefine as per the requirements. Log in to the Database Firewall instance through SSH as support user and then switch to root user. Run the following command watch ls -ltr /usr/local/dbfw/va/*/log/. Examine the logs displayed and search for the log files named kernel.nnnnnnnnnn.nnn.n.nnnn.dat.gz. In case the traffic is being logged, then these files are displayed. Later these files disappear in a few minutes as their contents are transferred to the Audit Vault Server. Check if the Audit Vault Server is requesting the transfer of the traffic log files from the Database Firewall. Follow these steps: The network traffic is being logged in Database Firewall and the log files are not removed after a while. Audit Vault Server may not be requesting the transfer of the traffic log files from the Database Firewall. Log in to the Database Firewall instance through SSH as support user and then switch to root user. Run the following command tail -f /var/log/httpd/ssl_request_log. Examine the output of the command. Some of the regular entries that end as follows exist: GET /logs/2/list HTTP/1.0" 78 GET /logs/2/kernel.1655297472.734.0.0000.dat.gz HTTP/1.0" 1321 If these entries are not displayed, then further troubleshooting in the Audit Vault Server is required. Raise a bug and attach the diagnostics of both Audit Vault Server and Database Firewall.

L.13 Agent Activation Request Returns 'host is not registered' Error

Read the troubleshooting advice if you receive a 'host is not registered' error.

Problem

I used the following two commands to register the Oracle Audit Vault Agent's host computer (where the agent is deployed), and to request Audit Vault Agent activation:

From the Audit Vault Server:

avcli> register host 'host_name'

From the host computer:

agentctl activate

But the agentctl activate command returns: Agent host is not registered

Solution

Your agent host may be multi homed. In this case, the agent hostname to IP address resolution may resolve to the NIC/IP that is not used by the agent while connecting to the AV server. To resolve this issue, try to register the agent host using the with ip option and then try activating the agent again.

From the Audit Vault Server, use the following command:

avcli> register host 'host_name' with ip 'host_ip_address'

If you still have issues, try finding the IP address used in the database session when you connect to the Audit Vault server from the agent host, using these commands:

Start SQL*Plus connection as sqlplus /nolog without the username or password.

In SQL*Plus execute the command: connect <user>. Enter the password when prompted.

sqlplus username/password@"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=Audit_Vault_Server_IP)(PORT=1521))(CONNECT_DATA= (SERVICE_NAME=dbfwdb)))"

sqlplus> select SYS_CONTEXT('USERENV','IP_ADDRESS') from dual;

Use the IP address from the above query to register your host.

L.14 Unable to Deploy Agent on the Secondary Audit Vault Server

Learn the resolution if you are unable to deploy an agent on a secondary Oracle Audit Vault server.

Problem

When I try to deploy the Audit Vault Agent on the secondary Audit Vault Server in a high availability pair, I get an error that the host is not registered.

Cause

After you pair two Audit Vault Servers for high availability, you do all configuration on the primary server in the pair only, including Audit Vault Agent deployment.

L.15 'java -jar agent.jar' Failed on Windows Machine

Review the resolution procedures when the java -jar agent.jar command fails on Windows machines.

Problem

The command java -jar agent.jar failed on my Windows target machine, and I noticed in the log files that the Audit Vault Agent services installation/un-installation failed.

Solution

1. Follow the instructions for unregistering the agent in Registering and Unregistering the Audit Vault Agent as a Windows Service.

   If Method 1 fails, then try Method 2.

2. Run the java -jar agent.jar command again.

L.16 Unable to Install the Agent or Generate the agent.jar File

Determine the steps to perform if you are unable to install the agent or generate the agent.jar file.

Problem

Unable to install the Audit Vault Agent. Attempts to regenerate the agent.jar file are also unsuccessful.

Solution

Follow these steps to regenerate the agent.jar file:

1. Log in to the Audit Vault Server through SSH as user oracle.

2. Go to the directory /var/lib/oracle/dbfw/av/conf/ location.

3. Delete the bootstrap.prop file.

4. Execute the following command:

   /var/lib/oracle/dbfw/bin/avca configure_bootstrap

5. Check the avca.log file that is available at /var/lib/oracle/dbfw/av/log/ to check if the above command was executed successfully.

6. Switch the user (su) to avsys.

7. Run the following query:

   select agent_gen_ts from file_repos where file_name='agent.jar';

8. The above query displays the current time in case the agent.jar file is generated successfully.

L.17 Unable to Un-install the Oracle Audit Vault Agent Windows Service

Review the troubleshooting advice if you are unable to un-install the Oracle Audit Vault Agent Windows Service.

Follow the instructions for unregistering the Agent inRegistering and Unregistering the Audit Vault Agent as a Windows Service.

If Method 1 fails, then try Method 2.

L.18 Access Denied Error While Installing Agent as a Windows Service

Learn how to resolve access denied errors when installing Oracle Audit Vault agent as a Windows service.

Problem

I got an error during installation of Oracle Audit Vault Agent on Windows, and I noticed the following error in the AGENT_HOME\av\log\av.agent.prunsvr log file:

[2013-05-02 11:55:53] [info] Commons Daemon procrun (1.0.6.0 32-bit) started
[2013-05-02 11:55:53] [error] Unable to open the Service Manager
[2013-05-02 11:55:53] [error] Access is denied.
[2013-05-02 11:55:53] [error] Commons Daemon procrun failed with exit value:
7 (Failed to )
[2013-05-02 11:55:53] [error] Access is denied. 

Solution

The above message means that the logged in user does not have privileges to install the Audit Vault Agent as a Windows Service. If you get the above message, try launching the command shell with the Run As Administrator option, and then execute java -jar agent.jar in that command shell.

L.19 Unable to Start the Agent Through the Services Applet on the Control Panel

Review how to resolve being unable to start the agent through the services applet on the control panel.

Problem

I did the following:

1. Installed the Audit Vault Agent using the java -jar agent.jar command.

2. Activated the Audit Vault Agent.

3. Started the Audit Vault Agent using the agentctl start -k key command.

   The agent started up and is in RUNNING state.

4. Stopped the Audit Vault Agent.

5. Tried to start the Audit Vault Agent using the Services Applet on the Windows Control Panel.

   The Audit Vault Agent errored out immediately.

Solution

This means that the Audit Vault Agent is configured to use a Windows account that does not have privileges to connect to the Audit Vault Server.

Take the following steps:

1. Go to Control Panel, then to Services Applet.

2. Select the Oracle Audit Vault Agent service.

3. Right click and select the Properties menu.

4. Click the Log on tab.

5. Select This account: and then enter a valid account name and password.

6. Save and exit.

7. Start the Audit Vault Agent through the Services Applet.

L.20 Error When Starting the Agent

Resolved errors that occur when starting the agent.

Problem

After I installed the Audit Vault Agent, I set the username and password in the OracleAVAgent Windows Service Properties Log On tab. However, when I try to start the OracleAVAgent service, I see the following error in the Agent_Home\av\log\av.agent.prunsvr.date.log file:

[info]  Commons Daemon procrun (1.0.6.0 32-bit) started
[info]  Running 'OracleAVAgent' Service...
[info]  Starting service...
[error] Failed creating java 
[error] ServiceStart returned 1
[info]  Run service finished.
[info]  Commons Daemon procrun finished

Solution

This means that the OracleAVAgent service is not able to launch the Java process. Try the following:

1. Uninstall all JDKs and/or JREs in the system.

2. Reinstall JDK SE or JRE and then start the OracleAVAgent service.

3. If this doesn't help, you can install 32 bit JDK SE or JRE and then start the OracleAVAgent service.

L.21 Alerts on Oracle Database Targets Are Not Triggered for Extended Periods of Time

Learn what to do when alerts on targets are not triggered for a long time.

Problem

I configured an Oracle Database target to audit to XML files, configured an audit trail in Oracle AVDF of type DIRECTORY, and then configured an alert to trigger on certain events. My alert did not get triggered for a long time.

Solution

This issue can occur if the Oracle Database target is not flushing the audit records to the file immediately. Contact Oracle Support in order to access support note 1358183.1 Audit Files Are Not Immediately Flushed To Disk.

L.22 Error When Creating an Audit Policy

Resolve errors that can occur when you create an audit policy.

Problem

I received this error message when I tried to create a new audit policy setting for Oracle Database:

-ORA-01400: cannot insert NULL into ("AVSYS"."AUDIT_SETTING_ARCHIVE_MAP"."ARCHIVE_ID")

Cause

The Oracle Database must have at least one audit policy setting before you can create and provision new audit settings using Oracle Audit Vault and Database Firewall. Oracle Database comes with a predefined set of audit policy settings. You must not manually remove these settings. If the audit settings have been removed, then you can manually create at least one audit setting in the Oracle Database. Then try again to create new audit settings using Oracle Audit Vault and Database Firewall.

See Also:

Oracle Database Security Guide for detailed information on Oracle Database auditing.

L.23 Connection Problems When Using Oracle Database Firewall Monitoring and Blocking

Resolve the connection problems that might occur when using Oracle Database Firewall monitoring and blocking.

Problem

In monitoring and blocking mode, my client application cannot connect to the target database.

Solution

1. Log in as root on the Database Firewall server.

2. Run this command using the target database IP address or host name:

   ping -I secured_target_ip_address_or_hostname

   If you do not receive a response, then ensure that the DNS is configured on Oracle Database Firewall.

   If a response is received, check:

   • The firewall policy to ensure that it is not blocking the connection attempt.

   • The client connection settings to ensure that the client is attempting to connect to the correct target database.

See Also:

• Configuring the Database Firewall and Its Traffic Sources on Your Network

• Configuring Network Services for Oracle Database Firewall

L.24 Audit Trail Does Not Start

Learn the resolution to use when the audit trail does not start.

Problem

An audit trail does not start. For example, in the Audit Vault Server console, in the Audit Trails page, the Collection Status column indicates that the trail is Stopped or Unreachable.

Solution

When a trail does not start, you can show the associated error in two ways:

• In the Audit Vault Server console:

  1. Click the Targets tab, and then from the Monitoring menu, click Audit Trails.

  2. Click the Actions button, and then click Select Columns.

  3. From the left-hand box, double-click Error Message so that it moves into the Display in Report box on the right.

  4. Click Apply.

  The Error Message column is displayed on the Audit Trails page and contains the error message for the stopped trail.

• On the Audit Vault Agent host computer:

  1. Go to the logs directory:

     cd %agenthome%/av/logs

  2. Run the following:

     grep -i 'error|warning|fail' *

  The error messages should indicate the cause of the problem.

If the cause is still unclear, or the grep command returns no results, raise an SR with Oracle Support and include Audit Vault Agent log files.

L.25 Cannot See Data for Targets

Learn what to do when you cannot see the data for a target.

Problem

Data for my Target does not appear on reports.

Solution

If you cannot see the data you expect to see in the Audit Vault Server, you can troubleshoot by trying one or more of the following:

• Confirm that Audit Vault Agent hosts are up and that the Audit Vault Agents are running.

• Confirm that audit trails are running and that the audit trail settings match the audit configuration of the Target database

  For example, the audit trail configuration in Oracle Audit Vault and Database Firewall should have the correct trail type and location.

  See Also:

  Configuring and Managing Audit Trail Collection

• Check the audit policy on the target to ensure you are auditing the activity that you are expecting to see in the reports.

• Check the firewall policy to ensure you are logging the activity you are expecting to see in reports.

• Clear any time filters on reports, and then check time settings on the target and on the AVS. If the time is incorrect, the time recorded against audit events will not be accurate. As a result, the audit events may not be displayed in the time window you expect.

• Check the /var/log/messages file on Audit Vault Server and on the Database Firewall for errors.

• Check that the Database Firewall monitoring point is created and running.

• Check that the Database Firewall monitoring point traffic source is correct.

• If the Database Firewall is in monitoring only mode, use the Database Firewall Live Capture utility to verify that traffic is being seen on the relevant traffic source. If necessary, use the File Capture utility to capture traffic to a file and verify (using Wireshark or a similar product) that the traffic being captured is consistent with the settings in the Target Addresses section of your Target configuration.

  See Also:

  Viewing Network Traffic for a Database Firewall

• Check that you have used the correct Oracle Database service name when configuring the Target Address in your Target configuration.

  Also, have you included all available Oracle Service names in the Target Addresses section of the Target configuration? Unless you intend to define a different firewall policy for each service name, Oracle recommends you omit service name and use only IP address and TCP ports in Target Addresses.

• On the Database Firewall, check the /var/log/httpd/ssl_access_log file to confirm that the Audit Vault Server is collecting logs.

• On the Audit Vault Server, check the /var/dbfw/tmp/processing* directories and make sure kernel*.dat files are arriving in the directory, and then being deleted once the Audit Vault Server has processed them.

• On the Audit Vault Server, check that the mwecsvc process is running. For example, run the command:

  ps -ef | grep mwecsvc

  If the process is not running, use this command to restart it:

  service controller start

L.26 Problems Pairing Oracle Database Firewall and Oracle Audit Vault Server

Review the procedure to follow when you have problems pairing Oracle Database Firewall with Oracle Audit Vault Server.

Problem

I encounter errors when I try to associate a Database Firewall with the Audit Vault Server.

Solution

Check the following:

• Ensure that you have entered the correct Audit Vault Server IP address in the Database Firewall Certificate page.

  Log in to the Audit Vault Server console, and click the Settings tab. Then click the Certificate tab on the main page.

• Ensure that both the Database Firewall server and the Audit Vault Server are configured to use NTP and that each machine is synced to the NTP time server.

See Also:

• Specifying the Server Date, Time, and Keyboard Settings

• Setting the Date and Time in Database Firewall

L.27 User Names Do Not Appear on Database Firewall Reports

Learn what to do when names do not appear on Database Firewall reports.

Problem

When I generate a Database Firewall report, I do not see user names.

Solution

Check the following possibilities:

• If this is occurring for a Microsoft SQL Server database target, check to make sure that retrieve session information is turned on.

• This problem may be caused by bad network traffic arriving at the Database Firewall. Check for duplicate or missing network packets. You can use the Database Firewall's Live Capture utility to capture network traffic to a file and analyze it.

Note:

Sometimes unknown_username is displayed in the User field of Database Firewall reports for SQL server. This can be resolved by enabling Retrieve session information from target DB option under the Advanced tab for the Database Firewall monitoring point. The report may also display unknown_osusername in the OS User field of Database Firewall reports for SQL server. This information is available to Database Firewall only if the client uses Windows authentication or a trusted connection.

See Also:

• Configuring Advanced Settings for Database Firewall

• Viewing Network Traffic for a Database Firewall

L.28 Alerts Are Not Generated

Review the resolution to use when alerts that you created are not generated.

Problem

Alerts I have created are not being generated.

Solution

Try the following:

• Examine the alert condition to make sure it is written correctly:

  Log in to the Audit Vault Server console as an auditor, click the Policy tab, click Alerts, and then click the name of the alert in question.

  See Also:

  • Oracle Audit Vault and Database Firewall Auditor's Guide for help in writing alert conditions.

  • Using Audit Vault Server Console for more information about logging in to the Audit Vault Server console.

• Restart the job framework on the Audit Vault Server. See My Oracle Support Doc ID 1592181.1.

L.29 Problems Retrieving or Provisioning Audit Settings on Oracle Target

Learn what to do when you encounter problems while retrieving or provisioning Oracle target audit settings.

Problem

I have a problem either retrieving audit settings form an Oracle Database target, or provisioning audit settings to an Oracle Database target.

Solution

If you have problems retrieving audit settings, try the following:

• Check the job status of the retrieval job for errors:

  Log in to the Audit Vault Server console as an auditor, click Settings, and then click Jobs in the System menu.

• Ensure you have entered the correct connect string in the Oracle Database's target configuration:

  Log in to the Audit Vault Server as an administrator, click the Targets tab, and then click the name of this Oracle target. Check the Target Location field for the connect string.

  See Also:

  Target Locations (Connect Strings)

If you have problems provisioning audit settings, and the Oracle Database target has Database Vault enabled, confirm that the Oracle Audit Vault and Database Firewall user you created on this database has the AUDIT SYSTEM and AUDIT ANY privileges.

L.30 Operation Failed Message Appears When Attempting to Enable Oracle Audit Vault and Database Firewall Policies

Learn how to resolve operation failures when you try to enable Oracle Audit Vault and Database Firewall policies.

Problem

I configured Oracle Audit Vault and Database Firewall for a backup and restore operation. After I completed the procedure, I could not enable an Oracle Audit Vault and Database Firewall policy. The error message Operation failed. Please contact Oracle Support appeared.

Solution

During the backup and restore process, Oracle Audit Vault and Database Firewall must perform a restart of the Oracle Audit Vault Server database. The internal tool Java Framework may need to be restarted. To remedy this problem:

1. Log in to Oracle Audit Vault Server.

2. At the command line, run the following command to check the status of the Java Framework:

   /usr/local/dbfw/bin/javafwk status
   

3. If the output says Java framework process is stopped, then restart it as follows:

   /usr/local/dbfw/bin/javafwk start 

L.31 Out of Memory Error Message During Restore

Learn the resolution when you receive an out of memory error message during a restore.

Problem

Encounter out of memory error while performing restore task.

Solution

Prior to initiating the restore task, ensure that the RAM size and Disk size in the new system is equal or bigger than the original system. This ensures that the out of memory error is not encountered while performing the restore task.

L.32 JAVA.IO.IOEXCEPTION Error

Learn how to resolve a JAVA.IO.IOEXCEPTION error.

Problem

SSL peer shuts down incorrectly with the following error:

JAVA.IO.IOEXCEPTION: IO ERROR:SSL PEER SHUT DOWN INCORRECTLY

Solution

1. Access the target through SSH.

2. Change to the following location using the command:

   cd $ORACLE_HOME/network/admin

3. Edit the sqlnet.ora file. Add parameter sqlnet.recv_timeout=100000 in the file.

4. Restart the target listener.

5. Once the target listener is started, start the agent, and the audit trail.

L.33 Failed to Start ASM Instance Error

Learn what to do when you receive a Failed to start ASM instance error.

Problem

The avdf-upgrade --confirm command stops and results in an error. The command may fail for many reasons. The error mainly occurs due to failure in starting or stopping of a service.

The following is an example of Failed to start ASM instance error:

{{{ 
[support@avs00161e637973 ~]$ su - root 
Password: 
[root@avs00161e637973 ~]# /usr/bin/avdf-upgrade --confirm 
Please wait while validating SHA256 checksum for 
/var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso 
Checksum validation successfull for 
/var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso 
Mounting /var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso on /images 
Successfuly mounted /var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso on /images 
Starting Oracle High Availability Service 
2016-08-05 15:32:09.097: 
CLSD: Failed to generate a fullname. Additional diagnostics: ftype: 2 
(:CLSD00167:) 
CRS-4639: Could not contact Oracle High Availability Services 
CRS-4000: Command Start failed, or completed with errors. 
Starting ASM instance 
Error: Failed to start ASM Instance 
Unmounted /var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso on /images 
Failed to start ASM Instance 
}}}

Solution

Rerun the command avdf-upgrade --confirm

Executing this command again will get past the Failed to start ASM instance error.

L.34 Internal Capacity Exceeded Messages Seen in the /var/log/messages file

Learn how to resolve Internal capacity exceeded messages that appear in the /var/log/messages file.

Problem

Not all the expected traffic is being captured or logged by the Database Firewall, and error messages are present in the /var/log/messages file containing the text Internal capacity exceeded.

Solution - 1

Increase the processing resources available for the target on which the issue is observed through the setting of the MAXIMUM_ENFORCEMENT_POINT_THREADS collection attribute.

See Also:

Registering Targets

Solution - 2

The size of the buffer used for inter-process communication on the Database Firewall can be increased to improve throughput, though at the cost of more memory being allocated by the relevant processes. Please note that this setting is in units of Megabytes, and has a default value of 16. To change the configuration for this value execute the following procedure:

1. Log in to the Audit Vault Server console as the root user.

2. Edit the file /usr/local/dbfw/etc/dbfw.conf. Look for an entry with the key IPC_PRIMARY_BUF_SIZE_MB. If it exists, this is the line to change. If it does not exist, add a new line beginning with IPC_PRIMARY_BUF_SIZE_MB.

3. Change the IPC_PRIMARY_BUF_SIZE_MB line to reflect the required buffer size. For example, if you wished to change the buffer size to 24 megabytes, the configuration line should be IPC_PRIMARY_BUF_SIZE_MB="24". Save the changes.

4. From the command line restart the Database Firewall processes so that the new setting is used with the command line /usr/local/dbfw/bin/dbfwctl restart.

There is also a second setting available to alter the maximum size that the inter-process communication buffer can grow to. It's units are in megabytes, and has a default value of 64 megabytes. To change the configuration for this value execute the following procedure:

1. Log in to the Audit Vault Server console as the root user.

2. Edit the file /var/dbfw/va/N/etc/appliance.conf, where N is the number of the Database Firewall monitoring points in question. Look for an entry with the key IPC_BUF_SIZ_MB. If it exists, this is the line to change. If it does not exist, add a new line beginning with IPC_BUF_SIZ_MB.

3. Change the IPC_BUF_SIZ_MB to reflect the desired maximum buffer size. For example, if you wished to change the buffer size to 80 megabytes, the configuration line should be IPC_BUF_SIZ_MB="80". Save the changes.

4. From the command line restart the Database Firewall processes so that the new setting is used with the command line /usr/local/dbfw/bin/dbfwctl restart.

If the problem persists and after altering the above settings the Internal capacity exceeded error is still encountered, then further investigation by support is required.

Perform the following:

1. Log in to the Audit Vault Server console as the root user.

2. Edit the file /usr/local/dbfw/etc/logging.conf

3. Find the line log4j.logger.com.oracle.dbfw.Metrics=ERROR

4. Comment out this line by placing a # character at the beginning of the line log4j.logger.com.oracle.dbfw.Metrics=ERROR. Save the changes.

5. From the command line restart the Database Firewall processes so that the new setting is used with the command line /usr/local/dbfw/bin/dbfwctl restart

6. Leave the Database Firewall running for several hours under load even while the Internal capacity exceeded error is still encountered.

7. After this period, get the diagnostics output from the Database Firewall as detailed in MOS note How to Collect Diagnostic Logs From Audit Vault Server (Doc ID 2144813.1). Provide the diagnostics output to support for further analysis.

L.35 First Archive Or Retrieve Job After Upgrade

Learn what to do if after an upgrade, the first archive or rervireve job submission displays the status of Starting.

Problem

After upgrade the first archive or retrieve job submission may display the status as Starting.

Solution

Submit the job again. This is a known issue and subsequent submission of job succeeds.

L.36 Audit Vault Agent Installation Fails After HA Pairing Or Separation

Learn what to do after the Oracle Audit Vault installation fails after an HA pairing or separation.

Problem

Installation of Audit Vault agent fails after performing pairing or separation (un-pairing) of Oracle Audit Vault server.

The following command generates agent debug logs during agent installations.

java -jar agent.jar -v

Symptoms

The following errors may be found during agent installation in the agent log file:

PKIX path validation failed

signature check failed

Solution

After the pairing or separating of Oracle Audit Vault servers, you must download the Audit Vault agent from the GUI and install the agent again after removing the existing Audit Vault Agent.

See Also:

Post High Availability Pairing Steps

If the Audit Vault agent fails to install after pairing or separating of Audit Vault server, then install the Audit Vault agent using -v option.

To resolve the above errors, follow the steps mentioned below:

1. Log in to the Audit Vault server as user root.

2. Run the following script to generate a new agent.jar file.

   /usr/local/dbfw/bin/priv/update_connect_string_ip.sh

3. Download the new agent.jar file from the GUI.

4. Install the newly downloaded agent.jar file.

L.37 Error in Restoring Files

Learn what to do when you encounter errors while restoring files.

Problem

An attempt to restore the data files results in a failure. The restore job completes successfully, however the data files are not restored. There is no information in the restore job log file.

Solution

Check for the following to troubleshoot the issue:

• The restore policy must follow the guidelines listed under the section Configuring Archive Locations and Retention Policies.

• Check the tablespace that needs to be archived and the corresponding tablespace that needs to be purged as per the policy defined.

• Restoring data into empty tablespaces is not possible. Check accordingly.

• In case the tablespace enters the delete period, it is deleted automatically from Oracle Audit Vault Server.

• Every tablespace is uniquely identified by the month it moves offline and the month during which it is purged. They are created automatically based on the policies that you create.

• When the retention policy is changed, the new policy is applied to the incoming data immediately. It does not affect existing tablespaces that adhere to the old policy.

• You can archive the tablespace when it enters the offline period.

• After restoring the tablespace, it is online. Once it is released, it goes offline. The tablespace must be rearchived once released.

L.38 DB2 Collector Fails Due to Source Version NULL Errors

If the DB2 collector fails due to source version NULL errors, then follow these steps.

Problem

The following error or trace is displayed in the collector log file.

Caused by: java.lang.ClassNotFoundException:

sun.io.MalformedInputException

at java.net.URLClassLoader.findClass(Unknown Source)

at java.lang.ClassLoader.loadClass(Unknown Source)

Solution

Check the Java version on the host system This failure is due to Java SE version 8. Attempt to use Java SE 7.

Note:

This issue may be encountered in releases prior to 12.2.0.11.0.

L.39 DB2 Collector Fails Due to Database Connection or Permission Issues

If the DB2 collector fails due to database connection or persmission issues, then follow these steps.

Problem

The following error or trace is displayed in the collector log file.

Caused by: oracle.ucp.UniversalConnectionPoolException: Cannot get Connection from Datasource: java.sql.SQLSyntaxErrorException: [Audit Vault][DB2 JDBC Driver][DB2]<User> DOES NOT HAVE PRIVILEGE TO PERFORM OPERATION EXECUTE ON THIS OBJECT NULLID.DDJC360B

Solution

Run the following command for successful execution of DB2 collector:

grant execute on package NULLID.DDJC360B to <User> (user while registering the target)

L.40 ORA-12660 Error While Registering Target

Learn how to resolve the ORA-12660 error.

Problem

Audit Vault agent fails with ORA-12660 error.

Solution

The server encryption is set to REQUIRED in on-premises by default. Set the server encryption to ACCEPTED or REQUESTED or REJECTED.

Note:

REJECTED is not a recommended option. The following table describes these options in detail.

Table L-1 Server Encryption Types

Option	Description
ACCEPTED	The server does not enable both encrypted and non-encrypted connections. This is the default value in case the parameter is not set.
REJECTED	The server does not enable encrypted traffic.
REQUESTED	The server requests encrypted traffic if it is possible, but accepts non-encrypted traffic if encryption is not possible.
REQUIRED	The server accepts only encrypted traffic.

L.41 Audit Trail Performance Issues Occur After Audit Vault Server Upgrade

Learn what to do when audit trail performance issues occur after upgrading Oracle Audit Vault Server.

Problem

You might experience audit trail performance issues after upgrading Oracle Audit Vault Server.

Solution

The audit_trail_id_idx index that is created resolves the performance issues encountered. However, you must retain sufficient disk space if there is large amount of event data for the period prior to upgrading Oracle Audit Vault Server. The amount of disk space required is about 5% of the total event log data size.

L.42 Failures Due to Dropping Users

Learn how to resolve failures that occur when dropping users.

Problem

Failed to drop the user with an error message and the user was not listed in the Audit Vault Server GUI.

Solution

Contact Oracle Support for the best workaround and to drop the user manually using SQL*Plus.

L.43 Failure of Agent Automatic Upgrades

Learn what to do when agent automatic upgrades fail.

Problem

The automatic upgrade of the Agent fails with the following error. This is because the Agent is unable to connect to the Audit Vault Database.

Message: Exception occurred while updating Agent.
Cause: Unable to connect to AV Server.
Note: Agent will try to re-connect automatically in 10 seconds.

Solution

The Agent attempts to connect to the Audit Vault Database and auto upgrade after 10 seconds. Check the Oracle Audit Vault Database connection or contact Oracle Support.

L.44 Some Services May Not Start After Backup

Learn what to do when services fail to start after a backup.

Problem

The system may not be stable after a cold backup operation failed to complete.

Solution

Oracle recommends that you reboot the system if there is a failure while performing a cold backup operation.

L.45 Data Overflow Issues in the Oracle Audit Vault UI

Learn how to resolve data overflow issues in the Oracle Audit Vault UI.

Problem

The Recently Raised Alerts Report region appears on your dashboard and displays the list of alerts with data overflowing in the Audit Vault GUI. This may occur when you launch the GUI using Internet Explorer and the Microsoft Windows Server operating system.

Solution

To fix this issue and to display the data properly on the Audit Vault GUI, you should make minor changes to the Internet Explorer browser settings. Press F12 and click the Emulation tab.

Change the Document mode and Browser profile fields from the default settings. For example, change the Document mode value to 10 from the drop down menu and change the Browser profile field to Desktop.

L.46 Oracle Audit Vault Agent is Unreachable and the Transaction Log Audit Trail is Frozen in Starting Status

Learn what to do when the Oracle Audit Vault Agent is unreachable and the transaction log audit trail is frozen in Starting status.

Problem

The status of Oracle Audit Vault Agent is unreachable from the AV GUI. The status of the Transaction Log audit trail persistently remains in the Starting status.

This may be due to a user application that is blocking the creation of streams by ORAAUDIT user.

Symptom

The Transaction Log audit trail does not start. The following information may be found in the thread dump that is taken using jstack tool:

oracle.av.platform.agent.collfwk.impl.redo.RedoCollector.sourceSetup(RedoCollector.java:634) 

Solution

Terminate the user application that is blocking the creation of streams. Restart the Transaction Log audit trail.

L.47 Scheduled PDF or XLS Reports Result in a Hung State

To resolve a hung state that occurs for scheduled PDF or XLS reports, follow these recommendations.

Problem

Scheduled PDF or XLS reports remain incomplete for an extended period of time or ramin in q RUNNING state.

Solution

You can schedule reports to be sent to other users in PDF or XLS formats. Avoid triggering or scheduling concurrent long-running reports at the same time. Producing PDF and XLS reports occupies a lot of system resources because there is a significant amount of data involved. Scheduled concurrent long-running reports can remain in a hung state indefinitely. The reports must be scheduled with staggered intervals in between. For example, run the reports at intervals of 5, 10, or 20 minutes.

L.48 Pending Reports Remain in Scheduled Status

To resolve pending reports that remain in scheduled status, follow these steps.

Problem

Many reports are stuck in scheduled or pending status. These reports may never be completed and may be stopped.

Solution

This may be due to an issue with the Java Framework process in the background. Use these steps to check and resolve this issue:

1. Log in to the CLI as support user.

2. Switch to root user using the command:

   su root

3. Run the following command to check the status of the Java Framework:

   systemctl status javafwk

4. Stop the Java Framework even if it is running. Run the following command:

   systemctl stop javafwk

5. Run the following command to start the Java Framework:

   systemctl start javafwk

6. Run the following command to restart the Java Framework:

   systemctl restart javafwk

Use the following procedure to check the status of the reports from the operating system logs after running one of the procedures mentioned above and restarting the Java Framework:

1. Log in to AVCLI as admin user.

2. Run the following command to enable diagnostics for the reports:

   ALTER SYSTEM SET loglevel=ReportLog:DEBUG|JfwkLog:DEBUG;

3. The diagnostics can also be enabled using the Oracle Audit Vault Server console by following these steps:

   1. Log in to the console as admin user.
   2. Click Settings tab.
   3. Click on Diagnostics on the left navigation menu.
   4. Select Debug against Report Generation.
   5. Click Save.

4. Run a PDF report. For example, Activity Overview.

   1. Log in to the Oracle Audit Vault Server console as auditor.
   2. Click Reports tab.
   3. Click Activity Reports under Built-in Reports.
   4. In the Activity Reports tab on the screen, you can schedule a report and view the generated report.

5. After a while, check on the /var/lib/oracle/dbfw/av/log file. For example, av.report* file. It contains the PDF/XLS report generation debug logs.

L.49 Audit Vault Log Displays a Message to Install WinPcap and OpenSSL

To resolve the Audit Vault log message to install WinPcap and OpenSSL, follow these steps.

Problem

The Host Monitor Agent can collect audit data from Windows 2016 servers. A message displays alerting you to install WinPcap and OpenSSL.

Solution

A set of DLL files may be causing issues. Run the following procedure to resolve this problem:

1. Search for the following files in the system:

   • ssleay32.dll
   • libeay32.dll
   • wpcap.dll
   • packet.dll

2. Append the file names with the .bk format notation.

3. Go to Control Panel then to Uninstall Programs and uninstall OpenSSL and WinPcap.

4. Reinstall WinPcap and OpenSSL 1.0.2.q (64-bit). The DLL files are restored to Windows system folder.

5. Check the Control Panel to verify that these two programs are installed.

6. Go to C:\Windows\System32 or C:\Windows\SysWOW64 folders and search for the above four DLL files. At least one file for each DLL must be present without the .bk extension.

7. Go to the OpenSSL installation location and search for libssl-1_1-x64.dll and libcrypto-1_1-x64.dll files. One for each type is available.

8. Upon confirmation, add the C:\Windows\System32 or C:\Windows\SysWOW64 to the path variable.

9. Restart the trail.

10. If the network audit does not start, then check the collfwk logs present at <AgentHome>\av\log location. If the following message is available in the collfwk log, then check the Host Monitor Agent logs present at <AgentHome>\hm\log location.

    <AgentHome> refers to the Audit Vault Agent installation directory.

    Note:

    Continue with the remaining steps if your installation is 12.2.0.10.0 or before. The steps are not required for release 12.2.0.11.0 and later.

11. If the following message is available in the Host Monitor Agent log, then execute the remaining procedure:

    Invalid AVS Credentials provided

12. Open the av/conf/bootstrap.prop file.

13. Copy the following line:

    CONNECT_STRING_PARAM_POSTFIX=9999

14. Paste this line in the hm/bootstrap.prop file.

15. Restart the trail.

16. In case the network audit trail starts without any errors, then the collection status on the Audit Vault Server console confirms the same.

17. Navigate to AVAUDIT then to Target then Firewall Policies and, finally, Log All.

18. Connect to the target database instance using SQL Developer, or any other tool.

19. Generate the traffic for collecting data.

20. It must be recorded in the reports of the event_log table.

L.50 Error OAV-47409 While Managing Archive Locations

Learn what to do when you receive the OAV-47409 error while managing archive locations.

Problem

The following error message displays in the Auto Archive Message column under Manage NFS Locations tab:

OAV-47409: Absolute path does not exist on remote filesystem
ORA-06510: PL/SQL: unhandled user-defined exception

The configured path of the archive location is either missing or outside of the remote filesystem.

Solution

The value under Auto Archive Order column is set to 0 [zero]. The system has set this value as the archive location is problematic. You must ensure that the NFS location issue is resolved to a valid directory on the remote filesystem. Upon resolving this issue, set the value under Auto Archive Order column to 1 or higher. This sets the appropriate priority for the auto archive order.

L.51 Error OAV-47402 While Defining Archive Locations Using NFS Mount Point

Learn what to do when you receive the OAV-47402 error while defining archive locations.

Problem

An error is observed after registering the archive location using NFS mount point through AVCLI. The created remote file system shows inaccessible when running the SHOW STATUS command. The following error is observed when running ALTER REMOTE FILESYSTEM <file system name> MOUNT command. However, the process of defining or creating the archive location is successful.

OAV-47402: Unable to mount export /exabackup from host <host Ip address>

Solution

This issue is observed when using NFS version v3 only. Reach out to the NAS storage support or NFS administrator support team to verify if the mount point in the NFS server is properly configured. It must support both v3 and v4 to integrate with Oracle AVDF.

Note:

NFS version v3 only is not supported for Oracle AVDF releases 20.3 and prior. It is supported starting Oracle AVDF release 20.4.

Follow the steps documented in My Oracle Support Doc ID 2232033.1 to verify if the mount point in the NFS server is properly configured.

See Defining Archive Locations for complete information.

L.52 Audit Trail Stopped After Relocating Windows Event Log Files

Use this procedure when the audit trail stops after you relocate the Windows event log files.

Problem

Windows event log relocation causes audit trail to be stopped.

Solution

Follow this procedure to resolve this problem:

1. Stop the audit trail.
2. Drop the audit trail.
3. Restart the audit trail. The new trail recognizes the new location for event logs.

L.53 Missing or Incomplete Client Information in Oracle Database Firewall Logs

Learn how to resolve missing or incomplete client information in Oracle Database Firewall logs.

Problem

Empty client information in the Oracle Database Firewall logs after upgrading Oracle Audit Vault and Database Firewall. The logs that are generated are missing some of the client information such as the user name.

Note:

This issue occurs only when you are in DAM mode deployment of Oracle Database Firewall. You will not experience this issue in the Proxy mode deployment.

Cause

Oracle Database Firewall records information that is related to the TCP sessions during inspection and it saves this data to disk. This recorded information includes client user names and other metadata about the connection. When Oracle Database Firewall processes are restarted after a configuration change or an upgrade, Oracle Database Firewall continues to generate logs accurately by re-reading this cached information.

The format of the cache file has changed in the recent releases. Oracle Database Firewall may not be able to read the file in the old format. Therefore, existing client connections to the database that were established before performing the upgrade may not retain certain information such as client user names. This can lead to logs missing information such as the client username.

Solution

Restart the database clients.

L.54 Issues with Retrieving Session Information Through Clients Connecting to Microsoft SQL Server

Learn what to do when you have issues retrieving session information through clients that connect through Microsoft SQL Server.

Problem

Database Firewall is unable to retrieve session information through some clients (for example, MS SQL Server Management Studio) as the information is encrypted. You can retrieve session information for non Oracle databases to obtain the name of the database user, operating system, and client program that originated a SQL statement.

Symptom

Audit Reports show unknown user names and unknown program names where the target is Microsoft SQL Server.

Solution

Ensure the following steps are accurate while registering Microsoft SQL Server as a target.

1. In the User Name field, enter the user name of the system administrator.
2. In the Password field, enter the password of the system administrator.
3. In the Host Name / IP Address field, enter the IP address of the SQL Server.
4. In the Port field, enter the port of the SQL server listening port.
5. In the Service Name field, enter a valid database service name on SQL Server. In case the database service name is not correct, then SQL server DDI requests fail on the SQL Server with invalid request error.

   Note:

   If the secured type is not Oracle, then the Service Name field must be empty. This field is designated for a specific Oracle Service Name (OSN) and is not applicable to any other database type. If this field is not blank, then no traffic will be recorded, as per the reported symptoms.

See Also:

• Registering Targets
• Setting Permissions to Retrieve Session Information in Microsoft SQL Server

L.55 Performance Issues Due to High Memory Usage

Learn how to address performance issues in Oracle AVDF with very large deployments.

Problem

Audit Vault Server in large deployments may have performance issues due to increased memory usage.

Solution

• Ensure the Audit Vault Server is sized as per the sizing guidelines documented in Audit Vault and Database Firewall Best Practices and Sizing Calculator for AVDF 12.2 and AVDF 20.1 (Doc ID 2092683.1).
• Audit Vault Server has Transparent Huge Pages set by default which should work in most cases. However, in some cases it has to be disabled by setting transparent_hugepages to never. This helps in improving the performance. For detailed the steps, refer to Oracle Linux 7 - How to disable Transparent HugePages for RHCK kernel? (Doc ID 2066217.1).
• If you still face performance issues after applying the above mentioned solution, contact Oracle Support.

L.56 httpd Crash Issue on Database Firewall

Learn how to fix httpd crash issue in Database Firewall.

Problem

The httpd process in Database Firewall may crash under some circumstances.

Symptom

The status of the Database Firewall instance appears Down in the Audit Vault Server console. The Database Firewall logs are not transferred to the Audit Vault Server.

The following is observed in the log files of the impacted Database Firewall instance. The httpd.service file in /etc is symlinked to the file in /usr path.

# ls -l /etc/systemd/system/multi-user.target.wants/httpd.service

lrwxrwxrwx. 1 root root 37 Nov 27 09:26 /etc/systemd/system/multi-user.target.wants/httpd.service -> /usr/lib/systemd/system/httpd.service

# ls -lL /etc/systemd/system/multi-user.target.wants/httpd.service

-rw-r--r--. 1 root root 752 Nov 10 20:33 /etc/systemd/system/multi-user.target.wants/httpd.service

#

Solution

Follow these steps to change the configuration of the system and restart the httpd process:

1. Log in to the Database Firewall instance as root user.

2. Check and confirm that the above mentioned symptom exists.

3. Copy the base file from /usr to /etc by running the following command:

   # install -m 0644 -o root -g root /usr/lib/systemd/system/httpd.service /etc/systemd/system/httpd.service

4. Edit the file in /etc and find the below mentioned Service block:

   # vi /etc/systemd/system/httpd.service

   ...

   [Service]

   Type=notify

   EnvironmentFile=/etc/sysconfig/httpd

   ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND

   ...

5. Modify the file and add the following code to include the restart failure directive. The file looks like the folllowing:

   
   ...
   [Service]
   Restart=on-failure
   Type=notify
   EnvironmentFile=/etc/sysconfig/httpd
   ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
   ...
   

6. Save the file.

7. Disable and re-enable the service to fully apply the following changes:

   
   # systemctl disable httpd
   Removed symlink /etc/systemd/system/multi-user.target.wants/httpd.service.
   # systemctl enable httpd
   Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /etc/systemd/system/httpd.service.
   #
   

8. Verify the following changes:

   
   # sha256sum -c - <<EOF
   eac607c17f2c122619b3e1459eafdfef6bde003d24964891aa506735df4f55c2  /etc/systemd/system/multi-user.target.wants/httpd.service
   EOF
   /etc/systemd/system/multi-user.target.wants/httpd.service: OK
   #
   

9. Reload the systemd configuration and restart httpd by running the following commands:

   # systemctl daemon-reload

   # systemctl restart httpd

10. Verifying the service is enabled by running the following command:

    # systemctl list-unit-files | grep http

11. Observe the following output:

    httpd.service enabled

    #

12. If the daemon subsequently fails, the systemd will restart it, and write the following example audit trail to the system log:

    Nov 27 08:38:09 example systemd: httpd.service: main process exited, code=killed, status=11/SEGV

    Nov 27 08:39:40 example systemd: httpd.service stop-sigterm timed out. Killing.

    Nov 27 08:39:40 example systemd: Unit httpd.service entered failed state.

    Nov 27 08:39:40 example systemd: httpd.service failed.

    Nov 27 08:39:40 example systemd: httpd.service holdoff time over, scheduling restart.

    Nov 27 08:39:40 example systemd: Stopped The Apache HTTP Server.

    Nov 27 08:39:40 example systemd: Starting The Apache HTTP Server...

    Nov 27 08:39:40 example systemd: Started The Apache HTTP Server.

L.57 Issue with Retrieval of Return Row Count

Learn how to fix the issue related to retrieval of return row count.

Problem

Database Firewall captures the number of rows returned by a SELECT query and display them in reports under the column Row Count.

If the database takes a while to generate response result set, then return row count may not be extracted due to timeout configuration.

Workaround

Follow these steps to adjust the timeout interval:

1. Log in to the Database Firewall through SSH and switch to the root user.

   See Logging In to Oracle AVDF Appliances Through SSH.

2. Change to /var/dbfw/va directory.
3. Identify the Database Firewall monitoring point by searching for the target name configured in the Audit Vault Server. Run the following command:

   grep -lr <TARGET NAME> *

4. Find the monitoring point number from the output which contains the name and path of the configuration file. For example: 1/etc/appliance.conf. In this example, 1 is the monitoring point number.

5. Change the directory to the identified monitoring point and open configuration file of the appliance.

6. Search for the following entry in the file:

   MAX_LOG_FILE_TIMERANGE

7. Modify the MAX_LOG_FILE_TIMERANGE line to reflect the required time range in seconds. For example, if you wish to change the time range to 5 minutes, then the configuration line should be MAX_LOG_FILE_TIMERANGE=="300".

8. Save the changes.

9. Run the following command to restart the Database Firewall processes so that the new setting takes effect:

   /usr/local/dbfw/bin/dbfwctl restart <monitoring point number>

   In this case the monitoring point number was 1.

   Hence, the command should be:

   /usr/local/dbfw/bin/dbfwctl restart 1

Note:

Increasing the timeout configuration delays the availability of captured SQL statements in the reports and any alerts configured for the same. Use your discretion while configuring the above value close to the actual query completion time.

L.58 Unable to Log in to the Oracle AVDF Appliance through SSH

Learn how to fix log in issue to Oracle AVDF appliance.

Problem

The user is unable to log in to the Oracle AVDF appliance through SSH. This may be because of using old SSH clients to log in to the Oracle AVDF appliance.

Workaround

Log in to ARU (Automated Release Updates). Apply the patch number 32287150 that solves the problem.

Note:

This patch must be applied on Oracle AVDF 20.3 and later only.

L.59 Error When Changing IP Address of Management Interface

Learn how to resolve the error encountered when changing the IP address of the Management Interface.

Problem

The Management Interface IP address is the IP address of the Database Firewall which was used to register the Database Firewall in the Audit Vault Server console.

In Oracle AVDF 20.1, the following error may be encountered while attempting to change the IP address of the Management Interface:

Operation failed OAV-46981: Unable to connect to Database Firewall with IP <ipaddress>

Solution

This error may come up because the IP Address of the Database Firewall is changed successfully. However, there may be a delay in the response from Database Firewall. It may take a few seconds for the network update on the Database Firewall and for the system to settle.

Click Save and Close buttons to exit the dialog. Do not click on the cross (X) mark in the top right corner of the dialog.

L.60 Unable to Configure Microsoft SQL Server XEL Audit Trail After Upgrade

Problem

The following error is observed while configuring Microsoft SQL Server XEL audit trail on Audit Vault Server after upgrading to Oracle AVDF 20.3:

[oracle][SQLServer JDBC Driver][SQLServer]VIEW SERVER STATE permission was denied on object 'server', database 'master'

Solution

Follow these steps to resolve this issue in Oracle AVDF 20.3:

1. Create a new user on Microsoft SQL Server target database.
2. Grant the necessary privileges. See Oracle AVDF Administrators Guide for complete information.
3. Modify the registered target with the newly created user credentials.
4. Configure the Microsoft SQL Server XEL audit trail.

This issue is resolved in Oracle AVDF 20.4. Follow these steps after upgrading to Oracle AVDF 20.4 (or later):

1. Revoke audit data collection privileges by running the mssql_drop_db_permissions.sql script as follows:

   sqlcmd -S server_name -U sa -i mssql_drop_db_permissions.sql -v username="username" mode="AUDIT_COLL" all_databases="NA" database="NA"

2. Run the mssql_user_setup.sql script as follows:

   sqlcmd -S server_name -U sa -i mssql_user_setup.sql -v username="username" mode="AUDIT_COLL" all_databases="NA" database="NA"

3. Configure the Microsoft SQL Server XEL audit trail.

L.61 Transaction Log Audit Trail Stops Due to an Error While Parsing XML File Containing Emoji

Problem

Transaction Log audit trail stops while parsing a file that contains emoji. The following error is observed in the Agent logs:

javax.xml.stream.XMLStreamException: ParseError at [row,col]

Solution

Follow these steps to resolve this error:

1. Run the following command to stop the Audit Vault Agent:

   AGENT_HOME/bin/agentctl stop

   .
2. Delete the sjsxp.jar file present in the AGENT_HOME/av/jlib directory.
3. Run the following command to start the Audit Vault Agent:

   AGENT_HOME/bin/agentctl start

   .

L.62 Unable to Find the FIPS Status for Database Firewall Instance

Learn how to fix the error when the FIPS status for a Database Firewall instance is not displayed in the Audit Vault Server console.

Problem

The FIPS status for the Database Firewall instance could not be determined from the Audit Vault Server console.

Solution

Perform the following checks to determine the root cause of the problem:

• The Database Firewall version is 20.4 or later.
• Check the network connectivity between the Audit Vault Server and the two Database Firewall instances.
• Ensure the Audit Vault Server's certificate is correctly copied or installed on the Database Firewall instance.
• Check if the Audit Vault Server can connect to the Database Firewall by confirming that the status of the Database Firewall instance is online.

If none of the above points are helpful in identifying the cause of the problem, then contact Oracle Support.

L.63 Unable to Modify the Database Firewall FIPS Mode Through Audit Vault Server Console

Learn how to fix the error when the FIPS mode cannot be modified through the Audit Vault Server console.

Problem

This could be caused due to a communication issue between the Audit Vault Server and the Database Firewall instances.

Solution

Perform the following checks to determine the root cause of the problem:

• The Database Firewall version is 20.4 or later.
• Check the network connectivity between the Audit Vault Server and the two Database Firewall instances.
• Ensure the Audit Vault Server's certificate is correctly copied or installed on the Database Firewall instance.
• Check if the Audit Vault Server can connect to the Database Firewall by confirming that the status of the Database Firewall instance is online.

If none of the above points are helpful in identifying the cause of the problem, then contact Oracle Support.

L.64 The FIPS Status on Both the Database Firewall Instances is Different

Learn how to fix the error when the FIPS mode is different on both the Database Firewall instances.

Problem

The FIPS mode is different on both the Database Firewall instances. This could be caused when FIPS mode is manually changed on one of the Database Firewall instances. It can also be caused when such an attempt to manually change the FIPS mode failed.

Solution

All the Database Firewall instances that are part of high availability must have the same FIPS 140-2 mode. The FIPS 140-2 status of the Database Firewall instances must either be Off or On.

FIPS 140-2 mode can be disabled or enabled on both the Database Firewall instances. In case, these two instances have different FIPS mode, then an error message is displayed on the screen.

Verify the high availability status of the Database Firewall instances, and change the FIPS mode again.

L.65 After Restarting Secondary Audit Vault Server, the Primary Instance Fails to Switchover

Learn how to fix a switchover issue on the primary Audit Vault Server, after the secondary instance is restarted.

Problem

After restarting the secondary Audit Vault Server, the switchover status of the primary Audit Vault Server shows NOT ALLOWED state.

This status of the primary Audit Vault Server is not recoverable and the following error messages appear and are repeated every 50 seconds on the secondary Audit Vault Server:

<Date> <avs-instance-name> observerctl: com.oracle.avs.observerctl DEBUG - DGMGRL:[W000 <date and timestamp>] The primary database has requested a transition to the UNSYNC/LAGGING state with the standby database DBFWDB_HA2.
<Date> <avs-instance-name> observerctl: com.oracle.avs.observerctl DEBUG - DGMGRL:[W000 <date and timestamp>] Permission granted to the primary database to transition to LAGGING state with the standby database DBFWDB_HA2.
<Date> <avs-instance-name> observerctl: com.oracle.avs.observerctl DEBUG - DGMGRL:[W000 <date and timestamp>] Reconnect interval expired, create new connection to primary database.
<Date> <avs-instance-name> observerctl: com.oracle.avs.observerctl DEBUG - DGMGRL:[W000 <date and timestamp>] The primary database has been in LAGGING state for 7138 seconds.

Solution

In case the primary Audit Vault Server's switchover status goes into NOT ALLOWED status after restarting the secondary instance, then follow the steps mentioned in MOS Note (Doc ID 1258074.1) to restart the standby Audit Vault Server.

L.66 Incorrect Syntax Near Connectivity Entry in Audit Logs

Learn how to fix incorrect syntax error entry in audit logs.

Problem

When attempting to add an audit trail for Microsoft SQL Server, the Audit Vault Agent attempts to acquire a target connection using JDBC driver. After the connection is established, a test query is sent to validate the connection by the JDBC driver.

This test query may generate the following error:

Incorrect syntax near ‘Connectivity’

This error is visible in the database audit records.

Solution

Starting Oracle AVDF release 20.6, to avoid unnecessary logging of records or events due test queries in the target database, define the collection attribute as follows:

av.collector.validateConnectionOnBorrow = false

See Also:

Microsoft SQL Server Plug-in for Oracle Audit Vault and Database Firewall

L.67 Certificate Regenerate Failure Error

Learn how to fix a certificate regenerate failure error.

Problem

In case the certificate regenerate operation fails, then one of the possible reasons can be the incorrect date and time of the appliance (Audit Vault Server or Database Firewall).

Solution

Specify the correct time, and then run the following command to regenerate the certificate:

/usr/local/bin/gensslcert create-certs

To retrieve the details about certificate expiry date, run the following command:

openssl x509 -enddate  -startdate -noout -in {certificate path}

For example:

openssl x509 -enddate  -startdate -noout -in /usr/local/dbfw/etc/cert.crt
notAfter=Oct 17 17:44:53 2022 GMT
notBefore=Sep 14 17:44:53 2021 GMT

Note:

The audit trails go to UNREACHABLE state for about 45 minutes after the certificates are rotated and all the relevant services are restarted. The trails continue to work normally after that. This behavior is observed in Oracle AVDF release 20.6 only.

L.68 User Entitlement or Audit Policy Job Stuck in Running State

Learn how to manage the user entitlement or audit policy job stuck in RUNNING state.

Problem

The user entitlement job or audit policy job is stuck in RUNNING state for a long time. This job is stuck and has to be manually stopped.

Workaround

This issue may be due to an issue with the Java Framework process in the background. Follow these steps and submit the job again:

1. Log in to the Audit Vault Server as support user through SSH.

2. Switch to root user by running the following command:

   su root

3. Restart the Java Framework by running the following command:

   systemctl restart javafwk

L.69 Audit Trails are Toggling Between COLLECTING and UNREACHABLE Status

Learn how to fix the incorrect audit trail status issue.

Problem

The Audit Trails tab in the Audit Vault Server console displays the status of all the audit trails. Some audit trails are continuously toggling between the status COLLECTING and UNREACHABLE.

The trails go to UNREACHABLE state if they take more than 120 seconds (2 heartbeat intervals) to update the trail status. This can happen if either the target or Audit Vault Server is temporarily loaded, causing the trails to take more time to update the trail status.

Solution

Consider increasing the heartbeat interval to 120 seconds. Currently, the default value is 60 seconds. Run the following command as avsys user:

exec avsys.adm.add_config_param('SYS.HEARTBEAT_INTERVAL', 120);

Note:

This scenario is applicable for Oracle AVDF releases 20.5 and earlier, where the default value is 60 seconds. Starting with Oracle AVDF 20.6, the default value is 120 seconds.

L.70 Displaying Job Status Takes Lot of Time in the Audit Vault Server Console

Learn how to resolve the Jobs dialog issue.

Problem

The Jobs dialog in the System tab takes lot of time to load and to display the jobs and their current status.

Solution

Delete unwanted or old data from the Status column. This resolves the issue and the Jobs dialog displays the required information.

For example: Delete unwanted or old data from the avsys.job_status table that is more than 30 days old using the following SQL query:

Delete from job_status
where status = 'Completed'
and status_time < sysdate - 30;

L.71 Microsoft SQL Server Database Audit Trails are in Stopped State After Upgrading Java

Learn how to fix issue when audit trails belonging to Microsoft SQL Server database go to stopped state after upgrading Java version u291 or greater.

Problem

Audit trails that belong to Microsoft SQL Server database are not collecting audit data. This issue is observed after upgrading the Java version to u291 or greater and when Microsoft SQL Server target’s connect string is one of the following:

• jdbc:av:sqlserver://<MSSQL Host name>:<Port number>;encryptionMethod=SSL; validateServerCertificate=false;
• jdbc:av:sqlserver://<MSSQL Host name>:<Port number>;encryptionMethod=SSL;validateServerCertificate=true; trustStore=<key store jks path>;trustStorePassword=<keystore password>;extendedOptions=enableCipherSuites=SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA

Solution

Modify the connect string for Microsoft SQL Server database (in Audit Vault Server console or AVCLI) to one of the following:

• jdbc:av:sqlserver://<MSSQL Host name>:<Port number>;encryptionMethod=SSL;validateServerCertificate=false;CryptoProtocolVersion=TLSv1.2;
• jdbc:av:sqlserver://<MSSQL Host name>:<Port number>;encryptionMethod=SSL;validateServerCertificate=true;CryptoProtocolVersion=TLSv1.2;trustStore=<key store jks path>;trustStorePassword=<keystore password>;extendedOptions=enableCipherSuites=SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA

L.72 Unable to Delete Database Firewall

Learn how to fix an issue observed when attempting to delete Database Firewall.

Problem

An error OAV-47704 is observed when attempting to delete Database Firewall. This issue is observed in the following scenario:

• Oracle AVDF releases 20.1 to 20.5
• Audit Vault Server is upgraded to Oracle AVDF 20, but Database Firewall is not upgraded to Oracle AVDF 20
• Error observed in the Audit Vault Server console or in AVCLI

Solution

This issue is fixed in Oracle AVDF release 20.6. In case the installed version is Oracle AVDF releases 20.5 and earlier, then follow these steps:

1. Log in to the Audit Vault Server through SSH.

2. Switch user to root:

   su root

3. Switch user to dvaccountmgr:

   su dvaccountmgr

4. Start SQL*Plus connection without the username or password:

   sqlplus /nolog

5. Unlock avsys user and assign a password by running the command:

   alter user avsys identified by <pwd> profile default account unlock;

6. Run the command:

   exit

7. Start SQL*Plus connection without the username or password:

   sqlplus /nolog

8. In SQL*Plus run the command:

   connect avsys

9. Enter the password when prompted. Alternatively, run the command:

   connect <avsys/password>

10. Run the following SQL query:

    select id from avsys.firewall where name= '<firewall_name> ' and deleted_at is null;

11. Make a note of the Database Firewall ID.

12. Run the command:

    update avsys.firewall set software_version=’<avs_version>’ where id=<firewall_id>;

    For example: update avsys.firewall set software_version=’20.5.0.0.0’ where id=<firewall_id>;

13. Run the command:

    commit;

14. Repeat the process for any other Database Firewall instance that needs to be deleted.

15. Run the command:

    exit

16. Attempt to delete the Database Firewall instance from the Audit Vault Server console or through AVCLI.

L.73 Issue in Language Setting of the Audit Vault Agent

Learn how to fix the language setting in Audit Vault Agent.

Problem

Unable to change or set the language in Audit Vault Agent. Audit Vault Agent supports languages other than English.

Audit Vault Agent uses the language specified in the locale settings of the host machine (Agent machine), provided the language is supported. In case the specific language is already set on the system, then there is no need to change the settings for the Agent to use the specific language.

Solution

The locale settings for the Windows platform can be changed through the Control Panel on the Windows host machine.

To change the locale settings on Linux/Unix/AIX/Solaris platform, set the LC_ALL and LANG environment variables.

For example:

export LC_ALL=fr_FR.iso88591

export LANG=fr_FR.iso88591

L.74 Unable to Create a Database Firewall Monitoring Point

Learn how to fix an error while creating a Database Firewall monitoring point.

Problem

An attempt to create a Database Firewall monitoring point using the target host name does not succeed.

Symptom

• Failure to create a Database Firewall monitoring point using the target host name displays the status as Starting. The status changes to Unreachable after a while.

• The /var/log/messages file in Database Firewall contains an error similar to the following:

  
  May 10 11:06:02 dbfw08002718dd46 hostname_lookup.rb[19691]:
  foobar.example.com.oracle.dbfw.hostname-lookup WARN - ODF-10505: Failed to resolve hostname:
  Unable to resolve the hostname ["hostname1.foobar.example.com"].
  Verify DNS settings. Hostname resolution will be tried every minute.
  

Solution

DNS is not configured and hence the above error is observed. Configure the DNS and attempt to create the Database Firewall monitoring point again.

In case DNS is configured, verify the DNS settings. Attempt to resolve the host name is made once every minute.

L.75 Issue with Configuring or Managing Oracle AVDF through Oracle Enterprise Manager Cloud Control

Learn how to solve an issue with configuring or managing Oracle AVDF through Oracle Enterprise Manager Cloud Control.

Problem

Unable to configure or manage Oracle AVDF through Oracle Enterprise Manager Cloud Control.

Solution

Oracle AVDF plug-in is an interface within Oracle Enterprise Manager Cloud Control for administrators to manage and monitor Oracle AVDF components. Refer to System Monitoring Plug-in User's Guide for Audit Vault and Database Firewall in case of any issues when configuring the Oracle EM plug-in.

Refer to Compatibility with Oracle Enterprise Manager to check the supported versions of Oracle Enterprise Manager with Oracle AVDF 20.

L.76 Unable to Connect to Audit Vault Server through Console or SSH

Learn how to resolve if you are unable to log in to Audit Vault Server through the console or SSH.

Problem

Unable to log in to the Audit Vault Server console or through SSH as opc user.

The following error is displayed when attempting to connect through SSH as opc user:

remote side unexpectedly closed network connection

The following error is displayed when attempting to connect through the Audit Vault Server console:

internal server error 500

Solution

Oracle AVDF OCI Marketplace image has a password expiry setting. Check if the password for the opc user has expired. The following message is displayed when attempting to connect through SSH from another Linux VM to the Audit Vault Server instance:

ssh -i av_key opc@<IP address>
Audit Vault Server 20.x.0.0.0
 
DO NOT CHANGE ANY CONFIGURATIONS IN Audit Vault Server APPLIANCE WITHOUT GUIDANCE FROM ORACLE SUPPORT.
ANY CHANGES SHOULD BE TRACEABLE TO APPROPRIATE SR REFERENCE.
Your account has expired; please contact your system administrator.
Authentication failed.

Follow these steps to resolve the issue on the:

• Audit Vault Server
• Audit Vault Server Console 20.11
• Audit Vault Server Console 20.12

1. Boot in single user mode.

2. Remove the password aging for the opc user.

3. Log in as usual by connecting to the local Audit Vault Server through VNC.

4. Reboot the Audit Vault Server appliance after connecting successfully.

5. When the boot screen appears, press e to edit the command line.

6. Add the following at the end of line starting linux16:

   init=/bin/bash

7. Press Ctrl+x to boot the appliance.

8. Remount the filesystem by running the following command:

   mount -o rw,remount /

9. Set the opc user to never expire:

   change -m 0 -M -1 -I -1 -E -1 opc

10. Reboot the appliance.

11. Log in as usual through SSH.

1. Log in to the Audit Vault Server through SSH and switch to the root user.

   See Logging In to Oracle AVDF Appliances Through SSH.

2. Switch to the oracle user.

   su - oracle

3. Start SQL*Plus as sysdba.

   sqlplus / as sysdba

4. Run the following command:

   alter package APEX_230100.WWV_FLOW_DYNAMIC_EXEC compile body;

1. Log in to the Audit Vault Server through SSH and switch to the root user.

   See Logging In to Oracle AVDF Appliances Through SSH.

2. Switch to the oracle user.

   su - oracle

3. Start SQL*Plus as sysdba.

   sqlplus / as sysdba

4. Run the following command:

   alter package APEX_230200.WWV_FLOW_DYNAMIC_EXEC compile body;

L.77 Audit Vault Agent Fails with the ORA-01745 Error

Learn how to resolve the ORA-01745 error for Audit Vault Agent.

Problem

Audit Vault Agent fails with the ORA-01745 error.

Solution

Modify the firewall rules to ensure that communication between Audit Vault Agent and Audit Vault Server is allowed.

L.78 Oracle Directory or Table Audit Trail Stops with Error PLS-00201

Learn how to resolve error PLS-00201 in the collector logs.

Problem

The Oracle directory or table audit trail stops and the collector logs display the following error:

PLS-00201: identifier 'SYS.DBMS_AUDIT_MGMT' must be declared

Solution

Grant permissions to the target user and start the trail again. See Oracle Database Setup Scripts.

L.79 Error with Potential Insecure Path

Learn how to solve error java.lang.IllegalArgumentException:Potential insecure path found : <path>.

Problem: Audit Vault agent fails with error java.lang.IllegalArgumentException: Potential insecure path found : <path>.

Solution:

• Ensure directories in path do not have write permission for other users.
• Ensure path does not have more than 5 levels of symbolic links.

L.80 Error "ORA-28000 the Account Is Locked" After Changing the Admin User Password

Learn what to do when you receive the ORA-28000 error when changing the admin user password.

Problem

The following error message appears after you change the admin user password:

ORA-28000 the account is locked

Solution

You might receive this error when the Oracle Enterprise Manager Agent is monitoring Audit Vault Server. Changing the admin password on the Audit Vault Server Console does not automatically update the password that Enterprise Manager Agent uses to connect to Audit Vault Server. Ensure that the Enterprise Manager Agent is connecting with the correct password.

L.81 Error OAV-47112 When Trying to Delete an Existing Archive Location

Learn what to do when you receive the OAV-47112 error when trying to delete an existing archive location.

Problem

The OAV-47112 error appears when you try to delete an existing archive location.

Solution

You might receive this error when you try to delete an archive location that is currently in use to store archive tablespaces or data files. Wait until the tablespace or data file archive period expires before deleting the archive location. If needed, you can create a new archive location to use for the tablespace or data file archiving and then retrieve the tablespaces from the previous location and archive them to the new location.

L.82 Transaction Log Audit Trail Stops Due to XML Parsing Error

Learn how to fix issue when Transaction Log Audit Trail goes to stopped state due to XML parsing error.

Problem: Transaction Log Audit Trail stops due to XML parsing error. This is because of invalid XML record generated by Oracle GoldenGate.

Solution:Contact Oracle Support to create a Merge Label Request for applying the patch 32175609, 32063871, 33701099, and 34014874. This patch needs to be applied on Oracle GoldenGate installation.

L.83 "-bash: permission denied" Error When Trying to Run Custom Backup Script from /home/oracle

Scripts in /home/oracle cause permission errors when trying to run the scripts.

Problem: "-bash: permission denied" error when trying to run script from /home/oracle. The reason is executing script under /home/oracle is not allowed.

Solution: Move scripts to a different location.

L.84 Issues Deleting Target Database With Audit Trail Still Running

Secured target could not be deleted as the audit trail had not officially been stopped via the console or command line.

Problem: Unable to delete a target while it's trails are running. This is a safety function to prevent removal of active audit trails by accident.

Solution: Stop all the trails on that target through GUI, or AVCLI before dropping the target.

L.85 Deleting Audit Records Requires Applying Retention Period to Purge Records

Learn how to apply retention periods to audit data so that it can be purged.

Problem: No mechanism to delete audit data.

Solution: Apply small retention period to audit data to be deleted so that it is purged with time. See Configuring Archive Locations and Retention Policies for retention periods.

L.86 Unable to Mount NFS on New AVDF 20.3 Server

Learn how to fix inability to Mount NFS on s System running with Oracle AVDF.

Problem: AVDF Client is unable to mount due to bug or parameter settings on NAS Storage and/or NFS Server.

Symptoms:

• Oracle Linux running with Oracle AVDF (Audit Vault and Database Firewall).
• NFS Archive mount point cannot be mounted on a Oracle Linux system with AVDF.
• Further details are found in the following command:

  [root@nfs-client01 ~]# mount -t nfs nfs-server01:/avdf_archive_vol01/avdf_archive_backup
  mount.nfs: rpc.statd is not running but is required for remote locking.
  mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
  mount.nfs: an incorrect mount option was specified
  [root@nfs-client01 ~]# service nfslock status
  rpc.statd (pid 25042) is running...

Note:

nfs-client01 is the Oracle AVDF system. Oracle AVDF has no mechanism to collect the sosreport.

Solution:

1. Engage Vendor NAS Storage Support or NFS Admin Support Team to verify if the mount point at the NFS Server side is properly set-up. See the output of the command below.

   [root@nfs-client01 ~]# showmount -e nfs-server01
   /avdf_archive_vol01 nfs-client01 <== This is the limited NFS configuration of the NFS Server, which can be seen from AVDF NFS Client.
   [root@nfs-server01 ~]# cat/etc/exports
   /avdf_archive_vol01 nfs-client01(rw,no_subtree_check,no_root_squash)

2. Test whether the NFS mount point can be mounted properly on NFS client, please use the command below.

   [root@nfs-client01 ~]# mount -vvvv -t nfs -o nolock nfs-server01:/avdf_archive_vol01/avdf_archive_backup
   mount.nfs: timeout set for Mon Oct 29 14:05:37 2018
   mount.nfs: trying text-based options 'nolock,vers=4.1,addr=<IP ADDR>,clientaddr=<IP ADDR>'
   [root@nfs-client01 ~]# df -hP
   Filesystem Size Used Avail Use% Mounted on
   devtmpfs 412M 0 412M 0% /dev
   tmpfs 432M 0 432M 0% /dev/shm
   tmpfs 432M 6.0M 426M 2% /run
   tmpfs 432M 0 432M 0% /sys/fs/cgroup
   /dev/mapper/ol-root 9.8G 7.3G 2.5G 75% /
   /dev/sdb1 16G 11G 5.7G 65% /yum
   /dev/sda1 1014M 171M 844M 17% /boot
   tmpfs 87M 0 87M 0% /run/user/0
   nfs-server01:/avdf_archive_vol01 11G 2.2G 7.4G 23% /avdf_archive_backup <== This the sample AVDF NFS mount point. The one in use might be different.

   Note:

   vvvv – this is the debugging mode in NFS to test which layer of NFS is failing.

   -o nolock – to test if the NFS mount can mount using nolock.

   If the above command is able to mount the ADVF NFS mount point, hence, there is no issue on the NFS at the Linux OS level.

For more information refer to My Oracle Support Doc ID 2466520.1.

L.87 Alert Email Notifications Are Not Received from Oracle AVDF Server

Learn what checks need to be performed when the email alerts are not received from the AVDF server.

Problem: Alert email notifications are not received from Oracle AVDF Server.

Solution:

1. Login with super administrator user in AVCLI and check SMTP server settings.

   AVCLI> connect avadmin;
   Connected.
   AVCLI> LIST ATTRIBUTE OF SMTP SERVER;
   --------------------------------------------------------------------------------------------------------
   | HOST | PORT | SENDER_ID | SENDER_EMAIL | SECURE | TRUSTSTORE | STATE |
   ========================================================================================================
   | <mail server host> | 25 | user1 | user1@<mail server host> | UNSECURED | | ENABLED |
   --------------------------------------------------------------------------------------------------------
   1 row(s) selected.
   The command completed successfully.

2. From AVCLI interface send an email to test if the connection with the SMTP server works:

   AVCLI> TEST SMTP SERVER SEND EMAIL TO user2@<mail server host>;
   Request submitted successfully.

3. There are multiple reasons why the connection to the SMTP server might not work including:
   • The SMTP server is configured using DNS name and it cannot be resolved by AVDF server
   • AVDF server cannot communicate with the mail server
   • There are invalid objects in the database
   • Java processes are stuck
   • There are scheduled jobs by auditor user to retrieve audit settings or user entitlements
   View other problem causes and their solutions at My Oracle Support Doc ID 2232033.1

L.88 Audit Vault Agent is Stuck in Starting State: Error OAV-46573

Problem:

Audit Vault agent installed as a service on Windows is stuck in starting state after restarting the agent host.

Error OAV-46573: Agent is UNREACHABLE on host "hostname". Please try after some time. Audit trail is not eligible for auto start.

Solution:

Set JAVA_HOME. Audit Vault agent needs to find Java Runtime Environment.

L.89 SSH Becomes Disabled After Enabling FIPS Mode

If SSH becomes disabled after enabling FIPS mode, update the SSH keys to be compliant with FIPS.

Problem

In Oracle AVDF 20.9, SSH becomes disabled after enabling FIPS mode.

Solution

Before enabling FIPS 140-2, ensure that your SSH keys are compliant with FIPS. If your SSH keys are not compliant with FIPS, the SSH connection with the appliance might be lost after enabling FIPS.

For Oracle AVDF on Oracle Cloud Infrastructure (OCI), before enabling FIPS mode, ensure that the opc user has FIPS-compliant keys registered to /home/opc/.ssh/authorized_keys.

Follow these steps to resolve this issue:

1. Log into the Audit Vault Server console and disable FIPS mode.

2. Log back into the appliance through SSH and check or update the user keys for SSH-enabled users in ~/.ssh/authorized_keys to be compliant with FIPS.

   It can take several minutes for the console to become available after enabling or disabling FIPS mode.

3. Enable FIPS mode.

L.90 Audit Vault Agent Is Not Reachable from the Audit Vault Server Console

Problem

The Audit Vault Server console reports an agent as "not reachable." When trying to start the agent, a message similar to the following appears:

C:\AUDIT_VAULT_AGENT_3\bin>agentctl.bat start An instance of the agent is already running. 
[2015-08-26T10:51:25.345+03:00] [agent] [ERROR] [] [] [tid: 10] [ecid: 
172.xx.1.xxx:69595:1440575485345:0,0] OAV-10: Failed to release connection to DB[[
Failed to release connection to DB at 
oracle.av.platform.common.dao.ConnectionManagerImpl.destroy(ConnectionManagerI 
mpl.java:578) at oracle.av.platform.agent.AgentController.doStop(AgentController.java:1966)
at oracle.av.platform.agent.AgentController.doProcess(AgentController.java:2037)
at oracle.av.platform.agent.AgentController.main(AgentController.java:2046) 
Nested Exception: oracle.ucp.UniversalConnectionPoolException: The Universal
Connection Pool cannot be null at 
oracle.ucp.util.UCPErrorHandler.newUniversalConnectionPoolException(UCPErrorHandler.java:368) at 
oracle.ucp.util.UCPErrorHandler.newUniversalConnectionPoolException(UCPErrorHandler.java:336) at 
oracle.ucp.util.UCPErrorHandler.newUniversalConnectionPoolException(UCPErrorHandler.java:350) at 
oracle.ucp.admin.UniversalConnectionPoolManagerBase.destroyConnectionPool
UniversalConnectionPoolManagerBase.java:469) at 
oracle.av.platform.common.dao.ConnectionManagerImpl.destroy(ConnectionManagerImpl.java:574) 
at oracle.av.platform.agent.AgentController.doStop(AgentController.java:1966)
at oracle.av.platform.agent.AgentController.doProcess(AgentController.java:2037)
at oracle.av.platform.agent.AgentController.main(AgentController.java:2046)
 

Cause

The lock file is still present. This is a protection mechanism to prevent starting multiple agents from the same host.

Solution

1. Make sure that the java.exe processes for the agent are terminated. Use Task Manager to terminate them, if necessary.

2. Remove the <agent home>\av\conf\agent.lck file. For example:

   del C:\AUDIT_VAULT_AGENT_3\av\conf\agent.lck

3. Start the agent normally. For example:

   C:\AUDIT_VAULT_AGENT_3\bin> agentctl.bat start

L.91 Proxy Error When Opening AVDF Console in Web Browser

Problem

While opening the AVDF console in a web browser, the following error is shown:

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /console/f.

Solution

To fix the proxy error:

1. Check if the database and the Automatic Storage Management (ASM) instance is running or not. If not, then reboot the Audit Vault Server once and then check again.
2. If Java framework is not running, then start it by running the following command:/usr/local/dbfw/bin/javafwk start

L.92 Prevent a Terminal Login Session from Expiring When Connecting to an Audit Vault Server or a Database Firewall Server

Problem

When performing an Audit Vault Server or Database Firewall Server backup or upgrade, sometimes the connection to the server through the terminal timeouts.

Solution

To prevent a terminal login session from expiring when connecting to an Audit Vault Server or a Database Firewall Server.

1. Connect to the AV Server as root using a terminal session (like putty).
2. Run the following command: cd /etc/ssh
3. Run the following command: vi sshd_config
4. Run the following command: /ClientAliveCountMax
5. Set the value from 0 to 1000
6. Save the file by running the following command: :wq!
7. Run the following command at the OS prompt: service sshd restart
8. Run the following command: cd /usr/local/dbfw/templates
9. Run the following command: vi template-ssh-sshd-conf
10. Run the following command: /ClientAliveCountMax
11. Set the value from 0 to 1000
12. Save the file by running the following command: :wq!
13. Exit out of the terminal session.
14. Connect to the Audit Vault server or Database Firewall server again.

L.93 Microsoft SQL Server Database Audit Trails Are Unreachable

Problem

When you start an audit trail, it fails with the following message:

OAV-46573: Agent is UNREACHABLE on host "****.XXX.com". Please try after some time. Audit trail is now eligible for auto start. 

This may occur for EVENT LOG and DIRECTORY audit trails for Microsoft SQL Server on Microsoft Windows Server 2012.

Cause

The Audit Vault Agent was stopped. To verify this, use the agentctl status command. For example:

PS C:\Agent_Home\bin> ./agentctl status
Agent is stopped.

Solution

Start the Audit Vault Agent by using the agentctl start command. For example:

PS C:\Agent_Home\bin> ./agentctl start 
Agent started successfully.

The audit trails are configured for automatic startup. After you start the Audit Vault Agent, the audit trails should start automatically. Check the status to verify that the audit trails are started and collecting audit data.

Note:

You can also configure the Audit Vault Agent to restart automatically. See Configuring Agent Auto Restart Functionality.

L.94 Database Firewall Error ODF-10507: TCP Session Re-use

Problem

The Database Firewall reports an error similar to the following in /var/log/messages:

Dec 23 08:41:20 dp-svif-odb-n001 dbfw2.0: 
com.oracle.dbfw.dbfw_server WARN - ODF-10507: TCP session re-use: 
Session reuse observed for session 10.8.130.107:35699-10.2.129.152:8521 
Connection observed 61 seconds since last access 

Cause

A closed TCP session to the database has been reopened. This could lead to the state from the previous session being applied to the new session.

Solution

No action is required.

L.95 Automate Archivelog Deletion in the Audit Vault Server Repository By Using the oracle User

Problem

You can't automate archivelog deletion in the Audit Vault Server repository because crontab is not enabled for the oracle user.

Cause

Crontab is disabled by default for the oracle user in Oracle AVDF.

Workaround

Use the root user to log in as the oracle user and issue the required command. For example: su -l oracle -c bash.

To enable the oracle user's crontab, as the root user, update /etc/cron.allow and change the command to ensure that the oracle user password has not expired. This results in configuration errors for using crontab.)

L.96 OAV-46511: Missing Plug-in for Trail at Agent on Host

Problem

Adding an audit trail fails after unregistering and re-registering a host. The following error appears:

OAV-46511: missing plugin for trail at agent on host "<hostname>"

Solution

1. Stop the Audit Vault Agent.
2. Make sure that no processes are running from the Audit Vault Agent home.
3. Log into the Audit Vault Server console and stop any audit trails that are using this Audit Vault Agent. These should already have been be stopped when the agent was stopped, but check again.
4. In the Audit Vault Server console, click the Agents tab.
5. Select the host name that appears in the error.
6. Click Deactivate.

7. Select the same host name, and click Activate.

   A new key is created.

8. Click Downloads in the left navigation menu.
9. Download the agent.jar file to the target host.
10. Create a new home (or remove all files from the old Audit Vault Agent home).

11. Redeploy the Audit Vault Agent.

    java -jar agent.jar -d <AGENT_HOME>

12. Start the Audit Vault Agent.

    cd <AGENT_HOME>/bin ./agentctl start -k

L.97 Initiate Pairing for High Availability Fails with OAV-46599: Internal Error

Problem

When setting up high availability, the Initiate Pairing command fails with the following error:

OAV-46599: internal error Error: Failed to execute HTTPS request on the remote Audit Vault Server.

The messages log shows errors similar to the following:

Jun 4 16:07:10 avs00001702d420 setup_ha.rb[5272]: com.oracle.avs.high_availability ERROR - ODF-10001: Internal error: Error: Failed to execute HTTPS request on the remote Audit Vault Server.
Jun 4 16:18:04 avs00001702d420 setup_ha.rb[9959]: com.oracle.avs.high_availability ERROR - ODF-10001: Internal error: Error: Failed to execute HTTPS request on the remote Audit Vault Server.

Cause

The ports that are required for network connectivity between the primary and secondary Audit Vault servers in high availability mode are not open in the firewall.

Solution

Open port 7443 in the firewall.

L.98 Archive Error OAV-46599 and Internal Error ORA-14400: Partition Key Not Mapped

Problem

When archiving the data, the following errors appear:

OAV-46599: Internal error ORA-14400: partition key not mapped to any partition

Cause

The EVENTDATA disk group doesn't have enough space.

Solution

1. Check the current status of the existing Oracle Automatic Storage Management (Oracle ASM) disks and disk groups.

   set pagesize 1000
   set linesize 1000
   COLUMN NAME format A25
   COLUMN MOUNT_STATUS format A10
   COLUMN HEADER_STATUS format A20
   COLUMN MODE_STATUS format A20
   COLUMN STATE format A20
   COLUMN PATH format A40
   COLUMN LABEL format A20
   SELECT GROUP_NUMBER,NAME,TOTAL_MB,FREE_MB FROM V$ASM_DISKGROUP;
   SELECT MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,TOTAL_MB,FREE_MB,NAME,PATH,LABEL FROM V$ASM_DISK;

2. Add space to the EVENTDATA disk group.

   ALTER DISKGROUP EVENTDATA add disk 'path';

L.99 SYSLOG Forwarding for Alerts Isn't Working

Problem

SYSLOG forwarding for alerts isn't working.

Cause

This may happen if the SYSLOG forwarding queue has many alerts without the old events backlog.

Solution

1. Bounce the database.

2. Purge the av_alert queue table as the AVSYS user.

   DECLARE
   po_t dbms_aqadm.aq$_purge_options_t;
   BEGINdbms_aqadm.purge_queue_table('AVSYS.AV_ALERT_QT', NULL, po_t);
   end;
   /

L.100 SYSLOG Forwarding to SIEM Isn't Working

Problem

SYSLOG forwarding to SIEM isn't working.

Cause

rsyslog.conf file has incorrect configuration entries.

The rsyslog.conf says to forward alerts, while also stating to filter out alerts. Because of this, the alert will never get written to local /var/log/messages and can't be forwarded to SIEM.

#FORWARDED SYSLOG CATEGORIES:system, alerts, debug, info
*.info;local0.none;local1.none;local2.none;user.crit;user.warn;*.=debug;local1.info @<SIEM
      server IP address>

# This filters out AVDF alerts, which are either user.crit, or user.warn
user.crit;user.warn ~

Solution

1. Modify /etc/rsyslog.conf to not filter out alerts. Change user.crit;user.warn ~ to:

   user.crit;user.warn /var/log/AVDF_alerts

   Note:

   This change to /etc/rsyslog.conf will revert after a server reboot.
2. As root services rsyslog restart.
3. If there are no alerts being forwarded but the following "logger" command works, then either no alerts are generated or SYSLOG forwarding is not setup fully in WebConsole.

   #Force it something log to rsyslog to process to send to remote @IP (and write to the AVDF_alerts file)
   logger -p user.crit AVDF Alert dummy test
   #You should see this dummy test being logged in the /var/log/dbalerts file

   For more information see, Configuring Audit Vault Server Syslog Destinations.

To monitor what is being sent by rsyslog off AVDF server via rsyslog, run one of the following commands on port 514:

tcpdump -nn -i <eth#> | grep <IP of SIEM>
tcpdump -i eth0 tcp port 514 

tcpdump -i lo -A udp and port 514

tcpdump -A dst
tcpdump -nnvvXS dst

L.101 Oracle AVDF Reports For Oracle Database Shows UNKNOWN For Session Info If Native Network Encryption Is Enabled On the Database

Problem

If Native Network Encryption is enabled on an Oracle Database the Oracle AVDF reports show UNKNOWN for session info. Examples include unknown_username and unknown_client.

Solution

See Monitor Native Network Encrypted Traffic Through Database Firewall for Oracle Databases for how to resolve this issue.

L.102 Error: Kernel Out of Memory

Problem

The following error appears:

kernel: Out of memory: Kill process nnnnn (oracle) score nnn or sacrifice child

The following output is a partial example:

<kern.warning> xxxxav01aud kernel: java invoked
      oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0<kern.info> xxxxav01aud
      kernel: java cpuset=/ mems_allowed=0-1<kern.warning> xxxxav01aud kernel: Pid: 19085, comm:
      java Not tainted 2.6.39-400.250.6.el5uek #1<kern.warning> xxxxav01aud kernel: Call
      Trace:<kern.warning> xxxxav01aud kernel: [<ffffffff81113e04>]
      dump_header+0x94/0xe0<kern.warning> xxxxav01aud kernel: [<ffffffff81113f4d>]
      

After this error occurs once, similar errors are logged intermittently and the audit trail may be stopped or a repository database may be terminated suddenly.

Cause

When this error occurs, the memory usage of oraagent.bin becomes very high. When this type of memory usage occurs, the Linux: Out-of-Memory (OOM) Killer may stop some processes. See Doc ID 452000.1 in My Oracle Support for more information about this process.

The root cause of the oraagent.bin high memory usage is related to an Oracle Database issue where the dependent listener is removed or renamed. See Doc ID 1640721.1 for more information about this issue.

Solution

Stop the oraagent.bin process periodically.

L.103 Increasing the Logical Volume Capacity for a File System

If an Oracle AVDF file system runs out of space, you can allocate more space to the logical volume that holds the file system.

Use the lvextend command to increase the logical volume capacity. The vg_root volume group normally has unallocated space for this purpose.

1. Log in to the appliance through SSH and switch to the root user.

   See Logging In to Oracle AVDF Appliances Through SSH.

2. Run vgs to check the volume group free space. For example:

   /usr/sbin/vgs  
   VG      #PV #LV #SN   Attr   VSize  VFree  
   vg_root   1  13   0 wz--n- 149.84G 10.72G

   For more detailed volume group information, run vgdisplay.

3. Increase the logical volume capacity.

   For example, the following command adds 2 GB to the /tmp folder from the VG_ROOT volume group:

   /usr/sbin/lvextend -r -L+2G /dev/mapper/vg_root-lv_tmp

L.104 Banner Is Incorrect When Logging In as the Support User

Problem

In Oracle AVDF 20.1, when you log in through SSH as the support user, the banner is incorrect. For example:

login as: support
\S
Kernel \r on an \ms
upport@'s Password:

support@ -] $

Solution

Note:

This issue was fixed in Oracle AVDF 20.2.

To resolve the issue, request a backport or apply the latest bundle patch. See bug 31715004 - BANNER WHILE LOGIN AS SUPPORT USER IS NOT CORRECT.

L.105 Can't Install Host Monitor with Error: Failed to Generate Executables for Host Monitor

Problem

When installing the Host Monitor Agent, you receive one of the following errors:

[root@hm]# ./hostmonsetup install
/usr/bin/ld: cannot find -lssl
collect2: ld returned 1 exit status
make: *** [hostmonitor] Error 1
Line 751: Failed to generate executables for Host monitor.

[root@hm]# ./hostmonsetup install
/usr/bin/ld: cannot find -lpcap
collect2: ld returned 1 exit status
make: *** [hostmonitor] Error 1
Line 751: Failed to generate executables for Host monitor.

[root@hm]# ./hostmonsetup install
/usr/bin/ld: cannot find -lcap
collect2: ld returned 1 exit status
make: *** [hostmonitor] Error 1
Line 751: Failed to generate executables for Host monitor.

The libcap, libpcap, and openssl packages are already installed. For example:

rpm -qa|grep cap

The output lists the following:

libcap-2.16-5.5.el6.x86_64
compat-libcap1-1.10-1.x86_64
libcap-ng-0.6.4-3.el6_0.1.x86_64
libpcap-1.0.0-6.20091201git117cb5.el6.x86_64
perl-Pod-Escapes-1.04-119.el6_1.1.x86_64

Cause

The libcap, libpcap, and openssl package should be installed.

Also, the -devel packages for libcap, libpcap, and openssl packages must be installed.

Solution

Run the following commands to verify whether the packages are installed:

rpm -q libcap

rpm -q libcap-devel

rpm -q libpcap

rpm -q libpcap-devel

rpm -q openssl

rpm -q openssl-devel

The output of each command should display the location where the library is installed. If any package is not installed, you should see a prompt stating that the package is not installed.

If a package isn't installed, then install it by using the following command:

yum -y install <package_name>

To install the -devel packages, use the following commands:

yum -y install libpcap libpcap-devel

yum -y install libcap libcap-devel

yum -y install openssl openssl-devel

L.106 OAV-47704 Error When Dropping a Firewall

Problem

In Oracle AVDF 20.5, when you try to drop (remove) a firewall, you receive ERROR OAV-47704. For example:

ERROR: OAV-47704: Database Firewall avdf001 is not on the latest version. Upgrade to the latest.

Cause

Oracle AVDF is not allowing you to configure or remove a older-versioned Database Firewall.

Solution

Note:

This issue was fixed in Oracle AVDF 20.6.

In Oracle AVDF 20.5, try the following workaround:

1. Unlock the avsys user.

   See Unlocking the AVSYS User.

   Note:

   Remember to relock the avsys account when you've completed this task.

2. Start SQL*Plus as the avsys user.

   sqlplus avsys

3. Enter the password at the prompt.

4. Update the firewall version to 20.5.0.0.0.

   1. Get the firewall ID.

      select id from avsys.firewall where name= '<firewall_name>' and deleted_at is null;

   2. Update the version for the firewall ID.

      update avsys.firewall set software_version='20.5.0.0.0' where id=<firewall_id>;

   3. Commit the change.

      commit;

   4. Repeat steps a-c for any other firewalls that you want to remove.

5. Exit SQL*Plus.

   exit

6. Try to remove the firewall by using the Audit Vault Server console or AVCLI.

7. If the preceding steps do not resolve the error, try the following additional steps:

   1. Start SQL*Plus as the avsys user.

      sqlplus avsys

   2. Enter the password at the prompt.

   3. Get the firewall ID again, if needed.

      select id from avsys.firewall where name= '<firewall_name>' and deleted_at is null;

   4. Run the following command:

      update avsys.enforcement_point set deleted_at = systimestamp where firewall_group_id = (select firewall_group_id from avsys.firewall where name='<firewall_name>' and deleted_at is null) and deleted_at is null;

   5. Run the following command:

      update avsys.enforcement_point_instance set deleted_at = systimestamp where firewall_id = (select id from avsys.firewall where name='<firewall_name>' and deleted_at is null) and deleted_at is null;

   6. Commit the changes.

      commit;

   7. Exit SQL*Plus.

      exit

8. Try to remove the firewall by using the Audit Vault Server console or AVCLI.

L.107 Installing the Oracle Enterprise Manager Management Agent for Oracle AVDF Fails with an Unzip Not Found Error

Problem

In Oracle AVDF release 20, when you install the Oracle Enterprise Manager Management Agent on the Audit Vault Server or Database Firewall server, the installation may fail with an error saying "unzip not found."

Cause

The unzip RPM is not present on the Audit Vault Server or Database Firewall server.

Solution

1. Access https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/index.html from a machine that has internet access.
2. Download unzip-6.0-21.el7.x86_64.rpm.
3. Use SCP to transfer the RPM file to the Audit Vault Server or Database Firewall server.

4. Enter the following command to install unzip:

   rpm -i unzip-6.0-21.el7.x86_64.rpm

5. Instal the Enterprise Manager Management Agent again. See Installing the Enterprise Manager Management Agent.

L.108 Audit Trail Error: Unable to Connect to Target to Get Timezone Offset

Problem

In Oracle AVD 20.5 and later, audit collection stops with the following error:

OAV-8015: Error initializing AuditEventCollector instanceCollectionController : run : AuditException from process()

In the Audit Vault Server console, when you start the audit trail, the state changes to "Stopped" with the following error:

Unable to connect to target to get Timezone Offset

Cause

The AV.COLLECTOR.TIMEZONEOFFSET audit collection attribute for the target is missing.

Solution

1. Run the following SQL query on the target database:

   select systimestamp from dual;

   The output should look like the following example:

   SYSTIMESTAMP
   ---------------------------------------------------------------------------
   AV.COLLECTOR.TIMEZONEOFFSET +7:00

2. Log in to the Audit Vault Server console and an administrator.

3. Modify the target and add the AV.COLLECTOR.TIMEZONEOFFSET audit collection attribute that you identified in the preceding step.

   For instructions, see Modifying Targets.

4. Stop and start the audit trail.

   For instructions, see Stopping, Starting, and Autostart of Audit Trails in Oracle Audit Vault Server.

L.109 Issue with Phusion Passenger Configuration

Problem

In Oracle AVDF 20.1-20.4, you may see communication attempts from Oracle AVDF to an external URL related to Phusion Passenger.

Cause

Oracle AVDF uses third-party open source software called Phusion Passenger. This software may anonymously send usage statistics to an external URL if anonymous telemetry reporting is enabled. For more information about this, see Anonymous Telemetry Reporting on the Phusion Passenger website.

Solution

To disable Passenger anonymous telemetry reporting in Oracle AVDF 20.1-20.4:

1. Log in to the Audit Vault Server through SSH and switch to the root user.

   See Logging In to Oracle AVDF Appliances Through SSH.

2. Edit the template-httpd-httpd.conf platform template.

   vi /usr/local/dbfw/templates/template-httpd-httpd.conf

3. Locate the following mod_passenger configuration text block:

   <IfModule mod_passenger.c>
   ...
   </IfModule>

4. At the end of that text block, add PassengerDisableAnonymousTelemetry on.

   <IfModule mod_passenger.c>
   …
   PassengerDisableAnonymousTelemetry on
   </IfModule>

5. Save and close the file.

6. To apply the updated configuration and restart Apache, run the following command:

   /usr/local/dbfw/bin/priv/configure-networking

L.110 Diagnostic Report: Checking for Unknown Keys in /usr/local/dbfw/etc/dbfw.conf

Problem

The diagnostic report has the following message:

Checking for unknown keys in /usr/local/dbfw/etc/dbfw.conf: ["duplex", "speed"] - WARN

Cause

This warning may appear if the following entries are not configured in /usr/local/dbfw/etc/dbfw.conf:

speed=""
duplex=""

Solution

You can safely ignore this warning.

L.111 ODF-10001: Internal Error: Failure in Read from <IP Address>:<Port>: Connection Timed Out in Firewall Server

Problem

In Oracle AVDF 20.1-20.5, the following error may appear multiple times in /var/log/messages on the Database Firewall server:

<hostname> fw7: com.oracle.dbfw.fw ERROR - ODF-10001: Internal error: Failure in  Read from <IP ADDRESS>:<PORT>: Connection timed out

Cause

This message may appear if a if TCP connection has been closed due to the TCP keep-alive mechanism detecting a terminated peer.

Solution

In Oracle AVDF 20.1-20.5, treat this message as a warning, rather than an error. It will not cause a loss of functionality.

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

L.112 Database Firewall Server /var/log Partition Is Full

Problem

The Database Firewall server /var/log partition is full. This issue may occur in Oracle AVDF 20.3 and earlier.

Solution

The issue does not happen in Oracle AVDF 20.4 and later.

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

As a workaround until you can patch Oracle AVDF, you can also restart rsyslog. For example:

systemctl restart rsyslog

L.113 The tuned.service Status Is Failed in the Database Firewall Health Check

Problem

In Oracle AVDF 20.3 and earlier, the Oracle Linux tuned-service process may appear with a Failed status in the Database Firewall health check job details.

# systemctl status
tuned.service - Dynamic System Tuning Daemon
Loaded: loaded (/usr/lib/systemd/system/tuned.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2021-02-05 06:21:12 UTC; 2min 27s ago
  Docs: man:tuned(8)
      man:tuned.conf(5)
      man:tuned-adm(8)
Process: 16912 ExecStart=/usr/sbin/tuned -l -P (code=exited, status=1/FAILURE)
Main PID: 16912 (code=exited, status=1/FAILURE)
Feb 05 06:21:12 dbfw0000abc00000 tuned[16912]: from tuned import storage, units, monitors, plugins, profiles, exports, hardware
Feb 05 06:21:12 dbfw0000abc00000 tuned[16912]: File "/usr/lib/python2.7/site-packages/tuned/exports/__init__.py", line 3, in <module>
Feb 05 06:21:12 dbfw0000abc00000 tuned[16912]: from . import dbus_exporter as dbus
Feb 05 06:21:12 dbfw0000abc00000 tuned[16912]: File "/usr/lib/python2.7/site-packages/tuned/exports/dbus_exporter.py", line 3, in <module>
Feb 05 06:21:12 dbfw0000abc00000 tuned[16912]: import dbus.service
Feb 05 06:21:12 dbfw0000abc00000 tuned[16912]: ImportError: No module named dbus.service
Feb 05 06:21:12 dbfw0000abc00000 systemd[1]: tuned.service: main process exited, code=exited, status=1/FAILURE
Feb 05 06:21:12 dbfw0000abc00000 systemd[1]: Failed to start Dynamic System Tuning Daemon.
Feb 05 06:21:12 dbfw0000abc00000 systemd[1]: Unit tuned.service entered failed state.
Feb 05 06:21:12 dbfw0000abc00000 systemd[1]: tuned.service failed.

You can use the following commands to get more details about this error:

• systemctl status tuned.service
• journalctl -xe

Cause

The following RPM is missing on the Database Firewall server:

dbus-python-1.1.1-9.el7.x86_64.rpm

Solution

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

As a workaround until you can patch Oracle AVDF, you can use the following steps:

1. As the root user, install dbus-python-1.1.1-9.el7.x86_64.rpm.

   Get the RPM from the following public yum (other locations may not be supported):

   http://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/dbus-python-1.1.1-9.el7.x86_64.rpm

   Use the following command:

   yum install dbus-python-1.1.1-9.el7.x86_64.rpm
   
   Plugin "ulninfo" can't be imported
   Installed:  dbus-python.x86_64  0:1.1.1-9.el7
   Complete!

2. 2) Restart tuned.service and then check the status.

   systemctl restart tuned

   systemctl status tuned
   
   tuned.service - Dynamic
   System Tuning Daemon
      Loaded: loaded (/usr/lib/systemd/system/tuned.service; enabled; vendor preset: enabled)
      Active: active (running) since Tue 2021-03-09 08:22:10 UTC; 12s ago     
      Docs: man:tuned(8)
            man:tuned.conf(5)
            man:tuned-adm(8)
   Main PID: 779 (tuned)
      CGroup: /system.slice/tuned.service
              └─779 /usr/bin/python2 -Es /usr/sbin/tuned -l -P
   
   Mar 09 08:22:09 dbfw0000abc00000 systemd[1]: Starting Dynamic System Tuning Daemon...
   Mar 09 08:22:10 dbfw0000abc00000 systemd[1]: Started Dynamic System Tuning Daemon...

3. Check the Database Firewall in the Audit Vault Server console and verify that tuned.service is running (green).

L.114 Agent IO Error: Network Adapter Can't Establish Connection

Problem

In Oracle AVDF 20.3 and later, the following error may occur after install when trying to start with this command: ./agentctl start -k:

Internal Error. See log files for detail.

Within the av.agent log, the following error may appear:

[2021-07-29T16:25:36.956+07:00][agent] [ERROR] [] [] [tid: 1] [ecid: 1918831609:74227:1627550704887:0,0] Unable to connect to AV Server after 10 retries 
[2021-07-29T16:25:36.959+07:00] [agent] [ERROR] [] [] [tid: 1] [ecid: 1918831609:74227:1627550704887:0,0] Error occurred in Agent.[[Failed to connect to DB at oracle.av.platform.common.dao.ConnectionManagerImpl.getConnection(ConnectionManagerImpl.java:548) at oracle.av.platform.agent.AgentController.doValidateKey(AgentController.java:3040) at oracle.av.platform.agent.AgentController.doProcess(AgentController.java:3595) at oracle.av.platform.agent.AgentController.main(AgentController.java:3614)]]

Similarly, within the av.common log, the following error may appear:

[2021-07-29T16:25:04.879+07:00][common] [ERROR] [] [] [tid: 1] [ecid: 1918831609:74227:1627550704887:0,0] [Thread]:main. Unable to get connection to the datasource through certificate and without credentials. Unable to start the Universal Connection Pool: oracle.ucp.UniversalConnectionPoolException: Cannot get Connection from Datasource: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

Cause

This error may occur if the external firewall is blocking network traffic from the secure target host to the audit vault server on port 1522.

Solution

To start correctly, you need to open the ports between Oracle AVDF and a secured target on 1521 and 1522. If there is a firewall in between the firewall ports, you must open it. After you open the ports, the error should no longer persist.

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

L.115 Error ORA-01403 No Data Found When Adding a Database Firewall Instance to a Target

Problem

In Oracle AVDF 20.3 and later, when configuring Database Firewall monitoring points for a target, you can add the first Database Firewall instance as a monitoring point, but when you try to add the second instance, you may get the following error:

ora-01403 no data found

If you remove the first Database Firewall instance and try to add the second instance as a new monitoring point, you may get the following error:

OAV-46593: secured target address does not exist. cannot drop secured target address.

Cause

This issue may happen when the ha_role for one of the Database Firewalls is set to 2 in the database. The ha_role needs to be set to 1.

This could happen if the Database Firewall instances were previously configured as a resilient pair.

Solution

1. Connect to the Audit Vault Server database.

2. Run the following SQL query:

   select id, name, is_active, ha_role from avsys.firewall where deleted_at is null;

3. Find the row where ha_role is set to 2 and make a note of the Database Firewall ID.

4. Run the following query by replacing the firewall_id with the ID that you identified in the preceding step.

   update avsys.firewall set ha_role = 1, is_active = 1, ha_role_changed_at = current_timestamp where id in (<firewall_id>);

   For example:

   update avsys.firewall set ha_role = 1, is_active = 1, ha_role_changed_at = current_timestamp where id in (2);

5. Run the following command:

   commit;

6. Add the second Database Firewall instance for the target by using the Audit Vault Server console.

L.116 The Order of IP Addresses Changes After Setting Up DNS Servers

Problem

After setting up DNS servers, the order of the IP addresses may change.

For example, you might set up the DNS servers in the following order:

1. DNS server 1: xx.xxx.xx.14
2. DNS server 2: xx.xxx.xx.15
3. DNS server 3: xx.xxx.xx.16

After the configuration, the IP addresses might change to the following order:

1. DNS server 1: xx.xxx.xx.14
2. DNS server 2: xx.xxx.xx.16
3. DNS server 3: xx.xxx.xx.15

Cause

The order depends on the behavior of the package that's operating internally when registering the DNS servers.

Solution

No action is required. The IP addresses are not always registered in the order in which they are set.

L.117 NTP Is Unreachable on the Audit Vault Server

Problem

When configuring Network Time Protocol (NTP) on the Audit Vault Server, NTP is unreachable. It may be working fine on a Database Firewall server in the same network.

Cause

This may occur if the browser that you're using to access the Audit Vault Server console is set to a language other than English.

Solution

To resolve this issue, change the browser to English and refresh the Audit Vault Server console.

L.118 Database Firewall Status Is Running but the Status Is Down on the Audit Vault Server Console

Problem

The Audit Vault Server console shows that the Database Firewall is Down even though it's up and running.

Cause

This may be caused by an application timeout that's related to a bug that was fixed in Oracle AVDF 20.8.

Solution

To resolve this issue, complete the following steps on the Database Firewall sever:

1. Rename /usr/local/dbfw/.bash_profile. For example:

   mv /usr/local/dbfw/.bash_profile /usr/local/dbfw/.bash_profile_old

2. Restart Apache.

   systemctl restart httpd

   This should make it possible for Apache to spawn the web server APIs and for the system to start working again.

3. Check the status of the Database Firewall in the Audit Vault Server console. The status should be Up.

L.119 Network Audit Trail Is Not Collecting Audit Data When Using the Host Monitor Agent

Problem

When using the Host Monitor Agent to capture network traffic, the network audit trail isn't collecting audit data, even though the audit trail is running.

Note:

The following instructions apply only when using the Host Monitor Agent to capture network traffic. Ensure that there are no connection issues between the Host Monitor Agent and the Database Firewall before proceeding.

Solution

1. Log in to the target machine where the Host Monitor Agent is installed.

2. Restart the Audit Vault Agent in debug mode by running the following command:

   <AVDF AGENT HOME>/bin>./agentctl stop
   <AVDF AGENT HOME>/bin>./agentctl start -l debug

3. Log in to the Audit Vault Server console as an administrator.
4. Start the network audit trail and see if the network audit trail status changes to Collecting.
5. Run the following command on the machine where the Host Monitor Agent is installed and ensure that the hostmonitor process is running.

   ps -ef | grep hostmonitor

6. Navigate to the folder that contains the hostmonitor logs (for example, AGENT_HOME/hm/log), and run the following command:

   grep "Successfully sent data to Firewall machine" *

   Note:

   If "Successfully sent data to Firewall machine" appears in a log file, then the Host Monitor Agent is collecting and sending the network traffic to the Database Firewall successfully.

7. If the preceding text doesn't appear in the entry is not present in the hostmonitor log file, run the following command on the secured target machine to see which IP addresses and ports the target database is listening to.

   lsnrctl status

8. Log in to to the Audit Vault Server console as an administrator and complete the following steps using the IP addresses and ports that you identified in the preceding step.
   1. Click the Targets tab.
   2. Select the target for which the network audit trail configured.
   3. Verify that all the IP addresses and ports that the target database is listening to appear in the Connection Details column in the Database Firewall Monitoring section.
   4. If an IP address or port doesn't appear, click the link under Connection Details.
   5. In the Database Firewall Monitor dialog box, click Add to add any missing IP addresses and ports.
   6. Click Save twice.

9. Run the following command on the target and verify the network interface card (NIC) to which all target database listening IP addresses belong.

   ifconfig -a

10. Navigate to the folder that contains the hostmonitor logs (for example, AGENT_HOME/hm/log), and run the following command:

    grep "network_device_name_for_hostmonitor" *

    The output should be similar to the following example:

    The selected network device for capturing is: eth0. To change the device update the network_device_name_for_hostmonitor attribute at Collection Attributes to any one value from the list: eth0, nflog, nfqueue, any, lo and restart the trail.

    In the preceding example, the log shows that hostmonitor is listening on the eth0 NIC.

    Verify that hostmonitor is listening on the same NIC to which the target database listening IP addresses belong.

11. If the target database listening IP addresses belong to a different NIC, perform the following steps:
    1. Log in to the Audit Vault Server console as an administrator.
    2. Click the Targets tab.
    3. Select the target for which the network audit trail is configured.
    4. Click Modify.
    5. Click the Audit Collection Attributes tab.

    6. Click Add and add the following attribute name and value pair:

       Name: network_device_name_for_hostmonitor

       Value: Enter the name of the NIC to which the target database listening IP addresses belong.

       Click Save twice.

12. Restart the network audit trail from the Audit Vault Server console.

13. Navigate to the folder that contains the hostmonitor logs (for example, AGENT_HOME/hm/log), and run the following command:

    grep "Successfully sent data to Firewall machine" *

    Note:

    If "Successfully sent data to Firewall machine" appears in a log file, then the Host Monitor Agent is collecting and sending the network traffic to the Database Firewall successfully.
14. Log in to the target machine where the Host Monitor Agent is installed.

15. Restart the Audit Vault Agent in normal mode by running the following command:

    <AVDF AGENT HOME>/bin>./agentctl stop
    <AVDF AGENT HOME>/bin>./agentctl start

L.120 Internal Error When Deploying the Audit Vault Agent

Problem

Deploying the Audit Vault Agent fails with the following error:

Internal Error
Error occurred during install/upgrade. Check log files for more information.

The log file may contain entries similar to the following:

Unable to get connection to the datasource through certificate and without credentials. Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException: Cannot get Connection from Datasource: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

Cause

The Audit Vault Agent was trying to connect to an incorrect IP address.

Solution

Cross-check the IP address of the Audit Vault Server and the secured target server on which you're installing the Audit Vault Agent.

L.121 Agent Host Is Not Registered

Problem

Deploying the Audit Vault Agent fails with the following message, even though the agent was already registered:

Agent host is not registered.
Agent host must be registered before an agent can be installed or upgraded. Agent deployment failed.

Cause

This might happen on a multi-homed system when there are multiple routes from the Audit Vault Agent host to the Audit Vault Server. The SQLNet traffic might use an IP address that's different from the one that was used to register the Audit Vault Agent host.

When registering a host in the Audit Vault Server, you have two choices:

• Provide both a host name and an IP address: In this case, the name is treated as a handle with no significance and only the IP address is used.
• Provide only the host name: In this case, when you don't provide an IP address, the Audit Vault Server tries to resolve the host name to an IP address using DNS, if configured. If DNS is not configured, you receive an error. If the name resolves correctly, the IP address is remembered and used. The host name is ignored for normal operations.

This means that you must register the host with the same IP address that you see when using SQL*Plus to connect from the Audit Vault Agent host to the Audit Vault Server.

Solution

To verify the IP address with which the host should be registered, use SQL*Plus and connect using the connect string that's defined in the <agent_home>/av/conf/bootstrap.prop file. For convenience, you can also add it to the tnsnames.ora file with the designation AV.

Use the following steps to determine which IP address to register:

1. Determine which network interface card (NIC) is used for the Audit Vault Agent communication.

   1. Connect to the AV Server Database from the agent host using the following command.

      sqlplus <username>/<password>@"`cat <agent_home>/av/conf/bootstrap.prop | grep "SYS.CONNECT_STRING" | sed -e 's/SYS.CONNECT_STRING=//g' | sed -e 's/\\\//g'`"

      For <username>, enter a valid user name in the database, such as avauditor.

      For <agent_home>, enter the path to the agent directory.

   2. Run the following query:

      select SYS_CONTEXT('USERENV','IP_ADDRESS') FROM dual;

2. Use the IP address that was returned by the preceding query to register the Audit Vault Agent in the Audit Vault Server console. See Registering Hosts on the Audit Vault Server.

   Alternative, you can use the following AVCLI command:

   register host <hostname> with ip <ip address from the query>

On a multi-homed system, to register with another IP address, contact your network administrator and change the TCP routing configuration.

L.122 A Database Firewall Policy Is Not Blocking Statements Correctly

Problem

After creating a Database Firewall policy to block or substitute all queries from a specific database user, that user may still be able to run the SQL statements freely.

Cause

This could happen if the protected address that's associated with the secured target doesn't have an Oracle service name.

Solution

Make sure that all protected addresses contain an Oracle service name.

L.123 Having Automatic Archiving Enabled Is Giving OAV-47116 Error

Problem

AUTOMATIC ARCHIVING ENABLE is giving error OAV-47116.

Cause

Auto archive order should be greater than 0 (i.e 1 or more) to enable automatic archiving.

Solution

Change auto archive order to be greater than 0 from UI then try to enable automatic archiving.

L.124 Network Trail Fails To Be Started Due To Insufficient Permissions Error

Problem

The following errors are spotted in agent host monitor logs when the network trail is started:

startHostMonitor : exception while starting HostMonitor[[
Failed to start collector {0}:{1}
at oracle.av.platform.agent.collfwk.impl.factory.HMCommandExecutor.execute(HMCommandExecutor.java:380)
at oracle.av.platform.agent.collfwk.impl.factory.HMCommandExecutor.execute(HMCommandExecutor.java:311)
at oracle.av.platform.agent.collfwk.impl.factory.HMCommandExecutor.startHostMonitor(HMCommandExecutor.java:111)
at oracle.av.platform.agent.collfwk.impl.factory.HMCommandManager.startHostMonitor(HMCommandManager.java:679)
at oracle.av.platform.agent.collfwk.impl.factory.HMCommandManager.startTrail(HMCommandManager.java:736)
at oracle.av.platform.agent.collfwk.impl.factory.CollectionFactory.createCollection(CollectionFactory.java:565)
at oracle.av.platform.agent.collfwk.impl.factory.CollectionFactory.createCollection(CollectionFactory.java:392)
at oracle.av.platform.agent.StartTrailCommandHandler.processMessage(StartTrailCommandHandler.java:63)
at oracle.av.platform.agent.AgentController.processMessage(AgentController.java:585)
at oracle.av.platform.agent.AgentController$MessageListenerThread.run(AgentController.java:3075)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot run program "/u02/app/oracle/product/avdf12/av_agent/hm/hostmonmanager" (in directory "/u02/app/oracle/product/avdf12/av_agent/hm"): error=13, 
Permission denied
Caused by: java.io.IOException: error=13, Permission denied

Cause

The AVDF agent and the hostmonitor have been deployed as root and oracle users are not allowed to run the executables due to the binaries' permissions and the hardcoded configuration.

Solution

1. Redeploy the AVDF agent with oracle user using documentation steps:

   Deactivate and Remove the Audit Vault Agent

2. Copy the zip file for host monitor deployment:

   cd <Agent Installation Directory>/stage/plugins
   cp agent-linux-x86-64-hmon-one.zip to /usr/local

3. Unzip the file as root user:

   unzip the agent-linux-x86-64-hmon-one.zip filecd hm

4. Install host monitor:

   ./hostmonsetup install agentuser=oracle agentgroup=oinstall

5. Start the agent

   cd <agent home>/bin
   ./agentctl start 

6. Start the network trail.

L.125 How To Start an Audit Trail for Audit Trail Type DIRECTORY if the Database is Down

This document explains how to start audit trail for OS audit files even when the database is down or only in MOUNT state.

1. For the collector to start you need to add 3 NLS attributes that the collector needs to parse the OS audit files.Collect the following information from the secured target database while it is running, if this is a standby database you can also collect this information from the primary database:

   select parameter, value from v$nls_parameters where parameter in ('NLS_LANGUAGE','NLS_TERRITORY','NLS_CHARACTERSET');

   For example:

   PARAMETER VALUE---------------- --------------------
   NLS_LANGUAGE AMERICAN
   NLS_TERRITORY AMERICA
   NLS_CHARACTERSET AL32UTF8

2. Add these NLS attributes of the secured target database to the collector:

   ORCLCOLL.NLS_TERRITORY
   ORCLCOLL.NLS_LANGUAGE
   ORCLCOLL.NLS_CHARSET

   Adding these attributes manually is necessary to be able to start the audit trail and collect audit records from OS files even if the database is in MOUNT state or down.

   See more information atOracle Database Audit Collection Attributes in the Plug-In Reference section.

L.126 After Setting the "SSH Acess" Setting, the SSH Connections are Dropped

Symptoms

After setting the "SSH Access" setting from the "Network Services" AVDF configuration page, the SSH connections are dropped.

Cause

When the SSH connection was being created a telnet connection protocol was used.

Solution

Use the SSH protocol to connect to the AVDF server.

L.127 AVDF Directory Audit Trail Stays Up Collecting Audit Data Even When Target Database Is Shutdown

Question

Why does AVDF directory Audit Trail stay up collecting audit data, even when target database is shutdown?

Answer

Directory audit trail collectors does not need target database to be up and running for collection. As long as the directory contains log files or audit files, directory trail collector collects. Hence non-availability of target database does not immediately translate to warning in AVDF UI. This is the inherent nature of directory trail collector.

If the target database is down, directory trail can continue running as long as it has access to directory or audit logs.

This behavior is unlike table audit trails, where non-availability of target database immediately translates to warning in AVDF UI.

L.128 ODF-10717 Is Logged In /var/log/messages File During The Starting Up of Database Firewall

Symptoms

ODF-10717 can be logged in /var/log/messages file during the starting up of database firewall.

Example)
Jan 18 00:45:51 <HOST> <EP>: com.oracle.dbfw.fw INFO - ODF-10102: Startup complete: Ready to process network traffic
Jan 18 00:45:51 <HOST> <EP>: com.oracle.dbfw.fw WARN - ODF-10717: Zero DAM packets processed: pcap_dispatch() processed zero packets out of 20 requested
Jan 18 00:45:51 <HOST> <EP>: com.oracle.dbfw.fw ERROR - ODF-10701: Network packets not intercepted: Maximum capacity of the system has been exceeded for Protected Databases '<SECURE TARGET1>', '<SECURE TARGET2>'

Cause

ODF-10717 can be logged when a empty network packet is detected on using DAM mode environment.

Also it is easly detected during the starting-up of database firewall or under heavy networking trafic.

This does not always mean that there are some kinds of crtical errors and it can be safely ignored in usual.

Solution

Safely ignore ODF-10717.

L.129 Error: Net::ReadTimeout occurred when executing Setup_ha.rb --disable_failover

Symptoms

Error: Net::ReadTimeout occurred when executing Setup_ha.rb --disable_failover.

$ /usr/local/dbfw/bin/setup_ha.rb --disable_ failover
Error: Net::ReadTimeout

Cause

After script was changing status to DISABLE, the other DBFW will be processed to reflect the settings, but at that point a timeout error has occurred.

Solution

If the result of /usr/local/dbfw/bin/setup_ha.rb --status is DISABLE, no other action is needed.

[root@avsxxxxx ~]#sudo -u oracle /usr/local/dbfw/bin/setup_ha.rb --status
HA mode: PRIMARYHA server 1: xx.xx.xx.xx
HA server 2: xx.xx.xx.xx
...
Automatic failover: DISABLED <<<<<

L.130 Audit Records Being Re-Read After Upgrade to 20.1

Problem

After upgrading to 20.1, audit records for SYSLOG that have been read prior to the upgrade are being re-read.

Solution

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

Note:

This issue is only found in Oracle AVDF 20.1 and is resolved in 20.2 (20 RU2) and subsequent releases.

If you are still encountering this problem, follow these steps to resolve the issue:

1. Stop the audit trail.

2. Unlock the avsys user.

   See Unlocking the AVSYS User.

   Note:

   Remember to relock the avsys account when you've completed this task.
3. Execute the provided SQL procedure:

   DECLARE
      v_count NUMBER;
   BEGIN
      FOR i IN (select audit_trail_id from audit_trail where location not like 'ÞLETED_%' and audit_trail_type like 'SYSLOG' and plugin_guid like 'com.oracle.av.plugin.oracle') LOOP
         select count(*) into v_count from avsys.checkpoint where audit_trail_id=i.audit_trail_id;
         if v_count = 0 then
            insert into avsys.checkpoint (audit_trail_id,checkpoint_time) (select i.audit_trail_id,max(event_time) from event_log where audit_trail_id=i.audit_trail_id);
         end if;
      END LOOP;
      COMMIT;
   END;
   /

4. Lock the avsys user.

   See Locking the AVSYS User.

5. Proceed with the upgrade to version 20.1 as intended.

L.131 Audit Records May Be Skipped After Upgrade to 20.1

Problem

Oracle DIRECTORY and SYSLOG audit trails may skip audit records during successive recoveries for XML and SYSLOG files. This behavior occurs after upgrading to 20.1.

Solution

This issue is only found in Oracle AVDF 20.1 and is resolved in 20.2 (20 RU2) or later.

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

L.132 Processes Still Run After Stopping Audit Trails

Problem

Before upgrading to 20.1, audit trails should stop; however, some of the processes continue to run even after stopping the audit trails. These processes may lead to problems.

Solution

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

To resolve this issue, begin by identifying and removing the avorclcoll processes that might be persisting on the host machine. For each host machine where an Oracle Directory trail is configured, execute the following command:

ps -ef | grep avorclcoll

If the avorclcoll process exists, then terminate the process by executing the following command:

kill -9 <pid>

Replace <pid> with the actual Process ID associated with the avorclcoll process.

L.133 Unable to Execute the Oracle User Setup Script

Problem

You may encounter an execution failure when attempting to run the oracle_user_setup.sql script. This may be attributed to the presence of an underscore (_) in the username, leading to the unsuccessful execution of the script.

Note:

This issue is only found in Oracle AVDF 20.1 and is resolved in 20.2 (20 RU2) and subsequent releases.

Solution

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

To fix this issue, you should create a user profile that does not contain an underscore in the name. Proceed with the execution of the oracle_user_setup.sql script using the newly created user account.

L.134 Loss of Bonding Between Network Interface Cards Upon Creation of Proxy Port

Problem

When a network interface card (NIC) bonding is established, the creation of a proxy port on one of the NICs through the Audit Vault Server console, performed as a super administrator, leads to the unintended removal of the established NIC bonding.

Solution

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

After setting the port in the UI and encountering the loss of bonding, follow these steps to reconfigure the bond using the command-line interface (CLI):

1. Configure Bonding via CLI: Use the CLI on the Database Firewall instance to configure the bonding between the relevant devices. See the CONFIG-BOND documentation for more details.
2. Configure Proxy Ports: Set up the necessary proxy ports for the bonded device as required for your configuration. See the CONFIG-PROXY documentation for more details.
3. Re-establish Bonding: Execute the bonding command, as outlined in Step 1, to re-establish the bond between the network interface cards.

L.135 Issue Between Returned Number of Rows and Database Response Monitoring Interaction

Problem

An issue occurs when database response monitoring is active and you have enabled the return number of rows for the Database Objects policy.

Note:

This issue has been eliminated in Oracle AVDF 20.4 and subsequent releases.

Symptoms

The following symptoms may occur when you experience this issue:

1. Successful extraction of returned number of rows for all SELECT queries.
2. Marking of returned number of rows as -1 on timeout for SELECT queries that match the policy.
3. A substantial influx of Database Firewall alerts is generated in Oracle AVDF 20.3. This occurs when both the Capture Database Response and Capture number of rows returned for SELECT queries field are enabled within the Database Firewall monitoring point.

Solution

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

To resolve this issue, implement the following workarounds:

1. If database response monitoring is not essential, consider turning off this feature to mitigate the encountered issue.
2. Adjust the timeout interval as guided in the solution described under Issue with Retrieval of Return Row Count.
3. Avoid enabling the Capture Database Response field while simultaneously activating the Capture number of rows returned for SELECT queries field within the Database Firewall monitoring point. This step helps alleviate the generation of excessive Database Firewall alerts.

L.136 Database Firewall Instance Status "Down" Post-Upgrade to 20.2

Problem

Upon upgrading from Oracle AVDF 20.1 to 20.2, an issue may occur where the status of the Database Firewall instance is incorrectly indicated as "Down" within the Audit Vault Server console. Additionally, the version of the Database Firewall instance is incorrectly displayed as 20.1, despite the upgrade.

Solution

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

To resolve this issue, implement the following step:

1. Reboot the Database Firewall host.

By performing a host reboot, you can rectify the inaccurately reported "Down" status and the version mismatch for the Database Firewall instance.

L.137 "Failed to Update" Error Encountered During Oracle AVDF 20.2 Upgrade

Problem

When upgrading to Oracle AVDF 20.2, a “Failed to Update Error” may be observed while running the pre-upgrade RPM.

The following error message is displayed:

Failed to apply update: Verifying pre-upgrade conditions failed.
Failed to apply update: /images/upgrade/lib/preconditions………

Note:

Losing power during an upgrade can result in the loss of data. Do not power off your machine while the upgrade is in progress for best results.

Solution

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

To address this issue, complete these steps:

1. Execute the following command, where <PID> represents the Process ID and is accessible within the directory path: /tmp/<directory name>/<PID>:

   kill -9 <PID>

2. Proceed by applying the pre-upgrade RPM once more.

L.138 Significant Time Delay in Captured Traffic by the Database Firewall For Reporting

There may be a significant time delay from the moment traffic is captured by the Database Firewall to the time it is available for report generation.

Symptoms

There may be a significant time difference between the time when the traffic is captured by the Database Firewall and the time when it is available at Audit Vault Server (AVS) for report generation. For example, if some SQL is captured, it may not be available in AVS for a few hours to generate the reports.

The corresponding time at which the data becomes available to AVS can be checked from the AVSYS.EVENT_LOG table on the AVS server. This can be done by the following SQL command:

SELECT MAX(EVENT_TIME) FROM AVSYS.EVENT_LOG;

Cause

The possible causes may be one of the following:

1. There could be a time zone mismatch on the UI and the visible time stamp is shifted.
2. There is a significant load on one of the Enforcement Points and the Audit Vault Server is not able to insert the data at the appropriate rate.
3. Some other problem.

Solution

The corresponding solutions to the above mentioned causes are as follows:

1. Connect directly to the Audit Vault Server database and run a query on the AVSYS.EVENT_LOG table. Compare the time stamp with the expected one.
2. To confirm there is a significant load on one of the Enforcement Points, check the number of files in the /usr/local/dbfw/va/*/log directories. If the number of kernel*.gz files is over 10, then this is a plausible cause.
3. Collect the diagnostic package for further investigation.

L.139 ODF-10719 Error Logged In Messages File After Starting Database Firewall

The error code ODF-10719 is logged in the messages file when starting Database Firewall, indicating difficulties in loading session information from a file.

Problem

When starting Database Firewall, a ODF-10719 error may be logged into the /var/log/messages file.

The following error message is an example of what is displayed:

<HOST> auditd[4106]: Audit daemon rotating log files 
<HOST> <EP>.24: com.oracle.dbfw.dbfw_server WARN - ODF-10719: Unable to load Session information from file: Could not acquire lock on file for instance=0 after 120 seconds.

Cause

Database Firewall collects the information of a session via a connect packet. Database Firewall manages the information of each session by using the connection information. A ODF-10719 error can occur when the Database Firewall cannot confirm the connect package information because it is unable to load the session information from its files. This may occur if sessions are already established before starting Database Firewall, which in turn means that Database Firewall cannot collect connect packets of the established session so it is missing this information in the logs.

Another possible cause is if the connection pooling feature is used on a target environment. This may result in an ODF-10719 error being logged in messages file after starting Database Firewall.

Solution

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

Please safely ignore the ODF-10719 error. Restart Database Firewall and then your session.

L.140 "Server Error 500" on Oracle AVDF Server after Setting Network Time Protocol (NTP)

After setting the Network Time Protocol (NTP) option on the Oracle AVDF server, the following error may occur: Server Error 500.

Problem

After setting the NTP using the setup page from the GUI, the following error message is observed:

Server Error 500

Cause

This error is caused by setting the NTP option on the Oracle AVDF server. This causes the server to stop working and show the above error message.

Solution

This issue can be solved by completing the following steps:

1. Disable the NTP setting.
2. Set the time manually.
3. Reboot the Oracle AVDF Server

After completing these steps, the database and all other services should start successfully. After the application starts successfully, the NTP service can be enabled again without issues.

L.141 Audit Vault Agent Logs Report IO Error: The Network Adapter Could Not Establish Connection Due To Inactive Database Listener

Audit Vault Agent logs report IO Error saying the network adapter could not establish connection due to the inactivity of the database listener which leads to the disruption in the audit trail.

Problem

The Audit Vault Agent logs return an IO Error that states The Network Adapter could not establish the connection.

Cause

When the database listener is inactive, the audit trail loses its ability to establish communication with the database through the agent, leading it to enter a stopped state. It is crucial to ensure that the database is available and that the connected listener is active. Additionally, you should verify that the database services are correctly registered with the listener. If the listener is down, or in instances where the listener is active, but without any associated database services registered, this situation triggers a shutdown of the audit trails with the above error message.

Solution

To resolve this issue, you must ensure that the database listener is active and the database service is up. Use the command ps -ef|grep tns to check the status of the listener. Below is the output of this command when the listener is down:

root 10 2 0 Aug30 ? 00:00:00 [netns]
oracle 1673 1245 0 07:47 pts/1 00:00:00 grep tns

Next, use the command ps -ef|grep pmon to see the status of the database service. Below is the output of this command when the database service is up:

oracle 1670 1245 0 07:47 pts/1 00:00:00 grep pmon 
oracle 3003 1 0 Aug30 ? 00:00:46 ora_pmon_orcl

See Creating and Configuring a Database Firewall Monitoring Point for more information.

Additionally, a TNSPING to the database service can verify the availability of the listener. If the listener is found to be inactive, start it using the LSNRCTL utility. Once the listener is running, the audit trail collector should initiate, and the status should display a green arrow pointing upwards.

Monitor the agent logs located at $AGENT_HOME/av/log. No further error logs should be reported upon successfully starting the collectors. Regularly checking these logs will help ensure the proper functioning of the collectors.

L.142 oracle_user_setup.sql Script Does Not Finish

Problem

When using the oracle_user_setup.sql script to grant AVAUDIT SETUP privilieges to a database user, the gets stuck and never finihses, but does not show any errors on screen.

Cause

In the script a grant select on SYS.GV_$INSTANCE to AVAUDIT is ran but it never finishes. You can see this from the logs generated on the target database by implementing the below traces:

alter session set max_dump_file_size = unlimited;
alter session set tracefile_identifier='&name_for_the_output';
alter session set events '10046 trace name context forever, level 12';
grant select on SYS.GV_$INSTANCE to AVAUDIT;
ALTER SESSION SET EVENTS '10046 trace name context off';

Solution

Determine what is preventing the grant on GV_$INSTANCE by running the following commands:

SQL>SELECT SID, OWNER, OBJECT, TYPE FROM V$ACCESS WHERE OBJECT = 'GV_$INSTANCE'
select * from V$LOCKED_OBJECT where OBJECT_ID in (select object_id from All_Objects where OBJECT_NAME = 'GV_$INSTANCE');
select distinct to_name object_locked from v$object_dependency where to_address in (select w.kgllkhdl address from dba_kgllock w, dba_kgllock h, v$session w1, v$session h1 where ((h.kgllkmod != 0) and (h.kgllkmod != 1) and ((h.kgllkreq = 0) or (h.kgllkreq = 1))) and (((w.kgllkmod = 0) or (w.kgllkmod= 1)) and ((w.kgllkreq != 0) and (w.kgllkreq != 1))) and w.kgllktype = h.kgllktype and w.kgllkhdl = h.kgllkhdl and w.kgllkuse = w1.saddr and h.kgllkuse = h1.saddr);

On a Real Application Cluster (RAC) database, run these commands on all nodes.

If no process can be identified, then a restart of the database might solve this issue. This way, any lock on the GV_$INSTANCE will be removed.

L.143 Authentication Processing Error When Logging in Due to Excessive Group String Length in Active Directory

An error in authentication processing is triggered by an excessive group string length in the Active Directory, leading to disruptions in user access.

Problem

Users receive an error message prompting them to contact their application administrator after there is an error processing authentication. This error disrupts the user's ability to successfully authenticate and access the AVDF system.

Cause

This issue arises when the group string associated with the AD/LDAP user in the Active Directory is too long. The group string length cannot exceed 8,000 characters; the system encounters an error processing authentication when the length surpasses this amount. See Integrating Oracle Audit Vault and Database Firewall with Microsoft Active Directory or OpenLDAP for more information.

Solution

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

To resolve this issue and ensure successful authentication, it is important to adhere to the current group string length requirement. Reduce the number of groups for the AD/LDAP users so that the group string length remains within the 8,000 character limit. Additionally, administrators should manage the user group assignments within the Active Directory to ensure that users are only added to necessary groups. After reducing the number of groups, login to AVDF with the AD user as planned.

L.144 Discrepancies When Registering a Target Using Internet Explorer as the Browser

Problem

When using Internet Explorer Audit Vault Server target registration screen is different from the manual.

Solution

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

You will additionally need to use a different broswer as Audit Vault Server console does not support Microsoft Internet Explorer 11 (and prior), starting with Oracle AVDF release 20.6

L.145 Datafiles Don't Change to Read Only Mode After Entering Archive Period

Problem

A tablespace is not entering READ ONLY status even after the archive period has started.

Cause

The definition of ALERT_EVENT_MAP_TRANS is incorrect. The definition of ALERT_EVENT_MAP_TRANS should be the same as the definition of ALERT_EVENT_MAP.

Check the definition of ALERT_EVENT_MAP_TRANS and ALERT_EVENT_MAP by running the following commands:

desc AVSYS.ALERT_EVENT_MAP_TRANS

desc AVSYS.ALERT_EVENT_MAP

Solution

Change the definition of the ALERT_EVENT_MAP_TRANS table to match that of the ALERT_EVENT_MAP table.

L.146 Datafiles Don't Change to Read Only Mode After Entering Archive Period

Problem

A tablespace is in ONLINE status even after the archive period has started, because the tablespace is not entering READ ONLY status.

Symptoms

You can use the following query to determine the date that the tablespace entered the archive period and it's current status:

SQL> select a.tablespace_name, a.status, to_char(b.bytes,'999,999,999,999')"BYTES",
to_char(add_months(to_date('01-01-1970','MM-DD,YYYY'),substr(a.tablespace_name,9,3)+1),'DD-MON-YYYY') "WHEN PLACE OFFLINE",
to_char(add_months(to_date('01-01-1970','MM-DD,YYYY'),substr(a.tablespace_name,14,3)+1),'DD-MON-YYYY') "WILL BE DELETED",
(-1)*months_between(to_char(add_months(to_date('01-01-1970','MM-DD,YYYY'),substr(a.tablespace_name,9,3)),'DD-MON-YYYY'),
to_char(add_months(to_date('01-01-1970','MM-DD,YYYY'),substr(a.tablespace_name,14,3)),'DD-MON-YYYY')) "MONTHNS BETWEEN"
from dba_tablespaces a, dba_data_files b
where a.tablespace_name = b.tablespace_name
and a.tablespace_name like '%ILM%'
order by a.tablespace_name

The output will be in the format: TABLESPACE_NAME STATUS BYTES DATE PLACED OFFLINE DATE IT WILL BE DELETED MONTHNS BETWEEN.

For example, if the output is TABLESPACE_ABC ONLINE 104,857,600 01-OCT-2020 01-APR-2021 6, then it means that the tablespace "TABLESPACE_ABC" is online, contains 104,857,600 bytes, was placed offline on October 1, 2020, was deleted on April 1, 2021, and was in the archive period for six months between October and April.

Solution

1. Check the following information:
   This should return 0:

   select count(*) from avsys.JOB_STATUS_TRANS;

   This should return AVSPACE:

   
   select tablespace_name from dba_tables where table_name ='JOB_STATUS_TRANS';

2. Disable AVS_MAINTENANCE_JOB by running the following:

   exec dbms_scheduler.disable ('AVSYS.AVS_MAINTENANCE_JOB');

   Ensure that it is disabled by running the following:

   SELECT STATE,enabled FROM dba_scheduler_jobs where job_name='AVS_MAINTENANCE_JOB';

3. Set event 14529 at level 512 by running the following:

   alter session set events '14529 trace name context forever, level 512';
   alter system set events '14529 trace name context forever, level 512';

4. Run the following on the Audit Vault database as the AVSYS user:

   DROP TABLE AVSYS.JOB_STATUS_TRANS;
   
   CREATE TABLE AVSYS.JOB_STATUS_TRANS as
       SELECT * FROM AVSYS.JOB_STATUS
           WHERE 1=0;
   
   ALTER TABLE AVSYS.JOB_STATUS_TRANS
       ADD CONSTRAINT CK_JOB_STATUS_TRANS_STATUS
       CHECK (STATUS IN ('Starting',           
                       'Running',           
                       'Stopping',           
                       'Completed',           
                       'Failed',           
                       'Waiting'));
   commit;

   Verify this completed successfully by running the following:

   This should return 0:

   select count(*) from avsys.JOB_STATUS_TRANS;

   This should return 1:

    select count(*)from dba_tables where table_name = 'JOB_STATUS_TRANS';

   This should return AVSPACE:

   select tablespace_name from dba_tables where table_name ='JOB_STATUS_TRANS';

5. Disable event 14529 by running the following:

   alter system set events '14529 trace name context off';
   alter session set events '14529 trace name context off';

   Confirm event 14529 is now disabled by running the following:

   SET SERVEROUTPUT ON
   DECLARE
   event_level NUMBER;
   BEGIN
   DBMS_SYSTEM.READ_EV(14529, event_level);
   dbms_output.put_line (' 14529 is set at level '||TO_CHAR (event_level));
   END;

6. Re-enable the AVS_MAINTENANCE_JOB by running the following:

   exec dbms_scheduler.enable ('AVSYS.AVS_MAINTENANCE_JOB');

   Ensure that it is enabled by running the following:

   SELECT STATE,enabled FROM dba_scheduler_jobs where job_name='AVS_MAINTENANCE_JOB';

L.147 OAV-46599 Internal Error: The Data Guard Observer Is Not Present When Performing Manual Switchover of Audit Vault Server

The internal error OAV-46599 occurs in a High Availability (HA) setup where the data guard observer is found to be absent, preventing the switchover process. To resolve this issue, you should enable automatic failover so that the data guard observer status is set to YES.

Problem

When attempting to perform a High Availability switchover, an OAV-46599 Internal Error occurs, indicating the absence of the Data Guard observer. The following error message is an example of what is displayed:

OAV-46599: Internal Error: The Data Guard Observer is not present

When the automatic failover is disabled, both the primary and secondary (standby) servers will display blank Data Guard observer statuses, which inhibits the switchover process. The following is an example of the first several lines of the status:

$ /usr/local/dbfw/bin/setup_ha.rb --status  
HA mode: PRIMARY  
HA server 1: <IP 1>  
HA server 2: <IP 2>  
Unique database name:   
Current database role: PRIMARY  
Data guard broker: ENABLED  
Data guard observer:  
...

Note:

Data Guard observer is blank instead of saying YES. This occurs in both the primary and secondary (standby) modes.

Cause

This error is caused by the absence of the Data Guard observer, which is necessary for the role switching process in the High Availability setup. The observer statuses are blank because automatic failover must be disabled.

Solution

To resolve this issue, please take the following steps:

1. Enable automatic failover. Please see Disabling or Enabling Failover of the Audit Vault Server for more information.
2. Ensure that the Data Guard observer status is set to YES.
3. Now perform the switchover process.

L.148 Mail Notification Fails When Mailing Server is Configured with TLS/SSL

Problem

When the mailing server is configured with TLS/SSL, the mail notifications fail due to a failure during secure handshake.

Solution

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

L.149 Upgrade To Oracle AVDF 20.5 Fails While Executing Database-Migrations.rb

When attempting to upgrade to Oracle AVDF 20.5, it fails due to executing database-migrations.rb simultaneously. Take the below steps to successfully complete the upgrade.

Problem

Upgrading to Oracle AVDF 20.5 fails due to executing database-migrations.rb simultaneously. First, you should confirm that the upgrade failed due to this issue. Below are the various ways to confirm:

• Check the status within the Oracle AVDF Server.
  1. Log in to the Oracle AVDF server as the root user.
  2. Run the following command: /opt/avdf/bin/privmigutl –status
     If the following error was produced, then the upgrade did fail for this reason:

     System state - recovery  
     Migration set 'AVS' - failed  
     Last migration 'Updating Oracle Audit Vault and Database Firewall data' - failed  
     Migrations will be resumed with 'Upgrading apex20'

• Check the output of this command: /opt/avdf/bin/privmigutl –history. The last three lines produced should be similar to below:

  Migration AVS:35, database-migrations.rb (as root) - failed  
  Migration APPLICATION:4, run-application-migrations["avs"] (as root; retry permitted) - failed  
  Migration TOP:11, run-privileged-migrations["application"] (as root; retry permitted) - failed

• Check the output of this command: /var/log/messages. The result should contain the following error message or similar:

  database-migrations.rb ERROR - ODF-10001: Internal error: Failed to execute: ["/usr/bin/sudo", "-u", "oracle", "-E", "-H", "/var/lib/oracle/dbfw/bin/sqlplus", "/", "as", "sysdba", "@/usr/local/dbfw/bin/migration/connector.sql", "/usr/local/dbfw/bin/migration/2021/changeset_210528_PIGYKICYSE/database.sql"]
  database-migrations.rb ERROR - ODF-10001: Internal error: Incremental migration of the system failed

• Check the output of this command: /var/log/debug. The result should contain the following error message or similar:

  database-migrations.rb DEBUG - Command output: alter table avsys.alert_event_map_arch add policy_name varchar2(4000 char)  
  database-migrations.rb DEBUG - Command output: *
  database-migrations.rb DEBUG - Command output: ERROR at line 1:  
  database-migrations.rb DEBUG - Command output: ORA-01658: unable to create INITIAL extent for segment in tablespace
  database-migrations.rb DEBUG - Command output: AV_ILM_0615_0621

Cause

The upgrade fails because you cannot upgrade the Oracle AVDF server while executing database-migrations.rb.

Solution

To resolve this issue, please take the following steps:

1. Log in to the AVDF server database as sysdba
2. Execute the following query:

   alter table avsys.alert_event_map_arch add policy_name varchar2(4000 char);

   The query should fail with the error:

   ERROR at line 1: ORA-01647: tablespace 'AV_ILM_XXXX_XXXX' is read-only, cannot allocate space in it

3. Make the AV_ILM_XXXX_XXXX tablespace online/read write by executing the below queries in the AV server database as sysdba:

   alter tablespace AV_ILM_XXXX_XXXX online;  
   alter tablespace AV_ILM_XXXX_XXXX read write;

4. Repeat steps 2-3 until the query in step 2 executes successfully.
5. Open the SQL file: /usr/local/dbfw/bin/migration/2021/changeset_210528_PIGYKICYSE/database.sql
6. Comment out the first two alter queries by adding -- at the start of each line.
7. Navigate to /usr/local/dbfw/etc/privileged-migrations/ as the root user.
8. Execute the database-migrations.rb script:

   cd /usr/local/dbfw/etc/privileged-migrations/
   ./database-migrations.rb

9. After the script successfully completes, execute the following command: echo $?.

   If the output is 2, the database-migrations.rb script has completed successfully.

10. Make all tablespaces read only/offline (revert changes from step 3). Do this by executing the following queries in the AV Server Database as sysdba:

    alter tablespace AV_ILM_XXXX_XXXX read only;     
    alter tablespace AV_ILM_XXXX_XXXX offline normal;

11. Log in to Oracle AVDF Server as the root user and resume the upgrade by executing the following command:/opt/avdf/bin/privmigutl --resume –confirm

L.150 How to Disable APEX Developer Console After Upgrading to Oracle APEX 20.1 in Oracle AVDF 20.4

Problem

When upgrading to Oracle APEX 20.1 in Oracle AVDF 20.4, the developer console may become available. The developer console should be disabled.

Solution

To disable the APEX developer console:

1. Log in to the Audit Vault Server through SSH as the support user.

   Note:

   If you're using the Oracle Cloud Infrastructure (OCI) marketplace image, connect through SSH as the OPC user.

   ssh support@<audit_vault_server_ip_address>

2. Switch to the root user.

   su - root

   Note:

   If you're using the OCI marketplace image, use the sudo su - command.

3. Switch to the oracle user.

   su - oracle

4. Start SQL*Plus as sysdba.

   sqlplus / as sysdba

5. Run the following:

    begin
       APEX_INSTANCE_ADMIN.SET_PARAMETER('DISABLE_ADMIN_LOGIN', 'Y');
       APEX_INSTANCE_ADMIN.SET_PARAMETER('DISABLE_WORKSPACE_LOGIN', 'Y');
   end;

L.151 AVDF Agent Deployment Failure: Unable to Get Connection from Datasource

The AVDF Agent deployment fails due to an error connecting to the datasource; the solution is to increase the init parameter processes value to 1000 for the AV repository database.

Problem

AVDF Agent deployment fails with the following error message:

Unable to get connection to the datasource through certificate and without credentials.
Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException:
Cannot get Connection from Datasource: java.sql.SQLRecoverableException: 
IO Error: Got minus one from a read call

Cause

The init parameter processes is set to the default value of 500. This value is too low for the AV repository database.

Solution

To resolve this issue, increase the init parameter processes value to 1000 for the AV repository database. You can do this by running the following SQL query:

ALTER SYSTEM SET processes = 1000;

Once you have increased the value of this parameter, restart the AV repository database. The AVDF Agent deployment should then succeed.

To learn more about the sizing guidelines, review the Audit Vault and Database Firewall Best Practices and Sizing Calculator for AVDF 12.2 and AVDF 20.1 (Doc ID 2092683.1).

L.152 Audit Vault Agent Installation Fails Due To File System Permissions

The Audit Vault agent installation fails when the file system on which the agent is being installed is mounted with the noexec option. This option prevents the execution of programs from the mounted file system.

Symptoms

When attempting to install the Audit Vault agent, the following error message appears:

Error occurred during install/upgrade. Check log files for more information.

The agent deployment log file contains the following error message:

java.io.IOException: Cannot run program "/home/audituser/avault/bin/agentctl": 
java.io.IOException: error=13, Permission denied

Cause

The error occurs because the file system on which the agent is being installed is mounted with the noexec option. This option prevents the execution of programs from the mounted file system.

Solution

To resolve this issue, take the following steps:

1. Check the Java version: Ensure that you have Java SE 6 or later installed on your machine. To check the Java version, run the following command: java -version
2. Verify the file system mount options: Check whether the file system on which the agent is being installed is mounted with the noexecoption. Run the following command to check the mount options: mount. Below is sample output of the mount command showing the file system mounted with noexec option:

   # mount 
   /dev/sda5 on / type ext4 (rw,errors=remount-ro)
   proc on /proc type proc (rw,noexec,nosuid,nodev)
   sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)

3. Remount the file system: If the file system is mounted with the noexec option, remount it without this option. The specific command for remounting the file system will depend on the operating system and file system type. For example, to remount an ext4 file system named /dev/sda5 without the noexec option, you would run the following command:

   mount -o remount,noexec=off /dev/sda5

4. Deploy the Audit Vault agent: After remounting the file system, deploy the Audit Vault agent. The installation should now proceed without encountering the permission error.

L.153 AVDF Agent Deployment Fails on Target Host with RAC DB Due to Incorrect IP Address Registration

The agent deployment on the target host fails due to an incorrect IP address being used when registering the 'secured host' in the AV server. The outgoing IP address of the cluster should be used instead.

Problem

The agent deployment process fails on the target host, resulting in the following errors:

java -jar agent.jar -d <Agent_Home_Path> 
Agent host is not registered. 
Agent host must be registered before an agent can be installed or upgraded. 
Agent deployment failed.

Cause

The target host has a RAC DB installed, and the host's physical IP address was used when registering the 'secured host' in the AV server. This causes the agent deployment to fail.

Solution

To resolve this issue, ensure that the outgoing IP address of the cluster, rather than the physical IP address, is specified when registering the 'secured host' in the AV server.

To determine the outgoing IP address of the host:

1. Connect to the AV database using SQLplus.
2. Execute the following query:

   select sys_context('userenv','ip_address') from dual;

   The result of this query will display the outgoing IP address of the host. Use this IP address when registering the 'secured host' in the AV server.

L.154 Host Monitoring Agent Installation Fails With Error About Inability to Retrieve Agent Details

Problem

Installation of Host Monitoring Agent fails with the following error:

/usr/local/hm# ./hostmonsetup install
Unable to retrieve - 1. Agent User 2. Agent Location 3. Platform Validation 4. HM Install State
Exception occured while creating AVS DB connection. Exception: Error while trying to retrieve text for error ORA-01804
Contact Oracle support.
:/usr/local/hm# ls -ltrh

/usr/local/hm/log# cat av.hmdeployer.log
[2023-09-16 11:06:21,784] [PID: 54294, TName: main] [ERROR] - Exception occured while creating AVS DB connection. Exception: Error while trying to retrieve text for error ORA-01804

[2023-09-16 11:06:21,784] [PID: 54294, TName: main] [ERROR] - Exception Occured: Unable to establish bootstrap connection to AV Server Database using connect string:(DESCRIPTION=(ENABLE=BROKEN)(FAILOVER=on)(R
[2023-09-16 11:06:21,851] [PID: 54300, TName: main] [ERROR] - Exception occured while creating AVS DB connection. Exception: Error while trying to retrieve text for error ORA-01804

Solution

Set the LD_LIBRARY_PATH environment variable as the Host Monitoring Agent installation path:

1. Log in to the Audit Vault Server through SSH and switch to the root user.

   See Logging In to Oracle AVDF Appliances Through SSH.

2. Change the directory to the location of the Host Monitoring Agent:

   cd /user/local/hm

3. Set the LD_LIBRARY_PATH environment variable:

   export LD_LIBRARY_PATH=/usr/local/hm

4. Run the installation of the Host Monitoring Agent:

   ./hostmonsetup install -verbose

L.155 Database Firewall Database Tablespace Growing Quickly in AVDF 20.5

Problem

The tablespace in the database firewall database is continuously growing.

Solution

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

L.156 AVDF 20.3 - 20.6: Cron File Message - Parent Directory Has Insecure Permissions

Problem

The cron file has the following messages:

<Date and Timestamp> <Server-name> CROND[127134]: (root) CMDOUT (error: skipping "/var/lib/oracle/dbfw/av/log/av.jfwk-<log-nujmber-0>.log"
because parent directory has insecure permissions (It's world writable or writable by group which is not "root")
Set "su" directive in config file to tell logrotate which user/group should be used for rotation.)

The log file rotation cron job fails with: because parent directory has insecure permissions.

Cause

The /var/lib/oracle/dbfw/av/log directory has drwxrwx--T 2 oracle dbfw as the ownership and permissions which causes log file rotation issues and stops the Java framework.

Solution

To fix this issue for AVDF 20.3 - 20.6:

1. Log in to the Audit Vault Server through SSH and switch to the root user.

   See Logging In to Oracle AVDF Appliances Through SSH.

2. Execute the following:

   chown oracle:oinstall /var/lib/oracle/dbfw/av/log
   chmod 750 /var/lib/oracle/dbfw/av/log

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

L.157 Audit Vault Agent Fails to Start from Windows Service

When trying to start the Audit Vault Agent from the Windows service, an error is returned.

Problem

The following error was logged in <AVDF AGENT HOME>/av/log/av.agent.prunsvr.YYYY-MM-DD.log:

Unable to find Java Runtime Environment.
The system could not find the environment option that was entered.
reportServiceStatusE: dwCurrentState = 1, dwWin32ExitCode = 0, dwWaitHint = 0, dwServiceSpecificExitCode = 0.

This is because the JAVA_HOME variable was not set in the environment of the Windows operating system (OS).

Solution

1. In the Windows OS, navigate to Control Panel.
2. Click System.
3. Click Advanced system settings.
4. In the Advanced tab, click on Environment Variables button.

   The Environment Variables dialog is displayed.

5. Add a new JAVA_HOME variable that points to your JDK or JRE installation path.

   For example, C:\Program Files\Java\jdk1.8.0_65

6. Start the Audit Vault Agent.

L.158 Error: "tee" Is Not Recognized When Registering Or Starting an Audit Vault Agent on Windows

When registering or starting AVDF AV Agent on Windows server, users may encounter an error stating "tee" is not recognized as an internal or external command.

Problem

When attempting to register or start an AV Agent on Windows Servers, users may encounter the following error:

$agentctl start -k 
Agent updated successfully 
'tee' is not recognized as an internal or external command, operable program or batch file

Cause

The "tee" command error occurs consistently during AV Agent registration or start-up on Windows Servers due to the absence of the "tee" command in the Windows OS.

Solution

The issue does not occur in Oracle AVDF 20.4 and later.

To prevent this issue, apply the patch to update Oracle AVDF to the latest release update (RU). See Patching Oracle Audit Vault and Database Firewall Release 20.

As a workaround until you can patch Oracle AVDF, follow these steps to modify agent.jar and resolve the "tee" command error:

1. Connect to the AV server as the Oracle user:

   su oracle

2. Navigate to $ORACLE_HOME/av/jlib (ORACLE_HOME path is /var/lib/oracle/dbfw by default):

   $cd $ORACLE_HOME/av/jlib

3. Take a backup of the existing agent.jar:

   $cp agent.jar agent_tee.jar

4. Create a script named agent-ch.sh with the provided entries:

   $ vi agent-ch.sh  
   #!/bin/sh 
   cd $ORACLE_HOME 
   cd av/jlib/ 
   /var/lib/oracle/dbfw/jdk/bin/jar -xvf agent.jar bin/agentctl.bat 
   sed -i 's: | tee -a "%OH%\\av\\log\\av.agent.log"::g' bin/agentctl.bat 
   /var/lib/oracle/dbfw/jdk/bin/jar -uvf agent.jar bin/agentctl.bat
   rm /var/lib/oracle/dbfw/av/conf/bootstrap.prop 
   $ORACLE_HOME/bin/avca configure_bootstrap

   Save the file.
5. Provide execute privileges on agent-ch.sh:

   $chmod 744 agent-ch.sh

6. Execute the script agent-ch.sh:

   $./agent-ch.sh

7. Download agent.jar from the AVDF console and use it to deploy the agent on the Windows server.

L.159 AVDF Agent Management after OS Upgrade

After an OS upgrade, users may encounter problems with the AVDF Agent; users should restart the agent to prevent problems.

Problem

When using Oracle AVDF 20.1 and later, users may encounter issues with the AVDF Agent after an operating system upgrade. The AVDF Agent may be affected if specific precautions are not followed.

Solution

To mitigate potential issues after an OS upgrade, follow these steps:

1. Stop AVDF Agent before an OS upgrade:

   <AVDF AGENT HOME>/bin>./agentctl stop

2. After an OS upgrade, start the Agent:

   <AVDF AGENT HOME>/bin>./agentctl start
   <AVDF AGENT HOME>/bin>./agentctl status

Additionally, ensure that the OS version being upgraded is certified and supported by the AVDF Agent.

L.160 Starting a Monitoring Point Causes Error OAV-46649

Problem

After successfully creating a monitoring point, attempting to start it fails. Starting through the AVCLI results in error OAV-46649: Enforcement point is in resume state.

Cause

DNS is not properly configured in the Database Firewall.

Solution

1. Log in to the Audit Vault Server Console as a super administrator.

2. Click Settings tab.
3. Click System in the left menu.
4. Under Status section, click System Settings.
5. Configure DNS settings.
6. Click Save.
7. Start the monitoring point.

L.161 Database Firewall Not Capturing in DAM Mode

Problem

Database Firewall is not capturing in database activity monitoring mode on a VMware installation due to network misconfiguration.

Solution

ESX/VMware virtual switch has a property that does not allow VLAN traffic.

1. The switch needs to be re-configured to allow VLAN traffic.
2. Live capture should start working at this point, test and verify that.
3. Check the reports to ensure they are being populated with data.
4. Check to verify that alerts in the Audit Vault Server console are being generated.

L.162 How to Use Linux to Send E-mails From an AVDF Appliance

Problem

How to use Linux to send e-mails from an AVDF appliance?

Solution

1. Log in to the appliance through SSH and switch to the root user.

   See Logging In to Oracle AVDF Appliances Through SSH.

2. Execute the following command:

   Example: echo TEST | mailx -s "subject " -S smtp=10.10.10.10:25 [e-mail]

L.163 Capture Bind Variables When Running the Database Firewall in DAM Mode

Problem

Is it possible to capture bind variables when running the Database Firewall in database activity monitoring (DAM) mode?

Solution

If the Database Firewall is only used to monitor the secured target through a monitoring point then the All Activity report will not capture bind variables involved in the SQL statement.

L.164 Audit Vault Agent Configuration for a Table Audit Trail in a RAC Environment

Problem

Learn how to configure the Audit Vault Agent for a <codeph>table</codeph> type audit trail in a real application cluster (RAC) environment.

Solution

Install the Audit Vault Agent in one of the following ways:

• The Audit Vault Agent is installed on one of the nodes. If one of the servers go down, the collection will stop.
• The Audit Vault Agent is installed on both of the nodes. If you register the same database twice, one on each node, then there will be duplicate records.
• The Audit Vault Agent is installed on a separate server. To do this:
  1. Register a separate server as host and install the Audit Vault Agent on the machine
  2. Register the RAC database as a secured target
  3. Add a Table type audit trail for this secured target using the same host.

     Since the Table audit trail makes a Java database connectivity (JDBC) connection to the secured target database to fetch the records from the AUD$ table, the audit trail running on a separate host will work without any issues.

L.165 Database Firewall Certificate Validation Failed

If in the Audit Vault Server console the Database Firewall page shows a status of Certificate Validation Failed, follow these steps to resolve the issue.

1. Update the certificate of the Database Firewall.

   For more information see Fetching an Updated Certificate from Database Firewall.

2. If updating the certificate does not resolve the issue, rotate the Database Firewall certificate.

   For more information see Rotating Database Firewall Certificates.

3. If the Database Firewall certificate can't be rotated, it may be because the Audit Vault Server certificate authority is no longer valid on the Database Firewall. Follow these steps to resolve the issue:

   1. Log in to the Audit Vault Server through SSH and switch to the root user.

      See Logging In to Oracle AVDF Appliances Through SSH.

   2. Run the following command:

      openssl x509 -noout -subject -in /usr/local/dbfw/etc/ca.crt

      Take note of the output.

   3. Log in to the Database Firewall through SSH and switch to the root user.

      See Logging In to Oracle AVDF Appliances Through SSH.

   4. Run the following command:

      openssl x509 -noout -subject -in /usr/local/dbfw/etc/controller.crt

   5. If the outputs of the commands are different, then you need to add the Audit Vault Server certificate to the Database Firewall.

      For more information see Specifying the Audit Vault Server Certificate and IP Address.

L.166 Configuring ERSPAN for SQL Traffic Auditing in Monitoring (Out of Band) Mode

If Monitoring (Out of Band) is not collecting any SQL traffic audit, follow these steps to resolve the issue.

Problem

Monitoring (Out of Band) in AVDF 20.7 is not collecting any SQL traffic audit. Even though the network interface cards (NICs) are correctly configured and the traffic is being captured in the pcap files, no SQL traffic audit is displayed on the AVDF web page.

Cause

The Database Firewall does not process ERSPAN traffic by default in Monitoring (Out of Band) mode. This has to be enabled on the Database Firewall monitoring points, otherwise, the SQL traffic audits will not be displayed despite being correctly mirrored and captured.

Solution

To resolve this issue, you need to enable ERSPAN processing by setting DAM_TRAFFIC_IS_ERSPAN=1. More information can be found in Configuring Encapsulated Remote Switched Port Analyzer with Database Firewall.

L.167 Recovery Disk Group is Getting Full with Archive Logs

Problem

Archive logs are not deleting and are causing the disk group of the standby Audit Vault Server in a high availability pairing to get full.

Run the following commands to determine if the archivelog has been applied to the standby:

select a.thread#, a.sequence#, a.applied
from v$archived_log a, v$database d
where a.activation# = d.activation#
and a.applied='YES'
/

Make sure the standby is in sync with the primary.

Solution

Ensure that your retention policies are set as this can help free up space in the fast recovery area. For more information see Creating and Deleting Archive and Retention Policies.

If no files are eligible for deletion based on their retention policy, then manual intervention is required. For more information see Managing Archival and Retrieval in High Availability Environments.

L.168 Cannot View the Updated Maintenance Job Schedule After Making Changes

After changing the maintenance job schedule in AVS, the updated time may incorrectly display as 0:00 due to a display issue, but the changes are correctly applied.

Problem

After changing the start time of the maintenance job schedule in the Audit Vault Server, the schedule displays as 0:00 instead of the updated time upon re-login.

Cause

This is a display issue only found in Oracle AVDF 20.7 and later.

Solution

Although 0:00 is displayed, the schedule changes have been successfully applied. You can verify the updated schedule by running the following SQL query in the AVS repository database:

select next_run_date,STATE,enabled FROM dba_scheduler_jobs where job_name='AVS_MAINTENANCE_JOB';

L.169 Oracle AVDF Does Not Failover When Primary Server Is Down

When the network connection to the Primary Oracle AVDF server is down, the system does not failover to the Standby server. The current design only triggers a failover if the Oracle AVDF Database or a critical process crashes.

Symptoms

When the network connection to the Primary setup is down and the Primary Oracle AVDF server becomes inaccessible, Oracle AVDF does not initiate a failover to the standby server. Once the network connection to the primary server is restored, Oracle AVDF becomes accessible again through the primary site.

Cause

Oracle AVDF is currently designed to failover only if the Oracle AVDF Database or a critical process crashes and triggers a failover to the Standby site. However, if the network connection is disabled or down, the Observer cannot determine the status of the processes, and as a result, failover will not occur.

Solution

The current Oracle AVDF failover mechanism does not guarantee High Availability, as failover only occurs during process crashes. In cases of network or system outages, the service may remain down. To maintain continuous availability, Oracle AVDF should be accessible from the Standby site when the Primary server is inaccessible. Implementing a Load Balancer could help by directing traffic to the Standby site in such scenarios. See Handling a Failover Scenario for more information on Failover Scenarios.

L.170 Upgrading AVDF from 20.7 to 20.8 Fails When Rebuilding the Index with UTLRP.SQL

When attempting to upgrade to Oracle AVDF 20.8, it fails while rebuilding the index by executing UTLRP.SQL after setting max_string_size to extended.

Problem

When upgrading Oracle AVDF from 20.7 to 20.8, it fails while rebuilding index with UTLRP.SQL. You may receive the following error logged in /var/log/messages:

HOSTNAME su: (to oracle) root on none
HOSTNAME com.oracle.privilegedMigration.max_string_size_extended: Failed utlrp.sql to setup MAX_STRING_SIZE to extended.
HOSTNAME run-application-migrations[28910]: com.oracle.dbfw.privilegedMigration ERROR - ODF-10001: Internal error: FAILEDmigration: max_string_size_extended (as root) (applied change)

Cause

The upgrade fails while rebuilding the index by executing UTLRP.SQL after setting max_string_size to extended.

Solution

To resolve this issue and resume upgrade, follow the steps below:

1. Connect to AVDF DB as sys

   sqlplus / as sysdba
   @/var/lib/oracle/dbfw/rdbms/admin/utlrp.sql
   DECLARE

   The query should fail with the following error:

   ERROR at line 1: 
   ORA-01502: index 'SYS.I_WRI$_OPTSTAT_HH_OBJ_ICOL_ST' or partition of such index is in unusable state
   Function created.
   PL/SQL procedure successfully completed.
   Function dropped.
   Warning: XDB now invalid
   PL/SQL procedure successfully completed.

2. Rebuild the package to fix the issue by executing the following query:

   alter index SYS.I_WRI$_OPTSTAT_HH_OBJ_ICOL_ST rebuild;

3. Executed the script again:

   @/var/lib/oracle/dbfw/rdbms/admin/utlrp.sql

4. Run the following commands:

   alter package objowner.object compile body;
   grant execute on DBMS_SQL to public;

5. Check if any other invalid objects are present:

   SELECT owner, object_type, object_name FROM dba_objects WHERE status !='VALID';

6. Drop the trigger which failed:

   drop trigger avsys.start_el_migration;

7. Resume the upgrade:

   /opt/avdf/bin/privmigutl --resume --confirm

L.171 Executing 'AVBACKUP BACKUP' Command Fails

Problem

When executing the '/var/lib/oracle/dbfw/bin/avbackup backup' command after running '/var/lib/oracle/dbfw/bin/avbackup config' , the operation fails with the following error "info.txt: No such file or directory."

Solution

The info.txt file should be located in the same directory. The user must ensure that the backup directory and its parent directories are owned by oracle:oinstall to prevent this error.

see Backup and Restore of Audit Vault Server for more information.

L.172 Error OAV-47411 "Export Path" Does Not Exist on Remote File System

Learn what to do when you receive the OAV-47411 error while registering a Network File System (NFS) export to Oracle AVDF.

Problem

While registering an NFS export by executing the following command:

AVCLI> REGISTER REMOTE FILESYSTEM <Remote FS name> OF TYPE NFS ON HOST <Hostname> USING EXPORT <Export path> MOUNT;

This error might be encountered:

OAV-47411: Export <Export path> does not exist on remote filesystem.

Cause

To identify the root cause, complete the following steps:

1. Run the following AVCLI commands and ensure the outputs are correct:
   • The output should display the export path:

     AVCLI> LIST EXPORT OF TYPE NFS ON HOST <Hostname>;

   • The output should display the Remote Filesystem name, along with export path:

     AVCLI> LIST REMOTE FILESYSTEM;

   • The output should be ACCESSIBLE:

     SHOW STATUS OF REMOTE FILESYSTEM "<Name of the Remote Filesystem>";

2. On the NFS server, execute the following command to check the existence and the permissions on the export path:

   ls -ld <Export path>

3. Check if the entry for the NFS location is located in the /etc/fstab within the Audit Vault Server:

   cat /etc/fstab

4. Check the output of following queries:

   select * from avsys.remote_filesystem;
   select * from avsys.remote_location;
   select * from avsys.archive_host;

5. Follow the below steps on the AV Server:
   1. Login to the AV Server as support.
   2. Execute the following commands:

      su root
      su oracle
      cd $ORACLE_HOME

   3. The output for the above command should display the export list for <IP address of the NFS server>, as shown below:

      [oracle@<AV host>]$ /usr/sbin/showmount --export <IP address of NFS server>
      clnt_create: RPC: Port mapper failure - Unable to receive: errno 111 (Connection refused)

      This means the ports (NFS) and 111 (port map) are blocked by the firewalls on the NFS server OR they are not open.

Solution

To resolve this error, check if the NFS server is reachable and all the required ports are open (no firewall is blocking the request on specific ports).

1. Turn off the firewall on the NFS machine.
2. Register the remote filesystem.
3. Mount and check the status.

L.173 AVDF 20.4 Error Accessing Target Report: "P107_FIRST_RUN_TIME_AUDIT"

While accessing the secured target on the AVDF Console, users receive an error "P107_FIRST_RUN_TIME_AUDIT" in Oracle AVDF 20.4.

Problem

When accessing a secured target in the Oracle AVDF 20.4 console, users encounter the following error message:

"Error computing item source value for page item P107_FIRST_RUN_TIME_AUDIT".

Cause

The error message indicates an issue in retrieving or calculating the source value for the specified page item.

Solution

This issue is resolved in Oracle AVDF 20.6. For earlier versions, the following workaround can be applied:

1. Schedule Audit Policy/User Entitlement (UE) Retrieval with One Auditor only: If the audit retrieval is already scheduled by multiple auditors, delete one.
2. Steps to perform in AVSYS:
   1. Find the Target ID of the affected target:

      select secured_Target_id from avsys.secured_Target where secured_Target_name='target name';

   2. Check schedules created for that target by auditors:

      select * FROM avsys.retrieval_schedule where secured_Target_id = < Target Id from first SQL Output>;

   3. Delete an entry for one auditor:

      DELETE FROM avsys.retrieval_schedule where secured_Target_id = < Target Id from first SQL Output> and user_name = 'username' ;

After completing these steps, the error should no longer occur when accessing the target report.

L.174 Error OAV-47487: Uploading a Certificate to AVDF Fails

Learn what to do when you receive the OAV-47487 error while uploading the certificates to AVDF.

Problem

Uploading a new certificate generated using a CSR from an external source (such as a third-party application) fails with the following error:

OAV-47487: Certificate is not compatible with server

Cause

Oracle AVDF 12 does not support CSRs that originate outside of its own system.

Solution

The only supported process is to generate the CSR directly from the AVDF application, signing it with a CA, and then uploading the signed certificate. Follow the steps below:

1. Download the CSR from the AVDF Server.
2. Have the CSR signed by the Certificate Authority (CA).
3. Then, upload only the newly signed certificate (excluding the CSR and any intermediate certificates). A certificate chain is not supported.

L.175 Troubleshooting Server Error 500 in AVDF

Learn how to identify the cause of "Server Error 500" in the Oracle AVDF environment.

Problem

When logging into the AVDF Web Console, users may encounter "Server Error 500." This error typically indicates a failure to connect to the repository database, a critical back-end required for the Web Console. Without this connection, login fails, and a Server Error 500 is triggered.

Cause

There are various possible reasons for the Server 500 Error:

1. User account or password issues: Incorrect password, locked account, or unsupported characters in passphrase.
2. Database unavailability: The repository database may be down or not correctly configured.
3. Service startup issues: The database services may not have started correctly, or other dependencies may be unavailable.
4. Database limitations: Connection may be restricted due to database limitations or session limits.
5. Timeout issues: Long loading times for dashboard or console due to performance delays.
6. File system iNode exhaustion: The iNode count on /var/lib/oracle is full, preventing login.
7. Other configuration issues: Other system-level configurations may block access.

Solution

To troubleshoot and resolve Server Error 500 in AVDF, follow these steps based on the potential causes listed above:

1. User Account or Password Issues:
   1. Verify login credentials by attempting to connect with sqlplus:

      su - oracle
      sqlplus <avadmin_user>/<password>

   2. If login fails, try changing the password or unlocking the account:

      su - oracle
      sqlplus "/as sysdba"
      SQL> ALTER USER <avadmin_user> IDENTIFIED BY <new_password> ACCOUNT UNLOCK;

   3. Ensure the passphrase does not contain special characters other than _, as unsupported characters may cause login issues.
2. Database Unavailability:
   1. Check /var/log/messages for specific errors like ORA-01034 or ORA-27101 indicating that the database is not available.
   2. For persistent issues, inspect alert.log and diagnostic files. Rebooting the AVS server may help restart the repository database and resolve the error.
3. Service Startup Issues:
   1. Verify that required services, including Grid Infrastructure (GI) resources, +ASM instance, and TNS listener, are running.
   2. Restart the AVS server if services are not initialized correctly, especially if running AVDF version 12.2.0.4 or later.
4. Database Limitations:
   1. Check for ORA-20 or ORA-18 errors, which indicate session limits. Reboot the AVS server if these limitations cause connectivity issues.
   2. If session limits continue to be problematic, contact Oracle Support for further investigation.
5. Timeout Issues:
   1. Long loading times for the dashboard or Web Console may result in Server Error 500. This is often cause by performance issues or large alert volumes.
   2. For AVDF versions before 12.2 BP#5, increase the TIMEOUT setting in /usr/local/dbfw/templates/template-httpd-httpd.conf and restart networking settings.
6. File System iNode Exhaustion:
   • Run df -i to check if /var/lib/oracle has reached 100% inode usage. If so, remove excess audit files.
7. Other Configuration Issues:
   • Review recent system or database changes. Incorrect configurations, such as manual host reboots without reconfiguration, may disrupt the database.

Following these steps should help diagnose and resolve the underlying cause of Server Error 500. If issues persist, consult Oracle Support with AVDF diagnostic files for additional assistance.

L.176 User Entitlement Retrieval Job Fails After Twelve Hours

Learn how to manage when a user entitlement job fails after running for twelve hours.

Problem

User Entitlement retrieval jobs consistently fail after running for an extended period. This issue may occur when retrieving data from databases with a large number of accounts, resulting in job termination before completion. The failure typically occurs with an error message indicating the inability to process privilege user data for the target database.

As a result, entitlement snapshots cannot be generated, which impacts reporting capabilities such as Privileged Users or other entitlement reports.

Cause

This issue is caused by a system-defined timeout setting that limits the maximum runtime for jobs. when the job exceeds this limit, it terminates prematurely. Logs may show errors such as: java.sql.SQLRecoverableException: IO Error: Socket read interrupted.

Solution

This issue is caused by a system-defined timeout setting that limits the maximum runtime for jobs. when the job exceeds this limit, it terminates prematurely. Logs may show errors such as: java.sql.SQLRecoverableException: IO Error: Socket read interrupted

To resolve this issue, follow the steps below:

1. Modify the job configuration by executing the following commands:

   su - oracle
   vi /var/lib/oracle/dbfw/bin/avjfwk

2. Update the relevant Java process line to include the following:

   /usr/bin/java -ea -Dfile.encoding=UTF-8 -DNLS_LANG="$NLS_LANG"-DORACLE_HOME="$OH" -Doracle.jdbc.javaNetNio=false -mx1024m -classpath$CLASSPATH oracle.av.platform.server.javafwk.JfwkProcess "$@" >/dev/null 2>&1

3. Restart the javafwk Service:

   restart javafwk  
   systemctl stop javafwk  
   systemctl status javafwk  
   systemctl start javafwk

4. Increase Timeout Settings:

   AVCLI>alter system set JFWK.THREAD_TIMEOUT_MINUTES=1440;

5. Resubmit the job. Once the changes are applied, resubmit the User Entitlement job from the AV UI to ensure it completes successfully.

L.177 Unable to Drop Audit Trail from Unreachable Host

If the error "OAV-46572: Agent is unreachable on host" appears during an audit trail drop operation, use the following steps to resolve it.

Symptoms

Attempts to drop an audit trail using the GUI or AVCLI fail with the error message:

OAV-46572: Agent is UNREACHABLE on host

The system reports that the audit trail from the secured target is still running, even though the host or agent has been removed.

Cause

This issue occurs if the agent and audit trails were removed without properly stopping them first. The Audit Vault server keeps the audit trail in an "UNREACHABLE" status, anticipating that the host or agent might recover. Since the agent is inactive, the audit trail cannot be fully stopped.

Solution

If the host and agent will not be restored, proceed with the following steps to manually update the audit trail status:

1. Connect to Audit Vault Server as the support user, switch to root, then to the dvaccountmgr, and access SQL*Plus.
2. Set a temporary password for the AVSYS user and unlock the account.
3. Execute the following SQL command to update the audit trail status to "STOPPED":

   UPDATE AVSYS.AUDIT_TRAIL SET COLLECTION_STATUS=0 WHERE COLLECTION_STATUS <> 0 AND ACTIVE='Y' AND HOST_NAME = '&unreachable_host' AND AUDIT_TRAIL_ID IN(SELECT AUDIT_TRAIL_ID FROM AVSYS.AUDIT_TRAIL WHERE HOST_NAME IN(SELECT HOST_NAME FROM AVSYS.AGENT_VIEW WHERE STATUS='UNREACHABLE'));

   COMMIT;

4. After executing this update, attempt to drop the audit trail again using the GUI or AVCLI.

Note:

In Oracle AVDF 20.5 and later, you can use the command DROP HOST <hostname> FORCE to force the host and its audit trails to be dropped directly.

L.178 Error OAV-47746: Sensitive Objects Data Upload Fails

Learn what to do when you receive the OAV-47411 error while uploading sensitive objects data in Oracle AVDF.

Problem

When attempting to upload sensitive object data in AVDF 20.8, users encounter the following error:

OAV-47746: "Input file with sensitive data is invalid format."

Cause

This error may occur when unsupported, invisible characters are present in the file. These characters can cause the file format to be unrecognized during the upload process.

Solution

To resolve this issue, follow the below steps:

1. Open the file with sensitive data and re-saved it.
2. Convert the file from DOS to Unix format using the following command:

   dos2unix <example.txt>

3. After converting the file, attempt to upload it again through the AVDF UI console.

For more information on converting files from DOS to Unix format, refer to "Convert DOS to Unix".

L.179 Status "Certificate Validation Failed" Error Shown in Audit Vault Server GUI

If Database Firewall page shows a status of Certificate Validation Failed in the Audit Vault Server GUI with the error OAV-46981: Unable to connect to Database Firewall with IP <ipaddress>, follow these steps to resolve the issue.

Problem

The Audit Vault Server GUI displays the status "Certificate Validation Failed" for the Database Firewall. Moreover, the following errors appear in the Host Monitor log:

• OAV-46981: Unable to connect to Database Firewall with IP <ipaddress>
• ORA-29273, ORA-28791, ORA-06512: Various errors indicating HTTP request failure and certificate verification failure.
• Log errors show that the certificate and key files for Host Monitor could not be loaded, and SSL handshake failed.

Cause

Database Firewall is down due to failed certificate validation.This may be caused by issues with the existing certificates or keys used for SSL communication.

Solution

To resolve the issue, perform the following steps:

1. Take a backup of /usr/local/dbfw/etc/avs/ folder.
2. Remove the existing certificates and wallet files. On the Audit Vault Server, as the root user, execute the following commands:

   rm -f /usr/local/dbfw/etc/avs/fwcerts/*

   rm -f /usr/local/dbfw/etc/avs/avswallet/*

3. Generate new SSL certificates. Run the following command to recreate the necessary certificates:

   /usr/local/bin/gensslcert create-certs

4. Update certificates in the AV Console:
   • Log in to the AV console as an administrator.
   • Go to the Database Firewall tab.
   • Select each Database Firewall and click the "Update Certificate" button.
   • Confirm that the Database Firewall status is now showing up as "Up."

   This solution should resolve the certificate validation issue and restore the connection with the Database Firewall.