G Troubleshooting Oracle Audit Vault and Database Firewall

This appendix describes common troubleshooting advice.

G.1 Audit Vault Agent or Host Monitor is not Upgraded to the Latest Bundle Patch

Learn how to upgrade the Audit Vault Agent or Host Monitor Agent manually.

Problem

After upgrading to Oracle AVDF 12.2.0.13.0 or later, some of the Audit Vault Agents or Host Monitor Agents are not upgraded.

Symptom - 1

Audit Vault Agent is in STOPPED state after Audit Vault Server upgrade.

Symptom - 2

Host Monitor Agent is in NEEDS UPGRADE or UPDATE FAILED state after Audit Vault Server upgrade.

Solution - 1

The symptom indicates that the Audit Vault Agent has failed to auto upgrade during the Audit Vault Server upgrade. Execute the following steps as the user who installed Agent previously:

  1. Check for any Agent processes on the host machine. Ensure there are no Agent related processes currently running.
  2. Remove the existing agent.jar file and the Agent folder from the host machine.
  3. Download the new agent.jar file from the upgraded Audit Vault Server.
  4. Execute the following command:

    java -jar agent.jar [-d <AgentHome>]
    
  5. Verify the Agent is in RUNNING state.

Solution - 2

The symptom indicates that the Host Monitor Agent has failed to auto upgrade during the Audit Vault Server upgrade. Execute the following steps as root user:

  1. Check for any Host Monitor related processes on the host machine. Ensure there are no hostmonitor, hmdeployer, or hostmonmanager processes currently running.
  2. Navigate to the directory outside of hm where the Host Monitor is installed.
  3. Execute the following command to uninstall the Host Monitor:

    ./hm/hostmonsetup uninstall
    
  4. Download the new Host Monitor installable bundle from the Audit Vault Server console, for the specific platform on which it will be reinstalled.
  5. Extract the Host Monitor bundle inside the hm directory.
  6. Execute the following command to reinstall the Host Monitor in a root owned location:

    ./hostmonsetup install
    

G.2 Enable Archiving Functionality Post Upgrade From BP11 to Later Releases

Learn how to fix disabled archiving functionality post upgrade from Oracle AVDF 12.2.0.11.0 to later releases (12.2.0.12.0 or 12.2.0.13.0).

Problem

Archiving functionality may be disabled after upgrading from Oracle AVDF 12.2.0.11.0 or 12.2.0.12.0, to 12.2.0.12.0 or 12.2.0.13.0.

Archiving functionality for high availability environment is supported starting Oracle AVDF release 12.2.0.11.0. This problem arises when you are upgrading from older releases where archiving functionality is supported only on the primary Audit Vault Server. Execute the steps only if you are upgrading from Oracle AVDF 12.2.0.11.0 to later releases.

Note:

If you are upgrading from any release prior to Oracle AVDF 12.2.0.11.0, then follow the steps documented in section Enable Archiving Functionality Post Upgrade.

Solution

If archive locations were present before upgrading to 12.2.0.13.0, execute the following steps to enable archiving functionality. These steps must be executed post upgrade process.

  1. Create a new archive location using the Audit Vault Server console. While creating this new archive location enter the details of both the primary and standby location. This will mount the new archive location on the primary or standby Audit Vault Server and update the fstab with both archive locations.

  2. SSH to the primary Audit Vault Server as support user and then unlock the avsys user by executing the following commands:

    su root
    
    su dvaccountmgr
    
    sqlplus /
    
    alter user avsys identified by <new password for avsys user> account unlock;
    
    exit;
    
    exit;
    
  3. Execute the following commands as avsys user:

    su oracle
    
    sqlplus avsys
    
  4. Enter the avsys password when prompted. Execute the following SQL commands:

    delete from avsys.system_configuration where property = '_ILM_ARCHIVING_DISABLED';
    
    COMMIT;
    
    insert into avsys.system_configuration values ('_ILM_HA_UPGRADE_COMPLETED', 'Y');
    
    COMMIT;
    
    exit;
    
    exit;
    
  5. Execute the following command to lock the avsys user:

    su dvaccountmgr
    
    sqlplus /
    
    alter user avsys account lock;
    
  6. Delete the new archive location that was created in the initial step. Navigate to Settings tab, then click Manage Archive Locations in the left navigation menu. Delete the specific archive location.

G.3 Partial or No Traffic Seen for an Oracle Database Monitored by Database Firewall

Problem

I see no traffic, or only partial traffic, captured in reports for an Oracle Database monitored by the Database Firewall.

Solutions

Go through the following checks to find the trouble:

  1. In the Audit Vault Server, check that the report filters are set correctly, including the time slot.

  2. Check that the system time on the Database Firewall is synchronized with the time on the Audit Vault Server and the secured target system.

  3. Check that the secured target's network traffic is visible to the Database Firewall using the Live Capture utility on the firewall.

  4. Check that the Oracle Database service name or SID is used correctly. If you specified an Oracle Database service name in the Enforcement Point settings for this secured target, you will only see traffic for that service name. To see all traffic, remove the service name from the Enforcement Point settings to see all traffic.

    If you have entered a service name in the Enforcement Point, and see no traffic, check to see that the service name is entered correctly in the Enforcement Point settings.

    For Enforcement Points set to use DAM mode, the Database Firewall may be monitoring traffic for existing client connections to the database. Since these connections were in place before you deployed the Database Firewall, it will not be able to detect the service name you specify in the Enforcement Point. In this case, restart the client connections to the database.

  5. Check that the correct Database Firewall policy is deployed.

See Also:

G.4 RPM Upgrade Failed

Problem

An RPM upgrade failed with the following error:

error: %post(dbfw-mgmtsvr-###) scriptlet failed, exit status 1

Solution

  1. Check that there is at least 10MB of free /tmp space.

  2. Remove the new RPM:

    rpm -e dbfw-mgmtsvr-###

  3. Retry the upgrade.

G.5 Agent Activation Request Returns 'host is not registered' Error

Problem

I used the following two commands to register the Audit Vault Agent's host computer (where the agent is deployed), and to request Audit Vault Agent activation:

From the Audit Vault Server:

avcli> register host 'host_name'

From the host computer:

agentctl activate

But the agentctl activate command returns: Agent host is not registered

Solution

Your agent host may be multi-homed. In this case, the agent hostname to IP address resolution may resolve to the NIC/IP that is not used by the agent while connecting to the AV server. To resolve this issue, try to register the agent host using the with ip option and then try activating the agent again.

From the Audit Vault Server, use the following command:

avcli> register host 'host_name' with ip 'host_ip_address'

If you still have issues, try finding the IP address used in the database session when you connect to the Audit Vault server from the agent host, using these commands:

sqlplus username/password@"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=Audit_Vault_Server_IP)(PORT=1521))(CONNECT_DATA= (SERVICE_NAME=dbfwdb)))"

sqlplus> select SYS_CONTEXT('USERENV','IP_ADDRESS') from dual;

Use the IP address from the above query to register your host.

G.6 Unable to Deploy Agent on the Secondary Audit Vault Server

Learn the resolution if you are unable to deploy an agent on a secondary Oracle Audit Vault server.

Problem

When I try to deploy the Audit Vault Agent on the secondary Audit Vault Server in a high availability pair, I get an error that the host is not registered.

Cause

After you pair two Audit Vault Servers for high availability, you do all configuration on the primary server in the pair only, including Audit Vault Agent deployment.

G.7 Operation Fails When I Try to Build Host Monitor or Collect Oracle Database Trail

Problem

This problem may manifest with various symptoms:

  • When I try to build a host monitor, the operation fails or cannot find the correct binaries.

  • When I try to collect audit data from an Oracle Database secured target, the operation fails.

  • The Audit Vault Agent cannot connect to the Audit Vault Server.

  • Audit trail does not start.

Solution

  1. Unset all environment variables except the following:

    • PATH

    • TERM

    • PS1

    • LANG

    • LC_*

    • JAVA_HOME

    Then run the java -jar agent.jar command again on the host machine.

  2. If you deployed the Audit Vault Agent in a Linux environment, ensure that the host machine name is present in the /etc/hosts file.

G.8 'java -jar agent.jar' Failed on Windows Machine

Problem

The command java -jar agent.jar failed on my Windows secured target machine, and I noticed in the log files that the Audit Vault Agent services installation/un-installation failed.

Solution

  1. Follow the instructions for unregistering the agent in Registering and Unregistering the Audit Vault Agent as a Windows Service.

    If Method 1 fails, then try Method 2.

  2. Run the java -jar agent.jar command again.

G.9 Unable to Install the Agent or Generate the agent.jar File

Determine the steps to perform if you are unable to install the agent or generate the agent.jar file.

Problem

Unable to install the Audit Vault Agent. Attempts to regenerate the agent.jar file are also unsuccessful.

Solution

Follow these steps to regenerate the agent.jar file:

  1. Log in to the Audit Vault Server through SSH as user oracle.

  2. Go to the directory /var/lib/oracle/dbfw/av/conf/ location.

  3. Delete the bootstrap.prop file.

  4. Execute the following command:

    /var/lib/oracle/dbfw/bin/avca configure_bootstrap

  5. Check the avca.log file that is available at /var/lib/oracle/dbfw/av/log/ to check if the above command was executed successfully.

  6. Switch the user (su) to avsys.

  7. Run the following query:

    select agent_gen_ts from file_repos where file_name='agent.jar';

  8. The above query displays the current time in case the agent.jar file is generated successfully.

G.10 Unable to Un-install the Oracle Audit Vault Agent Windows Service

Review the troubleshooting advice if you are unable to un-install the Oracle Audit Vault Agent Windows Service.

Follow the instructions for unregistering the Agent inRegistering and Unregistering the Audit Vault Agent as a Windows Service.

If Method 1 fails, then try Method 2.

G.11 Access Denied Error While Installing Agent as a Windows Service

Problem

I got an error during installation of the Audit Vault Agent on Windows, and I noticed the following error in the AGENT_HOME\av\log\av.agent.prunsvr log file:

[2013-05-02 11:55:53] [info] Commons Daemon procrun (1.0.6.0 32-bit) started
[2013-05-02 11:55:53] [error] Unable to open the Service Manager
[2013-05-02 11:55:53] [error] Access is denied.
[2013-05-02 11:55:53] [error] Commons Daemon procrun failed with exit value:
7 (Failed to )
[2013-05-02 11:55:53] [error] Access is denied. 

Solution

The above message means that the logged in user does not have privileges to install the Audit Vault Agent as a Windows Service. If you get the above message, try launching the command shell with the Run As Administrator option, and then execute java -jar agent.jar in that command shell.

G.12 Unable to Start the Agent Through the Services Applet On The Control Panel

Problem

I did the following:

  1. Installed the Audit Vault Agent using the java -jar agent.jar command.

  2. Activated the Audit Vault Agent.

  3. Started the Audit Vault Agent using the agentctl start -k key command.

    The agent started up and is in RUNNING state.

  4. Stopped the Audit Vault Agent.

  5. Tried to start the Audit Vault Agent using the Services Applet on the Windows Control Panel.

    The Audit Vault Agent errored out immediately.

Solution

This means that the Audit Vault Agent is configured to use a Windows account that does not have privileges to connect to the Audit Vault Server.

Take the following steps:

  1. Go to Control Panel, then to Services Applet.

  2. Select the Oracle Audit Vault Agent service.

  3. Right click and select the Properties menu.

  4. Click the Log on tab.

  5. Select This account: and then enter a valid account name and password.

  6. Save and exit.

  7. Start the Audit Vault Agent through the Services Applet.

G.13 Error When Starting the Agent

Problem

After I installed the Audit Vault Agent, I set the username and password in the OracleAVAgent Windows Service Properties Log On tab. However, when I try to start the OracleAVAgent service, I see the following error in the Agent_Home\av\log\av.agent.prunsvr.date.log file:

[info]  Commons Daemon procrun (1.0.6.0 32-bit) started
[info]  Running 'OracleAVAgent' Service...
[info]  Starting service...
[error] Failed creating java 
[error] ServiceStart returned 1
[info]  Run service finished.
[info]  Commons Daemon procrun finished

Solution

This means that the OracleAVAgent service is not able to launch the Java process. Try the following:

  1. Uninstall all JDKs and/or JREs in the system.

  2. Reinstall JDK SE or JRE and then start the OracleAVAgent service.

  3. If this doesn't help, you can install 32 bit JDK SE or JRE and then start the OracleAVAgent service.

G.14 Error When Running Host Monitor Setup

Problem

I am setting up a Host Monitor. When I run the command bin/hostmonsetup install, the following error is displayed:

[root@dbsec1 av]# bin/hostmonsetup install /usr/bin/ld: cannot find -lpcap collect2: ld returned 1 exit status make: *** [hostmonitor] Error 1 Line 105: Failed to generate executables for Host monitor.

Solution

This means the host computer does not have the required libraries for the Host Monitor. Install the required libraries listed in Host Monitor Requirements.

G.15 Alerts on Oracle Database Secured Target are not Triggered for a Long Time

Problem

I configured an Oracle Database secured target to audit to XML files, configured an audit trail in Oracle AVDF of type DIRECTORY, and then configured an alert to trigger on certain events. My alert did not get triggered for a long time.

Solution

This issue can occur if the Oracle Database secured target is not flushing the audit records to the file immediately. Contact Oracle Support in order to access support note 1358183.1 Audit Files Are Not Immediately Flushed To Disk.

G.16 Error When Creating an Audit Policy

Resolve errors that can occur when you create an audit policy.

Problem

I received this error message when I tried to create a new audit policy setting for Oracle Database:

-ORA-01400: cannot insert NULL into ("AVSYS"."AUDIT_SETTING_ARCHIVE_MAP"."ARCHIVE_ID")

Cause

The Oracle Database must have at least one audit policy setting before you can create and provision new audit settings using Oracle Audit Vault and Database Firewall. Oracle Database comes with a predefined set of audit policy settings. You must not manually remove these settings. If the audit settings have been removed, then you can manually create at least one audit setting in the Oracle Database. Then try again to create new audit settings using Oracle Audit Vault and Database Firewall.

See Also:

Oracle Database Security Guide for detailed information on Oracle Database auditing.

G.17 Connection Problems when Using Database Firewall DPE Mode

Problem

In DPE (blocking) mode, my client application cannot connect to the secured target database.

Solution 1

  1. Log in as root on the Database Firewall server.

  2. Execute this command, using the secured target database IP address or host name:

    ping -I secured_target_ip_address_or_hostname

    If no response is received, check that:

    • The bridge IP settings are correct.

    • The bridge IP address is on the same subnet as the secured target database.

    • DNS is configured on the Database Firewall

    If a response is received, check:

    • The firewall policy to ensure it is not blocking the connection attempt.

    • The client connection settings to ensure that the client is attempting to connect to the correct secured target database.

Solution 2

If your client application computer is on a different subnet than the secured target database, see document number 1566766.1 on My Oracle Support https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1566766.1.

G.18 Audit Trail Does Not Start

Problem

An audit trail does not start. For example, in the Audit Vault Server console, in the Audit Trails page, the Collection Status column indicates that the trail is Stopped or Unreachable.

Solution

When a trail does not start, you can show the associated error in two ways:

  • In the Audit Vault Server console:

    1. Click the Secured Targets tab, and then from the Monitoring menu, click Audit Trails.

    2. Click the Actions button, and then click Select Columns.

    3. From the left-hand box, double-click Error Message so that it moves into the Display in Report box on the right.

    4. Click Apply.

    The Error Message column is displayed on the Audit Trails page and contains the error message for the stopped trail.

  • On the Audit Vault Agent host computer:

    1. Go to the logs directory:

      cd %agenthome%/av/logs

    2. Run the following:

      grep -i 'error|warning|fail' *

    The error messages should indicate the cause of the problem.

If the cause is still unclear, or the grep command returns no results, raise an SR with Oracle Support and include Audit Vault Agent log files.

See also document number 1566766.1 on My Oracle Support https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1566766.1.

G.19 Cannot Access the Audit Vault Server UI

Problem

The Audit Vault Server console UI is not accessible.

Solution

There are two steps you can take depending on when this problem occurs:

  • The problem occurs immediately after Audit Vault Server installation.

    In this case the installation may not have been completed correctly. Perform the installation again.

  • The problem occurs after the system is already running.

    In this case, check that the disk is not full, and that the Audit Vault Server database is running. To check that the database is running, execute this command:

    /etc/init.d/dbfwdb status

    To restart the database, use execute this command as root:

    /etc/init.d/dbfwdb start

    If you have a problem restarting the database, contact Oracle Support.

G.20 Cannot See Data for My Secured Target

Problem

Data for my Secured Target does not appear on reports.

Solution

If you cannot see the data you expect to see in the Audit Vault Server, you can troubleshoot by trying one or more of the following:

  • Confirm that Audit Vault Agent hosts are up and that the Audit Vault Agents are running.

  • Confirm that audit trails are running and that the audit trail settings match the audit configuration of the Secured Target database

    For example, the audit trail configuration in Oracle Audit Vault and Database Firewall should have the correct trail type and location.

  • Check the audit policy on the secured target to ensure you are auditing the activity that you are expecting to see in the reports.

  • Check the firewall policy to ensure you are logging the activity you are expecting to see in reports.

  • Clear any time filters on reports, and then check time settings on the secured target and on the AVS. If the time is incorrect, the time recorded against audit events will not be accurate. As a result, the audit events may not be displayed in the time window you expect.

  • Check the /var/log/messages file on Audit Vault Server and on the Database Firewall for errors.

  • Check that the enforcement point is created and running.

  • Check that the enforcement point traffic source is correct.

  • If the Database Firewall is in DAM mode, use the Database Firewall Live Capture utility to verify that traffic is being seen on the relevant traffic source. If necessary, use the File Capture utility to capture traffic to a file and verify (using Wireshark or a similar product) that the traffic being captured is consistent with the settings in the Secured Target Addresses section of your Secured Target configuration.

  • Check that you have used the correct Oracle Database service name when configuring the Secured Target Address in your Secured Target configuration.

    Also, have you included all available Oracle Service names in the Secured Target Addresses section of the Secured Target configuration? Unless you intend to define a different firewall policy for each service name, Oracle recommends you omit service name and use only IP address and TCP ports in Secured Target Addresses.

  • On the Database Firewall, check the /var/log/httpd/ssl_access_log file to confirm that the Audit Vault Server is collecting logs.

  • On the Audit Vault Server, check the /var/dbfw/tmp/processing* directories and make sure kernel*.dat files are arriving in the directory, and then being deleted once the Audit Vault Server has processed them.

  • On the Audit Vault Server, check that the mwecsvc process is running. For example, run the command:

    ps -ef | grep mwecsvc

    If the process is not running, use this command to restart it:

    service controller start

G.21 Problems Pairing Oracle Database Firewall and Oracle Audit Vault Server

Review the procedure to follow when you have problems pairing Oracle Database Firewall with Oracle Audit Vault Server.

Problem

I encounter errors when I try to associate a Database Firewall with the Audit Vault Server.

Solution

Check the following:

  • Ensure that you have entered the correct Audit Vault Server IP address in the Database Firewall Certificate page.

    Log in to the Database Firewall administration console, and then in the Security menu, click Certificate.

  • Ensure that both the Database Firewall server and the Audit Vault Server are configured to use NTP and that each machine is synced to the NTP time server.

G.22 User Names Do Not Appear on Database Firewall Reports

Problem

When I generate a Database Firewall report, I do not see user names.

Solution

Check the following possibilities:

  • If this is occurring for a Microsoft SQL Server database secured target, check to make sure that database interrogation is turned on.

  • This problem may be caused by bad network traffic arriving at the Database Firewall. Check for duplicate or missing network packets. You can use the Database Firewall's Live Capture utility to capture network traffic to a file and analyze it.

G.23 Alerts Are Not Generated

Review the resolution to use when alerts that you created are not generated.

Problem

Alerts I have created are not being generated.

Solution

Try the following:

G.24 Problems Retrieving or Provisioning Audit Settings on Oracle Secured Target

Problem

I have a problem either retrieving audit settings form an Oracle Database secured target, or provisioning audit settings to an Oracle Database secured target.

Solution

If you have problems retrieving audit settings, try the following:

  • Check the job status of the retrieval job for errors:

    Log in to the Audit Vault Server console as an auditor, click Settings, and then click Jobs in the System menu.

  • Ensure you have entered the correct connect string in the Oracle Database's secured target configuration:

    Log in to the Audit Vault Server as an administrator, click the Secured Targets tab, and then click the name of this Oracle secured target. Check the Secured Target Location field for the connect string.

If you have problems provisioning audit settings, and the Oracle Database secured target has Database Vault enabled, confirm that the Oracle Audit Vault and Database Firewall user you created on this database has the AUDIT SYSTEM and AUDIT ANY privileges.

G.25 Operation Failed Message Appears When Attempting to Enable Oracle Audit Vault and Database Firewall Policies

Learn how to resolve operation failures when you try to enable Oracle Audit Vault and Database Firewall policies.

Problem

I configured Oracle Audit Vault and Database Firewall for a backup and restore operation. After I completed the procedure, I could not enable an Oracle Audit Vault and Database Firewall policy. The error message Operation failed. Please contact Oracle Support appeared.

Solution

During the backup and restore process, Oracle Audit Vault and Database Firewall must perform a restart of the Oracle Audit Vault Server database. The internal tool Java Framework may need to be restarted. To remedy this problem:

  1. Log in to Oracle Audit Vault Server.

  2. At the command line, run the following command to check the status of the Java Framework:

    /usr/local/dbfw/bin/javafwk status
    
  3. If the output says Java framework process is stopped, then restart it as follows:

    /usr/local/dbfw/bin/javafwk start 
    

G.26 Failure While Adding Disks

If you experience disk failures when adding disks during an upgrade, then use this procedure.

Problem

Failure while adding additional disk or failure during upgrade. The symptoms include, but are not limited to:

  • Two vg_root volume groups. This results in failure during install or upgrade.

  • Hard drive devices becoming unavailable during install or upgrade. This leads to input or output errors and failure.

Solution

Ensure that any disk added to the appliance has no pre-existing LVM or other device mapper metadata. To remove any such metadata, follow these steps:

  1. Execute the following command:

    dd of=/dev/<device name> if=/dev/zero bs=1024k

    Best Practice:

    To ensure you only erase the correct drive, place it in a standalone system to execute this command. On successful completion, add the drive to the Oracle Audit Vault and Database Firewall appliance.

  2. Reboot the device.

  3. Verify the partition table and metadata.

Note:

Fiber Channel based storage with multipath is not supported in Oracle Audit Vault and Database Firewall.

G.27 Out of Memory Error Message During Restore

Problem

Encounter out of memory error while performing restore task.

Solution

Prior to initiating the restore task, ensure that the RAM size and Disk size in the new system is equal or bigger than the original system. This ensures that the out of memory error is not encountered while performing the restore task.

G.28 JAVA.IO.IOEXCEPTION Error

Problem

SSL peer shuts down incorrectly with the following error:

JAVA.IO.IOEXCEPTION: IO ERROR:SSL PEER SHUT DOWN INCORRECTLY

Solution

  1. Access the secured target through SSH.

  2. Change to the following location using the command:

    cd $ORACLE_HOME/network/admin

  3. Edit the sqlnet.ora file. Add parameter sqlnet.recv_timeout=100000 in the file.

  4. Restart the secured target listener.

  5. Once the secured target listener is started, start the agent, and the audit trail.

G.29 Failed to Start ASM Instance Error

Learn what to do when you receive a Failed to start ASM instance error.

Problem

The avdf-upgrade --confirm command aborts and results in an error. The command may fail for many reasons. The error mainly occurs due to failure in starting or stopping of a service.

The following is an example of Failed to start ASM instance error:

{{{ 
[support@avs00161e637973 ~]$ su - root 
Password: 
[root@avs00161e637973 ~]# /usr/bin/avdf-upgrade --confirm 
Please wait while validating SHA256 checksum for 
/var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso 
Checksum validation successfull for 
/var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso 
Mounting /var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso on /images 
Successfuly mounted /var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso on /images 
Starting Oracle High Availability Service 
2016-08-05 15:32:09.097: 
CLSD: Failed to generate a fullname. Additional diagnostics: ftype: 2 
(:CLSD00167:) 
CRS-4639: Could not contact Oracle High Availability Services 
CRS-4000: Command Start failed, or completed with errors. 
Starting ASM instance 
Error: Failed to start ASM Instance 
Unmounted /var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso on /images 
Failed to start ASM Instance 
}}}

Solution

Rerun the command avdf-upgrade --confirm

Executing this command again will get past the Failed to start ASM instance error.

G.30 Internal capacity exceeded messages seen in the /var/log/messages file

Problem

Not all the expected traffic is being captured or logged by the Database Firewall, and error messages are present in the /var/log/messages file containing the text Internal capacity exceeded.

Solution - 1

Increase the processing resources available for the Secured Target on which the issue is observed through the setting of the MAXIMUM_ENFORCEMENT_POINT_THREADS collection attribute.

Solution - 2

The size of the buffer used for inter-process communication on the Database Firewall can be increased to improve throughput, though at the cost of more memory being allocated by the relevant processes. Please note that this setting is in units of Megabytes, and has a default value of 16. To change the configuration for this value execute the following procedure:

  1. Log in to the Database Firewall console as the root user.

  2. Edit the file /usr/local/dbfw/etc/dbfw.conf. Look for an entry with the key IPC_PRIMARY_BUF_SIZE_MB. If it exists, this is the line to change. If it does not exist, add a new line beginning with IPC_PRIMARY_BUF_SIZE_MB.

  3. Change the IPC_PRIMARY_BUF_SIZE_MB line to reflect the required buffer size. For example, if you wished to change the buffer size to 24 megabytes, the configuration line should be IPC_PRIMARY_BUF_SIZE_MB="24". Save the changes.

  4. From the command line restart the Database Firewall processes so that the new setting is used with the command line /etc/init.d/dbfw restart.

There is also a second setting available to alter the maximum size that the inter-process communication buffer can grow to. It's units are in megabytes, and has a default value of 64 megabytes. To change the configuration for this value execute the following procedure:

  1. Log in to the Database Firewall console as the root user.

  2. Edit the file /var/dbfw/va/N/etc/appliance.conf, where N is the number of the enforcement point in question. Look for an entry with the key IPC_BUF_SIZ_MB. If it exists, this is the line to change. If it does not exist, add a new line beginning with IPC_BUF_SIZ_MB.

  3. Change the IPC_BUF_SIZ_MB to reflect the desired maximum buffer size. For example, if you wished to change the buffer size to 80 megabytes, the configuration line should be IPC_BUF_SIZ_MB="80". Save the changes.

  4. From the command line restart the Database Firewall processes so that the new setting is used with the command line /etc/init.d/dbfw restart.

If the problem persists and after altering the above settings the Internal capacity exceeded error is still encountered, then further investigation by support is required.

Perform the following:

  1. Log in to the Database Firewall console as the root user.

  2. Edit the file /usr/local/dbfw/etc/logging.conf

  3. Find the line log4j.logger.com.oracle.dbfw.Metrics=ERROR

  4. Comment out this line by placing a # character at the beginning of the line log4j.logger.com.oracle.dbfw.Metrics=ERROR. Save the changes.

  5. From the command line restart the Database Firewall processes so that the new setting is used with the command line /etc/init.d/dbfw restart

  6. Leave the Database Firewall running for several hours under load even while the Internal capacity exceeded error is still encountered.

  7. After this period, get the diagnostics output from the Database Firewall as detailed in MOS note How to Collect Diagnostic Logs From Audit Vault Server (Doc ID 2144813.1). Provide the diagnostics output to support for further analysis.

G.31 A Client Is Unable To Connect To The AVS Using SSH With A Secondary Network Interface Card

Problem

The Audit Vault Server is configured with a secondary Network Interface card for SSH connections. The secondary Network Interface card uses a gateway to access the wider network. When attempting to connect from the client to the Audit Vault Server, the connection cannot be established.

Solution

The Audit Vault Server implements spurious IP rules that prevent server connections forming on the secondary network interfaces.

This issue can be resolved by following this procedure:

  1. Diagnose the connection issue by checking the incoming packets on the Audit Vault Server. Execute the following command while attempting to connect to the Audit Vault Server with the client using SSH:

    # tcpdump -e -i any host <client IP address> and host <AVS IP address>

    Result: The following output is displayed in case the client request is being received but dropped:

    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on any, link-type LINUX_SLL (Linux cooked), capture size
    65535 bytes
       11:01:25.455535  In <Client MAC> (oui Unknown) ethertype IPv4 (0x0800), length 100: <Client IP> >
    eth1.oracle_database_firewall_internal: ICMP echo request, id 24456, seq 1, length 64
       11:01:25.455627 Out <AVS MAC> (oui Unknown) ethertype IPv4 (0x0800), length 128: eth1.oracle_database_firewall_internal >
    <Client IP>: ICMP host eth1.oracle_database_firewall_internal unreachable - admin prohibited, length 92
    
  2. In case there is no output displayed, then check the command argument. If the command is valid, then the connection is not being established to the Audit Vault Server. This indicates a wider networking problem.

    In case the connection is established, then check the IP rules. There should be no specific IP rules on the Audit Vault Server. The IP rules enabled on the system can be checked with the following command:

    # ip rule show

  3. Systems with problem display the following output:

    0:     from all lookup local
    500:   from <Eth0 Address> lookup 1
    501:   from <Eth1 Address> lookup 1
    32766: from all lookup main
    32767: from all lookup default
    
  4. To fix this problem edit the following file:

    /usr/local/dbfw/templates/template-rule-ethN

  5. Delete the following line in the file:

    from <%= @appliance_address %>/32 tab 1 priority <%= @ip_rule_base %>

  6. Execute the following command for all IP addresses presented in the output received when the command ip rule show was executed:

    ip rule del from IPADDRESS lookup 1

  7. Restart every network interface.

  8. Execute the following command to check the rule status again:

    # ip rule show

    Result: The following output is displayed:

    0:     from all lookup local
    32766: from all lookup main
    32767: from all lookup default
    
  9. SSH will now connect to the AVS.

G.32 First Archive Or Retrieve Job After Upgrade

Learn what to do if after an upgrade, the first archive or rervireve job submission displays the status of Starting.

Problem

After upgrade the first archive or retrieve job submission may display the status as Starting.

Solution

Submit the job again. This is a known issue and subsequent submission of job succeeds.

G.33 Audit Vault Agent Installation Fails After HA Pairing Or Separation

Learn what to do after the Oracle Audit Vault installation fails after an HA pairing or separation.

Problem

Installation of Audit Vault agent fails after performing pairing or separation (un-pairing) of Oracle Audit Vault server.

The following command generates agent debug logs during agent installations.

java -jar agent.jar -v

Symptoms

The following errors may be found during agent installation in the agent log file:

PKIX path validation failed

signature check failed

Solution

After the pairing or separating of Oracle Audit Vault servers, you must download the Audit Vault agent from the GUI and install the agent again after removing the existing Audit Vault Agent.

If the Audit Vault agent fails to install after pairing or separating of Audit Vault server, then install the Audit Vault agent using -v option.

To resolve the above errors, follow the steps mentioned below:

  1. Log in to the Audit Vault server as user root.

  2. Run the following script to generate a new agent.jar file.

    /usr/local/dbfw/bin/priv/update_connect_string_ip.sh

  3. Download the new agent.jar file from the GUI.

  4. Install the newly downloaded agent.jar file.

G.34 Error in Restoring Files

Learn what to do when you encounter errors while restoring files.

Problem

An attempt to restore the data files results in a failure. The restore job completes successfully, however the data files are not restored. There is no information in the restore job log file.

Solution

Check for the following to troubleshoot the issue:

  • The restore policy must follow the guidelines listed under the section Configuring Archive Locations and Retention Policies.

  • Check the tablespace that needs to be archived and the corresponding tablespace that needs to be purged as per the policy defined.

  • Restoring data into empty tablespaces is not possible. Check accordingly.

  • In case the tablespace enters the delete period, it is deleted automatically from Oracle Audit Vault Server.

  • Every tablespace is uniquely identified by the month it moves offline and the month during which it is purged. They are created automatically based on the policies that you create.

  • When the retention policy is changed, the new policy is applied to the incoming data immediately. It does not affect existing tablespaces that adhere to the old policy.

  • You can archive the tablespace when it enters the offline period.

  • After restoring the tablespace, it is online. Once it is released, it goes offline. The tablespace must be rearchived once released.

G.35 DB2 Collector Fails Due to Source Version NULL Errors

If the DB2 collector fails due to source version NULL errors, then follow these steps.

Problem

The following error or trace is displayed in the collector log file.

Caused by: java.lang.ClassNotFoundException:

sun.io.MalformedInputException

at java.net.URLClassLoader.findClass(Unknown Source)

at java.lang.ClassLoader.loadClass(Unknown Source)

Solution

Check the Java version on the host system This failure is due to Java SE version 8. Attempt to use Java SE 7.

Note:

This issue may be encountered in releases prior to 12.2.0.11.0.

G.36 DB2 Collector Fails Due To Connection or Permission Issue From Database

Problem

The following error or trace is displayed in the collector log file.

Caused by: oracle.ucp.UniversalConnectionPoolException: Cannot get Connection from Datasource: java.sql.SQLSyntaxErrorException: [Audit Vault][DB2 JDBC Driver][DB2]<User> DOES NOT HAVE PRIVILEGE TO PERFORM OPERATION EXECUTE ON THIS OBJECT NULLID.DDJC360B

Solution

Run the following command for successful execution of DB2 collector:

grant execute on package NULLID.DDJC360B to <User> (user while registering the secured target)

G.37 ORA-12660 Error While Registering Secured Target

Problem

Audit Vault agent fails with ORA-12660 error.

Solution

The server encryption is set to REQUIRED in on-premises by default. Set the server encryption to ACCEPTED or REQUESTED or REJECTED.

Note:

REJECTED is not a recommended option. The following table describes these options in detail.

Table G-1 Server Encryption Types

Option Description

ACCEPTED

The server does not allow both encrypted and non-encrypted connections. This is the default value in case the parameter is not set.

REJECTED

The server does not allow encrypted traffic.

REQUESTED

The server requests encrypted traffic if it is possible, but accepts non-encrypted traffic if encryption is not possible.

REQUIRED

The server accepts only encrypted traffic.

G.38 Failure During High Availability Pairing in Oracle Audit Vault Server

Learn what to do when high availability pairinng in Oracle Audit Vault Server fails.

Problem

There may be some errors encountered while executing high availability pairing Oracle Audit Vault Server. The errors may be in stored script memory, failure to verify some of the files in the backup set, failure to verify some of the data files, and failure to read or create files.

Solution

Check if ILM archival was run before you perform the high availability pairing in Oracle Audit Vault Server. This is due to presence of archive files in the primary server.

To avoid this, ensure that you delete archive files from the primary Oracle Audit Vault Server and later run the high availability pairing.

G.39 Audit Trail Performance Issues Occur After Audit Vault Server Upgrade

Learn what to do when audit trail performance issues occur after upgrading Oracle Audit Vault Server.

Problem

You might experience audit trail performance issues after upgrading Oracle Audit Vault Server.

Solution

The audit_trail_id_idx index that is created resolves the performance issues encountered. However, you must retain sufficient disk space if there is large amount of event data for the period prior to upgrading Oracle Audit Vault Server. The amount of disk space required is about 5% of the total event log data size.

G.40 Failures Due to Dropping Users

Learn how to resolve failures that occur when dropping users.

Problem

Failed to drop the user with an error message and the user was not listed in the Audit Vault Server GUI.

Solution

Contact Oracle Support for the best workaround and to drop the user manually using SQL*Plus.

G.41 Failure of Agent Automatic Upgrades

Learn what to do when agent automatic upgrades fail.

Problem

The automatic upgrade of the Agent fails with the following error. This is because the Agent is unable to connect to the Audit Vault Database.

Message: Exception occurred while updating Agent.
Cause: Unable to connect to AV Server.
Note: Agent will try to re-connect automatically in 10 seconds.

Solution

The Agent attempts to connect to the Audit Vault Database and auto upgrade after 10 seconds. Check the Oracle Audit Vault Database connection or contact Oracle Support.

G.42 Some Services May Not Start After Backup

Learn what to do when services fail to start after a backup.

Problem

The system may not be stable after a cold backup operation failed to complete.

Solution

Oracle recommends that you reboot the system if there is a failure while performing a cold backup operation.

G.43 Data Overflow Issues in the Oracle Audit Vault UI

Learn how to resolve data overflow issues in the Oracle Audit Vault UI.

Problem

The Recently Raised Alerts Report region appears on your dashboard and displays the list of alerts with data overflowing in the Audit Vault GUI. This may occur when you launch the GUI using Internet Explorer and the Microsoft Windows Server operating system.

Solution

To fix this issue and to display the data properly on the Audit Vault GUI, you should make minor changes to the Internet Explorer browser settings. Press F12 and click the Emulation tab.

Change the Document mode and Browser profile fields from the default settings. For example, change the Document mode value to 10 from the drop down menu and change the Browser profile field to Desktop.

G.44 Oracle Audit Vault Agent is Unreachable and the Transaction Log Audit Trail is Frozen in Starting Status

Learn what to do when the Oracle Audit Vault Agent is unreachable and the transaction log audit trail is frozen in Starting status.

Problem

The status of Oracle Audit Vault Agent is unreachable from the AV GUI. The status of the Transaction Log audit trail persistently remains in the Starting status.

This may be due to a user application that is blocking the creation of streams by ORAAUDIT user.

Symptom

The Transaction Log audit trail does not start. The following information may be found in the thread dump that is taken using jstack tool:

oracle.av.platform.agent.collfwk.impl.redo.RedoCollector.sourceSetup(RedoCollector.java:634) 

Solution

Terminate the user application that is blocking the creation of streams. Restart the Transaction Log audit trail.

G.45 Scheduled PDF or XLS Reports Result in a Hung State

To resolve a hung state that occurs for scheduled PDF or XLS reports, follow these recommendations.

Problem

Scheduled PDF or XLS reports remain incomplete for an extended period of time or ramin in q RUNNING state.

Solution

You can schedule reports to be sent to other users in PDF or XLS formats. Avoid triggering or scheduling concurrent long-running reports at the same time. Producing PDF and XLS reports occupies a lot of system resources because there is a significant amount of data involved. Scheduled concurrent long-running reports can remain in a hung state indefinitely. The reports must be scheduled with staggered intervals in between. For example, run the reports at intervals of 5, 10, or 20 minutes.

G.46 Pending Reports In Scheduled Status

Problem

Many reports are stuck in scheduled or pending status. These reports may never be completed and may be stopped.

Solution

This may be due to an issue with the Java Framework process in the background. Use these steps to check and resolve this issue:

  1. Log in to the CLI as support user.

  2. Switch to root user using the command:

    su root

  3. Switch to oracle user using the command:

    su oracle

  4. Execute the following command to check the status of the Java Framework:

    /usr/local/bin/javafwk status

  5. Execute the following commands to stop and start the Java Framework:

    /usr/local/dbfw/bin/javafwk stop

    /usr/local/dbfw/bin/javafwk start

Use the following procedure to check the status of the reports from the operating system logs after executing one of the procedures mentioned above and restarting the Java Framework:

  1. Log in to AVCLI as admin user.

  2. Execute the following command to enable diagnostics for the reports:

    ALTER SYSTEM SET loglevel=ReportLog:DEBUG|JfwkLog:DEBUG;

  3. The diagnostics can also be enabled using the Audit Vault Server console by following these steps:

    1. Log in to the console as admin user.
    2. Click Settings tab.
    3. Click on Diagnostics on the left navigation menu.
    4. Select Debug against Report Generation.
    5. Click Save.
  4. Run a PDF report. For example, Activity Overview.

    1. Log in to the Audit Vault Server console as auditor.
    2. Click Reports tab.
    3. Click Activity Reports under Built-in Reports.
    4. In the Activity Reports tab on the screen, you can schedule a report and view the generated report.
  5. After a while, check on the /var/lib/oracle/dbfw/av/log file. For example, av.report* file. It contains the PDF/XLS report generation debug logs.

G.47 The Audit Vault Logs Display A Message To Install Npcap And OpenSSL

Problem

Host Monitor is capable of collecting audit data from Windows 2016 server. A message is displayed alerting the user to install Npcap and OpenSSL.

Solution

A set of DLL files may be causing an issue. Execute the following procedure to resolve this problem:

  1. Search for the following files in the system:

    • libssl-1_1-x64.dll
    • libcrypto-1_1-x64.dll
    • wpcap.dll
    • packet.dll
  2. Append the file names with .bk format.

  3. Go to Control Panel > Uninstall Programs and uninstall OpenSSL and WinPcap.

  4. Reinstall Npcap and OpenSSL 1.1.1g. The DLL files are restored to Windows system folder.

  5. Check the Control Panel to verify that these two programs are installed.

  6. Go to C:\Windows\System32 or C:\Windows\SysWOW64 folders and search for the above four DLL files. At least one file for each DLL must be present without the .bk extension.

  7. Go to the OpenSSL installation location and search for libssl-1_1-x64.dll and libcrypto-1_1-x64.dll files. One for each type is available.

  8. Upon confirmation, add the OpenSSL installation location to the path variable.

  9. Restart the trail.

  10. If the network audit trail does not start, then check the collfwk logs present at <AgentHome>\av\log location. If the following message is available in the collfwk log, then check the Host Monitor logs present at <AgentHome>\hm\log location.

    Prerequisites are installed and present in PATH variable or System directory

    <AgentHome> refers to the Audit Vault Agent installation directory.

    Note:

    Continue with the remaining steps if your installation is 12.2.0.10.0 or before. The steps are not required for release 12.2.0.11.0 and later.
  11. If the network trail does not start and continues to throw the above error, then ensure the Windows target machine has the latest update of Visual C++ Redistributable for Visual Studio 2010 (MSVCRT.dll (*) or later) package installed. This is a must to use Host Monitor on Windows.
  12. If the following message is available in the Host Monitor log, then execute the remaining procedure:

    Invalid AVS Credentials provided
    
  13. Open the av/conf/bootstrap.prop file.

  14. Copy the following line:

    CONNECT_STRING_PARAM_POSTFIX=9999
    
  15. Paste this line in the hm/bootstrap.prop file.

  16. Restart the trail.

  17. In case the network audit trail starts without any errors, then the collection status on the Audit Vault Server console confirms the same.

  18. Go to AVAUDIT > Secured Target > Firewall Policies > Log All.

  19. Connect to the secured target database instance using SQL Developer, or any other tool.

  20. Generate the traffic for collecting data.

  21. It must be recorded in the reports of the event_log table.

G.48 Host Monitor Agent Fails to Start

Learn what to do when the Host Monitor Agent fails to start.

Problem

The Host Monitor network trail does not start after installation. The collection framework (collfwk) log file contains one of the following errors:

  • java.io.IOException: Cannot run program "<AgentHome>/hm/hostmonmanager" (in directory "<AgentHome>/hm"): error=13, The file access permissions do not allow the specified action.
    
  • HMCommandExecutor : startTrail :  binary is not found here: <AgentHome>/hm/hostmonmanager
    

Solution

This issue may arise due to insufficient privileges while starting Host Monitor. Ensure the Audit Vault Agent user belongs to the group that owns hm (Host Monitor installation) directory. Also ensure that the group that owns Host Monitor installation (hm) directory has read and execute permission on the hm directory and execute permission on hostmonmanager binary.

Note:

  • AgentHome is the Audit Vault Agent installation directory.

  • hm is the Host Monitor installation directory.

G.49 Audit Trail Stopped After Relocating Windows Event Log Files

Use this procedure when the audit trail stops after you relocate the Windows event log files.

Problem

Windows event log relocation causes audit trail to be stopped.

Solution

Follow this procedure to resolve this problem:

  1. Stop the audit trail.
  2. Drop the audit trail.
  3. Restart the audit trail. The new trail recognizes the new location for event logs.

G.50 Network Audit Trail Does Not Start on Unix Platforms

Learn the resolution when the network audit trail failst to start on Unix platforms.

Problem

Network audit trail does not start on Unix platforms.

Symptoms

  • The Oracle Audit Vault Server console displays the following error:

    Unable to start Host Monitor process

  • The collection framework log displays the following error:

    <Host Monitor home>/hostmonmanager binary is not found here

Solution

  1. Connect to the host machine on which the Audit Vault Agent and Host Monitor are installed.
  2. In the Agent Home location there is an hm symlink pointing to Host Monitor installation location.
  3. Run the following command from the Agent Home as the user who installed Oracle Audit Vault Agent:

    ls -lrt hm

  4. Check if it is possible to list the contents of Host Monitor install directory.
  5. Check the permission of all directories in the hierarchy of the path under which Host Monitor is installed.

    Note:

    The entire directory hierarchy must be owned by the root user. All of the directories in this hierarchy must have read and execute permission for other users or groups, but not write permission.
  6. Grant the necessary permissions as stated above.
  7. Restart the network audit trail.

G.51 Audit Vault Agent in Unreachable state upon Failover

Problem

Audit Vault Agent goes into Unreachable state in the event of a failover of the Audit Vault Server or reboot of the primary Audit Vault Server.

Symptom

Audit Vault Agent and audit trails go into Unreachable state.

Solution

  1. For the Primary Audit Vault Server reboot instance - Restart the Oracle Database Listener process on the standby Audit Vault Server.

  2. For the failover instance, restart the Oracle Database Listener process on the primary (old) Audit Vault Server. This is the Audit Vault Server which was primary prior to failover.

G.52 Unable to Reach Gateway Error

Learn to fix incorrect Gateway details entered during installation.

Problem

Incorrect or invalid Gateway details entered while installing Audit Vault Sever or Database Firewall. The following error message may be encountered:

Gateway is not reachable from host

Solution

The Gateway details can to be corrected by following these steps:

  1. Log in to Terminal-1 as root user. Alternately, Terminal-1 can be accessed by pressing Ctrl+Alt+Right Arrow Key.
  2. Access and open the dbfw.conf file by executing this command:
    vi /usr/local/dbfw/etc/dbfw.conf
    
  3. Set the correct value for the GATEWAY field by overwriting the existing value.
  4. Save and close the file.
  5. Execute the command to apply the modified value:
    /usr/local/dbfw/bin/priv/configure-networking
    
  6. Return back to the appliance screen by pressing Ctrl+Alt+Left Arrow Key.

Note:

The network settings entered during installation can be modified, by choosing the Change IP Settings option in the installer or appliance screen.