Learn how to upgrade the Audit Vault Agent or Host Monitor Agent manually.
Problem
After upgrading to Oracle AVDF 12.2.0.13.0 or later, some of the Audit Vault Agents or Host Monitor Agents are not upgraded.
Symptom - 1
Audit Vault Agent is in STOPPED state after Audit Vault Server upgrade.
Symptom - 2
Host Monitor Agent is in NEEDS UPGRADE or UPDATE FAILED state after Audit Vault Server upgrade.
Solution - 1
The symptom indicates that the Audit Vault Agent has failed to auto upgrade during the Audit Vault Server upgrade. Execute the following steps as the user who installed Agent previously:
agent.jar file and the Agent folder from the host machine.agent.jar file from the upgraded Audit Vault Server.Execute the following command:
java -jar agent.jar [-d <AgentHome>]
RUNNING state.Solution - 2
The symptom indicates that the Host Monitor Agent has failed to auto upgrade during the Audit Vault Server upgrade. Execute the following steps as root user:
hostmonitor, hmdeployer, or hostmonmanager processes currently running.hm where the Host Monitor is installed.Execute the following command to uninstall the Host Monitor:
./hm/hostmonsetup uninstall
hm directory.Execute the following command to reinstall the Host Monitor in a root owned location:
./hostmonsetup install
Learn how to fix disabled archiving functionality post upgrade from Oracle AVDF 12.2.0.11.0 to later releases (12.2.0.12.0 or 12.2.0.13.0).
Problem
Archiving functionality may be disabled after upgrading from Oracle AVDF 12.2.0.11.0 or 12.2.0.12.0, to 12.2.0.12.0 or 12.2.0.13.0.
Archiving functionality for high availability environment is supported starting Oracle AVDF release 12.2.0.11.0. This problem arises when you are upgrading from older releases where archiving functionality is supported only on the primary Audit Vault Server. Execute the steps only if you are upgrading from Oracle AVDF 12.2.0.11.0 to later releases.
Note:
If you are upgrading from any release prior to Oracle AVDF 12.2.0.11.0, then follow the steps documented in section Enable Archiving Functionality Post Upgrade.Solution
If archive locations were present before upgrading to 12.2.0.13.0, execute the following steps to enable archiving functionality. These steps must be executed post upgrade process.
Create a new archive location using the Audit Vault Server console. While creating this new archive location enter the details of both the primary and standby location. This will mount the new archive location on the primary or standby Audit Vault Server and update the fstab with both archive locations.
SSH to the primary Audit Vault Server as support user and then unlock the avsys user by executing the following commands:
su root
su dvaccountmgr
sqlplus /
alter user avsys identified by <new password for avsys user> account unlock;
exit;
exit;
Execute the following commands as avsys user:
su oracle
sqlplus avsys
Enter the avsys password when prompted. Execute the following SQL commands:
delete from avsys.system_configuration where property = '_ILM_ARCHIVING_DISABLED';
COMMIT;
insert into avsys.system_configuration values ('_ILM_HA_UPGRADE_COMPLETED', 'Y');
COMMIT;
exit;
exit;
Execute the following command to lock the avsys user:
su dvaccountmgr
sqlplus /
alter user avsys account lock;
Delete the new archive location that was created in the initial step. Navigate to Settings tab, then click Manage Archive Locations in the left navigation menu. Delete the specific archive location.
Problem
I see no traffic, or only partial traffic, captured in reports for an Oracle Database monitored by the Database Firewall.
Solutions
Go through the following checks to find the trouble:
In the Audit Vault Server, check that the report filters are set correctly, including the time slot.
Check that the system time on the Database Firewall is synchronized with the time on the Audit Vault Server and the secured target system.
Check that the secured target's network traffic is visible to the Database Firewall using the Live Capture utility on the firewall.
Check that the Oracle Database service name or SID is used correctly. If you specified an Oracle Database service name in the Enforcement Point settings for this secured target, you will only see traffic for that service name. To see all traffic, remove the service name from the Enforcement Point settings to see all traffic.
If you have entered a service name in the Enforcement Point, and see no traffic, check to see that the service name is entered correctly in the Enforcement Point settings.
For Enforcement Points set to use DAM mode, the Database Firewall may be monitoring traffic for existing client connections to the database. Since these connections were in place before you deployed the Database Firewall, it will not be able to detect the service name you specify in the Enforcement Point. In this case, restart the client connections to the database.
Check that the correct Database Firewall policy is deployed.
See Also:
Oracle Audit Vault and Database Firewall Auditor's Guide for information on editing and deploying firewall policies.
Configuring Enforcement Points for information on Enforcement Points.
Read the troubleshooting advice if you receive a 'host is not registered' error.
Problem
I used the following two commands to register the Oracle Audit Vault Agent's host computer (where the agent is deployed), and to request Audit Vault Agent activation:
From the Audit Vault Server:
avcli> register host 'host_name'
From the host computer:
agentctl activate
But the agentctl activate command returns: Agent host is not registered
Solution
Your agent host may be multi homed. In this case, the agent hostname to IP address resolution may resolve to the NIC/IP that is not used by the agent while connecting to the AV server. To resolve this issue, try to register the agent host using the with ip option and then try activating the agent again.
From the Audit Vault Server, use the following command:
avcli> register host 'host_name' with ip 'host_ip_address'
If you still have issues, try finding the IP address used in the database session when you connect to the Audit Vault server from the agent host, using these commands:
Start SQL*Plus connection as sqlplus /nolog without the username or password.
In SQL*Plus execute the command: connect <user>. Enter the password when prompted.
sqlplus username/password@"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=Audit_Vault_Server_IP)(PORT=1521))(CONNECT_DATA= (SERVICE_NAME=dbfwdb)))"
sqlplus> select SYS_CONTEXT('USERENV','IP_ADDRESS') from dual;Use the IP address from the above query to register your host.
Learn the resolution if you are unable to deploy an agent on a secondary Oracle Audit Vault server.
Problem
When I try to deploy the Audit Vault Agent on the secondary Audit Vault Server in a high availability pair, I get an error that the host is not registered.
Cause
After you pair two Audit Vault Servers for high availability, you do all configuration on the primary server in the pair only, including Audit Vault Agent deployment.
Problem
This problem may manifest with various symptoms:
Solution
Unset all environment variables except the following:
PATH
TERM
PS1
LANG
LC_*
JAVA_HOME
Then run the java -jar agent.jar command again on the host machine.
If you deployed the Audit Vault Agent in a Linux environment, ensure that the host machine name is present in the /etc/hosts file.
Problem
The command java -jar agent.jar failed on my Windows secured target machine, and I noticed in the log files that the Audit Vault Agent services installation/un-installation failed.
Solution
Follow the instructions for unregistering the agent in Registering and Unregistering the Audit Vault Agent as a Windows Service.
If Method 1 fails, then try Method 2.
Run the java -jar agent.jar command again.
agent.jar FileDetermine the steps to perform if you are unable to install the agent or generate the agent.jar file.
Problem
Unable to install the Audit Vault Agent. Attempts to regenerate the agent.jar file are also unsuccessful.
Solution
Follow these steps to regenerate the agent.jar file:
Log in to the Audit Vault Server through SSH as user oracle.
Go to the directory /var/lib/oracle/dbfw/av/conf/ location.
Delete the bootstrap.prop file.
Execute the following command:
/var/lib/oracle/dbfw/bin/avca configure_bootstrap
Check the avca.log file that is available at /var/lib/oracle/dbfw/av/log/ to check if the above command was executed successfully.
Switch the user (su) to avsys.
Run the following query:
select agent_gen_ts from file_repos where file_name='agent.jar';
The above query displays the current time in case the agent.jar file is generated successfully.
Review the troubleshooting advice if you are unable to un-install the Oracle Audit Vault Agent Windows Service.
Follow the instructions for unregistering the Agent inRegistering and Unregistering the Audit Vault Agent as a Windows Service.
If Method 1 fails, then try Method 2.
Problem
I got an error during installation of the Audit Vault Agent on Windows, and I noticed the following error in the AGENT_HOME\av\log\av.agent.prunsvr log file:
[2013-05-02 11:55:53] [info] Commons Daemon procrun (1.0.6.0 32-bit) started [2013-05-02 11:55:53] [error] Unable to open the Service Manager [2013-05-02 11:55:53] [error] Access is denied. [2013-05-02 11:55:53] [error] Commons Daemon procrun failed with exit value: 7 (Failed to ) [2013-05-02 11:55:53] [error] Access is denied.
Solution
The above message means that the logged in user does not have privileges to install the Audit Vault Agent as a Windows Service. If you get the above message, try launching the command shell with the Run As Administrator option, and then execute java -jar agent.jar in that command shell.
Problem
Installed the Audit Vault Agent using the java -jar agent.jar command.
Activated the Audit Vault Agent.
Started the Audit Vault Agent using the agentctl start -k key command.
The agent started up and is in RUNNING state.
Stopped the Audit Vault Agent.
Tried to start the Audit Vault Agent using the Services Applet on the Windows Control Panel.
The Audit Vault Agent errored out immediately.
Solution
This means that the Audit Vault Agent is configured to use a Windows account that does not have privileges to connect to the Audit Vault Server.
Take the following steps:
Go to Control Panel, then to Services Applet.
Select the Oracle Audit Vault Agent service.
Right click and select the Properties menu.
Click the Log on tab.
Select This account: and then enter a valid account name and password.
Save and exit.
Start the Audit Vault Agent through the Services Applet.
Problem
After I installed the Audit Vault Agent, I set the username and password in the OracleAVAgent Windows Service Properties Log On tab. However, when I try to start the OracleAVAgent service, I see the following error in the Agent_Home\av\log\av.agent.prunsvr.date.log file:
[info] Commons Daemon procrun (1.0.6.0 32-bit) started [info] Running 'OracleAVAgent' Service... [info] Starting service... [error] Failed creating java [error] ServiceStart returned 1 [info] Run service finished. [info] Commons Daemon procrun finished
Solution
This means that the OracleAVAgent service is not able to launch the Java process. Try the following:
Uninstall all JDKs and/or JREs in the system.
Reinstall JDK SE or JRE and then start the OracleAVAgent service.
If this doesn't help, you can install 32 bit JDK SE or JRE and then start the OracleAVAgent service.
Problem
I am setting up a Host Monitor. When I run the command bin/hostmonsetup install, the following error is displayed:
[root@dbsec1 av]# bin/hostmonsetup install /usr/bin/ld: cannot find -lpcap collect2: ld returned 1 exit status make: *** [hostmonitor] Error 1 Line 105: Failed to generate executables for Host monitor.
Solution
This means the host computer does not have the required libraries for the Host Monitor. Install the required libraries listed in Host Monitor Requirements.
Problem
I configured an Oracle Database secured target to audit to XML files, configured an audit trail in Oracle AVDF of type DIRECTORY, and then configured an alert to trigger on certain events. My alert did not get triggered for a long time.
Solution
This issue can occur if the Oracle Database secured target is not flushing the audit records to the file immediately. Contact Oracle Support in order to access support note 1358183.1 Audit Files Are Not Immediately Flushed To Disk.
Resolve errors that can occur when you create an audit policy.
Problem
I received this error message when I tried to create a new audit policy setting for Oracle Database:
-ORA-01400: cannot insert NULL into ("AVSYS"."AUDIT_SETTING_ARCHIVE_MAP"."ARCHIVE_ID")
Cause
The Oracle Database must have at least one audit policy setting before you can create and provision new audit settings using Oracle Audit Vault and Database Firewall. Oracle Database comes with a predefined set of audit policy settings. You must not manually remove these settings. If the audit settings have been removed, then you can manually create at least one audit setting in the Oracle Database. Then try again to create new audit settings using Oracle Audit Vault and Database Firewall.
See Also:
Oracle Database Security Guide for detailed information on Oracle Database auditing.
Problem
In DPE (blocking) mode, my client application cannot connect to the secured target database.
Solution 1
Log in as root on the Database Firewall server.
Execute this command, using the secured target database IP address or host name:
ping -I secured_target_ip_address_or_hostname
If no response is received, check that:
The bridge IP settings are correct.
The bridge IP address is on the same subnet as the secured target database.
DNS is configured on the Database Firewall
If a response is received, check:
The firewall policy to ensure it is not blocking the connection attempt.
The client connection settings to ensure that the client is attempting to connect to the correct secured target database.
Solution 2
If your client application computer is on a different subnet than the secured target database, see document number 1566766.1 on My Oracle Support https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1566766.1.
Problem
An audit trail does not start. For example, in the Audit Vault Server console, in the Audit Trails page, the Collection Status column indicates that the trail is Stopped or Unreachable.
Solution
When a trail does not start, you can show the associated error in two ways:
In the Audit Vault Server console:
Click the Secured Targets tab, and then from the Monitoring menu, click Audit Trails.
Click the Actions button, and then click Select Columns.
From the left-hand box, double-click Error Message so that it moves into the Display in Report box on the right.
Click Apply.
The Error Message column is displayed on the Audit Trails page and contains the error message for the stopped trail.
On the Audit Vault Agent host computer:
Go to the logs directory:
cd %agenthome%/av/logs
Run the following:
grep -i 'error|warning|fail' *
The error messages should indicate the cause of the problem.
If the cause is still unclear, or the grep command returns no results, raise an SR with Oracle Support and include Audit Vault Agent log files.
See also document number 1566766.1 on My Oracle Support https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1566766.1.
Problem
The Audit Vault Server console UI is not accessible.
Solution
There are two steps you can take depending on when this problem occurs:
The problem occurs immediately after Audit Vault Server installation.
In this case the installation may not have been completed correctly. Perform the installation again.
The problem occurs after the system is already running.
In this case, check that the disk is not full, and that the Audit Vault Server database is running. To check that the database is running, execute this command:
/etc/init.d/dbfwdb status
To restart the database, use execute this command as root:
/etc/init.d/dbfwdb start
If you have a problem restarting the database, contact Oracle Support.
Problem
Data for my Secured Target does not appear on reports.
Solution
If you cannot see the data you expect to see in the Audit Vault Server, you can troubleshoot by trying one or more of the following:
Confirm that Audit Vault Agent hosts are up and that the Audit Vault Agents are running.
Confirm that audit trails are running and that the audit trail settings match the audit configuration of the Secured Target database
For example, the audit trail configuration in Oracle Audit Vault and Database Firewall should have the correct trail type and location.
Check the audit policy on the secured target to ensure you are auditing the activity that you are expecting to see in the reports.
Check the firewall policy to ensure you are logging the activity you are expecting to see in reports.
Clear any time filters on reports, and then check time settings on the secured target and on the AVS. If the time is incorrect, the time recorded against audit events will not be accurate. As a result, the audit events may not be displayed in the time window you expect.
Check the /var/log/messages file on Audit Vault Server and on the Database Firewall for errors.
Check that the enforcement point is created and running.
Check that the enforcement point traffic source is correct.
If the Database Firewall is in DAM mode, use the Database Firewall Live Capture utility to verify that traffic is being seen on the relevant traffic source. If necessary, use the File Capture utility to capture traffic to a file and verify (using Wireshark or a similar product) that the traffic being captured is consistent with the settings in the Secured Target Addresses section of your Secured Target configuration.
Check that you have used the correct Oracle Database service name when configuring the Secured Target Address in your Secured Target configuration.
Also, have you included all available Oracle Service names in the Secured Target Addresses section of the Secured Target configuration? Unless you intend to define a different firewall policy for each service name, Oracle recommends you omit service name and use only IP address and TCP ports in Secured Target Addresses.
On the Database Firewall, check the /var/log/httpd/ssl_access_log file to confirm that the Audit Vault Server is collecting logs.
On the Audit Vault Server, check the /var/dbfw/tmp/processing* directories and make sure kernel*.dat files are arriving in the directory, and then being deleted once the Audit Vault Server has processed them.
On the Audit Vault Server, check that the mwecsvc process is running. For example, run the command:
ps -ef | grep mwecsvc
If the process is not running, use this command to restart it:
service controller start
Review the procedure to follow when you have problems pairing Oracle Database Firewall with Oracle Audit Vault Server.
Problem
I encounter errors when I try to associate a Database Firewall with the Audit Vault Server.
Solution
Check the following:
Ensure that you have entered the correct Audit Vault Server IP address in the Database Firewall Certificate page.
Log in to the Database Firewall administration console, and then in the Security menu, click Certificate.
Ensure that both the Database Firewall server and the Audit Vault Server are configured to use NTP and that each machine is synced to the NTP time server.
Problem
When I generate a Database Firewall report, I do not see user names.
Solution
Check the following possibilities:
If this is occurring for a Microsoft SQL Server database secured target, check to make sure that database interrogation is turned on.
This problem may be caused by bad network traffic arriving at the Database Firewall. Check for duplicate or missing network packets. You can use the Database Firewall's Live Capture utility to capture network traffic to a file and analyze it.
Review the resolution to use when alerts that you created are not generated.
Problem
Alerts I have created are not being generated.
Solution
Try the following:
Examine the alert condition to make sure it is written correctly:
Log in to the Audit Vault Server console as an auditor, click the Policy tab, click Alerts, and then click the name of the alert in question.
See Also:
Oracle Audit Vault and Database Firewall Auditor's Guide for help in writing alert conditions.
Logging in to the Audit Vault Server Console UI for more information about logging in to the Audit Vault Server console.
Restart job framework on the Audit Vault Server. See document 1592181.1 on My Oracle Support https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1592181.1.
Problem
I have a problem either retrieving audit settings form an Oracle Database secured target, or provisioning audit settings to an Oracle Database secured target.
Solution
If you have problems retrieving audit settings, try the following:
Check the job status of the retrieval job for errors:
Log in to the Audit Vault Server console as an auditor, click Settings, and then click Jobs in the System menu.
Ensure you have entered the correct connect string in the Oracle Database's secured target configuration:
Log in to the Audit Vault Server as an administrator, click the Secured Targets tab, and then click the name of this Oracle secured target. Check the Secured Target Location field for the connect string.
If you have problems provisioning audit settings, and the Oracle Database secured target has Database Vault enabled, confirm that the Oracle Audit Vault and Database Firewall user you created on this database has the AUDIT SYSTEM and AUDIT ANY privileges.
Learn how to resolve operation failures when you try to enable Oracle Audit Vault and Database Firewall policies.
Problem
I configured Oracle Audit Vault and Database Firewall for a backup and restore operation. After I completed the procedure, I could not enable an Oracle Audit Vault and Database Firewall policy. The error message Operation failed. Please contact Oracle Support appeared.
Solution
During the backup and restore process, Oracle Audit Vault and Database Firewall must perform a restart of the Oracle Audit Vault Server database. The internal tool Java Framework may need to be restarted. To remedy this problem:
Log in to Oracle Audit Vault Server.
At the command line, run the following command to check the status of the Java Framework:
/usr/local/dbfw/bin/javafwk status
If the output says Java framework process is stopped, then restart it as follows:
/usr/local/dbfw/bin/javafwk start
If you experience disk failures when adding disks during an upgrade, then use this procedure.
Problem
Failure while adding additional disk or failure during upgrade. The symptoms include, but are not limited to:
Two vg_root volume groups. This results in failure during install or upgrade.
Hard drive devices becoming unavailable during install or upgrade. This leads to input or output errors and failure.
Solution
Ensure that any disk added to the appliance has no pre-existing LVM or other device mapper metadata. To remove any such metadata, follow these steps:
Execute the following command:
dd of=/dev/<device name> if=/dev/zero bs=1024k
Best Practice:
To ensure you only erase the correct drive, place it in a standalone system to execute this command. On successful completion, add the drive to the Oracle Audit Vault and Database Firewall appliance.
Reboot the device.
Verify the partition table and metadata.
Note:
Fiber Channel based storage with multipath is not supported in Oracle Audit Vault and Database Firewall.
Problem
Encounter out of memory error while performing restore task.
Solution
Prior to initiating the restore task, ensure that the RAM size and Disk size in the new system is equal or bigger than the original system. This ensures that the out of memory error is not encountered while performing the restore task.
Problem
SSL peer shuts down incorrectly with the following error:
JAVA.IO.IOEXCEPTION: IO ERROR:SSL PEER SHUT DOWN INCORRECTLY
Solution
Access the secured target through SSH.
Change to the following location using the command:
cd $ORACLE_HOME/network/admin
Edit the sqlnet.ora file. Add parameter sqlnet.recv_timeout=100000 in the file.
Restart the secured target listener.
Once the secured target listener is started, start the agent, and the audit trail.
Learn what to do when you receive a Failed to start ASM instance error.
Problem
The avdf-upgrade --confirm command stops and results in an error. The command may fail for many reasons. The error mainly occurs due to failure in starting or stopping of a service.
The following is an example of Failed to start ASM instance error:
{{{
[support@avs00161e637973 ~]$ su - root
Password:
[root@avs00161e637973 ~]# /usr/bin/avdf-upgrade --confirm
Please wait while validating SHA256 checksum for
/var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso
Checksum validation successfull for
/var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso
Mounting /var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso on /images
Successfuly mounted /var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso on /images
Starting Oracle High Availability Service
2016-08-05 15:32:09.097:
CLSD: Failed to generate a fullname. Additional diagnostics: ftype: 2
(:CLSD00167:)
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Start failed, or completed with errors.
Starting ASM instance
Error: Failed to start ASM Instance
Unmounted /var/dbfw/upgrade/avdf-upgrade-12.2.0.3.0.iso on /images
Failed to start ASM Instance
}}}Solution
Rerun the command avdf-upgrade --confirm
Executing this command again will get past the Failed to start ASM instance error.
Problem
Not all the expected traffic is being captured or logged by the Database Firewall, and error messages are present in the /var/log/messages file containing the text Internal capacity exceeded.
Solution - 1
Increase the processing resources available for the Secured Target on which the issue is observed through the setting of the MAXIMUM_ENFORCEMENT_POINT_THREADS collection attribute.
See Also:
Solution - 2
The size of the buffer used for inter-process communication on the Database Firewall can be increased to improve throughput, though at the cost of more memory being allocated by the relevant processes. Please note that this setting is in units of Megabytes, and has a default value of 16. To change the configuration for this value execute the following procedure:
Log in to the Database Firewall console as the root user.
Edit the file /usr/local/dbfw/etc/dbfw.conf. Look for an entry with the key IPC_PRIMARY_BUF_SIZE_MB. If it exists, this is the line to change. If it does not exist, add a new line beginning with IPC_PRIMARY_BUF_SIZE_MB.
Change the IPC_PRIMARY_BUF_SIZE_MB line to reflect the required buffer size. For example, if you wished to change the buffer size to 24 megabytes, the configuration line should be IPC_PRIMARY_BUF_SIZE_MB="24". Save the changes.
From the command line restart the Database Firewall processes so that the new setting is used with the command line /etc/init.d/dbfw restart.
There is also a second setting available to alter the maximum size that the inter-process communication buffer can grow to. It's units are in megabytes, and has a default value of 64 megabytes. To change the configuration for this value execute the following procedure:
Log in to the Database Firewall console as the root user.
Edit the file /var/dbfw/va/N/etc/appliance.conf, where N is the number of the enforcement point in question. Look for an entry with the key IPC_BUF_SIZ_MB. If it exists, this is the line to change. If it does not exist, add a new line beginning with IPC_BUF_SIZ_MB.
Change the IPC_BUF_SIZ_MB to reflect the desired maximum buffer size. For example, if you wished to change the buffer size to 80 megabytes, the configuration line should be IPC_BUF_SIZ_MB="80". Save the changes.
From the command line restart the Database Firewall processes so that the new setting is used with the command line /etc/init.d/dbfw restart.
If the problem persists and after altering the above settings the Internal capacity exceeded error is still encountered, then further investigation by support is required.
Perform the following:
Log in to the Database Firewall console as the root user.
Edit the file /usr/local/dbfw/etc/logging.conf
Find the line log4j.logger.com.oracle.dbfw.Metrics=ERROR
Comment out this line by placing a # character at the beginning of the line log4j.logger.com.oracle.dbfw.Metrics=ERROR. Save the changes.
From the command line restart the Database Firewall processes so that the new setting is used with the command line /etc/init.d/dbfw restart
Leave the Database Firewall running for several hours under load even while the Internal capacity exceeded error is still encountered.
After this period, get the diagnostics output from the Database Firewall as detailed in MOS note How to Collect Diagnostic Logs From Audit Vault Server (Doc ID 2144813.1). Provide the diagnostics output to support for further analysis.
Problem
The Audit Vault Server is configured with a secondary Network Interface card for SSH connections. The secondary Network Interface card uses a gateway to access the wider network. When attempting to connect from the client to the Audit Vault Server, the connection cannot be established.
Solution
The Audit Vault Server implements spurious IP rules that prevent server connections forming on the secondary network interfaces.
This issue can be resolved by following this procedure:
Diagnose the connection issue by checking the incoming packets on the Audit Vault Server. Execute the following command while attempting to connect to the Audit Vault Server with the client using SSH:
# tcpdump -e -i any host <client IP address> and host <AVS IP address>
Result: The following output is displayed in case the client request is being received but dropped:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes 11:01:25.455535 In <Client MAC> (oui Unknown) ethertype IPv4 (0x0800), length 100: <Client IP> > eth1.oracle_database_firewall_internal: ICMP echo request, id 24456, seq 1, length 64 11:01:25.455627 Out <AVS MAC> (oui Unknown) ethertype IPv4 (0x0800), length 128: eth1.oracle_database_firewall_internal > <Client IP>: ICMP host eth1.oracle_database_firewall_internal unreachable - admin prohibited, length 92
In case there is no output displayed, then check the command argument. If the command is valid, then the connection is not being established to the Audit Vault Server. This indicates a wider networking problem.
In case the connection is established, then check the IP rules. There should be no specific IP rules on the Audit Vault Server. The IP rules enabled on the system can be checked with the following command:
# ip rule show
Systems with problem display the following output:
0: from all lookup local 500: from <Eth0 Address> lookup 1 501: from <Eth1 Address> lookup 1 32766: from all lookup main 32767: from all lookup default
To fix this problem edit the following file:
/usr/local/dbfw/templates/template-rule-ethN
Delete the following line in the file:
from <%= @appliance_address %>/32 tab 1 priority <%= @ip_rule_base %>
Execute the following command for all IP addresses presented in the output received when the command ip rule show was executed:
ip rule del from IPADDRESS lookup 1
Restart every network interface.
Execute the following command to check the rule status again:
# ip rule show
Result: The following output is displayed:
0: from all lookup local 32766: from all lookup main 32767: from all lookup default
SSH will now connect to the AVS.
Learn what to do if after an upgrade, the first archive or rervireve job submission displays the status of Starting.
Problem
After upgrade the first archive or retrieve job submission may display the status as Starting.
Solution
Submit the job again. This is a known issue and subsequent submission of job succeeds.
Learn what to do after the Oracle Audit Vault installation fails after an HA pairing or separation.
Problem
Installation of Audit Vault agent fails after performing pairing or separation (un-pairing) of Oracle Audit Vault server.
The following command generates agent debug logs during agent installations.
java -jar agent.jar -v
Symptoms
The following errors may be found during agent installation in the agent log file:
PKIX path validation failed
signature check failed
Solution
After the pairing or separating of Oracle Audit Vault servers, you must download the Audit Vault agent from the GUI and install the agent again after removing the existing Audit Vault Agent.
If the Audit Vault agent fails to install after pairing or separating of Audit Vault server, then install the Audit Vault agent using -v option.
To resolve the above errors, follow the steps mentioned below:
Log in to the Audit Vault server as user root.
Run the following script to generate a new agent.jar file.
/usr/local/dbfw/bin/priv/update_connect_string_ip.sh
Download the new agent.jar file from the GUI.
Install the newly downloaded agent.jar file.
Learn what to do when you encounter errors while restoring files.
Problem
An attempt to restore the data files results in a failure. The restore job completes successfully, however the data files are not restored. There is no information in the restore job log file.
Solution
Check for the following to troubleshoot the issue:
The restore policy must follow the guidelines listed under the section Configuring Archive Locations and Retention Policies.
Check the tablespace that needs to be archived and the corresponding tablespace that needs to be purged as per the policy defined.
Restoring data into empty tablespaces is not possible. Check accordingly.
In case the tablespace enters the delete period, it is deleted automatically from Oracle Audit Vault Server.
Every tablespace is uniquely identified by the month it moves offline and the month during which it is purged. They are created automatically based on the policies that you create.
When the retention policy is changed, the new policy is applied to the incoming data immediately. It does not affect existing tablespaces that adhere to the old policy.
You can archive the tablespace when it enters the offline period.
After restoring the tablespace, it is online. Once it is released, it goes offline. The tablespace must be rearchived once released.
If the DB2 collector fails due to source version NULL errors, then follow these steps.
Problem
The following error or trace is displayed in the collector log file.
Caused by: java.lang.ClassNotFoundException:
sun.io.MalformedInputException
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
Solution
Check the Java version on the host system This failure is due to Java SE version 8. Attempt to use Java SE 7.
Note:
This issue may be encountered in releases prior to12.2.0.11.0.Problem
The following error or trace is displayed in the collector log file.
Caused by: oracle.ucp.UniversalConnectionPoolException: Cannot get Connection from Datasource: java.sql.SQLSyntaxErrorException: [Audit Vault][DB2 JDBC Driver][DB2]<User> DOES NOT HAVE PRIVILEGE TO PERFORM OPERATION EXECUTE ON THIS OBJECT NULLID.DDJC360B
Solution
Run the following command for successful execution of DB2 collector:
grant execute on package NULLID.DDJC360B to <User> (user while registering the secured target)
Problem
Audit Vault agent fails with ORA-12660 error.
Solution
The server encryption is set to REQUIRED in on-premises by default. Set the server encryption to ACCEPTED or REQUESTED or REJECTED.
Note:
REJECTED is not a recommended option. The following table describes these options in detail.
Table G-1 Server Encryption Types
| Option | Description |
|---|---|
|
|
The server does not allow both encrypted and non-encrypted connections. This is the default value in case the parameter is not set. |
|
|
The server does not allow encrypted traffic. |
|
|
The server requests encrypted traffic if it is possible, but accepts non-encrypted traffic if encryption is not possible. |
|
|
The server accepts only encrypted traffic. |
Learn what to do when high availability pairinng in Oracle Audit Vault Server fails.
Problem
There may be some errors encountered while executing high availability pairing Oracle Audit Vault Server. The errors may be in stored script memory, failure to verify some of the files in the backup set, failure to verify some of the data files, and failure to read or create files.
Solution
Check if ILM archival was run before you perform the high availability pairing in Oracle Audit Vault Server. This is due to presence of archive files in the primary server.
To avoid this, ensure that you delete archive files from the primary Oracle Audit Vault Server and later run the high availability pairing.
Learn what to do when audit trail performance issues occur after upgrading Oracle Audit Vault Server.
Problem
You might experience audit trail performance issues after upgrading Oracle Audit Vault Server.
Solution
The audit_trail_id_idx index that is created resolves the performance issues encountered. However, you must retain sufficient disk space if there is large amount of event data for the period prior to upgrading Oracle Audit Vault Server. The amount of disk space required is about 5% of the total event log data size.
Learn how to resolve failures that occur when dropping users.
Problem
Failed to drop the user with an error message and the user was not listed in the Audit Vault Server GUI.
Solution
Contact Oracle Support for the best workaround and to drop the user manually using SQL*Plus.
Learn what to do when agent automatic upgrades fail.
Problem
The automatic upgrade of the Agent fails with the following error. This is because the Agent is unable to connect to the Audit Vault Database.
Message: Exception occurred while updating Agent. Cause: Unable to connect to AV Server. Note: Agent will try to re-connect automatically in 10 seconds.
Solution
The Agent attempts to connect to the Audit Vault Database and auto upgrade after 10 seconds. Check the Oracle Audit Vault Database connection or contact Oracle Support.
Learn what to do when services fail to start after a backup.
Problem
The system may not be stable after a cold backup operation failed to complete.
Solution
Oracle recommends that you reboot the system if there is a failure while performing a cold backup operation.
Learn how to resolve data overflow issues in the Oracle Audit Vault UI.
Problem
The Recently Raised Alerts Report region appears on your dashboard and displays the list of alerts with data overflowing in the Audit Vault GUI. This may occur when you launch the GUI using Internet Explorer and the Microsoft Windows Server operating system.
Solution
To fix this issue and to display the data properly on the Audit Vault GUI, you should make minor changes to the Internet Explorer browser settings. Press F12 and click the Emulation tab.
Change the Document mode and Browser profile fields from the default settings. For example, change the Document mode value to 10 from the drop down menu and change the Browser profile field to Desktop.
Learn what to do when the Oracle Audit Vault Agent is unreachable and the transaction log audit trail is frozen in Starting status.
Problem
The status of Oracle Audit Vault Agent is unreachable from the AV GUI. The status of the Transaction Log audit trail persistently remains in the Starting status.
This may be due to a user application that is blocking the creation of streams by ORAAUDIT user.
Symptom
The Transaction Log audit trail does not start. The following information may be found in the thread dump that is taken using jstack tool:
oracle.av.platform.agent.collfwk.impl.redo.RedoCollector.sourceSetup(RedoCollector.java:634)
Solution
Terminate the user application that is blocking the creation of streams. Restart the Transaction Log audit trail.
To resolve a hung state that occurs for scheduled PDF or XLS reports, follow these recommendations.
Problem
Scheduled PDF or XLS reports remain incomplete for an extended period of time or ramin in q RUNNING state.
Solution
You can schedule reports to be sent to other users in PDF or XLS formats. Avoid triggering or scheduling concurrent long-running reports at the same time. Producing PDF and XLS reports occupies a lot of system resources because there is a significant amount of data involved. Scheduled concurrent long-running reports can remain in a hung state indefinitely. The reports must be scheduled with staggered intervals in between. For example, run the reports at intervals of 5, 10, or 20 minutes.
Problem
Many reports are stuck in scheduled or pending status. These reports may never be completed and may be stopped.
Solution
This may be due to an issue with the Java Framework process in the background. Use these steps to check and resolve this issue:
Log in to the CLI as support user.
Switch to root user using the command:
su root
Switch to oracle user using the command:
su oracle
Execute the following command to check the status of the Java Framework:
/usr/local/bin/javafwk status
Execute the following commands to stop and start the Java Framework:
/usr/local/dbfw/bin/javafwk stop
/usr/local/dbfw/bin/javafwk start
Use the following procedure to check the status of the reports from the operating system logs after executing one of the procedures mentioned above and restarting the Java Framework:
Log in to AVCLI as admin user.
Execute the following command to enable diagnostics for the reports:
ALTER SYSTEM SET loglevel=ReportLog:DEBUG|JfwkLog:DEBUG;
The diagnostics can also be enabled using the Audit Vault Server console by following these steps:
Run a PDF report. For example, Activity Overview.
After a while, check on the /var/lib/oracle/dbfw/av/log file. For example, av.report* file. It contains the PDF/XLS report generation debug logs.
Problem
Host Monitor is capable of collecting audit data from Windows 2016 server. A message is displayed alerting the user to install Npcap and OpenSSL.
Solution
A set of DLL files may be causing an issue. Execute the following procedure to resolve this problem:
Search for the following files in the system:
libssl-1_1-x64.dlllibcrypto-1_1-x64.dllwpcap.dllpacket.dllAppend the file names with .bk format.
Go to Control Panel > Uninstall Programs and uninstall OpenSSL and WinPcap.
Reinstall Npcap and OpenSSL 1.1.1g. The DLL files are restored to Windows system folder.
Check the Control Panel to verify that these two programs are installed.
Go to C:\Windows\System32 or C:\Windows\SysWOW64 folders and search for the above four DLL files. At least one file for each DLL must be present without the .bk extension.
Go to the OpenSSL installation location and search for libssl-1_1-x64.dll and libcrypto-1_1-x64.dll files. One for each type is available.
Upon confirmation, add the OpenSSL installation location to the path variable.
Restart the trail.
If the network audit trail does not start, then check the collfwk logs present at <AgentHome>\av\log location. If the following message is available in the collfwk log, then check the Host Monitor logs present at <AgentHome>\hm\log location.
Prerequisites are installed and present in PATH variable or System directory
<AgentHome> refers to the Audit Vault Agent installation directory.
Note:
Continue with the remaining steps if your installation is 12.2.0.10.0 or before. The steps are not required for release 12.2.0.11.0 and later.MSVCRT.dll (*) or later) package installed. This is a must to use Host Monitor on Windows.If the following message is available in the Host Monitor log, then execute the remaining procedure:
Invalid AVS Credentials provided
Open the av/conf/bootstrap.prop file.
Copy the following line:
CONNECT_STRING_PARAM_POSTFIX=9999
Paste this line in the hm/bootstrap.prop file.
Restart the trail.
In case the network audit trail starts without any errors, then the collection status on the Audit Vault Server console confirms the same.
Go to AVAUDIT > Secured Target > Firewall Policies > Log All.
Connect to the secured target database instance using SQL Developer, or any other tool.
Generate the traffic for collecting data.
It must be recorded in the reports of the event_log table.
Learn what to do when the Host Monitor Agent fails to start.
Problem
The Host Monitor network trail does not start after installation. The collection framework (collfwk) log file contains one of the following errors:
java.io.IOException: Cannot run program "<AgentHome>/hm/hostmonmanager" (in directory "<AgentHome>/hm"): error=13, The file access permissions do not allow the specified action.
HMCommandExecutor : startTrail : binary is not found here: <AgentHome>/hm/hostmonmanager
Solution
This issue may arise due to insufficient privileges while starting Host Monitor. Ensure the Audit Vault Agent user belongs to the group that owns hm (Host Monitor installation) directory. Also ensure that the group that owns Host Monitor installation (hm) directory has read and execute permission on the hm directory and execute permission on hostmonmanager binary.
Note:
AgentHome is the Audit Vault Agent installation directory.
hm is the Host Monitor installation directory.
Use this procedure when the audit trail stops after you relocate the Windows event log files.
Problem
Windows event log relocation causes audit trail to be stopped.
Solution
Follow this procedure to resolve this problem:
Learn the resolution when the network audit trail failst to start on Unix platforms.
Problem
Network audit trail does not start on Unix platforms.
Symptoms
The Oracle Audit Vault Server console displays the following error:
Unable to start Host Monitor process
The collection framework log displays the following error:
<Host Monitor home>/hostmonmanager binary is not found here
Solution
hm symlink pointing to Host Monitor installation location.ls -lrt hm
Note:
The entire directory hierarchy must be owned by theroot user. All of the directories in this hierarchy must have read and execute permission for other users or groups, but not write permission.Problem
Audit Vault Agent goes into Unreachable state in the event of a failover of the Audit Vault Server or reboot of the primary Audit Vault Server.
Symptom
Audit Vault Agent and audit trails go into Unreachable state.
Solution
For the Primary Audit Vault Server reboot instance - Restart the Oracle Database Listener process on the standby Audit Vault Server.
For the failover instance, restart the Oracle Database Listener process on the primary (old) Audit Vault Server. This is the Audit Vault Server which was primary prior to failover.
Learn to fix incorrect Gateway details entered during installation.
Problem
Incorrect or invalid Gateway details entered while installing Audit Vault Sever or Database Firewall. The following error message may be encountered:
Gateway is not reachable from host
Solution
The Gateway details can to be corrected by following these steps:
Ctrl+Alt+Right Arrow Key.vi /usr/local/dbfw/etc/dbfw.conf
/usr/local/dbfw/bin/priv/configure-networking
Ctrl+Alt+Left Arrow Key.Note:
The network settings entered during installation can be modified, by choosing the Change IP Settings option in the installer or appliance screen.