Pre-upgrade RPM Warnings

While patching or upgrading Oracle Database Security Central (Oracle DBSecCentral), the pre-upgrade RPM displays warnings to indicate issues that you need to resolve before proceeding with the update.

RPM Upgrade Failed

Read the troubleshooting advice if RPM upgrades fail.

Problem

An RPM upgrade failed with the following error:

error: %post(dbfw-mgmtsvr-###) scriptlet failed, exit status 1

Solution

1. Check that there is at least 10MB of free /tmp space.

2. Remove the new RPM:

   rpm -e dbfw-mgmtsvr-###

3. Retry the upgrade.

Uninstalling the Pre-Upgrade RPM for AVDF 20.12 and Later Doesn’t Remove Filesystem

If you currently have Oracle AVDF 20.11 or earlier and apply the pre-upgrade RPM for AVDF 20.12 or later, and decide to not proceed with the upgrade, the filesystem for Database Firewall doesn’t get removed. If you wish to reallocate the space reserved for upgrade, perform the following.

1. Run the following command to find the exact version of the pre-upgrade RPM:

   rpm -q avdf-pre-upgrade

2. Run the following command to uninstall and remove the pre-upgrade RPM:

   rpm -e {rpm name}

3. Run the following command to verify the filesystem remains mounted:

   # df

   You will see something similar to:

   [...]
   /dev/mapper/vg_root-lv_var_dbfw_upgrade on /var/dbfw/upgrade type ext4(rw,relatime,seclabel)
   [...]

4. Run the following command to unmount the filesystem:

   umount /var/dbfw/upgrade

5. Run the following command to remove the logical volume:

   lvremove /dev/vg_root/lv_var_dbfw_upgrade

6. Run the following command to confirm the logical volume is unmounted and removed:

   # df
   # lvs

   You will see something similar to:

   [no /var/dbfw/upgrade records]
   [no /var/dbfw/upgrade records]

Pre-upgrade RPM Failure Due to Insufficient Memory

Learn how to resolve pre-upgrade RPM failure due to insufficient memory.

Problem

Installing the pre-upgrade RPM places the system in a safe state, performs multiple checks, and rearranges free space on the appliance for a safe and successful installation or upgrade of Audit Vault Server and Database Firewall.

The following error may be observed:

AVDF::Installer::Upgrade::InvalidPreconditions
Recommended memory is x.yy GB; system only has xx.yy MB available
ERROR:
AVDF::Installer::Upgrade::InvalidPreconditions
Verifying pre-upgrade conditions failed.

Solution

Follow these steps to resolve this issue:

1. Run the following command to find the exact version of the pre-upgrade RPM:

   rpm -q avdf-pre-upgrade

2. Run the following command to uninstall and remove the pre-upgrade RPM:

   rpm -e {rpm name}

3. Power off the host machine.

4. Increase the memory as per the recommendation.

5. Power on the host machine.

6. Re-install the pre-upgrade RPM.

7. Ensure to check the warnings related to memory are resolved.

8. Proceed with the upgrade as per Oracle DBSecCentral documentation.

Insufficient Space Error in /var/lib/oracle File System Reported by Pre-upgrade RPM

Learn how to fix insufficient space error issue in /var/lib/oracle (lv_oracle) file system reported by pre-upgrade RPM.

Problem

An error or issue is observed when running pre-upgrade RPM. There is insufficient space in /var/lib/oracle (lv_oracle) file system.

Solution

The /var/lib/oracle file system needs a minimum of 31 GB free space for performing upgrade.

Follow these steps to clear space in /var/lib/oracle and to proceed with the upgrade process:

1. Run the following command as grid user:

   /usr/bin/find /var/lib/oracle/grid/rdbms/audit -name '*.aud' -mtime +1 -delete

   This process may take up to one hour to complete.

2. Create another terminal.

3. Run the following command as grid user to remove the trc and trm files:

   rm /var/lib/oracle/diag/asm/+asm/+ASM/trace/*.tr[cm]

4. As root user check if the /var/lib/oracle/upgrade_iso_file directory exists. Remove the ISO file in case it exists.

5. As root user check and remove these file in case they exist.

   rm /var/lib/oracle/software/database.tar.xz

   rm /var/lib/oracle/dbfw/av/grid[12].zip

6. Run the following command as oracle user and remove the trc and trm files:

   rm /var/lib/oracle/diag/rdbms/dbfwdb/dbfwdb/trace/*.tr[cm]

7. Clear diagnostic logs through the Oracle Database Security Central console. This process may also release some additional space. In case any of the components are set to Debug, then set them to Warning.

Note: Clearing Diagnostic Logs

Insufficient Space Error in / File System Reported by Pre-upgrade RPM

Learn how to fix insufficient space error issue in the / file system reported by pre-upgrade RPM.

Problem

An error similar to the below message is observed when running pre-upgrade RPM. There is insufficient space in the / file system.

Checking upgrade preconditions
This upgrade requires at least
2.35GiB free on / (actual: 2.29GiB)

    AVDF::Installer::Upgrade::InvalidPreconditions

Precondition: 'space-check.rb'
    Result: 'Please follow the instructions in the Administrator's Guide to add storage, then retry.
    Summary: AVDF::Installer::Upgrade::InvalidPreconditions
        System is not ready for upgrade.

Solution

Extend / using the free space from vg_root:

lvextend --resizefs -L+2.35G /dev/vg_root/lv_ol8root

Pre-upgrade RPM Could Not Stop Certain Processes During Oracle AVDF Upgrade

Learn how to fix warnings or errors pointed by pre-upgrade RPM while upgrading Oracle AVDF.

Problem

The pre-upgrade RPM performs necessary checks to prepare the appliance conducive for upgrade. It stops certain processes running on the appliance in due course. In some cases, some of the processes cannot be stopped by the pre-upgrade RPM. It results in the following errors or warnings:

Not all processes were stopped

target is busy

Solution

Follow these steps:

1. The pre-upgrade RPM suggests a possible way or solution to figure out the specific processes that are still running. Follow the instructions and stop the specific processes.

2. Uninstall the pre-upgrade RPM.

3. Reinstall the pre-upgrade RPM.

4. Proceed with the upgrade procedure.

Pre-upgrade RPM Fails with “Unable to Stop Observer”

Learn how to resolve the “unable to stop observer” warning in the pre-upgrade RPM.

Problem

The pre-upgrade RPM fails with the “unable to stop observer” warning.

Messages and debug files display one of the following errors when the observer was started:

‘DGMGRL:ORA-28000: The account is locked.’ or ‘DGMGRL:ORA-28001: the password has expired’

Solution

This can happen if the sys password has expired or the sys user is locked. To resolve this issue, update the sys user on the primary and standby systems. See Verify That the SYS User Is Unlocked and the Password Is Not Expired for instructions.

Pre-upgrade RPM Fails with “Not All Processes Were Stopped”

Problem

The pre-upgrade RPM fails with the following warning: Not all processes were stopped: 7378,7379.

For example:

rpm -ivh --force avdf-pre-upgrade-20.x.0.0.0-0_NNNNNN.NNNN.x86_64.rpm
Preparing... ########################################### [100%]
1:avdf-pre-upgrade ########################################### [100%]
Checking upgrade preconditions
/bin/df: '/var/dbfw/upgrade': No such file or directory
/bin/df: no file systems processed
Shutting down services.
Traceback (most recent call last):
3: from /usr/local/dbfw/bin/pre_upgrade.rb:642:in '<main>'
2: from /usr/local/dbfw/bin/pre_upgrade.rb:614:in 'process_command_line'
1: from /usr/local/dbfw/bin/pre_upgrade.rb:503:in 'post_install'
/usr/local/dbfw/lib/ruby/upgrade/common.rb:621:in 'stop_nonroot_processes':
Not all processes were stopped: 7378,7379

Cause

This issue could be caused by an idle SSH session, busy devices, or open temporary files.

Solution

1. Uninstall the RPM as the root user.

   1. Log in to the Audit Vault Server through SSH and switch to the root user.

      See Logging In to Oracle DBSecCentral Appliances Through SSH.

   2. Uninstall the pre-upgrade RPM by using one of the following commands:

      rpm -e avdf-pre-upgrade

      rpm -e avdf-pre-upgrade --noscripts

2. Check the pre-upgrade RPM listing.

   1. Enter the following command:

      rpm -qa \|grep avdf-pre-upgrade

   2. Ensure that there’s no entry for avdf-pre-upgrade RPM.

   3. Reboot the Audit Vault Server if it’s a STANDALONE system.

3. Check for other SSH sessions, busy devices, or temporary open files.

   1. Ensure that there are no other SSH sessions that are owned by the support user.

      To do this, identify idle notty (no tty) SSH sessions and try to stop them.

      Use the following commands to check the pid of sshd: support@notty.

      ps -ef \|grep support

      ps -ef \|grep notty

      For example:

      support 2480 2427 0 18:31 ? 00:00:00 sshd: support@notty
      support 2481 2480 0 18:31 ? 00:00:00 -bash
      kill -9 2481
      kill -9 2480

   2. Check again for support@notty processes in the system.

   3. Ensure that the system doesn’t have any busy devices or open temporary files. To do this, run lsof against /tmp and /usr/local/dbfw/tmp.

      For example:

      lsof /usr/local/dbfw/tmp
      lsof /tmp

      Note: Ensure that no logs are open when starting the patching or upgrade process.

4. Try to install the pre-upgrade RPM as the root user.

   1. Log in to the Audit Vault Server through SSH and switch to the root user.

      See Logging In to Oracle DBSecCentral Appliances Through SSH.

   2. Enter the following command:

      rpm -i /root/avdf-pre-upgrade-20.x.0.0.0-0_NNNNNN.NNNN.x86_64.rpm

Pre-upgrade RPM Check: Agent Failure Checks - Upgrade Prerequisites

Starting with Oracle AVDF 20.9, the pre-upgrade RPM verifies that the Audit Vault Agent and Host Monitor Agent configurations are compatible with Oracle AVDF 20.10 or later.

Problem

The agent_prereq_checks_failure_report.txt report indicates that a Audit Vault Agent or Host Monitor Agent doesn’t meet the prerequisites to update to Oracle AVDF 20.10 or later. You can find the agent success and failure reports in the following locations:

• Success report: /opt/avdf/report/agent_prereq_checks_success_report.txt

• Failure report: /opt/avdf/report/agent_prereq_checks_failure_report.txt

The following example shows a failure message:

Agent/HM Validation Failure statuses are as below :
------------------------------------------------------------------
Agent Name : agent-linux
Agent Validation Status : FAILURE
Agent Failure Checks : Upgrade Prerequisites check jar build with latest version. Please check the minimum java version required. - <Exception Message>
Agent Checks Warning Messages :
Validated at : 2022-12-02 09:11:24.774880

Solution

Resolve the issue that’s indicated in the report. For example, update the Audit Vault Agent machine to the minimum Java version that’s supported.

You can rerun the failure check scripts individually to verify that the issues are resolved. Run these scripts as the root user.

/usr/bin/python3 /usr/local/dbfw/bin/upgrade/pre_upgrade_validate_agent.py standalone

/usr/bin/python3 /usr/local/dbfw/bin/upgrade/pre_upgrade_download_agent_validation_status.py standalone