5 Troubleshooting Oracle Linux Upgrades

This chapter provides troubleshooting information and describes known issues that might affect the upgrade process.

Tools for Troubleshooting

Use the following options to generate more output when you are generating the preupgrade report or performing the actual upgrade:

  • --verbose displays warnings, error messages, and other critical information.

  • --debug adds debug information in addition to the same output as the --verbose option.

You can use the following resources and tools for obtaining troubleshooting information:

  • /var/log/leapp/leapp-report.txt
  • /var/log/leapp/leapp-upgrade.log

  • /var/log/leapp/dnf-debugdata/: a directory for debug information. Note that this directory is created only if you use the --debug option when issuing either the preupgrade or the upgrade command.

  • journalctl command

Known Issues

The following are known issues that you might encounter when upgrading an Oracle Linux 7 system to Oracle Linux 8.

Upgrade Issues

  • Optional Resilient Storage group is not supported with Leapp

    Leapp does not support the optional Resilient Storage group. The presence in the system of certain packages from that group might prevent a leapp upgrade to complete.

    For example, the Resilient Storage group includes the lvm2-cluster package. In an Oracle Linux 7 installation, this package can be optionally added as part of either the Infrastructure Server and File profile or the Print Server profile. However, the package is an inhibitor and would cause the Leapp upgrade to fail.

  • Leapp might report missing packages that are marked for installation

    The /var/log/leapp-preupgrade.log or /var/log/leapp-upgrade.log files might report a warning similar to the following:

    Warning: Packages marked by Leapp for install not found in repositories 
    metadata: rpcgen python3-pyxattr libnsl2-devel rpcsvc-proto-devel

    These packages are in the Oracle Linux 8 Codeready Builder repository, which is a developer repository and is disabled by default.

    If your installation requires these packages, then during the preupgrade or the upgrade phase, add the --enablerepo ol8_codeready_builder option to the appropriate Leapp command, for example:

    sudo leapp upgrade --oraclelinux --enablerepo ol8_codeready_builder

    Repositories that have been enabled during the Leapp upgrade remain enabled on your Oracle Linux 8 system after the upgrade completes.

    Alternatively, after completing the upgrade, you can manually install the packages required for your installation by using the dnf command.

  • MySQL-related *.el7 packages might remain after an upgrade

    In an Oracle Cloud Infrastructure instance, previously existing MySQL packages might remain after the upgrade is completed. You can verify their existence with the following command:

    rpm -qa | grep el7 | grep -v leapp | grep -v kernel
    mysql-community-client-plugins-8.0.25-1.el7.x86_64

    To ensure that these packages are properly updated, enable the ol8_MySQL80 repository also when you run the upgrade.

    sudo leapp upgrade --oci --enablerepo ol8_mysql80
  • Some el7 packages might not be upgraded

    The same rpm -qa command syntax in the previous item that detects MySQL-related *.el7 packages might also list additional *el7 packages on the system that were not upgraded. Packages might not be upgraded if they were installed from repositories that are not supported by Leapp, such as developer repositories. For such packages, do the following:

    1. Go to https://yum.oracle.com and determine the Oracle Linux 8 repositories that would serve the packages you need.

    2. After the upgrade is completed, manually install the packages from those Oracle Linux 8 repositories.

    3. After all of the necessary packages have been installed, remove the residual el7 packages from the system.

  • (aarch64) Upgrade log might report errors related to the vmd module

    After completing an upgrade on aarch64 systems, the Leapp upgrade log might report the following message:

    dracut-install: Failed to find module 'vmd'

    The VMD module does not apply to the Arm architecture and therefore, the error message can be safely ignored.

Security and Authentication Issues

  • TCP Wrappers not used in Oracle Linux 8

    TCP Wrappers are not available in Oracle Linux 8. If you are using TCP Wrappers to control network traffic in the Oracle Linux 7 system through /etc/hosts.deny and /etc/hosts.allow files, then after the upgrade you must create equivalent firewall rules to implement the same control in Oracle Linux 8.

    For more information about implementing firewall rules, see Oracle Linux 8: Configuring the Firewall.

  • Oracle Linux 7 authentication configuration not converted after upgrade

    Leapp does not convert Oracle Linux 7 authentication configuration that uses the deprecated authconfig utility to the equivalent Oracle Linux 8 authentication that uses the authselect command. After the upgrade, you need to reconfigure authentication appropriately in Oracle Linux 8.

    For more information about using the authselect utility, see Oracle Linux 8: Setting Up System Users and Authentication.

  • Upgrade blocked if system has configured pam_krb5 and pam_pkcs11 PAM modules

    The deprecated pam_krb5 and pam_pkcs11 PAM modules in Oracle Linux 7 are removed during the upgrade process. Before performing the upgrade, you must first reconfigure the system's PAM configuration to disable these modules. Otherwise, the system becomes locked.

    For instructions to disable these modules, see Oracle Linux 7: Setting Up System Accounts and Authentication. See also the discussion about these modules in Providing Information to the Leapp Answerfile.

System Management Issues

  • Ksplice Uptrack software displays error messages

    During the upgrade, the Oracle Ksplice Uptrack software might report errors similar to the following:

    [ 256.033527] upgrade[390]: Upgrading : uptrack-1.2.74-0.el8.noarch 577/1453
    [ 256.037151] upgrade[390]: Running scriptlet: uptrack-1.2.74-0.el8.noarch 577/1453
    ...
    [ 256.045914] upgrade[390]: Traceback (most recent call last):
    [ 256.049230] upgrade[390]: File "/usr/lib/uptrack/access-key-from-uln", line 9, in <module>
    [ 256.051376] upgrade[390]: from up2date_client import up2dateAuth, rpcServer
    [ 256.056490] upgrade[390]: File "/usr/share/rhn/up2date_client/up2dateAuth.py", line 74
    [ 256.059251] upgrade[390]: os.chmod(path, 0600)
    [ 256.060997] upgrade[390]: 
    [ 256.062842] upgrade[390]: SyntaxError: invalid token

    The report is a known but harmless issue, which you can ignore. After the upgrade is completed, Ksplice continues to operate normally.

File Systems and Storage Issues

  • Systems with Btrfs in a RAID configuration cannot be upgraded

    A system that uses the Btrfs file system in a RAID configuration cannot be upgraded. In the /var/log/leapp/leapp-report.txt that is generated by the preupgrade command, this configuration is flagged as an inhibitor and no remedy is provided. If you attempt to upgrade the system and that configuration is detected, the upgrade process halts.

  • Upgrade blocked if winbind and wins Samba modules are used

    If winbind and wins Samba modules are used in the /etc/nsswitch.conf, the upgrade is blocked. As a workaround, remove these modules from the file first, then perform the upgrade. After the upgrade is complete, restore these module entries to the file.

    For more information about configuring these modules, see Oracle Linux 8: Managing Shared File Systems.

  • Hosts with network mounted file systems cannot be upgraded

    Leapp does not support upgrading systems with mounted file systems on network storage, NFS, or iSCSI. As a workaround, unmount the file systems and comment out their entries from /etc/fstab. After the upgrade is completed, you can restore the entries and remount the file systems.

Networking Issues

  • Possible upgrade error if system has multiple NICs with the same prefix as NIC that is used by kernel

    The in-place upgrade process might cause an error if the system to be upgraded has more than one NIC that shares the same prefix as the NIC that is used by the kernel, for example eth. Consequently, after the upgrade, the system's network connectivity is lost.

    For more information, see About Network Interface Names in Oracle Linux 8: Setting Up Networking.

  • NetworkManager might not start after the upgrade completes

    After the upgrade, the system's NetworkManager might not start because of the failure of its name resolution service. The failure can be verified by checking the status of the service.

    systemctl status systemd-resolved.service
    ● systemd-resolved.service - Network Name Resolution 
       Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; 
    disabled; > 
       Active: inactive (dead) 
         Docs: man:systemd-resolved.service(8) 
               https://www.freedesktop.org/wiki/Software/systemd/resolved

    The /var/log/messages file also reports the following error:

    dbus-daemon[742]: [system] Activation via systemd failed for unit 
    'dbus-org.freedesktop.resolve1.service': Unit 
    dbus-org.freedesktop.resolve1.service not found.

    To resolve this issue, choose one of the following workarounds:

    • Configure NetworkManager to not use systemd-resolved.service.

      Add the following entries to the /etc/NetworkManager/conf.d/no-systemd-resolved.conf file:

      [main]
      systemd-resolved=false
    • Enable the systemd-resolved.service as follows:

      systemctl enable systemd-resolved.service
      Created symlink /etc/systemd/system/dbus-org.freedesktop.resolve1.service → 
      /usr/lib/systemd/system/systemd-resolved.service. 
      Created symlink 
      /etc/systemd/system/multi-user.target.wants/systemd-resolved.service → 
      /usr/lib/systemd/system/systemd-resolved.service.
      systemctl start systemd-resolved.service

    You can also adopt other methods that are more consistent with the network name resolution model that you are using for your specific setup. For useful information, refer to About Network Interface Names in Oracle Linux 8: Setting Up Networking.

Virtualization and Containers Issues

  • Docker removed during upgrade process

    Docker is removed as part of the upgrade process. If you intend to run containers on the upgraded system, be prepared to migrate your container infrastructure to Podman.

    For more information, see Oracle Linux: Podman User's Guide.

  • KVM virtual machine snapshots might not be listed after an upgrade

    After an upgrade, the libvirtd service might report snapshot-related error messages similar to the following:

    libvirtd[53328]: internal error: Failed to parse snapshot XML from file 
    '/var/lib/libvirt/qemu/snapshot/path-to-previous-snapshot-file'

    Furthermore, listing available snapshots from prior to the upgrade generates an empty list.

    sudo virsh snapshot-list previous-snapshot-file
     Name   Creation Time   State 
    -------------------------------

    As a workaround, reboot the system. At the end of the boot process, the snapshots should be listed and available again.

  • libvirtd service might fail to restart in nested virtualization configurations

    In nested virtualization setups, the libvrtd service might not restart in the nested KVM host after the upgrade.

    As a workaround, reboot the nested KVM host.

  • Some KVM virtual machines might not start after the upgrade

    If the KVM host is running the RHCK kernel, virtual machines or guests that existed prior to the upgrade might not start after the Leapp upgrade completes. When you attempt to start these guests, an error message similar to the following might be displayed:

    sudo virsh start domain-name
    error: Failed to start domain domain-name
    error: the CPU is incompatible with host CPU: Host CPU does not provide
    required features: hle, rtm

    This error is reported because the RHCK kernel disables Intel Transactional Synchronization Extensions (TSX) by default and cannot provide the Hardware Lock Elision (HLE) and Restricted Transactional Memory (RTM) features that the guest is expecting.

    As a workaround, perform the following steps:

    1. Edit /etc/default/grub by adding tsx=on to the GRUB_CMDLINE_LINUX directive, as shown in the following example in bold:

      ...
      GRUB_CMDLINE_LINUX="crashkernel=auto LANG=en_US.UTF-8 … tsx=on"
    2. Regenerate the grub.cfg file.

      sudo grub2-mkconfig -o /boot/grub2/grub.cfg
    3. Reboot the KVM host.

Development Tools Issues

  • Python 2 replaced by Python 3

    In Oracle Linux 8, the python command is not available by default. Python 3 is the primary Python version and is not backward compatible with Python 2. Additionally, Python 2 is available in Oracle Linux 8 but only with limited support. If you have Python packages or scripts on the system to be upgraded, you should migrate these to Python 3 as soon as possible.

    For more information, see Oracle Linux 8: Installing and Managing Python.

Hardware Related Issues

  • Systems with unsupported hardware cannot be upgraded

    Support for certain hardware, such as the e1000 driver, has been removed from RHCK 8. The upgrade cannot proceed on platforms that have such hardware installed. Even though UEK might continue to support the hardware, the upgrade procedure is still inhibited if the hardware is detected on the system.

Leapp Overlay Size Issues

  • Upgrading might require increased overlay size

    Upgrading Oracle Linux 7 systems with a large number of packages to Oracle Linux 8 might fail because of insufficient space in the Leapp overlay file systems that are used during the upgrade. You might encounter the following error message:

    Error: Transaction test error: 
      installing package package-name needs 4MB on the / filesystem 

    As a workaround, increase the LEAPP_OVL_SIZE variable. The default size is 4096. The actual size you would need might be larger depending on your specific setup. Use the following command:

    sudo export LEAPP_OVL_SIZE=new-size