Troubleshooting Oracle Linux Upgrades

5 Troubleshooting Oracle Linux Upgrades

This chapter provides troubleshooting information and describes known issues that might affect the upgrade process.

Tools for Troubleshooting

Use the following options to generate more output when you are generating the preupgrade report or performing the actual upgrade:

--verbose displays warnings, error messages, and other critical information.
--debug adds debug information in addition to the same output as the --verbose option.

You can use the following resources and tools for obtaining troubleshooting information:

/var/log/leapp/leapp-report.txt
/var/log/leapp/leapp-upgrade.log
/var/log/leapp/dnf-debugdata/: a directory for debug information. Note that this directory is created only if you use the --debug option when issuing either the preupgrade or the upgrade command.
journalctl command

Known Issues

The following are known issues that you might encounter when upgrading an Oracle Linux 7 system to Oracle Linux 8.

Upgrade Issues

Optional Resilient Storage group isn't supported with Leapp

Leapp doesn't support the optional Resilient Storage group. The presence in the system of certain packages from that group might prevent a leapp upgrade to complete.

For example, the Resilient Storage group includes the lvm2-cluster package. In an Oracle Linux 7 installation, this package can be optionally added as part of either the Infrastructure Server and File profile or the Print Server profile. However, the package is an inhibitor and would cause the Leapp upgrade to fail.

BugID 33573562
Leapp might report missing packages that are marked for installation

The /var/log/leapp-preupgrade.log or /var/log/leapp-upgrade.log files might report a warning similar to the following:
```
Warning: Packages marked by Leapp for install not found in repositories 
metadata: rpcgen python3-pyxattr libnsl2-devel rpcsvc-proto-devel
```
These packages are in the Oracle Linux8 Codeready Builder repository, which is a developer repository and is disabled by default.

If the system requires these packages, then during the preupgrade or the upgrade phase, add the --enablerepo ol8_codeready_builder option to the appropriate Leapp command, for example:
```
sudo leapp upgrade --oraclelinux --enablerepo ol8_codeready_builder
```
Repositories that have been enabled during the Leapp upgrade remain enabled on the Oracle Linux 8 system after the upgrade completes.

Alternatively, after completing the upgrade, you can manually install the packages required for your installation by using the dnf command.

Bug ID 32827043
MySQL-related *.el7 packages might remain after an upgrade

In an Oracle Cloud Infrastructure instance, existing MySQL packages might remain after the upgrade is completed. You can verify their existence with the following command:
```
rpm -qa | grep el7 | grep -v leapp | grep -v kernel
```
```
mysql-community-client-plugins-8.0.25-1.el7.x86_64
```
To ensure that these packages are correctly updated, enable the ol8_MySQL80 repository also when you run the upgrade.
```
sudo leapp upgrade --oci --enablerepo ol8_mysql80
```
Some el7 packages might not be upgraded

The same rpm -qa command syntax in the previous item that detects MySQL-related *.el7 packages might also list more *el7 packages on the system that weren't upgraded. Packages might not be upgraded if they were installed from repositories that aren't supported by Leapp, such as developer repositories. For such packages, do the following:
1. Go to https://yum.oracle.com and check the Oracle Linux 8 repositories that would serve the packages you need.
2. After the upgrade is completed, manually install the packages from those Oracle Linux 8 repositories.
3. After all the necessary packages have been installed, remove the residual el7 packages from the system.
Bug ID 32878386
(aarch64) Upgrade log might report errors related to the vmd module

After completing an upgrade on aarch64 systems, the Leapp upgrade log might report the following message:
```
dracut-install: Failed to find module 'vmd'
```
The VMD module doesn't apply to the Arm architecture and therefore, the error message can be safely ignored.

Bug ID 34172552

Security and Authentication Issues

TCP Wrappers not used in Oracle Linux 8

TCP Wrappers aren't available in Oracle Linux 8. If you're using TCP Wrappers to control network traffic in the Oracle Linux 7 system through /etc/hosts.deny and /etc/hosts.allow files, then after the upgrade you must create applicable firewall rules to implement the same control in Oracle Linux 8.

For more information about implementing firewall rules, see Oracle Linux 8: Configuring the Firewall.
Oracle Linux 7 authentication configuration not converted after upgrade

Leapp does not convert Oracle Linux 7 authentication configuration that uses the deprecated authconfig utility to the corresponding Oracle Linux 8 authentication that uses the authselect command. After the upgrade, you need to reconfigure authentication appropriately in Oracle Linux 8.

For more information about using the authselect utility, see Oracle Linux 8: Setting Up System Users and Authentication.
Upgrade blocked if system has configured pam_krb5 and pam_pkcs11 PAM modules

The deprecated pam_krb5 and pam_pkcs11 PAM modules in Oracle Linux 7 are removed during the upgrade process. Before performing the upgrade, you must first reconfigure the system's PAM configuration to disable these modules. Otherwise, the system becomes locked.

For instructions to disable these modules, see Oracle Linux 7: Setting Up System Accounts and Authentication. See also the discussion about these modules in Providing Information to the Leapp Answerfile.

System Management Issues

Ksplice Uptrack software displays error messages

During the upgrade, the Oracle Ksplice Uptrack software might report errors similar to the following:

[ 256.033527] upgrade[390]: Upgrading : uptrack-1.2.74-0.el8.noarch 577/1453
[ 256.037151] upgrade[390]: Running scriptlet: uptrack-1.2.74-0.el8.noarch 577/1453
...
[ 256.045914] upgrade[390]: Traceback (most recent call last):
[ 256.049230] upgrade[390]: File "/usr/lib/uptrack/access-key-from-uln", line 9, in <module>
[ 256.051376] upgrade[390]: from up2date_client import up2dateAuth, rpcServer
[ 256.056490] upgrade[390]: File "/usr/share/rhn/up2date_client/up2dateAuth.py", line 74
[ 256.059251] upgrade[390]: os.chmod(path, 0600)
[ 256.060997] upgrade[390]: 
[ 256.062842] upgrade[390]: SyntaxError: invalid token

The report is a known but harmless issue, which you can ignore. After the upgrade is completed, Ksplice continues to operate normally.

File Systems and Storage Issues

Systems with Btrfs in a RAID configuration can't be upgraded

A system that uses the Btrfs file system in a RAID configuration can't be upgraded. In the /var/log/leapp/leapp-report.txt that's generated by the preupgrade command, this configuration is flagged as an inhibitor and no remedy is provided. If you upgrade the system and that configuration is detected, the upgrade process halts.
Upgrade blocked if winbind and wins Samba modules are used

If winbind and wins Samba modules are used in the /etc/nsswitch.conf, the upgrade is blocked. As a workaround, remove these modules from the file first, then perform the upgrade. After the upgrade is complete, restore these module entries to the file.

For more information about configuring these modules, see Oracle Linux 8: Managing Shared File Systems.
Hosts with network mounted file systems can't be upgraded

Leapp doesn't support upgrading systems with mounted file systems on network storage, NFS, or iSCSI. As a workaround, unmount the file systems and comment out their entries from /etc/fstab. After the upgrade is completed, you can restore the entries and remount the file systems.

Networking Issues

Possible upgrade error if system has several NICs with the same prefix as NIC that's used by kernel

The in-place upgrade process might cause an error if the system to be upgraded has more than one NIC that shares the same prefix as the NIC that's used by the kernel, for example eth. After the upgrade, the system's network connectivity is lost.

For more information, see About Network Interface Names in Oracle Linux 8: Setting Up Networking.

NetworkManager might not start after the upgrade completes

After the upgrade, the system's NetworkManager might not start because of the failure of its name resolution service. The failure can be verified by checking the status of the service.

systemctl status systemd-resolved.service

● systemd-resolved.service - Network Name Resolution 
   Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; 
disabled; > 
   Active: inactive (dead) 
     Docs: man:systemd-resolved.service(8) 
           https://www.freedesktop.org/wiki/Software/systemd/resolved

The /var/log/messages file also reports the following error:

dbus-daemon[742]: [system] Activation via systemd failed for unit 
'dbus-org.freedesktop.resolve1.service': Unit 
dbus-org.freedesktop.resolve1.service not found.

To resolve this issue, choose one of the following workarounds:

Configure NetworkManager to not use systemd-resolved.service.

Add the following entries to the /etc/NetworkManager/conf.d/no-systemd-resolved.conf file:
```
[main]
systemd-resolved=false
```

Enable the systemd-resolved.service as follows:

systemctl enable systemd-resolved.service

Created symlink /etc/systemd/system/dbus-org.freedesktop.resolve1.service → 
/usr/lib/systemd/system/systemd-resolved.service. 
Created symlink 
/etc/systemd/system/multi-user.target.wants/systemd-resolved.service → 
/usr/lib/systemd/system/systemd-resolved.service.

systemctl start systemd-resolved.service

You can also adopt other methods that are more consistent with the network name resolution model that you're using for the specific setup. For useful information, see About Network Interface Names in Oracle Linux 8: Setting Up Networking.

Virtualization and Containers Issues

Docker removed during upgrade process

Docker is removed as part of the upgrade process. If you intend to run containers on the upgraded system, be prepared to migrate the container infrastructure to Podman.

For more information, see Oracle Linux: Podman User's Guide.
KVM virtual machine snapshots might not be listed after an upgrade

After an upgrade, the libvirtd service might report snapshot-related error messages similar to the following:
```
libvirtd[53328]: internal error: Failed to parse snapshot XML from file 
'/var/lib/libvirt/qemu/snapshot/path-to-previous-snapshot-file'
```
Furthermore, listing available snapshots from prior to the upgrade generates an empty list.
```
sudo virsh snapshot-list previous-snapshot-file
```
```
 Name   Creation Time   State 
-------------------------------
```
As a workaround, reboot the system. At the end of the boot process, the snapshots are listed and available again.
libvirtd service might fail to restart in nested virtualization configurations

In nested virtualization setups, the libvrtd service might not restart in the nested KVM host after the upgrade.

As a workaround, reboot the nested KVM host.
Some KVM virtual machines might not start after the upgrade

If the KVM host is running the RHCK kernel, virtual machines or guests that existed before the upgrade might not start after the Leapp upgrade completes. When start these guests, an error message similar to the following might be displayed:
```
sudo virsh start domain-name
```
```
error: Failed to start domain domain-name
error: the CPU is incompatible with host CPU: Host CPU does not provide
required features: hle, rtm
```
This error is reported because the RHCK kernel disables Intel Transactional Synchronization Extensions (TSX) by default and can't provide the Hardware Lock Elision (HLE) and Restricted Transactional Memory (RTM) features that the guest is expecting.

As a workaround, perform the following steps:
1. Edit /etc/default/grub by adding tsx=on to the GRUB_CMDLINE_LINUX directive, as shown in the following example in bold:
```
...
GRUB_CMDLINE_LINUX="LANG=en_US.UTF-8 … tsx=on"
```
2. Regenerate the grub.cfg file.
```
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
```
3. Reboot the KVM host.

Development Tools Issues

Python 2 replaced by Python 3

In Oracle Linux 8, the python command isn't available by default. Python 3 is the primary Python version and isn't backward compatible with Python 2. Also, Python 2 is available in Oracle Linux 8 but only with limited support. If you have Python packages or scripts on the system to be upgraded, migrate these to Python 3 as soon as possible.

For more information, see Oracle Linux 8: Installing and Managing Python.

Hardware Related Issues

Systems with unrecognized hardware can't be upgraded

Support for certain hardware, such as the e1000 driver, has been removed from RHCK 8. The upgrade can't proceed on platforms that have such hardware installed. Even though UEK might continue to support the hardware, the upgrade procedure is still inhibited if the hardware is detected on the system.

Leapp Overlay Size Issues

Upgrading might require increased overlay size

Upgrading Oracle Linux 7 systems with a huge number of packages to Oracle Linux 8 might fail because of insufficient space in the Leapp overlay file systems that are used during the upgrade. You might see the following error message:
```
Error: Transaction test error: 
  installing package package-name needs 4MB on the / filesystem 
```
As a workaround, increase the LEAPP_OVL_SIZE variable. The default size is 4096. The actual size you would need might be larger depending on the specific setup. Use the following command:
```
sudo export LEAPP_OVL_SIZE=new-size
```