Resolve Boot Failures After Shape Change on OCI Linux Instances

Introduction

This tutorial walks you through resolving boot failures on custom Oracle Cloud Infrastructure (OCI) images, which can occur after a shape change or even a simple reboot due to changes in underlying device mappings.

Before diving into the solution, let’s first understand what a custom image is.

A custom image in OCI is a snapshot of an instance’s boot volume that captures the operating system configuration, installed software, and environment settings. Custom images are commonly used to replicate pre-configured environments across multiple instances or regions, ensuring consistency at scale.

These images can introduce risks, however, especially if they are tightly coupled to the instance’s original hardware. If not designed with portability in mind, issues may arise when changing shapes (for example, switching from a VM.Standard3 to a VM.Standard4 instance), as underlying hardware changes can affect how devices are mapped.

This blog outlines a real-world boot failure scenario encountered by a customer using a Rocky Linux image from Oracle Cloud Marketplace. While this example is specific to Rocky Linux, the root cause and resolution apply broadly to any custom Linux-based image that uses static device names in its configuration.

Objectives

This guide helps OCI users identify and resolve boot failures on custom Linux images caused by shape changes or reboots, by correcting mount configurations and enhancing image portability.

Prerequisites

Task 1: Understand the Problem

  1. Review how OCI shape changes affect underlying hardware and device naming.
  2. Recognize that static device names in /etc/fstab (for example /dev/sdX) may become invalid after a shape change or even after a reboot.
    • When the shape changes, the OS may assign new names to attached block volumes. For example, a disk that was /dev/sdb might now appear as /dev/sdc.
    • If /etc/fstab relies on these static names, the volumes may fail to mount at boot.
    • To avoid this situation, use persistent identifiers such as the volume’s UUID or /dev/disk/by-uuid/ paths in /etc/fstab.

Task 2: Identify Symptoms and Errors

  1. Check serial console or system logs (journalctl -xb) for errors such as:

    • Welcome to emergency mode! After logging in, type “journalctl -xb”...
    • Failed to mount /mnt/data: No such device
    • Duplicate mount point /mnt/data detected
    • Cannot mount /dev/sdX: No such file or directory
    • systemd[1]: Failed to mount /mnt/data.
    • systemd[1]: Dependency failed for Local File Systems.
    • systemd[1]: Job local-fs.target/start failed with result ‘dependency’.

Task 3: Boot Into Rescue/Emergency Mode

  1. Access the instance’s serial console.
  2. Boot into rescue or emergency mode to perform diagnostics and corrective actions.

Task 4: Identify Correct Device Names

  1. Run blkid or lsblk commands to list current block devices and their UUIDs.

Task 5: Manually Mount Boot Volume

  1. Mount the boot volume using the correct device name:

    • mount /dev/sdXn /mnt
    • chroot /mnt

Task 6: Edit /etc/fstab

  1. Open /etc/fstab using an editor (for example vi /etc/fstab)

    • Replace all /dev/sdX entries with corresponding persistent UUID values found from blkid.
    • Remove any duplicate mount entries. Note: This is a one-time fix once persistent identifiers are used, future shape changes should not cause this issue.

Task 7: Exit and Reboot

  1. Exit the chroot environment: exit.
  2. Unmount volumes if needed.
  3. Reboot the instance.

Prevention Tips to Avoid Boot Failures After OCI Shape Changes

  1. Use Persistent Identifiers: Always specify disks in /etc/fstab using UUID= or LABEL= instead of device names like /dev/sdX. This ensures mount points remain consistent despite hardware changes.
  2. Perform blkid Audits: Run blkid before and after a shape change to verify that device-to-UUID mappings remain correct.
  3. Backup Before Changes: Always back up your instance or boot volume prior to any shape modifications to enable recovery if issues occur.
  4. Test Shape Changes in Non-Production: Especially when using marketplace or custom images that might have hard-coded mounts, test shape changes in a non-production environment first.

Conclusion

While this issue was observed on Rocky Linux, it can potentially affect any Linux-based custom image if best practices around disk mounting are not followed. The good news is that this is a one-time correction. After /etc/fstab is updated to use persistent identifiers like UUIDs or LABELs, future OCI shape changes or even reboots should not lead to boot failures. Proactively validating your configuration ensures smooth, reliable transitions across shapes.

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.