Working With Kernel Dumps

7 Working With Kernel Dumps

The Kdump feature provides a kernel crash information dumping mechanism in Oracle Linux. The kdump service saves the contents of the system’s memory for later analysis. The second kernel resides in a reserved part of the system memory.

Kdump uses the kexec system call to boot into the second kernel, called a capture kernel, without the need to reboot the system, and then captures the contents of the stopped kernel’s memory as a crash dump (vmcore) and saves it. The vmcore crash dump can help with finding the cause of the malfunction.

Enabling the Kdump feature is highly recommended because a crash dump might be the only information source that's available if a system failure occurs. Kdump is vital in many mission-critical environments.

Before enabling Kdump, ensure that the system meets all the memory requirements for using Kdump. To capture a kernel crash dump and save it for further analysis, reserve part of the system's memory permanently for that purpose. When you do so, that part of the system's memory is no longer be available to the main kernel.

The following table lists the minimum amount of reserved memory that's required to use Kdump, based on the system's architecture and the amount of available memory.

Table 7-1 Kdump Memory Requirements

Architecture	Available Memory	Minimum Reserved Memory
`x86_64`	1 GB to 64 GB	160 MB of RAM
	64 GB to 1 TB	256 MB of RAM
	1 TB and more	512 MB of RAM
Arm (aarch64)	2 TB and more	512 MB of RAM

For information about configuring Kdump by using the Cockpit web console, see Oracle Linux: Using the Cockpit Web Console

Note:

Kdump can also be used for troubleshooting in a cluster setup that uses the OCFS2 file system. For more information, see Configuring the Behavior of Fenced Nodes With Kdump in Oracle Linux 9: Managing Shared File Systems.

Installing Kdump

During an Oracle Linux interactive installation with the graphical installer, you have the option to enable Kdump and specify how much system memory is reserved for Kdump. The installer screen is titled Kdump and is available from the main Installation Summary screen of the installer.

If you don't enable Kdump at installation time, or it's not enabled by default during an installation, as in the case of a custom kickstart installation, you can install and enable the feature by using the command line.

Before you install and configure Kdump by using the command line, ensure that the system meets all the necessary memory specifications. For details, see Table 7-1.

Install the kdump package:
```
sudo dnf install kexec-tools
```
With the proper administrative privileges, edit the /etc/default/grub file and set the crashkernel= option to the required value.

For example, you would reserve 64 MB of memory as follows:
```
crashkernel=64M
```
You can also set the amount of reserved memory as a variable by using the following syntax: crashkernel=range1:size1,range2:size2.

For example, you might set the memory as a variable as follows:
```
crashkernel=512M-2G:64M,2G-:128M
```
(Optional) Consider defining an offset value for the reserved memory.

Because the crashkernel reservation occurs early in the boot process, some systems require that you reserve memory with a certain fixed offset. When a fixed offset is specified, the reserved memory begins at that point. For example, you would reserve 128 MB of memory, starting at 16 MB as follows:
```
crashkernel=128M@16M
```
If no offset value is set, Kdump offsets reserved memory automatically.
Refresh the grub configuration to apply changes:
```
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
```
Reboot the system and finish configuring Kdump.

For instructions, see Configuring the Kdump Output Location.

Enable the kdump service:

sudo systemctl enable --now kdump.service

Configuring Kdump

When you install and configure Kdump, the following files are changed:

/boot/grub2/grub.cfg: Appends the crashkernel option to the kernel line to specify the amount of reserved memory and any offset value.
/etc/kdump.conf: Sets the location in which the dump file can be written, the filtering level for the makedumpfile command, and the default behavior to take if the dump fails. See the comments in the file for information about the enabled parameters.

When you edit these files, you must reboot the system for the changes to take effect.

For more information, see the kdump.conf(5) manual page.

Configuring the Kdump Output Location

After installing Kdump, you can define the location in which the resulting output is saved. For Oracle Linux, Kdump files are stored in the /var/crash directory by default.

To save the result to other locations, such as NFS mounts, externally mounted drives, and remote file servers, edit the /etc/kdump.conf file and remove the # comment character at the beginning of each line that you want to enable.

For example, to add a new directory location, prefix it with the path keyword:

path /usr/local/cores

Use raw to output directly to a specific device in the /dev directory. You can also manually specify the output file system for a particular device by using its label, name, or UUID. For example:

ext4 UUID=5b065be6-9ce0-4154-8bf3-b7c4c7dc7365

Kdump files can also be transferred over a secure shell connection, as shown in the following example:

ssh user@example.com
sshkey /root/.ssh/mykey

You can also set the Kdump files to be exported to a compatible network share:

nfs example.com:/output

When you have finished configuring the output location for Kdump, enable the kdump service.

sudo systemctl enable --now kdump.service

Configuring the Default Kdump Failure State

By default, if kdump fails to send its result to the configured outlook locations, it reboots the server. This action deletes any data that has been collected for the dump. To prevent this outcome then you can uncomment and change the default value in the /etc/kdump.conf file as follows:

default dump_to_rootfs

The dump_to_rootfs option tries to save the result to a local directory, which can be useful if a network share is unreachable. You can use shell instead to copy the data manually from the command line.

Note:

The poweroff, restart, and haltoptions are also valid for the default kdump failure state. However, performing these actions causes you to lose the collected data if those actions are performed.

Analyzing Kdump Output

You can use the crash utility to analyze the contents of kdump core dumps in a shell prompt, which is useful when troubleshooting problems.

Install the crash package:
```
sudo dnf install crash
```
Identify the running kernel, for example:
```
uname -r
```
The output of the previous command is similar to the following:
```
4.18.0-80.el8.x86_64
```
Provide the location of the kernel debuginfo module and the location of the core dump as parameters to the crash utility, for example:
```
sudo crash /usr/lib/debug/lib/modules/4.18.0-80.el8.x86_64/vmlinux \
 /var/crash/127.0.0.1-2019-10-28-12:38:25/vmcore
```
In the previous command, 4.18.0-80.el8.x86_64 is the running kernel and 127.0.0.1-2019-10-28-12:38:25 represents the ipaddress-timestamp.
Inside the crash shell, use the help log command for information about how to use the log command.

You can also use the bt, ps, vm, and files commands to get more information about the core dump.
When you have finished analyzing the core dump, exit the shell.

For more detailed information about using the crash utility, see the crash(8) manual page.

Alternatively, you can use drgn to analyze core dumps. For more information, see Working With the drgn Kernel Debugging Utility.

Using Early Kdump

Early Kdump loads the crash kernel and initramfs early enough to capture vmcore information for early malfunctions.

Because the kdump service starts too late, early malfunctions don't trigger the kdump kernel to boot, which prevents the capture of diagnostic information. To address that problem, you can enable early Kdump by adding a dracut module so that the crash kernel and initramfs are loaded as early as possible.

Note:

The following limitations apply for early Kdump:

The feature doesn't work with Fadump.
Early Kdump becomes active the moment the system's initramfs begins to be processed. Any malfunction that occurs before that moment isn't captured even if early Kdump is enabled.

For more information about configuring early Kdump, see the step-by-step instructions in the /usr/share/doc/kexec-tools/early-kdump-howto.txt file.