5.1. Procedures and Tools

5.1.1. Categorizing and Isolating Problems

More often than not, a virtualized guest behaves like a physical system. Any problems that a physical machine would encounter, a virtual machine will encounter as well. If, for example, Internet connectivity is lost due to external issues, virtual machines will be affected just as much as physical ones.

If a true Oracle VM VirtualBox problem is encountered, it helps to categorize and isolate the problem first. Here are some of the questions that should be answered before reporting a problem:

  • Is the problem specific to a certain guest OS? Or a specific release of a guest OS? Especially with Linux guest related problems, the issue may be specific to a certain distribution and version of Linux.

  • Is the problem specific to a certain host OS? Problems are usually not host OS specific, because most of the Oracle VM VirtualBox code base is shared across all supported platforms, but especially in the areas of networking and USB support, there are significant differences between host platforms. Some GUI related issues are also host specific.

  • Is the problem specific to certain host hardware? This category of issues is typically related to the host CPU. Because of significant differences between VT-x and AMD-V, problems may be specific to one or the other technology. The exact CPU model may also make a difference, even for software virtualization, because different CPUs support different features, which may affect certain aspects of guest CPU operation.

  • Is the problem specific to a certain virtualization mode? Some problems may only occur in software virtualization mode, others may be specific to hardware virtualization.

  • Is the problem specific to guest SMP? That is, is it related to the number of virtual CPUs (VCPUs) in the guest? Using more than one CPU usually significantly affects the internal operation of a guest OS.

  • Is the problem specific to the Guest Additions? In some cases, this is obvious, such as a shared folders problem. In other cases such as display problems, it may be less obvious. If the problem is Guest Additions specific, is it also specific to a certain version of the Guest Additions?

  • Is the problem specific to a certain environment? Some problems are related to a particular environment external to the VM. This usually involves network setup. Certain configurations of external servers such as DHCP or PXE may expose problems which do not occur with other, similar servers.

  • Is the problem a regression? Knowing that an issue is a regression usually makes it significantly easier to find the solution. In this case, it is crucial to know which version is affected and which is not.

5.1.2. Collecting Debugging Information

For problem determination, it is often important to collect debugging information which can be analyzed by Oracle VM VirtualBox support. This section contains information about what kind of information can be obtained.

Every time Oracle VM VirtualBox starts up a VM, a so-called release log file is created, containing lots of information about the VM configuration and runtime events. The log file is called VBox.log and resides in the VM log file folder, which is $HOME/VirtualBox VMs/VM-name/Logs by default.

When starting a VM, the configuration file of the last run will be renamed to .1, up to .3. Sometimes when there is a problem, it is useful to have a look at the logs. Also when requesting support for Oracle VM VirtualBox, supplying the corresponding log file is mandatory.

For convenience, for each virtual machine, the VirtualBox Manager window can show these logs in a window. To access it, select a virtual machine from the list on the left and select Show Log from the Machine menu.

The release log file, VBox.log, contains a wealth of diagnostic information, such as Host OS type and version, Oracle VM VirtualBox version and build. It also includes a complete dump of the guest's configuration (CFGM), detailed information about the host CPU type and supported features, whether hardware virtualization is enabled, information about VT-x/AMD-V setup, state transitions (such as creating, running, paused, stopping), guest BIOS messages, Guest Additions messages, device-specific log entries and, at the end of execution, final guest state and condensed statistics.

In case of crashes, it is very important to collect crash dumps. This is true for both host and guest crashes. For information about enabling core dumps on Linux, Oracle Solaris, and Mac OS X systems, refer to the following core dump article on the Oracle VM VirtualBox website:

http://www.virtualbox.org/wiki/Core_dump.

You can also use VBoxManage debugvm to create a dump of a complete virtual machine. See VBoxManage debugvm.

For network related problems, it is often helpful to capture a trace of network traffic. If the traffic is routed through an adapter on the host, it is possible to use Wireshark or a similar tool to capture the traffic there. However, this often also includes a lot of traffic unrelated to the VM.

Oracle VM VirtualBox provides an ability to capture network traffic only on a specific VM's network adapter. Refer to the following network tracing article on the Oracle VM VirtualBox website for information on enabling this capture:

http://www.virtualbox.org/wiki/Network_tips.

The trace files created by Oracle VM VirtualBox are in .pcap format and can be easily analyzed with Wireshark.

5.1.3. Using the VBoxBugReport Command to Collect Debug Information Automatically

The VBoxBugReport command is used to collect debug information automatically for an Oracle VM VirtualBox installation. This command can be useful when you need to gather information to send to Oracle Support.

The following examples show how to use VBoxBugReport.

By default, the command collects VBoxSVC process logs, device settings, and global configuration data for an Oracle VM VirtualBox host. 

$ VBoxBugReport
  ...
  0% - collecting VBoxSVC.log.10...
  7% - collecting VBoxSVC.log.9...
  ...
 64% - collecting VBoxSVC.log.1...
 71% - collecting VBoxSVC.log...
 78% - collecting VirtualBox.xml...
 85% - collecting HostUsbDevices...
 92% - collecting HostUsbFilters...
100% - compressing...

Report was written to '2019-03-26-13-32-02-bugreport.tgz'

The results are saved as a compressed tar file archive in the same directory where the command is run.

To specify a different output file location: 

$ VBoxBugReport --output ~/debug/bug004.tgz

To output all debug information to a single text file, rather than a tgz file: 

$ VBoxBugReport --text

To collect information for a specific VM, called Windows_10: 

$ VBoxBugReport Windows_10

This command collects machine settings, guest properties, and log files for the specified VM. Global configuration information for the host is also included.

To collect information for several VMs, called Windows_7, Windows_8, and Windows_10: 

$ VBoxBugReport Windows_7 Windows_8 Windows_10

To collect information for all VMs: 

$ VBoxBugReport --all

To show a full list of the available command options, run VBoxBugReport --help.

5.1.4. The Built-In VM Debugger

Oracle VM VirtualBox includes a built-in VM debugger, which advanced users may find useful. This debugger enables you to examine and, to some extent, control the VM state.

Warning

Use the VM debugger at your own risk. There is no support for it, and the following documentation is only made available for advanced users with a very high level of familiarity with the x86/AMD64 machine instruction set, as well as detailed knowledge of the PC architecture. A degree of familiarity with the internals of the guest OS in question may also be very helpful.

The VM debugger is available in all regular production versions of Oracle VM VirtualBox, but it is disabled by default because the average user will have little use for it. There are two ways to access the debugger:

  • Using a debugger console window displayed alongside the VM

  • Using the telnet protocol on port 5000

The debugger can be enabled in the following ways:

  • Start the VM directly using VirtualBoxVM --startvm, with an additional --dbg, --debug, or --debug-command-line argument. See the VirtualBoxVM --help command usage help for details.

  • Set the VBOX_GUI_DBG_ENABLED or VBOX_GUI_DBG_AUTO_SHOW environment variable to true before launching the Oracle VM VirtualBox process. Setting these variables, only their presence is checked, is effective even when the first Oracle VM VirtualBox process is the VM selector window. VMs subsequently launched from the selector will have the debugger enabled.

  • Set the GUI/Dbg/Enabled extra data item to true before launching the VM. This can be set globally or on a per VM basis.

A new Debug menu entry is added to the Oracle VM VirtualBox application. This menu enables the user to open the debugger console.

The VM debugger command syntax is loosely modeled on Microsoft and IBM debuggers used on DOS, OS/2, and Windows. Users familiar with symdeb, CodeView, or the OS/2 kernel debugger will find the Oracle VM VirtualBox VM debugger familiar.

The most important command is help. This will print brief usage help for all debugger commands. The set of commands supported by the VM debugger changes frequently and the help command is always up-to-date.

A brief summary of frequently used commands is as follows:

  • stop: Stops the VM execution and enables single stepping

  • g: Continue VM execution

  • t: Single step an instruction

  • rg, rh, and r: Print the guest, hypervisor, and current registers

  • kg, kh, and k: Print the guest, hypervisor, and current call stack

  • da, db, dw, dd, dq: Print memory contents as ASCII, bytes, words, dwords, and qwords

  • u: Unassemble memory

  • dg: Print the guest's GDT

  • di: Print the guest's IDT

  • dl: Print the guest's LDT

  • dt: Print the guest's TSS

  • dp*: Print the guest's page table structures

  • bp and br: Set a normal and recompiler breakpoint

  • bl: List breakpoints

  • bc: Clear a breakpoint

  • writecore: Write a VM core file to disk. See Section 5.1.5, “VM Core Format”

See the built-in help for other available commands.

The VM debugger supports symbolic debugging, although symbols for guest code are often not available. For Oracle Solaris guests, the detect command automatically determines the guest OS version and locates kernel symbols in guest's memory. Symbolic debugging is then available. For Linux guests, the detect commands also determines the guest OS version, but there are no symbols in the guest's memory. Kernel symbols are available in the file /proc/kallsyms on Linux guests. This file must be copied to the host, for example using scp. The loadmap debugger command can be used to make the symbol information available to the VM debugger. Note that the kallsyms file contains the symbols for the currently loaded modules. If the guest's configuration changes, the symbols will change as well and must be updated.

For all guests, a simple way to verify that the correct symbols are loaded is the k command. The guest is normally idling and it should be clear from the symbolic information that the guest operating system's idle loop is being executed.

Another group of debugger commands is the set of info commands. Running info help provides complete usage information. The information commands provide ad-hoc data pertinent to various emulated devices and aspects of the VMM. There is no general guideline for using the info commands, the right command to use depends entirely on the problem being investigated. Some of the info commands are as follows:

  • cfgm: Print a branch of the configuration tree

  • cpuid: Display the guest CPUID leaves

  • ioport: Print registered I/O port ranges

  • mmio: Print registered MMIO ranges

  • mode: Print the current paging mode

  • pit: Print the i8254 PIT state

  • pic: Print the i8259A PIC state

  • ohci, ehci, xhci: Print a subset of the OHCI, EHCI, and xHCI USB controller state

  • pcnet0: Print the PCnet state

  • vgatext: Print the contents of the VGA framebuffer formatted as standard text mode

  • timers: Print all VM timers

The output of the info commands generally requires in-depth knowledge of the emulated device or Oracle VM VirtualBox VMM internals. However, when used properly, the information provided can be invaluable.

5.1.5. VM Core Format

Oracle VM VirtualBox uses the 64-bit ELF format for its VM core files created by VBoxManage debugvm, see VBoxManage debugvm. The VM core file contain the memory and CPU dumps of the VM and can be useful for debugging your guest OS. The 64-bit ELF object format specification can be obtained at:

http://downloads.openwatcom.org/ftp/devel/docs/elf-64-gen.pdf.

The overall layout of the VM core format is as follows: 

[ ELF 64 Header]
[ Program Header, type PT_NOTE ]
  → offset to COREDESCRIPTOR
[ Program Header, type PT_LOAD ] - one for each contiguous physical memory range
  → Memory offset of range
  → File offset
[ Note Header, type NT_VBOXCORE ]
[ COREDESCRIPTOR ]
  → Magic
  → VM core file version
  → VBox version
  → Number of vCPUs etc.
[ Note Header, type NT_VBOXCPU ] - one for each vCPU
[ vCPU 1 Note Header ]
  [ DBGFCORECPU - vCPU 1 dump ]
[ Additional Notes + Data ] - currently unused
[ Memory dump ]

The memory descriptors contain physical addresses relative to the guest and not virtual addresses. Regions of memory such as MMIO regions are not included in the core file.

The relevant data structures and definitions can be found in the Oracle VM VirtualBox sources under the following header files: include/VBox/dbgfcorefmt.h, include/iprt/x86.h and src/VBox/Runtime/include/internal/ldrELFCommon.h.

The VM core file can be inspected using elfdump and GNU readelf or other similar utilities.

Copyright © 2004, 2020 Oracle and/or its affiliates. All rights reserved. Legal Notices