7.2 Run-Time Problems

This section provides information to help you diagnose and solve problems that might occur when you run virtual machines (created by using Oracle JRockit Virtual Edition) on Oracle VM.

This section contains the following topics:

How to Diagnose and Solve Run-Time Problems
Fixing Problems in the File System of a Virtual Machine
Solutions to Specific Run-Time Problems
Frequently Asked Questions: Run-Time
Viewing the Virtual Machine Log File
Using Crash Files
Using Diagnostic Commands

7.2.1 How to Diagnose and Solve Run-Time Problems

This section describes a roadmap that you can follow to diagnose and solve Image Tool problems efficiently.

Verify whether you created the virtual machine as described in Chapter 3, "Creating Virtual Machines on Oracle VM."
Go to the Oracle VM Manager and check the status of the virtual machine.

If the status is Error, view the details of the virtual machine and click the link to the log file. The Oracle VM Manager User's Guide ("Troubleshooting" section) provides information to help you troubleshoot virtual machine problems by using log messages.
Fix problems in the file system of the virtual machine by following the procedure described in Section 7.2.2, "Fixing Problems in the File System of a Virtual Machine."
Check whether Section 7.2.3, "Solutions to Specific Run-Time Problems" contains the solution to the problem.
Check whether Section 7.2.4, "Frequently Asked Questions: Run-Time" contains the answer to your question.
Contact Oracle for help, as described in Section 7.3, "Contacting Oracle for Help."

7.2.2 Fixing Problems in the File System of a Virtual Machine

The virtual machine might crash due to problems in its file system. The problems could, for example, be caused by interruptions while executing Image Tool options (such as -f and -r) that access the file system, and result in the following error message:

EBUSY - Device or resource busy! (Filesystem already mounted!)

When the virtual machine restarts, Oracle JRockit Virtual Edition checks the file system automatically.

If the virtual machine cannot be restarted, you can find and fix file system problems by running the following Image Tool command.

Note:

Repairing a file system of a virtual machine image could lead to loss of data. So, before attempting to repair the file system, back up the contents in virtual machine disk by copying the virtual machine image.

java -jar jrockitve-imagetool.jar --repair vm_cfg [prompt_type]

In this command, vm_cfg is the path and name of the virtual machine configuration file (vm.cfg), and prompt_type indicates whether problems found should be fixed automatically (auto, the default value), not fixed at all (check), or fixed only after the user specifically confirms that each problem should be fixed (prompt).

For more information, see --repair in Appendix A, "Image Tool Command-Line Option Reference."

7.2.3 Solutions to Specific Run-Time Problems

ERROR: Configured IP [...] in use by MAC

When the IP address specified for the server is already in use, the following error message is displayed:

00:00:01 [net WRN] Configured IP ip_address in use by MAC: mac_address
00:00:02 [net WRN] Network stack initialization FAILED: 98

Use the -r option to verify the IP address. Make sure that the IP address is correct and none of the running virtual machines use the same address. If the problem persists, contact your system administrator to obtain a new IP address.

For more information about using the -r option to reconfigure a virtual machine, see Appendix A, "Image Tool Command-Line Option Reference."

My Virtual Machine Is Freezing

If you know the IP address of the virtual machine, ping it.

A failed ping indicates that the virtual machine is no longer running; it has crashed or hung (see "My Virtual Machine Has Crashed").

Note that the ping can also fail if network security features, such as a firewall, are enabled in your system or network.
A successful ping indicates that the network stack, scheduler, and other operating system functions are working.

Try to determine where the virtual machine is freezing by reviewing the thread dump. For information about generating a thread dump, see "How do I take thread dumps?".

My Virtual Machine Has Crashed

When a virtual machine crashes, the Oracle JRockit JVM creates a text crash file in the current working directory of the virtual machine; it also writes the crash file to stderr and to the virtual console log.

Review the crash file to determine the cause of the crash. For more information, see Section 7.2.5, "Using Crash Files."

Error: 2, 'Invalid kernel', 'xc_dom_parse_elf_kernel: corrupted ELF image\n'

When a virtual machine starts, Oracle VM copies the Oracle JRockit Virtual Edition ELF (executable and linking format) kernel image to the /var/run/xend/boot directory, and then reads the image. If the /var disk partition is full (for example, on account of frequent of core dumps in the /var/xen/dump directory), Oracle VM cannot copy the ELF kernel image to the /var/run/xend/boot directory; so it displays the error message.

Make sure that the /var disk partition has sufficient space.

7.2.4 Frequently Asked Questions: Run-Time

Where can I find crash files?

See "Location of the Oracle JRockit JVM and Oracle JRockit Virtual Edition Crash Files".

How do I take thread dumps?

A thread dump is a snapshot of the state of all threads that are part of a virtual machine image.

To take thread dumps in Oracle JRockit Virtual Edition, do one of the following:

Run the print_threads diagnostic command from Oracle JRockit Mission Control.
Press Ctrl+\ (or the 9 key) in the virtual machine console. This is equivalent to sending a SIGQUIT signal to the process in a Linux environment

If you cannot generate thread dumps, force the virtual machine to crash by using the xm dump-core command. Provide the core dump file to Oracle Support for further diagnosis and troubleshooting.

For more information about troubleshooting by using thread dumps, see the Oracle® JRockit Diagnostics and Troubleshooting Guide at http://download.oracle.com/docs/cd/E15289_01/doc.40/e15059/toc.htm.

How do I send a Diagnostic Command?

Use Oracle JRockit Mission Control.

For more information, see Section 7.2.6, "Using Diagnostic Commands."

Where should I store the ctrlhandler.act file?

Save the ctrlhandler.act file in the current working directory of the file system of the virtual machine. For information about finding out the current working directory, see "How do I find out the current working directory of a virtual machine image?".

How do I shut down the virtual machine from the virtual console?

Press the F5 (or 5) key and then press Ctrl+c in the virtual machine console. This is equivalent to sending a SIGINT signal.

How do I send a SIGABRT signal?

Oracle JRockit Virtual Edition does not provide an option to send a SIGABRT signal.

Use the xm dump-core command to generate a core dump.

How do I find out the current working directory of a virtual machine image?

You can find out the current working directory by running the following command:

Caution:

You must shut down the virtual machine before running this command.

java -jar jrockitve-imagetool.jar -r vm_cfg get working-dir

In this command, vm_cfg is the path and name of the virtual machine configuration file (vm.cfg).

7.2.5 Using Crash Files

When the JVM crashes, Oracle JRockit Virtual Edition creates a text crash file (.dump) and a binary crash file. Note that if the Oracle JRockit Virtual Edition kernel (rather than the JVM) notices the crash first, then the crash files are not generated.

For more information about crash files and crash file sizing, see the Oracle® JRockit Diagnostics and Troubleshooting Guide at http://download.oracle.com/docs/cd/E15289_01/doc.40/e15059/toc.htm.

Location of the Oracle JRockit JVM and Oracle JRockit Virtual Edition Crash Files

The name of the Oracle JRockit text crash file is jrockit.4711.dump. It is written to the current working directory in the file system of the virtual machine.

For information about finding out the current working directory, see "How do I find out the current working directory of a virtual machine image?"

Note:
Oracle JRockit Virtual Edition does not support using the environment variable JROCKIT_DUMP_PATH to specify a different location for the text crash file.
The Oracle JRockit Virtual Edition binary crash file is written to the /var/xen/dump directory on the Oracle VM Server.

Enabling Creation of Crash Files on Oracle VM

The virtual machine configuration file (vm.cfg) contains a parameter on_crash, which defines the action to be performed in the event of a system crash, as shown in the following example.

name="wls-ve"
bootloader="/usr/bin/pygrub"
memory=256
disk=['tap:aio:/OVS/seed_pool/wls-ve/system.img,sda1,w']
vif=['']
on_crash="coredump-destroy"

For a virtual machine image created by using Oracle JRockit Virtual Edition, the possible values for the on_crash parameter are restart, destroy, coredump-restart and coredump-destroy.

Note:

For crash files to be written, sufficient disk space must exist in the /var partition on the Oracle VM Server.

Disabling Creation of Crash Files

To disable creation of the text crash file, use the -XXnoJrDump option.
To disable creation of the binary crash file, change the value of the on_crash parameter in the vm.cfg file.

Caution:

Changing the on_crash parameter in the vm.cfg file renders the product unsupported.

7.2.6 Using Diagnostic Commands

You can send diagnostic commands to a running virtual machine in one of the following ways:

By using the Ctrl-Break handler from Oracle JRockit Mission Control.
By using the JRockit Management Console in Oracle JRockit Mission Control.

Note:

The default Ctrl-Break handler (triggered by pressing Ctrl+\) is the print_threads command. However, if the ctrlhandler.act file is available in the current working directory, the diagnostic commands listed in the file are executed.

For more information about using the Ctrl-Break handler and JRockit Management Console, see the Oracle® JRockit JDK Tools Guide at http://download.oracle.com/docs/cd/E15289_01/doc.40/e15061/toc.htm.

7.2.7 Viewing the Virtual Machine Log File

The log messages generated by Oracle JRockit Virtual Edition are useful for diagnosing and troubleshooting run-time problems.

The messages written to the standard output streams (stdout and stderr) are also written to a log file, which, by default, is /jrockitve/log/jrockitve.log within the file system of the virtual machine image. You can change the log file name and location by using the -r vm_cfg set console-log-path operation of the Image Tool.

Note:

Oracle VM Server logs are in the /var/log/xen directory of the Oracle VM server. The log file is named xend.log. It contains only Oracle VM server-related messages.

To view the contents of the virtual machine log file (if it exists within the file system of the virtual machine), shut down the virtual machine and then run the following command:

java -jar jrockitve-imagetool.jar --get-log vm_cfg [output-file]

In this command, vm_cfg is the path and name of the virtual machine configuration file (vm.cfg), and output-file is the path and name of the file to which you want the contents of the log file to be written. If you do not specify the output file, the output of the command is printed to stdout.