This chapter describes how to monitor and tune Java Virtual Machine (JVM) performance for Oracle Communications Converged Application Server engine servers.
Java Flight Recorder and Java Mission Control together create a complete tool chain to continuously collect low level and detailed runtime information enabling after-the-fact incident analysis.
Java Flight Recorder is a profiling and event collection framework built into the Oracle JDK that lets Converged Application Server administrators and developers to gather detailed low level information about how the Java Virtual Machine (JVM) and the Java application are behaving.
Java Mission Control is an advanced set of tools that enables efficient and detailed analysis of the extensive of data collected by Java Flight Recorder. The tool chain enables developers and administrators to collect and analyze data from Java applications running locally or deployed in production environments.
The jcmd utility is used to send diagnostic command requests to the JVM. It must be used on the same machine on which the JVM is running, and have the same effective user and group identifiers that were used to launch the JVM.
You can run multiple recordings concurrently and configure each recording using different settings; in particular, you can configure different recordings to capture different sets of events. However, in order to make the internal logic of Java Flight Recorder as streamlined as possible, the resulting recording always contains the union of all events for all recordings active at that time. This means that if more than one recording is running, you might end up with more information in the recording than you wanted. This can be confusing but has no other negative implications.
The easiest and most intuitive way to use JFR is through the Flight Recorder plug-in that is integrated into Java Mission Control. This plug-in enables access to JFR functionality through an intuitive GUI. For more information about using the JMC client to control JFR, see the Flight Recorder Plug-in section of the Java Mission Control help.
You can start and configure a recording from the command line using the -XX:StartFlightRecording option of the java command, when starting the application. To enable the use of JFR, specify the -XX:+FlightRecorder option. Because JFR is a commercial feature, you also have to specify the -XX:+UnlockCommercialFeatures option. The following example illustrates how to run the MyApp application and immediately start a 60-second recording which will be saved to a file named myrecording.jfr:
java -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:StartFlightRecording=duration=60s,filename=myrecording.jfr MyApp
To configure JFR, you can use the -XX:FlightRecorderOptions option.
You can also control recordings by using Java-specific diagnostic commands.
The simplest way to execute a diagnostic command is to use the jcmd tool (located in the Java installation directory). To issue a command, you have to pass the process identifier of the JVM (or the name of the main class) and the actual command as arguments to jcmd. For example, to start a 60-second recording on the running Java process with the identifier 5368 and save it to myrecording.jfr in the current directory, use the following:
jcmd 5368 JFR.start duration=60s filename=myrecording.jfr
To see a list of all running Java processes, run the jcmd command without any arguments. To see a complete list of commands available to a running Java application, specify help as the diagnostic command after the process identifier (or the name of the main class). The commands relevant to Java Flight Recorder are:
JFR.start
Start a recording.
JFR.check
Check the status of all recordings running for the specified process, including the recording identification number, file name, duration, and so on.
JFR.stop
Stop a recording with a specific identification number (by default, recording 1 is stopped).
JFR.dump
Dump the data collected so far by the recording with a specific identification number (by default, data from recording 1 is dumped).
Note:
These commands are available only if the Java application was started with the Java Flight Recorder enabled, that is, using the following options:-XX:+UnlockCommercialFeatures -XX:+FlightRecorder
You can configure an explicit recording in a number of other ways. These techniques work the same regardless of how you start a recording (that is, either by using the command-line approach or by using diagnostic commands).
You can configure an explicit recording to have a maximum size or age by using the following parameters:
maxsize=size
Append the letter k or K to indicate kilobytes, m or M to indicate megabytes, g or G to indicate gigabytes, or do not specify any suffix to set the size in bytes.
maxage=age
Append the letter s to indicate seconds, m to indicate minutes, h to indicate hours, or d to indicate days.
If both a size limit and an age are specified, the data is deleted when either limit is reached.
When scheduling a recording. you might want to add a delay before the recording is actually started; for example, when running from the command line, you might want the application to boot or reach a steady state before starting the recording. To achieve this, use the delay parameter:
delay=delay
Append the letter s to indicate seconds, m to indicate minutes, h to indicate hours, or d to indicate days.
Although the recording file format is very compact, you can compress it further by adding it to a ZIP archive. To enable compression, use the following parameter:
compress=true
Note that CPU resources are required for the compression, which can negatively impact performance.
When running with a default recording you can configure Java Flight Recorder to automatically save the current in-memory recording data to a file whenever certain conditions occur. If a disk repository is used, the current information in the disk repository will also be included.
To save the recording data to the specified path every time the JVM exits, start your application with the following option:
-XX:FlightRecorderOptions=defaultrecording=true,dumponexit=true,dumponexitpath=path
Set path to the location where the recording should be saved. If you specify a directory, a file with the date and time as the name is created in that directory. If you specify a file name, that name is used. If you do not specify a path, the recording will be saved in the current directory.
Creating a Recording Using Triggers
You can use the Console in Java Mission Control to set triggers. A trigger is a rule that executes an action whenever a condition specified by the rule is true. For example, you can create a rule that triggers a flight recording to commence whenever the heap size exceeds 100 MB. Triggers in Java Mission Control can use any property exposed through a JMX MBean as the input to the rule. They can launch many other actions than just Flight Recorder dumps.
Define triggers on the Triggers tab of the JMX Console. For more information on how to create triggers, see the Java Mission Control help.
You can collect a significant amount of diagnostic information from Java Flight Recorder by starting the JVM with one of the following options:
-XX:FlightRecorderOptions=loglevel=debug
-XX:FlightRecorderOptions=loglevel=trace.
For a listing of commands you can use with the Java Flight Recorder, see "Command Reference" in Java Platform, Standard Edition Java Flight Recorder Runtime Guide.
This section describes using Java Mission Control.
The JMC client executable file is located in the bin directory of the Java SE Development Kit (JDK) installation path (JAVA_HOME). If the JAVA_HOME/bin directory is in the PATH environment variable, you can start the JMC client by entering jmc at the command-line prompt (shell). Otherwise, you have to specify the full path to the JMC executable:
JAVA_HOME/bin/jmc (Linux)
Passing JVM Options To the JMC Launcher
JMC is a Java application, and the JMC client executable is a launcher for this application. JMC startup is controlled by options specified in the jmc.ini file, which is located in the JAVA_HOME/bin directory. Arguments to the -vmargs option in the jmc.ini file are options that are passed to the JVM running the JMC application. You can specify these options to control the way this JVM runs. If you do not want to modify the jmc.ini file, you can specify JVM options on the command line as arguments to the -vmargs option of the jmc command.
Note:
If other options are specified for thejmc command, the -vmargs option must be specified last.To start the JMC client with your own set of JVM options (overriding those specified in the jmc.ini file), run the following command (separate multiple arguments with spaces):
jmc -vmarg arguments
To start the JMC client with additional JVM options (appending them to those specified in the jmc.ini file), run the following command (separate multiple arguments with spaces):
jmc --launcher.appendVmargs -vmarg arguments
If you want to copy your settings for the JMC client to another computer or another user, or use different predefined settings for different applications, add the -data command-line option and define a workspace directory when you start the JMC client:
jmc -data workspace-directory
For detailed information on using the Java Mission Control GUI client, see "Java Mission Control Client GUI" in Java Platform, Standard Edition Java Mission Control User's Guide.
You can use the Java utility, jcmd, to diagnostic command requests directly to the JVM. For detailed information on using the jcmd utility, see "The jcmd Utility" in Java Platform, Standard Edition Troubleshooting Guide.
To create a heap dump using jcmd execute the following command, replacing Process_ID with the process ID of your JVM process and specifying a path and filename for the output file:
jcmd Process_ID GC.heap_sump /path/filename
Note:
JVM heap dumps are generated in the HPROF binary format. You can analyze HPROF heap dumps using the jhat utility. For more information see "jhat" in Java Platform, Standard Edition Tools Reference.To create a thread dump using jcmd execute the following command, replacing Process_ID with the process ID of your JVM process:
jcmd Process_ID Thread.print
Example 16-2 Creating a Thread Dump
jcmd 5216 Thread.print 5216: 2014-09-19 13:12:30 Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.45-b08 mixed mode): "Thread-21" daemon prio=6 tid=0x0000000016109800 nid=0x1d5c in Object.wait() [0x000000001c22f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method)
This section provides example jcmd commands.
Run jcmd without any parameters (or with -l) to list all JVM processes preceded by a process ID:
jcmd -l 6848 8120 sun.tools.jcmd.JCmd -l 3108 weblogic.Server
List jcmd Commands for a Particular Process
Run jcmd PID help to list the jcmd commands available for that process. Replace PID with the process ID of your JVM process:
jcmd 3108 help The following commands are available: VM.native_memory VM.commercial_features ManagementAgent.stop ManagementAgent.start_local ManagementAgent.start Thread.print GC.class_histogram GC.heap_dump GC.run_finalization GC.run VM.uptime VM.flags VM.system_properties VM.command_line VM.version help
Get More Information on a jcmd Command
To get more information on a jcmd command run jcmd help command_name where command_name is the name of the jcmd command:
jcmd help GC.heap_dump
GC.heap_dump
Generate a HPROF format dump of the Java heap.
Impact: High: Depends on Java heap size and content. Request a full GC
unless the '-all' option is specified.
Syntax : GC.heap_dump [options] <filename>
Arguments:
filename : Name of the dump file (STRING, no default value)
Options: (options must be specified using the <key> or <key>=<value> syntax)
-all : [optional] Dump all objects, including unreachable objects
(BOOLEAN, false)
For a listing of commands you can use with the jcmd utility, see "Command Reference" in Java Platform, Standard Edition Java Flight Recorder Runtime Guide.
On Oracle Solaris or Linux operating systems, the combination of pressing the Control key and the backslash (\) key at the application console (standard input) causes the Java HotSpot VM to print a thread dump to the application's standard output. On Windows, the equivalent key sequence is the Control and Break keys. The general term for these key combinations is the Control+Break handler.
On Oracle Solaris and Linux operating systems, a thread dump is printed if the Java process receives a QUIT signal. Therefore, the kill -QUIT pid command causes the process with the ID pid to print a thread dump to standard output.
For details on Control+Break output see the following sections in the Java Platform, Standard Edition Troubleshooting Guide:
This Section describes how to tune Java Virtual Machine (JVM) garbage collection performance for Oracle Communications Converged Application Server engine servers.
Production installations of Converged Application Server generally require extremely small response times (under 50 milliseconds) for clients even under peak server loads. A key factor in maintaining brief response times is the proper selection and tuning of the JVM's Garbage Collection (GC) algorithm for Converged Application Server instances.
Whereas certain tuning strategies are designed to yield the lowest average garbage collection times or to minimize the frequency of full GCs, those strategies can sometimes result in one or more very long periods of garbage collection (often several seconds long) that are offset by shorter GC intervals. With a production Converged Application Server installation, all long GC intervals must be avoided to maintain response time goals.
The sections that follow describe GC tuning strategies for Oracle's JVM that generally result in best response time performance.
If you use custom startup scripts to start Converged Application Server engines and replicas, simply edit those scripts to include the recommended JVM options described in the sections that follow.
The Configuration Wizard also installs default startup scripts when you configure a new domain. by default, these scripts are installed in the Middleware_Home/user_projects/domains/domain_name/bin directory, where Middleware_Home is where you installed the Converged Application Server software and domain_name is the name of the domain's directory. The /bin directory includes:
startWebLogic.cmd, startWebLogic.sh: These scripts start the Administration Server for the domain. The also contain a variety of Java configuration settings.
startManagedWebLogic.cmd, startManagedWebLogic.sh: These scripts start managed engines and replicas in the domain.
If you use the Oracle-installed scripts to start engines and replicas, you can override JVM memory arguments by first setting the USER_MEM_ARGS environment variable in your command shell.
Note:
Setting the USER_MEM_ARGS environment variable overrides all default JVM memory arguments specified in the Oracle-installed scripts. Always set USER_MEM_ARGS to the full list of JVM memory arguments you intend to use.When using Oracle's JDK, the goal in tuning garbage collection performance is to reduce the time required to perform a full garbage collection cycle. You should not attempt to tune the JVM to minimize the frequency of full garbage collections, because this generally results in an eventual forced garbage collection cycle that may take up to several full seconds to complete.
The simplest and most reliable way to achieve short garbage collection times over the lifetime of a production server is to use a fixed heap size with the collector and the parallel young generation collector, restricting the new generation size to at most one third of the overall heap.
Oracle recommends using the Garbage-First (G1) garbage collector. See "Getting Started with the G1 Garbage Collector" for more information on using the Garbage-First collector.
The following example JVM settings are recommended for most production engine servers:
-server -Xms24G -Xmx24G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=20 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70
For standalone installations, use the example settings:
-server -Xms32G -Xmx32G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=20 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70
The above options have the following effect:
-Xms, -Xmx: Places boundaries on the heap size to increase the predictability of garbage collection. The heap size is limited in replica servers so that even Full GCs do not trigger SIP retransmissions. -Xms sets the starting size to prevent pauses caused by heap expansion.
-XX:+UseG1GC: Use the Garbage First (G1) Collector.
-XX:MaxGCPauseMillis: Sets a target for the maximum GC pause time. This is a soft goal, and the JVM will make its best effort to achieve it.
-XX:ParallelGCThreads: Sets the number of threads used during parallel phases of the garbage collectors. The default value varies with the platform on which the JVM is running.
-XX:ConcGCThreads: Number of threads concurrent garbage collectors will use. The default value varies with the platform on which the JVM is running.
-XX:InitiatingHeapOccupancyPercent: Percentage of the (entire) heap occupancy to start a concurrent GC cycle. GCs that trigger a concurrent GC cycle based on the occupancy of the entire heap and not just one of the generations, including G1, use this option. A value of 0 denotes 'do constant GC cycles'. The default value is 45.
The library used for random number generation in Oracle's JVM relies on /dev/random by default for UNIX platforms. This can potentially block the Converged Application Server process because on some operating systems /dev/random waits for a certain amount of "noise" to be generated on the host system before returning a result.
To determine if your operating system exhibits this behavior, try displaying a portion of the file from a shell prompt:
head -n 1 /dev/random
If the command returns immediately, you need not continue. If the command does not return immediately, configure the rngd daemon to feed data to the kernel's random number entropy pool:
rngd -r /dev/urandom -o /dev/random -f -t .1
Note:
You may have to experiment with the value of the -t parameter. For more information on the rngd daemon, run the man rngd command from a shell to display this manual page.If your application's execution time becomes longer and longer, or if the operating system seems to be performing slower and slower, this could be an indication of a memory leak. In other words, virtual memory is being allocated but is not being returned when it is no longer needed. Eventually the application or the system runs out of memory, and the application terminates abnormally.
For more information on diagnosing Java memory leaks, see "Troubleshooting Memory Leaks" in Java Platform, Standard Edition Troubleshooting Guide.
In addition, you can also use the Eclipse Memory Analyzer Tool (MAT) during development to discover memory leaks as well as reduce memory consumption.
For details on the Eclipse MAT, see http://www.eclipse.org/mat/.