You can use the following Oracle and third-party tools to identify potential performance bottlenecks:
These tools can be particularly useful if your deployment uses custom Java classes.
The DTrace facility is a dynamic tracing framework for the Solaris 10 operating system that enables you to monitor JVM activity.
DTrace contains more than 30,000 probes and uses integrated user-level and kernel-level tracing to give you a view into your production system. You can also trace arbitrary data and expressions by using the D language, which is similar to C or awk. The DTrace facility also includes special support for monitoring the JVM, and enables you to watch your whole system and span outside the JVM.
DTrace is easiest to use with Java 6 because probes are built into the JVM. The facility also works with Java 1.4 and Java 5, but you must download JVM PI or JVM TI agents from the following URL:
https://solaris10-dtrace-vm-agents.dev.java.net/
The following example shows how to write a DTrace script.
#!/usr/sbin/dtrace -Zs #pragma D option quiet hotspot$1::: { printf("%s\n", probename); } |
In this example, you would replace $1 with the first argument to the script, which is the PID of the Java process you want to monitor. For example:
# ./all-jvm-probes.d 1234
The following table describes the commands you can use to enable different DTrace probes.
Table 4–3 DTrace Commands
Because DTrace causes additional work in the system, enabling this facility affects system performance. The effect is often negligible, but can become substantial if you enable many probes with costly enablings.
Instructions for minimizing the performance effect of DTrace are provided in the “Performance Considerations” chapter of the Solaris Dynamic Tracing Guide.
For more information about DTrace, see /usr/demo/dtrace and man dtrace.
Waveset enables you to use Java Management Extensions (JMX) to capture and expose operational statistics for certain resource adapter operations. You can use this data for diagnostic and predictive purposes, such as to monitor system health and reports.
This statistical data includes the following:
The number of times the action was performed
The minimum, maximum, and average duration of the operations
Objects |
Actions Monitored |
---|---|
For Accounts |
|
For Actions |
Run |
For Other Objects |
|
JMX creates MBeans for each resource adapter, by server, and registers these beans with names that match the following pattern:
serverName=server name, resourceAdapterType=Resource Adapter Type, resourceAdapterName=Resource Adapter Name |
Waveset records statistics for all completed operations, whether they completed successfully or with errors. However, Waveset does not record statistics for incomplete operations, such as any operations that throw exceptions.
You can configure excludes as follows:
From the Administrator interface, select Configure -> Servers.
On the Configure Servers page, perform one of the following tasks:
Click the Edit Default Server Settings button to edit the default server settings.
Click a server link to edit the policy for that server.
Click the JMX tab and enable the JMX Enable Resource Adapter Monitor box to turn on resource monitoring.
To exclude specific resources, add regular expressions to the JMX Resource Adapter Monitor Excludes list.
To exclude monitoring specific actions, add regular expressions to the JMX Resource Adapter Monitor Operation Excludes list.
All excludes use regular expressions. For excluding certain resources, JMX just matches on the resource name. For example, if you have adapters named
resource1 resource2 resource3 resource10 resource11 |
and you specify the following pattern
.*1$ |
which means, match any 0 or more of any character (.*) until something that ends with a 1 (1$). JMX will exclude resource1 and resource11.
For operations, the process is similar. If your operations have the following names, the patterns must match against those names.
ACCOUNT_CREATE ACCOUNT_UPDATE ACCOUNT_DELETE ACCOUNT_GET ACCOUNT_AUTHENTICATE OBJECT_CREATE OBJECT_UPDATE OBJECT_DELETE OBJECT_GET OBJECT_LIST ACTION_RUN |
For example, the ^ACCOUNT.* pattern excludes all operations that start with ACCOUNT. Or, using this pattern excludes updates and deletes:
.*UPDATE$ .*DELETE$ |
For more information about configuring and using JMX, see Configuring JMX Monitoring and The JMX Publisher Type in Oracle Waveset 8.1.1 Business Administrator’s Guide.
Waveset supplies some JMX MBeans that provide information about the following:
Data Exporter
Object Repository
Performance
Reconciliation
Resource Adapters and Connectors
Scheduler
Waveset Server Cluster
There are nine MBeans in the Performance category:
DataExporter
DataQueue
FormConverter
ObjectChangeNotification
Reconcile
Ruler
TaskInstanceCache
ViewMaster
WorkItemCache
These MBeans can capture the same performance data that is captured by the Waveset Profiler, but the MBeans have a lower runtime performance cost.
You enable the Profiler on a per-server basis. When enabled, the Profiler impacts both memory (to capture and hold the performance data) and performance.
Performance MBeans are disabled by default. When enabled, MBeans have a significantly lower operational impact, however they capture very specific performance data.
When enabled, the FormConverter and ViewMaster MBeans each have a configurable threshold and when processing takes longer than the threshold, a JMX Notification is produced. This notification indicates which view or form element was involved and how long it took to process that element. If there are no JMX notification listeners registered with the MBean server the notification is discarded, otherwise Waveset delivers the notification to the listener.
These MBeans are useful for tracking down performance problems with Waveset's GUI. Because you can customize Waveset's GUI, and rendering parts of the GUI can involve large amounts of data or significant computations. The FormConverter and ViewMaster MBeans can help you quickly identify whether the performance problem is caused by view processing or by processing a specific form field.
To use the FormConverter or ViewMaster MBeans, perform the following steps from a JMX console:
Enable the MBean by setting the Enabled attribute to true.
Specify an appropriate value (in mSecs) for the Limit attribute.
Subscribe to the MBean for notifications by using the Notifications tab.
After completing these steps, any View/Form processing that takes longer than the configured limit will cause a JMX notification to display in the JMX console. The notification will specify the Form, Field, or View ID and how long the operation took.
If the ViewMaster MBean indicates delays in processing a view, check these additional items:
For delays caused by runDerivations or runExpansions, the problem may still be in the form associated with the view.
For delays caused by getView, checkoutView, checkinView, refreshView, or getForm, the problem is probably in the viewer. You must turn on tracing in the viewer to further isolate the problem.
The FormConverter MBean shows processing delays for any field that is rendered to HTML during form processing. Only fields that have a <Display> element are candidates. Common reasons for fields taking a long time to process are:
The field has a lot of data
The field executes XPRESS or script code to compute its value
The FormConverter MBean indicates which Form/Field and how long it took to render. If a form has a lot of fields, the form might display slowly but no single field will exceed the limit.
The Rule MBean emits a notification any time the execution of a rule exceeds the configured limit. Rules can be executed in many places (in tasks, forms, or workflows), so having a single place to capture any rule execution that exceeds a specified time is useful for a high-level performance analysis.
The TaskInstanceCache and WorkItemCache MBeans show which cache operations are used when workflows contain ManualActions that are marked as transient. When a workflow contains a transient ManualAction, changes to the corresponding WorkItem and workflow are made in memory, bypassing the repository but removing the assurance that a workflow will survive a server restart. These MBeans are useful when diagnosing self-service workflow wizard performance.
The ObjectChangeNotification MBean issues a JMX notification any time the server exceeds the execLimit value by delivering a notification to Identity Manager code indicating that an object has changed. This MBean provides a useful diagnostic when the server appears to be processing tasks too slowly. If this MBean is producing JMX notifications with an execLimit of 50 milliseconds, the server is running slowly and you should capture both the JMX notification and the JVM thread dumps for analysis.
The Reconcile MBean shows how much data Reconcile has already processed, how much data is still queued, and the current processing rate. This bean is useful when measuring the impact of changing Reconcile processing thread counts. You can use this MBean with resource-specific MBeans to assess Reconciliation tunings. If the Reconcile.processRate is low and the resource account get is high, you can add more Reconciliation worker threads to increase throughput.
The DataExporter and DataQueue MBeans indicate both the size of the internally queued data and the rate at which the queue is being filled. The Data Exporter Queue provides an in-memory queue to buffer data that needs to be written to the repository, allowing separate threads to drain the queue without blocking the code that queued the data.
When looking at notifications using JConsole, you can hover the mouse pointer over the message field in the notification to see the details (such as viewId, form, field, and so on).
The Java Monitoring and Management Console (JConsole) is a Java Management Extension (JMX) technology-compliant graphical management tool that is co-packaged with at least JDK 5. JConsole connects to a running JVM and gathers information from the JVM MBeans in the connected JMX agent.
Specifically, you can use JConsole to perform the following tasks:
Detect low memory and deadlocks
JConsole accesses the memory system, memory pools, and MBeans garbage collector to provide information about memory use, including memory consumption, memory pools, and garbage collection statistics.
Enable or disable garbage collection
Enable or disable verbose tracing
Monitor local and remote applications
Monitor and manage MBeans including current heap memory use, non-heap memory use, and how many objects are pending for finalization
View information about performance, resource consumption, and server statistics
View summary information about the JVM and monitored values, threads running on the application, and loaded classes
View information about operating system resources (Waveset’s platform extension), such as:
CPU process time
How much total and free physical memory is available
The amount of committed virtual memory (how much virtual memory is guaranteed to be available to the running process)
How much total and free swap space is available
The number of open file descriptions (UNIXonly)
For more information about using JConsole to monitor applications on the Java platform, see the Oracle Developer Network (SDN) article titled Using JConsole to Monitor Applications, which is available from the following URL:
http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html
You can use the Java Runtime Analysis Toolkit (JRat), an open-source performance profiler for the Java platform, to identify potential performance bottlenecks, especially if your deployment uses custom Java classes. JRat monitors your application’s execution and persists the application’s performance measurements.
For example, if you have a custom workflow for provisioning, you can use JRat to see which classes are being invoked and how much time is required to run your workflow compared to the default Waveset provisioning workflow.
For more information about JRat, see http://jrat.sourceforge.net.