This section describes the different Waveset and third-party debugging tools you can use to debug performance issues.
The information is organized into the following sections:
Tracing affects system performance. To help ensure optimal performance, specify the minimum tracing level or turn tracing off after debugging the system.
This section provides instructions for accessing the Waveset Debug pages and describes how to use these pages to identify and debug Waveset performance issues.
See the following sections for information:
Provisioning Threads for Administrator Configurator (Show_Provisioning.jsp)
XML Resource Adapter Caches Flushed and Cleared (Clear_XMLResourceAdapter_Cache.jsp)
You must have the Debug, Security Administrator, or Waveset Administrator capabilities to access and execute operations from the Waveset Debug pages. Administrators and Configurator are assigned this capability by default.
If you do not have the Debug capability, an error message results.
Open a browser and log in to the Administrator interface.
Type the following URL:
http:// host:port /idm/debug
where:
host is the application server on which you are running Waveset.
port is the number of the TCP port on which the server is listening.
When the System Settings page displays, type the .jsp file name for the debug page you want to open.
For example:
http:// host:port /idm/debug/pageName.jsp
Some debugging utilities are not linked from the System Settings page, but you can use them to enhance your ability to gather data for product performance and usability. For a complete list of debug pages, open a command window and list the contents of the idm/debug directory.
Use the Control Timings page to collect and view call timer statistics for different methods. You can use this information to track bottlenecks to specific methods and invoked APIs. You can also use options on the Call Timings page to import or export call timer metrics.
Call timing statistics are only collected while trace is enabled.
Open the Control Timings page, and click Start Timing & Tracing to enable trace and timing.
To stop the timing, click Stop Timing & Tracing or click Stop Timing.
The page re-displays and populates the Show Timings table with a list of methods for which statistics are available and the methods’ aggregate call timer statistics (not broken down by caller).
This table contains the following information:
Method name (Click a method name to see which methods it calls)
Total time
Average time
Minimum time
Maximum time
Total calls
Total errors
To clear the list, click Clear Timing.
You can also use the callTimer command to collect call timer data from the Console. This command is useful when you are debugging performance issues during an upgrade or in other situations where Waveset is not running on the application server.
Use the Edit Trace Configuration page to enable and configure tracing for the Java classes provided with your Waveset installation.
Specifically, you can use this page to configure the following trace settings:
Choose methods, classes, or packages to trace and specify the level of trace you want to capture.
Send trace information to a file or to standard output.
Specify the maximum number of trace files to be stored and the maximum size for each file.
Specify how dates and times are formatted in the trace output file.
Specify the maximum number of methods to be cached.
Indicate how to write data to the trace file.
Write data to the trace file as the data is generated, or queue the data and then write it to the file.
If you are not using a data source, you can use the Host Connection Pool page to view connection pool statistics. These statistics include the pool version, how many connections were created, how many are active, how many connections are in the pool, how many requests were serviced from the pool, and how many connections were destroyed.
You can also use the Host Connection Pool page to view a summary of the connection pools used to manage connections to the Gateway. You can use this information to investigate low-memory conditions.
Use the List Cache Cleared page to clear recently used XML parsers from the cache and to investigate low memory conditions.
Use the Method Timings page to quickly detect and assess hotspots at a method level.
The following information is gathered from Waveset methods and displayed on the Method Timings page:
Method names
How many times the methods were called
How many times the methods exited with an error status
Average time consumed by the methods
Minimum and maximum times consumed by invocations of each method
The Method Timings page also contains a table with the following links. You can click these links to view additional information.
Details. Shows call stack information.
History. Shows a graph of call duration compared with the time of the most recent calls.
History data. Shows a list of the most recent calls, showing what time the call was made and the duration of the call.
To keep stack history and to control its depth, edit Waveset.properites and look at the MethodTimer keys.
Waveset does not keep stack history by default. Keeping stack history has a large negative impact on CPU and memory use.
The Clear ALL option on the Method Timings page clears all results. This option is enabled by default.
Use the Object Size Summary page to detect problematically large objects that can affect your system.
The Object Size Summary page shows information about the size of objects (in characters) stored in the repository. These objects are listed by type, along with the total number of objects of each type, and the objects’ total combined size, average size, maximum size, and minimum size.
Click an entry in the Type column to view additional size information about that object type. For example, click Configuration to view the ID, name, and size of the largest configuration objects in the repository.
You can also access this size information from the Console command line.
Open the console.
At the command prompt, type:
showSizes [ type[limit ]]
For upgrades, existing objects will report a size of 0 until they have been updated or otherwise refreshed.
Use the Provisioning Threads for Administrator Configurator to view a summary of the provisioning threads in use by the system. This summary is a subset of the information available in Show_Threads.jsp.
Looking at just a single thread dump can be misleading.
Use the System Cache Summary page to view information about the following items to help you investigate low-memory conditions:
Administrator-associated object caches
System object cache
User login sessions
XML parser cache
Use the System Memory Summary page to view how much total and free memory you have available in Mbytes. When you are using memory-intensive functionality such as Reconciliation, this information can help you determine whether there is sufficient memory allocated to the JVM.
You can also use this page to launch garbage collection or to clear unused memory in the JVM for investigating heap usage.
The System Properties page provides information about your environment, including software versions, paths and environmental variables.
Use the System Threads page to see which processes are running so you can verify that automated processes (such as reconciliation or Active Sync) are running.
This page includes information about the process type, process name, its priority, if the process is a daemon, and if the process is still running.
Looking at just a single thread dump can be misleading.
Use the Session Pool Cleared page to clear all of the cached sessions for users who have recently logged in and to investigate low memory conditions.
Use the Waveset Properties page to view and temporarily edit properties in the Waveset.properties file. You can test different property settings for a particular server on which the Waveset.properties file resides without having to restart the server to pick up the changes. The edited property settings only remain in effect until the next time you restart the server.
Use the XML Resource Adapter Caches Flushed and Cleared page to clear test XML resource adapters from the cache and to investigate low memory conditions.
Waveset provides a Profiler utility to help you troubleshoot performance problems with forms, Java, rules, workflows, and XPRESS in your deployment.
Customized forms, Java, rules, workflows, and XPRESS can cause performance and scale problems. The Profiler profiles how much time is spent in these different areas, enabling you to determine whether these forms, Java, rules, workflows, or XPRESS objects are contributing to performance and scale problems and, if so, which parts of these objects are causing the problems. You must use the Waveset IDE to view the profiled data.
When enabled, the Profiler has both a memory and a performance impact because this feature captures a significant amount of information across a wide range of services. If you are not having performance problems in your production system and you do not need the Profiler data, you can disable the Profiler as described in Disabling the Profiler.
This section explains how to use Waveset’s Profiler and provides a tutorial to help you learn how to troubleshoot performance issues in your deployment.
The information is organized into the following topics:
Waveset Profiler is only supported on version 7.1 Update 1 and later.
The section provides an overview of the Waveset’s Profiler’s features and functionality. The information is organized as follows:
The Profiler provides helpful information, but has a performance cost on its own. If you do not need the Profiler data, you can disable this utility using the instructions provided in .
You can use the Profiler utility to
Create “snapshots” of profiling data.
A snapshot is the cumulative result of profiling since the last time you reset all of your collected profile results.
Display snapshot results in four, different data views:
Call Tree view provides a tree table showing the call timing and invocations counts throughout the system.
Hotspots view provides a flattened list of nodes that shows the aggregate call timings regardless of parent.
Back Traces view provides an inverted call stack showing all the call chains from which that node (known as the root node) was called.
Callees view provides an aggregate call tree of the root node, regardless of its parent chain.
Specify what kinds of information to include in your snapshot:
You can include every element of form, workflow, and XPRESS or restrict the content to a set of specific elements.
You can pick specific Java methods and constructors to include or exclude from the instrumentation. Instrumentation of Identity Manager classes and custom classes is supported.
Manage your project snapshots as follows.
Save the snapshot in your project’s nbproject/private/idm-profiler directory or to an arbitrary location outside of your project.
You can view a list of all saved snapshots in the Saved Snapshots section of the IDM Profiler view.
Open snapshots from your project or load them from an arbitrary location outside your project.
Delete snapshots.
Search for specific nodes, by name.
This section describes how the Profiler looks up and manages the source for the following Waveset objects:
Forms, Rules, Workflows, and XPRESS objects: When you take a snapshot with the Profiler, the server evaluates all of the profiling data and discovers on which sources the data depends. The server then fetches all of these sources from the repository and includes them in the snapshot. Consequently, you can be sure that the Waveset objects displayed in the snapshot are accurately reflecting the point at which the snapshot was captured.
This process adds to the size of the snapshot, but the source size is actually a relatively small fraction of the total size. As a result, you can send a snapshot to Sun’s Customer Support without having to send your source files separately.
Java source: When you take a snapshot of Java source, the client downloads the snapshot and then goes through the snapshot to capture all referenced Java sources from the project. When you save the snapshot, the client zips the sources and attaches them to the end of the snapshot.
Then, when you view the snapshot and go to the Java source, the client first checks the content of the snapshot. If the client cannot find the content there, it checks the project’s content. This process allows you to send a snapshot containing profiling data from both your custom Java code and Waveset code.
In a Java source snapshot, do not assume the source is up-to-date with the server or always available.
In Call Tree view or Hotspots view, you can double-click any node that corresponds to a Java method, workflow, form, rule, or XPRESS to view the source for that node.
The following sections contain information to consider when you evaluate results provided by the Profiler:
Self Time Statistics: To compute a root node’s Self Time statistic, the Profiler subtracts the times of all children nodes from the root node’s total time. Consequently, an uninstrumented child node’s time is reflected in the root node’s self time. If a root node has a significant self time, you should certainly investigate why. You might not have the proper methods instrumented and so you are looking in the wrong place. For example, assume method A calls method B.
Method A takes a total time of 10 seconds (where total time includes the call to B) and the call to B takes a total time of 10 seconds.
If both A and B are instrumented, the call stack reflects that information. You will see that A has a self-time of 0 seconds and that B has a self-time of 10 seconds (where 10 seconds was actually spent in B). If, however, B is not instrumented, you only see that the call to A takes 10 seconds and that A’s self-time is 10 seconds. Consequently, you might assume the problem lies directly in A rather than in B.
In particular, you might notice large self times on JSPs during their initial compile. If you reset the collected results and then re-display the page, the self time value will be much less.
Constructor Calls: Because there are limitations in the Java instrumentation strategy, initial calls to this() or super() will appear as a sibling to the constructor call, rather than as a child. See the following example:
class A { public A() { this(0); } public A(int i) { } } and: class B { public static void test() { new A(); } } The call tree will look like this: B.test() -A.<init>(int) -A.<init>() Rather than this: B.test() -A.<init>() -A.<init>(int) |
Daemon Threads: Do not be misled by the seemingly large amount of time spent in a number of Waveset’s daemon threads, such as ReconTask.WorkerThread.run() or TaskThread.WorkerThread.run(). Most of this time is spent sleeping, while waiting for events. You must explore these traces to see how much time is actually spent when they are processing an event.
This section describes how to start the Profiler and how to work with various features of the Profiler’s graphical user interface. This information is organized as follows:
Because the Profiler is very memory intensive, you should significantly increase the memory for both your server and the Netbeans Java Virtual Machine (JVM).
To increase your server’s memory,
Open the Netbeans window and select the Runtime tab.
Expand the Servers node, right-click Bundled Tomcat, and select Properties from the menu.
When the Server Manager dialog displays, clear the Enable HTTP Monitor box on the Connection tab.
Select the Platform tab, set VM Options to -Xmx1024M, and then click Close.
To increase the Netbeans JVM memory,
Open the netbeans-installation-dir \etc\netbeans.conff file and locate the following line:
netbeans_default_options="-J-Xms32m -J-Xmx ...
Change the -J-Xmx value to -J-Xmx 1024M.
Save, and then close the file.
When you are finished, start the Profiler using the instructions provided in the Starting the Profiler section.
You can use any of the following methods to start the Profiler from the Waveset IDE window:
Click the Start Identity Manager Profiler on Main Project icon located on the menu bar.
Select Window -> IDM Profiler from the menu bar.
The Identity Manager Profiler window appears in the Explorer. From this window, select an Waveset project from Current Project drop-down menu, and then click the Start Identity Manager Profiler icon located in the Controls section.
The Start Identity Manager Profiler on Main Project icon is enabled when the main Waveset project is version 7.1 Update 1 or later.
Right-click a project in the Projects window, and then select Start Identity Manager Profiler from the pop-up menu.
Select a project in the Projects window, and then select IdM -> Start Identity Manager Profiler from the menu bar.
When you start the Profiler, the Profiler Options dialog displays so you can specify which profiling options you want to use. Instructions for setting these options are provided in Specifying the Profiler Options.
To disable the Waveset Profiler, import the following configuration update:
lh import file
where file contains
<?xml version='1.0' encoding='UTF-8'?> <!DOCTYPE Waveset PUBLIC 'waveset.dtd' 'waveset.dtd'> <Waveset> <ImportCommand class='com.waveset.session.SystemConfigurationUpdater'> <Object> <Attribute name='serverSettings.default.disableProfiling'> <Boolean>true</Boolean> </Attribute> </Object> </ImportCommand> </Waveset> |
This section describes the features of the Profiler graphical user interface, and how to use these features. The information is organized as follows:
The Profiler Options dialog consists of the following tabs:
Mode
The Mode tab provides the following options:
IDM Objects Only: Select to profile form, rule, workflow, and XPRESS objects. Excludes Java objects from the profile.
Java and IDM Objects: Select to profile form, Java, rule, workflow, and XPRESS objects.
The Java and IDM Objects option is not available if you are using a regular Waveset project with an external Identity Manager instance or using a remote Waveset project.
You cannot change the Mode option while the Profiler is running. You must stop the Profiler to change the option.
IDM Object Filters:
The IDM Object Filters tab provides the following options:
Show IDM Object details
Include Anonymous Sources
Anonymous sources are forms (or portions of a form) that are generated on the fly (such as Login forms and MissingFields forms) and do not correspond to a persistent form that resides in the Waveset repository.
Select this box to include Anonymous sources in the snapshot.
Clear this box to exclude Anonymous sources from the snapshot.
Java Filters
Select the Java Filters tab to
Include or exclude Java filters
Create new filters
Delete existing filters
Restore the default filters
Java filters are given in terms of method patterns, and they are expressed in patterns that include or exclude based on canonical method name. Where a canonical method name is:
fully-qualified-class-name.method-name(parameter-type-1, parameter-type-2, ...)
For constructors, method-name is <init>.
Here are a few examples:
To exclude all constructors, enable the Exclude box and add the following filter:
*.<init>(*)
To exclude all constructors with a single org.w3c.dom.Element parameter, enable the Exclude box and add the following filter:
*.<init>(org.w3c.dom.Element)
To exclude all Identity Manager classes, enable the Exclude box and add the following filters:
"com.waveset.*" "com.sun.idm.*"
To instrument your custom code only, disable the Exclude box, remove the initial * include filter, and then add the following filter:
"com.yourcompany.*"
The last two examples are currently equivalent because the filters are applied only to your custom classes and Waveset classes.
If necessary, you can instrument other jars by modifying the following lines in build.xml as appropriate. For example,
<instrument todir="${lighthouse-dir-profiler}/WEB-INF" verbose="${instrumentor.verbose}" includeMethods="${profiler.includes}" excludeMethods="${profiler.excludes}"> <fileset dir="${lighthouse-dir}/WEB-INF"> <include name="lib/idm*.jar"/> <include name="classes/**/*.class"/> </fileset> </instrument> |
By default, the configuration includes all your custom classes and most Waveset classes. A number of Waveset classes are forcibly excluded— because enabling them would break the Profiler.
For example, classes from the workflow, forms, and XPRESS engines are excluded or the Profiler would produce an unintelligible snapshot when profiling Java and Waveset objects.
Note that Java filters provide much more filtering granularity than IDM Object Filters. Java instrumentation adds significant overhead to the execution time, which can drastically skew the profiling results. Because Waveset objects are interpreted rather than compiled, the instrumentation overhead is negligible. So for example, there is basically no reason to exclude workflow A and include workflow B, and so forth.
You cannot modify Java filters while the Profiler is running. You must stop the Profiler before changing Java filters.
Miscellaneous
The Miscellaneous tab provides the following options:
Prune snapshot nodes where execution time is 0:
Disable this option (default) if you want the snapshot to include invocation information for all executed entities— even those whose execution time is zero.
It might be useful to have the number of invocations, even for nodes where there is no execution time.
Enable this option to remove these nodes, which allows you to focus on the most relevant profiling data. In addition, enabling this option can provide a large savings in Profiler snapshot size.
Automatically Open Browser Upon Profiler Start:
Enable this option (default) when you launch the Profiler to automatically open a browser that points to the Identity Manager instance being profiled.
Disable this option if you do not want to open a browser.
Include Java Sources in Snapshot:
Enable this option (default) to include Java sources for any Java methods referenced by the profiling data in the Snapshot. You should always use this setting for snapshots in the field. Custom Java is relatively small and it is very valuable to have for support.
Disable this option only if you are profiling Waveset and have the complete Waveset source available.
In this situation, you do not want to include the Waveset source because it can create extremely large snapshots. (See How the Profiler Locates and Manages Source for more information.)
Use the options on these tabs to indicate which objects to profile and which elements to display in the profile.
After specifying the Profiler options, click OK to start the Profiler. Depending on your project configuration, the Profiler does one of two things:
If you are using a regular Waveset project with an Embedded Identity Manager Instance, the Profiler performs a full build, deploys into the NetBean’s application server, and starts the Profiler.
If you are using a regular Waveset project with an External Identity Manager Instance or the remote Waveset project, the Profiler attaches to the Identity Manager instance configured for the project.
You can select IdM -> Set Identity Manager Instance to control the Identity Manager Instance action for the project.
The IDM Profiler view consists of the following areas:
Current Project Area: Consists of a drop-down menu that lists all of your current projects. Use this menu to select the project you want to profile.
Controls Area: Contains four icons, as described in the following table:
Icon |
Name |
Purpose |
---|---|---|
|
Start Waveset Profiler |
Starts the Profiler and opens the Profiler Options dialog. |
|
Stop Waveset Profiler |
Stops the Profiler. |
|
Reset Collected Results |
Resets all of the profile results you collected to this point. |
|
Modify Profiling |
Reopens the Profiler Options dialog so you can change any of the settings to modify your current profile results. |
Status Area: Reports whether you are connected to the Host and provides Status information as the Profiler is starting up, running, and stopping.
Profiling Results Area: Contains two icons, which are described in the following table:
Icon |
Name |
Purpose |
---|---|---|
|
Start Waveset Profiler |
Starts the Profiler and opens the Profiler Options dialog. |
|
Reset Collected Results |
Resets all of the profile results you collected to this point. |
Saved Snapshots Area: Provides a list of all saved snapshots.
Instructions for saving snapshots are provided in Saving a Snapshot.
In addition, you can use the following buttons to manage these snapshots:
Open: Click to open saved snapshots in the Snapshot View window.
You can also double-click a snapshot in the Saved Snapshots list to open that snapshot.
Delete: Select a snapshot in the Saved Snapshots list, and then click this button to delete the selected snapshot.
Save As: Select a snapshot in the list and then click this button to save that snapshot externally to an arbitrary location.
Load: Click to open a snapshot from an arbitrary location into the Snapshot View window.
When you open a snapshot, the results display in the Snapshot View window, located on the upper right side of Waveset IDE.
A snapshot provides the following views of your data:
Call Tree view: Consists of a tree table showing the call timing and invocation counts throughout your system.
This tree table contains three columns:
Call Tree column: Lists all nodes.
Top-level nodes are one of the following:
Thread.run() methods for various background threads in the system
For example, if you enabled Java profiling, you will see the ReconTask.WorkerThread.run() method.
Request timings
For example, if you viewed the idm/login.jsp URL, you will see a top-level entry for idm/login.jsp. The data displayed in the Time column for this entry represents the total time for that request (or requests). The data displayed in the Invocations column represents the total number of invocations to that page. You can then explore further into that data to see what calls contributed to its time.
The Call Tree also contains Self Time nodes. Self Time values represent how much time was spent in the node itself. (For more information, see Statistics Caveats.)
Time column: Lists the time spent in each node when that node was called from its parent. The percentages are given relative to parent time.
Invocations column: Lists how many times each node was invoked from its parent.
Hotspots view: Provides a flattened list of nodes that shows aggregate call timings regardless of parent.
This view contains the following columns:
Self Time: Lists the total amount of time spent in each node.
Invocations: Lists the total number of times each node was invoked from its parent.
Time: Lists the total amount of time spent in each node and in all of its children.
Back Traces view: Provides an inverted call stack showing all the call chains from where each node was called.
You can use these statistics to answer the question— How much time would I save if I eliminated this particular call chain from this node?
You can access the Back Traces view from any of the other snapshot views by right-clicking a node (known as the root node) and selecting Show Back Traces from the pop-up menu.
The Time and Invocations data values mean something different in Back Traces view:
Time: The values in this column represent the time spent in the root node when it is called from a given call chain.
Invocations: The values in this column represent how many times the root node was invoked from a given call chain.
Callees view: Provides an aggregate call tree for a node (known as the root node), regardless of its parent chain.
These statistics are helpful if you have a problem area that is called from many places throughout the master call tree and you want to see the overall profile for that node.
You can access the Callees view from any of the other snapshot views by right-clicking a node (known as the root node) and selecting Show Callees from the pop-up menu.
The Time and Invocations data values used in Callees view have the same meaning as those used in Call Tree view.
Right-click any node in Call Tree view or in Hotspots view and a pop-up menu displays with the options described the following table:
Menu Options |
Description |
---|---|
GoTo Source |
Select this option to view the XML source for a node that corresponds to a Java method, workflow, form, rule, or XPRESS. For detailed information about this view, see How the Profiler Locates and Manages Source. |
Show Back Traces |
Select this option to access the Back Traces view. For detailed information about this view, see Working with the Snapshot View. |
Show Callees |
Select this option to access the Callees view. For detailed information about this view, see Working with the Snapshot View. |
Find In Hotspots |
Select this option to find a node in the Hotspots view. For detailed information about this view, see Working with the Snapshot View. |
List Options -> Sort -> |
Select this option to
|
List Options -> Change Visible Columns |
Select this option to change the columns displayed in the Call Tree or Hotspots list. When the Change Visible Columns dialog displays, you can select one or more of the following options:
|
Use the Search icon , located at the top of the Snapshot View window to search for nodes by name the Call Tree view or Hotspots tree.
Alternatively, right-click any node in Call Tree view or Hotspots view and select Find in Call Tree or Find in Hotspots (respectively) from the pop-up menu to search for a node.
The Profiler provides several options for saving a snapshot. See the following table for a description of these options:
Icon |
Name |
Purpose |
---|---|---|
|
Save the Snapshot in the Project icon (located at the top of the Snapshot View window) |
Saves the snapshot in the nbproject/private/idm-profiler directory of your project. Snapshots saved in your project are listed in the Saved Snapshots section of the Profiler view. |
|
Save the Snapshot Externally icon (located at the top of the Snapshot View window) |
Saves a snapshot to an external, arbitrary location. |
|
Save As button (located in the Saved Snapshots area) |
Saves a snapshot to an external, arbitrary location. |
Waveset provides a tutorial (profiler-tutorial.zip) to help you learn how to use the Profiler to troubleshoot forms, Java rules, workflows, and XPRESS.
Use the following steps to complete the tutorial.
Select File -> New Project.
When the New Project wizard displays, specify the following, and then click Next:
Complete the following fields on the Name and Location panel, and then click Next:
Project Name: Enter Idm80 as the project name.
Project Location: Use the default location or specify a different location.
Project Folder: Use the default folder or specify a different folder.
When the Waveset WAR File Location panel displays, enter the location of the Waveset 8.1.1 war file. Typically, unzipping this file creates an idm.war file in the same directory.
Click Next to continue to the Repository Setup panel.
You should not have to change the default settings on this panel, just click Finish. When you see the BUILD SUCCESSFUL message in the Waveset IDE Output window, you can extract the Profiler tutorial files. See Step 2: Unzip the Profiler Tutorial for instructions.
Unzip profiler-tutorial.zip in the project root. The extracted files include:
<project root>/custom/WEB-INF/config/ProfilerTutorial1.xml <project root>/custom/WEB-INF/config/ProfilerTutorial2.xml <project root>/src/org/example/ProfilerTutorialExample.java <project root>/PROFILER_TUTORIAL_README.txt
Start the Profiler. Proceed to Step 3: Start the Profiler.
Use the instructions provided in Before You Begin to increase the memory for your server and Netbeans JVM.
Use any of the methods described in Starting the Profiler to start the Profiler.
When the Profiler Options dialog displays, you can specify profiling options.
Continue to Step 4: Set the Profiler Options
For detailed information about all of the different Profiler options, see Specifying the Profiler Options.
For the purposes of this tutorial, specify the following Profiler options:
On the Mode tab, select Java and IDM Objects to profile form, Java, rule, workflow, and XPRESS objects.
Select the Java Filters tab.
Use the following steps to disable all Waveset Java classes except your custom Java classes (in this case, org.example.ProfilerTutorialExample):
Click OK to run the Profiler.
The Profiler takes a few minutes to complete the first time you run it on a project or if you have recently performed a Clean Project action.
When the Profiler finishes processing, you are prompted to Log In.
Enter the password configurator, select the Remember Password box, and then click OK to continue.
When the Waveset window displays, log in.
Typically, you should log in to Waveset as a different user instead of logging in as configurator again. You are already logged into the Profiler as configurator, and the Waveset session pool only allows one entry per user. Using multiple entries can result in the appearance of a broken session pool and might skew your profiling results for finer-grained performance problems.
However, for this simple example the session pool is of no consequence so you can login as configurator/configurator.
In Waveset, select Server Tasks -> Run Tasks, and then click ProfilerTutorialWorkflow1.
The tutorial might take a few moments to respond.
Although you could take a snapshot now; you are going to reset your results instead, run the Profiler, run it again, and then take a snapshot.
It is a best practice to run the Profiler a couple of times before taking a snapshot to be sure all the caches are primed, all the JSPs are compiled, and so forth.
Running the Profiler several times enables you to focus on actual performance problems. The only exception to this practice is if you are having a problem populating the caches themselves.
Return to the IDM Profiler view in the Waveset IDE. Click the Reset Collected Results icon in the Profiling Results section (or in the Controls section) to reset all of the results collected so far.
In Waveset, select Server Tasks -> Run Tasks again, and click ProfilerTutorialWorkflow1.
When the Process Diagram displays, return to the Waveset IDE and click Take Snapshot in the Profiling Results section.
The Waveset IDE downloads your snapshot and displays the results on the right side of the window.
This area is the Call Tree view. At the top of the Call Tree, you should see a /idm/task/taskLaunch.jsp with a time listed in the Time column. The time should indicate that the entire request took six+ seconds.
Expand the /idm/task/taskLaunch.jsp node, and you can see that ProfilerTutorialWorkflow1 took six seconds.
Expand the ProfilerTutorialWorkflow1 node. Note that activity2 took four seconds and activity1 took two seconds.
Expand activity2.
Note that action1 took two seconds and action2 took two seconds.
Expand action1 and note that the <invoke> also took two seconds.
Double-click the <invoke> to open ProfilerTutorialWorkflow1.xml and highlight the following line:
<invoke name=’example’ class=’org.example.ProfilerTutorialExample’/> |
You should see that a call to the ProfilerTutorialExample method took two seconds.
You are actually browsing XML source that was captured in the snapshot, rather than source in the project. Snapshots are completely self-contained. (For more information, see How the Profiler Locates and Manages Source.)
Select the CPU:<date><time> tab to return to your snapshot.
Expand the <invoke> node, and note that the Profiler spent two seconds in the Java ProfilerTutorialExample.example() method.
Double-click the method name to open the ProfilerTutorialExample.java source and highlight the following line:
Thread.sleep(2000); |
There’s the problem! This method contains a two-second thread sleep.
If you return to the Call Tree, you can see that all of the two second paths lead to this method. (You should see three paths; for a total of six seconds.)
Select the Hotspots tab (located at the bottom of the Call Tree area) to open the Hotspots view. Notice that ProfilerTutorialExample.example() has a total self time of six seconds.
(For more information about Hotspots, see Working with the Snapshot View.)
Right-click ProfilerTutorialExample.example() and select Show Back Traces from the pop-up menu.
A new Back Traces tab displays at the bottom of the area.
Expand the ProfilerTutorialExample.example() node on the Back Traces tab to see that this method was called from three places, and that the method took two seconds when it was called from each place.
(For more information about Back Traces, see Working with the Snapshot View.)
Click the Save the snapshot in the project icon to save your snapshot and close it.
If you check the Saved Snapshots section on the IDM Profiler tab, you should see your snapshot. (You might have to scroll down.)
Select the saved snapshot, and then click Open to re-open it.
You can use the Save As button to save your snapshots externally and use the Load button to load a snapshot from outside your project.
Close the snapshot again.
The next part of this tutorial illustrates how to profile a workflow ManualAction.
In Waveset, select Server Tasks -> Run Tasks, and then click ProfilerTutorialWorkflow2.
After a few moments, an empty form displays.
Click Save and the process diagram displays.
Select Server Tasks -> Run Tasks again.
Return to the Waveset IDE IDM Profiler view and click the Reset Collected Results icon in the Profiling Results section.
Now click ProfilerTutorialWorkflow2 in Waveset.
When the blank form displays again, click Save.
In the IDM Profiler view, click Take Snapshot.
After a few seconds, a snapshot should display in the Call Tree area. You should see that /idm/task/workItemEdit.jsp took six+ seconds. (This result corresponds to the manual action in the workflow.)
Expand the /idm/task/workItemEdit.jsp node and note that running all Derivations in the ManualAction form took a total of six seconds.
Expand the Derivation, displayNameForm, variables.dummy, and <block> nodes.
You should see that the <block> took six seconds and, of that time, the Profiler spent two seconds in each of the three invokes to the ProfilerTutorialExample.example(). method.
You can double-click <block> to view the source.
You can use the following Oracle and third-party tools to identify potential performance bottlenecks:
These tools can be particularly useful if your deployment uses custom Java classes.
The DTrace facility is a dynamic tracing framework for the Solaris 10 operating system that enables you to monitor JVM activity.
DTrace contains more than 30,000 probes and uses integrated user-level and kernel-level tracing to give you a view into your production system. You can also trace arbitrary data and expressions by using the D language, which is similar to C or awk. The DTrace facility also includes special support for monitoring the JVM, and enables you to watch your whole system and span outside the JVM.
DTrace is easiest to use with Java 6 because probes are built into the JVM. The facility also works with Java 1.4 and Java 5, but you must download JVM PI or JVM TI agents from the following URL:
https://solaris10-dtrace-vm-agents.dev.java.net/
The following example shows how to write a DTrace script.
#!/usr/sbin/dtrace -Zs #pragma D option quiet hotspot$1::: { printf("%s\n", probename); } |
In this example, you would replace $1 with the first argument to the script, which is the PID of the Java process you want to monitor. For example:
# ./all-jvm-probes.d 1234
The following table describes the commands you can use to enable different DTrace probes.
Table 4–3 DTrace Commands
Because DTrace causes additional work in the system, enabling this facility affects system performance. The effect is often negligible, but can become substantial if you enable many probes with costly enablings.
Instructions for minimizing the performance effect of DTrace are provided in the “Performance Considerations” chapter of the Solaris Dynamic Tracing Guide.
For more information about DTrace, see /usr/demo/dtrace and man dtrace.
Waveset enables you to use Java Management Extensions (JMX) to capture and expose operational statistics for certain resource adapter operations. You can use this data for diagnostic and predictive purposes, such as to monitor system health and reports.
This statistical data includes the following:
The number of times the action was performed
The minimum, maximum, and average duration of the operations
Objects |
Actions Monitored |
---|---|
For Accounts |
|
For Actions |
Run |
For Other Objects |
|
JMX creates MBeans for each resource adapter, by server, and registers these beans with names that match the following pattern:
serverName=server name, resourceAdapterType=Resource Adapter Type, resourceAdapterName=Resource Adapter Name |
Waveset records statistics for all completed operations, whether they completed successfully or with errors. However, Waveset does not record statistics for incomplete operations, such as any operations that throw exceptions.
You can configure excludes as follows:
From the Administrator interface, select Configure -> Servers.
On the Configure Servers page, perform one of the following tasks:
Click the Edit Default Server Settings button to edit the default server settings.
Click a server link to edit the policy for that server.
Click the JMX tab and enable the JMX Enable Resource Adapter Monitor box to turn on resource monitoring.
To exclude specific resources, add regular expressions to the JMX Resource Adapter Monitor Excludes list.
To exclude monitoring specific actions, add regular expressions to the JMX Resource Adapter Monitor Operation Excludes list.
All excludes use regular expressions. For excluding certain resources, JMX just matches on the resource name. For example, if you have adapters named
resource1 resource2 resource3 resource10 resource11 |
and you specify the following pattern
.*1$ |
which means, match any 0 or more of any character (.*) until something that ends with a 1 (1$). JMX will exclude resource1 and resource11.
For operations, the process is similar. If your operations have the following names, the patterns must match against those names.
ACCOUNT_CREATE ACCOUNT_UPDATE ACCOUNT_DELETE ACCOUNT_GET ACCOUNT_AUTHENTICATE OBJECT_CREATE OBJECT_UPDATE OBJECT_DELETE OBJECT_GET OBJECT_LIST ACTION_RUN |
For example, the ^ACCOUNT.* pattern excludes all operations that start with ACCOUNT. Or, using this pattern excludes updates and deletes:
.*UPDATE$ .*DELETE$ |
For more information about configuring and using JMX, see Configuring JMX Monitoring and The JMX Publisher Type in Oracle Waveset 8.1.1 Business Administrator’s Guide.
Waveset supplies some JMX MBeans that provide information about the following:
Data Exporter
Object Repository
Performance
Reconciliation
Resource Adapters and Connectors
Scheduler
Waveset Server Cluster
There are nine MBeans in the Performance category:
DataExporter
DataQueue
FormConverter
ObjectChangeNotification
Reconcile
Ruler
TaskInstanceCache
ViewMaster
WorkItemCache
These MBeans can capture the same performance data that is captured by the Waveset Profiler, but the MBeans have a lower runtime performance cost.
You enable the Profiler on a per-server basis. When enabled, the Profiler impacts both memory (to capture and hold the performance data) and performance.
Performance MBeans are disabled by default. When enabled, MBeans have a significantly lower operational impact, however they capture very specific performance data.
When enabled, the FormConverter and ViewMaster MBeans each have a configurable threshold and when processing takes longer than the threshold, a JMX Notification is produced. This notification indicates which view or form element was involved and how long it took to process that element. If there are no JMX notification listeners registered with the MBean server the notification is discarded, otherwise Waveset delivers the notification to the listener.
These MBeans are useful for tracking down performance problems with Waveset's GUI. Because you can customize Waveset's GUI, and rendering parts of the GUI can involve large amounts of data or significant computations. The FormConverter and ViewMaster MBeans can help you quickly identify whether the performance problem is caused by view processing or by processing a specific form field.
To use the FormConverter or ViewMaster MBeans, perform the following steps from a JMX console:
Enable the MBean by setting the Enabled attribute to true.
Specify an appropriate value (in mSecs) for the Limit attribute.
Subscribe to the MBean for notifications by using the Notifications tab.
After completing these steps, any View/Form processing that takes longer than the configured limit will cause a JMX notification to display in the JMX console. The notification will specify the Form, Field, or View ID and how long the operation took.
If the ViewMaster MBean indicates delays in processing a view, check these additional items:
For delays caused by runDerivations or runExpansions, the problem may still be in the form associated with the view.
For delays caused by getView, checkoutView, checkinView, refreshView, or getForm, the problem is probably in the viewer. You must turn on tracing in the viewer to further isolate the problem.
The FormConverter MBean shows processing delays for any field that is rendered to HTML during form processing. Only fields that have a <Display> element are candidates. Common reasons for fields taking a long time to process are:
The field has a lot of data
The field executes XPRESS or script code to compute its value
The FormConverter MBean indicates which Form/Field and how long it took to render. If a form has a lot of fields, the form might display slowly but no single field will exceed the limit.
The Rule MBean emits a notification any time the execution of a rule exceeds the configured limit. Rules can be executed in many places (in tasks, forms, or workflows), so having a single place to capture any rule execution that exceeds a specified time is useful for a high-level performance analysis.
The TaskInstanceCache and WorkItemCache MBeans show which cache operations are used when workflows contain ManualActions that are marked as transient. When a workflow contains a transient ManualAction, changes to the corresponding WorkItem and workflow are made in memory, bypassing the repository but removing the assurance that a workflow will survive a server restart. These MBeans are useful when diagnosing self-service workflow wizard performance.
The ObjectChangeNotification MBean issues a JMX notification any time the server exceeds the execLimit value by delivering a notification to Identity Manager code indicating that an object has changed. This MBean provides a useful diagnostic when the server appears to be processing tasks too slowly. If this MBean is producing JMX notifications with an execLimit of 50 milliseconds, the server is running slowly and you should capture both the JMX notification and the JVM thread dumps for analysis.
The Reconcile MBean shows how much data Reconcile has already processed, how much data is still queued, and the current processing rate. This bean is useful when measuring the impact of changing Reconcile processing thread counts. You can use this MBean with resource-specific MBeans to assess Reconciliation tunings. If the Reconcile.processRate is low and the resource account get is high, you can add more Reconciliation worker threads to increase throughput.
The DataExporter and DataQueue MBeans indicate both the size of the internally queued data and the rate at which the queue is being filled. The Data Exporter Queue provides an in-memory queue to buffer data that needs to be written to the repository, allowing separate threads to drain the queue without blocking the code that queued the data.
When looking at notifications using JConsole, you can hover the mouse pointer over the message field in the notification to see the details (such as viewId, form, field, and so on).
The Java Monitoring and Management Console (JConsole) is a Java Management Extension (JMX) technology-compliant graphical management tool that is co-packaged with at least JDK 5. JConsole connects to a running JVM and gathers information from the JVM MBeans in the connected JMX agent.
Specifically, you can use JConsole to perform the following tasks:
Detect low memory and deadlocks
JConsole accesses the memory system, memory pools, and MBeans garbage collector to provide information about memory use, including memory consumption, memory pools, and garbage collection statistics.
Enable or disable garbage collection
Enable or disable verbose tracing
Monitor local and remote applications
Monitor and manage MBeans including current heap memory use, non-heap memory use, and how many objects are pending for finalization
View information about performance, resource consumption, and server statistics
View summary information about the JVM and monitored values, threads running on the application, and loaded classes
View information about operating system resources (Waveset’s platform extension), such as:
CPU process time
How much total and free physical memory is available
The amount of committed virtual memory (how much virtual memory is guaranteed to be available to the running process)
How much total and free swap space is available
The number of open file descriptions (UNIXonly)
For more information about using JConsole to monitor applications on the Java platform, see the Oracle Developer Network (SDN) article titled Using JConsole to Monitor Applications, which is available from the following URL:
http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html
You can use the Java Runtime Analysis Toolkit (JRat), an open-source performance profiler for the Java platform, to identify potential performance bottlenecks, especially if your deployment uses custom Java classes. JRat monitors your application’s execution and persists the application’s performance measurements.
For example, if you have a custom workflow for provisioning, you can use JRat to see which classes are being invoked and how much time is required to run your workflow compared to the default Waveset provisioning workflow.
For more information about JRat, see http://jrat.sourceforge.net.