This chapter explains how you can use tracing to help fix errors, solve performance issues, and understand the flow within Waveset. This chapter also describes methods for troubleshooting problems that might occur with different Waveset components.
This chapter covers the following topics:
Review the information in the following sections before tracing or troubleshooting Waveset:
This chapter is intended for application server and database administrators, front-line support engineers, and partners who are responsible for maintaining Waveset in a deployment environment.
Before troubleshooting problems with Waveset, you must
Be familiar with Java 5.0 (required for Oracle Waveset 8.1.1)
Understand the components that you are trying to troubleshoot
Before you start tracing or troubleshooting Waveset, note the following:
Tracing affects system performance. To ensure optimal performance, use the minimum tracing level needed or turn off tracing after debugging the systems.
Do not enable tracing for the com.waveset class. The com.waveset class is verbose and has many classes, so tracing this class might cause your server to hang.
Do not configure Waveset to trace at the advanced level unless instructed to do so by Oracle Support.
If you set exception.trace in the Waveset.properties file to debug a specific problem, do not use it for an extended period of time and be sure you disable it after debugging. Setting exception.trace in the Waveset.properties file can significantly affect system performance.
By default, Waveset returns stack trace information as HTML comments when certain errors occur. Generally, a stack trace is not visible to the end-user, but the trace can be seen by viewing the page source in the browser.
Stack trace information can be very useful when diagnosing a problem, but exposes information about the execution of the web application that could be considered a security risk. Production deployments should consider turning off stack trace information rendering by configuring the following setting in Waveset.properties:
ui.web.disableStackTraceComments=true |
You must perform this change manually on each Waveset server.
Beginning with Waveset version 8.1.1, the default setting for ui.web.disableStackTraceComments is true. In previous releases, the default setting for ui.web.disableStackTraceComments was false.
However, because version 8.1.1 is an upgrade and not an install, the new setting does not automatically take place because Waveset.properties is not overridden during an upgrade.
Before calling Oracle Technical Support or Sustaining for assistance, you must prepare for the call by collecting as much information as possible about your environment and the problem. Instructions are provided in the following sections:
Unless specifically noted otherwise, the commands described in the following sections will work from a properly sourced terminal on UNIX or Windows.
Try using one or more of the following methods to narrow the problem area:
Use web browser search tools to research the problem.
Use resource-specific support options when available. Your problem might be related to a resource and fixed by using a resource patch.
Remove Waveset from the equation by using an appropriate client tool when possible. For example, by using a Java LDAP browser.
You must have the following information about your environment and Waveset installation available for all calls to your Support or Sustaining representative.
If you are experiencing issues related to the following, specific problem areas, you must gather some additional information. Be sure to read the pertinent sections before calling Support.
If you are having problems with your Waveset installation, you must have the following information available for Support:
Information about your repository configuration
You can type the lh setRepo -c -v command in a terminal window to view detailed information about your repository configuration.
Database information
You should be able to answer the following questions about your database:
What type of database is installed?
What database version, including any patches, are you using?
What JDBC driver is being used to connect?
If you do not know this information, consult your database administrator.
For upgrade issues, Support asks for this information on a case-by-case basis:
Current level. What version of Waveset are you upgrading from?
Target level. What version of Waveset are you upgrading to?
If your system is hung, try to collect following information:
Thread dump. A thread dump creates a stack trace for every running thread. Support can use thread dumps to determine the state of the system.
To collect a thread dump, send a SIGQUIT signal to the Java process running Waveset, as follows:
In UNIX: Type kill -3 pid.
Where pid is the process ID of the JVM running Waveset (usually the same as Application Server).
In Windows: Press CTRL-BREAK in the JVM console.
Heap dump. A heap dump contains the entire status of what is in memory for a Java process.
To force a heap dump, type the jmap -dump:file=filenamepid command.
Where:
filename is the heap dump file to be created. Be aware that this file can be very large.
pid is the process ID of the JVM running Waveset (usually the same as Application Server).
Screen shots or saved HTML pages. Use these tools to preserve information about the last action you performed before the system hung. You might capture information about the state of the system that is worth noting and saving for review.
Recent activity report. If you can restart Waveset, you can retrieve an activity report that might help you determine what was happening in the system before the hang occurred.
Select the Reports tab in Waveset; and generate a Recent Systems Messages report and a Today's Activities report.
Instructions for accessing the Waveset debug pages mentioned in this section are described in Accessing the Debug Pages.
If you are having performance problems, try gathering the following information for Support:
Call Timer data. Waveset's Call Timer debug page (debug/callTimer.jsp) enables you to collect call timer statistics for different methods, which can be useful performance data.
Use the following steps to collect the most-relevant data:
Open the Clear Timing Statistics page and click Start Timing and Tracing to enable tracing and timing.
Waveset can only collect call timing statistics when tracing is enabled.
Return to Waveset and perform the action that is causing performance issues.
Refresh the Call Timer debug page.
Verify that you captured performance data.
The Show Timings table should list the methods for which statistics are available and the methods' aggregate call timer statistics. (These statistics are not broken down by caller).
Click the Export button to export the data to a calltimer.xml file.
Save calltimer.xml and deliver the file to Support for analysis.
For more information about the Call Timer debug page, see Control Timings (callTimer.jsp).
Java Profiler data. Java profilers are third-party applications that attach to running Java processes and track all operations.
Support might ask you to run a memory profiler if you are having extreme or low-level performance issues in Waveset or in your custom code. Running Java profilers can cause additional performance degradation, but they provide useful insight about what is happening at a low level. If running a profiler is necessary, your Support representative will give you detailed instructions.
Main/Run Loop Tracing data. You can use Waveset Trace to capture the state of your system but to be useful, the trace must be targeted where the problem occurs. Identifying this location can be difficult, so consult your Support team if you need help.
For common performance issues, it's likely that Support already knows which area is affected. For example, following are several major Waveset systems and which areas to trace:
Process Level Profiling data. You must sometimes gather profiling data using native tools on the server or resource having issues.
For UNIX: Invoke the top command from the command line to view a list of all processes with their CPU and memory consumption.
For Windows: Type Control—Alt—Del, click Task Manager to open the Windows Task Manager dialog. Select the Processes tab to view a list of all processes with their CPU and memory consumption. Alternatively, you can launch the Windows Task Manager by right-clicking the Task Bar and selecting Task Manager.
Query Timer data. This utility facilitates connecting to Waveset's back-end repository and allows you to issue JDBC SQL statements directly to the database. As the name suggests, this utility times statements issued to the repository SQL database.
When you have performance issues, the Query Timer can check the performance of specific SQL statements or check connectivity using the JDBC connection that Waveset uses from a command line. The Query Timer uses the JDBC driver, user, and password configured for Waveset.
The Query Timer utility has been hard coded to use the default Configurator authentication. If you modified Configurator, you will not be able to run the Query Timer in your environment.
You invoke the Query Timer from the command line, by using a $WSHOME/bin/lh script. The following example is for a UNIX system. If necessary, modify the script appropriately for Windows.
sh ${WSHOME}/bin/lh com.waveset.repository.QueryTimer [ sql statement ] |
Issue this script with no options to perform a query that summarizes the statistics of each object type in the repository. You can use this query to determine the size of the objects in the database.
If you are running Waveset 7.0, this script outputs additional information for every 1000th object of each object type in the repository, such as:
Type
numObjects
numAttrVals
objectBytes
attrBytes
objIdxBytes
attrIdxBytes
Show Timings data. The Method Timings page (debug/Show_Timings.jsp) enables you to determine at a glance which processes are being run most frequently. You might use this page instead of the Call Timings page if you are only interested in a quick view of the processes being run, and where an in-depth investigation is not likely to be necessary.
For the best results, click the Clear ALL button and then perform the action that is causing poor performance. After checking the page results, save the HTML page from the JSP and send the page to Support.
See Method Timings (Show_Timings.jsp) for more information.
If you are reporting a system crash, try to collect the following information for Support:
Stack traces. When Java crashes, a stack trace is sometimes printed to Standard Error. If a stack trace is available, forward the trace to Oracle Support.
Java core files. If the JVM dumps a core file, the file might be useful to Oracle Support or Sustaining. Although core files are large, collect and store them in a safe place so you can deliver them to Support upon request.
Memory issues. If your system is running out of memory, try to gather the following information for Support before your program crashes:
Heap dump. You can automate heap dump generation by adding the -XX:+HeapDumpOnOutOfMemoryError option to the application server's JVM options. You can also force a heap dump on a running process as described in Hung Systems.
Histogram. A histogram is a list of class objects that are occupying memory for the Java application.
You must collect histograms at set intervals over a period of time to track memory growth over time. For example, starting a reconcile consumes all system memory in ten hours, so collecting a histogram every hour results in ten histograms, or ten points of data from which to track memory growth. Save the output from each histogram to a different file, with an appropriate timestamp, and deliver this information to Support.
To collect histograms for Support, type jmap -histo pid in a terminal window. Where pid is the Process ID of the JVM running Waveset (usually the same as application server).
Exception tracing. Support can frequently use an exception trace to determine the cause of a fatal error. However, Waveset's exception tracing is disabled by default. To collect a useful exception trace for Support, you must enable exception tracing and then coerce Waveset into crashing again in the same way.
To enable exception tracing, use the instructions in Tracing Exceptions.
In addition to the information provided in this chapter, consult the documents and web sites listed in this section for information related to tracing and troubleshooting Waveset.
See the following documents for information related to tracing and troubleshooting Waveset:
Testing Your Customized Form in Oracle Waveset 8.1.1 Deployment Reference for the recommended method of tracing XPRESS functions.
The article titled Using JConsole to Monitor Applications for information about using JConsole to monitor applications that run on the Java platform. This article is available from the following URL, http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html.
The following table describes web sites related to troubleshooting Waveset.
Table 5–1 Useful Web Sites
Generally, you can identify and resolve problems in your deployment if you follow the steps shown in the following illustration.
See Tracing Waveset Objects and Activities for information about how to enable tracing for different Waveset objects and activities.
See Troubleshooting and Fixing Common Problems for information about how to troubleshoot problems that commonly occur for those objects and activities.
Trace output can be very helpful when you are trying to identify and resolve problems, or when you are developing custom resource adapters.
This section describes how to enable tracing for a variety of Waveset objects and activities. The information is organized as follows:
Tracing affects system performance. To ensure optimal performance, specify the minimum tracing level or turn tracing off after debugging the system.
This section describes how to enable and configure tracing from several locations within Waveset. Instructions are organized as follows:
The System Settings page is the primary Waveset Debug page.
From this page, you can
View and edit objects in the repository
Clear caches
Set specialized traces
Reload the Waveset.properties file
Use the following steps to enable tracing from the Waveset System Settings page:
Open a browser and log in to the Waveset Administrative interface.
Type the following URL:
http://hostport/idm/debug
where:
host is the local server on which you are running Waveset.
port is the number of the TCP port on which the server is listening.
When the System Settings page displays, click Show Trace (located near the bottom of the page) to manage a single trace configuration, where you can create, modify, or delete up to ten trace settings.
The remaining instructions assume you have only one trace configuration.
To manage multiple trace configurations, click the Show Trace List button instead. When the Trace Configuration page displays, click the configuration name to edit the current settings.
Waveset supplies one configuration, called Global, by default. However, if you have multiple servers in an Waveset instance, defining different configurations for particular servers might be useful. If the name of a trace configuration matches the name of the current host, the host configuration overrides the Global configuration.
After you configure tracing, you can view and edit the default global trace configuration object or create a new configuration object as described in the following sections:
From the Edit Trace Configuration page, click the Trace Enabled box to enable tracing.
Deselecting this box stops tracing, but keeps your configuration. You can turn tracing on and off without having to remember and retype the classes you were tracing.
Specify which classes, packages, or methods to trace by typing their names into the table.
For example:
To trace all classes in the waveset.repository package, enter com.waveset.repository.
To trace the AbstractDataStore class in the waveset.repository package, enter com.waveset.repository.AbstractDataStore.
To trace the list method in the AbstractDataStore class in the waveset package, enter com.waveset.repository.AbstractDataStore#list.
Do not enable trace for the com.waveset class. The com.waveset class is verbose and has many classes, so tracing this class might cause your server to hang.
In the same table, choose a Method/Class tracing level from the Level menu.
Each level captures different types of information, as described in the following table:
Trace Level |
Description |
---|---|
0 |
Minimum debugging output, traces exceptions and error information only |
1 |
Trace Level 0 events, plus entries and exits for public methods |
2 |
Trace Level 1 events, plus entries and exits for non-public methods |
3 |
Trace Level 2 events, plus decision points and significant variables |
4 |
Maximum debugging output |
Method/Class tracing produces a predictable, but possibly very large volume of trace output. Try to be as specific as possible when specifying methods and classes to trace.
(Optional) To enable subcall tracing, choose a level from the Subcall Trace menu. This menu uses the same trace numbering scale described in the previous table.
The default Subcall Tracing level is none, which disables subcall tracing on a per-method or per-class basis.
Subcall Tracing levels are independent of the Method/Class tracing levels you specified in the previous step.
When you enable Subcall Tracing for a particular method that supports subcall tracing, you are automatically setting the tracing level for methods that are called by this method. Subcall Tracing enables you to produce a short, but detailed burst of trace output that is triggered by the entry into a specified method and terminated by the method’s exit.
For example, assume you created a trace configuration setting for the com.waveset.adapter.NewRes#init method, set Method/Class tracing to level one and set Subcall tracing to level three.
Also, assume that the init method calls two other methods:
NewRes#subcallA
NewRes#subcallB
When the init method runs, the com.waveset.adapter.NewRes#init method produces trace output at level one until reaching subcallA. When subcallA begins executing, the trace level changes to three and continues at that level until subcallA exits. The com.waveset.adapter.NewRes#init method returns to the init method and restores the trace level to one. Later, when init calls subcallB, there is another burst of level three trace detail that lasts until subcallB exits. Finally, when init exits, level one tracing stops.
Send the trace results to a specified file location or to stdout.
If you choose output to a file, the Trace File field displays. Use this field to specify an alternate location and file name for trace output files. The default location and file name is
path_to_idm_install\export\pipeline\config\WSTrace.log |
Specify the maximum number of trace files to store (default is 2).
Specify the maximum size for each file (default is 512K).
Specify whether to write the trace output file as generated (synchronously) or to queue the data and then write it to the trace file (asynchronously).
Save your changes.
Use the following steps to create a new trace configuration object:
Decide which package or method you want to trace.
Generally, you specify a resource adapter name or use information that was revealed in an error message.
Log in to the Waveset Administrator interface and open the System Settings page as described in To Enable Tracing from the System Settings Page.
On the Systems Setting page, click Show Trace List.
When the Waveset Trace Configuration page displays, click New.
Enable tracing from the Edit Trace Configuration page.
Choose one of the following options from the Trace Configuration menu:
Global. Select to enable tracing for all servers.
Server_name. Select a server name to enable tracing for a particular server.
Select the Trace Enabled box to enable tracing for this object and configure the remaining parameters on this page as described in To Edit the Default Configuration Object.
Save your changes.
This section describes how to enable tracing from an Waveset Debug page.
Open a browser and log in to the Waveset Administrative interface.
Type the following URL:
http:// host:port /idm/debug/pageName.jsp
where:
host is the local server on which you are running Waveset.
port is the number of the TCP port on which the server is listening.
pageName.jsp is the particular Debug page you want to open.
For example, to trace adapter classes and methods for a custom adapter, you can open the Edit Trace Configuration page by typing the following URL:
http://host :port/idm/debug/Show_Trace.jsp
This section explains how to enable tracing from the Waveset console.
Set $WSHOME.
For example, to set the variable to the default installation directory:
set WSHOME=C:\Program Files\tomcat\webapps\idm |
Type lh console to open the Waveset console from the bin directory.
From the console, type trace to see a detailed summary of available trace options, including enable and disable.
The syntax for this command is:
trace [ -s server ] $subcommand
By default, Waveset “sanitizes” trace information by replacing each occurrence of sensitive data with a single asterisk. Sensitive data includes any data that is inbound to encryption methods, outbound from decryption methods, and cipher-related metadata such as data lengths, padding lengths, and so forth. However, you can configure a property in the Waveset.properties file to include this information in your trace output for debugging purposes.
Use the following steps to enable sensitive cipher data in your trace logs:
From a command line, open the Waveset.properties file.
Locate the cipher.data.trace property file comments, and uncomment the following line:
cipher.data.trace=true |
Restart your server for this change to take effect.
By default, Waveset sends tracing information to a file named WSTrace#.log that is stored in the path_to_idm_install\export\pipeline\config\ directory. If necessary, you can specify alternate file names and locations when you configure tracing for an object.
Each log file is numbered sequentially, up to the maximum number of files specified on the Edit Trace Configuration page. For example, if you specify a maximum of three files, the files are named WSTrace1.log, WSTrace2.log, and WSTrace3.log.
Use one of the following methods to view these log files:
If you are sending trace information to a file, open the trace file from the specified location. For example:
path_to_idm_install \export\pipeline\config\WSTrace2.log |
If you are sending trace information to stdout, open your application server’s stdout file in a text editor to view the trace output logs.
Waveset is a Java-based product whose executables consist of Java classes grouped as packages. When the code is implemented, many classes can supply trace output.
Tracing the Waveset server can provide helpful information such as where a server is failing, having problems, or not running. You can use the Waveset Debug pages to enable package-level and method-level tracing in a running Waveset server.
Configure Waveset to trace at this advanced level only if instructed by Oracle Support.
You can use trace information to verify that a resource adapter has started, to verify that all setting changes were saved, and to diagnose problems with the adapter.
This section contains information to help you configure adapter tracing, and is organized into the following topics:
When you enable tracing for an adapter, you must identify the methods that you want to trace by typing:
com.waveset.adapter.sample.MyResourceAdapter |
You must also provide calls to create log entries for any new methods in your custom resource adapter so the adapter can write its resource settings to a log file.
In some cases, you can specify additional logging parameters for the resource instance, such as:
Maximum Log Archives
Maximum Active Log Age
Log File Path
Maximum Log File Size
Log Level
To further debug the synchronization process, you can configure synchronization logging for an Active Sync adapter. Instructions are provided in the Waveset online help.
Tracing custom Java code can be useful when you are writing your own resource adapter. See Tracing Custom Code for more information.
You can instrument code with trace points to help assess and resolve problems more effectively. See Instrumenting Code with Trace Points for more information.
For more information about tracing specific adapter methods, see the Oracle Waveset 8.1.1 Resources Reference.
The following tasks were moved from the adapter to the task package in Identity Manager version 8.0. You must update the paths if you have tracing enabled for any of these tasks, or if you have customized task definitions referencing these packages.
Old Package Name |
New Package Name |
---|---|
com.waveset.adapter.ADSyncFailoverTask |
com.waveset.task.ADSyncFailoverTask |
com.waveset.adapter.ADSyncRecoveryCollectorTask |
com.waveset.task.ADSyncRecoveryCollectorTask |
com.waveset.adapter.SARunner |
com.waveset.task.SARunner |
com.waveset.adapter.SourceAdapterTask |
com.waveset.task.SourceAdapterTask |
The Waveset trace facility enables you to instrument code with trace points in the adapters component (resource adapters). This section contains some guidelines to help you to consistently use trace points to assess and resolve adapter problems.
Some guiding principles for instrumenting code with trace points are as follows:
Make trace points as useful as possible for troubleshooting in both production and development environments by displaying relevant information.
Create one entry point and one exit or exception trace point for every significant method.
Use trace points as efficiently as possible during normal operations and when tracing is enabled by
Minimizing the number of trace level checks
Minimizing object creation when tracing is enabled
You can specify a trace level for a trace point to control how much information displays. The following table describes each of the trace levels that are defined in the com.sun.idm.logging.TraceManager interface.
Table 5–2 Defined Trace Levels
Trace Level |
Trace Variable |
Usage |
---|---|---|
1 |
Trace.LEVEL1 |
Entry and exit points of public methods of the resource adapter interface |
2 |
Trace.LEVEL2 |
Entry and exit points of non-public methods or methods not included in the Waveset component external interface |
3 |
Trace.LEVEL3 |
Decision points or significant variables |
4 |
Trace.LEVEL4 |
Very detailed information, such as significant variables within a loop |
n/a |
Trace.ALWAYS |
Do not perform a trace-level check Note: Use this option if you have already specified the trace level in a condition. |
Use the com.sun.idm.logging.Trace class to prevent circular dependencies on the com.sun.idm.logging package that implements the com.sun.idm.logging.TraceManager interface.
When adding trace points to code, be aware of the following information:
Trace arguments to the method and return values from the method.
If the arguments “print large,” where the text representation is not brief, specify whether the Object is null or not. If the argument or return value is an array or a list, and the value could print large, showing the array size or list size is sufficient. You can use the com.waveset.util.Util.length() utility method to guard against null values.
To format trace information or to create objects that present trace information in a meaningful way, put an if statement around the trace statements, using the Trace.getLevel method and the isLogging(level) method, so that Waveset does not marshal the trace point data unless the specified trace level is enabled.
When using this construct, specify the Trace.ALWAYS trace level for trace points inside the condition. This trace level prevents an unnecessary check for the trace level when that check has already been performed in the condition.
If you are tracing an object value that requires a method call to get a printable representation of the object, such as obj.getName(), guard against a null object to prevent a NullPointerException occurring if trace becomes enabled.
In general, do not use trace points for simple getter and setter methods or for any other trivial methods.
When tracing a WSUser object, use the toString() method instead of the getName() method to present the value of the object. The toString()) method is more robust and prints more useful information than the getName() method. Instead of tracing user.getName(), trace user, which calls toString() implicitly.
If a method or constructor is just a wrapper, where that method or constructor does nothing significant except call another method or constructor with the same name but a different signature, just put trace points into the destination method or constructor.
If performance is critical and the method is called many times, use information trace points to indicate errors. Do not add entry or exit trace points to the method. Use this option carefully.
Before adding a trace point, ask yourself if that trace point will provide useful information when determining problems in the field.
The following sections describe specific trace levels in greater detail and provide examples showing how to use trace points in code.
Consider these guidelines before adding entry and exit trace points to your code:
Use Trace.Level1 entry or exit trace points for all Waveset component external interface methods, which are the public methods declared in the resource adapter interface.
Use Trace.Level2 entry or exit trace points for significant non-public methods including constructors and static methods or methods not included in the Waveset component external interface.
Specify arguments in the trace points.
If the argument is not a primitive type or java.lang.String, and the argument requires any kind of object creation or method calls, conditionalize the entry trace point so that the formatting work is only done when tracing is enabled.
When specifying an exit trace point, show the return value if the method is a primitive type or java.lang.String. If the return value requires some sort of formatting, conditionalize the object formatting as specified in these guidelines. Use the following:
int getLevel() int getLevel (Method) boolean isLogging(level,method) boolean level1 (method) boolean level2 (method) boolean level31 (method) boolean level4 (method) |
If you are tracing information that requires appending several java.lang.Strings together, use java.lang.StringBuffer instead of java.lang.String. Appends are faster using java.lang.StringBuffer. Also, conditionalize as in the preceding bullet item.
When adding entry or exit trace points for significant constructors, remember that you cannot place a trace point before a this() or super() method call within a constructor. Put the entry trace point immediately after the this() or super() method call.
Do not use an exit trace point to trace an exception condition. If the method throws an exception, use an exception trace point at the same trace level as the entry or exit trace point just before the exception is thrown. See Using Exception Trace Pointsfor more information.
If a method contains an entry trace point, that method also must contain a code path that executes either a throwing method, a caught method, or an exit.
Following are some simple entry and exit trace statements. For these examples, assume the following CLASS variables are declared for each statement.
private static final String CLASS = "com.waveset.adapter.MyResourceAdapter"; protected static Trace _trace = Trace.getTrace();
final String METHOD = methodName; _trace.entry(_trace.LEVEL1, CLASS, METHOD); _trace.entry(_trace.LEVEL1, CLASS, METHOD, user); if (_trace.level1(CLASS, METHOD)) { _trace.entry(_trace.ALWAYS, CLASS, METHOD, user= + user); } // Show the size of an array argument // ASSUME: password is an argument to the method // Note the use of the Util.length() method. It is // a convenience method that guards against null. if (_trace.level1(CLASS, METHOD)) { StringBuffer sb = new StringBuffer(32); sb.append(password length=); ab.append(Util.length(password)); _trace.entry(_trace.ALWAYS, CLASS, METHOD, sb.toString()); }
_trace.exit(_trace.LEVEL1, CLASS, METHOD); _trace.exit(_trace.LEVEL1, CLASS, METHOD, returnValue); if (_trace.level1(CLASS, METHOD)) { _trace.exit(_trace.ALWAYS, CLASS, METHOD, returnValue != null ? returnValue.getName() : returnValue); } // Show the size of an array String[] accounts = ... if (_trace.level1(CLASS, METHOD)) { StringBuffer sb = new StringBuffer(32) sb.append(accounts length=); ab.append(accounts.length); _trace.exit(_trace.ALWAYS, CLASS, METHOD, sb.toString()); }
Consider these guidelines before adding information trace points to your code:
Generally, use Trace.Level3 or Trace.Level4 for information, variable and data trace points. The trace level depends upon the significance, expense, and the verbosity of the information presented.
Use Trace.Level3 and Trace.Level4 information and variable trace points to trace significant conditions, branches, and other information as they occur in the code path.
A variable trace point is a convenience method that is the same as an information trace point, except that a variable trace point prints out variable=<variable> instead of just printing the argument value. In addition, this method does not format the information unless the trace level specified in the argument is matched.
Do not use an information trace point if the method contains an exception condition but does not rethrow the exception. Use the caught method. For an example, see Using Exception Trace Points.
Use Trace.Level3 and Trace.Level4 data trace points to show information in byte arrays. You can use a lower trace level if the information is appropriate for that trace level and the information presented is brief.
You can use Trace.Level1 and Trace.Level2 information, variable, and data trace points to trace significant, but brief, information. However, you should rarely use these trace points at these levels.
Following is an example of a simple information trace statement. For this example, assume the following CLASS variables are declared.
private static final String CLASS = "com.waveset.adapter.MyResourceAdapter"; protected static Trace _trace = Trace.getTrace();
_trace.info(_trace.LEVEL3, CLASS, METHOD, Some Message); WavesetResult result = new WavesetResult(); try { someMethod(); } catch(Exception e) { String msg = Some Error Message; WavesetException we = new Waveset(msg, e); _trace.caught(_trace.LEVEL3, CLASS, METHOD, e); result.addException(we); }
Consider these guidelines before adding exception trace points to your code:
Use exception trace points in all methods where entry or exit trace points are specified, and in methods that catch and rethrow exceptions. Waveset sometimes throws exceptions in which a new exception wraps the original exception. Use the same trace level being used by the entry and exit trace points.
Use an exception trace point if an exception is created and thrown from the current method.
Use an exception trace point caught method if an exception is caught and handled. For example, when the exception is not thrown from the current method. This situation typically occurs when Waveset catches and handles the exception by placing it in a container, such as a WavesetResult, to be examined later.
If the exception thrown from the current method is not a checked exception, which is an exception that extends java.lang.RuntimeException or java.lang.Error, the method exit point will not be traced unless other measures are taken. One way to set an exception trace point in this situation is to use a try/catch block around the critical section of the method and rethrow the exceptions that occur. In general, this means that you should use Trace.Level3 and higher when it is critical to know whether a method has succeeded or failed.
Following is an example of a simple exception trace statement. For this example, assume the following CLASS variables are declared:
private static final String CLASS = "com.waveset.adapter.MyResourceAdapter"; protected static Trace _trace = Trace.getTrace();
try { someMethod(); } catch(Exception e) { _trace.throwing(_trace.ALWAYS, CLASS, METHOD, e); throw e; } try { someMethod(); } catch(Exception e) { _trace.throwing(_trace.ALWAYS, CLASS, METHOD, e); WavesetException we = new WavesetException(Some Message, e); throw we; } if (error) { WavesetException e = new WavesetException(Some Error Message.; _trace.throwing(_trace.LEVEL3, CLASS, METHOD, e); throw e; } try { someMethod(); } catch(Exception e) { _trace.caught(_trace.LEVEL1, CLASS, METHOD, e); } // execution continues. someOtherMethod();
Using trace point templates can help you provide consistent and useful trace points in Waveset resource adapters. Waveset provides Eclipse templates in the following location:
src/wps/doc/eclipse-trace-templates.xml
To import these templates into Eclipse, select Window -> Preferences -> Java -> Editor -> Templates.
You can create similar templates if you are using Emacs or IDEA.
This section describes some additional tracing guidelines, including the following:
Remember to trace significant methods in inner classes. Be sure to declare a final static CLASS variable if there are any trace methods in the inner class.
private final static String CLASS = com.waveset.adapter.SomeResourceAdapter$AdapterInnerClass;
In general, do not use the trace facility in static initializers. Waveset executes these initializers when the class is loaded, which can occur when the repository is initialized. Use the Debug class to trace anything significant in static initializers.
You can trace the following methods to troubleshoot issues with Identity Auditor:
com.sun.idm.auditor.policy to trace issues with Audit Scans.
com.sun.idm.auditor.accessreview to trace issues with Access Reviews.
com.sun.idm.auditor.report to trace issues with Audit Reports.
com.sun.idm.auditor.view to trace issues with Auditor Views.
Open a browser and log in to the Administrator interface.
Select Configure -> Servers.
When the Configure Servers page displays, click the server name in the Server column to edit the settings.
On the Edit Server Settings page, click the Scheduler tab.
Select the Tracing Enabled box to activate Scheduler debug tracing and write the results to stdout.
Save your changes.
You can use the Waveset tracing facility in custom-written code to provide seamless integration when troubleshooting your deployment.
Use the following classes to provide this tracing facility:
Use com.sun.idm.logging.trace.Trace objects to interface with the tracing subsystem.
Use com.sun.idm.logging.trace.TraceManager as a factory for these objects.
You can also use these com.sun.idm.logging.trace facilities to record trace output. See the Javadoc for more information about these facilities and classes.
Be aware that the com.sun.idm.logging.trace facilities were not available prior to the Waveset Version 6.0 SP1 release. For earlier versions of Waveset, you must use the com.waveset.util.Trace class instead.
Exception logs are stack traces that you can view from the Waveset Debug pages or from the config/Waveset.properties file. Trace data does not include all exceptions by default, but exception logging can be an important and informative troubleshooting tool.
Use one of the following methods to enable exception logging:
Open the Waveset Properties page (debug/Show_WSProp.jsp) in the Waveset Administrator interface. Locate the exception.trace key and change the Value to true.
Open the config/Waveset.properties file in a text editor, and change the exception.trace key value to true. The following figure shows an example exception trace.
Waveset sends exception logs to stdout on the web application instance, which is often the application server console.
When you are finished, disable exception logging to stop unnecessary output to the application server logs. To disable exception logging, set the exception.trace key value back to false.
You can enable tracing to troubleshoot edited forms and to check for expression statement errors within your form fields.
Use either of the following methods to enable tracing:
Open the Waveset Properties page (debug/Show_WSProp.jsp) in the Waveset Administrator interface. Locate the form.trace key and change the Value to true.
Open the config/Waveset.properties file in a text editor, and change the form.trace key value to true.
Waveset reports any problems with form expression syntax to standard output.
The form.trace key is disabled by default because it produces trace information for every field on every page, including the Accounts List page, which affects system performance. Consider using a more targeted form and field tracing method.
When you are finished troubleshooting your forms, remember to disable tracing by changing the form.trace key value back to false.
Global XPRESS tracing might also be helpful while you are developing and updating forms and form processes. Although Global XPRESS tracing produces a large amount of output that affects system performance, this tracing method shows XPRESS output and might expose where problems are happening in your form.
For more information, see
While not generally recommended, you can use global XPRESS tracing to trace any and all XPRESS code, wherever the code is located. For example, you can trace XPRESS code in forms, views, and workflows. The resulting trace shows XPRESS output that can expose potential problems.
XPRESS tracing is disabled by default because it produces a large amount of output, which affects system performance.
See Testing Your Customized Form in Oracle Waveset 8.1.1 Deployment Reference for more information about tracing XPRESS functions.
Open a command window.
Change directories to config/Waveset.properties in the default Waveset installation directory.
Open the config/Waveset.properties file and edit the xpress.trace line to read:
xpress.trace=true |
Save the Waveset.properties file.
Restart your application server or reload the Waveset.properties file from the Waveset debug pages.
Replicate the XPRESS trace output to a file by adding this line to the Waveset.properties file:
xpress.traceFile=FileName.txt |
xpress.traceFileOnly=true |
When you set xpress.traceFileOnly=true in Waveset.properties, all XPRESS statement evaluations will generate trace messages to a file specified by xpress.traceFile. Otherwise, when xpress.traceFile has a value, trace messages are redirected to both the console and a file.
This section describes how to enable tracing for PasswordSync and how to configure tracing in Direct access or JMS modes.
You can use the following methods to configure tracing for Waveset’s PasswordSync feature:
This section describes how to configure tracing from the PasswordSync Configuration Trace tab.
For more information about installing and configuring PasswordSync, see Chapter 11, PasswordSync, in Oracle Waveset 8.1.1 Business Administrator’s Guide.
The first time you run the configuration tool, the wizard does not allow you to configure tracing. Subsequently, when you run the configuration tool, the wizard displays a Trace tab where you can configure tracing.
The following figure shows the PasswordSync Configuration tool Trace tab.
From this tab, you can specify the following:
Use the Trace Level field to specify the level of detail you want PasswordSync to provide when writing to the trace log. A value of 0 turns tracing off, while a value of 4 shows full detail.
Use the Max File Size field to specify a maximum size for the log file.
When the trace file exceeds the size specified in the Max File Size (MB) field, PasswordSync starts a new trace file and appends .bk to the old trace file name. For example, if your trace level is set to 100 Mbytes, and your trace file writes to C:\logs\pwicsvc.log, when the trace file exceeds 100 Mbytes PasswordSync renames the file to C:\logs\pwicsvc.log.bk. PasswordSync then creates a new C:\logs\pwicsvc.log file where new trace file messages are written.
Use the Trace File field to specify a location for the PasswordSync trace file.
To enable additional PasswordSync configuration settings, edit the following PasswordSync registry keys using the PasswordSync configuration tool.
Using PasswordSync configuration tool is the safest method for editing PasswordSync registry keys. Editing these keys directly in the Windows Registry is discouraged.
Key Name |
Type |
Description |
---|---|---|
dumpFilebase |
REG_SZ |
Set this registry key to enable Windows to generate a dump file if the PasswordSync DLL displays an exception. You must set this registry key to the fully qualified directory path where you want to write the memory dump. For example: c:\temp Set the registry value to write the memory dump each time Waveset catches an exception during password processing. Note: On Windows 2000 server (any service pack), you also must install in the configured directory DbgHelp.dll, which is available from Microsoft. The minimum release version for the DbgHelp.dll file must be Version 5.1. Download the DbgHelp.dll file here: http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx If DbgHelp.dll is not installed, no dump files will be generated on Windows 2000. The format for dump file names is lhpwic-YYYYMMDD-HHmm-xxxxx.dmp In this name, YYYYMMDD will be the date of the dump, HHmm is the time of the dump (24-hour clock), and xxxxx is the thread number of the application. You must manually remove dump files! Dump files range in size from 20 MB to more than 100 MB, depending on the size of the Windows Local Security Authority Subsystem (LSASS) process. Over time, systems with limited disk space could fill up if these dump files are not removed. |
installdir |
REG_SZ |
Directory where the PasswordSync application is installed. |
The PasswordSync registry keys are located in the following location:
HKEY_LOCAL_MACHINE\SOFTWARE\Sun Microsystems\Sun Identity Manager\Passwordsync
Other keys are present in this location.
PasswordSync trace logs are the same, whether you are using a direct access mode or JMS mode configuration. However, these trace logs might only provide partial information. You must configure different classes for each configuration to collect logs on the server side, as described in the following sections.
When using PasswordSync with a direct access mode configuration, the trace logs show failures, but not all logged failures are real failures. For example, in some circumstances the view check-in takes a long time, which shows as a failure in the log. You must trace on the server side to see this information.
In Direct mode, PasswordSync talks to the servlet that generates the view to be checked into the repository. You can trace the com.waveset.rpc.GenericMessageHandler class at level 4 to view all phases of password synchronization, from receiving the password change to the response generated and returned to the servlet. Level 4 is the only level that supplies enough detail for troubleshooting.
When using PasswordSync with a JMS mode configuration, the logs only show successful or failed deliveries to the JMS server. From this point on, you must rely on server side logs. JMS tracing is a little more complex.
You can trace the com.waveset.rpc.PasswordSyncHandler class at level 4 to convert the messages generated by the PasswordSync dll into a JMS message and add those messages to the JMS queue. Limited tracing is available in this class, and only level 4 can provide enough information to help with troubleshooting.
If PasswordSync successfully delivers the JMS message to the JMS queue, the tracing will not help you find the cause of a problem. The next, and final step is to trace the JMS adapter. See the Oracle Waveset 8.1.1 Resources Reference for instructions.
You can trace rule-driven members caches and use the results to tune cache properties in Waveset.properties.
If you trace com.waveset.server.RuleDrivenMembersCache at Level 1, the resulting information includes the number of adds, removes, hits to the cache, and so forth. Use this information to evaluate cache sizes and decide whether tuning the cache properties in Waveset.properties is necessary.
You can use the following properties in Waveset.properties to control the rule-driven members cache:
Use ruledrivenmemberslistcache.size = value to specify the maximum number of object lists to be cached per subject. (Default is 20.)
Use ruledrivenmemberslistcache.rowlimit = value to specify the maximum total number of objects to be cached per subject. (Default is 100000.)
By default, Waveset evaluates the User Members Rule associated with a specified organization and creates a user members list cache containing a dynamic list of users. However, you can also create a user members list cache containing a specified user's set of dynamic member organizations for a given subject. The key to this cache is the object type concatenated with the object ID. For example, the User object type concatenated with the User#ID#Configurator object ID. The value for each key is a list of object groups for which this object is dynamically a member.
To determine whether the object being evaluated is a dynamic member, the cache evaluates the same User Members Rule per organization as that used by the list cache. If the object is a dynamic member, Waveset adds that object to the list and then caches the list. Waveset caches both empty and non-empty lists to ensure the highest cache hit rate.
For this cache, you can use these properties in Waveset.properties to control memory requirements affecting performance:
Use ruledrivenmembersobjectcache.size = value to specify the number of member object group lists to be cached per subject. The default value is 100.
Use ruledrivenmembersobjectcache.rowlimit = value to specify the maximum total number of member object groups to cache per subject. The default value is 100000.
Enabling Waveset tracing at Method/Class Level 2 for the following classes allows you to trace authorization flows when listing or accessing Oracle Waveset Service Provider (Service Provider) users and when AdminRoles are dynamically assigned when Service Provider users login.
Trace com.sun.idm.idmx.view.IDMXBrowseViewer when searching for Service Provider users.
Trace com.sun.idm.idmx.view.IDMXUserViewer when creating, editing, or deleting Service Provider users.
Trace com.sun.idm.idmx.api.IDMXServerUtils when using both of the preceding classes.
Trace com.waveset.security.authn.WSSPELoginModule when you are logging in as a Service Provider user.
You configure Service Provider tracing from the Waveset debug pages. If necessary, review Tracing the Waveset Server for instructions.
If you are having problems with a reconciliation task, you can use the standard tracing facility on com.waveset.task.Reconciler to trace the Reconciler.
Use either of the following methods to enable tracing:
Open the Waveset Properties page (debug/Show_WSProp.jsp) in the Waveset Administrator interface. Locate the exception.trace key and change the Value to true.
Open the config/Waveset.properties file in a text editor, and change the exception.trace key value to true.
You can also enable tracing from the System Settings page and trace the following reconciliation methods, at the Method/Class trace Levels noted, to view useful debugging information.
Table 5–4 Reconciliation Methods/Classes to Trace
Trace this com.waveset.recon.Method/Class |
At this Trace Level |
To See |
---|---|---|
ReconTask$WorkerThread#reconcileAccount |
2 |
The individual account being reconciled |
ReconTask$WorkerThread#performResponse |
2 |
The individual account and user during response |
ReconUtil#deleteAccountIndex |
2 |
User information to be deleted from the account index |
UserContext#aquireRepoLock |
2 |
The user who is being locked for update |
UserContext#releaseRepoLock |
2 |
The user who is being unlocked in the repository |
ReconUtil#deleteAccountIndex |
2 |
User information to be deleted from the account index |
UserContext#aquireRepoLock |
2 |
The user who is being locked for update |
ReconTask$WorkerThread#failUserExamination |
2 |
All user examination requests that failed with the error |
ReconTask$WorkerThread#failUserResponses |
2 |
All user response requests that failed with the error |
UserContext#releaseRepoLock |
2 |
The user who is being unlocked in the repository |
ReconTask$ResourceThread#examineResource |
3 |
How many accounts read from the resource |
ReconTask$ResourceThread#queueAccountReconciles |
3 |
Information about each account read from the resource such as accountId, accountGUID, accountDisabled |
ReconTask$ResourceThread#examineLighthouse |
3 |
How many Waveset users claim to own an account on the reconciled resource queued |
ReconTask$WorkerThread#findClaimants |
3 |
All Waveset users who claim to have an account on the resource |
ReconTask$WorkerThread#confirmPossibleOwners |
3 |
A list of all confirmed owners of resource accounts |
ReconTask$WorkerThread#applyResponse |
3 |
The response list that is being applied |
AccountContext#processAttributeWorkflow |
3 |
The attribute changes and the formatted changes during the launch of the attribute change workflow |
ReconUtil#getRuleState |
3 |
Full User view with the user’s attribute during rule processing |
ReconUtil#evaluateCorrelationRule |
3 |
The value of the correlation rule state and result for examination |
ReconUtil#confirmPotentialOwners |
3 |
A list of users who have been confirmed using the confirmation rule |
ReconUtil#getIfExistsAccountIndexEntry |
3 |
To output info related to examination of the account index for a specified entry |
ReconUtil#launchWorkflow |
3 |
The task instance and task definition information when launched |
ReconUtil#indexFoundAccount |
3 |
The account and situation recorded during a create or update of the index for an account known to exist |
ReconUtil#indexMissingAccount |
3 |
The account and situation recorded during a create or update of the index for an account NOT known to exist |
ReconUtil#listAccountsIndexSaysExist |
3 |
Account information that the index says exists |
ReconTask$ResourceThreadwaitForLighthouseWorkItems# |
4 |
How many users queued and processed during the Waveset user examination process |
ReconTask$ResourceThread#waitForReconcileWorkItems |
4 |
How many reconciles and responses queued and processed for the specified resource |
ReconTask$WorkerThread#correlateUsers |
4 |
A list of correlated/confirmed users |
ReconTask$WorkerThread#respondOrRequeue |
4 |
The error message if there is a problem performing the previous response method for an account |
Response#perform |
4 |
The user, response, and the user’s resource info XML before and after the modifications took place |
Response#perform |
4 |
A summary of the response action on the user |
Response#createNewUserFromAccount |
4 |
Full details of the response action on the user |
ReconUtil#evaluateConfirmationRule |
4 |
The value of the confirmation rule state and result for examination |
ReconUtil#getCorrelatedUsers |
4 |
A list of correlated users matching the specified rule result |
Remember, the higher the tracing level, the larger the trace file. Also tracing all of these methods at the same time will create a very large trace file.
When you are finished troubleshooting, remember to disable tracing by setting the exception.trace key value back to false.
If you see errors while you are using the setRepo command to configure the Waveset repository, use the following flags to isolate and debug the problem:
-Dtrace.enabled=true -Dtrace.level.com.waveset.repository.AbstractDataStore=2 -Dtrace.level.com.waveset.repository.DefaultTypeHandler=4 // Use one of the following based on your repository type -Dtrace.level.com.waveset.repository.OracleDataStore=4 -Dtrace.level.com.waveset.repository.SqlServerDataStore=4 -Dtrace.level.com.waveset.repository.MysqlDataStore=4 -Dtrace.level.com.waveset.repository.DB2DataStore=4 |
Waveset sends output from the setRepo command to the default $WSHOME/config/WSTrace.log file.
This section describes methods for enabling trace for SPML Version 1.0 and SPML Version 2.0.
SPML 1.0 contains the following options for turning on trace output so you can log Waveset’s SPML traffic and diagnose problems.
You can use the setTrace method, provided by the SpmlClient and LighthouseClient classes, to enable tracing for SPML 1.0.
When you enable this setTrace method, the XML for the request sent by the client and the XML for the response received from the server are printed to the client console as they are sent and received.
The setTrace method takes a Boolean argument. For example:
SpmlClient client = new SpmlClient(); client.setURL("http://localhost:8080/idm/spml"); client.setTrace(true); |
You can enable tracing when initializing the org.openspml.server.SOAPRouter servlet, which is a third-party, open source class from the OpenSPML organization. This servlet controls the output of RPC traffic information for the servlet handling SPML requests.
To enable this tracing method, add the following to the WEB-INF/web.xml file:
<servlet> <servlet-name>rpcrouter2</servlet-name> <display-name>OpenSPML SOAP Router</display-name> <description>no description</description> <servlet-class> org.openspml.server.SOAPRouter </servlet-class> <init-param> <param-name>trace</param-name> <param-value>true</param-value> </init-param> ... </servlet> |
The following is sample output for an SPML 1.0 trace:
SpmlClient: sending to http://example.com:8080/idm/servlet/rpcrouter2 <spml:addRequest xmlns:spml='urn:oasis:names:tc:SPML:1:0' xmlns:dsml='urn:oasis:names:tc:DSML:2:0:core'> <spml:operationalAttributes> <dsml:attr name='session'> <dsml:value>session token</dsml:value> </dsml:attr> </spml:operationalAttributes> <spml:identifier type='urn:oasis:names:tc:SPML:1:0#GUID'> <spml:id>suetonius</spml:id> </spml:identifier> <spml:attributes> <dsml:attr name='objectclass'> <dsml:value>person</dsml:value> </dsml:attr <dsml:attr name='password' <dsml:value>password</dsml:value> </dsml:attr> <dsml:attr name='gn'> <dsml:value>Suetonius</dsml:value> </dsml:attr> <dsml:attr name='sn'> <dsml:value>Tranquillus</dsml:value> </dsml:attr> <dsml:attr name='email'> <dsml:value>twelve@example.com</dsml:value> </dsml:attr> </spml:attributes> </spml:addRequest> SpmlClient: received <?xml version='1.0' encoding='UTF-8'?> <SOAP-ENV:Envelope xmlns:SOAP-ENV='http://schemas.xmlsoap.org/soap/envelope/'> <SOAP-ENV:Body> <spml:addResponse xmlns:spml='urn:oasis:names:tc:SPML:1:0' xmlns:dsml='urn:oasis:names:tc:DSML:2:0:core' result='urn:oasis:names:tc: SPML:1:0#success'> <spml:operationalAttributes> <dsml:attr name='session'> <dsml:value>session token</dsml:value> </dsml:attr> </spml:operationalAttributes> <spml:identifier type='urn:oasis:names:tc:SPML:1:0#GUID'> <spml:id>suetonius</spml:id> </spml:identifier> </spml:addResponse> /SOAP-ENV:Body> </SOAP-ENV:Envelope> |
For more information about the SOAP rpcrouter servlet, refer to your OpenSPML Toolkit documentation.
You can enable tracing for an individual SPML RPC request by passing a trace operational attribute to the RPC request on the server side.
Tracing occurs during servlet initialization, and it controls how information is output for the RPC traffic of a servlet handling SPML Version 1.0 requests. For example, the trace prints the raw XML that is sent back and forth on whatever the System.out is for that servlet (which is a function of the Application container). For example:
AddRequest ar = new AddRequest(); ar.setOperationalAttribute("trace", "true"); |
When you use the trace attribute, how the attribute affects server operation is vendor-specific. Currently, Waveset prints the raw request and response data to the server console, which is useful if the client application is not associated with a console window.
For more information consult your OpenSPML Toolkit product documentation.
SPML 2.0 contains the following options for turning on trace output so you can log Waveset’s SPML traffic and diagnose problems.
As with SPML 1.0, you can enable tracing for SPML 2.0 when initializing the org.openspml.v2.transport.RPCRouterServlet class, which controls the output of RPC traffic information for the servlet handling SPML 2.0 requests.
To enable this tracing method, add the following to the WEB-INF/web.xml file:
<servlet> <servlet-name>openspmlRouter</servlet-name> <display-name>OpenSPML SOAP Router</display-name> <description>A router of RPC traffic - nominally SPML 2.0 over SOAP</description> <servlet-class> org.openspml.v2.transport.RPCRouterServlet </servlet-class> <init-param> <param-name>trace</param-name> <param-value>true</param-value> </init-param> ... </servlet> |
The following example illustrates output from an rpcrouter servlet trace:
RPCRouterServlet: <?xml version='1.0' encoding='UTF-8'?><SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header/> <SOAP-ENV:Body><lookupRequest xmlns='urn:oasis:names:tc:SPML:2:0' requestID='random name' executionMode='synchronous' returnData='everything'> <openspml:operationalNameValuePair xmlns:openspml='urn:org:openspml:v2:util:xml' name=' session'value=session token'/> <psoID ID='random name' targetID='spml2-DSML-Target'/> </lookupRequest> </SOAP-ENV:Body></SOAP-ENV:Envelope> RPCRouterServlet: response: <?xml version='1.0' encoding='UTF-8'?> <SOAP-ENV:Envelope xmlns:SOAP-ENV='http://schemas.xmlsoap.org/soap/envelope/'> <SOAP-ENV:Body> <lookupResponse xmlns='urn:oasis:names:tc:SPML:2:0' status='failure' requestID='random name'error='noSuchIdentifier'> <openspml:operationalNameValuePair xmlns:openspml='urn:org:openspml:v2:util:xml' name='session' value=session token/> <errorMessage>Item User:random name was not found in the repository, it may have been deleted in another session.</errorMessage> </lookupResponse> </SOAP-ENV:Body> </SOAP-ENV:Envelope>
For more information see Method 2: Initializing the org.openspml.server.SOAPRouter Servlet, in the To Enable Tracing for SPML 1.0 section.
SPML 2.0 has a simple text access log that can be a useful troubleshooting tool. This log is always available and it enables you to view information, such as what kind of requests have been received, how long it took to process those requests, and whether the requests were successful without having to enable tracing.
Instructions for configuring this SPML text access log are provided in Configuring SPML Tracing in Oracle Waveset 8.1.1 Web Services
If a Scheduler task is having problems, you can use the standard tracing facility on com.waveset.task.Scheduler to trace the task scheduler. The output shows detailed information about task scheduling.
Open a browser and log in to the Administrator interface.
Select Configure -> Servers.
When the Configure Servers page displays, click the server name in the Server column to edit the settings.
On the Edit Server Settings page, click the Scheduler tab.
Select the Tracing Enabled box to activate Scheduler debug tracing and write the results to stdout.
Save your changes.
In a clustered environment, tracing occurs on each instance.
See Tracing the Waveset Server for more information about defining and editing trace configuration objects, and about viewing trace files.
Enabling workflow tracing can help you resolve problems with workflow activities and understand the workflow process.
In a clustered environment, tracing occurs on each instance.
To debug workflows in a multiple server deployment environment, consider shutting down all but one server. If all servers are running, you cannot determine which server in the environment is executing the workflow, which causes troubleshooting problems.
Use the following steps to enable workflow tracing:
Open a browser and log in to the Waveset Administrator interface.
From the System Setting page, choose Configuration from the List Objects Type menu.
Click the List Objects button.
When the List Objects of type: Configuration page displays, scroll down the list of objects to locate the System Configuration object and click the edit link.
When the Checkout Object: Configuration, #ID#CONFIGURATION:SYSTEMCONFIGURATION page displays, you can edit any of the following workflow options in the SystemConfiguration object:
Typically, you enable only one option, but it is possible to enable more than one option at a time.
These attributes are not dependent on each other. You can turn on one type of trace while the other types are turned off.
Specify workflow.consoleTrace=true to redirect workflow trace messages to the application server console, which can be useful when workflows are terminating due to a fatal exception because this attribute prints more trace output than workflow.fileTrace. (Default value is false.)
Specify workflow.fileTrace=PathtoFileto redirect workflow trace messages to a file that is easy to read. This attribute does not have a value by default.
<Attribute name=’fileTrace’/>
Add a value tag to the workflow.fileTrace attribute and, using Unix-style forward slashes, enter the path to a log file. Waveset stores relative pathnames relative to the application server’s installation directory. For example:
On Windows. <Attribute name=’fileTrace’ value=’C:\mydir\workflow.txt’/>
On Solaris/UNIX. <Attribute name=’fileTrace’ value=’/mydir/workflow.txt’/>
Specify workflow.traceLevel=tracingLevel to specify the level of workflow tracing you want to see.
Specify workflow.Trace=true to trace workflow processing. You must restart the application server to start tracing with this option. Waveset stores the trace results in the task’s WavesetResult object. Use this tracing option when file system access is unavailable. (Default value is false.)
Trace messages in the task’s WavesetResult. (Default value is 1.)
Specifying workflow.Trace=true appends trace messages into one long, unformatted string that is difficult to read. Use this option only when you do not have access to the file system.
With the first two options, you might lose some of the workflow trace if a fatal exception occurs.
Save the SystemConfiguration object.
Restart your application server, or reload the SystemConfiguration object from the Waveset debug area.
Use one of the following methods to enable tracing for a designated workflow:
Editing the WFProcess definition. To enable trace unconditionally for a particular process, edit the XML of the TaskDefinition object by adding the trace=’console ’ attribute to the <WFProcess> element.
Editing the workflow variable. Gives you more control over the timing of tracing by using a workflow variable. The workflow engine will look for a top-level workflow variable named trace.
The following example shows how to trace a workflow variable.
<Variable name=’trace’> <cond><eq><ref>accountId</ref><s>jfaux</s></eq> <s>workflowTrace.txt</s> </cond> </Variable> |
The trace variable turns tracing on only if the workflow is operating on a user named jfaux. You could also specify trace in a form field to control tracing interactively.
In this example, the trace output is written to the workflowTrace.txt file.
You can use one of the following methods to get information about the Waveset version you are currently using:
Hover your cursor above the Help button in the upper right corner of the Waveset application window. A pop-up is displayed that contains the version number.
Open the Waveset.properties file, and check the information provided at the top of the file.
Use the Waveset System Properties page (/debug/SysInfo.jsp) to get information about which JVM versions, and other system software versions, you are currently using.
This section describes how to trace objects and activities in Oracle Waveset Gateway, the information is organized as follows:
When viewing or editing a Gateway trace file, use Notepad to avoid file restrictions.
When you start the Gateway, the program appends new trace entries to the trace file instead of deleting entries. To locate the point at which the Gateway trace entries begin, look for a Gateway version string.
The Gateway version is output in the trace automatically when you start the Gateway. You can also type gateway -v from the command line to get the version.
If you cannot access the Gateway, you can enable tracing to debug problems with Windows accounts on Waveset. You can enable or disable tracing from the Gateway Debug page (Gateway.jsp) or, if you want a wider range of options, you can enable tracing from the command line
Use the following instructions to enable and configure tracing from the Gateway Debug page (Gateway.jsp).
Log in to the Waveset Administrator interface.
Type the following URL in to your browser to open the Gateway Debug page:
http://host :port/idm/debug/Gateway.jsp
Choose a resource to trace from the Gateway Resource list.
If necessary, modify existing settings.
Click the following buttons to modify the settings:
Get Version. Returns the Gateway version and the operating system of the machine on which you are running the Gateway.
Get Trace File. Returns the contents of the trace file.
Get Trace Parameters. Returns the path of the trace file, the trace level, and the maximum size of the trace file.
Set Trace Parameters. See To Create a New Trace Configuration Object for information about these options.
Get Loaded Modules. Returns the load addresses of modules (DLLs) being used by the Gateway.
The Get Loaded Modules list consists of load addresses, followed by module names and only includes loaded modules. The list does not include delay-loaded modules that have not been called.
The Get Loaded Modules option only supports Active Directory and Domino.
Open a command window.
Start the Gateway.
To start the gateway as a service, type the following command:
#gateway -s
To start the gateway as a regular application rather than a service, type the following command:
#gateway —d
Do not use the -d and -s options at the same time on the same command line.
—p. Use this argument to specify a TCP/IP port number. (The default port is 9278.) For example,
#gateway -d -p 9278
Specify the necessary trace command arguments.
The following table describes the Gateway tracing command line arguments.
Usage: #gateway -l level -m size -f name -s
For example:
cd %WSHOME%\bin\winnt #gateway -p 11319 -l 2 -m 500 -f %CD%\gateway.trc -s |
The preceding invocation starts the Gateway with the following characteristics:
-p 11319 – Use port 11319
You must configure this port for Gateway resources from the Waveset resource configuration. For example, for an Active Directory resource
-l 2 – Trace the gateway at output level 2.
-m 500 – Limit the size of the trace log file to 500 Kilobytes.
-f C:\gateway.trc – Write the trace output to a file called gateway.trc in the C:\ directory.
-s – Start the gateway service.
If specified, Waveset saves -p, -l, -m, and -f values in the registry, so the next time you run Gateway from the command line or as a service, the same values are used.
If you started the gateway using the -d option, Waveset sends the Gateway trace output to the console and to a trace file.
Log in to the Waveset Administrator interface.
Type the following URL in to your browser to open the Gateway Debug page:
http://host:port/idm/debug/Gateway.jsp
Deselect the Show Trace option or delete the file name from the Get Trace File text box.
If necessary, open a command window.
Stop the Gateway service.
If you started the Gateway service using the -s option, stop the service by typing the following command:
#gateway -k
If you used the -d option to start the gateway as a regular application, stop the gateway by typing Control-C in the same command window you used to start the gateway.
Start the gateway with trace level 0 using the -l option. (You do not have to specify the -m or -f option.) For example
#gateway -d -l 0
Waveset encrypts all messages between the gateway and the Waveset server.
You can configure Waveset to log encryption and decryption message details while tracing at level 4 in the gateway, but this capability is disabled by default. Tracing encryption information at level 4 can cause a large amount of overhead because Waveset logs each message multiple times, once on entry to and once on exit from the encryption routines. In addition, Waveset limits the size of logged informational messages to 4000 characters, which can truncate large messages.
To enable message tracing, you must add the traceEncryptedInfo registry entry to the gateway registry key. This entry triggers a full log of each message before and after encryption and decryption. You must still use trace level 4 to trace this information.
You can use the following values for the traceEncryptedInfo registry entry.
1. Trace input and output messages for all encryption and decryption routines.
Any other value. Disables tracing.
No entry value (Default). Disables tracing.
Waveset preserves password encryption in the logs. Tracing encrypted information does not expose passwords in clear text.
The PowerShellExecutor.dll is an add-on that implements communication between the gateway and Microsoft PowerShell. You can use PowerShell to manage Exchange Server 2007 accounts. This add-on cannot share tracing facilities with the rest of the gateway and provides a similar stand-alone tracing facility as the rest of the gateway.
The trace configuration for the PowerShellExecutor is stored in the same registry key as the other gateway registry keys:
HKEY_LOCAL_MACHINE\Software\Waveset\Lighthouse\Gateway
You create this base key when you configure tracing through the Waveset debug pages or when you start the gateway with trace command arguments.
On shut down, the gateway writes the current PowerShellExecutor settings for the tracing to the registry. These settings include:
traceFileName
Content. File name for the trace output (registry type REG_SZ)
Default. " "
Name of the trace file to generate for the PowerShellExecutor tracing. Where the name:
Can be a fully qualified path, including the filename
Cannot end in a slash (\)
The full path, except the file, provided in the traceFileName must exist.
If configured, log rotation adds a timestamp to the configured filename after rotation, when the file is no longer active. This timestamp displays in the following format:
yyyyMMddHHmmss
traceLevel
Content. Trace level (registry type REG_DWORD)
Default. 0 (no tracing)
Allowed. 0–4
This key is shared with the rest of the gateway. The whole gateway always provides tracing at the same level.
traceMaxSize
Content. Maximum file size in bytes (registry type REG_DWORD or REG_QWORD)
Default. 100000 bytes
Minimum. 100000 bytes
Tracing text is written as UTF–8 encoded text with a byte order mark to make it portable to other systems.
traceMaxFiles
Content. Number of trace files (registry type REG_DWORD)
Default. 2
Minimum. 1
This setting controls the number of trace files to keep on the system. Setting the maximum number of files to keep to 1, causes the file to be overwritten when the maximum size is reached. The oldest file, based on last write time, is removed when the maximum number of files is reached.
traceConfigInterval
Content. Time out in milliseconds (registry type REG_DWORD)
Default. 300000 ms (5 minutes)
Minimum. 60000 ms (1 minute)
All trace settings are reread from the registry based on this timeout value. In a production environment, consider setting this value to a large value, such as 24 hours, to minimize overhead.
If the Gateway encounters a serious problem and exits abnormally, you can send the resulting Dr. Watson logs to Oracle Support for analysis.
You must have administrator privileges on the system to view these logs.
Open the Windows Event Viewer.
Open the application log.
Look for an event with DrWatson source.
Open the event to view detailed information.
Ensure that the Bytes option is selected for Data.
Right-click in the display dialog and choose Select all from the menu.
Type Ctrl-C to copy the information.
Paste the information into Notepad and save the file.
Send the file in an email to Oracle Support with a detailed description of the problem. Be sure to indicate which version of the Waveset Gateway you are running.
Use the information provided in the following sections to help diagnose and fix problems you might encounter as you work with Waveset:
For additional troubleshooting information, including the Waveset FAQ, go to the following URL:
https://sharespace.sun.com/gm/document-1.26.2296/IdM_FAQ.html?
You must sign up for a Share Space ID to access information provided on this site.
You can use several different debugging tools to help identify and fix problems in your Waveset deployment. These tools include:
You can use Waveset Debug pages to help identity and fix problems in your deployment. For example, you can enable or disable tracing for various activities and objects, collect statistical information, verify that processes are running, or investigate bottlenecks and memory problems.
The following table describes the most commonly used Debug pages and their actual .jsp file names.
For a comprehensive list of all Waveset Debug pages, open a command window and list the contents of the idm/debugdirectory.
See Working With Waveset Debug Pages for more information about these Debug pages.
You must have the Debug capability to access and execute operations from the Waveset Debug pages. If you do not have the Debug capability, an error message results. Administrators and the Configurator are assigned the Debug capability by default.
Open a browser and log into the Administrator interface.
Type the following URL to open the System Settings page:
http://host:port/idm/debug
where:
host is the local server on which you are running Waveset.
port is the number of the TCP port on which the server is listening.
From this page, you can enable or disable tracing for various Waveset activities and objects and use the information displayed on these pages to troubleshoot problems in your deployment.
Some Debug pages are not linked to the System Settings page, and you must type the page’s .jsp file name to open the page. For example:
http:// host:port/idm/debug/pageName.jsp
Where pageName.jsp is the particular Debug page you want to open.
The Oracle Waveset Integrated Development Environment (Waveset IDE) is Java application that enables you to view, customize, and debug Oracle Waveset (Waveset) objects in your deployment.
Specifically, the Waveset IDE provides a graphical Debugger that you can use to debug &Waveset forms, rules, and workflows. You can use this Debugger to set breakpoints and watches, step through code, examine and modify variables, examine classes and the callstack, follow threads, and run multiple sessions.
Instructions for installing and configuring the Oracle Waveset Integrated Development Environment (Waveset IDE) are now available from the following URL:https://identitymanageride.dev.java.net.
You can configure Waveset system monitoring to track system events. System monitoring collects and aggregates statistics at various levels to present a real-time view of system events, based on your specifications.
Viewing this information in dashboard graphs enables you to quickly assess system resources, view abnormalities, understand historical performance trends, and interactively isolate problems before looking at audit logs. Although dashboards do not provide as much detail as audit logs, dashboards can indicate where to look for problems in the logs.
For more information about dashboards and system monitoring, see Chapter 8, Reporting, in Oracle Waveset 8.1.1 Business Administrator’s Guide.
Adapter logs capture information about the adapter that is currently processing. You can use this information to monitor the adapter’s progress and to diagnose and debug adapter problems.
You must enable tracing and identify the methods for which tracing is requested before any logging can occur. Also, your customized adapter must include calls that create log entries for new methods.
Nearly every adapter has its own log file, path, and log level. You can specify the level of detail captured by the adapter log, along with these other values in the Logging section of the Synchronization Policy for the appropriate Waveset or Service Provider user type.
For more information about using adapter log files as a debugging tool, see Troubleshooting Adapters.
DTrace is a comprehensive, dynamic tracing framework for the Solaris operating environment. DTrace provides more than 30,000 probes into your production system and integrates user- and kernel-level tracing. You can use DTrace to monitor JVM activity. This facility also allows you to use the D language (similar to C or awk) to trace arbitrary data and expressions.
The Java Monitoring and Management Console (JConsole) is a Java Management Extension (JMX) technology-compliant graphical management tool bundled with JDK 5 (and later). JConsole connects to a running JVM and gathers information from the JVM MBeans in the connected JMX agent.
For example, you can use JConsole to
Detect low memory
Enable or disable garbage collection
Enable or disable verbose tracing
Detect deadlocks
Control Waveset log levels
Access information about operating systems resources (Oracle’s platform extension)
Monitor and manage MBeans
View information about the JVM and monitored values, threads running on the application, and class loading
For more information about JConsole, see the article titled, Using JConsole to Monitor Applications. You can view this article from the following URL:
http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html
If a red error message displays in the Waveset interface after you have performed an action, you might be able to view more complete information and further analyze the error by viewing and saving the page source.
To view the page source
If you are using Internet Explorer, select View -> Source from the menu bar.
If you are using Netscape, select View -> Page Source from the menu bar.
If you still need help resolving the problem,
View the page source, and then select File -> Save to save the file to your system.
Locate the error in your saved file.
Send the error information, the URL from the page where the problem occurred, and a description of the problem in an email to Oracle Support for resolution assistance.
To troubleshoot an adapter, review the adapter’s log file. Almost all adapters write their resource settings to a log file, and you can use this information to confirm that the adapter started and that all setting changes have been saved.
You must enable tracing and identify the methods for which tracing is requested before any logging can occur. Also, your custom adapter must include calls that create log entries for new methods.
If necessary, review Tracing the Waveset Server for instructions about how to enable tracing.
Most adapter log files are located in the $WSHOME/config directory and they are named WSTrace1.log.
Active Sync-enabled adapters that make log calls to the ActiveSyncUtil instance create a log file or set of log files in a directory specified by the Log File Path resource attribute. Be sure to check these log files for additional Active Sync-related log entries.
The information in this section is organized as follows:
Follow these general steps to debug your custom adapter.
Create a test program for your adapter, and be sure this Java file performs the following basic functions:
Create a new resource
Create a user
Get a user
Update a user
Delete a user
Perform create, get, update and delete operations on multiple users
A sample test file (SkeletonResourceTests.java) is provided in the /REF directory on your installation CD.
Set an appropriate logging level for debugging.
For example, for the first debugging pass, increase the logging level to 4 (maximum debugging output), set the log file path, and specify a maximum file size.
When you start the adapter, all of the resource settings are written to the log file. You can use this information to validate that the adapter started and that all setting changes were saved.
Compile and test your adapter.
To compile the test program, open a command window and enter the javac -d . test/filename.java command. This command creates the class file in the appropriate com/waveset/adapter/test directory.
To test your new adapter using this class file, be sure that your compiled adapter is in the com/waveset/adapter directory and use the following command to run the adapter:
java– D waveset.home=path com.waveset.adapter.test. MyResourceAdapter |
Create an HTML help file for your resource.
Example help files are supplied in the idm.jar file located in the com/waveset/msgcat/help/resources directory.
See Oracle Waveset 8.1.1 Deployment Reference for information about how to include online help with the application.
(For Active Sync-enabled adapters only) To reset synchronization on the last resource, delete the XmlData SYNC_resourceName object.
Read the error log and modify the adapter.
Reset the logging level.
For example, specifying Level 2 debugging yields information about the adapter settings and any errors, but limits the amount of log detail to a manageable level.
Before starting Waveset, you must identify the new adapter in the $WSHOME/config/Waveset.properties file by placing the adapter name under the resource.adapters entry or Waveset cannot recognize the adapter.
Install your adapter and its associated help file into Waveset.
Before Waveset can recognize an instance of a new adapter in the display, you must create a new resource of that type from the List Resource page.
From this page, select New -> new adapter and use the Resource Wizard to create the new adapter.
Use Waveset to create a resource and a user on that resource.
When troubleshooting an Active Sync-enabled adapter, if you edit the XmlData SYNC_resourceName object to remove the MapEntry for the Active Sync synchronization process from the Debug page, the adapter starts over from the first detected change.
If you used the IAPI event, you must set the Property() method to store synchronization state for the resource, such as a last change processed value. Setting this method is very useful for troubleshooting adapters. You can set the adapter to run and ignore past changes. Subsequently, you can modify the adapter and see your change results in the adapter log file.
If your resource is an Active Sync resource, you might see additional information if you enable logging on the resource edit page. Set the logging level (0-4) and the file path where the log file will be written (as resource_name.log).
(For Active Sync-enabled adapters only) Restart synchronization for the last resource.
To debug LoginConfig-related changes to your adapter, you must
Enable trace for the selected files and trace the following classes at Method/Class Level 1 trace:
com.waveset.security.authn.WSResourceLoginModule
com.waveset.session.LocalSession
com.waveset.session.SessionFactory
com.waveset.ui.LoginHelper
com.waveset.ui.web.common.ContinueLoginForm
com.waveset.ui.web.common.LoginForm
Test Single Sign-On (SSO) pass-through authentication login through Telnet as follows:
After correctly configuring the SSO login module, telnet directly to the HTTP port and send an HTTP request to login.jsp.
Paste the following request, which contains an SSO login module that looks for the sm_user HTTP header, into your telnet session:
HEAD /idm/login.jsp HTTP/1.0 Accept: text/plain,text/html,*/* Accept-Language: en-us Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 Host: LOCALHOST sm_user: Configurator |
A trace displays to indicate that your user has logged in correctly. For example:
2003.07.08 14:14:16.837 Thread-7 WSResourceLoginModule#checkForAuthenticatedResourceInfo() Found authenticated resource accountId, ’Configurator@Netegrity SiteMinder’on Identity Manager user ’Configurator’. null null 2003.07.08 14:14:16.837 Thread-7 WSResourceLoginModule#checkForAuthenticatedResourceInfo() Exit null null 2003.07.08 14:14:16.837 Thread-7 WSResourceLoginModule#login() Exit, return code = true null null 2003.07.08 14:14:16.847 Thread-7 LocalSession#login() Login succeeded via Netegrity SiteMinder null null 2003.07.08 14:14:16.847 Thread-7 LocalSession#login() Overall authentication succeeded null null 2003.07.08 14:14:16.897 Thread-7 LocalSession#checkIfUserDisabled() Entry null null 2003.07.08 14:14:16.897 Thread-7 LocalSession#checkIfUserDisabled() Exit null null 2003.07.08 14:14:16.927 Thread-7 LocalSession#login() Exit null null |
This section describes methods for debugging some common adapter connection problems.
The topics in this section are organized as follows:
Generally, you can identify adapter connection issues by tracing the adapter class com.waveset.adapter.adapter_classname. For example:
com.waveset.adapter.ADSIResourceAdapter
If necessary, review instructions for enabling trace in Tracing the Waveset Server.
Some common authentication problems include
Missing authentication properties
You must include a property name for the specified DataSource type in the set of names output in the trace.
An Waveset user is missing a matching resource account
If authentication succeeds for the resource adapter, but an exception occurs indicating that no Waveset user could be found with a matching resource account ID, be sure that the resource accountId associated with the user is the same as the accountId returned by your resource adapter’s authenticate method.
To verify the Waveset user’s resource accountId, review Waveset’s Edit Trace Configuration page (debug/Show_Trace.jsp). If a mismatch exists, change the content of the name being returned by your authenticate method or change your resource’s ID template. The template must generate a resource accountId that matches the accountId being returned by the authenticate method.
The most common problems with custom Active Sync adapters are form-related. These errors generally occur because you have not provided necessary information, such as password or email information, in a required field.
Waveset prints form validation errors to the adapter log after the final XML of the view. For example:
20030414 17:23:57.469: result from submit (blank means no errors): 20030414 17:23:57.509: Validation error: missing required field password |
Waveset also prints all messages to the adapter log. These messages include account creation and update times, adapter errors, and a summary of the schema map data.
Active Sync resource adapters store information about the last change processed in the SYNC.resourceName XMLData object.
Following are some common Domino gateway and adapter configuration errors and instructions for fixing these problems:
If an error message states the ’New Domino Gateway’ resource is not accessible and the connection is refused, try stopping and restarting the Oracle Waveset Gateway.
If an error message states no ID file name is specified and the path to the userID file is set incorrectly, specify a target location for the userID file and edit the resource adapter to set this attribute to a correct path. Typically, the target location for the userID file is the directory into which you installed the Gateway.
If an error message states you are not authorized to use the server, you have not set the correct access permissions for the ID file. Specify the correct permissions for this file and retry.
When RACF, ACF2, or TopSecret host adapters fail to reuse or cache connections, users are forced to log in frequently, which negatively impacts performance. Generally, the cache timeout setting causes this problem.
To check the cache timeout setting, trace Waveset’s adapter connection pool as follows:
From Waveset’s Edit Configuration Object page, trace the com.waveset.adapter.HostConnPool#reapConnections method at level 4.
If necessary, review instructions for enabling trace in Tracing the Waveset Server.
Capture trace for a sufficiently long period of time (at least 30-60 minutes), while the adapter performs operations.
Review the trace output in the application server stdout or trace file and look for Info reaping connection entries.
If this entry occurs more than once every 30 minutes, you have a good indication that connections are being timed out unnecessarily.
To resolve this problem, increase the Idle Timeout resource attribute value to prevent connections from being reaped too frequently. The Idle Timeout attribute controls how long a connection remains idle before the connection is logged out. The default value is 90 seconds, which causes new logins to occur frequently.
Ideally, specify a value that is greater than the average idle time for your deployment environment. For example, adjust the Idle Timeout attribute to 30 minutes (1800000 milliseconds) or more.
This section describes methods for troubleshooting the following PeopleSoft adapter problems:
Executing a “Test Connection” from the PeopleSoft resource adapter causes a failure with a generic exception.
Open the Waveset.properties file and set exception.trace=true.
Retry the test connection, and if you see the following results:
FormState: expansion complete java.lang.NullPointerException: PSProperties not loaded from file WavesetException: WavesetException: ==> com.waveset.util.WavesetException: FormState: derivation Log in to the PeopleSoft web interface to verify that you are using the correct UID and password. |
The PeopleSoft application server logs are not showing login attempts.
This problem generally occurs because you did not use the psjoa.jar file supplied with the PeopleSoft installation to which you are connecting. (See the Oracle Waveset 8.1.1 Resources Reference for more information about the PeopleSoft resource adapter).
If an error results when you try to test the connection from an SAP or SAP HR Active Sync adapter to the SAP system, open a command window and run this command from your installation directory WEB-INF/lib:
java -jar sapjco.jar
The sapjco.jar command shows which version of the SAP Java Connector (JCO) is installed and whether the adapter is installed correctly. The command also returns the Java Native Interface (JNI) platform-dependent and the RFC libraries that communicate with the SAP system.
If these platform-dependent libraries are not found, consult the SAP documentation to find out how to correctly install the SAP Java Connector.
This section contains information about debugging some common problems with UNIX adapters.
If you see timeout errors when provisioning to a UNIX resource adapter, you can determine where the provisioning process is failing by tracing the com.waveset.adapter.ScriptedConnection method at Method/Class level 4, which gives you the maximum logging output.
When becoming root to perform administrative commands, if the resource adapter executes a su root command instead of a su - root command, the environment does not inherit any of the custom environment variables defined for root; including any custom prompts (environment variable of PS1).
When configuring a UNIX adapter, you can determine which prompt to enter into the Root Shell Prompt field as follows:
Telnet or ssh to the system as the user you specified in the Login User field.
After typing the password and logging in, type su root without a dash and press return.
Type the root password.
The next prompt displayed is the prompt you must enter into the Root Shell Prompt field.
You can trace the following methods to troubleshoot issues with Identity Auditor:
com.sun.idm.auditor.policy to trace issues with Audit Scans.
com.sun.idm.auditor.accessreview to trace issues with Access Reviews.
com.sun.idm.auditor.report to trace issues with Audit Reports.
com.sun.idm.auditor.view to trace issues with Auditor Views.
In addition, you can set the following hidden flags by modifying Forms or TaskDefinitions:
Audit Policy Scan Task - maxThreads (int). Number of concurrent threads used. (Defaults to 5).
Audit Policy Scan Task - userLock (int). Time (in milliseconds) to wait to acquire a lock on the User object. (Defaults to 5000)
Audit Policy Scan Task - scanDelay (int). Time (in milliseconds) to delay between launching a new scan thread. (Defaults to 0, no delay)
Audit Policy Scan Task - clearuserLocks (boolean). If true, the scanner attempts to clear the user lock before scanning.
In addition, the Show Timings page (/debug/callTimer.jsp) provides the following information:
How long it takes to fetch the initial User view, with no resources, during the scan
How long it takes to refresh the User view, including resources, during the scan
How long it takes to evaluate the policy on the User view
How long each user scan takes, including User view fetch, policy evaluation, and so forth
How long it takes to fetch a list of users for access review
How long it takes to evaluate the attestation rule in access review
In the event of a ClassNotFound exception error, verify that the missing class is included in the server's classpath. If the classpath is configured properly, try configuring your application server such that the application class loader loads before the parent class loader. Sometimes loading the application classpath before the server classpath can resolve this issue. Consult your application server documentation for instructions.
This section describes some common form problems and how to fix these problems.
You added a <script> tag to your user form, but see the following exception when trying to access the user form:
java.lang.NoClassDefFoundError: org/mozilla/javascript/NativeScript
You must put the WEB-INF/lib/javascript.jar file into the application server’s classpath. For example:
If you installed Tomcat as a service (Windows only), edit the following registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tomcat\Parameters\JVM Option Number 0
Append the path to your jar file to the end of the JVM Option Number 0 key string value, wherever it is located. For example:
;C:\tomcat\lib\javascript.jar
If you just started Tomcat from the command line, move the jar file into the tomcat/lib directory.
You customized a Tabbed User Form, but when you try accessing the form through User Creation, you see the following exception:
com.waveset.util.WavesetException: Maximum form stack depth exceeded
You are limited to pushing 1000 hard-coded elements on the stack and you exceeded this limit.
If you added a check to detect Field containers that include a FieldRef or Field that matches the name of the Field container, you might have inadvertently created a circular reference on the form. To fix this problem, change the FieldRef or Field name.
You used a MultiSelect field in a form for LDAP users, assigned groups to a user in this field, edited the user again, and then see the same group on both sides of the MultiSelect. For example:
Left side value. cn=Group1,dc=test,dc=Com
Right side value. cn=Group1,dc=test,dc=com
In this example, the LDAP baseContext resource field is set to dc=test,dc=com and the LDAP groups are listed as dc=test,dc=Com, which causes a problem because LDAP is not case sensitive, but the MultiSelect widget is case-sensitive.
To alleviate this problem, change the baseContext on the LDAP resource to match the case in your LDAP resource dc=test,dc=Com or use the <upcase> XPRESS function to uppercase both the left and right side displays.
When troubleshooting the Oracle Waveset Gateway, it is often useful to run the Gateway from the command line. Using command line options allows you to input a wider range of start-up options, which includes starting the Gateway as a normal application instead of a service and running the Gateway on a different port.
You must kill the Waveset Gateway as a service before running it from the command line. For example, type
gateway.exe -k
The following table describes the Gateway command line arguments.
Argument |
Description |
---|---|
-i |
Install this program as an NT service, with specified startup |
-r |
Remove this program from the Service Manager |
-s |
Start the service |
-k |
Kill the service |
-t |
Set start-up for an existing service |
-d |
Debug, and run as a regular application |
-p |
Specify a TCP/IP port number (Default is 9278) |
-f |
Specify the path to the trace file |
-l |
Specify the level of tracing (Default is 0, no information) |
-m |
Specify the maximum trace file size in kilobytes |
-v |
Display the version |
Usage: gateway -i n -r -s -k -t n -d -p n -f name -l n -m n -v.
The -d and -s options are mutually exclusive.
See To Enable Tracing from the Command Line for more information about the Gateway tracing levels.
You can also use the Waveset Gateway Debug page (debug/Gateway.jsp ) to troubleshoot the Gateway. See How to Configure Gateway Tracing for more information.
If you have the basic Java programming skills required to work with Waveset, you should be able to diagnose and resolve most Java code problems.
However, a fairly common problem occurs where someone opens a connection to the database but does not close the connection properly. If you do not close the connection properly, performance issues result.
This section describes several methods you can use to investigate low memory conditions, including:
If you start experiencing slow response times when accessing users or when logging into accounts, you might have exceeded the admin cache size, which results in the admin cache being disabled.
The admin cache is a cache maintained on each server that contains all WSUser objects that are considered to be administrators. Waveset loads this cache when the server starts up, and automatically maintains this cache by watching for WSUser object changes.
Users are generally considered administrators when they have at least one capability and control an object group. If the number of administrator users exceeds the number specified for the cache, Waveset disables the cache, which causes poor system performance.
The default size of the admin cache is 10K. If you have more than 10K administrators, Waveset does not initialize the admin cache. You can adjust this limit by changing the admincache.rowlimit property in Waveset.properties. The cost of increasing admin cache size is the additional use of heap memory. The WSUser objects in the cache consume memory for the entire lifecycle of the server.
Waveset reads the admincache.rowlimit property at server startup, so any change to this property requires the server to be restarted.
You must set this property on each server because Waveset.properties is not shared across servers.
Waveset no longer requires approvers to be administrators, as in earlier versions of the product, which potentially reduces the number of administrators in the cache and improves system performance.
Out-of-memory conditions can be difficult to troubleshoot, but you can start by ensuring that sufficient memory was initialized at start up and by checking that Waveset is tracing memory usage levels and alerts.
If you are debugging problems with the admin cache, tracing com.waveset.server.InitAdminCacheThread at level 4 can show the following information:
What were the start-up timings
How long did it take to load the admin and auth caches
What data was loaded
What happened when you added or edited a user from the Waveset Administrator interface
You can use com.waveset.server.InitAdminCacheThread to trace start-up timings. For example,
20091124 12:02:56.203 InitAdminCacheThread(0x010dae16) InitAdminCacheThread#initAdminCache() Info Getting 9 admins from admin cache took 0m0.266s 20091124 12:02:56.437 InitAdminCacheThread(0x010dae16) InitAdminCacheThread#initAdminCache() Info Loading auth cache for 9 admins took 0m0.234s
Changing admincache.rowlimit in Waveset.properties only takes affect after you restart the server. Increasing the cache size to accommodate all of the administrators in the repository may increase server performance if there is enough memory to hold the cache without resulting in excessive JVM garbage collection.
Be aware that this solution does not scale.
Available JVM heap memory is a limitation to consider when using the admincache.rowlimit property to configure cache size, particularly when implementing and tuning an x64 system.
To test your configuration,
Set admincache.rowlimit=xxxxx on VM (where xxxxx denotes the cache size setting).
Trace com.waveset.server.InitAdminCacheThread at level 4.
Review the trace file output.
Verify that the cache was loaded.
Confirm that you can disable the admincache.rowlimit setting.
Compare the machine's memory usage before and after you disable the admincache.rowlimit setting.
Ensure that the operating system has sufficient memory available to run effectively. For example, check for excess paging, CPU activity, and so forth.
Check the JVM memory usage.
Verify how long it takes the cache to load and compare this time with that of a smaller cache, such as 1000.
You must have the Debug capability to access and execute operations from the Waveset Debug pages. Administrators and the Configurator are assigned this capability by default.
If you do not have the Debug capability, an error message results.
You can open the following Waveset Debug pages from the Administrator interface to monitor how much memory is being used by your system:
Host Connection Pool page (debug/Show_ConnectionPools.jsp). View a summary of connection pool statistics (if you are not using a data source), including the pool version, how many connections were created, how many are active, how many connections are in the pool, how many requests were serviced from the pool, and how many connections were destroyed.
You can also use the Host Connection Pool page to view a summary of the connection pools used to manage connections to the Gateway. You can use this information to investigate low-memory conditions.
List Cache Cleared (debug/Clear_XMLParser_Cache.jsp ). Clear the cache of recently used XML parsers.
Private collection pool (debug/Show_JDBC.jsp). View a summary of the cache of Java DataBase Connectivity (JDBC) connections being used by the repository and some resource adapters.
System Memory Summary page (debug/Show_Memory.jsp). View the used and total memory in the system. You must click the Garbage Collect button to get the most current used memory value.
System Memory Summary page (debug/Show_Memory2.jsp). View an updated Show_Memory.jsp page that allow you to clear all unused memory in the JVM so you can investigate heap usage.
User Session Pool Cleared (Clear_User_Cache.jsp). Clear the cached sessions for recently logged in users.
XML Resource Adapter Caches Flushed and Cleared (Clear_XMLResourceAdapter_Cache.jsp). Clear the cache of the test XML resource adapter.
Use the Java Monitoring and Management Console (JConsole) to detect low memory and deadlocks. JConsole is a Java Management Extension (JMX) technology-compliant graphical management tool that is co-packaged with JDK 5 (and later).
JConsole accesses the memory system, memory pools, and MBeans garbage collector to provide information about memory use such as memory consumption, memory pools, and garbage collection statistics. In addition, You can use JConsole to monitor MBeans for information about current heap memory use and non-heap memory use.
For information about using JConsole to monitor applications that run on the Java platform, see Using JConsole to Monitor Applications. This document is available from the following URL:
http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html
When you are trying to troubleshoot problems with PasswordSync, review the following logs for information:
PasswordSync Error Logs. PasswordSync writes all failures to the Windows Event Viewer. (For more information about Event Viewer, see Windows’ Help.) The source name for error log entries is PasswordSync.
PasswordSync Trace Logs. PasswordSync writes all trace logs to the file location specified when you configured tracing. See Using the PasswordSync Configuration Tool.
Some common PasswordSync problems and solutions include
PasswordSync is not propagating password changes from the Windows server to Waveset.
PasswordSync relies on the registry settings when creating a connection from Active Directory to the Waveset Server. PasswordSync reads the registry and processes the settings, but PasswordSync does not perform any checks to see if it can create a connection.
The following example shows a registry entry for a PasswordSync server. This example includes the default registry setting values, but does not show all of the settings used by PasswordSync.
[HKEY_LOCAL_MACHINE\SOFTWARE\Sun Microsystems\Sun Identity Manager\PasswordSync] "reinitIntervalMilli"=dword:1B77400 "securityIgnoreCertRevoke"=dword:00000000 "securityAllowInvalidCert"=dword:00000000 "directMode"=dword:00000001 "lhuser"="config" "lhcred"="rsVtQZpa5Ys=" "endpointURL"="http://10.10.10.10:8080/idm/servlet/PasswordSync" "installdir"="C:\\Program Files\\Sun Microsystems\\Sun Identity Manager PasswordSync" "tracelevel"=dword:00000000 "tracemaxKB"=dword:00002710 "tracefile"="C:\\Program Files\\Sun Microsystems\\Sun Identity Manager PasswordSync\\trace.log" |
If you have not enabled tracing at an appropriate level, PasswordSync does not log connection failures in much detail. To see more detailed trace information, edit the PasswordSync registry settings as described on Editing the Registry Keys. Specify tracelevel 4 to output the maximum trace information, and change the tracefile value to point to a writable file. For example:
"tracelevel"=dword:00000004 "tracefile"="C:\\Program Files\\Sun\\IdentityManager\\PasswordSync\\pwicsvc.log"
The registry settings will be reread based on the <i>reinitIntervalMilli</i> setting in the registry. After rereading the registry settings, PasswordSync automatically starts or stops tracing, depending on the trace parameters set in the registry. For each intercepted password change, PasswordSync logs the actions taken to push the password to Waveset.
If a connection fails during creation, you might encounter the following situations:
Each of these situations has its own error code and set of log entries. Waveset removes the date, time stamp, and process number from these entries to keep them short.
An incorrect or unreachable URL error that occurs when the server cannot be reached, is not running, or does not reply with a correct response.
Check that PasswordSync can access the server and page.
Be sure the server is running and that you have configured your firewalls and routers correctly.
Check the application server to be sure it is running, and that PasswordSync can connect to the endpointURL without the application path. If PasswordSync can does not return a page or an error, the application server is not running.
Check the servlet response by opening the endpointURL in a standard browser. If you do not see an error that starts with: com.waveset.util.WavesetException see if the servlet is compiling and available..
JMS usage for PasswordSync relies on the jms.jar being available in the classpath. The following exception message displays if you access the endpointURL without the correct file in place:
com.waveset.util.WavesetException: A JMS request arrived, but JMS PasswordSync is unavailable. Is JMS jar file available? |
An incorrect user name error generally occurs when the userID stored in the lhuser entry is incorrect. Use the Configure.exe utility to replace the user or replace the lhuser registry key value with a valid userID.
An incorrect password error generally occurs when the password stored in the lhcred entry is not correct when used in combination with the userID stored in lhuser. Use the Configure.exe utility to replace the password, but do not manually edit the lhcred registry key.
A garbage in the password entry error generally occurs when the registry key is corrupted and or when the registry key is manually edited, which causes garbage in the password entry.
This situation causes the process to hang in RAEncryptor::Decrypt3DES and PasswordSync cannot decrypt the entry. Use the Configure.exe utility to replace the password.
When you are trying to troubleshoot problems with a reconciliation task, review the Reconciliation Status Debug page (debug/Show_Reconciler.jsp) to see what the resource threads are working on.
Some common reconciliation problems include
Reconciliation does not start.
Open the System Threads Debug page (debug/Show_Threads.jsp) to see if the Reconcilerserver_name task is running.
If the task is not running, open the System Settings page and click Trace Scheduler.
Running the Scheduler restarts the Reconciler server_name task.
Check the System Threads page again, and if the Reconciler server_name task has not restarted, the Scheduler might be hung.
Restart your application server.
Reconciliation fails for a certain user object.
If the object exists in Waveset, try editing the object directly. If the Waveset account does not exist, load the one account from the resource.
Verify the reconciliation user has the proper access rights.
Try extracting the object to a file using the same code path as reconciliation.
Open the System Memory Summary Debug page (debug/Show_Memory.jsp or debug/Show_Memory2.jsp) and verify you have enough free memory.
Waveset’s lh commands are very useful when you are troubleshooting connection problems. These commands use Waveset’s web application installation, but remove the application server from the equation.
This section describes the following
This section describes how to use the lh commands; starting with using the more basic commands and progressing to using commands that exercise most of Waveset.
After becoming familiar with these debugging tools, you can develop your own variations for using these lh commands.
Use the lh setRepo -c -n command to perform the most basic connection test, which allows you to examine the current repository location without connecting. You can use this command to verify that parameters, such as URL and JDBC driver, are correct.
If the connection is successful, you can read the ServerRepository.xml bootstrap file.
If the connection fails, try to solve this failure first. A decryption error is the most common cause of this failure. For example, you might have a J2EE mismatch or classpath conflict.
Use the lh setRepo -c -v command to connect to and examine the current repository location. (The -v provides verbose output.) You can use this command to exercise almost all of the Repository code without requiring the Waveset server.
If the connection is successful, then you are successfully connected to the current repository location.
If the connection fails, try to solve this problem first. Solving your connection problem can be very helpful in resolving DNS, firewall, or remote connection privilege problems.
For more information, see Testing DataSource Connections.
Use the setRepo command throughout the debugging process, to specify a new repository location or to set the repository to the same location.
You can use this command to confirm that all of the necessary components, such as the JAR files, are in place. The setRepo command also lets you vary connection information, such as userid and password, to debug table ownership or privilege problems.
Use this command to actually start an Waveset Server using the JAR files in the WEB-INF/lib and the classes in WEB-INF/classes under WSHOME. The lh console command uses your Waveset installation environment and actually starts the Waveset application, but removes the application server from the equation.
If the connection is successful, the problem is specific to the application server environment or configuration.
If the connection fails, review the failure messages.
If the connection failure is the same as the application server failure, you have reproduced this failure with significantly fewer variables.
If the failure appears to be different from the application server failure, try fixing the Waveset connection problem first because there are fewer variables and more of these variables are under Waveset control.
If you are testing a DataSource connection, the lh setRepo -c command might fail.
This failure is especially likely if you configured Waveset to use the application server’s database connectivity service or the application server’s directory service. These services often work only in the environment that a running application server provides to a web application.
Initially, approach the DataSource configuration you want in a step-by-step manner. Once you are comfortable with these steps, you can adapt your approach to suit your needs.
Try using a direct JDBC DriverManager connection, such as a non-DataSource connection, that bypasses the application server’s database connectivity service.
Use a DataSource, but store the DataSource object in a directory service other than application server’s directory service.
If you have no other directory service available, you can download a free directory service, including the reference implementation of JNDI that uses only the local file system.
If these steps work, you have localized the problem to the application server.
Then, if useful, you can add the application server’s database connectivity service or the application server’s directory service, whichever service works outside of the environment that the application server provides to web applications.
This section describes how to diagnose and fix some common server-related problems.
You can analyze your application server logs for fatal errors and other server-related problems.
To troubleshoot server problems, use the application server’s Administrative Console to increase the logging level for each module. For more information, see the product documentation supplied with your server.
Most application servers have a standard location for standard out files (stdout) and standard error files (stderr) for the JVM running the application server. To analyze your application server logs, locate the logs directory or the log files specified for your Waveset application server.
Open the stdoutfile to view minor messages and other tracings.
Open the stderr file to view fatal and critical exceptions.
You will see Waveset start up and shut down the messages in this trace output.
Beginning with Sun Identity Manager Version 7.1, a sealing violation exception occurs in the application server log when you use Waveset with Oracle 10g on Sun Java System Application Server Enterprise Edition 8.2.
This exception generally occurs if you are using more than one Java Archive file (JAR file) containing Oracle JDBC drivers.
To prevent this problem, be sure the CLASSPATH contains only one JDBC driver JAR file, and that you use the ojdbc14_g.jar provided with Oracle 10g. In addition, you must use the ojdbc14_g.jar provided by the Oracle 10g installation to ensure correct operation.
Waveset depends on, and includes, some jar files that may also be provided by the application server, which can cause a conflict.
For example, an error can result when you deploy:
Sun Identity Manager 8.1 on jBoss
Sun Identity Manager 7.0 on Apache Tomcat 4.1
Tomcat ships with a commons-logging jar file in $TOMCAT_HOME/server/lib/commons-logging-1.1.jar and Identity Manager 7.0 ships with a commons-logging file in $TOMCAT_HOME/webapps/idm/WEB-INF/lib/commons-logging.jar.
The version of the jar file provided by the application server must be newer than the one provided by Waveset. To determine which jar file is newer, inspect the Specification-Version and Implementation-Version meta tags in the META-INF/MANIFEST.MF file.
If you deploy Waveset on certain application servers and errors result, you can remove the Waveset commons-*.jar files from the Waveset WEB-INF/lib directory to fix the problem.
If you are using the Oracle Waveset Service Provider End User Login page in WebSphere, and a javax.servlet.UnavailableException occurs with a 404 error displayed in your browser, you must reset some properties in the IBM 1.5 JDK.
Use the following steps:
In the was-install/java/jre/lib directory, rename the jaxb.properties.sample to jax.properties and uncomment these two lines:
javax.xml.parsers.SAXParserFactory= org.apache.xerces.jaxp.SAXParserFactoryImpl javax.xml.parsers.DocumentBuilderFactory= org.apache.xerces.jaxp.DocumentBuilderFactoryImpl |
Save the file and restart the application server.
To test an SPML configuration:
Open the Connect page and click Test.
A dialog indicating that the connection was successful pops up.
Open the Schema page and click Submit.
The system displays a hierarchical view of the schemas supported by the Waveset server.
If you cannot establish a successful connection
Double-check the URL you entered.
If the error you receive contains phrases such as no response or connection refused, then the problem is most likely the host or port used in the connection URL.
If the error suggests that a connection was made, but the web application or servlet could not be located, the problem is most likely in the WEB-INF/web.xml file.
If you encounter problems during the upgrade, check the upgrade log files located in the $WSHOME/patches/logs directory. The file names for the logs are based on a timestamp and the stage of the upgrade.
If, following an upgrade, Waveset fails to start with the following exception, your JDK/JRE may be the problem:
java.lang.IllegalStateException: Error attempting to decrypt: Given final block not properly padded |
Verify that you are using a JDK/JRE supplied by the same vendor that you were using previously. For example, you cannot upgrade to a Sun JDK if previously you were using a JDK from IBM. To fix this problem, uninstall the JDK/JRE and install the JDK or JRE from your previous vendor.