You can significantly improve the performance of Sun Identity Manager (Identity Manager) software across nearly all activities with the proper tuning. In addition to changing settings within the software, you can make performance improvements by tuning the application server, the JavaTM Virtual Machine (JVMTM machine), hardware, operating system, and network topology.
Additionally, you can use several tools to diagnose and monitor performance. Several of these tools exist within Identity Manager, such as trace and method timers. You can also use other Sun Microsystems and third-party tools to debug performance issues with Identity Manager.
This chapter describes tools, methodologies, and references you can use to improve performance and to debug performance-related issues.
The tuning process spans many entities and is specific to your deployment environment. This chapter describes different tuning methods for optimizing Identity Manager performance, but these methods are only intended as guidelines. You might have to adapt these methods for your deployment environment.
This chapter covers the following topics:
Review all of the information in this section before you start tuning Identity Manager.
The tuning methods described in this chapter are only provided as guidelines. You might have to modify some of these tunings for your deployment. In addition, Be sure to validate tunings before applying changes in a production environment.
Before you can tune Identity Manager, you must:
Be familiar with tuning application servers
Be familiar with Java 5.0 (required for Sun Identity Manager 8.1)
Understand performance limitations within your deployment environment
Be able to identify areas needing performance improvements
Understand the checklists provided in this chapter
In addition to the information provided in this chapter, consult the documents and web sites listed in this section for information related to tuning Identity Manager.
See the following documents for information related to performance tuning.
Table 4–1 Related Documentation
document Title |
Description |
---|---|
IBM Developer Kit and Runtime Environment, Java Technology Edition, Version 5.0 Diagnostics Guide |
Explains how to use AIX JVM to diagnose performance problems. |
Contains information, techniques, and pointers related to Java performance tuning. |
|
Oracle MetaLink: Note:114671.1: Gathering Statistics for the Cost Based Optimizer |
Explains how to use system statistics and Oracle®’s Cost-Based Optimizer (CBO). Note: This document is available to Oracle Metalink subscribers. Registration is required. |
Solaris Dynamic Tracing Guide |
Explains how to use DTrace to observe, debug, and tune your system's behavior. |
Sun JavaTM System Application Server Performance Tuning Guide |
Describes how to obtain optimal performance from your Sun Java System Application Server. Download the necessary version of this book from the Sun Microsystems documentation web site. |
Describes how to tune your garbage collection application by using JVM. |
|
Explains how to download and use the PrintGCStats script and how to collect statistics to derive optimal JVM tunings. |
|
Describes how Oracle’s Cost-Based Optimizer uses system statistics. |
|
Describes how to use JConsole to monitor applications that run on the Java platform. |
The following table describes some web sites that you might find useful when trying to tune Identity Manager performance.
Table 4–2 Useful Web Sites
Web Site URL |
Description |
---|---|
Sun web site containing diagnostic tools, forums, features and articles, security information, and patch contents. Note: The information on this site is divided into three areas:
|
|
Sun Developer Network (SDN) web site where you can browse forums and post questions. |
|
JRat web site that describes how to use the Java Runtime Analysis Toolkit, an open source performance profiler for the Java platform. |
|
Oracle’s internal forum site that contains information about tuning Oracle databases. Note: You must be an Oracle Metalink subscriber to access the information provided on this site. |
|
http://performance.netbeans.org/howto/jvmswitches/index.html |
NetBeansTM web site containing information about tuning JVM switches for performance. |
Identity Manager link on Sun’s Share Space. Note: You must sign up for a Share Space ID to access information provided on this site. |
|
Identity Manager FAQ on Sun’s Share Space. Note: You must sign up for a Share Space ID to access this FAQ. |
|
SLAMD Distributed Load Generation Engine web site. |
|
OpenSolaris Community: DTrace web page. |
|
Web sites containing information related to tuning the Solaris OS. |
How well your Identity Manager solution performs can depend on the following deployment-specific settings:
Resource configuration
How many resources are connecting
What type of resources are connecting
How attributes are mapped on the resources
Exact resource version
Network topology
Number (and distribution) of domain controllers
Number of installed Identity Manager Gateways
Number of concurrent settings
Number of concurrent processes (running workflows)
Number of concurrent users
Number of concurrent Identity Manager administrators
Total number of users under management
When you are trying to debug performance problems, start by analyzing and describing the problem. Ask yourself the following questions:
Where do you see the performance issue? In reconciliation, workflows, custom workflows, GUI page loading, provisioning, Access Review?
Are you CPU bound, memory bound, resource bound, or network bound?
Have you examined your configuration (hardware, network, parameters, and so forth) for problems?
Have you recently changed anything in your deployment environment?
Have you tried profiling troublesome resources natively to see if the problem is on the resource side and not with Identity Manager?
What size are the views?
Are you provisioning to several resources?
Are resources on slow networks connecting to Identity Manager?
Are you running additional applications on the server with Identity Manager?
Do your organizations have a Rule-Driven Members rule?
Have you run a series of thread dumps to see if there is a consistent pattern?
Looking at just a single thread dump can be misleading.
Have you recently turned on tracing?
Have you checked your JVM garbage collection?
Have you added organizations that are adding load to memory management?
This section provides information about tuning your deployment environment, including:
This section describes some tuning suggestions you can use to optimize your Java Platform, Enterprise Edition (Java EE platform) environment.
These tuning suggestions were derived from a series of experiments in which a considerable increase in throughputs was observed for the use cases tested. The increases were attributed to JVM sizing and to switches that affected garbage collector behavior.
For more information about tuning Java, JConsole, or JVM, visit the web sites noted in Table 4–1 and Table 4–2.
The following sections provide information about tuning Java and the JVM in your Java EE environment.
For information, best practices, and examples related to Java performance tuning, see the Java Tuning White Paper at:
http://java.sun.com/performance/reference/whitepapers/tuning.html
The following tuning scripts were used to derive the tuning suggestions noted in this section. These scripts were added to the domain.xml file (located in the domain configuration directory, which is typically domain-dir/config ) on a Sun Java System Application Server.
PrintGCStats – A data mining shell script that collects data from verbose:gc logs and displays information such as garbage collection pause times, parameter calculations, and timeline analyses over the application’s runtime by sampling the data at user-specified intervals.
For more information about how to use this script and garbage collection statistics to derive optimal JVM tunings, see the following web site:
http://java.sun.com/developer/technicalArticles/Programming/turbo/#PrintGCStats
PrintGCDetails – A shell script that can provide more verbose garbage collection statistics.
PrintGCTimeStamps – A shell script that adds time-stamp information to the garbage collection statistics collected by using the PrintGCDetails script.
To help ensure the best JVM performance, verify the following:
Be sure that you are using the required Java version noted in the “Supported Software and Environments” section of the Sun Identity Manager 8.1 Release Notes to ensure you are using the most current features, bug fixes, and performance enhancements.
Be sure that you are using a newer version of garbage collection.
Frequently, customers do not remove the older, default garbage collection scheme when installing an application server. Running Identity Manager with an older garbage collector creates many objects, which forces the JVM to constantly collect garbage.
If you deployed Identity Manager on Sun Java System Application Server, you can increase throughput by adding garbage collection elements to the deployed Identity Manager instance server.xml file.
If you expect a peak load of more than 300 users, try modifying the following settings to increase performance:
For HTTP listeners configured for the deployed Identity Manager instance, edit the listener definition element in the server.xml file and set the number of acceptor threads to the Number of active CPUs on the host divided by the Number of active HTTP listeners on the host.
For example:
<http-listener id=”http-listener-1” \address=”0.0.0.0” port=”80” \acceptor threads=”Calculated Acceptor Threads” ...>
Because the static content of most Identity Manager deployments is not projected to change frequently, you can edit the File Cache settings (on the File Cache Configuration page) for static content. Specify a high number (such as the number of seconds in 24 hours) for the maximum age of content within the file cache before the content is reloaded.
To access the File Cache Configuration page, click the File Caching tab on the web-based Administrative Console for the HTTP server node. (See the latest Sun Java System Web Server Administrator’s Guide for detailed instructions.)
Sun Java System Application Server exposes tunables that affect the size of various thread pools and connection queues that are maintained by the HTTP container.
By default, most of these tunables are set for a concurrent user load of 300 users or less.
The following guidelines are provided to help you tune your application server:
Other than heap size, you can use the default parameter settings for most application servers. You might want to modify the server’s heap size, depending on the release being used.
The “Tuning the Application Server” chapter, in the latest Sun Java System Application Server Performance Tuning Guide, contains information about tuning a Sun Java System Application Server. This document is available from the following URL at http://docs.sun.com/app/docs.
In addition, if you are using Sun Java System Application Server 8.2 Enterprise Edition, the following changes solve “concurrent mode failures,” and should give you better and more predictable performance:
If you are constantly running old generation collections, review your application’s heap footprint and consider increasing the size. For example:
500 Mbytes is considered a modest size, so increasing this value to 3 Gbytes might improve performance.
With a 2–Gbyte young generation collection, each scavenge promotes about 70 Mbytes. Consider giving this 70 Mbytes at least one more scavenge before promoting. For example, you might need the following SurvivorRatio:
2 GB/70 M X 2 = 4096/70 = 55
where:
-XX:SurvivorRatio=32 -XX:MaxTenuringThreshold=1
This ratio prevents premature promotion and the added problem of “nepotism,” which can degrade scavenge performance.
If you specified -XX:CMSInitiatingOccupancyFraction=60, and the CMS collections are still starting before they reach that threshold. For example:
56402.647: [GC [1 CMS-initial-mark: 265707K(1048576K)] 1729560K(3129600K), 3.4141523 secs]
Try removing -XX:CMSInitiatingOccupancy=60 (using the default value of 69percent), and add the following line:
-XX:UseCMSInitiatingOccupancyOnly
If your young generation collection is 2 Gbytes and the old generation collection is 1 Gbyte, this situation might also be causing premature CMS collections. Consider reversing this ratio. Use a 1–Gbyte young generation collection and a 2–Gbyte old generation collection, as follows
-Xms3G -Xmx3G -Xmn1G
Also, remove -XX:NewRatio. This ratio is redundant when you have explicitly specified young generation and overall heap sizes.
If you are using a 5uXX version of the Java Development Kit (JDKTM software), and have excessively long “abortable preclean” cycles, you can use -XX:CMSMaxAbortablePrecleanLoops=5 as a temporary workaround.
You might have to adjust this value further.
Add the following line to view more information about garbage collector performance.
-XX:+PrintHeapAtGC
Use this command with caution because it increases how much verbose garbage collection data is produced. Be sure that you have enough disk space on which to save the garbage collector output.
If you are using the Sun FireTM T2000 server, large-heap data Translation Look-aside Buffers (DTLBs) can become a scarce resource. Using large pages for the Java heap often helps performance. For example:
-XX:+UseLargePages
If you are tuning Identity Manager on an IBM WebSphere® application server, consider limiting how much memory is allocated for the heap because heap memory can affect the memory used by threads.
If many threads are created simultaneously and the heap size increases, the application’s space limit can be quickly impacted and the following error results:
JVMCI015:OutOfMemoryError
Identity Manager relies on the repository database to store and manage its identity and configuration data. For this reason, database performance can greatly influence Identity Manager’s performance.
Detailed information about performance tuning and managing databases is beyond the scope of this document because this information is dataset-specific and vendor-specific. In addition, customer database administrators (DBAs) should already be experts on their own databases.
This section characterizes the Identity Manager application and provides general information about the nature of Identity Manager data and its typical usage patterns to help you plan, tune, and manage your databases.
This information is organized into the following sections:
The Identity Manager repository contains three types of tables, and each table has slightly different usage characteristics. Information about these tables is organized into the following sections:
Attribute tables enable you to query for predefined single-valued or multi-valued object attributes.
For most object types, stored attributes are hard-coded.
The User and Role object types are exceptions to this rule. The inline attributes that are stored in the object table for User and Role are configurable, so you can configure additional custom attributes as queryable.
When you search for objects based on attribute conditions, Identity Manager accesses attribute tables in joins with the corresponding object tables. Some form of join (such as a JOIN, an EXISTS predicate, or a SUB-SELECT) occurs for each attribute condition.
The number of rows in the attribute table are proportional to the number of rows in the corresponding object table. The values distribution might exhibit skew, where multi-valued attributes have a row per value and some objects might have more attributes or more attribute values than others. Typically, there is a many-to-one relation between rows in the attribute table and rows in the object table.
Attribute tables have ATTR in the table name.
Identity Manager uses a change table to track changes made to a corresponding object table. These table sizes are proportional to the rate of object change, but the tables are not expected to grow without bound. Identity Manager automatically truncates change tables.
Change tables can be highly volatile because the lifetime of a row is relatively short and new rows can be created frequently.
Access to a change table is typically performed by a range scan on the time-stamp field.
Change tables have CHANGE in the table name.
The Identity Manager repository uses object tables to hold serialized data objects, such as Large Objects (LOBs). Object tables can also hold commonly queried, single-valued object attributes.
For most object types, stored attributes are hard-coded.
The User and Role object types are exceptions to this rule. The inline attributes that are stored in the object table are configurable, and you can configure additional custom attributes as queryable for User and Role.
The number of rows in an object table equals the number of objects being stored. The number of objects stored in each object table depends on which object types are being stored in the table. Some object types are numerous, while other types are few.
Generally, Identity Manager accesses an object table by object ID or name, though Identity Manager can also access the table by using one of the attributes stored in the table. Object IDs and names are unique across a single object type, but attribute values are not unique or evenly distributed. Some attributes have many values, while other attributes have relatively few values. In addition, several object types can expose the same attribute. An attribute may have many values for one object type and few values for another object type. The uneven distribution of values might cause an uneven distribution of index pages, which is a condition known as skew.
Object tables are tables that do not have ATTR or CHANGE suffixes in the table name.
Every object table contains an XML column, which is used to store each serialized object except the LOG table-set. Certain LOG table-set optional attributes are stored in the XML column if these attributes are present. For example, if digital signing is enabled.
You can roughly divide Identity Manager data into a number of classes that exhibit similar properties with respect to access patterns, cardinality, lifetime, volatility, and so forth. Each of the following classes corresponds to a set of tables in the repository:
User data consists of user objects.
You can expect this data to grow quite large because there is an object for each managed identity. After an initial population phase, you can expect a proportionally small number of creates because the majority of operations will be updates to existing objects.
User objects are generally long-lived and they are removed at a relatively low rate.
User data is stored in USEROBJ, USERATTR, and USERCHANGE tables.
Role data consists of Role objects, including Roles subtypes such as Business Roles, IT Roles, Applications, and Assets.
Role data is similar to organization data, and these objects are relatively static after a customer deploys Identity Manager.
An exception to the preceding statement is a deployment that is integrated with an external source containing an authoritative set of roles. One integration style might be to feed role changes into Identity Manager, which causes Identity Manager Role data to be more volatile.
Generally, the number of role objects is small when compared to the number of identity objects such as users (assuming that multiple users share each role), but this depends on how each enterprise defines its roles.
Role data is stored in ROLEOBJ, ROLEATTR, and ROLECHANGE tables.
Account data solely consists of account objects in the Account Index.
As with user data, account data can become rather large, with an object for each known resource account. Account objects are generally long-lived, removed at a relatively low rate, and after initial population, are created infrequently. Unless you frequently add or remove native accounts, or enable native change detection, account object modifications occur infrequently.
Identity Manager stores account data in ACCOUNT, ACCTATTR, and ACCTCHANGE tables.
Compliance Violation data contains violation records that indicate when the evaluation of an Audit Policy failed. These violation records exist until the same Audit Policy is evaluated against the same User and the policy passes. Violation records are created, modified, or deleted as part of an Audit Policy Scan or as part of an Access Review.
The number of violation records is proportional to the number of Audit Policies that are used in scans and the number of Users. An installation with 5000 users and 10 Audit Policies might have 500 violation records (5000 x 10 x 0.01), where the 0.01 multiplier depends on how strict the policies are and how user accounts are changed.
Identity Manager stores Compliance Violation records in OBJECT, ATTRIBUTE, and OBJCHANGE tables.
Entitlement data predominately consists of user entitlement objects, which are only created if you are doing compliance access reviews.
Entitlement records are created in large batches, modified slowly (days) after initial creation, and are then untouched. These records are deleted after an Access Review is deleted.
Identity Manager stores entitlement data in ENTITLE, ENTATTR, and ENTCHANGE tables.
Organization data consists of object group or organization objects.
Object group data is similar to configuration data, and this data is relatively static after being deployed. Generally, the number of objects is small (one for each defined organization) when compared to task objects or to identity objects such as users or accounts, however, the number can become large compared to other configuration objects.
Organization data is stored in ORG, ORGATTR, and ORGCHANGE tables.
Task data consists of objects that are related to tasks and workflows, including state and result data.
The data contained in these tables is short-lived compared to other classes because objects are created, modified, and deleted at a high rate. The volume of data in this table is proportional to the amount of activity on the system.
Task data is stored in TASK, TASKATTR, and TASKCHANGE tables.
Configuration data consists of objects related to Identity Manager system configuration, such as forms, roles, and rules.
Generally, configuration data is:
Relatively small compared to other classes
Only expected to change during deployment and upgrade, and changes occur in large batches
Not expected to change much after being deployed
Identity Manager stores configuration data in ATTRIBUTE, OBJCHANGE, and OBJECT tables.
If you enable Data Exporting, some records are queued inside Identity Manager until the export task writes those records to the Data Warehouse. The number of records that are queued is a function of Data Exporting configuration and the export interval for all queued types.
The following data types are queued by default, and all other data types are not:
ResourceAccount
WorkflowActivity
TaskInstace
WorkItem
The number of records in these tables grows until the export task drains the queue. The current table size is visible through a JMXTM Bean.
Records added to this table are never modified. These records are written during other Identity Manager activities, such as reconciliation, provisioning, and workflow execution. When the Data Exporter export task runs, the task drains the table.
Identity Manager stores Export Queue data records in QUEUE, QATTR, and QCHANGE tables.
Log data consists of audit and error log objects. Log data is write-once only, so you can create new audit and error log objects, but you cannot modify these objects.
Log data is long-lived and can potentially become very large because you can only purge log data by explicit request. Access to log data frequently relies on attributes that are stored in the object table instead of in the attribute table. Both the distribution of attribute values and queries against the log specifically depend on how you are using Identity Manager.
For example, the distribution of attribute values in the log tables depends on the following:
What kind of changes are made
Which Identity Manager interface was used to make the changes
Which types of objects were changed
The pattern of queries against the log table also depends on which Identity Manager reports, which custom reports, or which external data mining queries a customer runs against the log table.
Identity Manager stores audit log records in LOG and LOGATTR tables, and error log records in SYSLOG and SLOGATTR tables. This data does not have corresponding change tables.
Identity Manager generates globally unique identifiers (GUIDs) for objects by using the VMID class provided in the JDK software.
These GUID values exhibit a property that gets sorted by its string representations, based on the order in which the objects are created. For example, when you create new objects with Identity Manager, the newer objects have object IDs that are greater than the older objects. Consequently, when Identity Manager inserts new objects into the database, the index based on object IDs can encounter contention for the same block or blocks.
Generally, Identity Manager uses prepared statements for activities (such as inserting and updating database rows), but does not use prepared statements for queries.
If you are using Oracle, this behavior can create issues with the library cache. In particular, the large number of statements versions can cause contention on the library cache latch.
To address this contention, change the Oracle CURSOR_SHARING parameter value from EXACT to SIMILAR. Changing this value causes Oracle to replace literals in SQL statements with bind variables, thereby reducing the number of versions.
Because Identity Manager is a Java application that generally reads and writes character data rather than bytes, it does not restrict which encoding the database uses.
Identity Manager only requires that the data is sent and returned correctly. For example, the data does not become corrupted when written or reread. Use an encoding that supports multi-byte characters and is appropriate for the customer’s data. Generally, UTF-8 encoding is sufficient, but enterprises with a large number of true multi-byte characters, such as Asian or Arabic, might prefer UTF-16.
Most database administrators prefer to use an encoding that supports multi-byte characters because of the following:
Their deployments often grow to support international characters.
Their end users cut-and-paste from a Microsoft application’s text containing characters that look like ASCII but are actually multi-byte, such as em dashes (—).
This section describes some general guidelines for tuning a repository database:
DBAs must frequently run statistics to monitor what is happening with the repository database.
If you are using a data source, set the connectionPoolDisable attribute to true in the RepositoryConfiguration object to disable automatic internal connection pooling in the Identity Manager repository.
For example, setting <RepositoryConfiguration connectionPoolDisable=’true’> allows you to avoid having two connection pools (one for Identity Manager and one for your application server).
You can edit the RepostioryConfiguration object to enhance the performance of searches against specific, single-valued attributes. For example, you might edit this object to add an extended attribute, such as employeeID, that is used to search for Users or as a correlation key.
The default RepositoryConfiguration object looks like the following example:
<RepositoryConfiguration ... > <TypeDataStore Type="User" ... attr1="MemberObjectGroups", attr2="lastname" attr3="firstname" attr4="" attr5=""> </TypeDataStore> </RepositoryConfiguration> |
The ellipses represent XML attributes that are not relevant here.
Each of the attr1, attr2, attr3, attr4, and attr5 XML attributes specifies a single-valued attribute to be copied into the waveset.userobj table. The waveset.userobj table can contain up to five inline attributes. The attribute value named by attr1 in RepositoryConfiguration will be copied into the “attr1” database column in this table.
Inline attributes are stored in the base object table for a Type (rather than as rows in the child attribute table).
Using inline attributes improves the performance of repository queries against those attributes. (Because inline attributes reside in the main “object” table, queries against inline attributes are faster than those against non-inline attributes, which are stored in the child “attribute” table. A query condition against a non-inline attribute requires a “join” to the attribute table.)
By default, Identity Manager uses the MemberObjectGroups, lastname, and firstname inline attributes.
You can add two more attributes to enable faster searching, as long as those attributes are queryable.
For example, if your deployment contains an employeeID extended attribute, adding that attribute inline will improve the performance of repository searches against that attribute.
If you do not need lastname or firstname, you can remove them or replace them with other attributes.
Do not remove MemberObjectGroups. Identity Manager uses this attribute internally to speed up authorization checks.
For more information about which object types are stored in each set of tables, see Data Classes.
This section describes some vendor-specific guidelines for tuning Oracle and SQL Server repository databases.
Currently, MySQLTM databases are only supported in development and for demonstrations.
This section describes guidelines for tuning Oracle repository databases:
The Identity Manager application does not require Oracle database features or options.
If you are using an Oracle repository database and Identity ManagerService Provider or Identity Manager, you might encounter problems with object table fragmentation because Identity Manager uses LONG, rather than LOB, data types by default. Using LONG data types can result in large amounts of “unallocated” extent space, which cannot be made into usable space.
To mitigate this problem, do the following:
Take EXPORT dumps of the Object table and re-import them to free up unallocated extent space. After importing, you must stop and restart the database.
Use LOB data types and DataDirect Technologies’ Merant drivers, which provide a standard LOB implementation for Oracle.
Use Locally Managed Tablespaces (LMTs), which offer automatic free space management. LMTs are available in Oracle 8.1.5.
Identity Manager does not require Oracle init.ora parameter settings for SGA sizing, buffer sizing, open cursors, processes, and so forth.
While the Identity Manager repository is a general-purpose database, it is best described as an object database.
Of the Identity Manager tables, the TASK table-set comes closest to having transaction-processing characteristics. The LOG and SYSLOG table-sets are also exceptional because these tables do not store serialized objects.
See Repository Table Types and Data Classes for descriptions of the tables, the object types stored in each table, and the general access pattern for each table.
If you have performance issues with the Oracle database, check for issues related to poor query plans being chosen for what Identity Manager expects to be relatively efficient queries.
For example, Identity Manager is configured to perform a full table-scan when an index is available for use. These issues are often visible in Automated Workload Repository (AWR) reports provided in the SQL by the buffer gets table. You can also view issues in the Enterprise Manager tool.
Performance problems typically appear to be the result of bad or missing database table statistics. Addressing this problem improves performance for both the database and Identity Manager.
The following articles (available from Oracle) are a good source of information about the cost-based optimizer (CBO) in Oracle:
Oracle MetaLink: Note:114671.1: Gathering Statistics for the Cost Based Optimizer
You might also investigate using SQL Profiles, which are another method for choosing the best query plans. You can use the SQL Advisor within Enterprise Manager to create these profiles when you identify poorly performing SQL.
If you detect unexpected growth in the Oracle redo log, you might have workflows that are caught in an infinite loop with a manual action. The loop causes constant updates to the repository, which in turn causes the size of each TaskInstance to grow substantially. The workflow errors are caused by improper handling of WF_ACTION_TIMEOUT and by users closing their browser in the middle of a workflow.
To prevent problematic workflows, preview each manual action before a production launch and verify the following:
Have you set a timeout?
Have you created appropriate transition logic to handle a timeout for the activity with the manual action?
Is the manual action using the exposed variables tag when there is a large amount of data in the TaskInstance?
Frequently, you can significantly improve Identity Manager performance if you change the CURSOR_SHARING parameter value from EXACT to SIMILAR.
Identity Manager uses prepared statements for some activities (such as inserting and updating database rows), but does not use these statements for most queries.
When you use Oracle, this behavior can cause issues with the library cache. In particular, the large number of statement versions can create contention on the library cache latch. Changing CURSOR_SHARING to SIMILAR causes Oracle to replace literals in SQL statements with bind variables, which greatly reduces the number of versions.
See Prepared Statements for more information.
Some customers who used an SQL Server 2000 database as a repository reported that as concurrency increased, SQL Server 2000 reported deadlocking problems that were related to SQL Server’s internal use of pessimistic locking (primarily lock escalation).
These deadlock errors display in the following format:
com.waveset.util.IOException: ==> com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 51) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction. |
To prevent or address deadlocking problems, do the following:
Use the SQL Server 2005 database.
Configure the READ_COMMITTED_SNAPSHOT parameter by formatting the command as follows:
ALTER DATABASE waveset SET READ_COMMITTED_SNAPSHOT ON
Enabling the READ_COMMITTED_SNAPSHOT parameter does the following:
Removes contention during the execution of SELECT statements that can cause blocks, which greatly reduces the potential for deadlocks internal to SQL Server.
Prevents uncommitted data from being read and guarantees that SELECT statements receive a consistent view of committed data.
For more information about the READ_COMMITTED_SNAPSHOT parameter, see: http://msdn2.microsoft.com/en-us/library/ms188277.aspx.
Suggestions for optimizing Identity Manager’s performance are organized into the following areas:
In general, you can optimize Identity Manager performance if you do the following:
Turn off tracing (such as Java class, userform, and workflow tracing). Tracing can add substantial overhead.
Run the Identity Manager built-in Audit Log Maintenance Task and System Log Maintenance Task to configure log record expirations. Log records can grow without bound, so use these tasks to prevent the repository database from running out of space. For information, see the Sun Identity Manager 8.1 Business Administrator’s Guide.
Check the README file in Identity Manager updates (formerly called service packs or installation packs) to see if any performance improvements have been made to the product. If so, schedule an upgrade.
Consider the performance impact when fetching data from one or more remote systems, including the Identity Manager repository.
Increase the number of application server instances running Identity Manager, either on the same server or by adding servers, and use a load-balancing tool to distribute the requests between instances.
Keep the size of files referenced in a binary attribute as small as possible. Loading extremely large graphics files, for example, can decrease Identity Manager performance.
Write robust and readable XML code that minimizes duplication (for example, refactored), that uses memory efficiently, and that mitigates the impact to overall system performance.
Configure Identity Manager system monitoring to track events in real time.
You can view these events in dashboard graphs to quickly assess system resources, spot abnormalities, understand historical performance trends (based on the time of day, the day of week, and so forth), and interactively isolate problems before looking at audit logs. Dashboards do not provide as much detail as audit logs, but can provide hints about where to look for problems in the logs.
For more information about dashboards, see Chapter 8, Reporting, in Sun Identity Manager 8.1 Business Administrator’s Guide.
Because synchronization is a background task, how you configure an Active Sync adapter can affect server performance.
Use the Resources list to manage Active Sync adapters. Choose an Active Sync adapter and access start, stop, and status refresh control actions from the Synchronization section of the Resource Actions list.
To improve Active Sync adapter performance, do the following:
Evaluate and adjust polling intervals based on the type of activity being performed.
The polling interval determines when the Active Sync adapter will start processing new information. For example, if the adapter reads in a large list of users from a database and updates these users in Identity Manager each time, you could run this process in the early morning every day. Some adapters have a quick search for new items to process and can be set to run every minute.
Edit the synchronization file for the resource to specify the host where the adapters will run.
You can configure Active Sync adapters that require more memory and CPU cycles to run on dedicated servers to help load balance the systems.
If you have the appropriate administrator capability, you can change Active Sync resources to disable, manually start, or automatically start Active Sync adapters.
When you set an adapter to automatic, the adapter restarts when the application server starts. When you start an adapter, it runs immediately and executes at the specified polling interval. When you stop an adapter, it stops the next time the adapter checks for the stop flag.
Adjust the level of detail captured by the synchronization logs.
Synchronization logs capture information about the resource that is currently processing. Each resource has its own log file, path, and log level. The amount of detail captured by the adapter log depends on the specified logging level. You specify these values in the Logging section of the Synchronization Policy for the appropriate user type (Identity Manager or Service Provider).
To improve performance during bulk load operations, do the following:
Simplify default workflows to improve processing time (especially for bulk processing actions such as Active Sync, bulk actions, and reconciliation) by removing the callout to the Approval subprocess.
Keep user forms that are assigned to administrators as simple as possible. For example:
When creating a form for data loading, remove any code that is designed to display data.
When using bulk add actions, be sure that your CSV file defines basic attributes such as firstname and lastname. You can then remove these attributes from the administrator user form.
Do not modify the default forms provided with Identity Manager. Instead, make a copy of the form, give the copy a unique name, and modify the renamed copy. This approach prevents your customized forms from being overwritten during upgrades and product updates.
See Chapter 3, Identity Manager Forms, in Sun Identity Manager Deployment Reference for more information about creating and editing forms.
Implement the following features in deployment environments where you have NIS (Network Information Service) implemented:
Add an account attribute named user_make_nis to the schema map and use this attribute in your reconciliation or other bulk provisioning workflow. Specifying this attribute causes the system to bypass the step of connecting to the NIS database after each user update on the resource.
To write the changes to the NIS database after provisioning has completed, create a ResourceAction named NIS_password_make in the workflow.
Configurable XML objects offer a broad spectrum of user interface specifications that enable you to define how data is presented to users for different tasks and to automate complex business processes. However, this same flexibility can affect efficiency, performance, and reliability.
This section describes some guidelines for tuning Identity Manager’s configurable XML objects, which consist of forms, rules, and workflows. The information is organized into the following sections:
You can use Identity Manager forms to define interfaces to interact with views or variable contexts in an executing task. Forms also provide an execution context for business and transformation logic on a set of data elements. Although you can create very powerful, dynamic forms that perform a variety of tasks, reducing the complexity of forms increases efficiency.
The following sections describe some methods for improving the performance of your customized forms:
When designing new Identity Manager forms, system integrators can optimize a form’s performance by doing the following:
Performing “expensive” queries only one time, wherever possible. To minimize these queries,
Use <Field> <Default> elements to execute and store query results.
Use field names to reference values in later fields.
For custom tasks
Calculate the value in the task before a ManualAction, then store that value in a task variable.
Use variables.tmpVar to reference variables in the form.
Use <setvar name=’tempVar’/> to clear the variable after a ManualAction.
Using <defvar> for calculations that are performed for the initial display and with each refresh.
To improve the performance of administrator forms, do the following:
Specify TargetResources that only fetch specific resources for editing. (See Tuning Workflows for more information.)
Use cacheList and cacheTimeout caching parameters for objects that change infrequently if you are working with FormUtil.getResourceObjects or FormUtil.listResourceObjects.
Store the results of time-consuming calculations and fetches in <Field> elements and evaluate in the <Default> expression to help ensure that an operation occurs only one time.
Use update.constraints to limit which resources are fetched at runtime (see Dynamic Tabbed User Form in Sun Identity Manager Deployment Reference).
Use background approval (ManualAction with different owners and one-second timeouts) for faster page submissions.
Be aware that Identity Manager refreshes all fields defined on all panels of a Tab Panel Form when the page reloads, regardless of which panel is selected.
To improve the performance of end-user forms, do the following:
Use TargetResources to limit view checkouts to just those resource accounts of interest, which reduces fetch time for view and the memory consumed by TaskInstance and WorkItems.
Consider using Session.getObject(Type, name) to return a WSUser if just the view properties and attributes of the Identity Manager user object are of interest (useful for managing multiple deferred task triggers).
Be aware that end-user tasks typically have more WorkItems than Provisioning tasks, so end user tasks are especially susceptible to WorkItem size.
Consider using temporary generic objects for “view” editing that is constructed on view check-out then merged back into a full view for check-in.
Consider using scalable forms instead of the default Create and Edit User interfaces.
When you use the default User forms to edit a user, Identity Manager fetches the resources owned by that user the moment you start editing the user’s account. In deployment environments where users have accounts on many resources, this potentially time-intensive operation can result in performance degradation.
Some activities performed in forms call resources that are external to Identity Manager. Accessing these resources can affect Identity Manager performance, especially if the results contain long lists of values, such as compiling a list of groups or email distribution lists.
To improve performance during these calls, follow the guidelines in “Using a Java Class to Obtain Field Data” in Sun Identity Manager Deployment Reference.
Also, avoid using JavaScriptTM in performance-critical expressions such as <Disable> expressions. Short XPRESS expressions are easier to debug if you use the built-in tracing facilities. Use JavaScript for complex logic in workflow actions.
If a form is slow to display, you can use the debug/Show_Timings.jsp page to determine the problem. Look for calls to Formconvert.convertField(). This method shows how long each field took to compute its value.
You use Identity Manager rules to encapsulate constants and XPRESS logic that can be reused in forms, workflows, and other configurable components in the product.
When writing rules, use the following guidelines (as applicable) to obtain optimal performance:
Use static declarations to return a constant value.
Use defvar methods to implement algorithms with temporary values for incremented values or for values that are referenced only one time.
Use putmap, setlist, or setvar methods for complex or expensive calculations whose value must be returned multiple times. Be sure to eventually set the value to <null>.
You customize Identity Manager workflows to facilitate and automate complex business processes with various human and electronic touchpoints.
You can use the following methods to improve custom workflow performance:
Simplify default workflows to improve processing time (especially for bulk processing actions such as Active Sync, bulk actions, and reconciliation) by removing the callout to the Approval subprocess.
Ensure that no infinite loops exist in your workflows. In particular, be sure that break flags are updated and properly checked in the loops that exist in approval subprocesses.
Put fetched objects into a variable for use later if you must contact the repository for the same object multiple times.
Using a variable is necessary because Identity Manager does not cache all objects.
Specify TargetResources options in WorkflowServers checkoutView to restrict the number of resources that are queried for account information.
The following example shows how to restrict the number of resources being queried for account information.
<Argument name=’TargetResources’> <list> <string>resource name[| #]</string> </list> </Argument> |
In the preceding example, [| #] is an optional parameter that you can use when more than one account exists on a particular resource. In most cases, the resource name is sufficient.
Clear unnecessary view variables left by forms, especially large maps and lists. For example:
<setvar name=’myLargeList’></null></setvar>
The view is copied multiple times in a TaskInstance object, so large views greatly increase the size of each TaskInstance and corresponding TaskResult.
Use resultLimit (in seconds) in the TaskDefinition, or set this option during task execution to quickly dispose of completed tasks. Large numbers of TaskInstances impact the following:
How taskResults.jsp in footers and some JSPTM tasks in the Administrator interface are displayed
How JSP tasks are displayed
Querying each TaskInstance for task renaming
Database size
Set the following options as needed:
(Preferred selection) delete — Causes an older TaskInstance of the same name to be deleted before the new task begins execution.
wait — Suspends the current TaskInstance until the older TaskInstance is deleted or expired due to reaching its resultLimit.
rename — Inserts a time stamp into the TaskInstance name to avoid naming collisions.
terminate — Deletes an older TaskInstance of the same name. Each currently executing TaskInstance of the same name is terminated.
The number and size of WorkItems (indicated by ManualActions in a workflow) can affect memory and system performance significantly. By default, Identity Manager copies an entire workflow context into a WorkItem, then writes the workflow context back out after submission.
To improve performance for WorkItems and ManualActions do the following:
Reduce the size of WorkItems.
By default, ManualAction creates a WorkItem, then copies each variable in the task context into WorkItem.variables. Limiting task context variables prevents overwrites from parallel approvals.
Use ExposedVariables to limit which variables are copied back into WorkItem. For example:
<ExposedVariables><List><String>user ...
Use EditableVariables to limit the variables assimilated back into the executing task from WorkItem. For example:
<EditableVariables><List><String>user ...
Remember to include an approval flag, a form button value, and the actual approver’s name.
Change the confirmation page and background processing to improve user interface response time.
Create a confirmation ManualAction or background ManualAction, owned by another user such as Configurator.
Set timeout=’-5’ (5 seconds) and ignoreTimeout=’true’ to prevent an error message if a user submits an action after the task is executed and the WorkItems are deleted.
Optimize memory use by setting large attribute values, such as value maps and lists, to null on submission or instantiate them as Activity-scoped variables that quickly pass out of scope.
Shorten the lifetime of finished tasks.
Prevent dead-end tasks by ensuring that each WorkItem specifies a Timeout and that the workflow anticipates a Timeout for each WorkItem.
Consider using the resultLimit and resultOption options in the TaskDefinition to determine how the Scheduler handles a task after the task completes.
Use resultLimit to control how many seconds a task is allowed to live after the task has completed. The default is zero (0), which means that the task instance will be deleted immediately after task completion.
Use resultOption to specify what action to take when repeated instances of a task are started (such as wait, delete, rename, or terminate). The default is delete.
If you want to immediately delete tasks that complete successfully, but you also want to keep tasks containing errors long enough to debug, you can conditionally delete finished tasks.
Set a resultLimit in the TaskDefinition to a sufficient time period to debug issues. You can set resultLimit to zero (or a small value) if no errors are reported at runtime (such as WF_ACTION_ERROR is <null/>) after a WorkflowServices call.
Evaluate and fix poorly scoped variables. Scope variables according to where they are declared, as follows:
Global variables are values that must be used across many activities (such as the case owner, view) and as approval flags in subprocesses.
If a variable is declared as an element of <TaskDefinition>, scope the variable globally. If a variable is declared external, its value is resolved up the call stack.
Activity variables of expensive values (such as those variables that require a resource fetch or that store a large list or map of values) can be referenced in a WorkItem.
If a variable is declared as an element of <Activity>, ensure that the variable is visible to actions and transition elements in Activity.
Beginning with Identity Manager Version 2005Q3M1 SP1, use <Activity> variable values in Forms, rather than in workflows, to avoid copying values on WorkItem creates.
Activity variables are values used in transition logic.
If a variable is declared as an element of <Action>, the variable should pass out of scope on completion of the action. Action variables are typically used in WorkflowApplication invocations to “change” the names of variables set by the application (such as View -> User).
Do not specify synchronous execution (syncExec='true') for the last page in a wizard workflow.
If set to true, the task will run to completion when the user executes the task. The interface will hang until the task completes or encounters another stopping point (such as another ManualAction).
Remove unnecessary approval checks.
For Active Sync, use a streamlined provisioning task in place of the system-specified provisioning task specified by viewOptions.Process.
Do not modify the provisioning tasks provided with Identity Manager.
You must create a new task, then identify that task in the form and in the process mappings configuration (unless the task is not listed).
As a database administrator, you should frequently run statistics to monitor your repository database.
Performance problems are often caused by bad or missing database table statistics. Fixing this problem improves performance for both the database and Identity Manager performance.
See the following Oracle articles for more information:
Also consider using SQL Profiles, which is another method for choosing the best query plans. You can use the SQL Advisor within Enterprise Manager to create these profiles when you identify poorly performing SQL.
Data Exporter enables you to export new, changed, or deleted Identity Manager data to an external repository that is suitable for reporting or analytic work. The actual exporting of data is done in batches, where each type of data to be exported is able to specify its own export cycle. The data to be exported comes from the Identity Manager repository and, depending on the length of the export cycle and the amount of changed data, the volume of exported data can be large.
Some Identity Manager data types are queued into a special table for later export. Specifically, WorkflowActivity and ResourceAccount data is queued because this data is not persisted otherwise. Any persisted data type can also be queued if the warehouse needs to see all changes to the type, or if the type has a lifecycle that does not correspond to the export cycle, such as TaskInstance and WorkItem data.
To maximize performance, only queue and export the types of data that you require in the warehouse. Data exporting is disabled by default, but if you enable data exporting, it exports all data types. Turn off any data types that you do not need.
When the export task exports data, the task attempts to complete the export as quickly as possible, using multiple threads to achieve as much throughput as possible. Depending on the I/O speed of the Identity Manager repository and the warehouse, the export task can fully utilize the processors on the Identity Manager server, which causes any interactive performance to degrade. Ideally, the export should occur on a machine dedicated to that task or at least occur during periods when there is no interactive activity on the machine.
The export task supports the following tuning parameters:
Queue read block size
Queue write block size
Queue drain thread count
The drain thread count is the most important throughput. If a large number of records are in the queue table, increasing the number of threads (up to 24) tends to increase throughput. However, if the queue is dominated by one type of record, fewer drain threads might actually be faster. The export task attempts to divide the queue table contents into as many sets as there are threads allocated, and to give each thread a set to drain. Note that these threads are in addition to the drain threads that are draining the other repository tables.
You can usually optimize the general XML by using static XMLObject declarations wherever possible. For example, use:
<List> instead of <list>
<String> instead of <s>
<Map><MapEntry ...></Map> instead of <map>
Also, depending on the context, you might have to wrap objects instead of using the <o></o> element.
You can use Identity Manager dashboard graphs to quickly assess the current system, spot abnormalities, and understand historical trends (such as concurrent users or resource operations over a time period) for SunTM Identity Manager Service Provider(Service Provider).
Service Provider does not have an Administrator interface. You use the Identity Manager Administrator interface to perform almost all administrative tasks (such as viewing dashboard graphs).
For more information about tuning Service Provider see Sun Identity Manager Service Provider 8.1 Deployment.
When you are working with the Identity Manager Web Interface, you can optimize performance by using the OpenSPML toolkit that is co-packaged with Identity Manager.
Using the openspml.jar file from the http://openspml.org/ web site might cause memory leaks.
To improve performance during a large, initial user load, follow this procedure:
Disable all Audit Events from the Identity Manager Administrator interface.
Audit Logging can add several records per operation, making future audit reports perform more slowly.
Choose Configure -> Audit.
On the Audit Configuration page, deselect the Enable auditing box and click Save.
Disable the list cache by shutting down the web server or by changing the ChangeNotifier.updateSearchIntervalCount property (on the debug/Show_WSProp.jsp debug page) to 0..
The list cache keeps lists of users in frequently accessed organizations in memory. To maintain these lists, the list cache searches for and checks all newly created users.
Clear the current list cache on the debug/Clear_List_Cache.jsp page.
Ensure that the workflow being used to process the users does not contain approvals.
Use alternative load methods, which include:
Splitting the load and running the data in zones
Using bulk loads, which are much faster
Loading from a file
Disable Data Exporter for the WorkflowActivity type.
You must determine your memory needs and set values in your application server’s JVM by adding maximum and minimum heap size to the Java command line. For example:
java -Xmx512M -Xms512M
To improve performance do the following:
Set the maximum and minimum heap size values to the same size.
Depending on your specific implementation, you might want to increase these values if you run reconciliation.
For performance tuning purposes, you may also set the following in the waveset.property file:
max.post.memory.size value
The max.post.memory.size specifies the maximum number of bytes that a posted file (for example by using an HTML FileSelect control) might contain without being spooled to the disk. For cases where you do not have permission to write to temporary files, increase the max.post.memory.size to avoid having to spool to the disk. The default value is 8 Kbytes.
For additional information about system requirements, see the Sun Identity Manager 8.1 Release Notes.
For information about tuning SolarisTM and Linux operating system kernels, see the “Tuning the Operating System” chapter in the Sun Java System Application Server Enterprise Edition Performance Tuning Guide.
For information about tuning Oracle operating system kernels, see the product documentation provided with your Oracle system.
Network latency tends to be a common cause for performance issues when dealing with view provisioning. Tracing individual resource adapters can help you determine what is causing performance problems.
To improve provisioner performance, do the following:
Set provisioner.maxThreads in the Waveset.properties file to control the number of simultaneous account provisioning threads where a thread is started for each resource operation.
Generally, you can achieve optimal performance by setting this value to 10. Specifying a value greater than 20 significantly degrades the provisioner’s performance.
Configure quota settings in the Waveset.properties file to control the number of concurrent operations (such as reprovisioning) a user can execute for a specific task. Increasing the number of concurrent actions can help more operations complete faster, but trying to process too many actions at once might cause bottlenecks.
You can create configuration sets on a per-pool basis. For example, if you create configuration A, configuration B, and configuration C, when you create a TaskDefinition (workflow), you can assign a specific pool configuration to the workflow from the configurations that you defined.
The following example shows the quota settings that limit user bob to running one reprovisioning task at a time:
Quota.poolNames=ReProvision,Provision Quota.pool.ReProvision.defaultLimit=1 Quota.pool.ReProvision.unlimitedItems=Configurator Quota.pool.ReProvision.items=bob,jan,ted Quota.pool.ReProvision.item.bob.limit=1 |
To enforce the task quota, reference poolName in a TaskDefinition. The format is as follows:
<TaskDefinition ... quotaName=’{poolName}’..>
Most users start only one task at a time. For proxy administrators who perform reconciliation or Active Sync tasks, set the task quota higher.
Avoid using the Configurator user for reconciliation and Active Sync tasks. The Configurator has access to unlimited tasks and can monopolize available resources, which adversely affects concurrent processes.
The Reconciler is the Identity Manager component that performs reconciliation. This section suggests methods for improving Reconciler performance, including:
In general, you can improve Reconciler performance if you do the following:
Avoid using the Configurator user for reconciliation tasks. The Configurator has access to unlimited tasks and can monopolize available resources, which adversely affects concurrent processes.
Instead, use a streamlined, minimal user for reconciliation and Active Sync tasks. Because the subject executing the task is serialized as part of the task, a minimal user takes less space, or overhead, for each task and update in the repository.
Use information on the Reconciler status page (debug/Show_Reconciler.jsp ) to decide which settings to adjust based on queue sizes, available system resources, and performance benchmarks. Be aware that these settings are dependent on the environment.
Use the System Memory Summary page (debug/Show_Memory.jsp ) to see how much total and free memory is available. Reconciliation is a memory-intensive function, and you can use this information to determine whether there is sufficient memory allocated to the JVM. You can also use this page to launch garbage collection or to clear unused memory in the JVM for investigating heap usage.
When you assign user forms to proxy administrators who are performing reconciliations, keep the user forms as simple as possible and only use essential fields. Depending on the schema map, including a field that calculates the waveset.organization attribute is generally sufficient.
Administrators who need to view or edit the Identity Manager schema for Users or Roles must be in the IDM Schema Configuration AdminGroup and must have the IDM Schema Configuration capability.
Use per-account workflows judiciously. The reconciliation process does not start provisioning tasks for performance reasons by default.
If you must use a per-account workflow task, edit the reconciliation policy to limit the Reconciler’s automatic responses to events of interest only. (See the Situation area of the Edit Reconciliation Policy page.)
Although the default settings are usually adequate, you can sometimes improve Reconciler performance if you adjust the following settings on the Edit Server Settings page:
Parallel Resource Limit. Specifies the maximum number of resource threads that the Reconciler can process in parallel.
Resource threads allocate work items to worker threads, so if you add additional resource threads, you might also have to increase the maximum number of worker threads.
Minimum Worker Threads. Specifies the number of processing threads that the Reconciler always keeps open.
Maximum Worker Threads. Specifies the maximum number of processing threads that the Reconciler can use. The Reconciler starts only as many threads as the workload requires, which places a limit on that number. Worker threads automatically close if they are idle for a short duration.
During idle times, the threads stop if they have no work to do, but only down to the minimum number of threads specified. As the load increases, the Reconciler adds more threads until the maximum number of threads is reached. The Reconciler never has less than the minimum number of threads or more than the maximum.
Generally, more threads allow more concurrency. However, at some point, too many threads can put too much load on the machine or just do not provide additional benefit.
Recommending generic, optimal settings is not possible because deployments are so different. Reconciler settings must be adjusted differently for each deployment environment.
Perform the following steps to change the Reconciler server settings:
Log into the Administrator interface.
Click the Configure -> Servers -> Reconciler tabs.
When the Edit Server Settings page is displayed, adjust the settings as necessary.
See Editing Default Server Settings for more information.
If you are configuring reconciliation for multiple resources in Identity Manager, you have several options:
All of the resources on the same server, all at the same time.
This option is the most efficient from the Identity Manager perspective, but if you have many resources (for example more than 20), you are likely to experience Java resource issues.
All of the resources on the same server, each at a different time.
This option is easier on Java resource loading, but puts a significant burden on your schedule configuration.
Each resource on a different server, all at the same time.
This option minimizes elapsed time, but increases the number of servers.
An ideal solution does not exist for this configuration because deployments are so different. You might have to mix and match these options to find an acceptable solution for your deployment.
Preparing a usage survey, based on the business reasons behind this functionality, might help you decide how to proceed.
Address these questions:
Why are you reconciling these resources?
Do you have the same the goal for each of these resources?
Are each of these resources equally important or critical?
Must all resources be reconciled on the same schedule, or can you spread out the reconciliations?
How often must each resource be reconciled?
Also, remember that the reconciliation server does not have to be one of the pools that handles web traffic. You can add a server that you never interact with directly because this server exists solely for transaction processing. Having a server dedicated to transaction processing might make the first option more attractive for very large systems.
Network latency tends to be a common cause of performance issues during view provisioning. Tracing individual resource adapters can help you determine what is causing performance problems.
You can improve resource query performance if you use FormUtil.getResourceObjects to implement the query.
Use one of the following methods to cache query results:
getResourceObjects(Session session, String objectType, String resID, Map options, String cacheList, String cacheTimeout, String cacheIfExists)
getResourceObjects(String subjectString, String objectType, String resId, Map options, String cacheList, String cacheTimeout, String clearCacheIfExists)
Set cacheTimeout in milliseconds.
Restrict searches to specific searchContext, if applicable.
Return the minimum number of attributes in options.searchAttrsToGet.
The Scheduler component controls task scheduling in Identity Manager.
This section suggests methods for improving Scheduler performance, including:
The following TaskDefinition options determine how the Scheduler handles tasks after they are completed:
resultLimit — Controls how many seconds a task is allowed to run after the task has completed. The default setting varies for different tasks. A setting of zero immediately removes tasks after completion.
resultOption — Controls what action is taken when repeated instances of a task are started. The default setting is delete, which removes extra task instances.
These default settings are designed to optimize memory by shortening the lifetime of finished Scheduler tasks. Unless there is a compelling reason to change these settings, use the defaults.
If you want to immediately delete tasks that completed successfully, but you also want to keep tasks containing errors long enough to debug, you can do the following:
Conditionally delete finished tasks by setting the resultLimit value to a time period that is sufficient for debugging issues.
Set the resultLimit to zero (or a small value) if no errors (such as WF_ACTION_ERROR is <null/>) are reported at runtime after a WorkflowServices call.
You can sometimes improve Scheduler performance by adjusting the following settings on the Edit Server Settings page:
Maximum Concurrent Tasks. Specifies the maximum number of tasks that the Scheduler can run at one time.
When more tasks are ready to run than the Maximum Concurrent Tasks setting allows, the extra tasks must wait until there is room available or until they are run on another server.
If too many tasks are being swapped out of memory and sharing CPU time, the overhead slows down performance. Alternatively, setting the maximum too low results in idle time. The Scheduler checks for available tasks every minute, so a waiting task waits at least a minute before being run.
The default Maximum Concurrent Tasks setting (100) is usually adequate. You can decide whether to adjust this setting up or down based on which tasks are being run in the deployment and by profiling the runtime behavior after the deployment is otherwise complete.
In some cases, you might want to suspend or disable the Scheduler. For example, if you want a server dedicated to handling the End User interface, disabling the Scheduler will prevent tasks from running on that server. The server would only serve the End User interface pages and store launched tasks for other servers to execute.
Task Restrictions. Specifies the set of tasks that can execute on the server.
The Task Restrictions setting can provide a finer granularity of control over what tasks are allowed to run on a server. You can restrict tasks individually or through the server settings.
Recommending generic, optimal settings is not possible because deployments are so different. Scheduler settings must be adjusted differently for each deployment environment.
Log in to the Administrator interface.
Click the Configure -> Servers -> Scheduler tabs.
When the Edit Server Settings page is displayed, adjust the settings as necessary.
See Editing Default Server Settings for more information.
Identity Manager maintains a least recently used (LRU) cache of authenticated sessions for use by authenticated users. By using existing authenticated sessions, you can speed up repository access for objects and actions that require a session.
To optimize the authentication pool size, change the session.userPoolSize value in the Waveset.properties file to the maximum number of expected, concurrent user sessions on the server.
The Sun Identity Manager Gateway generates a thread for each connection, and uses a different pool for each unique combination of resource type, Gateway host, and Gateway port. The Gateway checks for idle connections every five minutes. When a connection has been idle for 60 seconds, the Gateway closes and removes that connection from the pool.
When the Gateway receives a request, it does the following:
If there are no idle connections in the corresponding pool, the Gateway creates a new connection.
If an idle connection exists in the pool, the Gateway retrieves and reuses that connection.
You must configure the maximum number of connections on the resource, and you must configure these connections the same way for all resources of the same type, that are using the same Gateway. For that resource type, the first connection made to the Gateway on a given host and port uses that resource’s maximum connections value.
When you change the maximum number of connections on a resource, you must start and stop the server for the change to take effect.
The following example shows how connections, requests, and Gateway threads are related.
If you set the maximum number of connections to 10 on an Active Directory resource, and you are using two Identity Manager servers, then you can have up to 20 simultaneous connections (10 from each Identity Manager server) to the Gateway for that Active Directory resource. The Gateway can have 10 simultaneous requests outstanding from each server, and the Gateway processes each request on a different thread. When the number of simultaneous requests exceeds the maximum number of Gateway connections, additional requests are queued until the Gateway completes a request and returns the connection to the pool.
Although the Gateway code is multi-threaded, this characteristic does not address the APIs or services being used by the Gateway. For Active Directory, the Gateway uses the ADSI interface provided by Microsoft. No investigation has been done to determine whether this interface handles Gateway requests in parallel.
Other methods for improving Gateway performance, include:
Locating the Gateway near (from a network connectivity perspective) the domain controllers of the managed domain
Increasing the block size on a Gateway resource can increase throughput during reconciliation or load operations
Increased throughput results have been noted for basic reconciliations with no custom workflows and in which no attribute reconciliations are being performed. Initially, the Gateway does consume more system memory, but this memory is eventually released.
Be aware that there is a diminishing return. At some point, larger block sizes do not result in proportionately increased performance. For example, the following data shows the speed observed for a Load from Resource of 10,000 users from an Active Directory resource. Also, the peak memory usage for the Gateway process during the load is included.
Block Setting |
Users Created Per Hour |
Peak Gateway Memory Usage |
---|---|---|
100 |
500 |
20 MB |
200 |
250 |
25 MB |
500 |
9690 |
60 MB |
1000 |
10044 |
92 MB |
For Exchange Server 2007, the PowerShellExecutor performs actions for Exchange Server 2007. You can modify the following registry settings to change the behavior of the PowerShellExecutor inside the Gateway.
Both settings can have a large impact on the behavior and memory usage of the Gateway. Changes to these parameters should only be considered after careful testing.
powerShellTimeout
Content. Timeout for PowerShell actions (registry type REG_DWORD)
Default. 60000 ms (1 minute)
When the powerShellTimeout setting times out, any RunSpace actions are interrupted and canceled to prevent runaway actions in the PowerShell environment that cause the Gateway to become unresponsive.
Decreasing the powerShellTimeout value to a small value can prematurely cancel actions, and can prevent the RunSpace initialization from finishing correctly. Observed startup times for the first RunSpace in the pool range from 2—5 seconds.
The powerShellTimeout value is read-only on startup, and you cannot change it without restarting the gateway.
runSpacePoolSize
Content. Number of RunSpaces in the pool (registry type REG_DWORD)
Default. 5
Minimum. 5
Maximum. 25
The number of RunSpaces in the pool allow for parallel execution of PowerShell actions by the gateway. One provisioning action or update of a user in Exchange 2007 can result in multiple PowerShell actions being executed.
A started RunSpace can consume a large amount of memory. For the first RunSpace, the typical size is approximately 40 MB. Subsequent RunSpaces normally use between 10—20 MB.
The preceding figures can differ in specific environments and are only given as guidelines, so be careful when changing this value.
The runSpacePoolSize value is read-only on startup, and you cannot change the pool size value without restarting the Gateway.
The Administrator interface task bar displays links to previously performed provisioning tasks, which causes the interface to render more slowly when there are a large number of tasks.
To improve interface performance, remove the taskResults.jsp link from each JSP by deleting the <List>...</List> element from the UserUIConfig object.
The following example shows <List>...</List> entries within <TaskBarPages>.
<TaskBarPages> <List> <String>account/list.jsp</String> <String>account/find.jsp</String> <String>account/dofindexisting.jsp</String> <String>account/resourceReprovision.jsp></String> <String>task/newresults.jsp</String> <String>home/index.jsp</String> </List> </TaskBarPages> |
This section describes the different Identity Manager and third-party debugging tools you can use to debug performance issues.
The information is organized into the following sections:
Tracing affects system performance. To help ensure optimal performance, specify the minimum tracing level or turn tracing off after debugging the system.
This section provides instructions for accessing the Identity Manager Debug pages and describes how to use these pages to identify and debug Identity Manager performance issues.
See the following sections for information:
Provisioning Threads for Administrator Configurator (Show_Provisioning.jsp)
XML Resource Adapter Caches Flushed and Cleared (Clear_XMLResourceAdapter_Cache.jsp)
You must have the Debug, Security Administrator, or Waveset Administrator capabilities to access and execute operations from the Identity Manager Debug pages. Administrators and Configurator are assigned this capability by default.
If you do not have the Debug capability, an error message results.
Open a browser and log in to the Administrator interface.
Type the following URL:
http:// host:port /idm/debug
where:
host is the application server on which you are running Identity Manager.
port is the number of the TCP port on which the server is listening.
When the System Settings page displays, type the .jsp file name for the debug page you want to open.
For example:
http:// host:port /idm/debug/pageName.jsp
Some debugging utilities are not linked from the System Settings page, but you can use them to enhance your ability to gather data for product performance and usability. For a complete list of debug pages, open a command window and list the contents of the idm/debug directory.
Use the Control Timings page to collect and view call timer statistics for different methods. You can use this information to track bottlenecks to specific methods and invoked APIs. You can also use options on the Call Timings page to import or export call timer metrics.
Call timing statistics are only collected while trace is enabled.
Open the Control Timings page, and click Start Timing & Tracing to enable trace and timing.
To stop the timing, click Stop Timing & Tracing or click Stop Timing.
The page re-displays and populates the Show Timings table with a list of methods for which statistics are available and the methods’ aggregate call timer statistics (not broken down by caller).
This table contains the following information:
Method name (Click a method name to see which methods it calls)
Total time
Average time
Minimum time
Maximum time
Total calls
Total errors
To clear the list, click Clear Timing.
You can also use the callTimer command to collect call timer data from the Console. This command is useful when you are debugging performance issues during an upgrade or in other situations where Identity Manager is not running on the application server.
Use the Edit Trace Configuration page to enable and configure tracing for the Java classes provided with your Identity Manager installation.
Specifically, you can use this page to configure the following trace settings:
Choose methods, classes, or packages to trace and specify the level of trace you want to capture.
Send trace information to a file or to standard output.
Specify the maximum number of trace files to be stored and the maximum size for each file.
Specify how dates and times are formatted in the trace output file.
Specify the maximum number of methods to be cached.
Indicate how to write data to the trace file.
Write data to the trace file as the data is generated, or queue the data and then write it to the file.
If you are not using a data source, you can use the Host Connection Pool page to view connection pool statistics. These statistics include the pool version, how many connections were created, how many are active, how many connections are in the pool, how many requests were serviced from the pool, and how many connections were destroyed.
You can also use the Host Connection Pool page to view a summary of the connection pools used to manage connections to the Gateway. You can use this information to investigate low-memory conditions.
Use the List Cache Cleared page to clear recently used XML parsers from the cache and to investigate low memory conditions.
Use the Method Timings page to quickly detect and assess hotspots at a method level.
The following information is gathered from Identity Manager methods and displayed on the Method Timings page:
Method names
How many times the methods were called
How many times the methods exited with an error status
Average time consumed by the methods
Minimum and maximum times consumed by invocations of each method
The Method Timings page also contains a table with the following links. You can click these links to view additional information.
Details. Shows call stack information.
History. Shows a graph of call duration compared with the time of the most recent calls.
History data. Shows a list of the most recent calls, showing what time the call was made and the duration of the call.
Identity Manager does not keep stack history by default. To keep stack history and to control its depth, edit Waveset.properites and look at the MethodTimer keys.
The Clear ALL option on the Method Timings page clears all results. This option is enabled by default.
Use the Object Size Summary page to detect problematically large objects that can affect your system.
The Object Size Summary page shows information about the size of objects (in characters) stored in the repository. These objects are listed by type, along with the total number of objects of each type, and the objects’ total combined size, average size, maximum size, and minimum size.
Click an entry in the Type column to view additional size information about that object type. For example, click Configuration to view the ID, name, and size of the largest configuration objects in the repository.
You can also access this size information from the Console command line.
Open the console.
At the command prompt, type:
showSizes [ type[limit ]]
For upgrades, existing objects will report a size of 0 until they have been updated or otherwise refreshed.
Use the Provisioning Threads for Administrator Configurator to view a summary of the provisioning threads in use by the system. This summary is a subset of the information available in Show_Threads.jsp.
Looking at just a single thread dump can be misleading.
Use the System Cache Summary page to view information about the following items to help you investigate low-memory conditions:
Administrator-associated object caches
System object cache
User login sessions
XML parser cache
Use the System Memory Summary page to view how much total and free memory you have available in Mbytes. When you are using memory-intensive functionality such as Reconciliation, this information can help you determine whether there is sufficient memory allocated to the JVM.
You can also use this page to launch garbage collection or to clear unused memory in the JVM for investigating heap usage.
The System Properties page provides information about your environment, including software versions, paths and environmental variables.
Use the System Threads page to see which processes are running so you can verify that automated processes (such as reconciliation or Active Sync) are running.
This page includes information about the process type, process name, its priority, if the process is a daemon, and if the process is still running.
Looking at just a single thread dump can be misleading.
Use the Session Pool Cleared page to clear all of the cached sessions for users who have recently logged in and to investigate low memory conditions.
Use the Waveset Properties page to view and temporarily edit properties in the Waveset.properties file. You can test different property settings for a particular server on which the Waveset.properties file resides without having to restart the server to pick up the changes. The edited property settings only remain in effect until the next time you restart the server.
Use the XML Resource Adapter Caches Flushed and Cleared page to clear test XML resource adapters from the cache and to investigate low memory conditions.
You can use the following Sun Microsystems’ and third-party tools to identify potential performance bottlenecks:
These tools can be particularly useful if your deployment uses custom Java classes.
Identity Manager provides a Profiler utility to help you troubleshoot performance problems in your deployment.
Customized forms, Java, rules, workflows, and XPRESS can cause performance and scale problems. The Profiler profiles how much time is spent in these different areas, enabling you to determine whether these forms, Java, rules, workflows, or XPRESS objects are contributing to performance and scale problems and, if so, which parts of these objects are causing the problems.
For more information about Profiler, see Working with the Identity Manager Profiler in Sun Identity Manager 8.1 Release Notes.
The DTrace facility is a dynamic tracing framework for the Solaris 10 operating system that enables you to monitor JVM activity.
DTrace contains more than 30,000 probes and uses integrated user-level and kernel-level tracing to give you a view into your production system. You can also trace arbitrary data and expressions by using the D language, which is similar to C or awk. The DTrace facility also includes special support for monitoring the JVM, and enables you to watch your whole system and span outside the JVM.
DTrace is easiest to use with Java 6 because probes are built into the JVM. The facility also works with Java 1.4 and Java 5, but you must download JVM PI or JVM TI agents from the following URL:
https://solaris10-dtrace-vm-agents.dev.java.net/
The following example shows how to write a DTrace script.
#!/usr/sbin/dtrace -Zs #pragma D option quiet hotspot$1::: { printf("%s\n", probename); } |
In this example, you would replace $1 with the first argument to the script, which is the PID of the Java process you want to monitor. For example:
# ./all-jvm-probes.d 1234
The following table describes the commands you can use to enable different DTrace probes.
Table 4–3 DTrace Commands
Because DTrace causes additional work in the system, enabling this facility affects system performance. The effect is often negligible, but can become substantial if you enable many probes with costly enablings.
Instructions for minimizing the performance effect of DTrace are provided in the “Performance Considerations” chapter of the Solaris Dynamic Tracing Guide.
For more information about DTrace, see /usr/demo/dtrace and man dtrace.
Identity Manager enables you to use Java Management Extensions (JMX) to capture and expose operational statistics for certain resource adapter operations. You can use this data for diagnostic and predictive purposes, such as to monitor system health and reports.
This statistical data includes the following:
The number of times the action was performed
The minimum, maximum, and average duration of the operations
Objects |
Actions Monitored |
---|---|
For Accounts |
|
For Actions |
Run |
For Other Objects |
|
JMX creates MBeans for each resource adapter, by server, and registers these beans with names that match the following pattern:
serverName=server name, resourceAdapterType=Resource Adapter Type, resourceAdapterName=Resource Adapter Name |
Identity Manager records statistics for all completed operations, whether they completed successfully or with errors. However, Identity Manager does not record statistics for incomplete operations, such as any operations that throw exceptions.
You can configure excludes as follows:
From the Administrator interface, select Configure -> Servers.
On the Configure Servers page, perform one of the following tasks:
Click the Edit Default Server Settings button to edit the default server settings.
Click a server link to edit the policy for that server.
Click the JMX tab and enable the JMX Enable Resource Adapter Monitor box to turn on resource monitoring.
To exclude specific resources, add regular expressions to the JMX Resource Adapter Monitor Excludes list.
To exclude monitoring specific actions, add regular expressions to the JMX Resource Adapter Monitor Operation Excludes list.
All excludes use regular expressions. For excluding certain resources, JMX just matches on the resource name. For example, if you have adapters named
resource1 resource2 resource3 resource10 resource11 |
and you specify the following pattern
.*1$ |
which means, match any 0 or more of any character (.*) until something that ends with a 1 (1$). JMX will exclude resource1 and resource11.
For operations, the process is similar. If your operations have the following names, the patterns must match against those names.
ACCOUNT_CREATE ACCOUNT_UPDATE ACCOUNT_DELETE ACCOUNT_GET ACCOUNT_AUTHENTICATE OBJECT_CREATE OBJECT_UPDATE OBJECT_DELETE OBJECT_GET OBJECT_LIST ACTION_RUN |
For example, the ^ACCOUNT.* pattern excludes all operations that start with ACCOUNT. Or, using this pattern excludes updates and deletes:
.*UPDATE$ .*DELETE$ |
For more information about configuring and using JMX, see Configuring JMX Monitoring and The JMX Publisher Type in Sun Identity Manager 8.1 Business Administrator’s Guide.
The Java Monitoring and Management Console (JConsole) is a Java Management Extension (JMX) technology-compliant graphical management tool that is co-packaged with at least JDK 5. JConsole connects to a running JVM and gathers information from the JVM MBeans in the connected JMX agent.
Specifically, you can use JConsole to perform the following tasks:
Detect low memory and deadlocks
JConsole accesses the memory system, memory pools, and MBeans garbage collector to provide information about memory use, including memory consumption, memory pools, and garbage collection statistics.
Enable or disable garbage collection
Enable or disable verbose tracing
Monitor local and remote applications
Monitor and manage MBeans including current heap memory use, non-heap memory use, and how many objects are pending for finalization
View information about performance, resource consumption, and server statistics
View summary information about the JVM and monitored values, threads running on the application, and loaded classes
View information about operating system resources (Sun’s platform extension), such as:
CPU process time
How much total and free physical memory is available
The amount of committed virtual memory (how much virtual memory is guaranteed to be available to the running process)
How much total and free swap space is available
The number of open file descriptions (UNIX®only)
For more information about using JConsole to monitor applications on the Java platform, see the Sun Developer Network (SDN) article titled Using JConsole to Monitor Applications, which is available from the following URL:
http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html
Identity Manager supplies some JMX MBeans that provide information about the following:
Identity Manager Server Cluster
Data Exporter
Scheduler
You can use the Java Runtime Analysis Toolkit (JRat), an open-source performance profiler for the Java platform, to identify potential performance bottlenecks, especially if your deployment uses custom Java classes. JRat monitors your application’s execution and persists the application’s performance measurements.
For example, if you have a custom workflow for provisioning, you can use JRat to see which classes are being invoked and how much time is required to run your workflow compared to the default Identity Manager provisioning workflow.
For more information about JRat, see http://jrat.sourceforge.net.