test

Sun Identity Manager 8.1 System Administrator's Guide

Chapter 4 Performance Tuning

You can significantly improve the performance of Sun Identity Manager (Identity Manager) software across nearly all activities with the proper tuning. In addition to changing settings within the software, you can make performance improvements by tuning the application server, the JavaTM Virtual Machine (JVMTM machine), hardware, operating system, and network topology.

Additionally, you can use several tools to diagnose and monitor performance. Several of these tools exist within Identity Manager, such as trace and method timers. You can also use other Sun Microsystems and third-party tools to debug performance issues with Identity Manager.

This chapter describes tools, methodologies, and references you can use to improve performance and to debug performance-related issues.

Note –

The tuning process spans many entities and is specific to your deployment environment. This chapter describes different tuning methods for optimizing Identity Manager performance, but these methods are only intended as guidelines. You might have to adapt these methods for your deployment environment.

This chapter covers the following topics:

Before You Begin Tuning

Review all of the information in this section before you start tuning Identity Manager.

Important Notes

Note –

The tuning methods described in this chapter are only provided as guidelines. You might have to modify some of these tunings for your deployment. In addition, Be sure to validate tunings before applying changes in a production environment.

Before you can tune Identity Manager, you must:

Related Documentation and Web Sites

In addition to the information provided in this chapter, consult the documents and web sites listed in this section for information related to tuning Identity Manager.

Recommended Reading

See the following documents for information related to performance tuning.

Table 4–1 Related Documentation

document Title 

Description 

IBM Developer Kit and Runtime Environment, Java Technology Edition, Version 5.0 Diagnostics Guide

Explains how to use AIX JVM to diagnose performance problems.  

Java Tuning White Paper

Contains information, techniques, and pointers related to Java performance tuning.  

Oracle MetaLink: Note:114671.1: Gathering Statistics for the Cost Based Optimizer

Explains how to use system statistics and Oracle®’s Cost-Based Optimizer (CBO).

Note: This document is available to Oracle Metalink subscribers. Registration is required.

Solaris Dynamic Tracing Guide

Explains how to use DTrace to observe, debug, and tune your system's behavior. 

Sun JavaTM System Application Server Performance Tuning Guide

Describes how to obtain optimal performance from your Sun Java System Application Server. Download the necessary version of this book from the Sun Microsystems documentation web site.

Tuning Garbage Collection with the 5.0 Java Virtual Machine

Describes how to tune your garbage collection application by using JVM. 

Turbo-charging Java HotSpot Virtual Machine, v1.4.x to Improve the Performance and Scalability of Application Servers

Explains how to download and use the PrintGCStats script and how to collect statistics to derive optimal JVM tunings.

Understanding System Statistics

Describes how Oracle’s Cost-Based Optimizer uses system statistics. 

Using JConsole to Monitor Applications

Describes how to use JConsole to monitor applications that run on the Java platform. 

Useful Web Sites

The following table describes some web sites that you might find useful when trying to tune Identity Manager performance.

Table 4–2 Useful Web Sites

Web Site URL 

Description 

http://sunsolve.sun.com

Sun web site containing diagnostic tools, forums, features and articles, security information, and patch contents. 

Note: The information on this site is divided into three areas:

  • Internal. Available only to Sun employees

  • Contract. Available only to customers with contract access

  • Public. Available to everyone

http://forum.java.sun.com/

Sun Developer Network (SDN) web site where you can browse forums and post questions. 

http://jrat.sourceforge.net/

JRat web site that describes how to use the Java Runtime Analysis Toolkit, an open source performance profiler for the Java platform. 

https://metalink.oracle.com/

Oracle’s internal forum site that contains information about tuning Oracle databases. 

Note: You must be an Oracle Metalink subscriber to access the information provided on this site.

http://performance.netbeans.org/howto/jvmswitches/index.html

NetBeansTM web site containing information about tuning JVM switches for performance.

https://sharespace.sun.com/gm/folder-1.11.60181?

Identity Manager link on Sun’s Share Space. 

Note: You must sign up for a Share Space ID to access information provided on this site.

https://sharespace.sun.com/gm/document-1.26.2296

Identity Manager FAQ on Sun’s Share Space. 

Note: You must sign up for a Share Space ID to access this FAQ.

http://www.slamd.com/

SLAMD Distributed Load Generation Engine web site. 

http://www.opensolaris.org/os/community/dtrace/

OpenSolaris Community: DTrace web page. 

http://www.solarisinternals.com/

http://docs.sun.com/app/docs/prod/solaris.10

Web sites containing information related to tuning the Solaris OS. 

Tuning Roadmap

How well your Identity Manager solution performs can depend on the following deployment-specific settings:

When you are trying to debug performance problems, start by analyzing and describing the problem. Ask yourself the following questions:

Tuning Your Deployment Environment

This section provides information about tuning your deployment environment, including:

Tuning Your Java EE Environment

This section describes some tuning suggestions you can use to optimize your Java Platform, Enterprise Edition (Java EE platform) environment.

These tuning suggestions were derived from a series of experiments in which a considerable increase in throughputs was observed for the use cases tested. The increases were attributed to JVM sizing and to switches that affected garbage collector behavior.

Note –

For more information about tuning Java, JConsole, or JVM, visit the web sites noted in Table 4–1 and Table 4–2.

The following sections provide information about tuning Java and the JVM in your Java EE environment.

Tuning Java

For information, best practices, and examples related to Java performance tuning, see the Java Tuning White Paper at:

http://java.sun.com/performance/reference/whitepapers/tuning.html

Tuning the JVM

The following tuning scripts were used to derive the tuning suggestions noted in this section. These scripts were added to the domain.xml file (located in the domain configuration directory, which is typically domain-dir/config ) on a Sun Java System Application Server.

To help ensure the best JVM performance, verify the following:

Tuning Your Application Server

The following guidelines are provided to help you tune your application server:

Note –

Other than heap size, you can use the default parameter settings for most application servers. You might want to modify the server’s heap size, depending on the release being used.

Tuning a Sun Java System Application Server

The “Tuning the Application Server” chapter, in the latest Sun Java System Application Server Performance Tuning Guide, contains information about tuning a Sun Java System Application Server. This document is available from the following URL at http://docs.sun.com/app/docs.

In addition, if you are using Sun Java System Application Server 8.2 Enterprise Edition, the following changes solve “concurrent mode failures,” and should give you better and more predictable performance:

You might have to adjust this value further.

Tuning a WebSphere Application Server

If you are tuning Identity Manager on an IBM WebSphere® application server, consider limiting how much memory is allocated for the heap because heap memory can affect the memory used by threads.

If many threads are created simultaneously and the heap size increases, the application’s space limit can be quickly impacted and the following error results:

JVMCI015:OutOfMemoryError

Tuning Your Repository Database

Identity Manager relies on the repository database to store and manage its identity and configuration data. For this reason, database performance can greatly influence Identity Manager’s performance.

Note –

Detailed information about performance tuning and managing databases is beyond the scope of this document because this information is dataset-specific and vendor-specific. In addition, customer database administrators (DBAs) should already be experts on their own databases.

This section characterizes the Identity Manager application and provides general information about the nature of Identity Manager data and its typical usage patterns to help you plan, tune, and manage your databases.

This information is organized into the following sections:

Repository Table Types

The Identity Manager repository contains three types of tables, and each table has slightly different usage characteristics. Information about these tables is organized into the following sections:

Attribute Tables

Attribute tables enable you to query for predefined single-valued or multi-valued object attributes.

For most object types, stored attributes are hard-coded.

Note –

The User and Role object types are exceptions to this rule. The inline attributes that are stored in the object table for User and Role are configurable, so you can configure additional custom attributes as queryable.

When you search for objects based on attribute conditions, Identity Manager accesses attribute tables in joins with the corresponding object tables. Some form of join (such as a JOIN, an EXISTS predicate, or a SUB-SELECT) occurs for each attribute condition.

The number of rows in the attribute table are proportional to the number of rows in the corresponding object table. The values distribution might exhibit skew, where multi-valued attributes have a row per value and some objects might have more attributes or more attribute values than others. Typically, there is a many-to-one relation between rows in the attribute table and rows in the object table.

Attribute tables have ATTR in the table name.

Change Tables

Identity Manager uses a change table to track changes made to a corresponding object table. These table sizes are proportional to the rate of object change, but the tables are not expected to grow without bound. Identity Manager automatically truncates change tables.

Change tables can be highly volatile because the lifetime of a row is relatively short and new rows can be created frequently.

Access to a change table is typically performed by a range scan on the time-stamp field.

Change tables have CHANGE in the table name.

Object Tables

The Identity Manager repository uses object tables to hold serialized data objects, such as Large Objects (LOBs). Object tables can also hold commonly queried, single-valued object attributes.

For most object types, stored attributes are hard-coded.

Note –

The User and Role object types are exceptions to this rule. The inline attributes that are stored in the object table are configurable, and you can configure additional custom attributes as queryable for User and Role.

The number of rows in an object table equals the number of objects being stored. The number of objects stored in each object table depends on which object types are being stored in the table. Some object types are numerous, while other types are few.

Generally, Identity Manager accesses an object table by object ID or name, though Identity Manager can also access the table by using one of the attributes stored in the table. Object IDs and names are unique across a single object type, but attribute values are not unique or evenly distributed. Some attributes have many values, while other attributes have relatively few values. In addition, several object types can expose the same attribute. An attribute may have many values for one object type and few values for another object type. The uneven distribution of values might cause an uneven distribution of index pages, which is a condition known as skew.

Object tables are tables that do not have ATTR or CHANGE suffixes in the table name.

XML Columns

Every object table contains an XML column, which is used to store each serialized object except the LOG table-set. Certain LOG table-set optional attributes are stored in the XML column if these attributes are present. For example, if digital signing is enabled.

Data Classes

You can roughly divide Identity Manager data into a number of classes that exhibit similar properties with respect to access patterns, cardinality, lifetime, volatility, and so forth. Each of the following classes corresponds to a set of tables in the repository:

User Data

User data consists of user objects.

You can expect this data to grow quite large because there is an object for each managed identity. After an initial population phase, you can expect a proportionally small number of creates because the majority of operations will be updates to existing objects.

User objects are generally long-lived and they are removed at a relatively low rate.

User data is stored in USEROBJ, USERATTR, and USERCHANGE tables.

Role Data

Role data consists of Role objects, including Roles subtypes such as Business Roles, IT Roles, Applications, and Assets.

Role data is similar to organization data, and these objects are relatively static after a customer deploys Identity Manager.

Note –

An exception to the preceding statement is a deployment that is integrated with an external source containing an authoritative set of roles. One integration style might be to feed role changes into Identity Manager, which causes Identity Manager Role data to be more volatile.

Generally, the number of role objects is small when compared to the number of identity objects such as users (assuming that multiple users share each role), but this depends on how each enterprise defines its roles.

Role data is stored in ROLEOBJ, ROLEATTR, and ROLECHANGE tables.

Account Data

Account data solely consists of account objects in the Account Index.

As with user data, account data can become rather large, with an object for each known resource account. Account objects are generally long-lived, removed at a relatively low rate, and after initial population, are created infrequently. Unless you frequently add or remove native accounts, or enable native change detection, account object modifications occur infrequently.

Identity Manager stores account data in ACCOUNT, ACCTATTR, and ACCTCHANGE tables.

Compliance Violation Data

Compliance Violation data contains violation records that indicate when the evaluation of an Audit Policy failed. These violation records exist until the same Audit Policy is evaluated against the same User and the policy passes. Violation records are created, modified, or deleted as part of an Audit Policy Scan or as part of an Access Review.

The number of violation records is proportional to the number of Audit Policies that are used in scans and the number of Users. An installation with 5000 users and 10 Audit Policies might have 500 violation records (5000 x 10 x 0.01), where the 0.01 multiplier depends on how strict the policies are and how user accounts are changed.

Identity Manager stores Compliance Violation records in OBJECT, ATTRIBUTE, and OBJCHANGE tables.

Entitlement Data

Entitlement data predominately consists of user entitlement objects, which are only created if you are doing compliance access reviews.

Entitlement records are created in large batches, modified slowly (days) after initial creation, and are then untouched. These records are deleted after an Access Review is deleted.

Identity Manager stores entitlement data in ENTITLE, ENTATTR, and ENTCHANGE tables.

Organization Data

Organization data consists of object group or organization objects.

Object group data is similar to configuration data, and this data is relatively static after being deployed. Generally, the number of objects is small (one for each defined organization) when compared to task objects or to identity objects such as users or accounts, however, the number can become large compared to other configuration objects.

Organization data is stored in ORG, ORGATTR, and ORGCHANGE tables.

Task Data

Task data consists of objects that are related to tasks and workflows, including state and result data.

The data contained in these tables is short-lived compared to other classes because objects are created, modified, and deleted at a high rate. The volume of data in this table is proportional to the amount of activity on the system.

Task data is stored in TASK, TASKATTR, and TASKCHANGE tables.

Configuration Data

Configuration data consists of objects related to Identity Manager system configuration, such as forms, roles, and rules.

Generally, configuration data is:

Identity Manager stores configuration data in ATTRIBUTE, OBJCHANGE, and OBJECT tables.

Export Queue Data

If you enable Data Exporting, some records are queued inside Identity Manager until the export task writes those records to the Data Warehouse. The number of records that are queued is a function of Data Exporting configuration and the export interval for all queued types.

The following data types are queued by default, and all other data types are not:

The number of records in these tables grows until the export task drains the queue. The current table size is visible through a JMXTM Bean.

Records added to this table are never modified. These records are written during other Identity Manager activities, such as reconciliation, provisioning, and workflow execution. When the Data Exporter export task runs, the task drains the table.

Identity Manager stores Export Queue data records in QUEUE, QATTR, and QCHANGE tables.

Log Data

Log data consists of audit and error log objects. Log data is write-once only, so you can create new audit and error log objects, but you cannot modify these objects.

Log data is long-lived and can potentially become very large because you can only purge log data by explicit request. Access to log data frequently relies on attributes that are stored in the object table instead of in the attribute table. Both the distribution of attribute values and queries against the log specifically depend on how you are using Identity Manager.

For example, the distribution of attribute values in the log tables depends on the following:

The pattern of queries against the log table also depends on which Identity Manager reports, which custom reports, or which external data mining queries a customer runs against the log table.

Identity Manager stores audit log records in LOG and LOGATTR tables, and error log records in SYSLOG and SLOGATTR tables. This data does not have corresponding change tables.

Object IDs

Identity Manager generates globally unique identifiers (GUIDs) for objects by using the VMID class provided in the JDK software.

These GUID values exhibit a property that gets sorted by its string representations, based on the order in which the objects are created. For example, when you create new objects with Identity Manager, the newer objects have object IDs that are greater than the older objects. Consequently, when Identity Manager inserts new objects into the database, the index based on object IDs can encounter contention for the same block or blocks.

Prepared Statements

Generally, Identity Manager uses prepared statements for activities (such as inserting and updating database rows), but does not use prepared statements for queries.

If you are using Oracle, this behavior can create issues with the library cache. In particular, the large number of statements versions can cause contention on the library cache latch.

To address this contention, change the Oracle CURSOR_SHARING parameter value from EXACT to SIMILAR. Changing this value causes Oracle to replace literals in SQL statements with bind variables, thereby reducing the number of versions.

Character Sets and Encodings

Because Identity Manager is a Java application that generally reads and writes character data rather than bytes, it does not restrict which encoding the database uses.

Identity Manager only requires that the data is sent and returned correctly. For example, the data does not become corrupted when written or reread. Use an encoding that supports multi-byte characters and is appropriate for the customer’s data. Generally, UTF-8 encoding is sufficient, but enterprises with a large number of true multi-byte characters, such as Asian or Arabic, might prefer UTF-16.

Most database administrators prefer to use an encoding that supports multi-byte characters because of the following:

General Guidelines for Tuning a Repository Database

This section describes some general guidelines for tuning a repository database:

For more information about which object types are stored in each set of tables, see Data Classes.

Vendor-Specific Database Tuning Guidelines

This section describes some vendor-specific guidelines for tuning Oracle and SQL Server repository databases.

Note –

Currently, MySQLTM databases are only supported in development and for demonstrations.

Oracle Databases

This section describes guidelines for tuning Oracle repository databases:

Identity Manager does not require Oracle init.ora parameter settings for SGA sizing, buffer sizing, open cursors, processes, and so forth.

SQL Server Databases

Some customers who used an SQL Server 2000 database as a repository reported that as concurrency increased, SQL Server 2000 reported deadlocking problems that were related to SQL Server’s internal use of pessimistic locking (primarily lock escalation).

These deadlock errors display in the following format:


com.waveset.util.IOException:
  ==> com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 51) 
was deadlocked on lock | communication buffer resources with another 
process and has been chosen as the deadlock victim. Rerun the transaction.

    To prevent or address deadlocking problems, do the following:

  1. Use the SQL Server 2005 database.

  2. Configure the READ_COMMITTED_SNAPSHOT parameter by formatting the command as follows:

    ALTER DATABASE waveset SET READ_COMMITTED_SNAPSHOT ON

    Enabling the READ_COMMITTED_SNAPSHOT parameter does the following:

    • Removes contention during the execution of SELECT statements that can cause blocks, which greatly reduces the potential for deadlocks internal to SQL Server.

    • Prevents uncommitted data from being read and guarantees that SELECT statements receive a consistent view of committed data.

    For more information about the READ_COMMITTED_SNAPSHOT parameter, see: http://msdn2.microsoft.com/en-us/library/ms188277.aspx.

Tuning Identity Manager Performance

Suggestions for optimizing Identity Manager’s performance are organized into the following areas:

General Performance Tunings

In general, you can optimize Identity Manager performance if you do the following:

Tuning Active Sync Adapter Performance

Because synchronization is a background task, how you configure an Active Sync adapter can affect server performance.

Use the Resources list to manage Active Sync adapters. Choose an Active Sync adapter and access start, stop, and status refresh control actions from the Synchronization section of the Resource Actions list.

To improve Active Sync adapter performance, do the following:

Tuning Bulk Loading

To improve performance during bulk load operations, do the following:

Tuning Configurable XML Objects

Configurable XML objects offer a broad spectrum of user interface specifications that enable you to define how data is presented to users for different tasks and to automate complex business processes. However, this same flexibility can affect efficiency, performance, and reliability.

This section describes some guidelines for tuning Identity Manager’s configurable XML objects, which consist of forms, rules, and workflows. The information is organized into the following sections:

Tuning Forms

You can use Identity Manager forms to define interfaces to interact with views or variable contexts in an executing task. Forms also provide an execution context for business and transformation logic on a set of data elements. Although you can create very powerful, dynamic forms that perform a variety of tasks, reducing the complexity of forms increases efficiency.

The following sections describe some methods for improving the performance of your customized forms:

Optimizing New Forms

When designing new Identity Manager forms, system integrators can optimize a form’s performance by doing the following:

Optimizing Administrator Forms

To improve the performance of administrator forms, do the following:

Optimizing End-User Forms

To improve the performance of end-user forms, do the following:

Optimizing Expressions in Form Fields

Some activities performed in forms call resources that are external to Identity Manager. Accessing these resources can affect Identity Manager performance, especially if the results contain long lists of values, such as compiling a list of groups or email distribution lists.

To improve performance during these calls, follow the guidelines in “Using a Java Class to Obtain Field Data” in Sun Identity Manager Deployment Reference.

Also, avoid using JavaScriptTM in performance-critical expressions such as <Disable> expressions. Short XPRESS expressions are easier to debug if you use the built-in tracing facilities. Use JavaScript for complex logic in workflow actions.

If a form is slow to display, you can use the debug/Show_Timings.jsp page to determine the problem. Look for calls to Formconvert.convertField(). This method shows how long each field took to compute its value.

Tuning Rules

You use Identity Manager rules to encapsulate constants and XPRESS logic that can be reused in forms, workflows, and other configurable components in the product.

When writing rules, use the following guidelines (as applicable) to obtain optimal performance:

Tuning Workflows

You customize Identity Manager workflows to facilitate and automate complex business processes with various human and electronic touchpoints.

You can use the following methods to improve custom workflow performance:

Tuning WorkItems (ManualActions)

The number and size of WorkItems (indicated by ManualActions in a workflow) can affect memory and system performance significantly. By default, Identity Manager copies an entire workflow context into a WorkItem, then writes the workflow context back out after submission.

To improve performance for WorkItems and ManualActions do the following:

Tuning Database Statistics

As a database administrator, you should frequently run statistics to monitor your repository database.

Performance problems are often caused by bad or missing database table statistics. Fixing this problem improves performance for both the database and Identity Manager performance.

See the following Oracle articles for more information:

Also consider using SQL Profiles, which is another method for choosing the best query plans. You can use the SQL Advisor within Enterprise Manager to create these profiles when you identify poorly performing SQL.

Tuning Data Exporter

Data Exporter enables you to export new, changed, or deleted Identity Manager data to an external repository that is suitable for reporting or analytic work. The actual exporting of data is done in batches, where each type of data to be exported is able to specify its own export cycle. The data to be exported comes from the Identity Manager repository and, depending on the length of the export cycle and the amount of changed data, the volume of exported data can be large.

Some Identity Manager data types are queued into a special table for later export. Specifically, WorkflowActivity and ResourceAccount data is queued because this data is not persisted otherwise. Any persisted data type can also be queued if the warehouse needs to see all changes to the type, or if the type has a lifecycle that does not correspond to the export cycle, such as TaskInstance and WorkItem data.

To maximize performance, only queue and export the types of data that you require in the warehouse. Data exporting is disabled by default, but if you enable data exporting, it exports all data types. Turn off any data types that you do not need.

When the export task exports data, the task attempts to complete the export as quickly as possible, using multiple threads to achieve as much throughput as possible. Depending on the I/O speed of the Identity Manager repository and the warehouse, the export task can fully utilize the processors on the Identity Manager server, which causes any interactive performance to degrade. Ideally, the export should occur on a machine dedicated to that task or at least occur during periods when there is no interactive activity on the machine.

The export task supports the following tuning parameters:

The drain thread count is the most important throughput. If a large number of records are in the queue table, increasing the number of threads (up to 24) tends to increase throughput. However, if the queue is dominated by one type of record, fewer drain threads might actually be faster. The export task attempts to divide the queue table contents into as many sets as there are threads allocated, and to give each thread a set to drain. Note that these threads are in addition to the drain threads that are draining the other repository tables.

Tuning the General XML

You can usually optimize the general XML by using static XMLObject declarations wherever possible. For example, use:

Also, depending on the context, you might have to wrap objects instead of using the <o></o> element.

Tuning Sun Identity Manager Service Provider

You can use Identity Manager dashboard graphs to quickly assess the current system, spot abnormalities, and understand historical trends (such as concurrent users or resource operations over a time period) for SunTM Identity Manager Service Provider(Service Provider).

Note –

Service Provider does not have an Administrator interface. You use the Identity Manager Administrator interface to perform almost all administrative tasks (such as viewing dashboard graphs).

For more information about tuning Service Provider see Sun Identity Manager Service Provider 8.1 Deployment.

Tuning the Identity Manager Web Interface

When you are working with the Identity Manager Web Interface, you can optimize performance by using the OpenSPML toolkit that is co-packaged with Identity Manager.

Note –

Using the openspml.jar file from the http://openspml.org/ web site might cause memory leaks.

Tuning Initial Loads

    To improve performance during a large, initial user load, follow this procedure:

  1. Disable all Audit Events from the Identity Manager Administrator interface.

    Note –

    Audit Logging can add several records per operation, making future audit reports perform more slowly.

    1. Choose Configure -> Audit.

    2. On the Audit Configuration page, deselect the Enable auditing box and click Save.

  2. Disable the list cache by shutting down the web server or by changing the ChangeNotifier.updateSearchIntervalCount property (on the debug/Show_WSProp.jsp debug page) to 0..

    The list cache keeps lists of users in frequently accessed organizations in memory. To maintain these lists, the list cache searches for and checks all newly created users.

  3. Clear the current list cache on the debug/Clear_List_Cache.jsp page.

  4. Ensure that the workflow being used to process the users does not contain approvals.

  5. Use alternative load methods, which include:

    • Splitting the load and running the data in zones

    • Using bulk loads, which are much faster

    • Loading from a file

  6. Disable Data Exporter for the WorkflowActivity type.

Tuning Memory Requirements

You must determine your memory needs and set values in your application server’s JVM by adding maximum and minimum heap size to the Java command line. For example:

java -Xmx512M -Xms512M

To improve performance do the following:

Note –

The max.post.memory.size specifies the maximum number of bytes that a posted file (for example by using an HTML FileSelect control) might contain without being spooled to the disk. For cases where you do not have permission to write to temporary files, increase the max.post.memory.size to avoid having to spool to the disk. The default value is 8 Kbytes.

For additional information about system requirements, see the Sun Identity Manager 8.1 Release Notes.

Tuning Operating System Kernels

For information about tuning SolarisTM and Linux operating system kernels, see the “Tuning the Operating System” chapter in the Sun Java System Application Server Enterprise Edition Performance Tuning Guide.

For information about tuning Oracle operating system kernels, see the product documentation provided with your Oracle system.

Tuning Provisioner

Network latency tends to be a common cause for performance issues when dealing with view provisioning. Tracing individual resource adapters can help you determine what is causing performance problems.

To improve provisioner performance, do the following:

Tuning Reconciliation

The Reconciler is the Identity Manager component that performs reconciliation. This section suggests methods for improving Reconciler performance, including:

General Suggestions for Tuning Reconciliation

In general, you can improve Reconciler performance if you do the following:

Tuning the Reconciler Server Settings

Although the default settings are usually adequate, you can sometimes improve Reconciler performance if you adjust the following settings on the Edit Server Settings page:

During idle times, the threads stop if they have no work to do, but only down to the minimum number of threads specified. As the load increases, the Reconciler adds more threads until the maximum number of threads is reached. The Reconciler never has less than the minimum number of threads or more than the maximum.

Generally, more threads allow more concurrency. However, at some point, too many threads can put too much load on the machine or just do not provide additional benefit.

Note –

Recommending generic, optimal settings is not possible because deployments are so different. Reconciler settings must be adjusted differently for each deployment environment.

ProcedureTo Change the Reconciler Server Settings

Perform the following steps to change the Reconciler server settings:

  1. Log into the Administrator interface.

  2. Click the Configure -> Servers -> Reconciler tabs.

  3. When the Edit Server Settings page is displayed, adjust the settings as necessary.

    See Editing Default Server Settings for more information.

Tuning Reconciliation for Multiple Resources

If you are configuring reconciliation for multiple resources in Identity Manager, you have several options:

An ideal solution does not exist for this configuration because deployments are so different. You might have to mix and match these options to find an acceptable solution for your deployment.

Preparing a usage survey, based on the business reasons behind this functionality, might help you decide how to proceed.

Address these questions:

Also, remember that the reconciliation server does not have to be one of the pools that handles web traffic. You can add a server that you never interact with directly because this server exists solely for transaction processing. Having a server dedicated to transaction processing might make the first option more attractive for very large systems.

Tuning Resource Queries

Note –

Network latency tends to be a common cause of performance issues during view provisioning. Tracing individual resource adapters can help you determine what is causing performance problems.

You can improve resource query performance if you use FormUtil.getResourceObjects to implement the query.

Use one of the following methods to cache query results:

Note –

Tuning the Scheduler

The Scheduler component controls task scheduling in Identity Manager.

This section suggests methods for improving Scheduler performance, including:

General Suggestions for Tuning the Scheduler

The following TaskDefinition options determine how the Scheduler handles tasks after they are completed:

These default settings are designed to optimize memory by shortening the lifetime of finished Scheduler tasks. Unless there is a compelling reason to change these settings, use the defaults.

If you want to immediately delete tasks that completed successfully, but you also want to keep tasks containing errors long enough to debug, you can do the following:

Tuning the Scheduler Server Settings

You can sometimes improve Scheduler performance by adjusting the following settings on the Edit Server Settings page:

Note –

Recommending generic, optimal settings is not possible because deployments are so different. Scheduler settings must be adjusted differently for each deployment environment.

ProcedureTo Change the Scheduler Server Settings

  1. Log in to the Administrator interface.

  2. Click the Configure -> Servers -> Scheduler tabs.

  3. When the Edit Server Settings page is displayed, adjust the settings as necessary.

    See Editing Default Server Settings for more information.

Tuning Sessions

Identity Manager maintains a least recently used (LRU) cache of authenticated sessions for use by authenticated users. By using existing authenticated sessions, you can speed up repository access for objects and actions that require a session.

To optimize the authentication pool size, change the session.userPoolSize value in the Waveset.properties file to the maximum number of expected, concurrent user sessions on the server.

Tuning the Sun Identity Manager Gateway

The Sun Identity Manager Gateway generates a thread for each connection, and uses a different pool for each unique combination of resource type, Gateway host, and Gateway port. The Gateway checks for idle connections every five minutes. When a connection has been idle for 60 seconds, the Gateway closes and removes that connection from the pool.

When the Gateway receives a request, it does the following:

You must configure the maximum number of connections on the resource, and you must configure these connections the same way for all resources of the same type, that are using the same Gateway. For that resource type, the first connection made to the Gateway on a given host and port uses that resource’s maximum connections value.

Note –

When you change the maximum number of connections on a resource, you must start and stop the server for the change to take effect.

The following example shows how connections, requests, and Gateway threads are related.

If you set the maximum number of connections to 10 on an Active Directory resource, and you are using two Identity Manager servers, then you can have up to 20 simultaneous connections (10 from each Identity Manager server) to the Gateway for that Active Directory resource. The Gateway can have 10 simultaneous requests outstanding from each server, and the Gateway processes each request on a different thread. When the number of simultaneous requests exceeds the maximum number of Gateway connections, additional requests are queued until the Gateway completes a request and returns the connection to the pool.

Note –

Although the Gateway code is multi-threaded, this characteristic does not address the APIs or services being used by the Gateway. For Active Directory, the Gateway uses the ADSI interface provided by Microsoft. No investigation has been done to determine whether this interface handles Gateway requests in parallel.

Other methods for improving Gateway performance, include:

Tuning the Task Bar

The Administrator interface task bar displays links to previously performed provisioning tasks, which causes the interface to render more slowly when there are a large number of tasks.

To improve interface performance, remove the taskResults.jsp link from each JSP by deleting the <List>...</List> element from the UserUIConfig object.

The following example shows <List>...</List> entries within <TaskBarPages>.

Example 4–1 Modifying the UserUIConfig Object


<TaskBarPages>
  <List>
    <String>account/list.jsp</String>
    <String>account/find.jsp</String>
    <String>account/dofindexisting.jsp</String>
    <String>account/resourceReprovision.jsp></String>
    <String>task/newresults.jsp</String>
    <String>home/index.jsp</String>
  </List>
 </TaskBarPages>

Debugging Performance Issues

This section describes the different Identity Manager and third-party debugging tools you can use to debug performance issues.

The information is organized into the following sections:

Working With Identity Manager Debug Pages

Note –

Tracing affects system performance. To help ensure optimal performance, specify the minimum tracing level or turn tracing off after debugging the system.

This section provides instructions for accessing the Identity Manager Debug pages and describes how to use these pages to identify and debug Identity Manager performance issues.

See the following sections for information:

Accessing the Debug Pages

Note –

You must have the Debug, Security Administrator, or Waveset Administrator capabilities to access and execute operations from the Identity Manager Debug pages. Administrators and Configurator are assigned this capability by default.

If you do not have the Debug capability, an error message results.

ProcedureTo Access the Identity Manager Debug Pages

  1. Open a browser and log in to the Administrator interface.

  2. Type the following URL:

    http:// host:port /idm/debug

    where:

    • host is the application server on which you are running Identity Manager.

    • port is the number of the TCP port on which the server is listening.

  3. When the System Settings page displays, type the .jsp file name for the debug page you want to open.

    For example:

    http:// host:port /idm/debug/pageName.jsp

    Note –

    Some debugging utilities are not linked from the System Settings page, but you can use them to enhance your ability to gather data for product performance and usability. For a complete list of debug pages, open a command window and list the contents of the idm/debug directory.

Control Timings (callTimer.jsp)

Use the Control Timings page to collect and view call timer statistics for different methods. You can use this information to track bottlenecks to specific methods and invoked APIs. You can also use options on the Call Timings page to import or export call timer metrics.

Note –

Call timing statistics are only collected while trace is enabled.

ProcedureTo View Call Timer Statistics

  1. Open the Control Timings page, and click Start Timing & Tracing to enable trace and timing.

  2. To stop the timing, click Stop Timing & Tracing or click Stop Timing.

    The page re-displays and populates the Show Timings table with a list of methods for which statistics are available and the methods’ aggregate call timer statistics (not broken down by caller).

    This table contains the following information:

    • Method name (Click a method name to see which methods it calls)

    • Total time

    • Average time

    • Minimum time

    • Maximum time

    • Total calls

    • Total errors

  3. To clear the list, click Clear Timing.

    Note –

    You can also use the callTimer command to collect call timer data from the Console. This command is useful when you are debugging performance issues during an upgrade or in other situations where Identity Manager is not running on the application server.

Edit Trace Configuration (Show_Trace.jsp )

Use the Edit Trace Configuration page to enable and configure tracing for the Java classes provided with your Identity Manager installation.

Specifically, you can use this page to configure the following trace settings:

Host Connection Pool (Show_ConnectionPools.jsp)

If you are not using a data source, you can use the Host Connection Pool page to view connection pool statistics. These statistics include the pool version, how many connections were created, how many are active, how many connections are in the pool, how many requests were serviced from the pool, and how many connections were destroyed.

You can also use the Host Connection Pool page to view a summary of the connection pools used to manage connections to the Gateway. You can use this information to investigate low-memory conditions.

List Cache Cleared (Clear_XMLParser_Cache.jsp)

Use the List Cache Cleared page to clear recently used XML parsers from the cache and to investigate low memory conditions.

Method Timings (Show_Timings.jsp)

Use the Method Timings page to quickly detect and assess hotspots at a method level.

The following information is gathered from Identity Manager methods and displayed on the Method Timings page:

The Method Timings page also contains a table with the following links. You can click these links to view additional information.

Note –

The Clear ALL option on the Method Timings page clears all results. This option is enabled by default.

Object Size Summary (Show_Sizes.jsp)

Use the Object Size Summary page to detect problematically large objects that can affect your system.

The Object Size Summary page shows information about the size of objects (in characters) stored in the repository. These objects are listed by type, along with the total number of objects of each type, and the objects’ total combined size, average size, maximum size, and minimum size.

Click an entry in the Type column to view additional size information about that object type. For example, click Configuration to view the ID, name, and size of the largest configuration objects in the repository.

You can also access this size information from the Console command line.

ProcedureTo Access Object Size Information from the Command Line

  1. Open the console.

  2. At the command prompt, type:

    showSizes [ type[limit ]]

    Note –

    For upgrades, existing objects will report a size of 0 until they have been updated or otherwise refreshed.

Provisioning Threads for Administrator Configurator (Show_Provisioning.jsp)

Use the Provisioning Threads for Administrator Configurator to view a summary of the provisioning threads in use by the system. This summary is a subset of the information available in Show_Threads.jsp.

Note –

Looking at just a single thread dump can be misleading.

System Cache Summary (Show_CacheSummary.jsp)

Use the System Cache Summary page to view information about the following items to help you investigate low-memory conditions:

System Memory Summary (Show_Memory.jsp)

Use the System Memory Summary page to view how much total and free memory you have available in Mbytes. When you are using memory-intensive functionality such as Reconciliation, this information can help you determine whether there is sufficient memory allocated to the JVM.

You can also use this page to launch garbage collection or to clear unused memory in the JVM for investigating heap usage.

System Properties (SysInfo.jsp)

The System Properties page provides information about your environment, including software versions, paths and environmental variables.

System Threads (Show_Threads.jsp)

Use the System Threads page to see which processes are running so you can verify that automated processes (such as reconciliation or Active Sync) are running.

This page includes information about the process type, process name, its priority, if the process is a daemon, and if the process is still running.

Note –

Looking at just a single thread dump can be misleading.

User Session Pool Cleared (Clear_User_Cache.jsp)

Use the Session Pool Cleared page to clear all of the cached sessions for users who have recently logged in and to investigate low memory conditions.

Waveset Properties (Show_WSProp.jsp)

Use the Waveset Properties page to view and temporarily edit properties in the Waveset.properties file. You can test different property settings for a particular server on which the Waveset.properties file resides without having to restart the server to pick up the changes. The edited property settings only remain in effect until the next time you restart the server.

XML Resource Adapter Caches Flushed and Cleared (Clear_XMLResourceAdapter_Cache.jsp)

Use the XML Resource Adapter Caches Flushed and Cleared page to clear test XML resource adapters from the cache and to investigate low memory conditions.

Working With Other Debugging Tools

You can use the following Sun Microsystems’ and third-party tools to identify potential performance bottlenecks:

These tools can be particularly useful if your deployment uses custom Java classes.

Identity Manager Profiler

Identity Manager provides a Profiler utility to help you troubleshoot performance problems in your deployment.

Customized forms, Java, rules, workflows, and XPRESS can cause performance and scale problems. The Profiler profiles how much time is spent in these different areas, enabling you to determine whether these forms, Java, rules, workflows, or XPRESS objects are contributing to performance and scale problems and, if so, which parts of these objects are causing the problems.

Note –

For more information about Profiler, see Working with the Identity Manager Profiler in Sun Identity Manager 8.1 Release Notes.

Using DTrace

The DTrace facility is a dynamic tracing framework for the Solaris 10 operating system that enables you to monitor JVM activity.

DTrace contains more than 30,000 probes and uses integrated user-level and kernel-level tracing to give you a view into your production system. You can also trace arbitrary data and expressions by using the D language, which is similar to C or awk. The DTrace facility also includes special support for monitoring the JVM, and enables you to watch your whole system and span outside the JVM.

DTrace is easiest to use with Java 6 because probes are built into the JVM. The facility also works with Java 1.4 and Java 5, but you must download JVM PI or JVM TI agents from the following URL:

https://solaris10-dtrace-vm-agents.dev.java.net/

The following example shows how to write a DTrace script.

Example 4–2 Example DTrace Script


#!/usr/sbin/dtrace -Zs 
#pragma D option quiet
hotspot$1::: 
{
  printf("%s\n", probename); 
}

In this example, you would replace $1 with the first argument to the script, which is the PID of the Java process you want to monitor. For example:

# ./all-jvm-probes.d 1234

The following table describes the commands you can use to enable different DTrace probes.

Table 4–3 DTrace Commands

Command 

Description 

-XX:+DTraceMonitorProbes

Enables JVM support in Java 6 (patches for Java 1.4 and Java 5) 

-XX:+ExtendedDTraceProbes

Provides the following information: 

  • JVM startup (begin and end) and shutdown

  • Thread starting and stopping

  • Class loading and unloading

  • Garbage collection (several options available)

  • JIT compilation begin and end

  • Compiled method loading and unloading

  • Monitor contention, wait, and notify

  • Method entry, method return, and object allocation

/usr/sbin/dtrace -n ’hotspot*:::’

Enables all JVM probes for all Java processes on the system 

/usr/sbin/dtrace -n ’hotspot1234:::’

Enables all JVM probes for only the Java process with PID 1234

/usr/sbin/dtrace -n ’hotspot1234:::gc-begin’

Enables only the probe that starts when garbage collection for process 1234 begins

Note –

Because DTrace causes additional work in the system, enabling this facility affects system performance. The effect is often negligible, but can become substantial if you enable many probes with costly enablings.

Instructions for minimizing the performance effect of DTrace are provided in the “Performance Considerations” chapter of the Solaris Dynamic Tracing Guide.

For more information about DTrace, see /usr/demo/dtrace and man dtrace.

Using JMX

Identity Manager enables you to use Java Management Extensions (JMX) to capture and expose operational statistics for certain resource adapter operations. You can use this data for diagnostic and predictive purposes, such as to monitor system health and reports.

This statistical data includes the following:

Objects 

Actions Monitored 

For Accounts 

  • Create

  • Update

  • Delete

  • Get

  • Authenticate

For Actions 

Run 

For Other Objects 

  • Create

  • Update

  • Delete

  • Get

  • List

JMX creates MBeans for each resource adapter, by server, and registers these beans with names that match the following pattern:


serverName=server name, resourceAdapterType=Resource Adapter Type,
resourceAdapterName=Resource Adapter Name

Identity Manager records statistics for all completed operations, whether they completed successfully or with errors. However, Identity Manager does not record statistics for incomplete operations, such as any operations that throw exceptions.

You can configure excludes as follows:

  1. From the Administrator interface, select Configure -> Servers.

  2. On the Configure Servers page, perform one of the following tasks:

    • Click the Edit Default Server Settings button to edit the default server settings.

    • Click a server link to edit the policy for that server.

  3. Click the JMX tab and enable the JMX Enable Resource Adapter Monitor box to turn on resource monitoring.

    • To exclude specific resources, add regular expressions to the JMX Resource Adapter Monitor Excludes list.

    • To exclude monitoring specific actions, add regular expressions to the JMX Resource Adapter Monitor Operation Excludes list.

All excludes use regular expressions. For excluding certain resources, JMX just matches on the resource name. For example, if you have adapters named


resource1
resource2
resource3
resource10
resource11

and you specify the following pattern


.*1$

which means, match any 0 or more of any character (.*) until something that ends with a 1 (1$). JMX will exclude resource1 and resource11.

For operations, the process is similar. If your operations have the following names, the patterns must match against those names.


ACCOUNT_CREATE
ACCOUNT_UPDATE
ACCOUNT_DELETE
ACCOUNT_GET
ACCOUNT_AUTHENTICATE
OBJECT_CREATE
OBJECT_UPDATE
OBJECT_DELETE
OBJECT_GET
OBJECT_LIST
ACTION_RUN

For example, the ^ACCOUNT.* pattern excludes all operations that start with ACCOUNT. Or, using this pattern excludes updates and deletes:


.*UPDATE$
.*DELETE$
Note –

For more information about configuring and using JMX, see Configuring JMX Monitoring and The JMX Publisher Type in Sun Identity Manager 8.1 Business Administrator’s Guide.

Using JConsole

The Java Monitoring and Management Console (JConsole) is a Java Management Extension (JMX) technology-compliant graphical management tool that is co-packaged with at least JDK 5. JConsole connects to a running JVM and gathers information from the JVM MBeans in the connected JMX agent.

Specifically, you can use JConsole to perform the following tasks:

Note –

For more information about using JConsole to monitor applications on the Java platform, see the Sun Developer Network (SDN) article titled Using JConsole to Monitor Applications, which is available from the following URL:

http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html

Identity Manager supplies some JMX MBeans that provide information about the following:

Using JRat

You can use the Java Runtime Analysis Toolkit (JRat), an open-source performance profiler for the Java platform, to identify potential performance bottlenecks, especially if your deployment uses custom Java classes. JRat monitors your application’s execution and persists the application’s performance measurements.

For example, if you have a custom workflow for provisioning, you can use JRat to see which classes are being invoked and how much time is required to run your workflow compared to the default Identity Manager provisioning workflow.

For more information about JRat, see http://jrat.sourceforge.net.