Suggestions for optimizing Waveset’s performance are organized into the following areas:
In general, you can optimize Waveset performance if you do the following:
Turn off tracing (such as Java class, userform, and workflow tracing). Tracing can add substantial overhead.
Run the Waveset built-in Audit Log Maintenance Task and System Log Maintenance Task to configure log record expirations. Log records can grow without bound, so use these tasks to prevent the repository database from running out of space. For information, see the Oracle Waveset 8.1.1 Business Administrator’s Guide.
Check the README file in Waveset updates (formerly called service packs or installation packs) to see if any performance improvements have been made to the product. If so, schedule an upgrade.
Consider the performance impact when fetching data from one or more remote systems, including the Waveset repository.
Increase the number of application server instances running Waveset, either on the same server or by adding servers, and use a load-balancing tool to distribute the requests between instances.
Keep the size of files referenced in a binary attribute as small as possible. Loading extremely large graphics files, for example, can decrease Waveset performance.
Write robust and readable XML code that minimizes duplication (for example, refactored), that uses memory efficiently, and that mitigates the impact to overall system performance.
Configure Waveset system monitoring to track events in real time.
You can view these events in dashboard graphs to quickly assess system resources, spot abnormalities, understand historical performance trends (based on the time of day, the day of week, and so forth), and interactively isolate problems before looking at audit logs. Dashboards do not provide as much detail as audit logs, but can provide hints about where to look for problems in the logs.
For more information about dashboards, see Chapter 8, Reporting, in Oracle Waveset 8.1.1 Business Administrator’s Guide.
Because synchronization is a background task, how you configure an Active Sync adapter can affect server performance.
Use the Resources list to manage Active Sync adapters. Choose an Active Sync adapter and access start, stop, and status refresh control actions from the Synchronization section of the Resource Actions list.
To improve Active Sync adapter performance, do the following:
Evaluate and adjust polling intervals based on the type of activity being performed.
The polling interval determines when the Active Sync adapter will start processing new information. For example, if the adapter reads in a large list of users from a database and updates these users in Waveset each time, you could run this process in the early morning every day. Some adapters have a quick search for new items to process and can be set to run every minute.
Edit the synchronization file for the resource to specify the host where the adapters will run.
You can configure Active Sync adapters that require more memory and CPU cycles to run on dedicated servers to help load balance the systems.
If you have the appropriate administrator capability, you can change Active Sync resources to disable, manually start, or automatically start Active Sync adapters.
When you set an adapter to automatic, the adapter restarts when the application server starts. When you start an adapter, it runs immediately and executes at the specified polling interval. When you stop an adapter, it stops the next time the adapter checks for the stop flag.
Adjust the level of detail captured by the synchronization logs.
Synchronization logs capture information about the resource that is currently processing. Each resource has its own log file, path, and log level. The amount of detail captured by the adapter log depends on the specified logging level. You specify these values in the Logging section of the Synchronization Policy for the appropriate user type (Waveset or Service Provider).
Waveset provides two different queries that you can use to determine which Waveset users have Administrative rights. One of these queries can be slow to execute when there are lots of users in the Oracle Waveset repository. Waveset executes this slow query during AdminCache initialization, which occurs during Waveset startup.
Waveset now uses the faster query by default; resulting in much faster server start-up times for installations with large numbers of users in the repository. However, under certain conditions the faster query may not produce the correct results. If you upgraded your installation from version 5.0, some Administrative users may not have the attribute used by the new query correctly set.
Refresh all Administrative users by importing the following XML, which sets the new query attribute properly:
<ImportCommand type='refreshType' targetType='User'> <List> <AttributeCondition attrName='MemberAdminGroups' operator='isPresent' operand=''/> <AttributeCondition attrName='ControlledObjectGroups' operator='isPresent' operand=''/> <AttributeCondition attrName='AdminRoles' operator='notPresent' operand=''/> </List> </ImportCommand> <ImportCommand type='refreshType' targetType='User'> <List> <AttributeCondition attrName='AdminRoles' operator='isPresent' operand=''/> </List> </ImportCommand> |
Disable the new query, and continue to use the old query by adding the following line to Waveset.properties on each Waveset server: admincache.fastinit=false.
To improve performance during bulk load operations, do the following:
Simplify default workflows to improve processing time (especially for bulk processing actions such as Active Sync, bulk actions, and reconciliation) by removing the callout to the Approval subprocess.
Keep user forms that are assigned to administrators as simple as possible. For example:
When creating a form for data loading, remove any code that is designed to display data.
When using bulk add actions, be sure that your CSV file defines basic attributes such as firstname and lastname. You can then remove these attributes from the administrator user form.
Do not modify the default forms provided with Waveset. Instead, make a copy of the form, give the copy a unique name, and modify the renamed copy. This approach prevents your customized forms from being overwritten during upgrades and product updates.
See Chapter 2, Waveset Forms, in Oracle Waveset 8.1.1 Deployment Reference for more information about creating and editing forms.
Implement the following features in deployment environments where you have NIS (Network Information Service) implemented:
Add an account attribute named user_make_nis to the schema map and use this attribute in your reconciliation or other bulk provisioning workflow. Specifying this attribute causes the system to bypass the step of connecting to the NIS database after each user update on the resource.
To write the changes to the NIS database after provisioning has completed, create a ResourceAction named NIS_password_make in the workflow.
Configurable XML objects offer a broad spectrum of user interface specifications that enable you to define how data is presented to users for different tasks and to automate complex business processes. However, this same flexibility can affect efficiency, performance, and reliability.
This section describes some guidelines for tuning Waveset’s configurable XML objects, which consist of forms, rules, and workflows. The information is organized into the following sections:
You can use Waveset forms to define interfaces to interact with views or variable contexts in an executing task. Forms also provide an execution context for business and transformation logic on a set of data elements. Although you can create very powerful, dynamic forms that perform a variety of tasks, reducing the complexity of forms increases efficiency.
The following sections describe some methods for improving the performance of your customized forms:
When designing new Waveset forms, system integrators can optimize a form’s performance by doing the following:
Performing “expensive” queries only one time, wherever possible. To minimize these queries,
Use <Field> <Default> elements to execute and store query results.
Use field names to reference values in later fields.
For custom tasks
Calculate the value in the task before a ManualAction, then store that value in a task variable.
Use variables.tmpVar to reference variables in the form.
Use <setvar name=’tempVar’/> to clear the variable after a ManualAction.
Using <defvar> for calculations that are performed for the initial display and with each refresh.
To improve the performance of administrator forms, do the following:
Specify TargetResources that only fetch specific resources for editing. (See Tuning Workflows for more information.)
Use cacheList and cacheTimeout caching parameters for objects that change infrequently if you are working with FormUtil.getResourceObjects or FormUtil.listResourceObjects.
Store the results of time-consuming calculations and fetches in <Field> elements and evaluate in the <Default> expression to help ensure that an operation occurs only one time.
Use update.constraints to limit which resources are fetched at runtime (see Dynamic Tabbed User Form in Oracle Waveset 8.1.1 Deployment Reference).
Use background approval (ManualAction with different owners and one-second timeouts) for faster page submissions.
Be aware that Waveset refreshes all fields defined on all panels of a Tab Panel Form when the page reloads, regardless of which panel is selected.
To improve the performance of end-user forms, do the following:
Use TargetResources to limit view checkouts to just those resource accounts of interest, which reduces fetch time for view and the memory consumed by TaskInstance and WorkItems.
Consider using Session.getObject(Type, name) to return a WSUser if just the view properties and attributes of the Waveset user object are of interest (useful for managing multiple deferred task triggers).
Be aware that end-user tasks typically have more WorkItems than Provisioning tasks, so end user tasks are especially susceptible to WorkItem size.
Consider using temporary generic objects for “view” editing that is constructed on view check-out then merged back into a full view for check-in.
Consider using scalable forms instead of the default Create and Edit User interfaces.
When you use the default User forms to edit a user, Waveset fetches the resources owned by that user the moment you start editing the user’s account. In deployment environments where users have accounts on many resources, this potentially time-intensive operation can result in performance degradation.
Some activities performed in forms call resources that are external to Waveset. Accessing these resources can affect Waveset performance, especially if the results contain long lists of values, such as compiling a list of groups or email distribution lists.
To improve performance during these calls, follow the guidelines in “Using a Java Class to Obtain Field Data” in Oracle Waveset 8.1.1 Deployment Reference.
Also, avoid using JavaScript in performance-critical expressions such as <Disable> expressions. Short XPRESS expressions are easier to debug if you use the built-in tracing facilities. Use JavaScript for complex logic in workflow actions.
If a form is slow to display, you can use the debug/Show_Timings.jsp page to determine the problem. Look for calls to Formconvert.convertField(). This method shows how long each field took to compute its value.
You can use the FormConverter JMX MBean to identify specific fields in a form that are slow to compute or render to HTML.
You use Waveset rules to encapsulate constants and XPRESS logic that can be reused in forms, workflows, and other configurable components in the product.
When writing rules, use the following guidelines (as applicable) to obtain optimal performance:
Use static declarations to return a constant value.
Use defvar methods to implement algorithms with temporary values for incremented values or for values that are referenced only one time.
Use putmap, setlist, or setvar methods for complex or expensive calculations whose value must be returned multiple times. Be sure to eventually set the value to <null>.
You can use the Rule JMX MBean to identify rules that are executing slowly.
You customize Waveset workflows to facilitate and automate complex business processes with various human and electronic touchpoints.
You can use the following methods to improve custom workflow performance:
Simplify default workflows to improve processing time (especially for bulk processing actions such as Active Sync, bulk actions, and reconciliation) by removing the callout to the Approval subprocess.
Ensure that no infinite loops exist in your workflows. In particular, be sure that break flags are updated and properly checked in the loops that exist in approval subprocesses.
Put fetched objects into a variable for use later if you must contact the repository for the same object multiple times.
Using a variable is necessary because Waveset does not cache all objects.
Specify TargetResources options in WorkflowServers checkoutView to restrict the number of resources that are queried for account information.
The following example shows how to restrict the number of resources being queried for account information.
<Argument name=’TargetResources’> <list> <string>resource name[| #]</string> </list> </Argument> |
In the preceding example, [| #] is an optional parameter that you can use when more than one account exists on a particular resource. In most cases, the resource name is sufficient.
Clear unnecessary view variables left by forms, especially large maps and lists. For example:
<setvar name=’myLargeList’></null></setvar>
The view is copied multiple times in a TaskInstance object, so large views greatly increase the size of each TaskInstance and corresponding TaskResult.
Use resultLimit (in seconds) in the TaskDefinition, or set this option during task execution to quickly dispose of completed tasks. Large numbers of TaskInstances impact the following:
How taskResults.jsp in footers and some JSP tasks in the Administrator interface are displayed
How JSP tasks are displayed
Querying each TaskInstance for task renaming
Database size
Set the following options as needed:
(Preferred selection) delete — Causes an older TaskInstance of the same name to be deleted before the new task begins execution.
wait — Suspends the current TaskInstance until the older TaskInstance is deleted or expired due to reaching its resultLimit.
rename — Inserts a time stamp into the TaskInstance name to avoid naming collisions.
terminate — Deletes an older TaskInstance of the same name. Each currently executing TaskInstance of the same name is terminated.
Using the SaveOnlyOnError attribute (see the following description) with the resultLimit attribute can significantly improve performance.
Use the SaveOnlyOnError attribute in TaskDefinitions and TaskInstances to configure a task that is persisted only when errors or warnings occur.
If you set SaveOnlyOnError to true, and the resultLimit value is non-zero, Waveset persists the task if there is an error or a warning. If the resultLimit value is 0, Waveset will not persist the task, even if an error or warning occurs.
If you set SaveOnlyOnError to false, Waveset just considers the resultLimit value. (The default value is false.)
The number and size of WorkItems (indicated by ManualActions in a workflow) can affect memory and system performance significantly. By default, Waveset copies an entire workflow context into a WorkItem, then writes the workflow context back out after submission.
To improve performance for WorkItems and ManualActions do the following:
Reduce the size of WorkItems.
By default, ManualAction creates a WorkItem, then copies each variable in the task context into WorkItem.variables. Limiting task context variables prevents overwrites from parallel approvals.
Use ExposedVariables to limit which variables are copied back into WorkItem. For example:
<ExposedVariables><List><String>user ...
Use EditableVariables to limit the variables assimilated back into the executing task from WorkItem. For example:
<EditableVariables><List><String>user ...
Remember to include an approval flag, a form button value, and the actual approver’s name.
Change the confirmation page and background processing to improve user interface response time.
Create a confirmation ManualAction or background ManualAction, owned by another user such as Configurator.
Set timeout=’-5’ (5 seconds) and ignoreTimeout=’true’ to prevent an error message if a user submits an action after the task is executed and the WorkItems are deleted.
Optimize memory use by setting large attribute values, such as value maps and lists, to null on submission or instantiate them as Activity-scoped variables that quickly pass out of scope.
Shorten the lifetime of finished tasks.
Prevent dead-end tasks by ensuring that each WorkItem specifies a Timeout and that the workflow anticipates a Timeout for each WorkItem.
Consider using the resultLimit and resultOption options in the TaskDefinition to determine how the Scheduler handles a task after the task completes.
Use resultLimit to control how many seconds a task is allowed to live after the task has completed. The default is zero (0), which means that the task instance will be deleted immediately after task completion.
Use resultOption to specify what action to take when repeated instances of a task are started (such as wait, delete, rename, or terminate). The default is delete.
If you want to immediately delete tasks that complete successfully, but you also want to keep tasks containing errors long enough to debug, you can conditionally delete finished tasks.
Set a resultLimit in the TaskDefinition to a sufficient time period to debug issues. You can set resultLimit to zero (or a small value) if no errors are reported at runtime (such as WF_ACTION_ERROR is <null/>) after a WorkflowServices call.
Evaluate and fix poorly scoped variables. Scope variables according to where they are declared, as follows:
Global variables are values that must be used across many activities (such as the case owner, view) and as approval flags in subprocesses.
If a variable is declared as an element of <TaskDefinition>, scope the variable globally. If a variable is declared external, its value is resolved up the call stack.
Activity variables of expensive values (such as those variables that require a resource fetch or that store a large list or map of values) can be referenced in a WorkItem.
If a variable is declared as an element of <Activity>, ensure that the variable is visible to actions and transition elements in Activity.
Beginning with Waveset Version 2005Q3M1 SP1, use <Activity> variable values in Forms, rather than in workflows, to avoid copying values on WorkItem creates.
Activity variables are values used in transition logic.
If a variable is declared as an element of <Action>, the variable should pass out of scope on completion of the action. Action variables are typically used in WorkflowApplication invocations to “change” the names of variables set by the application (such as View -> User).
Do not specify synchronous execution (syncExec='true') for the last page in a wizard workflow.
If set to true, the task will run to completion when the user executes the task. The interface will hang until the task completes or encounters another stopping point (such as another ManualAction).
Remove unnecessary approval checks.
For Active Sync, use a streamlined provisioning task in place of the system-specified provisioning task specified by viewOptions.Process.
Do not modify the provisioning tasks provided with Waveset.
You must create a new task, then identify that task in the form and in the process mappings configuration (unless the task is not listed).
Sometimes automating a process requires a wizard. A wizard is a multi-step GUI operation where each step presents the user with a page used to capture or display data. Waveset provides two techniques for building a wizard:
Form-Based wizards. These wizards use form processing to change what the user sees, allowing sets of fields to be visible or invisible based on data held in the view.
Workflow-Based wizards. These wizards require task execution and suspension to provide page transitions.
A Form-Based wizard is four-times more efficient than the best Workflow-Based wizard. Workflow-Based wizards can vary by up to 10x in performance, depending on how they are constructed. Wizard efficiency is typically not noticeable until many wizards are run concurrently.
Because view processing does not necessarily require object repository access, a single Waveset server can process many Form-Based wizards concurrently without contention. However, Form-Based wizards are limited in what they can do between steps. The only processing these wizards can perform between steps is done by normal form or view processing. Form derivation and expansion, which are done when the view is being refreshed, is more limiting than the processing that is possible in a workflow. If a Form-Based wizard's processing limits preclude its use, you must use a Workflow-Based wizard. However, you should consider using a Form-Based wizard first.
Because Workflow-Based wizards require task execution and suspension to provide page transitions, and because each page in a Workflow-Based wizard corresponds to a workflow ManualAction object, these wizards are less efficient than Form-Based wizards. A problem occurs when the wizard has hundreds of concurrent invocations because all of the task start, suspend, and resume operations must contend for the object repository task table.
The object repository is being accessed synchronously with each HTTP request; consequently, a Workflow-Based wizard does not scale to large numbers of concurrent executions due to repository contention. Each start, suspend, and resume operation involves several reads and writes to the task table in the repository, and under a large concurrent load results in the page-to-page response of the GUI slowing down. Unlike a Form-Based wizard that scales by adding more Waveset servers and balancing the HTTP load between them, Workflow-Based wizards slow down due to repository contention. Adding more Waveset servers actually makes the problem worse because the repository is shared between all servers.
If you must use a Workflow-Based wizard because you need processing between page transitions, consider using transient ManualAction. A ManualAction in a Workflow-Based wizard is the mechanism used to display pages to the user. A five-step wizard typically has at least five ManualActions.
If you construct the wizard so that the user must either complete or abort the entire flow, then you can mark the ManualActions with the transient='true' attribute. Adding this attribute allows Waveset to bypass the repository instead of keeping the task in memory and sequencing it without accessing the object repository. This construction decreases the load the wizard puts on the repository, allowing it to scale to higher concurrency loads. However, using this attribute has some drawbacks. If you set transient to true (it is false by default), you cannot restart the workflow after the task enters the transient section because the in-memory state of the workflow does not match the repository state. Also, each HTTP request for the wizard must use the same server because the true state of the wizard is now only kept in memory on the initiating server. As soon as Waveset encounters a ManualAction with transient=false, Waveset writes the workflow to the repository normally, and normal workflow behavior resumes.
Consider using a Workflow-Based wizard with the following structure:
Begin Activity 1 ManualAction 1 (transient = true) Activity 2 ManualAction 2 (transient = true) ManualAction 3 (transient = true) ManualAction 4 (transient = true) ManualAction 5 (transient = false) Activity 3 ... |
When launched, this wizard creates a task that is initially stored in the object repository. All processing between ManualAction 1 and ManualAction 5 is done without any further repository work (the transient section). When Waveset executes ManualAction 5, the task is again stored in the repository (normal behavior), which is a significant performance savings because each normal suspend/resume for a ManualAction does the following:
Creates and stores a WorkItem
Saves the task state
Reads/writes the WorkItem
Locks the task
Resumes the task
Deletes the WorkItem
Each suspend/resume pair (when in wizard mode) results in more than 20 repository read/writes, all on the task table. The state of the task in the repository is running, so if the server crashes or shuts down during the execution of the wizard, the task will be deleted from the repository within a few minutes by another server or when the crashed server restarts, whichever comes first.
If you use a transient ManualAction, you can observe the effects by looking at the JMX TaskInstanceCache and WorkItemCache MBeans. These MBeans show the number of Store (repository write) operations compared to Cache (memory only) operations occurring. Each Cache operation means a Store was avoided, thus reducing object repository contention.
As a database administrator, you should frequently run statistics to monitor your repository database.
Performance problems are often caused by bad or missing database table statistics. Fixing this problem improves performance for both the database and Waveset performance.
See the following Oracle articles for more information:
Also consider using SQL Profiles, which is another method for choosing the best query plans. You can use the SQL Advisor within Enterprise Manager to create these profiles when you identify poorly performing SQL.
Data Exporter enables you to export new, changed, or deleted Waveset data to an external repository that is suitable for reporting or analytic work. The actual exporting of data is done in batches, where each type of data to be exported is able to specify its own export cycle. The data to be exported comes from the Waveset repository and, depending on the length of the export cycle and the amount of changed data, the volume of exported data can be large.
Some Waveset data types are queued into a special table for later export. Specifically, WorkflowActivity and ResourceAccount data is queued because this data is not persisted otherwise. Any persisted data type can also be queued if the warehouse needs to see all changes to the type, or if the type has a lifecycle that does not correspond to the export cycle, such as TaskInstance and WorkItem data.
To maximize performance, only queue and export the types of data that you require in the warehouse. Data exporting is disabled by default, but if you enable data exporting, it exports all data types. Turn off any data types that you do not need.
When the export task exports data, the task attempts to complete the export as quickly as possible, using multiple threads to achieve as much throughput as possible. Depending on the I/O speed of the Waveset repository and the warehouse, the export task can fully utilize the processors on the Waveset server, which causes any interactive performance to degrade. Ideally, the export should occur on a machine dedicated to that task or at least occur during periods when there is no interactive activity on the machine.
The export task supports the following tuning parameters:
Queue read block size
Queue write block size
Queue drain thread count
The drain thread count is the most important throughput. If a large number of records are in the queue table, increasing the number of threads (up to 24) tends to increase throughput. However, if the queue is dominated by one type of record, fewer drain threads might actually be faster. The export task attempts to divide the queue table contents into as many sets as there are threads allocated, and to give each thread a set to drain. Note that these threads are in addition to the drain threads that are draining the other repository tables.
You can usually optimize the general XML by using static XMLObject declarations wherever possible. For example, use:
<List> instead of <list>
<String> instead of <s>
<Map><MapEntry ...></Map> instead of <map>
Also, depending on the context, you might have to wrap objects instead of using the <o></o> element.
You can use Waveset dashboard graphs to quickly assess the current system, spot abnormalities, and understand historical trends (such as concurrent users or resource operations over a time period) for Oracle Waveset Service Provider (Service Provider).
Service Provider does not have an Administrator interface. You use the Waveset Administrator interface to perform almost all administrative tasks (such as viewing dashboard graphs).
For more information about tuning Service Provider see Oracle Waveset Service Provider 8.1.1 Deployment.
When you are working with the Waveset Web Interface, you can optimize performance by using the OpenSPML toolkit that is co-packaged with Waveset.
Using the openspml.jar file from the http://openspml.org/ web site might cause memory leaks.
To improve performance during a large, initial user load, follow this procedure:
Disable all Audit Events from the Waveset Administrator interface.
Audit Logging can add several records per operation, making future audit reports perform more slowly.
Choose Configure -> Audit.
On the Audit Configuration page, deselect the Enable auditing box and click Save.
Disable the list cache by shutting down the web server or by changing the ChangeNotifier.updateSearchIntervalCount property (on the debug/Show_WSProp.jsp debug page) to 0.
The list cache keeps lists of users in frequently accessed organizations in memory. To maintain these lists, the list cache searches for and checks all newly created users.
Clear the current list cache on the debug/Clear_List_Cache.jsp page.
Ensure that the workflow being used to process the users does not contain approvals.
Use alternative load methods, which include:
Splitting the load and running the data in zones
Using bulk loads, which are much faster
Loading from a file
Disable Data Exporter for the WorkflowActivity type.
You must determine your memory needs and set values in your application server’s JVM by adding maximum and minimum heap size to the Java command line. For example:
java -Xmx512M -Xms512M
To improve performance do the following:
Set the maximum and minimum heap size values to the same size.
Depending on your specific implementation, you might want to increase these values if you run reconciliation.
For performance tuning purposes, you may also set the following in the waveset.property file:
max.post.memory.size value
The max.post.memory.size specifies the maximum number of bytes that a posted file (for example by using an HTML FileSelect control) might contain without being spooled to the disk. For cases where you do not have permission to write to temporary files, increase the max.post.memory.size to avoid having to spool to the disk. The default value is 8 Kbytes.
For additional information about system requirements, see the Oracle Waveset 8.1.1 Release Notes.
For information about tuning Solaris and Linux operating system kernels, see the “Tuning the Operating System” chapter in the Sun Java System Application Server Enterprise Edition Performance Tuning Guide.
For information about tuning Oracle operating system kernels, see the product documentation provided with your Oracle system.
Each Waveset server captures profiling data by default. You can use this data with the Waveset IDE to diagnose a large range of performance problems. However, capturing and storing this profiling data adds a measurable load to the server, which consumes both memory and CPU. In a stable production environment, disable the profiler in production servers. Enable the profiler only when you are investigating a performance problem.
For information about using the Waveset IDE, see Identity Manager IDE Frequently Asked Questions (FAQ) in Oracle Waveset 8.1.1 Release Notes.
The following example shows how to disable the Profiler from capturing data.
Import the following XML to disable the profiler.
Setting the attribute value to true disables the profiler, and setting it to false enables the profiler.
Use the lh import command to import the following XML
<?xml version='1.0' encoding='UTF-8'?> <!DOCTYPE Waveset PUBLIC 'waveset.dtd' 'waveset.dtd'> <Waveset> <ImportCommand class='com.waveset.session.SystemConfigurationUpdater' > <Object> <Attribute name='server.default.disableProfiling'> <Boolean>true</Boolean> </Attribute> </Object> </ImportCommand> </Waveset> |
Restart all Waveset servers.
Network latency tends to be a common cause for performance issues when dealing with view provisioning. Tracing individual resource adapters can help you determine what is causing performance problems.
To improve provisioner performance, do the following:
Set the provisioner.maxThreads parameter in the Waveset.properties file to limit the maximum number of threads that are started to perform parallel resource provisioning operations each time a user is created, modified, or deleted.
The default value is 10, which generally provides optimal performance. Specifying a value greater than 20 significantly degrades the provisioner’s performance.
For example, if a user has 15 resource accounts, a maximum of ten provisioner threads are started to simultaneously perform resource provisioning operations on ten resource accounts. The remaining five resource accounts will not be modified until the first ten threads have completed resource provisioning operations.
In a different example, if you are modifying two users, and each user has two resource accounts, four provisioner threads might be used to update those resources.
Increasing the provisioner.maxThreads value can help throughput when users have many resource accounts. However, specifying a very large value might create many provisioner threads, which can degrade performance on the server as a whole.
Configure quota settings in the Waveset.properties file to control the number of concurrent operations (such as reprovisioning) a user can execute for a specific task. Increasing the number of concurrent actions can help more operations complete faster, but trying to process too many actions at once might cause bottlenecks.
You can create configuration sets on a per-pool basis. For example, if you create configuration A, configuration B, and configuration C, when you create a TaskDefinition (workflow), you can assign a specific pool configuration to the workflow from the configurations that you defined.
The following example shows the quota settings that limit user bob to running one reprovisioning task at a time:
Quota.poolNames=ReProvision,Provision Quota.pool.ReProvision.defaultLimit=1 Quota.pool.ReProvision.unlimitedItems=Configurator Quota.pool.ReProvision.items=bob,jan,ted Quota.pool.ReProvision.item.bob.limit=1 |
To enforce the task quota, reference poolName in a TaskDefinition. The format is as follows:
<TaskDefinition ... quotaName=’{poolName}’..>
Most users start only one task at a time. For proxy administrators who perform reconciliation or Active Sync tasks, set the task quota higher.
Avoid using the Configurator user for reconciliation and Active Sync tasks. The Configurator has access to unlimited tasks and can monopolize available resources, which adversely affects concurrent processes.
The Reconciler is the Waveset component that performs reconciliation. This section suggests methods for improving Reconciler performance, including:
In general, you can improve Reconciler performance if you do the following:
Avoid using the Configurator user for reconciliation tasks. The Configurator has access to unlimited tasks and can monopolize available resources, which adversely affects concurrent processes.
Instead, use a streamlined, minimal user for reconciliation and Active Sync tasks. Because the subject executing the task is serialized as part of the task, a minimal user takes less space, or overhead, for each task and update in the repository.
Use information on the Reconciler status page (debug/Show_Reconciler.jsp) to decide which settings to adjust based on queue sizes, available system resources, and performance benchmarks. Be aware that these settings are dependent on the environment.
The Reconciler JMX MBean provides much of the information in Show_Reconciler.jsp, and shows a processing rate estimate.
Use the System Memory Summary page (debug/Show_Memory.jsp) to see how much total and free memory is available. Reconciliation is a memory-intensive function, and you can use this information to determine whether there is sufficient memory allocated to the JVM. You can also use this page to launch garbage collection or to clear unused memory in the JVM for investigating heap usage.
When you assign user forms to proxy administrators who are performing reconciliations, keep the user forms as simple as possible and only use essential fields. Depending on the schema map, including a field that calculates the waveset.organization attribute is generally sufficient.
Administrators who need to view or edit the Waveset schema for Users or Roles must be in the IDM Schema Configuration AdminGroup and must have the IDM Schema Configuration capability.
Use per-account workflows judiciously. The reconciliation process does not start provisioning tasks for performance reasons by default.
If you must use a per-account workflow task, edit the reconciliation policy to limit the Reconciler’s automatic responses to events of interest only. (See the Situation area of the Edit Reconciliation Policy page.)
Reconciliation of a resource goes through two phases. In the first phase, Waveset gets a list of all users in its internal repository that are known to have accounts on the resource. This first phase does not involve the physical resource at all, and typically happens very quickly.
The second phase requests a list of all accounts from the resource, and then processes those accounts; potentially linking them to users or even creating new users. Performance of the second phase, indicated as reconciling accounts in the resource status message, is proportional to the speed of the resource and the number of worker threads. You can compensate for a slow resource by adding more worker threads; assuming the resource can handle more concurrent AccountGet operations.
There is a JMX MBean for each resource that shows the average, minimum, and maximum response times for each resource operation. Reconciliation phase two involves lots of Account_Get operations, so the average time for each Account_Get strongly influences the overall reconciliation performance. To compensate for resources with longer Account_Get times, use more worker threads. However, because the same number of worker threads are used for all resources, setting a maximum worker thread too high might overwhelm the Waveset object repository on faster resources.
Although the default settings are usually adequate, you can sometimes improve Reconciler performance if you adjust the following settings on the Edit Server Settings page:
Parallel Resource Limit. Specifies the maximum number of resource threads that the Reconciler can process in parallel.
Resource threads allocate work items to worker threads, so if you add additional resource threads, you might also have to increase the maximum number of worker threads.
Minimum Worker Threads. Specifies the number of processing threads that the Reconciler always keeps open.
Maximum Worker Threads. Specifies the maximum number of processing threads that the Reconciler can use. The Reconciler starts only as many threads as the workload requires, which places a limit on that number. Worker threads automatically close if they are idle for a short duration.
During idle times, the threads stop if they have no work to do, but only down to the minimum number of threads specified. As the load increases, the Reconciler adds more threads until the maximum number of threads is reached. The Reconciler never has less than the minimum number of threads or more than the maximum.
Generally, more threads allow more concurrency. However, at some point, too many threads can put too much load on the machine or just do not provide additional benefit. Because the worker threads are typically reading and writing User and Account objects, having too much concurrency might overload the Waveset repository RDBMS.
Recommending generic, optimal settings is not possible because deployments are so different. Reconciler settings must be adjusted differently for each deployment environment.
Perform the following steps to change the Reconciler server settings:
Log into the Administrator interface.
Click the Configure -> Servers -> Reconciler tabs.
When the Edit Server Settings page is displayed, adjust the settings as necessary.
See Editing Default Server Settings for more information.
If you are configuring reconciliation for multiple resources in Waveset, you have several options:
All of the resources on the same server, all at the same time.
This option is the most efficient from the Waveset perspective, but if you have many resources (for example more than 20), you are likely to experience Java resource issues.
All of the resources on the same server, each at a different time.
This option is easier on Java resource loading, but puts a significant burden on your schedule configuration.
Each resource on a different server, all at the same time.
This option minimizes elapsed time, but increases the number of servers.
An ideal solution does not exist for this configuration because deployments are so different. You might have to mix and match these options to find an acceptable solution for your deployment.
Preparing a usage survey, based on the business reasons behind this functionality, might help you decide how to proceed.
Address these questions:
Why are you reconciling these resources?
Do you have the same the goal for each of these resources?
Are each of these resources equally important or critical?
Must all resources be reconciled on the same schedule, or can you spread out the reconciliations?
How often must each resource be reconciled?
Also, remember that the reconciliation server does not have to be one of the pools that handles web traffic. You can add a server that you never interact with directly because this server exists solely for transaction processing. Having a server dedicated to transaction processing might make the first option more attractive for very large systems.
Network latency tends to be a common cause of performance issues during view provisioning. Tracing individual resource adapters can help you determine what is causing performance problems.
You can improve resource query performance if you use FormUtil.getResourceObjects to implement the query.
Use one of the following methods to cache query results:
getResourceObjects(Session session, String objectType, String resID, Map options, String cacheList, String cacheTimeout, String cacheIfExists)
getResourceObjects(String subjectString, String objectType, String resId, Map options, String cacheList, String cacheTimeout, String clearCacheIfExists)
Set cacheTimeout in milliseconds.
Restrict searches to specific searchContext, if applicable.
Return the minimum number of attributes in options.searchAttrsToGet.
The Scheduler component controls task scheduling in Waveset.
This section suggests methods for improving Scheduler performance, including:
The following TaskDefinition options determine how the Scheduler handles tasks after they are completed:
resultLimit — Controls how many seconds task results are kept after the task has completed. The default setting varies for different tasks. A setting of zero immediately removes tasks after completion.
resultOption — Controls what action is taken when repeated instances of a task are started. The default setting is delete, which removes previous task instances.
These default settings are designed to optimize memory by shortening the lifetime of finished Scheduler tasks. Unless there is a compelling reason to change these settings, use the defaults.
If you want to immediately delete tasks that completed successfully, but you also want to keep tasks containing errors long enough to debug, you can do the following:
Set the resultLimit to a small value, such as 3600 seconds.
Set the saveOnlyOnError value to true.
With these two settings, Waveset will only store the task results if the task has an error or warning Result Item. This configuration can improve the performance of some tasks by allowing them to bypass being stored in the repository when they complete.
You can sometimes improve Scheduler performance by adjusting the following settings on the Edit Server Settings page:
Maximum Concurrent Tasks. Specifies the maximum number of tasks that the Scheduler can run at one time.
When more tasks are ready to run than the Maximum Concurrent Tasks setting allows, the extra tasks must wait until there is room available or until they are run on another server.
If too many tasks are being swapped out of memory and sharing CPU time, the overhead slows down performance. Alternatively, setting the maximum too low results in idle time. The Scheduler checks for available tasks every minute, so a waiting task waits at least a minute before being run.
The default Maximum Concurrent Tasks setting (100) is usually adequate. You can decide whether to adjust this setting up or down based on which tasks are being run in the deployment and by profiling the runtime behavior after the deployment is otherwise complete.
Before you change this setting, you may want to observe the Scheduler using JMX. The Scheduler.executingTaskCount attribute shows how many tasks the Scheduler is running.
In some cases, you might want to suspend or disable the Scheduler. For example, if you want a server dedicated to handling the End User interface, disabling the Scheduler will prevent tasks from running on that server. The server would only serve the End User interface pages and store launched tasks for other servers to execute.
Task Restrictions. Specifies the set of tasks that can execute on the server.
The Task Restrictions setting can provide a finer granularity of control over what tasks are allowed to run on a server. You can restrict tasks individually or through the server settings.
Recommending generic, optimal settings is not possible because deployments are so different. Scheduler settings must be adjusted differently for each deployment environment.
Log in to the Administrator interface.
Click the Configure -> Servers -> Scheduler tabs.
When the Edit Server Settings page is displayed, adjust the settings as necessary.
See Editing Default Server Settings for more information.
To improve Scheduler performance when the component is under a heavy backlog, you must modify the SystemConfiguration configuration object to enable all optimizations. The Scheduler can pick up any changes to the SystemConfiguration object while the server is running.
The Waveset Scheduler is responsible for executing scheduled tasks, resuming suspended tasks, and cleaning up task results for completed tasks. The Scheduler is single-threaded, meaning it processes its work with a single control thread. However, when the Scheduler starts or resumes a task, it runs that task in a new thread and can have several tasks running at the same time.
You can directly start tasks without any Scheduler processing. For example, workflow and report tasks are often started as a result of a HTTP request. Starting, suspending, and resuming a task causes a modest amount of object repository processing. The repository can become congested if lots of concurrent HTTP requests are starting or resuming tasks. Because an application server may service hundreds of HTTP requests simultaneously, it is easy to create a large backlog for the Scheduler.
For example, if hundreds of HTTP requests started workflows that were subsequently suspended, the Scheduler would be responsible for resuming those workflows.
Because the Scheduler has a single control thread, but many different tasks to perform, it might sometimes seem like the Scheduler is not keeping up with one task or another. Waveset uses the sleepingTaskLimit and readyTaskLimit attributes to control how long the Scheduler can spend processing sleeping or ready tasks during each control loop. These limits ensure that a Scheduler, when presented with thousands of ready tasks, does not spend too much time starting these tasks and ignoring sleeping tasks.
When you have multiple Waveset servers, each server typically runs a Scheduler. By default, these Schedulers compete for the same work by polling the object repository. Configuring blockProcessing allows each Scheduler to process a different block of tasks, resulting in somewhat less object repository contention for specific records. The blockProcessing attribute is enabled by default, but all servers process the same slot. To enable cooperative (rather than competitive) task processing, you can assign each server a different slot (starting with 0). When you assign slots (other than 0) to the servers, they segment the tasks into buckets and only process the tasks in their assigned bucket.
The Scheduler has a JMX MBean that is very useful in diagnosing what is perceived as slow Scheduler performance. The ExecuteTime attribute in the MBean is often the key to understanding the Scheduler's performance. ExecuteTime is the time (in milliseconds) it took the Scheduler to start or resume the last task it processed. On a healthy system, this time should be less then 150 milliseconds. When this value starts to get large, the server is having trouble starting tasks, typically because there is congestion on the task table in the object repository, or because of internal synchronization in the Waveset code itself. Viewing the Scheduler's thread stack in the JMX console usually reveals the problem.
The ExecutingTaskCount attribute in the MBean shows how many tasks the Scheduler is currently managing. By default, the task limit is 100, which is almost always sufficient unless the tasks being executed run for a long time without suspending (such as report tasks). The ExecutingTaskCount value does not reflect all tasks running on the server. Remember that HTTP requests can also start tasks, so the total number of tasks running on the server is unknown to the Scheduler.
One of the Scheduler's many jobs is to resume tasks. Because each Waveset server can have a Scheduler running, the Schedulers also periodically look to process work that was being handled by another server that is not currently running. When a server goes into the recovered state, Schedulers on other servers attempt to process or clean up work that was being done on the recovered server. Schedulers go into a recovered state when one server observes another server has not issued a heartbeat within the last five minutes.
This check compares the timestamp of the last heartbeat message in the repository to the current server's clock. If Waveset servers have more than five minutes of system clock skew, the server with the clock that is farthest ahead marks servers with clocks that are behind as recovered. To avoid this situation, keep the system clocks on your Waveset servers synchronized.
If necessary, review Editing Waveset Configuration Objects in Oracle Waveset 8.1.1 Business Administrator’s Guide.
To control the Scheduler's behavior, add one or more of the following attributes to the SystemConfiguration configuration object. Use the serverSettings.default.scheduler.attribute path to create these attributes.
blockProcessing (Boolean). Indicates whether servers should work on independent blocks of tasks or not. Defaults to true, which means Scheduler will work on independent blocks of tasks.
blockProcessing does not apply to task deletion.
blockProcessingSize (Integer). Represents how many tasks a Scheduler will process in its block. Defaults to 50.
slots (List of Strings). Indicates slots of Waveset server instance names. Defaults to an empty list.
If the blockProcessing attribute is true, each Scheduler processes the block at the server's position in this list.
If you do not provide a list of slots, or do not include a server in the list, every Scheduler assumes it has slot 0 and each Scheduler processes the same block of tasks concurrently.
slots do not apply to task deletion.
fastResume (Boolean). Instructs the Scheduler to immediately execute tasks that are sleeping with an expired restart time. Defaults to true.
If true, the Scheduler executes sleeping tasks without first placing them in the ready state, which reduces load on the Identity Manager repository.
If false, the Scheduler first puts the task in a ready state, and then executes the task on the next Scheduler poll.
threadPriority (Boolean). Indicates the Scheduler thread should be run at an elevated priority. Defaults to false.
If true, the Scheduler threads run with an elevated priority.
If false, the Scheduler threads are only read at server startup.
sleepingTaskLimit (Integer). Indicates how many milliseconds the Scheduler can spend processing sleeping tasks before reentering the polling loop. Defaults to 60 seconds.
The Scheduler is single-threaded and does not perform any other operations when it is processing sleeping tasks. If a large number of tasks are sleeping and ready to run, the Scheduler only processes as many tasks as it can within this time limit.
readyTaskLimit (Integer). Indicates how many milliseconds the Scheduler can spend processing ready tasks before reentering the polling loop. Defaults to 60 seconds.
The Scheduler is single-threaded and does not perform any other operations when it is processing ready tasks. If a large number of tasks are ready to run, the Scheduler only processes as many tasks as it can within this time limit.
Waveset maintains a least recently used (LRU) cache of authenticated sessions for use by authenticated users. By using existing authenticated sessions, you can speed up repository access for objects and actions that require a session.
To optimize the authentication pool size, change the session.userPoolSize value in the Waveset.properties file to the maximum number of expected, concurrent user sessions on the server.
The Oracle Waveset Gateway generates a thread for each connection, and uses a different pool for each unique combination of resource type, Gateway host, and Gateway port. The Gateway checks for idle connections every five minutes. When a connection has been idle for 60 seconds, the Gateway closes and removes that connection from the pool.
When the Gateway receives a request, it does the following:
If there are no idle connections in the corresponding pool, the Gateway creates a new connection.
If an idle connection exists in the pool, the Gateway retrieves and reuses that connection.
You must configure the maximum number of connections on the resource, and you must configure these connections the same way for all resources of the same type, that are using the same Gateway. For that resource type, the first connection made to the Gateway on a given host and port uses that resource’s maximum connections value.
When you change the maximum number of connections on a resource, you must start and stop the server for the change to take effect.
The following example shows how connections, requests, and Gateway threads are related.
If you set the maximum number of connections to 10 on an Active Directory resource, and you are using two Waveset servers, then you can have up to 20 simultaneous connections (10 from each Waveset server) to the Gateway for that Active Directory resource. The Gateway can have 10 simultaneous requests outstanding from each server, and the Gateway processes each request on a different thread. When the number of simultaneous requests exceeds the maximum number of Gateway connections, additional requests are queued until the Gateway completes a request and returns the connection to the pool.
Although the Gateway code is multi-threaded, this characteristic does not address the APIs or services being used by the Gateway. For Active Directory, the Gateway uses the ADSI interface provided by Microsoft. No investigation has been done to determine whether this interface handles Gateway requests in parallel.
Other methods for improving Gateway performance, include:
Locating the Gateway near (from a network connectivity perspective) the domain controllers of the managed domain
Increasing the block size on a Gateway resource can increase throughput during reconciliation or load operations
Increased throughput results have been noted for basic reconciliations with no custom workflows and in which no attribute reconciliations are being performed. Initially, the Gateway does consume more system memory, but this memory is eventually released.
Be aware that there is a diminishing return. At some point, larger block sizes do not result in proportionately increased performance. For example, the following data shows the speed observed for a Load from Resource of 10,000 users from an Active Directory resource. Also, the peak memory usage for the Gateway process during the load is included.
Block Setting |
Users Created Per Hour |
Peak Gateway Memory Usage |
---|---|---|
100 |
500 |
20 MB |
200 |
250 |
25 MB |
500 |
9690 |
60 MB |
1000 |
10044 |
92 MB |
For Exchange Server 2007, the PowerShellExecutor performs actions for Exchange Server 2007. You can modify the following registry settings to change the behavior of the PowerShellExecutor inside the Gateway.
Both settings can have a large impact on the behavior and memory usage of the Gateway. Changes to these parameters should only be considered after careful testing.
powerShellTimeout
Content. Timeout for PowerShell actions (registry type REG_DWORD)
Default. 60000 ms (1 minute)
When the powerShellTimeout setting times out, any RunSpace actions are interrupted and canceled to prevent runaway actions in the PowerShell environment that cause the Gateway to become unresponsive.
Decreasing the powerShellTimeout value to a small value can prematurely cancel actions, and can prevent the RunSpace initialization from finishing correctly. Observed startup times for the first RunSpace in the pool range from 2—5 seconds.
The powerShellTimeout value is read-only on startup, and you cannot change it without restarting the gateway.
runSpacePoolSize
Content. Number of RunSpaces in the pool (registry type REG_DWORD)
Default. 5
Minimum. 5
Maximum. 25
The number of RunSpaces in the pool allow for parallel execution of PowerShell actions by the gateway. One provisioning action or update of a user in Exchange 2007 can result in multiple PowerShell actions being executed.
A started RunSpace can consume a large amount of memory. For the first RunSpace, the typical size is approximately 40 MB. Subsequent RunSpaces normally use between 10—20 MB.
The preceding figures can differ in specific environments and are only given as guidelines, so be careful when changing this value.
The runSpacePoolSize value is read-only on startup, and you cannot change the pool size value without restarting the Gateway.
The Administrator interface task bar displays links to previously performed provisioning tasks, which causes the interface to render more slowly when there are a large number of tasks.
To improve interface performance, remove the taskResults.jsp link from each JSP by deleting the <List>...</List> element from the UserUIConfig object.
The following example shows <List>...</List> entries within <TaskBarPages>.
<TaskBarPages> <List> <String>account/list.jsp</String> <String>account/find.jsp</String> <String>account/dofindexisting.jsp</String> <String>account/resourceReprovision.jsp></String> <String>task/newresults.jsp</String> <String>home/index.jsp</String> </List> </TaskBarPages> |