test

Oracle Waveset 8.1.1 System Administrator's Guide

Tuning Waveset Performance

Suggestions for optimizing Waveset’s performance are organized into the following areas:

General Performance Tunings

In general, you can optimize Waveset performance if you do the following:

Tuning Active Sync Adapter Performance

Because synchronization is a background task, how you configure an Active Sync adapter can affect server performance.

Use the Resources list to manage Active Sync adapters. Choose an Active Sync adapter and access start, stop, and status refresh control actions from the Synchronization section of the Resource Actions list.

To improve Active Sync adapter performance, do the following:

Tuning Admin Cache Initialization

Waveset provides two different queries that you can use to determine which Waveset users have Administrative rights. One of these queries can be slow to execute when there are lots of users in the Oracle Waveset repository. Waveset executes this slow query during AdminCache initialization, which occurs during Waveset startup.

Waveset now uses the faster query by default; resulting in much faster server start-up times for installations with large numbers of users in the repository. However, under certain conditions the faster query may not produce the correct results. If you upgraded your installation from version 5.0, some Administrative users may not have the attribute used by the new query correctly set.

Tuning Bulk Loading

To improve performance during bulk load operations, do the following:

Tuning Configurable XML Objects

Configurable XML objects offer a broad spectrum of user interface specifications that enable you to define how data is presented to users for different tasks and to automate complex business processes. However, this same flexibility can affect efficiency, performance, and reliability.

This section describes some guidelines for tuning Waveset’s configurable XML objects, which consist of forms, rules, and workflows. The information is organized into the following sections:

Tuning Forms

You can use Waveset forms to define interfaces to interact with views or variable contexts in an executing task. Forms also provide an execution context for business and transformation logic on a set of data elements. Although you can create very powerful, dynamic forms that perform a variety of tasks, reducing the complexity of forms increases efficiency.

The following sections describe some methods for improving the performance of your customized forms:

Optimizing New Forms

When designing new Waveset forms, system integrators can optimize a form’s performance by doing the following:

Optimizing Administrator Forms

To improve the performance of administrator forms, do the following:

Optimizing End-User Forms

To improve the performance of end-user forms, do the following:

Optimizing Expressions in Form Fields

Some activities performed in forms call resources that are external to Waveset. Accessing these resources can affect Waveset performance, especially if the results contain long lists of values, such as compiling a list of groups or email distribution lists.

To improve performance during these calls, follow the guidelines in “Using a Java Class to Obtain Field Data” in Oracle Waveset 8.1.1 Deployment Reference.

Also, avoid using JavaScript in performance-critical expressions such as <Disable> expressions. Short XPRESS expressions are easier to debug if you use the built-in tracing facilities. Use JavaScript for complex logic in workflow actions.

If a form is slow to display, you can use the debug/Show_Timings.jsp page to determine the problem. Look for calls to Formconvert.convertField(). This method shows how long each field took to compute its value.

Note –

You can use the FormConverter JMX MBean to identify specific fields in a form that are slow to compute or render to HTML.

Tuning Rules

You use Waveset rules to encapsulate constants and XPRESS logic that can be reused in forms, workflows, and other configurable components in the product.

When writing rules, use the following guidelines (as applicable) to obtain optimal performance:

Note –

You can use the Rule JMX MBean to identify rules that are executing slowly.

Tuning Workflows

You customize Waveset workflows to facilitate and automate complex business processes with various human and electronic touchpoints.

You can use the following methods to improve custom workflow performance:

Tuning WorkItems (ManualActions)

The number and size of WorkItems (indicated by ManualActions in a workflow) can affect memory and system performance significantly. By default, Waveset copies an entire workflow context into a WorkItem, then writes the workflow context back out after submission.

To improve performance for WorkItems and ManualActions do the following:

Working with Waveset Wizards

Sometimes automating a process requires a wizard. A wizard is a multi-step GUI operation where each step presents the user with a page used to capture or display data. Waveset provides two techniques for building a wizard:

A Form-Based wizard is four-times more efficient than the best Workflow-Based wizard. Workflow-Based wizards can vary by up to 10x in performance, depending on how they are constructed. Wizard efficiency is typically not noticeable until many wizards are run concurrently.

Because view processing does not necessarily require object repository access, a single Waveset server can process many Form-Based wizards concurrently without contention. However, Form-Based wizards are limited in what they can do between steps. The only processing these wizards can perform between steps is done by normal form or view processing. Form derivation and expansion, which are done when the view is being refreshed, is more limiting than the processing that is possible in a workflow. If a Form-Based wizard's processing limits preclude its use, you must use a Workflow-Based wizard. However, you should consider using a Form-Based wizard first.

Because Workflow-Based wizards require task execution and suspension to provide page transitions, and because each page in a Workflow-Based wizard corresponds to a workflow ManualAction object, these wizards are less efficient than Form-Based wizards. A problem occurs when the wizard has hundreds of concurrent invocations because all of the task start, suspend, and resume operations must contend for the object repository task table.

The object repository is being accessed synchronously with each HTTP request; consequently, a Workflow-Based wizard does not scale to large numbers of concurrent executions due to repository contention. Each start, suspend, and resume operation involves several reads and writes to the task table in the repository, and under a large concurrent load results in the page-to-page response of the GUI slowing down. Unlike a Form-Based wizard that scales by adding more Waveset servers and balancing the HTTP load between them, Workflow-Based wizards slow down due to repository contention. Adding more Waveset servers actually makes the problem worse because the repository is shared between all servers.

If you must use a Workflow-Based wizard because you need processing between page transitions, consider using transient ManualAction. A ManualAction in a Workflow-Based wizard is the mechanism used to display pages to the user. A five-step wizard typically has at least five ManualActions.

If you construct the wizard so that the user must either complete or abort the entire flow, then you can mark the ManualActions with the transient='true' attribute. Adding this attribute allows Waveset to bypass the repository instead of keeping the task in memory and sequencing it without accessing the object repository. This construction decreases the load the wizard puts on the repository, allowing it to scale to higher concurrency loads. However, using this attribute has some drawbacks. If you set transient to true (it is false by default), you cannot restart the workflow after the task enters the transient section because the in-memory state of the workflow does not match the repository state. Also, each HTTP request for the wizard must use the same server because the true state of the wizard is now only kept in memory on the initiating server. As soon as Waveset encounters a ManualAction with transient=false, Waveset writes the workflow to the repository normally, and normal workflow behavior resumes.

Consider using a Workflow-Based wizard with the following structure:


Begin Activity 1 
ManualAction 1 (transient = true) 
Activity 2 
ManualAction 2 (transient = true) 
ManualAction 3 (transient = true) 
ManualAction 4 (transient = true) 
ManualAction 5 (transient = false) 
Activity 3 
...

    When launched, this wizard creates a task that is initially stored in the object repository. All processing between ManualAction 1 and ManualAction 5 is done without any further repository work (the transient section). When Waveset executes ManualAction 5, the task is again stored in the repository (normal behavior), which is a significant performance savings because each normal suspend/resume for a ManualAction does the following:

  1. Creates and stores a WorkItem

  2. Saves the task state

  3. Reads/writes the WorkItem

  4. Locks the task

  5. Resumes the task

  6. Deletes the WorkItem

Each suspend/resume pair (when in wizard mode) results in more than 20 repository read/writes, all on the task table. The state of the task in the repository is running, so if the server crashes or shuts down during the execution of the wizard, the task will be deleted from the repository within a few minutes by another server or when the crashed server restarts, whichever comes first.

Note –

If you use a transient ManualAction, you can observe the effects by looking at the JMX TaskInstanceCache and WorkItemCache MBeans. These MBeans show the number of Store (repository write) operations compared to Cache (memory only) operations occurring. Each Cache operation means a Store was avoided, thus reducing object repository contention.

Updating Database Statistics

As a database administrator, you should frequently run statistics to monitor your repository database.

Performance problems are often caused by bad or missing database table statistics. Fixing this problem improves performance for both the database and Waveset performance.

See the following Oracle articles for more information:

Also consider using SQL Profiles, which is another method for choosing the best query plans. You can use the SQL Advisor within Enterprise Manager to create these profiles when you identify poorly performing SQL.

Tuning Data Exporter

Data Exporter enables you to export new, changed, or deleted Waveset data to an external repository that is suitable for reporting or analytic work. The actual exporting of data is done in batches, where each type of data to be exported is able to specify its own export cycle. The data to be exported comes from the Waveset repository and, depending on the length of the export cycle and the amount of changed data, the volume of exported data can be large.

Some Waveset data types are queued into a special table for later export. Specifically, WorkflowActivity and ResourceAccount data is queued because this data is not persisted otherwise. Any persisted data type can also be queued if the warehouse needs to see all changes to the type, or if the type has a lifecycle that does not correspond to the export cycle, such as TaskInstance and WorkItem data.

To maximize performance, only queue and export the types of data that you require in the warehouse. Data exporting is disabled by default, but if you enable data exporting, it exports all data types. Turn off any data types that you do not need.

When the export task exports data, the task attempts to complete the export as quickly as possible, using multiple threads to achieve as much throughput as possible. Depending on the I/O speed of the Waveset repository and the warehouse, the export task can fully utilize the processors on the Waveset server, which causes any interactive performance to degrade. Ideally, the export should occur on a machine dedicated to that task or at least occur during periods when there is no interactive activity on the machine.

The export task supports the following tuning parameters:

The drain thread count is the most important throughput. If a large number of records are in the queue table, increasing the number of threads (up to 24) tends to increase throughput. However, if the queue is dominated by one type of record, fewer drain threads might actually be faster. The export task attempts to divide the queue table contents into as many sets as there are threads allocated, and to give each thread a set to drain. Note that these threads are in addition to the drain threads that are draining the other repository tables.

Tuning the General XML

You can usually optimize the general XML by using static XMLObject declarations wherever possible. For example, use:

Also, depending on the context, you might have to wrap objects instead of using the <o></o> element.

Tuning Oracle Waveset Service Provider

You can use Waveset dashboard graphs to quickly assess the current system, spot abnormalities, and understand historical trends (such as concurrent users or resource operations over a time period) for Oracle Waveset Service Provider (Service Provider).

Note –

Service Provider does not have an Administrator interface. You use the Waveset Administrator interface to perform almost all administrative tasks (such as viewing dashboard graphs).

For more information about tuning Service Provider see Oracle Waveset Service Provider 8.1.1 Deployment.

Tuning the Waveset Web Interface

When you are working with the Waveset Web Interface, you can optimize performance by using the OpenSPML toolkit that is co-packaged with Waveset.

Note –

Using the openspml.jar file from the http://openspml.org/ web site might cause memory leaks.

Tuning Initial Loads

    To improve performance during a large, initial user load, follow this procedure:

  1. Disable all Audit Events from the Waveset Administrator interface.

    Note –

    Audit Logging can add several records per operation, making future audit reports perform more slowly.

    1. Choose Configure -> Audit.

    2. On the Audit Configuration page, deselect the Enable auditing box and click Save.

  2. Disable the list cache by shutting down the web server or by changing the ChangeNotifier.updateSearchIntervalCount property (on the debug/Show_WSProp.jsp debug page) to 0.

    The list cache keeps lists of users in frequently accessed organizations in memory. To maintain these lists, the list cache searches for and checks all newly created users.

  3. Clear the current list cache on the debug/Clear_List_Cache.jsp page.

  4. Ensure that the workflow being used to process the users does not contain approvals.

  5. Use alternative load methods, which include:

    • Splitting the load and running the data in zones

    • Using bulk loads, which are much faster

    • Loading from a file

  6. Disable Data Exporter for the WorkflowActivity type.

Tuning Memory Requirements

You must determine your memory needs and set values in your application server’s JVM by adding maximum and minimum heap size to the Java command line. For example:

java -Xmx512M -Xms512M

To improve performance do the following:

Note –

The max.post.memory.size specifies the maximum number of bytes that a posted file (for example by using an HTML FileSelect control) might contain without being spooled to the disk. For cases where you do not have permission to write to temporary files, increase the max.post.memory.size to avoid having to spool to the disk. The default value is 8 Kbytes.

For additional information about system requirements, see the Oracle Waveset 8.1.1 Release Notes.

Tuning Operating System Kernels

For information about tuning Solaris and Linux operating system kernels, see the “Tuning the Operating System” chapter in the Sun Java System Application Server Enterprise Edition Performance Tuning Guide.

For information about tuning Oracle operating system kernels, see the product documentation provided with your Oracle system.

Tuning the Waveset Profiler

Each Waveset server captures profiling data by default. You can use this data with the Waveset IDE to diagnose a large range of performance problems. However, capturing and storing this profiling data adds a measurable load to the server, which consumes both memory and CPU. In a stable production environment, disable the profiler in production servers. Enable the profiler only when you are investigating a performance problem.

Note –

For information about using the Waveset IDE, see Identity Manager IDE Frequently Asked Questions (FAQ) in Oracle Waveset 8.1.1 Release Notes.

Example 4–1 Modifying the SystemConfiguration Configuration Object

    The following example shows how to disable the Profiler from capturing data.

  1. Import the following XML to disable the profiler.

    Note –

    Setting the attribute value to true disables the profiler, and setting it to false enables the profiler.

  2. Use the lh import command to import the following XML

    <?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE Waveset PUBLIC 'waveset.dtd' 'waveset.dtd'>

  <Waveset>
     <ImportCommand class='com.waveset.session.SystemConfigurationUpdater' >
       <Object>
         <Attribute name='server.default.disableProfiling'>
           <Boolean>true</Boolean>
         </Attribute>
       </Object>
     </ImportCommand>
   </Waveset>

  3. Restart all Waveset servers.

Tuning Provisioner

Network latency tends to be a common cause for performance issues when dealing with view provisioning. Tracing individual resource adapters can help you determine what is causing performance problems.

To improve provisioner performance, do the following:

Tuning Reconciliation

The Reconciler is the Waveset component that performs reconciliation. This section suggests methods for improving Reconciler performance, including:

General Suggestions for Tuning Reconciliation

In general, you can improve Reconciler performance if you do the following:

Reconciliation of a resource goes through two phases. In the first phase, Waveset gets a list of all users in its internal repository that are known to have accounts on the resource. This first phase does not involve the physical resource at all, and typically happens very quickly.

The second phase requests a list of all accounts from the resource, and then processes those accounts; potentially linking them to users or even creating new users. Performance of the second phase, indicated as reconciling accounts in the resource status message, is proportional to the speed of the resource and the number of worker threads. You can compensate for a slow resource by adding more worker threads; assuming the resource can handle more concurrent AccountGet operations.

Note –

There is a JMX MBean for each resource that shows the average, minimum, and maximum response times for each resource operation. Reconciliation phase two involves lots of Account_Get operations, so the average time for each Account_Get strongly influences the overall reconciliation performance. To compensate for resources with longer Account_Get times, use more worker threads. However, because the same number of worker threads are used for all resources, setting a maximum worker thread too high might overwhelm the Waveset object repository on faster resources.

Tuning the Reconciler Server Settings

Although the default settings are usually adequate, you can sometimes improve Reconciler performance if you adjust the following settings on the Edit Server Settings page:

During idle times, the threads stop if they have no work to do, but only down to the minimum number of threads specified. As the load increases, the Reconciler adds more threads until the maximum number of threads is reached. The Reconciler never has less than the minimum number of threads or more than the maximum.

Generally, more threads allow more concurrency. However, at some point, too many threads can put too much load on the machine or just do not provide additional benefit. Because the worker threads are typically reading and writing User and Account objects, having too much concurrency might overload the Waveset repository RDBMS.

Note –

Recommending generic, optimal settings is not possible because deployments are so different. Reconciler settings must be adjusted differently for each deployment environment.

ProcedureTo Change the Reconciler Server Settings

Perform the following steps to change the Reconciler server settings:

  1. Log into the Administrator interface.

  2. Click the Configure -> Servers -> Reconciler tabs.

  3. When the Edit Server Settings page is displayed, adjust the settings as necessary.

    See Editing Default Server Settings for more information.

Tuning Reconciliation for Multiple Resources

If you are configuring reconciliation for multiple resources in Waveset, you have several options:

An ideal solution does not exist for this configuration because deployments are so different. You might have to mix and match these options to find an acceptable solution for your deployment.

Preparing a usage survey, based on the business reasons behind this functionality, might help you decide how to proceed.

Address these questions:

Also, remember that the reconciliation server does not have to be one of the pools that handles web traffic. You can add a server that you never interact with directly because this server exists solely for transaction processing. Having a server dedicated to transaction processing might make the first option more attractive for very large systems.

Tuning Resource Queries

Note –

Network latency tends to be a common cause of performance issues during view provisioning. Tracing individual resource adapters can help you determine what is causing performance problems.

You can improve resource query performance if you use FormUtil.getResourceObjects to implement the query.

Use one of the following methods to cache query results:

Note –

Tuning the Scheduler

The Scheduler component controls task scheduling in Waveset.

This section suggests methods for improving Scheduler performance, including:

General Suggestions for Tuning the Scheduler

The following TaskDefinition options determine how the Scheduler handles tasks after they are completed:

These default settings are designed to optimize memory by shortening the lifetime of finished Scheduler tasks. Unless there is a compelling reason to change these settings, use the defaults.

If you want to immediately delete tasks that completed successfully, but you also want to keep tasks containing errors long enough to debug, you can do the following:

With these two settings, Waveset will only store the task results if the task has an error or warning Result Item. This configuration can improve the performance of some tasks by allowing them to bypass being stored in the repository when they complete.

Tuning the Scheduler Server Settings

You can sometimes improve Scheduler performance by adjusting the following settings on the Edit Server Settings page:

Note –

Recommending generic, optimal settings is not possible because deployments are so different. Scheduler settings must be adjusted differently for each deployment environment.

ProcedureTo Change the Scheduler Server Settings

  1. Log in to the Administrator interface.

  2. Click the Configure -> Servers -> Scheduler tabs.

  3. When the Edit Server Settings page is displayed, adjust the settings as necessary.

    See Editing Default Server Settings for more information.

Tuning Scheduler Attributes in the SystemConfiguration Object

To improve Scheduler performance when the component is under a heavy backlog, you must modify the SystemConfiguration configuration object to enable all optimizations. The Scheduler can pick up any changes to the SystemConfiguration object while the server is running.

The Waveset Scheduler is responsible for executing scheduled tasks, resuming suspended tasks, and cleaning up task results for completed tasks. The Scheduler is single-threaded, meaning it processes its work with a single control thread. However, when the Scheduler starts or resumes a task, it runs that task in a new thread and can have several tasks running at the same time.

You can directly start tasks without any Scheduler processing. For example, workflow and report tasks are often started as a result of a HTTP request. Starting, suspending, and resuming a task causes a modest amount of object repository processing. The repository can become congested if lots of concurrent HTTP requests are starting or resuming tasks. Because an application server may service hundreds of HTTP requests simultaneously, it is easy to create a large backlog for the Scheduler.

For example, if hundreds of HTTP requests started workflows that were subsequently suspended, the Scheduler would be responsible for resuming those workflows.

Because the Scheduler has a single control thread, but many different tasks to perform, it might sometimes seem like the Scheduler is not keeping up with one task or another. Waveset uses the sleepingTaskLimit and readyTaskLimit attributes to control how long the Scheduler can spend processing sleeping or ready tasks during each control loop. These limits ensure that a Scheduler, when presented with thousands of ready tasks, does not spend too much time starting these tasks and ignoring sleeping tasks.

When you have multiple Waveset servers, each server typically runs a Scheduler. By default, these Schedulers compete for the same work by polling the object repository. Configuring blockProcessing allows each Scheduler to process a different block of tasks, resulting in somewhat less object repository contention for specific records. The blockProcessing attribute is enabled by default, but all servers process the same slot. To enable cooperative (rather than competitive) task processing, you can assign each server a different slot (starting with 0). When you assign slots (other than 0) to the servers, they segment the tasks into buckets and only process the tasks in their assigned bucket.

The Scheduler has a JMX MBean that is very useful in diagnosing what is perceived as slow Scheduler performance. The ExecuteTime attribute in the MBean is often the key to understanding the Scheduler's performance. ExecuteTime is the time (in milliseconds) it took the Scheduler to start or resume the last task it processed. On a healthy system, this time should be less then 150 milliseconds. When this value starts to get large, the server is having trouble starting tasks, typically because there is congestion on the task table in the object repository, or because of internal synchronization in the Waveset code itself. Viewing the Scheduler's thread stack in the JMX console usually reveals the problem.

The ExecutingTaskCount attribute in the MBean shows how many tasks the Scheduler is currently managing. By default, the task limit is 100, which is almost always sufficient unless the tasks being executed run for a long time without suspending (such as report tasks). The ExecutingTaskCount value does not reflect all tasks running on the server. Remember that HTTP requests can also start tasks, so the total number of tasks running on the server is unknown to the Scheduler.

One of the Scheduler's many jobs is to resume tasks. Because each Waveset server can have a Scheduler running, the Schedulers also periodically look to process work that was being handled by another server that is not currently running. When a server goes into the recovered state, Schedulers on other servers attempt to process or clean up work that was being done on the recovered server. Schedulers go into a recovered state when one server observes another server has not issued a heartbeat within the last five minutes.

This check compares the timestamp of the last heartbeat message in the repository to the current server's clock. If Waveset servers have more than five minutes of system clock skew, the server with the clock that is farthest ahead marks servers with clocks that are behind as recovered. To avoid this situation, keep the system clocks on your Waveset servers synchronized.

Note –

If necessary, review Editing Waveset Configuration Objects in Oracle Waveset 8.1.1 Business Administrator’s Guide.

To control the Scheduler's behavior, add one or more of the following attributes to the SystemConfiguration configuration object. Use the serverSettings.default.scheduler.attribute path to create these attributes.

Tuning Sessions

Waveset maintains a least recently used (LRU) cache of authenticated sessions for use by authenticated users. By using existing authenticated sessions, you can speed up repository access for objects and actions that require a session.

To optimize the authentication pool size, change the session.userPoolSize value in the Waveset.properties file to the maximum number of expected, concurrent user sessions on the server.

Tuning the Oracle Waveset Gateway

The Oracle Waveset Gateway generates a thread for each connection, and uses a different pool for each unique combination of resource type, Gateway host, and Gateway port. The Gateway checks for idle connections every five minutes. When a connection has been idle for 60 seconds, the Gateway closes and removes that connection from the pool.

When the Gateway receives a request, it does the following:

You must configure the maximum number of connections on the resource, and you must configure these connections the same way for all resources of the same type, that are using the same Gateway. For that resource type, the first connection made to the Gateway on a given host and port uses that resource’s maximum connections value.

Note –

When you change the maximum number of connections on a resource, you must start and stop the server for the change to take effect.

The following example shows how connections, requests, and Gateway threads are related.

If you set the maximum number of connections to 10 on an Active Directory resource, and you are using two Waveset servers, then you can have up to 20 simultaneous connections (10 from each Waveset server) to the Gateway for that Active Directory resource. The Gateway can have 10 simultaneous requests outstanding from each server, and the Gateway processes each request on a different thread. When the number of simultaneous requests exceeds the maximum number of Gateway connections, additional requests are queued until the Gateway completes a request and returns the connection to the pool.

Note –

Although the Gateway code is multi-threaded, this characteristic does not address the APIs or services being used by the Gateway. For Active Directory, the Gateway uses the ADSI interface provided by Microsoft. No investigation has been done to determine whether this interface handles Gateway requests in parallel.

Other methods for improving Gateway performance, include:

Tuning the Task Bar

The Administrator interface task bar displays links to previously performed provisioning tasks, which causes the interface to render more slowly when there are a large number of tasks.

To improve interface performance, remove the taskResults.jsp link from each JSP by deleting the <List>...</List> element from the UserUIConfig object.

The following example shows <List>...</List> entries within <TaskBarPages>.

Example 4–2 Modifying the UserUIConfig Object


<TaskBarPages>
  <List>
    <String>account/list.jsp</String>
    <String>account/find.jsp</String>
    <String>account/dofindexisting.jsp</String>
    <String>account/resourceReprovision.jsp></String>
    <String>task/newresults.jsp</String>
    <String>home/index.jsp</String>
  </List>
 </TaskBarPages>