Tuning the Scheduler

The Scheduler component controls task scheduling in Waveset.

This section suggests methods for improving Scheduler performance, including:

• General Suggestions for Tuning the Scheduler

• Tuning the Scheduler Server Settings

• Tuning Scheduler Attributes in the SystemConfiguration Object

General Suggestions for Tuning the Scheduler

The following TaskDefinition options determine how the Scheduler handles tasks after they are completed:

• resultLimit — Controls how many seconds task results are kept after the task has completed. The default setting varies for different tasks. A setting of zero immediately removes tasks after completion.

• resultOption — Controls what action is taken when repeated instances of a task are started. The default setting is delete, which removes previous task instances.

These default settings are designed to optimize memory by shortening the lifetime of finished Scheduler tasks. Unless there is a compelling reason to change these settings, use the defaults.

If you want to immediately delete tasks that completed successfully, but you also want to keep tasks containing errors long enough to debug, you can do the following:

• Set the resultLimit to a small value, such as 3600 seconds.

• Set the saveOnlyOnError value to true.

With these two settings, Waveset will only store the task results if the task has an error or warning Result Item. This configuration can improve the performance of some tasks by allowing them to bypass being stored in the repository when they complete.

Tuning the Scheduler Server Settings

You can sometimes improve Scheduler performance by adjusting the following settings on the Edit Server Settings page:

• Maximum Concurrent Tasks. Specifies the maximum number of tasks that the Scheduler can run at one time.

  When more tasks are ready to run than the Maximum Concurrent Tasks setting allows, the extra tasks must wait until there is room available or until they are run on another server.

  If too many tasks are being swapped out of memory and sharing CPU time, the overhead slows down performance. Alternatively, setting the maximum too low results in idle time. The Scheduler checks for available tasks every minute, so a waiting task waits at least a minute before being run.

  The default Maximum Concurrent Tasks setting (100) is usually adequate. You can decide whether to adjust this setting up or down based on which tasks are being run in the deployment and by profiling the runtime behavior after the deployment is otherwise complete.

  Before you change this setting, you may want to observe the Scheduler using JMX. The Scheduler.executingTaskCount attribute shows how many tasks the Scheduler is running.

  In some cases, you might want to suspend or disable the Scheduler. For example, if you want a server dedicated to handling the End User interface, disabling the Scheduler will prevent tasks from running on that server. The server would only serve the End User interface pages and store launched tasks for other servers to execute.

• Task Restrictions. Specifies the set of tasks that can execute on the server.

  The Task Restrictions setting can provide a finer granularity of control over what tasks are allowed to run on a server. You can restrict tasks individually or through the server settings.

Recommending generic, optimal settings is not possible because deployments are so different. Scheduler settings must be adjusted differently for each deployment environment.

To Change the Scheduler Server Settings

1. Log in to the Administrator interface.

2. Click the Configure -> Servers -> Scheduler tabs.

3. When the Edit Server Settings page is displayed, adjust the settings as necessary.

   See Editing Default Server Settings for more information.

Tuning Scheduler Attributes in the SystemConfiguration Object

To improve Scheduler performance when the component is under a heavy backlog, you must modify the SystemConfiguration configuration object to enable all optimizations. The Scheduler can pick up any changes to the SystemConfiguration object while the server is running.

The Waveset Scheduler is responsible for executing scheduled tasks, resuming suspended tasks, and cleaning up task results for completed tasks. The Scheduler is single-threaded, meaning it processes its work with a single control thread. However, when the Scheduler starts or resumes a task, it runs that task in a new thread and can have several tasks running at the same time.

You can directly start tasks without any Scheduler processing. For example, workflow and report tasks are often started as a result of a HTTP request. Starting, suspending, and resuming a task causes a modest amount of object repository processing. The repository can become congested if lots of concurrent HTTP requests are starting or resuming tasks. Because an application server may service hundreds of HTTP requests simultaneously, it is easy to create a large backlog for the Scheduler.

For example, if hundreds of HTTP requests started workflows that were subsequently suspended, the Scheduler would be responsible for resuming those workflows.

Because the Scheduler has a single control thread, but many different tasks to perform, it might sometimes seem like the Scheduler is not keeping up with one task or another. Waveset uses the sleepingTaskLimit and readyTaskLimit attributes to control how long the Scheduler can spend processing sleeping or ready tasks during each control loop. These limits ensure that a Scheduler, when presented with thousands of ready tasks, does not spend too much time starting these tasks and ignoring sleeping tasks.

When you have multiple Waveset servers, each server typically runs a Scheduler. By default, these Schedulers compete for the same work by polling the object repository. Configuring blockProcessing allows each Scheduler to process a different block of tasks, resulting in somewhat less object repository contention for specific records. The blockProcessing attribute is enabled by default, but all servers process the same slot. To enable cooperative (rather than competitive) task processing, you can assign each server a different slot (starting with 0). When you assign slots (other than 0) to the servers, they segment the tasks into buckets and only process the tasks in their assigned bucket.

The Scheduler has a JMX MBean that is very useful in diagnosing what is perceived as slow Scheduler performance. The ExecuteTime attribute in the MBean is often the key to understanding the Scheduler's performance. ExecuteTime is the time (in milliseconds) it took the Scheduler to start or resume the last task it processed. On a healthy system, this time should be less then 150 milliseconds. When this value starts to get large, the server is having trouble starting tasks, typically because there is congestion on the task table in the object repository, or because of internal synchronization in the Waveset code itself. Viewing the Scheduler's thread stack in the JMX console usually reveals the problem.

The ExecutingTaskCount attribute in the MBean shows how many tasks the Scheduler is currently managing. By default, the task limit is 100, which is almost always sufficient unless the tasks being executed run for a long time without suspending (such as report tasks). The ExecutingTaskCount value does not reflect all tasks running on the server. Remember that HTTP requests can also start tasks, so the total number of tasks running on the server is unknown to the Scheduler.

One of the Scheduler's many jobs is to resume tasks. Because each Waveset server can have a Scheduler running, the Schedulers also periodically look to process work that was being handled by another server that is not currently running. When a server goes into the recovered state, Schedulers on other servers attempt to process or clean up work that was being done on the recovered server. Schedulers go into a recovered state when one server observes another server has not issued a heartbeat within the last five minutes.

This check compares the timestamp of the last heartbeat message in the repository to the current server's clock. If Waveset servers have more than five minutes of system clock skew, the server with the clock that is farthest ahead marks servers with clocks that are behind as recovered. To avoid this situation, keep the system clocks on your Waveset servers synchronized.

If necessary, review Editing Waveset Configuration Objects in Oracle Waveset 8.1.1 Business Administrator’s Guide.

To control the Scheduler's behavior, add one or more of the following attributes to the SystemConfiguration configuration object. Use the serverSettings.default.scheduler.attribute path to create these attributes.

• blockProcessing (Boolean). Indicates whether servers should work on independent blocks of tasks or not. Defaults to true, which means Scheduler will work on independent blocks of tasks.

  ---

  Note –

  blockProcessing does not apply to task deletion.

  ---

• blockProcessingSize (Integer). Represents how many tasks a Scheduler will process in its block. Defaults to 50.

• slots (List of Strings). Indicates slots of Waveset server instance names. Defaults to an empty list.

  • If the blockProcessing attribute is true, each Scheduler processes the block at the server's position in this list.

  • If you do not provide a list of slots, or do not include a server in the list, every Scheduler assumes it has slot 0 and each Scheduler processes the same block of tasks concurrently.

  ---

  Note –

  slots do not apply to task deletion.

  ---

• fastResume (Boolean). Instructs the Scheduler to immediately execute tasks that are sleeping with an expired restart time. Defaults to true.

  • If true, the Scheduler executes sleeping tasks without first placing them in the ready state, which reduces load on the Identity Manager repository.

  • If false, the Scheduler first puts the task in a ready state, and then executes the task on the next Scheduler poll.

• threadPriority (Boolean). Indicates the Scheduler thread should be run at an elevated priority. Defaults to false.

  • If true, the Scheduler threads run with an elevated priority.

  • If false, the Scheduler threads are only read at server startup.

• sleepingTaskLimit (Integer). Indicates how many milliseconds the Scheduler can spend processing sleeping tasks before reentering the polling loop. Defaults to 60 seconds.

  The Scheduler is single-threaded and does not perform any other operations when it is processing sleeping tasks. If a large number of tasks are sleeping and ready to run, the Scheduler only processes as many tasks as it can within this time limit.

• readyTaskLimit (Integer). Indicates how many milliseconds the Scheduler can spend processing ready tasks before reentering the polling loop. Defaults to 60 seconds.

  The Scheduler is single-threaded and does not perform any other operations when it is processing ready tasks. If a large number of tasks are ready to run, the Scheduler only processes as many tasks as it can within this time limit.