Schedulable services are useful for a wide variety of tasks in Dynamo, including session expiration, content and component indexing, session backup, JMS message polling, log file management, and reporting. Typically, these are tasks that are performed periodically, and only affect the Dynamo server where they run. There are certain recurring tasks, however, that should be performed no more than once under any circumstances. There are many examples of at-most-once behavior, such as:

The typical approach to implementing such tasks in Dynamo has been to create a scheduled service, then configure it on only one Dynamo server within the site. This provides at-most-once behavior, but has the disadvantage of introducing a single point of failure into the system. If the server handling order placement goes down, orders are not placed. If the server handling report generation goes down, report generation stops. How easily one can recover from this situation depends largely on the complexity of the scheduled services involved.

The SingletonSchedulableService, a subclass of the standard SchedulableService, works in conjunction with a client lock manager to guarantee that only one instance of a scheduled service is running at any given time throughout a cluster of cooperating Dynamo servers. This provides the foundation for an architecture where scheduled services can be configured on multiple Dynamo servers to provide fault tolerance, but the system can still ensure that each task is performed at most once.

SingletonSchedulableService is an abstract base class that implements the Schedulable interface by subclassing SchedulableService, and that automatically checks to see if any other instance of the service is running anywhere on the local Dynamo cluster before performing its regularly scheduled task. Applications can subclass SingletonSchedulableService to provide concrete implementations that perform whatever application-specific work is required.

Note: Singleton behavior of the scheduled service is necessary, but not in itself sufficient, to ensure at-most-once behavior of the overall system. Consider the case of periodic order fulfillment:

Obviously, two scheduled fulfillment services should never wake up at the same time on two different Dynamo servers and fulfill the same orders at the same time. The result would be to ship two copies of each order to every customer, and to bill for both of them as well. On the other hand, two scheduled fulfillment services should not wake up at completely different times and fulfill the same orders. Even though the services might not overlap at all, another mechanism is necessary to keep the second service instance from shipping orders that the first instance already handled.

SingletonSchedulableService is designed to work in situations where the job performed by the system can be broken down into discrete work units, and where each work unit has some status indicator that tells whether or not it currently requires processing. Examples of such systems include:

The assumption behind SingletonSchedulableService is that on each run the service wakes up, obtains a lock to ensure that it is the only instance of the service running, fetches the work units that require processing, processes them, and then updates their status before relinquishing the lock.

The following sequence must be performed atomically from the service’s point of view:

The automatic locking mechanism provided by SingletonSchedulableService ensures this atomicity.

Guaranteeing that only one instance of the service runs at a time throughout the Dynamo cluster prevents two instances of the service that happen to wake up at the same time from trying to process the same work units. The fact that the service updates the status of the work units before releasing its lock prevents a subsequent instance of the service from trying to process those same work units again. The overall result is that each work unit should be processed once and only once, without the need to introduce a single point of failure into the system.

Configuring a SingletonSchedulableService

SingletonSchedulableService is an abstract class that extends SchedulableService and implements the performScheduledTask method in a way that tests for other instances of the service before proceeding.

To implement a SingletonSchedulableService, create a subclass of this class, and then create a component based on that subclass. Configure the component with a reference to a client lock manager, a lock name, and a timeout period. For example:

#Thu Aug 09 19:15:14 EDT 2001

The code in performScheduledTask contacts the client lock manager and requests a write lock using the specified lock name and timeout. If a lock is obtained successfully, the SingletonSchedulableService checks to see whether it is a global write lock or a local write lock.

If a global write lock is returned the service can assume that it is the only instance running anywhere on the Dynamo cluster, at least if all instances of the service point to client lock managers than in turn point to the same server lock manager. In this case SingletonSchedulableService calls the doScheduledTask method to do whatever work the service does, then releases the global lock.

If a local write lock is returned it means that the client lock manager was unable to contact the server lock manager. The implementation then makes the pessimistic assumption that another instance of the service might be running, so it logs a debugging message, releases the local lock without doing any work, and goes back to sleep until the scheduler calls it again.

If the request to obtain a lock times out, the implementation assumes that another instance of the service is running and taking a long time about it, so it logs a debugging message and goes back to sleep until the scheduler calls it again.

Finally, if a DeadlockException is thrown, the implementation logs an error message and makes the pessimistic assumption that another instance of the service might be running, so it logs an error message and goes back to sleep.

Note: If the client lock manager’s useLockServer property is set to false, it means that global locking is disabled for that lock manager. In this case, SingletonSchedulableService accepts a local lock in place of the global lock, which at least ensures that only one instance of the service runs at a time on the current Dynamo server.

loading table of contents...