This document describes how to handle an Active Directory synchronization failover. Implementing this customization can help limit the number of repeated events that occur when you switch to a new domain controller.
The Active Directory synchronization failover uses a task to periodically collect and maintain a history of the HighestCommittedUSN from a configurable set of domain controllers to which it can fail over. If the Active Sync domain controller goes down, another task can be run that will change the configuration of the Active Directory resource to point to one of the failover domain controllers. Because changes made in Active Directory can take a while to replicate to all domain controllers, Active Directory Active Sync cannot just start processing only new changes on the failover domain controller. Instead, it must also look at older changes made on the failover domain controller that might not have been replicated to the domain controller before it went down. To this end, it will use a saved HighestCommittedUSN for the failover domain controller that is far enough in the past to account for any replication delay. This prevents Active Sync from missing events, but some changes will likely be processed twice.
This procedure involves the following components:
The Active Directory Synchronization Failure Process, which is defined on the Active Directory resource by the On Synchronization Failure Process Active Directory resource attribute
Active Directory Recovery Collector Task
Active Directory Failover Task
The On Synchronization Failure Process Active Directory active synchronization resource attribute specifies the name of a process to be executed on a synchronization failure. By default, the value of this resource attribute is empty.
This attribute gives Waveset administrators the ability to execute a process when Active Directory synchronization failures occur.
The process specified by the resource attribute is launched by the resource on failure. You should invoke a process that sends email to the Active Directory administrator that alerts them to a synchronization failure. The body of the email might contain the error messages that were returned from the adapter poll method.
You can also design a business process that, when a specified error occurs, automatically calls the Synchronization Failover task after an approval by an administrator is given.
The following arguments are available to the native process.
Argument |
Description |
---|---|
resourceName |
Identifies the resource where the failure occurred |
resultErrors |
Lists strings that represent the errors returned by the poll method |
failureTimestamp |
Indicates when the failure occurred |
You can schedule and launch the Active Directory Recovery Collector task from the Task Schedule pages of the Waveset Administrator interface. This process uses the resource object interface to contact each domain controller’s rootDSE object. The task’s schedule determines the frequency at which the data is collected from the domain controllers.
This task collects and stores resource recovery information in a Configuration object named ADSyncRecovery_resourceName. The extension to this configuration object is a GenericObject that stores a list of HighestCommittedUSN and the timestamp (milliseconds) that was collected for each domain controller.
During each execution, the task prunes old values for HighestCommittedUSN from the recovery data. You can configure the length of time to store this data through the daysToKeepUSNS argument.
Argument |
Description |
---|---|
resourceName |
Specifies the Active Directory resource for which Waveset collects backup data. |
backupDCs |
Lists the fully qualified domain controller hostnames that should be contacted for recovery data. This can and should include the original host, which permits Waveset to include the source resource host if Waveset must fail over to the resource. When synchronizing against a global catalog, back up hosts in this list will be assumed to be global catalogs. |
daysToKeepUSNS |
Specifies the number of days for which Waveset stores the data (default is 7 days). |
This task reconfigures the failed resource and the IAPI Object to use an alternate domain controller and usnChanged starting point. The task input form displays the available usn-changed times for a given host from the stored failover data.
Certain errors can identify conditions where failover is appropriate. One example of the potential difficulty of automatically calling the failover task is the java.net.UnknownHostException error message. The failure indicated by this message can occur for at least two reasons:
The host cannot be reached from the gateway machine due to a temporary routing issue.
The host cannot be reached and will be down for the next eight hours due to a planned outage.
You can take one of two approaches towards implementing Active Directory failover resolution:
Manual mode. When a problem occurs, the administrator specifies which backup domain controller and USN to use. This is the only mode available when running tasks from the Waveset interface.
Semi-auto mode. Semi-auto mode permits you to semi-automate the fail-over resolution process. In semi-auto mode, the task uses the collected data to identify the best backup domain controller and USN to use. It computes this by looking for a collection point that is closest to a derived TargetTimestamp without exceeding this value
where TargetTimestamp = (FailureTimestamp - replicationTime)
Semi-auto mode is not available from the Waveset Administrator interface.
If you have determined that launching semi-auto failover is appropriate for a particular error, set the following task arguments. (The on-error workflow must launch the Active Directory Synchronization failover task.) Setting these arguments reconfigures the failed resource and the IAPI Object to use an alternate domain controller and usnChanged starting point.
Argument |
Description |
---|---|
resourceName |
Identifies (by name or resource ID) where the failure has occurred. |
autoFailover |
Specifies whether auto failure is set. Must be set to true. |
failureTimestamp |
Indicates when the failure occurred. This value is derived from the onSync failure process. |
replicationTime |
Specifies the maximum time in hours for data to replicate across an Active Directory environment. |
To manually specify which domain controller to fail over to and which saved HighestCommittedUSN number to start from, set the following arguments.
Argument |
Description |
---|---|
resourceName |
Specifies the name or ID of the resource where the failure has occurred. |
backupDC |
Specifies the name of the host with which to begin the synchronization process. |
usnDate |
The timestamp to use that correlates to a collected HighestCommittedUSN changed value from the collected data. This would be computed just as targetTime was computed in the semi-auto mode. |
restartActiveSync |
Specifies whether to start Active Sync after the switch to the new domain controller is complete. |
The Active Directory Recovery Collector task updates either the LDAPHostname or the GlobalCatalog resource attribute value (depending on which value is in use). If the search subdomains resource attribute is set to true, and the global catalog attribute value is not empty, the global catalog server attribute is changed. Otherwise, the LDAPHostname is changed to the name of the backup domain controller.
The Active Directory Recovery Collector task also updates the IAPI object so that the Active Directory resource adapter knows which changes to look for the next time it runs. The task updates the HighCommitedUSN value for both lastUpdated and lastDeleted attribute values.
Configure the maximum number of hours to retain data. The default value is seven days. This value controls how far back the HighestCommittedUSN values are kept.
You must configure one workflow per Active Sync resource that must be configured.
Schedule this task from the Task page of the Waveset Administrator interface. The polling interval, which establishes how often to contact each host for their HighestCommittedUSN value, is set by the task schedule.
When this task is executed, it calls out to the Active Directory adapter to retrieve the HighestCommittedUSN number from each domain controller’s rootDSE. It then stores this value in an Waveset configuration object. The task generates one configuration object per defined Active Sync resource to store alternate domain controller HighestCommittedUSN values.
On each Active Directory Active Sync resource, Waveset defines an onError process that is called when a failure occurs during the synchronization of a resource. If an Active Directory resource defines an on-error process, this process is called if there errors occur when the poll method is called on the resource during active synchronization. This process checks the result from the IAPI objects, and if an error occurs, calls the defined process.
Configure this process to notify an administrator through email when an error occurs. Include the error text in the email body so that the administrator can determine if the error warrants that Waveset fails over to another domain controller.
Using the error text, the administrator is alerted to a potentially lengthy outage or an outrage due to a temporary, quickly resolved issue (such as a temporary routing issue that is resolved by the next poll attempt).
If the domain controller returns an error that warrants failing over to another domain controller, run the Active Directory Synchronization Failover task from the Task page.
For manual fail-over mode, the fail-over task requests
The name of the downed domain controller or resource
The name of the DC hostname to fail over
The timestamp of the collected HighestCommittedUSN value to use
You also must choose whether to restart Active Sync after the switch to a new domain controller is complete.
When executed, the Active Directory Synchronization Failover task
Stops the Active Sync process on the failed resource
Reads in the fail over configuration object
Changes necessary resource attribute values
Optionally restarts the Active Sync process.
You can configure the following example workflow as the On Synchronization Failure Process resource attribute of an Active Directory resource. The workflow looks for the java.net.UnknownHostException error message. If it finds this message, the workflow launches a notification email to the administrator.
<TaskDefinition name=’Sample AD Sync On Error Workflow’ executor=’com.waveset.workflow.WorkflowExecutor’ syncControlAllowed=’true’ execMode=’sync’ taskType=’Workflow’> <Extension> <WFProcess title=’Example AD Sync OnError Workflow’> <Variable name=’resultErrors’ input=’true’> <Comments>Errors returned from the resource. </Comments> </Variable> <Variable name=’resourceName’ input=’true’> <Comments>Name of the AD resource that returned the errors. </Comments> </Variable> <Variable name=’failureTimestamp’ input=’true’> <Comments>Failure timestamp, when it occurred. </Comments> </Variable> <Activity name=’start’> <Transition to=’checkErrors’/> </Activity> <Activity name=’checkErrors’> <Variable name=’criticalError’> <Comments>Local variable to hold if we need to notify </Comments> </Variable> <Action name=’iterateMessage’> <dolist name=’msg’> <ref>resultErrors</ref> <cond> <match> <ref>msg</ref> <s>java.net.UnknownHostException</s> </match> <set name=’criticalError’> <s>true</s> </set> </cond> </dolist> </Action> <Transition to=’notify’> <notnull> <ref>criticalError</ref> </notnull> </Transition> <Transition to=’end’/> </Activity> <Activity name=’notify’> <Action application=’notify’> <Argument name=’template’ value=’#ID#EmailTemplate:ADSyncFailoverSample’/> <Argument name=’resultErrors’ value=’$(resultErrors)’/> </Action> <Transition to=’end’/> </Activity> <Activity name=’end’/> </WFProcess> </Extension> </TaskDefinition>