Chapter 52 Active Directory Synchronization Failover

This document describes how to handle an Active Directory synchronization failover. Implementing this customization can help limit the number of repeated events that occur when you switch to a new domain controller.

The Active Directory synchronization failover uses a task to periodically collect and maintain a history of the HighestCommittedUSN from a configurable set of domain controllers to which it can fail over. If the Active Sync domain controller goes down, another task can be run that will change the configuration of the Active Directory resource to point to one of the failover domain controllers. Because changes made in Active Directory can take a while to replicate to all domain controllers, Active Directory Active Sync cannot just start processing only new changes on the failover domain controller. Instead, it must also look at older changes made on the failover domain controller that might not have been replicated to the domain controller before it went down. To this end, it will use a saved HighestCommittedUSN for the failover domain controller that is far enough in the past to account for any replication delay. This prevents Active Sync from missing events, but some changes will likely be processed twice.

Architectural Components

This procedure involves the following components:

• The Active Directory Synchronization Failure Process, which is defined on the Active Directory resource by the On Synchronization Failure Process Active Directory resource attribute

• Active Directory Recovery Collector Task

• Active Directory Failover Task

On Synchronization Failure Process Resource Attribute

The On Synchronization Failure Process Active Directory active synchronization resource attribute specifies the name of a process to be executed on a synchronization failure. By default, the value of this resource attribute is empty.

This attribute gives Identity Manager administrators the ability to execute a process when Active Directory synchronization failures occur.

Active Directory On Failure Process

The process specified by the resource attribute is launched by the resource on failure. You should invoke a process that sends email to the Active Directory administrator that alerts them to a synchronization failure. The body of the email might contain the error messages that were returned from the adapter poll method.

You can also design a business process that, when a specified error occurs, automatically calls the Synchronization Failover task after an approval by an administrator is given.

Process Context

The following arguments are available to the native process.

Active Directory Recovery Collector Task

You can schedule and launch the Active Directory Recovery Collector task from the Task Schedule pages of the Identity Manager Administrator interface. This process uses the resource object interface to contact each domain controller’s rootDSE object. The task’s schedule determines the frequency at which the data is collected from the domain controllers.

This task collects and stores resource recovery information in a Configuration object named ADSyncRecovery_resourceName. The extension to this configuration object is a GenericObject that stores a list of HighestCommittedUSN and the timestamp (milliseconds) that was collected for each domain controller.

During each execution, the task prunes old values for HighestCommittedUSN from the recovery data. You can configure the length of time to store this data through the daysToKeepUSNS argument.

Arguments

Active Directory Failover Task

This task reconfigures the failed resource and the IAPI Object to use an alternate domain controller and usnChanged starting point. The task input form displays the available usn-changed times for a given host from the stored failover data.

Certain errors can identify conditions where failover is appropriate. One example of the potential difficulty of automatically calling the failover task is the java.net.UnknownHostException error message. The failure indicated by this message can occur for at least two reasons:

• The host cannot be reached from the gateway machine due to a temporary routing issue.

• The host cannot be reached and will be down for the next eight hours due to a planned outage.

Failover Modes

You can take one of two approaches towards implementing Active Directory failover resolution:

• Manual mode. When a problem occurs, the administrator specifies which backup domain controller and USN to use. This is the only mode available when running tasks from the Identity Manager interface.

• Semi-auto mode. Semi-auto mode permits you to semi-automate the fail-over resolution process. In semi-auto mode, the task uses the collected data to identify the best backup domain controller and USN to use. It computes this by looking for a collection point that is closest to a derived TargetTimestamp without exceeding this value

  where TargetTimestamp = (FailureTimestamp - replicationTime)

  Semi-auto mode is not available from the Identity Manager Administrator interface.

Arguments

If you have determined that launching semi-auto failover is appropriate for a particular error, set the following task arguments. (The on-error workflow must launch the Active Directory Synchronization failover task.) Setting these arguments reconfigures the failed resource and the IAPI Object to use an alternate domain controller and usnChanged starting point.

To manually specify which domain controller to fail over to and which saved HighestCommittedUSN number to start from, set the following arguments.

Resource Object Changes

The Active Directory Recovery Collector task updates either the LDAPHostname or the GlobalCatalog resource attribute value (depending on which value is in use). If the search subdomains resource attribute is set to true, and the global catalog attribute value is not empty, the global catalog server attribute is changed. Otherwise, the LDAPHostname is changed to the name of the backup domain controller.

IAPI Object Changes

The Active Directory Recovery Collector task also updates the IAPI object so that the Active Directory resource adapter knows which changes to look for the next time it runs. The task updates the HighCommitedUSN value for both lastUpdated and lastDeleted attribute values.

Setting Up Active Directory Synchronization Failover

Step 1: Configure the Active Directory Synchronization Recovery Collector Task

• Configure the maximum number of hours to retain data. The default value is seven days. This value controls how far back the HighestCommittedUSN values are kept.

  You must configure one workflow per Active Sync resource that must be configured.

• Schedule this task from the Task page of the Identity Manager Administrator interface. The polling interval, which establishes how often to contact each host for their HighestCommittedUSN value, is set by the task schedule.

  When this task is executed, it calls out to the Active Directory adapter to retrieve the HighestCommittedUSN number from each domain controller’s rootDSE. It then stores this value in an Identity Manager configuration object. The task generates one configuration object per defined Active Sync resource to store alternate domain controller HighestCommittedUSN values.

Step 2: Define the Active Directory On-Error Process Active Sync Attribute

On each Active Directory Active Sync resource, Identity Manager defines an onError process that is called when a failure occurs during the synchronization of a resource. If an Active Directory resource defines an on-error process, this process is called if there errors occur when the poll method is called on the resource during active synchronization. This process checks the result from the IAPI objects, and if an error occurs, calls the defined process.

Configure this process to notify an administrator through email when an error occurs. Include the error text in the email body so that the administrator can determine if the error warrants that Identity Manager fails over to another domain controller.

Using the error text, the administrator is alerted to a potentially lengthy outage or an outrage due to a temporary, quickly resolved issue (such as a temporary routing issue that is resolved by the next poll attempt).

Step 3: Run Active Directory Synchronization Failover Task for the Failed Resource

If the domain controller returns an error that warrants failing over to another domain controller, run the Active Directory Synchronization Failover task from the Task page.

For manual fail-over mode, the fail-over task requests

• The name of the downed domain controller or resource

• The name of the DC hostname to fail over

• The timestamp of the collected HighestCommittedUSN value to use

You also must choose whether to restart Active Sync after the switch to a new domain controller is complete.

How the Task Works

When executed, the Active Directory Synchronization Failover task

Task Actions

1. Stops the Active Sync process on the failed resource

2. Reads in the fail over configuration object

3. Changes necessary resource attribute values

4. Optionally restarts the Active Sync process.

Example of Synchronization Failure Workflow

You can configure the following example workflow as the On Synchronization Failure Process resource attribute of an Active Directory resource. The workflow looks for the java.net.UnknownHostException error message. If it finds this message, the workflow launches a notification email to the administrator.

<TaskDefinition name=’Sample AD Sync On Error Workflow’
  executor=’com.waveset.workflow.WorkflowExecutor’
   syncControlAllowed=’true’ execMode=’sync’
   taskType=’Workflow’>
   <Extension>
      <WFProcess title=’Example AD Sync OnError Workflow’>
         <Variable name=’resultErrors’ input=’true’>
           <Comments>Errors returned from the resource.
           </Comments>
         </Variable>
        <Variable name=’resourceName’ input=’true’>
           <Comments>Name of the AD resource that returned the errors.
            </Comments>
         </Variable>
        <Variable name=’failureTimestamp’ input=’true’>
           <Comments>Failure timestamp, when it occurred.
           </Comments>
         </Variable>
        <Activity name=’start’>
           <Transition to=’checkErrors’/>
         </Activity>
         <Activity name=’checkErrors’>
           <Variable name=’criticalError’>
            <Comments>Local variable to hold if we need to notify
            </Comments>
           </Variable>
        <Action name=’iterateMessage’>
           <dolist name=’msg’>
             <ref>resultErrors</ref>
               <cond>
                 <match>
             <ref>msg</ref>
               <s>java.net.UnknownHostException</s>
                 </match>
                   <set name=’criticalError’>
                    <s>true</s>
                   </set>
               </cond>
           </dolist>
         </Action>
        <Transition to=’notify’>
           <notnull>
             <ref>criticalError</ref>
           </notnull>
         </Transition>
         <Transition to=’end’/>
         </Activity>
        <Activity name=’notify’>
           <Action application=’notify’>
             <Argument name=’template’    
                       value=’#ID#EmailTemplate:ADSyncFailoverSample’/>
             <Argument name=’resultErrors’ value=’$(resultErrors)’/>
           </Action>
         <Transition to=’end’/>
         </Activity>
        <Activity name=’end’/>
      </WFProcess>
    </Extension>
</TaskDefinition>