Chapter 6 Data Synchronization and Loading

Chapter 6
Data Synchronization and Loading

This chapter provides information and procedures for using Identity Manager data synchronization and loading features. You will learn about data synchronization tools (discovery, reconciliation, and synchronization) and how to use them to keep data current.

Data Synchronization Tools: Which to Use?

Discovery

Reconciliation

Active Sync Adapters

Data Synchronization Tools: Which to Use?

Follow these guidelines when selecting Identity Manager data synchronization tools to perform a task.

Table 6-1 Tasks to Use with the Data Synchronization Tools
If you want to:	Then choose this feature:
Initially pull resource accounts into Identity Manager, without viewing before loading	Load from Resource
Initially pull resource accounts into Identity Manager, optionally viewing and editing data before loading	Extract to File, Load from File
Periodically pull resource accounts into Identity Manager, taking action on each account according to configured policy	Reconcile with Resources
Push or pull resource account changes into Identity Manager	Synchronization using Active Sync adapters (multiple resource implementations)

Discovery

Identity Manager account discovery features help facilitate rapid deployment and speed account creation tasks. These features are:

Extract to File — Extracts the resource accounts returned by a resource adapter to a file (in CSV or XML format). You can manipulate this file before importing the data into Identity Manager.

Load from File — Reads accounts in a file (in CSV or XML format) and loads them into Identity Manager.

Load from Resource — Combines the other two discovery features, extracting accounts from a resource and loading them directly into Identity Manager.

Using these tools, you can create new Identity Manager users or correlate accounts on a resource with existing Identity Manager user accounts.

Extract to File

Use this feature to extract resource accounts from a resource to an XML or CSV text file. Doing this allows you to view and make changes to extracted data before importing it into Identity Manager.

To extract accounts:

From the menu bar, select Accounts, and then select Extract to File.

Select a resource from which to extract accounts.

Select a file format for the output account information. You can extract data to an XML file, or to a text file with account attributes arranged in comma-separated value (CSV) format.

Click Download. Identity Manager displays a File Download dialog, in which you may choose to save or view the extracted file.

If you choose to open the file, you might have to select a program to view it.

Load from File

Use this feature to load resource accounts — either those extracted from a resource through Identity Manager, or from another file source — into Identity Manager. A file created by the Identity Manager Extract to File feature is in XML format. If you are loading a list of new users, the data file typically is in CSV format.

About CSV File Format

Often, accounts to be loaded are listed in a spreadsheet and saved in comma-separated-value (CSV) format for loading into Identity Manager. CSV file contents must follow these format guidelines:

Line 1 — Lists column headings or schema attributes for each field, separated by commas.

Lines 2 to end — Lists values for each attribute defined in line 1, separated by commas. If data does not exist for a field value, that field must be represented by adjacent commas.

For example, the first three lines of a file might look like the file entries in the following figure:

firstname,middleinitial,lastname,accountId,asciipassword,EmployeeID,Depart ment,Phone
John,Q,Example,E1234,E1234,1234,Operations,555-222-1111
Jane,B,Doe,E1111,E1111,1111,,555-222-4444

Figure 6-1 Example of Properly Formatted CSV File for Loading Data
CVS formatted file content

In this example, the second user (Jane Doe) does not have a department. The missing value is represented by adjacent commas (,,).



Note	To enable the ability to apply identity attributes during a load operation, add Load from File to the list of enabled applications for the Identity Attributes, using the Meta View. When enabled, the load operation does not display the following options: User Form Update Attributes Merge Attributes If you select the Update Accounts option, then all identity attributes are processed fully and accounts are reprovisioned. Otherwise, only attributes that are sourced from the file being loaded and that flow to the Identity user are processed.

To load accounts:

From the menu bar, select Accounts, and then select Load from File.

Identity Manager displays the Load from File page, which lets you specify load options before continuing:

User Form — When load creates an Identity Manager user, the user form assigns an organization as well as roles, resources, and other attributes. Select the user form to apply to each resource account.

Account Correlation Rule — An account correlation rule selects Identity Manager users that might own each unowned resource account. Given the attributes of an unowned resource account, a correlation rule returns a list of names or a list of attribute conditions that will be used to select potential owners. Select a rule to look for Identity Manager users that may own each unowned resource account.

Account Confirmation Rule — An account confirmation rule eliminates any non-owner from the list of potential owners that the correlation rule selects. Given the full View of an Identity Manager user and the attributes of an unowned resource account, a confirmation rule returns true if the user owns the account, and false otherwise. Select a rule to test each potential owner of a resource account. If you select No Confirmation Rule, Identity Manager accepts all potential owners without confirmation.



Note	In your environment, if the correlation rule will select at most one owner for each account, then you do not need a confirmation rule.

Load Only Matching — Select to load into Identity Manager only those accounts that match an existing Identity Manager user. If you select this option, load will discard any unmatched resource account.

Update Attributes — Select to replace the current Identity Manager user attribute values with the attribute values from the account being loaded.

Merge Attributes — Enter one or more attribute names, separated by commas, for which values should be combined (eliminating duplicates) rather than overwritten. Use this option only for list-type attributes, such as groups and mailing lists. You must also select the Update Attributes option.

Result Level — Select a threshold at which the load process will record an individual result for an account:

Errors only — Record an individual result only when loading an account produces an error message.

Warnings and errors — Record an individual result when loading an account produces a warning or an error message.

Informational and above — Record an individual result for every account. This causes the load process to run more slowly.

In the File to Upload field, specify a file to load, and then click Load Accounts.



Note	If the input file does not contain a user column, you must select a confirmation rule for the load to proceed correctly. The task instance name associated with the load process is based on the input file name; therefore, if you re-use a file name, then the task instance associated with the latest load process will overwrite any previous task instances.

Figure 6-2 illustrates the fields and options available in the Load from File screen.

Figure 6-2 Load from File
Use the load process to load accounts from a file.

If an account matches (or correlates with) an existing user, the load process will merge the account into the user. The process will also create a new Identity Manager user from any input account that does not correlate (unless Correlation Required is specified).

The bulkAction.maxParseErrors configuration variable sets a limit on the number of errors that can be found when a file is loaded. By default, the limit is 10 errors. If the maxParseErrors number of errors is found, then parsing stops.

Load from Resource

Use this feature to directly extract and import accounts into Identity Manager according to the load options you specify.

To import accounts, select Accounts from the menu bar, and then select Load from Resource.

Identity Manager lets you specify load options before continuing. Load options available from the Load from Resource page, and the actions that result, are the same as those on the Load from File page.



Note	To enable the ability to apply identity attributes during a load operation, add Load from Resource to the list of enabled applications for the Identity Attributes. When enabled, the load operation does not display the following options: User Form Update Attributes Merge Attributes If you select the Update Accounts option, then all identity attributes are processed fully and accounts are reprovisioned. Otherwise, only attributes being loaded that are sourced from the resource and that flow to the Identity user are processed.

Reconciliation

Use the reconciliation feature to highlight inconsistencies between the resource accounts on Identity Manager and the accounts that actually exist on a resource, and to periodically correlate account data.

Because reconciliation is designed for ongoing comparison, it has the following characteristics:

Diagnoses account situations more specifically and supports a wider range of responses than the discovery process

Can be scheduled (discovery cannot)

Offers an incremental mode (discovery is always full mode)

Can detect native changes (discovery cannot)

You can also configure reconciliation to launch an arbitrary workflow at each of the following points in processing a resource:

Before reconciling any account

For each account

After reconciling all accounts

Access Identity Manager reconciliation features from the Resources area. The Resources list shows when each resource was last reconciled and its current reconciliation status.

About Reconciliation Policies

Reconciliation policies allow you to establish a set of responses, by resource, for each reconciliation task. Within a policy, you select the server to run reconciliation, determine how often and when reconciliation takes place, and set responses to each situation encountered during reconciliation. You can also configure reconciliation to detect changes made natively (not made through Identity Manager) to account attributes.

Editing Reconciliation Policies

To edit a reconciliation policy:

Select Resources from the menu bar.

Select a resource in the Resources list hierarchy.

Select Edit Reconciliation Policy from the Resource Actions options list.

Identity Manager displays the Edit Reconciliation Policy page, where you can make these policy selections:

Reconciliation Server — In a clustered environment, each server may run reconciliation. Specify which Identity Manager server will run reconciliation against resources in the policy.

Reconciliation Modes — Reconciliation can be performed in different modes, which optimize different qualities:

Full reconciliation — Optimizes for thoroughness at a cost of speed.

Incremental reconciliation — Optimizes for speed at the expense of some thoroughness.

Select the mode in which Identity Manager should run reconciliation against resources in the policy. Select Do not reconcile to disable reconciliation for targeted resources.

Full Reconciliation Schedule — If full mode reconciliation is enabled, it is performed automatically on a fixed schedule. Specify how frequently full reconciliation should be run against resources in the policy. Select the Inherit option to inherit the indicated schedule from a higher-level policy. Deselect the Inherit option to specify the schedule or specify a Task Schedule Repetition rule.

Incremental Reconciliation Schedule — If incremental mode reconciliation is enabled, it is performed automatically on a fixed schedule. The schedule is inherited from a higher-level policy, when the inherit default policy option is selected. To specify how frequently to run incremental reconciliation against resources in the policy, or to select a Task Schedule Repetition rule, deselect the inherit option.



Note	Not all resources support incremental reconciliation.

Attribute-level Reconciliation — Reconciliation can be configured to detect changes made natively (that is, not made through Identity Manager) to account attributes. Specify whether reconciliation should detect native changes to the attributes specified in Reconciled Account Attributes.

Account Confirmation Rule — An account confirmation rule eliminates any non-owner from the list of potential owners that the correlation rule selects. Given the full View of an Identity Manager user and the attributes of an unowned resource account, a confirmation rule returns true if the user owns the account and false otherwise. Select a rule to test each potential owner of a resource account. If you select No Confirmation Rule, Identity Manager accepts all potential owners without confirmation.



Note	In your environment, if the correlation rule will select at most one owner for each account, then you do not need a confirmation rule.

Proxy Administrator — Specify the administrator to use when reconciliation responses are performed. The reconciliation can perform only those actions that the designated proxy administrator is permitted to do. The response will use the user form (if needed) associated with this administrator.

You can also select the No Proxy Administrator option. When selected, reconciliation results are available to view, but no response actions or workflows are run.

Situation Options (and Response)— Reconciliation recognizes several types of situations. Specify in the Response column any action reconciliation should take:

CONFIRMED — The expected account exists.

DELETED — The expected account does not exist.

FOUND — The reconciliation process found a matching account on an assigned resource.

MISSING — No matching account exists on a resource assigned to the user.

COLLISION — Two or more Identity Manager users are assigned the same account on a resource.

UNASSIGNED — The reconciliation process found a matching account on a resource not assigned to the user.

UNMATCHED — The account does not match any users.

DISPUTED — The account matches more than one user.

Select from one of these response options (available options vary by situation):

Create new Identity Manager user based on resource account — Runs the user form on the resource account attributes to create a new user. The resource account is not updated as a result of any changes.

Create resource account for Identity Manager user — Recreates the missing resource account, using the user form to regenerate the resource account attributes.

Delete resource account and Disable resource account — Deletes/disables the account on the resource.

Link resource account to Identity Manager user and Unlink resource account from Identity Manager user — Adds or removes the resource account assignment to or from the user. No form processing is performed.

Pre-reconciliation Workflow — Reconciliation can be configured to run a user-specified workflow prior to reconciling a resource. Specify the workflow that reconciliation should run. Select Do not run workflow if no workflow should be run.

Per-account Workflow — Reconciliation can be configured to run a user-specified workflow after responding to the situation of a resource account. Specify the workflow that reconciliation should run. Select Do not run workflow if no workflow should be run.

Post-reconciliation Workflow — Reconciliation can be configured to run a user-specified workflow after completing reconciliation for a resource. Specify the workflow that reconciliation should run. Select Do not run workflow if no workflow should be run.

Click Save to save policy changes.

Starting Reconciliation

Two options are available for starting reconciliation tasks:

Reconciliation schedule — You can set a reconciliation schedule on the Edit Reconciliation Policy page, which runs reconciliation at regular intervals.

Immediate reconciliation — Runs reconciliation immediately. To do this, select a resource in the resources list, and then select one of the following options in the Resource Actions list:

Full Reconcile Now

Incremental Reconcile Now

Reconciliation will run according to the parameters you have set in the policy. If the policy has a regular schedule set for reconciliation, it will continue to run as specified.

Canceling Reconciliation

To cancel reconciliation, select the resource, and then select Cancel Reconciliation from the Resource Actions list.

Viewing Reconciliation Status

The Status column in the Resources list reports several reconciliation status conditions. These are:

unknown — Status is not known. Results for the latest reconciliation task are not available.

disabled — Reconciliation is disabled.

failed — The latest reconciliation failed to complete.

success — The latest reconciliation completed successfully.

completed with errors — The latest reconciliation completed, but with errors.



Note	You must refresh this page to view changes to status (the information does not automatically refresh).

Detailed status information for each account on a resource is available. Select a resource in the list, and then select View Reconciliation Status from the Resource Actions list.

Working with the Account Index

The Account Index records the last known state of each resource account known to Identity Manager. It is primarily maintained by reconciliation, but other Identity Manager functions will also update the Account Index, as needed.

Discovery tools do not update the Account Index.

Searching the Account Index

To search the account index, select Search Account Index from the Resource Actions list.

Select a search type, and then enter or select search attributes. Click Search to find accounts that match all search criteria.

Resource account name — Select this option, select one of the modifiers (starts with, contains, or is), and then enter part or all of an account name.

Resource is one of — Select this option, and then select one or more resources from the list to find reconciled accounts that reside on the specified resources.

Owner — Select this option, select one of the modifiers (starts with, contains, or is), and then enter part or all of an owner name. To search for unowned accounts, search for accounts in the UNMATCHED or DISPUTED situation.

Situation is one of — Select this option, and then select one or more situations from the list to find reconciled accounts in the specified situations.

Click Search to search for accounts according to your search parameters. To limit the results of the search, optionally specify a number in the Limit results to first field. The default limit is the first 1000 accounts found.

Click Reset Query to clear the page and make new selections.

Examining the Account Index

It is also possible to view all Identity Manager user accounts and optionally reconcile them on a per-user basis. To do this, select Resources, and then select Examine Account Index.

The table displays all of the resource accounts that Identity Manager knows about (whether or not an Identity Manager user owns the account). This information is grouped by resource or by Identity Manager organization. To change this view, make a selection from the Change index view list.

Working with Accounts

To work with the accounts on a resource, select the Group by resource index view. Identity Manager displays folders for each type of resource. Navigate to a specific resource by expanding a folder. Click + or - next to the resource to display all resource accounts that Identity Manager knows about.

Accounts that have been added directly to the resource since the last reconciliation on that resource are not displayed.

Depending on the current situation of a given account, you may be able to perform several actions. You can also view account details or choose to reconcile that one account.

Working with Users

To work with Identity Manager users, select the Group by user index view. In this view, Identity Manager users and organizations are displayed in a hierarchy similar to the Accounts List page. To see accounts currently assigned to a user in Identity Manager, navigate to the user and click the indicator next to the user name. The user’s accounts and the current status of those accounts that Identity Manager knows about are displayed under the user name.

Depending on the current situation of a given account, you may be able to perform several actions. You can also view account details or choose to reconcile that one account.

Active Sync Adapters

The Identity Manager Active Sync feature allows information that is stored in an authoritative external resource (such as an application or database) to synchronize with Identity Manager user data. Configuring synchronization for an Identity Manager resource enables it to listen or poll for changes to the authoritative resource.

You can configure how resource attribute changes are flowed into Identity Manager by using the Meta View, or by specifying the Input Form in the resource’s synchronization policy (for the appropriate target object type).

Using the Meta View to specify how data will be updated, specify the identity attributes to enable for the Active Sync application. For more information about configuring identity attributes, see Configuring Identity Attributes and Events.

Continue to the next section to configure synchronization.

Configuring Synchronization

Identity Manager uses a Synchronization Policy to enable synchronization for resources. To configure synchronization, on the Resources tab select the resource for which you want to configure synchronization and then select Edit Synchronization Policy from the Resource Actions list.

Editing the Synchronization Policy

Specify the following options in the Edit Synchronization Policy page to configure synchronization:

Target Object Type — Select the type of users to which the policy applies, either Identity Manager Users or Service Provider Edition Users.



Note	In a Service Provider implementation you must configure a synchronization policy (with Service Provider Edition Users specified as the object type) to enable synchronization of data for those users. For more information about service provider users, see Chapter 13, "Service Provider Administration."

Scheduling Settings — Use this section to specify the startup method and polling schedule.

Startup Type can be Manual, Automatic, Automatic with Failover, or Disabled:

Automatic or Automatic with failover — Starts the authoritative source when the Identity system is started.

Manual — Requires that an administrator start the authoritative source.

Disabled — Disables the resource.

Use the Start Date and Start Time options to specify when polling begins. Specify the polling cycles by selecting an interval and entering a value for the interval (seconds, minutes, hours, days, weeks, months).

If you set a polling start date and time that is in the future, polling will begin when specified. If you set a polling start date and time that is in the past, Identity Manager determines when to begin polling based on this information and the polling interval. For example:

You configure active synchronization for the resource on July 18, 2005 (Tuesday)

You set the resource to poll weekly, with a start date of July 4, 2005 (Monday) and time of 9:00 a.m.

In this case, the resource will begin polling on July 25, 2005 (the following Monday).

If you do not specify a start date or time, then the resource will poll immediately. If you take this approach, each time the application server is restarted, all resources configured for active synchronization will begin polling immediately. The typical approach, is to set a start date and time.

Synchronization Servers — In a clustered environment, each server can run synchronization. Select an option to specify which servers will be used to run synchronization for the resource.

Select Use any available server if it does not matter where synchronization runs. A server will be chosen from the set of possible servers when synchronization starts.

Select Use the settings in waveset.properties to use servers specified there to run synchronization. (This feature is deprecated.)

Select Use specified servers, and then select one or more available servers from the Synchronization Servers list, to select specific servers to run synchronization.

Resource Specific Settings — Use this section to specify how synchronization will determine the data to be processed for the resource.

Common Settings — Specify the following general settings for data synchronization activities:

Proxy Administrator — Select the administrator who will process updates. All actions will be authorized through capabilities assigned to this administrator. You should select a proxy administrator with an empty user form.

Input Form — Select an input form that will process data updates. This optional configuration item allows attributes to be transformed before they are saved on the accounts.

Rules — You have the option of specifying rules to use during the data synchronization process:

Process Rule — Select this rule to specify a process rule to run for each incoming account. This selection overrides all other options. If you specify a process rule, the process will be run for every row, regardless of other settings on the resource. It can be either a process name, or a rule evaluating to a process name.

Correlation Rule — Select a correlation rule to override the correlation rule specified in the resource's reconciliation policy. Correlation rules correlate resource accounts to Identity system accounts.

Confirmation Rule — Select a confirmation rule to override the confirmation rule specified in the resource's reconciliation policy.

Resolve Process Rule — Select this rule to specify the name of a Task Definition to run in case of multiple matches to a record in the data feed. This should be a process that prompts an administrator for manual action. It can be a process name or a rule evaluating to a process name.

Delete Rule — Select a rule, which returns true or false, that will be evaluated for each incoming user update to determine if a delete operation should occur.

Create Unmatched Accounts — When this option is enabled (true), the adapter will attempt to create accounts that it does not find in the Identity Manager system. If not enabled, the adapter will run the account through the process returned by the Resolve Process Rule.

Logging Settings — Specify a value for the following logging options:

Maximum Log Archives — If greater than zero, retain the latest N log files. If zero, then a single log file is re-used. If -1, then log files are never discarded.

Maximum Active Log Age — After this period of time has elapsed, the active log will be archived. If the time is zero, then no time-based archival will occur. If Maximum Log Archives is zero, then the active log will instead be truncated and re-used after this time period. This age criteria is evaluated independently of the time criteria specified by Maximum Log File Size.

Enter a number, and then select the unit of time (Days, Hours, Minutes, Months, Seconds, or Weeks). Days is the default unit.

Log File Path — Enter the path to the directory in which to create the active and archived log files. Log file names begin with the resource name.

Maximum Log file Size — Enter the maximum size, in bytes, of the active log file. The active log file will be archived when it reaches maximum size. If Maximum Log Archives is zero, then the active log will instead be truncated and re-used after this time period. This size criteria is evaluated independently of the age criteria specified by Maximum Active Log Age.

Log Level — Enter the level of logging:

0 — no logging

1 — error

2 — information

3 — verbose

4 — debug

Click Save to save the policy settings for the resource.

Editing Active Sync Adapters

Before editing an Active Sync adapter, stop synchronization. From the Edit Synchronization Policy page, select Disabled as Startup Type for Identity Manager users; for Service Provider users deselect the Enable Synchronization option. A warning message will appear to indicate that active synchronization is disabled.

Disabling synchronization for a resource will result in stopping the synchronization task when the changes are saved.

Tuning Active Sync Adapter Performance

Since synchronization is a background task, ActiveSync adapter configuration can affect server performance. Tuning ActiveSync adapter performance involves these tasks:

Changing Polling Intervals

Specifying the Host Where the Adapter Will Run

Starting and Stopping

Adapter Logging

Manage Active Sync adapters through the resources list. Select an Active Sync adapter, and then access start, stop, and status refresh controls actions from the Synchronization section of the Resource Actions list.

Changing Polling Intervals

The polling interval determines when the Active Sync adapter will start processing new information. Polling intervals should be determined based on the type of activity being performed. For example, if the adapter reads in a large list of users from a database and updates all users in Identity Manager each time, consider running this process daily in the early morning hours. Some adapters may have a quick search for new items to process and could be set to run every minute.

Specifying the Host Where the Adapter Will Run

To specify the host where the adapters will run, edit the waveset.properties file. Edit the sources.hosts property to either of the following options:

Set sources.hosts=hostname1,hostname2,hostname3. This lists the host names of machines to run Active Sync adapters. The adapter will run on the first available host listed in this field.



Note	The hostname you enter must match an entry in the Identity Manager list of servers. View the list of servers from the Configure tab.

Set sources.hosts=localhost. With this setting the adapter will run on the first Identity Manager server that attempts to start Active Sync for the resource.



Note	In a cluster you should use the first option if you need to specify a specific server. This property setting applies only to Identity Manager user synchronization. Host configuration for Service Provider user synchronization is determined by the Synchronization Policy.

Active Sync adapters that require more memory and CPU cycles can be configured to run on dedicated servers to help load balance the systems.

Starting and Stopping

Active Sync adapters can be disabled, manually started, or automatically started. You must have the appropriate administrator capability to change Active Sync resources in order to start or stop Active Sync adapters. For information about administrator capabilities, see Capabilities Categories.

When an adapter is set to automatic, the adapter restarts when the application server does. When you start an adapter, it will run immediately and execute at the specified polling interval. When you stop an adapter, the next time the adapter checks for the stop flag, it will stop.

Adapter Logging

Adapter logs capture information about the adapter currently processing. The amount of detail that the log captures depends upon the logging level of the logging you have set. Adapter logs are useful for debugging problems and watching the adapter process progress.

Each adapter has its own log file, path, and log level. You specify these values in the Logging section of the Synchronization Policy for the appropriate user type (Identity Manager or Service Provider).

Deleting Adapter Logs

Adapter logs should be deleted only when the adapter has been stopped. In most cases, make a copy of the log for archive purposes before deleting a log.

Previous Contents Index Next

Previous Contents Index Next
Sun Java[TM] System Identity Manager 7.1 Admininstration