Data Exporter Architecture

When Data Exporter is enabled, Identity Manager stores each detected change to a specified object (data type) as a record in a table in the repository. At a configurable interval for each data type, the system executes two queries that select the records to export.

• The first query looks for persistent objects of the specified type in the repository that have changed since the last export. The Warehouse Task exports these records, determines the timestamp of the most recently-changed record, and uses this value as the starting point the next time the query is run.

• The second query searches the queue table. It locates all records of the specified data type, exports them, then deletes the records from the queue. Any records added after the query completes will be exported at the next cycle.

The exported records are not ordered. However, there are fields in the exported data that allow a subsequent query of the warehouse to put the data in chronological order.

In a typical deployment, Data Exporter writes data to a set of staging tables. Identity Manager provides SQL scripts that define these tables for each type of supported database. You do not need to modify these tables, unless your Identity Manager deployment contains extended attributes that need to be exported. However, if you have extended attributes that will be exported, then you must customize your export schema and compile your own factory class for handling these attributes. For more information, see Customizing Data Exporter.

Exporting data to staging tables allows you to write your own Extract, Transform, and Load (ETL) infrastructure so that the data can be processed for storage in a data warehouse, and ultimately, in a datamart. Timestamp manipulation is a commonly-implemented transformation. The system uses the java.sql.Timestamp format of YYYY-MM-DD hh:mm:ss. Although the day of the week is not explicitly specified in the timestamp, it can be extracted using a transformation.

If you do not need to transfer information to a warehouse and datamart, then you can consider the staging tables to be the final destination. In this case, be sure to use the same connection information for read and write operations. See the Business Administrator's Guide for information about configuring Data Exporter.

Forensic queries allow Identity Manager to read data that has been stored in the data warehouse (or staging tables in a simple environment). They can identify users or roles based on current or historical values of the user, role, or related data types. A forensic query is similar to a Find User or Find Role report, but it differs in that the matching criteria can be evaluated against historical data, and because it allows you to search attributes that are of data types other than the user or role being queried. See the Business Administrator's Guide for information about defining forensic query.

The following diagram illustrates the data flow when Data Exporter is enabled.

Figure 5–1 Data Exporter Data Flow