About the Flat File Connector

1 About the Flat File Connector

The Flat File connector integrates Oracle Identity Governance with files of formats such as CSV and XML.

The following topics provide a high-level overview of the connector:

1.1 Introduction to the Connector

Oracle Identity Governance is a centralized identity management solution that provides self service, compliance, provisioning and password management services for applications residing on-premises or on the Cloud. Oracle Identity Governance connectors are used to integrate Oracle identity Governance with the external identity-aware applications.

The Flat File connector lets you onboard flat file-based applications in Oracle Identity Governance.

Note:

In this guide, the connector that is deployed using the Applications option on the Manage tab of Identity Self Service is referred to as an AOB application. The connector that is deployed using the Manage Connector option in Oracle Identity System Administration is referred to as a CI-based connector (Connector Installer-based connector).

Application onboarding is the process of registering or associating an application with Oracle Identity Governance and making that application available for provisioning and reconciliation of user information.

Enterprise applications generally support the export of users in the form of a file. Some widely used file formats are CSV, LDIF, and XML. The connector consumes the information in a flat file, thereby enabling the import of this data as Oracle Identity Governance user accounts or entitlements. You can use the flat file connector in a number of situations for offline data loading or when a predefined connector is not available.

By default, this connector supports processing of flat files in the CSV format. If you want to use this connector to process flat files in formats other than CSV, then you must create a custom parser and integrate it with the connector.

1.2 Certified Components

These are the software components and their versions required for installing and using the connector.

Table 1-1 Certified Components

Item	Requirement for AOB Application	Requirement for CI-Based Connector
Oracle Identity Governance or Oracle Identity Manager	Oracle Identity Governance 12c PS4 (12.2.1.4.0)	You can use one of the following releases: Oracle Identity Governance 12c PS3 (12.2.1.3.0) Oracle Identity Manager 11g Release 2 PS3 (11.1.2.3.0) and later
Target System	Any enterprise system that can export users, accounts, or entitlements to a flat file.	Any enterprise system that can export users, accounts, or entitlements to a flat file.
Connector Server	11.1.2.1.0	11.1.1.5.0 or later Note: You can download the necessary Java Connector Server from the Oracle Technology Network web page.
Connector Server JDK	JDK 1.8 or later	JDK 1.8 or later
Flat File format	CSV Note: Formats other than CSV are supported through the use of custom parsers.	CSV Note: Formats other than CSV are supported through the use of custom parsers.

1.3 Certified Languages

The connector supports the languages that are supported by Oracle Identity Governance.

Resource bundles are not part of the connector installation media as the resource bundle entries vary depending on the flat file being used. You can localize field labels in UI forms as described in Localizing Field Labels in UI Forms.

1.4 Usage Recommendation

These are the recommendations for the Flat File connector versions that you can deploy and use depending on the Oracle Identity Governance or Oracle Identity Manager version that you are using.

If you are using Oracle Identity Governance 12c PS4 (12.2.1.4.0), then use the latest 12.2.1.x version of this connector. Deploy it using the Applications option on the Manage tab of Identity Self Service.
If you are using Oracle Identity Governance 12c PS3 (12.2.1.3.0) or Oracle Identity Manager 11g Release 2 PS3 (11.1.2.3.0) and later, then use the 11.1.1.x version of the Flat File connector. If you want to use the 12.2.1.x version of this connector, then you can install and use it only in the CI-based mode. If you want to use the AOB application, then you must upgrade to Oracle Identity Governance release 12.2.1.4.0.

Note:

If you are using the latest 12.2.1.x version of the Flat File connector in the CI-based mode, then see Oracle Identity Manager Connector Guide for Flat File, Release 11.1.1 for complete details on connector deployment, usage, and customization.

1.5 Connector Architecture

The Flat File connector is a generic solution to retrieve records from flat files that are exported from various enterprise target systems. This connector is implemented using the Identity Connector Framework (ICF) component.

These flat files can be of various formats such as CSV, LDIF, XML, and so on. The connector focuses only on the reconciliation of records from a flat file. The connector installation package contains scheduled jobs that you can use to load users, accounts, and entitlements from a flat file into an existing resource in Oracle Identity Governance.

Figure 1-1 shows the connector integrating the flat files exported from an enterprise target system with Oracle Identity Governance.

Figure 1-1 Architecture of the Connector

Description of "Figure 1-1 Architecture of the Connector"

You must place the flat files exported from the enterprise target system in a directory that is accessible from Oracle Identity Governance. If you are using a Connector Server, then place the exported flat files on the computer hosting the Connector Server. The connector sorts the exported flat files within the directory in an alphanumeric manner, and then processes each file based on this order.

The location of the directory containing the flat file is specified in the attributes of a scheduled job. When a scheduled job is run, it calls the connector's search implementation, which in turn returns the connector objects to Oracle Identity Governance.

1.6 Use Cases Supported by the Connector

These are the scenarios in which you can use the connector.

1.6.1 Reconciliation of Records

Reconciling records from a flat file exported from an enterprise target system involves loading data from a flat file into Oracle Identity Manager.

You can perform the following operations in this scenario:

Reconciliation
Certification

Here, the Flat File connector can be used to perform reconciliation runs.

The following example shows how the Flat File connector can be used to load data from a flat file into Oracle Identity Governance to perform certification tasks.

Suppose John works as a Compliance Administrator in ACME Corporation. He uses Oracle Identity Governance to define roles, automate certification processes, and generate business structure reports for auditing. He has a list of users in his enterprise and their entitlements in the form of a CSV file, and he wants to import this data into Oracle Identity Governance, to use this data purely for certification purposes. He needs to create resource objects and forms for all the users, and import the data into these tables.

In the preceding example, by using the flat file connector, John can load accounts from a flat file into a Flat File Resource. He can run the corresponding reconciliation jobs of the flat file to import data from the CSV file into Oracle Identity Governance.

1.6.2 Disconnected Resource

Disconnected resources are targets for which a predefined connector does not exist. Therefore, the provisioning fulfillment for disconnected resources is not automated, but manual.

You can perform the following operations in this scenario:

Request
Manual fulfillment or provisioning
Reconciliation
Certification

Here, the Flat File connector can be used to perform reconciliation runs and provisioning operations.

The following example shows how the Flat File connector can be used to load data from a flat file into Oracle Identity Governance for disconnected resources.

Suppose Smith is the chief librarian in the University of Utopia. His responsibilities include providing library access cards to the students of the university. He has a file with the list of students who already have library cards. He wants to transfer this list to Oracle Identity Governance after which he can automate the library transactions for existing members.

In the preceding example, as library cards are modeled as a disconnected resource in Oracle Identity Governance, he can create an application for the disconnected resource, and then load accounts from a flat file into a Library Card Resource using the corresponding reconciliation jobs. By defining a disconnected resource through Oracle Identity Governance, Smith can start reconciling users from the flat file and link them to the desired disconnected resource.

1.6.3 Connected Resource

Connected resources are targets for which a predefined connector is available, for example, Microsoft Active Directory.

You can perform the following operations in this scenario:

Request
Automatic fulfillment or provisioning
Reconciliation
Certification

Here, the Flat File connector can be used only to perform reconciliation runs.

The following example shows how the Flat File connector can be used to load data from a flat file into Oracle Identity Governance, although a predefined connector is available.

Suppose Jane works as a Network Administrator at Example Multinational Inc. In Example Multinational Inc., she performs identity and access management tasks on users within the organization. One of Jane's responsibilities is to create and maintain users in Oracle Identity Governance, and to provision these users with resources. At Example Multinational Inc., all the employee details are maintained in the Microsoft Active Directory target system. Jane wants to reconcile about 100,000 user records from the target system to her Oracle Identity Governance instance, as soon as possible. As the AD Server is planned for a maintenance shutdown, she is looking for a means for offline loading of all the user data which has been exported in the form of an LDIF file. Given the time and network constraints, Jane needs a solution for the initial on-boarding of the users into Oracle Identity Governance.

In the preceding example, performing an initial reconciliation or full reconciliation, is a performance and time-intensive operation. Using the Microsoft Active Directory User Management connector to perform the reconciliation operation requires the connection between the target system and Oracle Identity Governance to remain active. In other words, offline loading of users cannot be performed. In this scenario, a native flat file dump from the target system can be used by the Flat File connector to quickly reconcile the users into Oracle Identity Governance.

1.7 Features of the Connector

The features of the connector include support for custom parsers, fault handling, archival, connector server, transformation and validation of account data, full, incremental, limited, and batched reconciliation, and so on.

1.7.1 Support for Both Target Resource and Trusted Source Reconciliation

You can configure the exported flat file as a Target application or an Authoritative application for reconciliation of records into Oracle Identity Governance.

See Providing Settings Information for a Disconnected Resource and Providing Settings Information for a Connected Resource for more information about the reconciliation jobs that are created when you create the application and their details.

1.7.2 Full and Incremental Reconciliation

After you create the application, you can perform full reconciliation to load all existing user data from the flat file to Oracle Identity Governance.

Any new files that are added after the first full reconciliation run are considered as a source of incremental data. Alternatively, incremental reconciliation can also be performed by explicitly providing the incremental data alone.

You can perform a full reconciliation run at any time. See Performing Full and Incremental Reconciliation for more information.

1.7.3 Limited Reconciliation

You can set a reconciliation filter as the value of the Filter attribute of the scheduled jobs. This filter specifies the subset of newly added and modified enterprise target system records that must be reconciled.

See Performing Limited Reconciliation for more information.

1.7.4 Support for Disconnected Resources

The connector provides support for disconnected resources by generating all artifacts associated with disconnected resources.

In addition, it generates process definitions associated with the default SOA composites that are required for performing manual provisioning. This eliminates the need to manually create disconnected resources and mappings between fields in Oracle Identity Governance and corresponding target system attributes.

To configure your flat file as a disconnected resource, see Creating an Application for a Disconnected Resource By Using the Flat File Connector

1.7.5 Support for Archival

The connector supports archival of the processed flat files.

You can specify the archive directory location in the Archive Directory parameter while configuring the reconciliation jobs, and the connector moves the files from the source directory to the specified location, once all files are processed.

If you do not specify a value for this parameter, then the connector creates a directory named "archived" within the directory containing the flat file, and saves the processed files in this location.

The Oracle Identity Governance Administrator must have read and write permissions on the Archive directory location.

The connector saves the processed flat file in the following format:

FILENAME_dd-MM-yyyy_HH-mm-ss.zip

In this format:

FILENAME is the name of the flat file being archived. If the directory with the flat file that is being processed contains more than one flat file, then FILENAME is the name of the first flat file from the alpha-numerically sorted list of flat files in the directory.
dd-MM-yyyy_HH-mm-ss is the date and time at which the flat file was archived.

For example, if the flat file has been exported from an enterprise target system, then the filename is saved in the following format:

acmeusers_29-08-2013_22-44-12.zip

When the archive location is specified, the connector moves all the files from the source directory irrespective of whether the file processing was successful or not. In case of errors, the connector writes the failed records to a separate file and this file is saved in the "failed" directory under the Flat File directory.

See Support for Fault Handling for more information about files in the "failed" directory.

1.7.6 Support for Custom Parsers

By default, the connector supports processing of flat files exported in the CSV format. To support the processing of flat files exported in formats other than CSV, you must create a custom parser and integrate it with the connector.

By default, the connector installation media contains the CSVParser.

See Configuring Custom Parsers for more information about custom parsers.

1.7.7 Support for Reconciling Complex Multivalued Data

The connector supports the reconciliation of complex multivalued data in the form of child forms containing single and multiple fields.

The child form data must be in the same file as the parent form data. The child form values are separated by customizable delimiters.

For example, in CSV files, every line in the flat file represents a single record which includes the parent and the child form data.

See Configuring the Flat File for Reconciliation of Complex Multivalued Data for more information.

1.7.8 Support for Delimiters

The connector supports the use of single character delimiters, which are used to separate values in a record.

By default, the connector supports comma (,) as a fieldDelimiter, semicolon (;) as a multiValueDelimiter, and number sign (#) as a subFieldDelimiter. If the exported flat file uses other characters as delimiters, then specify them as the values for the fieldDelimiter, multiValueDelimiter, and subFieldDelimiter parameters of the Advanced Settings section.

You must specify the Space or tab characters as space or tab respectively. Other multibyte characters (characters in different locale) can be directly entered in the Advanced Settings section of the respective locale.

Note:

The connector does not support multicharacter delimiters. For example, the use of characters $# together as a delimiter is not supported.

In the following sample multivalued data, the data has been presented in the following format, separated by delimiters:

AccountID,FirstName,LastName,Email,Languages,Roles

"111","John","Doe","john.doe@example.com","English;French;Spanish","Administrator#6-Dec-2013;Backup Operator#7-Nov-2013"

Here, comma (,) is a fieldDelimiter, semicolon (;) is a multiValueDelimiter, and number sign (#) is a subFieldDelimiter.

Figure 1-2 shows sample multivalued data separated by delimiters.

Figure 1-2 Sample Multivalued Data Separated by Delimiters

Description of "Figure 1-2 Sample Multivalued Data Separated by Delimiters"

See Providing Basic Information for a Connected Resource or Providing Basic Information for a Disconnected Resource for more information about the fieldDelimiter, multiValueDelimiter, and subFieldDelimiter parameters.

1.7.9 Support for Comment Characters

You can configure the connector to ignore the processing of lines that begin with certain characters such as #,$, and so on.

These configurable characters are considered as comment characters, and sentences beginning with such characters are considered as comments. The connector implementation skips any lines that start with the configured comment character.

You can configure this by adding an attribute named commentCharacter in the Advanced Settings section of your flat file application as described in Configuring the Connector to Ignore Comment Characters.

1.7.10 Support for Fault Handling

The connector logs record-level errors in a separate file while parsing the flat file. This log file is saved in a directory named "failed" that the connector creates, within the flat file directory.

The connector saves the processed flat file in the following format:

FILENAME_dd-MM-yyyy_HH-mm-ss.EXT

In this format:

FILENAME is the name of the flat file being archived.
dd-MM-yyyy_HH-mm-ss is the date and time at which the connector started processing the file.
EXT is the extension of the file.

For example, the filename will be saved in the following format:

acmeusers_29-08-2013_22-44-12.csv

The error file contains all those records that were not processed due to validation or data errors. The connector also appends the reason for failure as a separate attribute in the error file for future reference. Since the error file contains the existing attributes of the failed record, you can modify the same file to fix the data errors and load it back using the connector to reconcile the failed records alone. The Oracle Identity Governance Administrator must have read and write permissions on the Flat File directory and Archive directory locations.

1.7.11 Support for Preprocess and Postprocess Handlers

Preprocess and postprocess tasks can be run both before and after the reconciliation of accounts respectively.

You can use these tasks to perform any job on the flat file directory, like zipping and unzipping files, encryption and decryption of the complete file dumps or specific fields in the files, virus scan of the files, or any other tasks limited only by the implementation of these tasks.

See Configuring Preprocess and Postprocess Tasks for more information.

1.7.12 Transformation and Validation of Account Data

You can configure transformation and validation of account data that is brought into or sent from Oracle Identity Governance during reconciliation and provisioning operations by writing Groovy scripts while creating your application.

For more information, see Validation and Transformation of Provisioning and Reconciliation Attributes in Oracle Fusion Middleware Performing Self Service Tasks with Oracle Identity Governance.

1.7.13 Support for the Connector Server

Connector Server is one of the features provided by ICF. By using one or more connector servers, the connector architecture permits your application to communicate with externally deployed bundles.

A Java connector server is useful when you do not wish to execute a Java connector bundle in the same VM as your application. It can be beneficial to run a Java connector on a different host for performance improvements.

For information about installing, configuring, and running the Connector Server, and then installing the connector in a Connector Server, see Using an Identity Connector Server in Oracle Fusion Middleware Developing and Customizing Applications for Oracle Identity Governance.