| Oracle® Fusion Middleware Developer's Guide for Oracle Identity Manager 11g Release 1 (11.1.1) Part Number E14309-09 |
|
|
PDF · Mobi · ePub |
| Oracle® Fusion Middleware Developer's Guide for Oracle Identity Manager 11g Release 1 (11.1.1) Part Number E14309-09 |
|
|
PDF · Mobi · ePub |
Oracle Identity Manager may be one among many repositories of user data in your organization. When you start using Oracle Identity Manager, you might want to load data from the other repositories into Oracle Identity Manager. The Bulk Load utility offers a solution to this requirement.
The Bulk Load utility is aimed at automating the process of loading a large amount of data into Oracle Identity Manager. It helps reduce the downtime involved in loading data. You can use this utility after you install Oracle Identity Manager or at any time during the production lifetime of Oracle Identity Manager.
This document is divided into the following sections:
The following are features of the bulk load utility:
The utility is compatible with Oracle Identity Manager release 9.1.0 and later.
Data can be loaded into Oracle Identity Manager as OIM Users, accounts allocated (provisioned) to OIM Users, roles, role hierarchies, role memberships, or role categories.
Data can be loaded from a single or multiple CSV files or a database table. Data imported into Oracle Identity Manager is automatically converted into OIM Users, accounts provisioned to OIM Users, roles, role hierarchies, role memberships, or role categories.
Data can be loaded from a single or multiple trusted sources.
Data can be loaded into either an empty Oracle Identity Manager repository or an Oracle Identity Manager repository that already contains data about OIM Users and resources. In other words, user data can be loaded at any time, either immediately after Oracle Identity Manager installation or when the system is already in production.
Exceptions generated during user data loading are handled, and records that fail the loading process can be retried.
Audit snapshots can be generated after a bulk load operation.
After bulk loading of OIM User data, password change at first login is enforced because a dummy password is used during the operation.
Note:
You cannot use the utility to encrypt user attributes. In other words, if a user field in Oracle Identity Manager is encrypted, then the utility cannot be used to encrypt data that is loaded into that field.
To install the utility:
Zip and copy the following directory from the installation package into a directory on the Oracle Identity Manager database host computer:
MIDDLEWARE_HOME/Oracle_IDM1/server/db/oim/oracle/Utilities/oimbulkload
Note:
You can run the utility from a remote host. It is not mandatory to run the utility from a directory in the Oracle Identity Manager database host.
Extract the contents of the ZIP file.
The oimbulkload directory is created when you extract the contents of the ZIP file. The following directories are created inside this directory:
sqls: This directory contains SQL scripts used during bulk load operations.
scripts: This directory contains the .sh and .bat scripts used during bulk load operations.
csv_files: If you are going to use a single or multiple CSV files as the input source, then the CSV files must be stored in this directory.
lib: The directory contains the oimBulkLoad.jar file.
sample_data: This directory contains the following sample CSV files:
For OIM User load operations:
master.txt
OIDusers.csv
HRusers.csv
For account load operations:
parentAD.csv
childAD.csv
For role-related load operations:
Role.csv (Role load)
Rolec.csv (Role category)
Roleh.csv (Role hierarchy)
Rolem.csv (Role membership)
Logs_ YYYYMMDD_hhmi: The log directory contains the log files that store the summary of the bulk load operation. This directory is created at run time.
The following sections provide additional information about the utility and bulk load operations:
The following are the main scripts that constitute the utility:
oim_blkld.bat and oim_blkld.sh
This script contains the code to perform bulk load operations. When it is run, this script calls other scripts and stored procedures.
oim_blkld_setup.sql
This script is used to add a datafile in the Oracle Identity Manager tablespace. The "Creating a Datafile in the Oracle Identity Manager Tablespace" section of this document provides more information.
The following temporary tables are used during a bulk load operation:
OIM_BLKLD_TMP_SUFFIX
If you are using a CSV file as the input source, then the utility automatically creates the OIM_BLKLD_TMP_SUFFIX table and first loads data from the CSV file into this table. The suffix for the table name is determined as follows:
The first 6 characters of the file name are taken into account.
Special characters in the file name and the file extension (.csv) are ignored while determining the first 6 characters.
A unique number is appended to the first 6 characters.
For example, if the name of the file is acc_Data.csv, then the table that is created during the bulk load operation is named oim_blkld_tmp_accDat1.
If there are multiple CSV files, then one table is created for each file. Because the first six characters of each CSV file name are appended to the table name, you must ensure that the first six characters of each file's name are unique. This guideline is explained later in this document.
Note:
if you are using a database table as the input source, then you can specify any name for the table. You provide the name of this table as one of the input parameters of the utility.
OIM_BLKLD_EX_SUFFIX
The OIM_BLKLD_EX_SUFFIX table is used to hold data records that fail (are not loaded into Oracle Identity Manager) during a bulk load operation. One OIM_BLKLD_EX_SUFFIX table is created for each OIM_BLKLD_TMP_SUFFIX table. The EXCEPTION_MSG column of the table stores the reason for failure of each record in the table.
If you are using CSV files as the input source, then the first six characters of the CSV file name are added as a suffix to the table name. For example, if the name of the CSV file is usrdt120508.csv, then the name of the table is OIM_BLKLD_EX_ usrdt1. If there are multiple CSV files, then one temporary table is created for each CSV file.
Note:
If there are multiple CSV files, then you must ensure that the first six characters of each CSV file name are unique.
OIM_BLKLD_LOG
During a bulk load operation, the utility inserts progress and error messages in the OIM_BLKLD_LOG table. You can query this table to monitor the progress of a bulk load operation. This procedure is described in detail later in this document.
When you run the bulk load utility, it prompts you to select one of the following options:
Note:
The utility prompts for more input depending on the option you select.
Load User Data
You select this option if you want the utility to load OIM User data. In other words, data is imported into the USR table of Oracle Identity Manager. You can select the input source, CSV files or database tables, for the data that you want to load.
Load Account Data
You select this option if you want the utility to load account data. In other words, data is imported into the relevant UD_ tables of Oracle Identity Manager. You can select the input source, CSV files or database tables, for the data that you want to load.
Load Role Data
You select this option if you want the utility to load role data. In other words, data is imported into the UGP table of Oracle Identity Manager. You can select the input source, CSV files, or database tables, for the data that you want to load.
Load Role Membership
You select this option if you want the utility to load role membership data. In other words, data is imported into the USG table of Oracle Identity Manager. You can select the input source, CSV files or database tables, for the data that you want to load.
Load Role Hierarchy
You select this option if you want the utility to load role hierarchy data. In other words, data is imported into the GPG table of Oracle Identity Manager. You can select the input source, CSV files, or database tables, for the data that you want to load.
Load Role Category
You select this option if you want the utility to load role data. In other words, data is imported into the ROLE_CATEGORY tables of Oracle Identity Manager. You can select the input source, CSV files, or database tables, for the data that you want to load.
Generate Audit Snapshot
You select this option if you want the utility to generate an audit snapshot of data that you have loaded.
Preparing your database for a bulk load operation involves the following:
As mentioned in "Temporary Tables Used During a Bulk Load Operation", temporary database tables are used during the bulk load operation. It is recommended that you create a tablespace to accommodate these temporary tables instead of using the default tablespace of the Oracle Identity Manager database.
Follow the instructions in the database documentation to create a tablespace.
The default size of the datafile in the Oracle Identity Manager tablespace created during Oracle Identity Manager installation is 500 MB. You may need to add space to this datafile to accommodate the data that you are going to load. The alternative is to create a datafile.
To create a datafile in the Oracle Identity Manager tablespace:
Start a SQL*Plus session.
Connect to the Oracle Identity Manager database as SYSDBA.
Run the oim_blkld_setup.sql script. The script will prompt for the following:
Name of the Oracle Identity Manager tablespace
Full path and name for the datafile to be added in the Oracle Identity Manager tablespace
Oracle Identity Manager database user name
Note:
If there are name conflicts with existing tables, then the utility overwrites existing temporary tables at the start of each run. If required, rename temporary database tables created during an earlier run of the utility.
To run the utility:
Stop Oracle Identity Manager.
Run one of the following scripts:
Note:
To load CSV file with non-ASCII data, before running the oim_blkld.sh or oim_blkld.bat script, set the NLS_LANG environment parameter to the UTF8 characterset, in the following format:
NLS_LANG = LANGUAGE_TERRITORY.UTF8
For example:
NLS_LANG = American_America.UTF8
On UNIX computers:
OIMBulkload/script/oim_blkld.sh
On Microsoft Windows computers:
OIMBulkload\script\oim_blkld.bat
From the main menu, select one of the options depending on the data you want to load, such as user, account, or role-related data, as described in "Options Offered by the Utility".
From the second menu:
Select CSV File if you are using CSV files as the input source.
Select DB Table if you are using a database table as the input source.
When prompted, provide values for the input parameters described in "Determining Values for the Input Parameters of the Utility".
Note:
See "Determining Values for the Input Parameters of the Utility" for information about the input parameters required for loading OIM User data. See corresponding sections for information about the input parameters required to load account, role, role hierarchy, role membership, and role category data.
Monitor the performance of the operation by following the steps given in "Monitoring the Progress of the Operation".
The following is a summary of the steps involved in loading OIM User data:
Prepare your database for bulk load if not done already. See "Preparing Your Database for a Bulk Load Operation" for details.
Create the OIM User whose password will be used as the default password for all OIM Users created during the bulk load operation.
Create the input source for the bulk load operation.
If you want to use a database table as the input source, then create the table and copy user data into the table.
If you want to use CSV files as the input source, then create the CSV files and copy user data into the files. In addition, create a master.txt file containing the names of the files in the sequence in which you want to load data from them.
Determine values for the input parameters of the utility.
Stop Oracle Identity Manager.
Run the oim_blkld.sh or oim_blkld.bat script. See "Running the Utility" for information about running the oim_blkld.sh or oim_blkld.bat scripts.
Monitor the progress of the bulk load operation.
Determine the outcome of the bulk load operation.
If required, reload data that was not loaded during the first run.
Restart Oracle Identity Manager.
Verify the outcome of the bulk load operation.
Gather performance data from the operation.
Remove temporary tables and files created during the operation.
Generate an audit snapshot.
The following sections provide detailed information about the steps involved in loading OIM User data:
The utility does not encrypt passwords that it assigns to OIM Users created during the bulk load operation. Instead, it assigns the password of an existing OIM User to all OIM Users that are created during the operation.
Note:
Each OIM User is required to change the password at first login.
When you run the utility, it prompts for the login name of the existing OIM User whose password you want to use as the default password for the new OIM Users. Before you run the utility, create this OIM User as follows:
Note:
You can create a user in Oracle Identity Manager dedicated for the bulk load operation, and later delete the user if it not required any more. Otherwise, any existing OIM User can be used to perform bulk load operations.
Log in to the Oracle Identity Manager Administrative and User Console as a user with Create User privileges.
Click Administration.
On the Welcome page, click Create User.
Specify values for the following fields:
User Login
First Name (optional)
Last Name
Organization: Select Xellerate Users.
Password
Confirm Password
Click Save.
Depending on the input source that you want to use, apply the guidelines given in one of the following sections:
If you want to use CSV files as the input source for the bulk load operation, then apply the following guidelines while creating the CSV files:
The CSV files must be placed in the oimbulkload/csv_files directory.
The first line in the CSV file is called the control line. This line must contain a comma-separated list of column names of the USR table in the Oracle Identity Manager database.
Note:
Ensure that the Password column or any other encrypted column is not included in the list of columns. As mentioned earlier in this document, the utility assigns the password of an existing OIM User that you specify to all OIM Users that it loads into Oracle Identity Manager.
From the second line onward, the file must contain values for the columns in the control line. The order of columns in the first line and the values in the rest of the lines must be the same.
The following are sample contents of a CSV file:
USR_LOGIN,USR_FIRST_NAME,USR_LAST_NAME,UD_ADUSER_OBJECTGUID john_doe, John, Doe, jdoe jane_doe, Jane, Doe, janedoe richard_roe, Richard, Roe, rroe
If the value in any column contains a comma, then that value must be enclosed in double quotation marks (").
The CSV file must contain values for all columns that are designated as mandatory in the USR table. The following table lists the mandatory columns required to load the USR table:
| Mandatory Column | Description |
|---|---|
|
USR_FIRST_NAME |
The first name of the user |
|
USR_LAST_NAME |
The last name of the user |
Note:
USR_LOGIN is not a mandatory column in Oracle Identity Manager 11g Release 1 (11.1.1).
There are some key mandatory columns that you can ignore. For example, the ACT_KEY clumn in the USR table, which is populated by ORG_NAME.
Each row in the CSV file must have a unique value for the USR_LOGIN column in the USR table. If there are multiple files, you must ensure that USR_LOGIN values are unique across the CSV files. This check for uniqueness of USR_LOGIN values must also cover existing OIM Users in Oracle Identity Manager.
Ensuring that USR_LOGIN values are unique can be a time-consuming exercise. As an alternative, you can first perform the bulk load operation, fix USR_LOGIN values that are not unique, and then retry the loading operation for the modified user records. This is possible because the utility checks for uniqueness of USR_LOGIN values at run time and copies records that fail this check into the OIM_BLKLD_EX table. Later in this document, there are instructions on retrying the bulk load operation for records that are not loaded during the first run.
If you want to include an organization name in each user record, then add ORG_NAME in the control line and enter the organization name for each user from the second line onward. If ORG_NAME is not included, then the users must be assigned to the Xellerate Users organization.
Note:
All organization names listed under the ORG_NAME column in the CSV file must exist in Oracle Identity Manager.
If you want to include a manager name in each user record, then add MANAGER_NAME in the control line and enter the USR_LOGIN value of the manager for each user from the second line onward.
The utility looks up the USR_LOGIN values for managers after all user data, from all CSV files, is loaded into Oracle Identity Manager. If a USR_LOGIN value given in the MANAGER_NAME column does not exist in Oracle Identity Manager, then the lookup for that user record fails and the record is copied into the exception table, OIM_BLKLD_EX. At the end of the bulk load operation, you can perform the procedure described in "Fixing Exceptions and Reloading Data Records" to reload user records that fail the first run.
Note that the following default values are inserted into Oracle Identity Manager if the CSV file does not contain values for these columns:
ORG_NAME: Xellerate Users
USR_TYPE: End-User
USR_STATUS: Active
USR_EMP_TYPE: Full-Time
Create a master TXT file containing the names of the CSV files containing user data to be loaded. You can specify any name for the file, for example, master.txt. Save the master file in the oimbulkload/csv_files directory.
If you want to load multiple CSV files, then enter the name of each data CSV file on a separate line in the master file. Order the list of CSV file names in the sequence in which you want the utility to load data from the files. For example, suppose you have created three data CSV files, London_Users.csv, NewYork_Users.csv, and Tokyo_Users.csv. In the master file, you enter the names of the data CSV files in the following order:
Tokyo_Users.csv London_Users.csv NewYork_Users.csv
When you run the utility, data is loaded in this order. This is because the user data in London and New York may have a dependency on the Tokyo users. This is to ensure the manager-user hierarchy.
If the CSV file is generated on Microsoft Windows and is to be loaded on Linux environment, then remove the special characters, such as '\n\r', to avoid run-time errors.
If you want to use a database table as the input source for loading OIM User data, then apply the following guidelines while creating the database table:
Create the table in the Oracle Identity Manager database.
The table must contain the following primary key column:
OIM_BLKLD_USRSEQ NUMBER(19)
The utility uses this column as the primary key. If required, you can use a database sequence to populate this column.
The rest of the columns must be the same as the ones in the USR table that you want to use. In other words, ignore optional USR_ columns that you do not want to include in the table that you create.
Note that the following default values are inserted into Oracle Identity Manager if the table does not contain values for these columns:
ORG_NAME: Xellerate Users
USR_TYPE: End-User
USR_STATUS: Active
USR_EMP_TYPE: Full-Time
Table 34-1 shows the structure of a sample database table.
The following are input parameters of the utility:
Oracle Home
Value of the ORACLE_HOME environment variable on the host computer for the Oracle Identity Manager database
Database Connection String
Connection string to connect to the database that must be entered in the following format:
//HOST_IP_ADDRESS:PORT_NUMBER/SERVICE_NAME
OIM DB User
Database login ID of the Oracle Identity Manager database user
OIM DB Pwd
Password of the Oracle Identity Manager database user
The database user password is to be entered twice when prompted.
Master file name
Name of the file containing names of the CSV data files to be loaded
This parameter is used only if the input source is a single or multiple CSV files. You place the master file and CSV data files in the oimbulkload/csv_files directory. See "Using CSV Files As the Input Source" for more information.
Tmp table name
Name of the temporary table to be used as the input source
This parameter is used only if the input source for the bulk load operation is a database table. See "Creating Database Tables As the Input Source" for more information.
Control Line
Comma-separated list of names of columns to be loaded from the database table into Oracle Identity Manager
This parameter is used only if the input source for the bulk load operation is a database table.
Tablespace Name
Name of the tablespace in which temporary tables are to be created during the bulk load operation. If the user does not provide the tablespace name, then it will pick the default tablespace.
See "Preparing Your Database for a Bulk Load Operation" for more information.
Date format
Date format used by date columns in the CSV files
This parameter is used only if the input source is a single or multiple CSV files.
The date format must match the following:
Oracle supported date formats, such as dd-mm-yyyy or MM-DD-YYYY
The date format specified in the CSV file
Batch Size
Number of user records that must be processed by the utility as a single transaction
The batch size can influence the performance of the bulk load operation. The default value of this parameter is 10000.
Debug Flag
You can specify Y or N as the value of this parameter. If this parameter is set to Y, then the utility records detailed information about events that occur during the bulk load operation. See "Data Recorded During the Operation" for more information.
User ID for default password
Login name of the OIM User that you create by performing the procedure described in "Setting a Default Password for OIM Users Added by the Utility".
During the bulk load operation, you can query the OIM_BLKLD_LOG table for information about the progress of the operation. For example, you can run the following query to see progress messages generated during the bulk load operation to load OIM User data:
SELECT MSG FROM OIM_BLKLD_LOG WHERE MODULE = 'USER' AND LOG_LEVEL = 'PROGRESS_MSG' ORDER BY MSG_SEQ_NO;
Errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table. The following is an example of the query to retrieve error messages:
SELECT MSG FROM OIM_BLKLD_LOG WHERE MODULE = 'USER' AND LOG_LEVEL = 'ERROR' ORDER BY MSG_SEQ_NO;
At the end of a bulk load operation, the utility records statistics related to the operation in the following file:
oimbulkload/logs_YYYYMMDD_hhmm/oim_blkld_user_load_summary.log
To determine if there were exceptions during the operation, open this log file and look for the number against the Number of Records Rejected label. If the number of rejected records is greater than zero, then exceptions were thrown during the operation. User records that are rejected by the utility are recorded in the exception table (OIM_BLKLD_EX_SUFFIX). For each rejected record, the EXCEPTION_MSG column in the OIM_BLKLD_EX_SUFFIX table stores information about the reason the record could not be loaded.
Example 34-1 shows sample statistics recorded in the log file at the end of a bulk load operation to store OIM User data.
Example 34-1 Sample Log File Generated After Loading OIM User Data
****************************************************************
Processing File: u10.csv ================================================================ U S E R L O A D S T A T I S T I C S F O R F I L E : u10.csv ================================================================ Start Time: 08-AUG-08 11.44.12.228000 AM End Time: 08-AUG-08 11.44.13.368000 AM Number of Records Processed: 10 Number of Records Loaded: 8 Number of Records Rejected: 2 ================================================================ The name of the TMP table used during the load: OIM_BLKLD_TMP_U101 The name of the Exception table used during the load: OIM_BLKLD_EX_U101 **************************************************************** Processing File: u10b.csv ================================================================ U S E R L O A D S T A T I S T I C S F O R F I L E : u10b.csv ================================================================ Start Time: 08-AUG-08 11.44.15.368000 AM End Time: 08-AUG-08 11.44.15.540000 AM Number of Records Processed: 16 Number of Records Loaded: 15 Number of Records Rejected: 1 ================================================================ The name of the TMP table used during the load: OIM_BLKLD_TMP_U10B2 The name of the Exception table used during the load: OIM_BLKLD_EX_U10B2 ================================================================ ================================================================ Time taken in re-building indexes and enabling FK constraints ================================================================ Start time: 08-AUG-08 11.44.15.556000 AM End Time: 08-AUG-08 11.46.50.586000 AM ================================================================
In this sample, the number of rejected records is 2. If the log file shows that any records were rejected by the utility, then see "Fixing Exceptions and Reloading Data Records" for information about retrying the load operation for these records.
Note:
At the end of each bulk load operation, it is recommended that you create a backup of the exception tables.
As mentioned earlier, errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table. The following is an example of the query to retrieve error messages:
SELECT MSG FROM OIM_BLKLD_LOG WHERE MODULE = 'USER' AND LOG_LEVEL = 'ERROR' ORDER BY MSG_SEQ_NO;
An exception table OIM_BLKLD_EX_SUFFIX is created for each data table used as the input source during the bulk load operation. Records that do not meet the criteria for the operation are copied into this exception table. The suffix appended to the name of each exception table is the same as suffix appended to the name of the corresponding data table.
To reload rejected records:
Create a backup of the exception table in which rejected records are stored.
Note:
Although this is an optional step, it is recommended that you create a backup.
Review each record in the exception table, and fix errors in the data based on the message recorded in the EXCEPTION_MSG column.
After you fix errors in all the rejected records in an exception table, rename the table to OIM_BLKLD_TMP_SUFFIX and then use it as the input source.
Load records from the OIM_BLKLD_TMP_SUFFIX table by running the utility. See "Running the Utility" for more information.
Repeat Steps 1 through 4 until the Number of Records Rejected label in the oim_blkld_user_load_summary.log file shows the value 0.
Restart Oracle Identity Manager.
To verify the outcome of the bulk load operation, check if you are able to perform the following steps for one of the OIM User added by the utility:
Note:
These steps leave footprints in the system, and therefore, the bulk load verification must be performed by using a test user. If you do not want to leave the footprints in the system, then revert the changes. For example, if you have provisioned a resource to a OIM User, then deprovision the resource after testing the outcome of the bulk load operation.
Log in as the OIM User. The system should prompt you to change the password.
Provision a resource for the OIM User.
Add the OIM User to a role.
Modify the account profile of the OIM User.
Revoked the resource provisioned to the OIM User.
Unassign the OIM User from the role to which the user was added earlier.
Modify the account profile again to restore the profile to its original state.
Check if the User Resource Access report (an operational report) and the User Resource Access History report can be generated for the user.
Create an Attestation and check its status using the Diagnostic Dashboard.
The following is a summary of the steps involved in loading account data:
Prepare your database for a bulk load operation, if not already done. See "Preparing Your Database for a Bulk Load Operation" for details.
Create the input source for the bulk load operation.
If you want to use a database table as the input source, then create the table and copy account data into the table.
If you want to use CSV files as the input source, then create the CSV files and copy account data into the files.
Determine values for the input parameters of the utility.
Stop Oracle Identity Manager.
Run the oim_blkld.sh or oim_blkld.bat script.
Monitor the progress of the bulk load operation.
Determine the outcome of the bulk load operation.
If required, reload data that was not loaded during the first run.
Restart Oracle Identity Manager.
Verify the outcome of the bulk load operation.
Gather performance data from the operation.
Remove temporary tables and files created during the operation.
Generate an audit snapshot.
Requirements and Features of the Bulk Load Operation for Account Data
The following are requirements and features of the bulk load operation for account data:
Reconciliation must be set up and you should be able to test reconciliation by importing a few accounts from the target system.
Only accounts for which there are corresponding OIM Users can be loaded.
A target system that requires multiple IT resources is not supported.
Duplicate accounts cannot be detected during a bulk load operation. If there are multiple entries for the same account in the input source, then multiple accounts are created for the corresponding OIM User.
For a particular target system, if there are multiple provisioning processes/process forms in Oracle Identity Manager, then the utility uses the default provisioning process for the resource object.
Information about the stage up to which earlier bulk load operations progressed is not stored. In other words, the utility cannot resume a bulk load operation. You must backup the Oracle Identity Manager database before a bulk load operation. If you want to retry a bulk load operation, you must first restore the database and then rerun the procedure.
The following sections provide detailed information about the steps involved in loading account data:
Depending on the input source that you want to use, apply the guidelines given in one of the following sections:
If you want to use CSV files as the input source for the bulk load operation, then apply the following guidelines while creating the CSV files:
The CSV files must be placed in the oimbulkload/csv_files directory.
The first line in the CSV file is called the control line. This line must contain a comma-separated list of column names in the account (UD_*) table into which you want to load the account data. To find out the UD_ table, go to the process form in the Design Console. See Chapter 12, "Developing Provisioning Processes" for information about process forms.
Note:
Ensure that the Password column or any other encrypted column is not included in the list of columns.
From the second line onward, the file must contain values for the columns in the control line. The order of columns in the first line and the values in the rest of the lines must be the same.
If the value in any column contains a comma, then that value must be enclosed in double quotation marks (").
The CSV file must contain values for all columns that are designated as mandatory in the account table. The key mandatory columns in the account table must be ignored.
If you want to load account data into parent and child tables, then you must create one parent CSV file and one child CSV file for each child table. For example if you are loading data into one parent table and three child tables, then you must create one parent CSV file and three child CSV files.
If you want to load account data into parent and child tables, then at least one column must be the same in both tables. This column corresponds to the link attribute between the parent and child CSV files. The following example illustrates this:
The following are sample contents of a parent CSV file:
UD_ADUSER_UID,UD_ADUSER_ORGNAME,UD_ADUSER_FNAME,UD_ADUSER_LNAME,UD_ADUSER_MNAME,UD_ADUSER_FULLNAME,UD_ADUSER_OBJECTGUID ADTEST1,"7~CN=ForeignSecurityPrincipals,DC=vivek01,DC=com",adtest1,adtest1,,adtest1,102
The following are sample contents of a child CSV file:
UD_ADUSER_UID,UD_ADUSER_ORGNAME,UD_ADUSRC_GROUPNAME ADTEST1,"7~CN=ForeignSecurityPrincipals,DC=vivek01,DC=com",group2
The UD_ADUSER_UID column is common to both the parent file and the child file.
If the CSV file is generated on Microsoft Windows and is to be loaded on Linux environment, then remove the special characters, such as '\n\r', to avoid run-time errors.
If you want to use a database table as the input source for loading account data, then apply the following guidelines while creating the database table:
Create the table in the Oracle Identity Manager database.
The table must contain the following primary key column:
OIM_BLKLD_USRSEQ NUMBER(19)
The utility uses this column as the primary key. If required, you can use a database sequence to populate this column.
The rest of the columns must be the same as the ones in the account (UD_) table that you want to use. In other words, ignore optional UD_ columns that you do not want to include in the table that you create.
Table 34-2 shows the structure of a sample parent table.
Table 34-2 Structure of a Sample Database Table
| Name | Null? | Type |
|---|---|---|
|
UD_ADUSER_UID |
VARCHAR2(20) |
|
|
UD_ADUSER_ORGNAME |
VARCHAR2(256) |
|
|
UD_ADUSER_FNAME |
VARCHAR2(80) |
|
|
UD_ADUSER_LNAME |
VARCHAR2(80) |
|
|
UD_ADUSER_MNAME |
VARCHAR2(80) |
|
|
UD_ADUSER_FULLNAME |
VARCHAR2(240) |
|
|
OIM_BLKLD_SEQ |
NOT NULL |
NUMBER(19) |
Table 34-3 shows the structure of a sample child table.
The following are input parameters of the utility:
Oracle Home
Value of the ORACLE_HOME environment variable on the host computer for the Oracle Identity Manager database
Database Connection String
Connection string to connect to the database that must be entered in the following format:
//HOST_IP_ADDRESS:PORT_NUMBER/SERVICE_NAME
OIM DB User
Database login ID of the Oracle Identity Manager database user
OIM DB Pwd
Password of the Oracle Identity Manager database user
Object name (OBJ_NAME)
Name of the resource object corresponding to the account data to be loaded
CSV file names
Names of the CSV files to be used as the input source
This parameter is used only if the input source is CSV files. See "Using CSV Files As the Input Source" for more information. If you are loading data from parent and child CSV file, then use a comma-delimited list to enter the names of the files. The name of the parent CSV file must be provided first, and it must be followed by the names of the child CSV files.
Tmp table name
Name of the temporary table to be used as the input source
This parameter is used only if the input source for the bulk load operation is a database table. See "Creating Database Tables As the Input Source" for more information.
Control Line
Comma-separated list of names of columns to be loaded from the database table into Oracle Identity Manager
This parameter is used only if the input source for the bulk load operation is a database table.
Tablespace Name
Name of the tablespace in which temporary tables are to be created during the bulk load operation (if end user won't provide the tablespace name then it will pick the default tablespace)
See "Preparing Your Database for a Bulk Load Operation" for more information.
Date format
Date format used by date columns in the CSV files
This parameter is used only if the input source is a single or multiple CSV files.
The date format must match the following:
Oracle supported date formats, such as dd-mm-yyyy or MM-DD-YYYY
The date format specified in the CSV file
Batch Size
Number of user records that must be processed by the utility as a single transaction
The batch size can influence the performance of the bulk load operation. The default value of this parameter is 10000.
Debug Flag
You can specify Y or N as the value of this parameter. If this parameter is set to Y, then the utility records detailed information about events that occur during the bulk load operation. See "Data Recorded During the Operation" for more information.
IT Resource Name
Name of the IT resource created for the target system
During the bulk load operation, you can query the OIM_BLKLD_LOG table for information about the progress of the operation. For example, you can run the following query to see progress messages generated during the bulk load operation to load account data:
SELECT MSG FROM OIM_BLKLD_LOG WHERE MODULE = 'ACCOUNT' AND LOG_LEVEL = 'PROGRESS_MSG' ORDER BY MSG_SEQ_NO;
For example, you can run the following query to see progress messages generated during the bulk load operation to load account data:
SELECT MSG FROM OIM_BLKLD_LOG WHERE MODULE = 'ACCOUNT' AND LOG_LEVEL = 'PROGRESS_MSG' ORDER BY MSG_SEQ_NO;
Errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table. The following is an example of the query to retrieve error messages:
SELECT MSG FROM OIM_BLKLD_LOG WHERE MODULE = 'ACCOUNT' AND LOG_LEVEL = 'ERROR' ORDER BY MSG_SEQ_NO;
At the end of a bulk load operation, the utility records statistics related to the operation in the following file:
oimbulkload/logs_YYYYMMDD_hhmm/oim_blkld_account_load_summary.log
To determine if there were exceptions during the operation, open this log file and look for the number against the Number of Records Rejected label. If the number of rejected records is greater than zero, then exceptions were thrown during the operation. User records that are rejected by the utility are recorded in the exception table (OIM_BLKLD_EX_SUFFIX). For each rejected record, the EXCEPTION_MSG column in the OIM_BLKLD_EX_SUFFIX table stores information about the reason the record could not be loaded.
Example 34-2 shows sample statistics recorded in the log file at the end of a bulk load operation to store account data.
Example 34-2 Sample Log File Generated After Loading Account Data
============================================================= A C C O U N T L O A D S T A T I S T I C S ============================================================= Start Time: 22-JUL-08 03.59.30.206000 PM End Time: 22-JUL-08 04.03.21.126000 PM Number of Records Processed: 100026 Number of Records Loaded: 100000 Number of Records Rejected: 26 ============================================================= The names of the TMP tables used during the load: OIM_BLKLD_TMP_P100001 OIM_BLKLD_TMP_C100002 The names of the Exception tables used during the load: OIM_BLKLD_EX_P100001 OIM_BLKLD_EX_C100002
In this sample, the number of rejected records is 26. If the log file shows that any records were rejected by the utility, then see "Fixing Exceptions and Reloading Data Records" for information about retrying the load operation for these records.
Note:
At the end of each bulk load operation, it is recommended that you create a backup of the exception tables.
Note:
If you want to load data from CSV files for multiple target systems, then you can apply one of the following approaches:
Approach 1: Run the utility for all the sets of CSV files, and then perform the procedure described in this section.
Approach 2: Run the utility for one set of CSV files, and perform the procedure described in this section. Then, repeat this procedure for the next set of CSV files.
As mentioned earlier, errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table. The following is an example of the query to retrieve error messages:
SELECT MSG FROM OIM_BLKLD_LOG WHERE MODULE = 'ACCOUNT' AND LOG_LEVEL = 'ERROR' ORDER BY MSG_SEQ_NO;
An exception table OIM_BLKLD_EX_SUFFIX is created for each data table used as the input source during the bulk load operation. Records that do not meet the criteria for the operation are copied into this exception table. The suffix appended to the name of each exception table is the same as suffix appended to the name of the corresponding data table.
To reload rejected records:
Create a backup of the exception table in which rejected records are stored.
Note:
Although this is an optional step, it is recommended that you create a backup.
Review each record in the exception table, and fix errors in the data based on the message recorded in the EXCEPTION_MSG column.
After you fix errors in all the rejected records in an exception table, rename the table to OIM_BLKLD_TMP_SUFFIX and then use it as the input source.
Load records from the OIM_BLKLD_TMP_SUFFIX table by running the utility. See "Running the Utility" for more information.
Repeat Steps 1 through 4 until the Number of Records Rejected label in the oim_blkld_account_load_summary.log file shows the value 0.
Restart Oracle Identity Manager.
To verify the outcome of the bulk load operation, check if you are able to perform the following steps for one of the OIM Users for whom an account has been added by the utility:
Log in as the OIM User, and check if the newly created account is displayed in the resource profile of the user.
Log in to the target system by using the credentials of the newly created account.
The following is a summary of the steps involved in loading role-related data:
Prepare your database for a bulk load operation, if not already done. See "Preparing Your Database for a Bulk Load Operation" for details.
Create the input source for the bulk load operation.
If you want to use a database table as the input source, then create the table and copy role-related data into the table.
If you want to use CSV files as the input source, then create the CSV files and copy role-related data into the files. In addition, create a master.txt file containing the names of the files in the sequence in which you want to load data from them.
Determine values for the input parameters of the utility.
Stop Oracle Identity Manager.
Run the oim_blkld.sh or oim_blkld.bat script.
Monitor the progress of the bulk load operation.
Determine the outcome of the bulk load operation.
If required, reload data that is not loaded during the first run.
Restart Oracle Identity Manager.
Verify the outcome of the bulk load operation.
Gather performance data from the operation.
Remove temporary tables and files created during the operation.
Generate an audit snapshot.
The following sections provide detailed information about the steps involved in loading OIM User data:
Depending on the input source that you want to use, apply the guidelines given in one of the following sections:
If you want to use CSV files as the input source for the bulk load operation, then apply the following guidelines while creating the CSV files:
The CSV files must be placed in the oimbulkload/csv_files directory.
The first line in the CSV file is called the control line.
This line must contain a comma-separated list of column names based on the selected role upload (role, role hierarchy, role membership, and role category) in the Oracle Identity Manager database.
From the second line onward, the file must contain values for the columns in the control line. The order of columns in the first line and the values in the rest of the lines must be the same. The following is a sample content of a role (UGP) CSV file:
USR_ROLE_NAME,UGP_NAMESPACE,USR_LOGIN Administrators, null, XELSYSADM operators, null, XELSYSADM
If the value in any column contains a comma, then that value must be enclosed in double quotation marks (").
The CSV file must contain values for all columns that are designated as mandatory in the respective role tables.
The CSV file must contain values for all columns that are designated as mandatory depending on the upload role data, role hierarchy data, role membership data, and role category data.
Role (UGP): UGP_ROLENAME, UGP_NAMESPACE, USR_LOGIN (UGP_NAMESPACE can be left as null when not required)
Role Hierarchy (GPG): UGP_NAME, GPG_UGP_NAME
Role Membership (USG): UGP_NAME, USR_LOGIN
Role Category (ROLE_CATEGORY): ROLE_CATEGORY_NAME
Each row in the CSV file must have a unique value for the combinationation of manadatory columns.
The following default values are inserted into Oracle Identity Manager if the CSV file does not contain values for these columns:
For Role (UGP)
UGP_ROLE_CATEGORY_KEY: ROLE_CATEGORY_KEY from ROLE_CATEGORY table with ROLE_CATEGORY_NAME as 'Default'.
UGP_DISPLAY_NAME: Defaults to UGP_NAME.
For Role Hierarchy (GPG)
For Role Membership (USG)
RUL_KEY: RUL_KEY from RUL table with RUL_NAME as 'Default'
USG_PRIORITY: group and rank based on UGP_KEY based on the rows given for upload.
Role Category (ROLE CATEGORY)
None
Create a master TXT file containing the names of the CSV files containing role data to be loaded. You can specify any name for the file, for example, master.txt. Save the master file in the oimbulkload/csv_files directory.
If you want to load multiple CSV files, then enter the name of each data CSV file on a separate line in the master file. Order the list of CSV file names in the sequence in which you want the utility to load data from the files. For example, suppose you have created three data CSV files, Role1.csv, Role2.csv, and Role3.csv. In the master file, enter the names of the data CSV files in the following order:
Role1.csv
Role2.csv
Role3.csv
When you run the utility, data is loaded in this order.
If the CSV file is generated on Microsoft Windows and is to be loaded on Linux environment, then remove the special characters, such as '\n\r', to avoid run-time errors.
If you want to use a database table as the input source for loading OIM User data, then apply the following guidelines while creating the database table:
Create the table in the Oracle Identity Manager database.
The table must contain the following primary key column:
OIM_BLKLD_USRSEQ NUMBER(19)
The utility uses this column as the primary key. If required, you can use a database sequence to populate this column.
The rest of the columns must be the same as the ones in the respective role tables that you want to use.
Table 34-4 shows the structure of a sample database role table.
The following are input parameters of the utility:
Oracle Home
Value of the ORACLE_HOME environment variable on the host computer for the Oracle Identity Manager database
Database Connection String
Connection string to connect to the database that must be entered in the following format:
//HOST_IP_ADDRESS:PORT_NUMBER/SERVICE_NAME
OIM DB User
Database login ID of the Oracle Identity Manager database user
OIM DB Pwd
Password of the Oracle Identity Manager database user
Object name (OBJ_NAME)
Name of the resource object corresponding to the account data to be loaded
CSV file names
Names of the CSV files to be used as the input source
This parameter is used only if the input source is CSV files. See "Using CSV Files As the Input Source" for more information. If you are loading data from parent and child CSV file, then use a comma-delimited list to enter the names of the files. The name of the parent CSV file must be provided first, and it must be followed by the names of the child CSV files.
Tmp table name
Name of the temporary table to be used as the input source
This parameter is used only if the input source for the bulk load operation is a database table. See "Creating Database Tables As the Input Source" for more information.
Control Line
Comma-separated list of names of columns to be loaded from the database table into Oracle Identity Manager
This parameter is used only if the input source for the bulk load operation is a database table.
Tablespace Name
Name of the tablespace in which temporary tables are to be created during the bulk load operation (if end user won't provide the tablespace name then it will pick the default tablespace)
See "Preparing Your Database for a Bulk Load Operation" for more information.
Date format
Date format used by date columns in the CSV files
This parameter is used only if the input source is a single or multiple CSV files.
The date format must match the following:
Oracle supported date formats, such as dd-mm-yyyy or MM-DD-YYYY
The date format specified in the CSV file
Batch Size
Number of user records that must be processed by the utility as a single transaction
The batch size can influence the performance of the bulk load operation. The default value of this parameter is 10000.
Debug Flag
You can specify Y or N as the value of this parameter. If this parameter is set to Y, then the utility records detailed information about events that occur during the bulk load operation. See "Data Recorded During the Operation" for more information.
IT Resource Name
Name of the IT resource created for the target system
During the bulk load operation, you can query the OIM_BLKLD_LOG table for information about the progress of the operation. For example, you can run the following query to see progress messages generated during the bulk load operation to load OIM Role data:
SELECT MSG FROM OIM_BLKLD_LOG WHERE MODULE = 'ROLE' AND LOG_LEVEL = 'PROGRESS_MSG' ORDER BY MSG_SEQ_NO;
Errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table. The following is an example of the query to retrieve error messages:
SELECT MSG FROM OIM_BLKLD_LOG WHERE MODULE = 'ROLE' AND LOG_LEVEL = 'ERROR' ORDER BY MSG_SEQ_NO;
At the end of a bulk load operation, the utility records statistics related to the operation in the following file:
oimbulkload/logs_YYYYMMDD_HHMM/oim_blkld_ENTITY_NAME_load_summary.log
In the log file name, ENTITY_NAME stands for the entity being loaded. For example:
For roles, the log file name is oim_blkld_role_load_summary.log.
For role memberships, the log file name is oim_blkld_rolemem_load_summary.log.
To determine if there were exceptions during the operation, open this log file and look for the number against the Number of Records Rejected label. If the number of rejected records is greater than zero, then exceptions were thrown during the operation. User records that are rejected by the utility are recorded in the exception table (OIM_BLKLD_EX_SUFFIX). For each rejected record, the EXCEPTION_MSG column in the OIM_BLKLD_EX_SUFFIX table stores information about the reason the record could not be loaded.
Example 34-3 shows sample statistics recorded in the log file at the end of a bulk load operation to store OIM Role data.
Example 34-3 Sample Log File Generated After Loading OIM Role Data
*************************************************************************************************** Processing File: Role.csv ========================================================================================== R O L E L O A D S T A T I S T I C S F O R F I L E : Role.csv ========================================================================================== Start Time: 17-NOV-09 02.48.18.447767 AM End Time: 17-NOV-09 02.48.19.228710 AM Number of Records Processed: 2 Number of Records Loaded: 2 Number of Records Rejected: 0 ========================================================================================== The name of the TMP table used during the load: OIM_BLKLD_TMP_ROLE1 The name of the Exception table used during the load: OIM_BLKLD_EX_ROLE1 ========================================================================================== =============================================================================== Time taken in re-building indexes and enabling FK constraints =============================================================================== Start time: 17-NOV-09 02.48.19.243781 AM
In this sample, the number of rejected loaded is 2. If the log file shows that any records have been rejected by the utility, then see "Fixing Exceptions and Reloading Data Records" for information about retrying the load operation for these records.
Note:
You cannot use the utility to load data into a remote Oracle Identity Manager database.
As mentioned earlier, errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table. The following is an example of the query to retrieve error messages:
SELECT MSG FROM OIM_BLKLD_LOG WHERE MODULE = 'ROLE' AND LOG_LEVEL = 'ERROR' ORDER BY MSG_SEQ_NO;
An exception table OIM_BLKLD_EX_SUFFIX is created for each data table used as the input source during the bulk load operation. Records that do not meet the criteria for the operation are copied into this exception table. The suffix appended to the name of each exception table is the same as suffix appended to the name of the corresponding data table.
To reload rejected records:
Create a backup of the exception table in which rejected records are stored.
Note:
Although this is an optional step, it is recommended that you create a backup.
Review each record in the exception table, and fix errors in the data based on the message recorded in the EXCEPTION_MSG column.
After you fix errors in all the rejected records in an exception table, rename the table to OIM_BLKLD_TMP_SUFFIX and then use it as the input source.
Load records from the OIM_BLKLD_TMP_SUFFIX table by running the utility. See "Running the Utility" for more information.
Repeat Steps 1 through 4 until the Number of Records Rejected label shows the value 0 in the oim_blkld_role_load_summary.log file or the corresponding log file for role membership, role hierarchy, and role category.
Restart Oracle Identity Manager.
To verify the outcome of the bulk load operation, check if you are able to perform the following steps for one of the OIM Role added by the utility:
Log in to Oracle Identity Manager Administrative and User Console and verify that the newly created role is displayed in the search result for roles.
For the newly created role hierarchy and role members, click the Hierarchy and Members tabs respectively on the role details page.
To verify the newly created role category, in the Welcome page of Oracle Identity Administration, click Advanced Search - Role Categories. Then, perform an advanced search to find the newly created role.
During the bulk load operation, the utility inserts progress and error messages in the OIM_BLKLD_LOG table. Data in this table is not deleted at the start of a new bulk load operation. One of the columns in this table holds the time stamp at which messages are recorded in the table.
Table 34-5 describes the structure of the OIM_BLKLD_LOG table.
Table 34-5 Structure of the OIM_BLKLD_LOG Table
| Column | NULL | Type | Description |
|---|---|---|---|
|
MSG_SEQ_NO |
NULL |
NUMBER(19) |
This column stores the number that denotes the order in which messages are inserted in this table. The column is populated by using the OIM_BLKLD_LOG_SEQ sequence. You can use this column to query for messages in the order in which they are recorded in the table. |
|
MODULE |
NOT NULL |
VARCHAR2(20) |
This column stores one of the following values: ROLE: This value indicates that the message has been recorded while loading OIM Role data. ROLE HIERARCHY: This value indicates that the message has been recorded while loading role hierarchy data. ROLE MEMBERSHIP: This value indicates that the message has been recorded while loading OIM role membership data. ROLE CATEGORY: This value indicates that the message has been recorded while loading OIM role category data. |
|
LOG_LEVEL |
NOT NULL |
VARCHAR2(20) |
This column stores one of the following values: ERROR: Designates fine-grained informational events that are useful to debug. DEBUG: Designates error events that might allow the application to continue running. Error is used to log all unhandled exceptions. PROGRESS_MSG: Designates intermediate progress messages. |
|
LOAD_SOURCE |
NOT NULL |
VARCHAR2(40) |
This column indicates the source of data for the bulk load operation during which the row was inserted. The value can be one of the following:CSV File: FILE_NAME DB Table |
|
MSG |
NOT NULL |
VARCHAR2(4000) |
This column stores a message corresponding to the value stored in the LOG_LEVEL column. |
|
CREATE_DATE |
DATE |
This column holds the time stamp at which the record was created. The format for entries in this column is as follows: yyyy/mm/dd hh24:mi:ss For example: 2008/06/23 21:49:16:32 |
As mentioned earlier in this document, the following log files are created during the bulk load operation:
For OIM Users:
oimbulkload/logs_YYYYMMDD_HHMM/oim_blkld_user_load_summary.log
For accounts:
oimbulkload/logs_YYYYMMDD_HHMM/oim_blkld_account_load_summary.log
For roles, role hierarchies, memberships, and role categories:
oimbulkload/logs_YYYYMMDD_HHMM/oim_blkld_ENTITY_NAME_load_summary.log
In the log file name, ENTITY_NAME stands for the entity being loaded. For example:
For roles, the log file name is oim_blkld_role_load_summary.log.
For role memberships, the log file name is oim_blkld_rolemem_load_summary.log.
Data recorded in this file can be used to collate performance-related information about the bulk load operation. The following information can be collected after the bulk load operation:
Start time
Input source
Number of records in the system before the load
Number of records successfully loaded
Number of records rejected
Total time taken
You can use this information during future runs of the utility.
See Also:
Table 34-5, "Structure of the OIM_BLKLD_LOG Table" for information about the log levels that stores error events
If you do not want to save the results of a bulk load operation, then:
Remove the OIM_BLKLD_TMP_SUFFIX, OIM_BLKLD_EX_SUFFIX, and OIM_BLKLD_LOG tables.
Remove any files that you created or used during the operation.
If you created a tablespace for the operation, then remove the tablespace.
See "Gathering Performance Data from the Bulk Load Operation" before you remove log files created in the logs_timestamp directory.
Note:
At this point, you can restart Oracle Identity Manager if you have not already done so.
If required, you can generate an audit snapshot of Oracle Identity Manager data after a bulk load operation, or at any time during the bulk load operation. You can also generate audit snapshots by selecting option 7 in the Bulk Load utility. The utility uses the audit engine shipped with Oracle Identity Manager. Internally, the GenerateSnapshot script is called when you run the audit utility. Similarly, the GenerateSnapshot script is called when you select the option to generate an audit snapshot.
Note:
Oracle Identity Manager must be up and running when you run the audit utility.
If you encounter a problem in running the Bulk Load utility by selecting option 7 to generat audit snapshots, then run the GenerateSnapshot.sh script from the OIM_HOME/server/bin/ directory.
Before you generate an audit snapshot:
Open the OIM_HOME/bin/GenerateSnapshot.sh (or GenerateSnapshot.bat) file in a text editor.
In this file, search for the following line:
In the GenerateSnapshot.bat file:
SET XEL_HOME=..
In the GenerateSnapshot.sh file:
XEL_HOME=..
Replace this line with the following line:
In the GenerateSnapshot.bat file:
SET XEL_HOME=OIM_HOME_DIRECTORY_PATH
In the GenerateSnapshot.sh file:
XEL_HOME=OIM_HOME_DIRECTORY_PATH
Save and close the file.
See "Auditing" in the Oracle Fusion Middleware User's Guide for Oracle Identity Manager for information about the procedure to generate audit snapshots.
|
 Copyright © 1991, 2013, Oracle and/or its affiliates. All rights reserved. Legal Notices |
|
