Using the Bulk Load Utility

13 Using the Bulk Load Utility

The Bulk Load utility is used to automate the process of loading a large amount of data into Oracle Identity Manager to reduce the downtime.

Oracle Identity Manager may be one among many repositories of entity data in your organization. When you start using Oracle Identity Manager, you might want to load data from the other repositories into Oracle Identity Manager. The Bulk Load utility offers a solution to this requirement.

The Bulk Load utility is aimed at automating the process of loading a large amount of data into Oracle Identity Manager. It helps reduce the downtime involved in loading data. You can use this utility after you install Oracle Identity Manager or at any time during the production lifetime of Oracle Identity Manager. The Bulk Load utility can load users, accounts, roles, role hierarchy, role membership, role category data, and organizations.

This chapter contains the following topics:

13.1 Modes of Running the Utility

The Bulk Load utility can be run in offline mode or online mode.

The Bulk Load utility can be run in one of the following modes:

Offline mode: This is the traditional or existing mode. To run the utility in offline mode, Oracle Identity Manager must be running.
Online mode: In online mode, there is no need to shut down Oracle Identity Manager. online mode only implies that the utility can be run when Oracle Identity Manager is up and running. It is still a command-line utility and no other interface is available for online mode.

By default, Bulk Load utility runs in online mode.

To choose between the online or offline mode, consider the following factors:

At times, service availability is more important for business reasons. Choose default online mode in this case.
If volume of new entities loaded is not huge, cost of service restart and index rebuild after offline bulkload is higher than slight performance degradation in online mode. Choose default online mode in this case.
When load volume is high and existing system data is less in comparison, offline bulk load might have some advantages.

13.2 Features of the Bulk Load Utility

The Bulk Load utility can load users, accounts, roles, role hierarchy, role membership, and role category data.

The following are features of the bulk load utility:

Data can be loaded into Oracle Identity Manager as OIM Users, accounts allocated (provisioned) to OIM Users, roles, role hierarchies, role memberships, role categories, or organizations.
Data can be loaded from a single or multiple CSV files or a database table. Data imported into Oracle Identity Manager is automatically converted into OIM Users, accounts provisioned to OIM Users, roles, role hierarchies, role memberships, role categories, or organizations.
Data can be loaded from a single or multiple trusted sources.
Data can be loaded into either an empty Oracle Identity Manager repository or an Oracle Identity Manager repository that already contains data about OIM Users and resources. In other words, user data can be loaded at any time, either immediately after Oracle Identity Manager installation or when the system is already in production.
The utility is for creating new entities only. It cannot be used to update or delete existing entities.
Exceptions generated during user data loading are handled, and records that fail the loading process can be retried.
Audit snapshots can be generated after a bulk load operation for users.
After bulk loading of OIM User data, password change at first login is enforced because a dummy password is used during the operation.

Note:

You cannot use the utility to encrypt user attributes. In other words, if a user field in Oracle Identity Manager is encrypted, then the utility cannot be used to encrypt data that is loaded into that field.
The Bulk Load utility can be used in offline or online modes.

13.3 Prerequisites for Running the Bulk Load Utility

Before running the Bulk Load utility, you must install the utility and prepare your database for the bulk load operation, and familiarize yourself with the various options of the Bulk Load utility.

Running the Bulk Load utility has the following prerequisites:

13.3.1 Installing the Bulk Load Utility

Install the Bulk Load utility by copying the oimbulkload directory from the installation package and by extracting the contents.

To install the utility complete the steps provided in the following approaches:

Approach 1:

Zip and copy the bulk load utility folder from OIM Machine to the DBMachine.
Copy the following MW Home related files in the same structure to the DBMachine:

<MW_HOME>/wlserver/server/lib/wlfullclient.jar

<MW_HOME>/oracle_common/modules/javax.management.j2ee.jar

<MW_HOME>/oracle_common/modules/clients/com.oracle.webservices.wls.jaxws-owsm-

client.jar

<MW_HOME>/oracle_common/modules/thirdparty/spring-context-5.1.3.RELEASE.jar

<MW_HOME>/oracle_common/modules/org.apache.commons.logging_1.2.jar

<MW_HOME>/wlserver/modules/com.bea.core.diagnostics.flightrecorder.jar

<MW_HOME>/wlserver/modules/com.oracle.weblogic.rjvm.jar

<MW_HOME>/wlserver/modules/com.oracle.weblogic.security.crypto.utils.jar

<MW_HOME>/wlserver/server/lib/wlthint3client.jar

<MW_HOME>/idm/server/apps/oim.ear/APP-INF/lib/OIMServer.jar

<MW_HOME>/idm/server/apps/oim.ear/APP-INF/lib/iam-platform-utils.jar

<MW_HOME>/idm/server/idmdf/idmdf-common.jar

<MW_HOME>/idm/server/idmdf/event-recording-client.jar

<MW_HOME>/idm/server/client/oimclient.jar

<MW_HOME>/idm/server/config/authwl.conf
Set MW_HOME & OIM_HOME as copied in the above step and run the Utility.

13.3.1.1

Approach 2

Install the Oracle Instant Client, either using yum.

yum install oracle-instantclient-release-el
yum install oracle-instantclient-basic
yum install oracle-instantclient-sqlplus
Yum install oracle-instantclient-tools.x86_64

OR
Use the same zip files given in the OTN.

Create the following folders in the ORACLE_HOME:
$ORACLE_HOME mkdir rdbms: This is a blank folder

mkdir -p jdbc/lib: Copy the file

ojdbc8.jar filecp $ORACLE_HOME/lib/ojdbc8.jar jdbc/lib
Set the following environment variables for the Client:
export ORACLE_HOME=/usr/lib/oracle/21/client64
export LD_LIBRARY_PATH=$CLIENT_HOME/lib

export PATH=$PATH:$CLIENT_HOME/bin
Run the Utility.

13.3.2 Understanding Bulk Load Options and Additional Details

Additional information about the Bulk Load utility consists of utility scripts, temporary database tables used for bulk load operations, and bulk load options for loading various entity data.

The following sections provide additional information about the utility and bulk load operations:

13.3.2.1 Scripts That Constitute the Utility

The following are the main scripts that constitute the utility:

oim_blkld.bat and oim_blkld.sh

This script contains the code to perform bulk load operations. When it is run, this script calls other scripts and stored procedures.
oim_blkld_setup.sql

This script is used to add a datafile in the Oracle Identity Manager tablespace and provide additional grants to the Oracle Identity Manager database user to perform required operations during Bulk Load. See Creating a Datafile in the Oracle Identity Governance Tablespace for more information about creating a datafile in the Oracle Identity Manager tablespace.

13.3.2.2 Temporary Tables Used During a Bulk Load Operation

The following temporary tables are used during a bulk load operation:

OIM_BLKLD_TMP_SUFFIX

If you are using a CSV file as the input source, then the utility automatically creates the OIM_BLKLD_TMP_SUFFIX table and first loads data from the CSV file into this table. The suffix for the table name is determined as follows:
- The first 6 characters of the file name are taken into account.
- Special characters in the file name and the file extension (.csv) are ignored while determining the first 6 characters.
- A unique number is appended to the first 6 characters.
- For example, if the name of the file is acc_Data.csv, then the table that is created during the bulk load operation is named oim_blkld_tmp_accDat1.
If there are multiple CSV files, then one table is created for each file. Because the first six characters of each CSV file name are appended to the table name, you must ensure that the first six characters of each file's name are unique. This guideline is explained later in this document.

Note:

if you are using a database table as the input source, then you can specify any name for the table. You provide the name of this table as one of the input parameters of the utility.
OIM_BLKLD_EX_SUFFIX

The OIM_BLKLD_EX_SUFFIX table is used to hold data records that fail (are not loaded into Oracle Identity Manager) during a bulk load operation. One OIM_BLKLD_EX_SUFFIX table is created for each OIM_BLKLD_TMP_SUFFIX table. The EXCEPTION_MSG column of the table stores the reason for failure of each record in the table.

If you are using CSV files as the input source, then the first six characters of the CSV file name are added as a suffix to the table name. For example, if the name of the CSV file is usrdt120508.csv, then the name of the table is OIM_BLKLD_EX_ usrdt1. If there are multiple CSV files, then one temporary table is created for each CSV file.

Note:

If there are multiple CSV files, then you must ensure that the first six characters of each CSV file name are unique.
OIM_BLKLD_LOG

During a bulk load operation, the utility inserts progress and error messages in the OIM_BLKLD_LOG table. You can query this table to monitor the progress of a bulk load operation. This procedure is described in detail later in this document.

13.3.2.3 Options Offered by the Utility

When you run the bulk load utility, it prompts you to select one of the following options:

Note:

The utility prompts for more input depending on the option you select.

Load User Data

You select this option if you want the utility to load OIM User data. In other words, data is imported into the USR table of Oracle Identity Manager. You can select the input source, CSV files or database tables, for the data that you want to load.
Load Account Data

You select this option if you want the utility to load account data. In other words, data is imported into the relevant UD_ tables of Oracle Identity Manager. You can select the input source, CSV files or database tables, for the data that you want to load.
Load Role Data

You select this option if you want the utility to load role data. In other words, data is imported into the UGP table of Oracle Identity Manager. You can select the input source, CSV files, or database tables, for the data that you want to load.
Load Role Membership

You select this option if you want the utility to load role membership data. In other words, data is imported into the USG table of Oracle Identity Manager. You can select the input source, CSV files or database tables, for the data that you want to load.
Load Role Hierarchy

You select this option if you want the utility to load role hierarchy data. In other words, data is imported into the GPG table of Oracle Identity Manager. You can select the input source, CSV files, or database tables, for the data that you want to load.
Load Organizations

You select this option if you want the utility to load organizations. In other words, data is imported into the ACT table of Oracle Identity Manager. You can select the input source, CSV files, or database tables, for the data that you want to load.
Load Role Category

You select this option if you want the utility to load role data. In other words, data is imported into the ROLE_CATEGORY tables of Oracle Identity Manager. You can select the input source, CSV files, or database tables, for the data that you want to load.
Generate Audit Snapshot

You select this option if you want the utility to generate an audit snapshot of users that you have loaded.

13.3.3 Preparing Your Database for a Bulk Load Operation

Prepare the database for bulk load operation by creating a tablespace for temporary tables and by creating a datafile in the tablespace.

Preparing your database for a bulk load operation involves the following:

13.3.3.1 Creating a Tablespace for Temporary Tables

As mentioned in Temporary Tables Used During a Bulk Load Operation, temporary database tables are used during the bulk load operation. It is recommended that you create a tablespace to accommodate these temporary tables instead of using the default tablespace of the Oracle Identity Manager database.

Follow the instructions in the database documentation to create a tablespace.

13.3.3.2 Creating a Datafile in the Oracle Identity Governance Tablespace

The default size of the datafile in the Oracle Identity Manager tablespace created during Oracle Identity Manager installation is 500 MB. You may need to add space to this datafile to accommodate the data that you are going to load. The alternative is to create a datafile.

To create a datafile in the Oracle Identity Manager tablespace:

Start a SQL*Plus session.
Connect to the Oracle Identity Manager database as SYSDBA.
Run the oim_blkld_setup.sql script. The script will prompt for the following:
- Name of the Oracle Identity Manager tablespace
- Full path and name for the datafile to be added in the Oracle Identity Manager tablespace
- Oracle Identity Manager database user name

After providing input to prompted Oracle Identity Manager database user name, appropriate grants to perform required operations during Bulk Load are provided to the database user.

13.4 Running the Utility

Running the Bulk Load utility involves running the bulk load scripts, selecting options for bulk load, selecting input source, and monitoring the progress of the operation.

To run the utility:

Note:

If there are name conflicts with existing tables, then the utility overwrites existing temporary tables at the start of each run. If required, rename temporary database tables created during an earlier run of the utility.

Start the Oracle Identity Manager.
Run one of the following scripts:
Note:

To load CSV file with non-ASCII data, before running the oim_blkld.sh or oim_blkld.bat script, set the NLS_LANG environment parameter to the UTF8 characterset, in the following format:

NLS_LANG = LANGUAGE_TERRITORY.UTF8

For example:

NLS_LANG = American_America.UTF8
- On UNIX computers:
  - To run in online mode, run:
    
    OIMBulkload/scripts/oim_blkld.sh
    
    OR:
    
    OIMBulkload/scripts/oim_blkld.sh —online
  - To run in offline mode, run:
    
    OIMBulkload/scripts/oim_blkld.sh
- On Microsoft Windows computers:
  - To run in online mode, run:
    
    OIMBulkload\scripts\oim_blkld.bat
  - To run in offline mode, run:
    
    OIMBulkload\script\oim_blkld.bat -offline
Note:

OIMBlukload is the directory in which the scripts/sqls/csv_files/lib/sample_data directories are present.
From the main menu, select one of the options depending on the data you want to load, such as user, account, or role-related data, as described in Options Offered by the Utility.
From the second menu:
- Select CSV File if you are using CSV files as the input source.
- Select DB Table if you are using a database table as the input source.
When prompted, provide values for the input parameters described in Determining Values for the Input Parameters of the Utility.

Note:

See Determining Values for the Input Parameters of the Utility for information about the input parameters required for loading OIM User data. See corresponding sections for information about the input parameters required to load account, role, role hierarchy, role membership, and role category data.
Monitor the performance of the operation by following the steps given in Monitoring the Progress of the Operation.

13.5 Performance Best Practices for Bulk Load

Follow best practices for loading large volume of data during the bulk load operation.

To enhance the performance of the Account Bulk Load operation:

Split the data load in phases for a high-volume entity data load for Users/Accounts/Role Membership, for example, when data load is greater than 1 million for Users and greater than 250 thousand for Accounts.
The phase-wise load can be in the initial size of 500 thousand, and thereafter, in the size of 2 to 3 million Entity data.
Perform Stats gathering operation essentially after the first and second batch of data load.

For information about Stats gathering operation, see Monitoring Oracle Identity Manager Performance in the Performance and Tuning Guide.
For Account data load, when the source is database table, then make sure that relevant indexes are present on the columns as per the reconciliation matching rules. For more information, see Selecting Additional Indexes for Matching Module.
Oracle recommends loading organization data in online mode.

13.6 Loading OIM User Data

When you load OIM User data by using the Bulk Load utility, data is imported into the USR table of the database.

This section provides detailed information about the steps involved in loading OIM User data. It contains the following topics:

13.6.1 Overview of Loading OIM User Data

Loading OIM User data involves creating the input source for the bulk load operation, running the Bulk Load utility, and handling exceptions, if any.

The following is a summary of the steps involved in loading OIM User data:

Prepare your database for bulk load if not done already. See Preparing Your Database for a Bulk Load Operation for details.
Create the OIM User whose password will be used as the default password for all OIM Users created during the bulk load operation.
Create the input source for the bulk load operation.

If you want to use a database table as the input source, then create the table and copy user data into the table.

If you want to use CSV files as the input source, then create the CSV files and copy user data into the files. In addition, create a master.txt file containing the names of the files in the sequence in which you want to load data from them.
Determine values for the input parameters of the utility.
Stop Oracle Identity Manager.
Run the oim_blkld.sh or oim_blkld.bat script. See Running the Utility for information about running the oim_blkld.sh or oim_blkld.bat scripts.
Monitor the progress of the bulk load operation.
Determine the outcome of the bulk load operation.
If required, reload data that was not loaded during the first run.
Restart Oracle Identity Manager.
Verify the outcome of the bulk load operation.
Gather diagnostic data from the operation.
Remove temporary tables and files created during the operation.
Generate an audit snapshot.

13.6.2 Setting a Default Password for OIM Users Added by the Utility

The Bulk Load utility assigns the password of an existing OIM User to all OIM Users that are created during the operation.

The utility does not encrypt passwords that it assigns to OIM Users created during the bulk load operation. Instead, it assigns the password of an existing OIM User to all OIM Users that are created during the operation.

Note:

Each OIM User is required to change the password at first login.

When you run the utility, it prompts for the login name of the existing OIM User whose password you want to use as the default password for the new OIM Users. Before you run the utility, create this OIM User as follows:

Note:

You can create a user in Oracle Identity Manager dedicated for the bulk load operation, and later delete the user if it not required any more. Otherwise, any existing OIM User can be used to perform bulk load operations.

Log in to the Oracle Identity Self Service as a user with Create User privileges.
On the left navigation pane, under Administration, click Users. The Search Users page is displayed.
From the Actions menu, select Create. The Create User page is displayed with input fields for user profile attributes.
Specify values for the following fields:
- User Login
- First Name (optional)
- Last Name
- Organization: Select Xellerate Users.
- Password
- Confirm Password
Click Submit.

13.6.3 Creating the Input Source for the Bulk Load Operation

Depending on the input source that you want to use, you must apply the guidelines for using CSV files or creating database tables as the input source.

This section contains the following topics:

13.6.3.1 Using CSV Files As the Input Source

If you want to use CSV files as the input source for the bulk load operation, then apply the following guidelines while creating the CSV files:

The CSV files must be placed in the oimbulkload/csv_files directory.
The first line in the CSV file is called the control line. This line must contain a comma-separated list of column names of the USR table in the Oracle Identity Manager database.

Note:

Ensure that the Password column or any other encrypted column is not included in the list of columns. As mentioned earlier in this document, the utility assigns the password of an existing OIM User that you specify to all OIM Users that it loads into Oracle Identity Manager.
From the second line onward, the file must contain values for the columns in the control line. The order of columns in the first line and the values in the rest of the lines must be the same.

The following are sample contents of a CSV file:
```
USR_LOGIN,USR_FIRST_NAME,USR_LAST_NAME,UD_ADUSER_OBJECTGUID
john_doe, John, Doe, jdoe
jane_doe, Jane, Doe, janedoe
richard_roe, Richard, Roe, rroe
```
If the value in any column contains a comma, then that value must be enclosed in double quotation marks (").

The CSV file must contain values for all columns that are designated as mandatory in the USR table. The following table lists the mandatory columns required to load the USR table:

Mandatory Column	Description
USR_FIRST_NAME	The first name of the user
USR_LAST_NAME	The last name of the user

Note:

USR_LOGIN is not a mandatory column.
There are some key mandatory columns that you can ignore. For example, the ACT_KEY column in the USR table, which is populated by ORG_NAME.

Each row in the CSV file must have a unique value for the USR_LOGIN column in the USR table. If there are multiple files, you must ensure that USR_LOGIN values are unique across the CSV files. This check for uniqueness of USR_LOGIN values must also cover existing OIM Users in Oracle Identity Manager.

Ensuring that USR_LOGIN values are unique can be a time-consuming exercise. As an alternative, you can first perform the bulk load operation, fix USR_LOGIN values that are not unique, and then retry the loading operation for the modified user records. This is possible because the utility checks for uniqueness of USR_LOGIN values at run time and copies records that fail this check into the OIM_BLKLD_EX table. Later in this document, there are instructions on retrying the bulk load operation for records that are not loaded during the first run.
If you want to include an organization name in each user record, then add ORG_NAME in the control line and enter the organization name for each user from the second line onward. If ORG_NAME is not included, then the users must be assigned to the Xellerate Users organization.

Note:

All organization names listed under the ORG_NAME column in the CSV file must exist in Oracle Identity Manager.
If you want to include a manager name in each user record, then add MANAGER_NAME in the control line and enter the USR_LOGIN value of the manager for each user from the second line onward.

The utility looks up the USR_LOGIN values for managers after all user data, from all CSV files, is loaded into Oracle Identity Manager. If a USR_LOGIN value given in the MANAGER_NAME column does not exist in Oracle Identity Manager, then the lookup for that user record fails and the record is copied into the exception table, OIM_BLKLD_EX. At the end of the bulk load operation, you can perform the procedure described in Fixing Exceptions and Reloading Data Records to reload user records that fail the first run.
Note that the following default values are inserted into Oracle Identity Manager if the CSV file does not contain values for these columns:

ORG_NAME: Xellerate Users

USR_TYPE: End-User

USR_STATUS: Active

USR_EMP_TYPE: Full-Time
Create a master TXT file containing the names of the CSV files containing user data to be loaded. You can specify any name for the file, for example, master.txt. Save the master file in the oimbulkload/csv_files directory.

If you want to load multiple CSV files, then enter the name of each data CSV file on a separate line in the master file. Order the list of CSV file names in the sequence in which you want the utility to load data from the files. For example, suppose you have created three data CSV files, London_Users.csv, NewYork_Users.csv, and Tokyo_Users.csv. In the master file, you enter the names of the data CSV files in the following order:
```
Tokyo_Users.csv
London_Users.csv
NewYork_Users.csv
```
When you run the utility, data is loaded in this order. This is because the user data in London and New York may have a dependency on the Tokyo users. This is to ensure the manager-user hierarchy.
If the CSV file is generated on Microsoft Windows and is to be loaded on Linux environment, then remove the special characters, such as '\n\r', to avoid run-time errors.
Note:

While copying a CSV file from Windows to UNIX, Solaris, or Linux systems, some special characters, such as ^M, are appended to the file. This is because, the file from Windows is in DOS (ASCII) format and must be converted to ISO format.

Solaris preinstalls the dos2unix utility into the system to do this job. But for UNIX/Linux systems, the CSV file must be converted from DOS format to UNIX format to ensure sanity of the input file before being used in the Bulk Load operation. To do this, the syntax is:
```
# dos2unix CSV_FILE_NAME
```
If the dos2unix utility does not exist in the UNIX/Linux systems, then the administrator can install the utility for the respective UNIX/Linux versions by using the relevant documentation.

13.6.3.2 Creating Database Tables As the Input Source

If you want to use a database table as the input source for loading OIM User data, then apply the following guidelines while creating the database table:

Create the table in the Oracle Identity Manager database.
The table must contain the following primary key column:

OIM_BLKLD_USRSEQ NUMBER(19)

The utility uses this column as the primary key. If required, you can use a database sequence to populate this column.
The rest of the columns must be the same as the ones in the USR table that you want to use. In other words, ignore optional USR_ columns that you do not want to include in the table that you create.
Note that the following default values are inserted into Oracle Identity Manager if the table does not contain values for these columns:

ORG_NAME: Xellerate Users

USR_TYPE: End-User

USR_STATUS: Active

USR_EMP_TYPE: Full-Time
If you want to include an organization name in each user record, then add ORG_NAME in the control line and enter the organization name for each user from the second line onward. If ORG_NAME is not included, then the users must be assigned to the Xellerate Users organization.
If you want to include a manager name in each user record, then add MANAGER_NAME in the control line and enter the USR_LOGIN value of the manager for each user from the second line onward.

13.6.3.3 Structure of a Sample Database Table

Table 13-1 shows the structure of a sample database table.

Table 13-1 Structure of a Sample Database Table

Name	Null?	Type
USR_LOGIN	NOT NULL	VARCHAR2(256)
USR_FIRST_NAME		VARCHAR2(150)
USR_LAST_NAME	NOT NULL	VARCHAR2(150)
. . .	. . .	. . .
OIM_BLKLD_USRSEQ	NOT NULL	NUMBER(19)

13.6.4 Determining Values for the Input Parameters of the Utility

You must set correct values for the input parameters of the Bulk Load utility for the type of data being loaded.

The following are input parameters of the utility:

Oracle Home

Value of the ORACLE_HOME environment variable on the host computer for the Oracle Identity Manager database
Database Connection String

Connection string to connect to the database that must be entered in the following format:

//HOST_IP_ADDRESS:PORT_NUMBER/SERVICE_NAME
OIM DB User

Database login ID of the Oracle Identity Manager database user
OIM DB Pwd

Password of the Oracle Identity Manager database user

The database user password is to be entered twice when prompted.
Master file name

Name of the file containing names of the CSV data files to be loaded

This parameter is used only if the input source is a single or multiple CSV files. You place the master file and CSV data files in the oimbulkload/csv_files directory. See Using CSV Files As the Input Source for more information.
Tmp table name

Name of the temporary table to be used as the input source

This parameter is used only if the input source for the bulk load operation is a database table. See Creating Database Tables As the Input Source for more information.
Control Line

Comma-separated list of names of columns to be loaded from the database table into Oracle Identity Manager

This parameter is used only if the input source for the bulk load operation is a database table.
Tablespace Name

Name of the tablespace in which temporary tables are to be created during the bulk load operation. If the user does not provide the tablespace name, then it will pick the default tablespace.

See Preparing Your Database for a Bulk Load Operation for more information.
Date format

Date format used by date columns in the CSV files

This parameter is used only if the input source is a single or multiple CSV files.

The date format must match the following:
- Oracle supported date formats, such as dd-mm-yyyy or MM-DD-YYYY
- The date format specified in the CSV file
Batch Size

Number of user records that must be processed by the utility as a single transaction

The batch size can influence the performance of the bulk load operation. The default value of this parameter is 10000.
Debug Flag

You can specify Y or N as the value of this parameter. If this parameter is set to Y, then the utility records detailed information about events that occur during the bulk load operation. See Data Recorded During the Operation for more information.
User ID for default password

Login name of the OIM User that you create by performing the procedure described in Setting a Default Password for OIM Users Added by the Utility.

Note:

While using the ATPS database, if you use the Bulk Load utility then follow the steps provided below. This is available after applying the Oracle Identity Governance Bundle Patch 12.2.1.4.2204XX.

Complete the following steps while using the ATPS database:

Install the DB version 18c or 19c and configure only the DB without instance on the machine while running the OIM Bulk Load utility and provide the installed location that is, base location as ORACLE_HOME such as, export ORACLE_HOME=/home/opc/db19c.
Copy the wallet files (cwallet.sso, ewallet.p12 and ojdbc.properties) to a different folder, say 'wallet1', and use it while running the oimBulkload utility.

Update the ojdbc.properties value in the wallet as follows:

# Connection property while using Oracle wallets.
#oracle.net.wallet_location=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=${TNS
_ADMIN})))
SSL_SERVER_DN_MATCH=yes
# FOLLOW THESE STEPS FOR USING JKS
# (1) Uncomment the following properties to use JKS.
# (2) Comment out the oracle.net.wallet_location property above
# (3) Set the correct password for both trustStorePassword and
keyStorePassword.
# It's the password you specified when downloading the wallet from OCI
Console or the Service Console.
javax.net.ssl.trustStoreType=JKS
javax.net.ssl.trustStore=/home/opc/idm/truststore.jks
javax.net.ssl.trustStorePassword=<truststorepwd>
javax.net.ssl.keyStoreType=JKS
javax.net.ssl.keyStore=/home/opc/idm/keystore.jks
javax.net.ssl.keyStorePassword=<keystorepwd>

Set TNS_ADMIN in the CMD prompt to the above wallet export as TNS_ADMIN=/home/opc/idm/wallet1.
During execution of bulk load utility, for the option ATP-S setup, provide 'y' as input.
For connection string, instead of "//HostIPAddress:Port/ServiceName", provide the tnsname only. This is visible in thetnsnames.ora file.
Sample Value:

###### Get the database connect string ######## Enter the OIM database hostname/ip from walletfmwatps2_tp

13.6.5 Monitoring the Progress of the Operation

During the bulk load operation, you can query the OIM_BLKLD_LOG table for information about the progress of the operation.

For example, you can run the following query to see progress messages generated during the bulk load operation to load OIM User data:

SELECT MSG FROM OIM_BLKLD_LOG
WHERE MODULE = 'USER' AND LOG_LEVEL = 'PROGRESS_MSG'
ORDER BY MSG_SEQ_NO;

Errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table. The following is an example of the query to retrieve error messages:

SELECT MSG FROM OIM_BLKLD_LOG 
WHERE MODULE = 'USER' AND LOG_LEVEL = 'ERROR'
ORDER BY MSG_SEQ_NO;

13.6.6 Handling Exceptions Recorded During the Operation

At the end of a bulk load operation, the utility records statistics related to the operation.

The statistics are recorded in the following file:

oimbulkload/logs_YYYYMMDD_hhmm/oim_blkld_user_load_summary.log

To determine if there were exceptions during the operation, open this log file and look for the number against the Number of Records Rejected label. If the number of rejected records is greater than zero, then exceptions were thrown during the operation. User records that are rejected by the utility are recorded in the exception table (OIM_BLKLD_EX_SUFFIX). For each rejected record, the EXCEPTION_MSG column in the OIM_BLKLD_EX_SUFFIX table stores information about the reason the record could not be loaded.

The following example shows sample statistics recorded in the log file at the end of a bulk load operation to store OIM User data.

****************************************************************
Processing File: u10.csv
================================================================
U S E R    L O A D    S T A T I S T I C S    F O R   F I L E : u10.csv
================================================================
Start Time:   08-AUG-08 11.44.12.228000 AM
End Time:     08-AUG-08 11.44.13.368000 AM
Number of Records Processed:  10
Number of Records Loaded:     8
Number of Records Rejected:   2
================================================================
The name of the TMP table used during the load:
OIM_BLKLD_TMP_U101

The name of the Exception table used during the load:
OIM_BLKLD_EX_U101

****************************************************************
Processing File: u10b.csv

================================================================
U S E R    L O A D    S T A T I S T I C S    F O R   F I L E : u10b.csv
================================================================
Start Time:   08-AUG-08 11.44.15.368000 AM
End Time:     08-AUG-08 11.44.15.540000 AM
Number of Records Processed:  16
Number of Records Loaded:     15
Number of Records Rejected:   1
================================================================
The name of the TMP table used during the load:
OIM_BLKLD_TMP_U10B2

The name of the Exception table used during the load:
OIM_BLKLD_EX_U10B2
================================================================

================================================================
Time taken in re-building indexes and enabling FK constraints
================================================================
Start time:      08-AUG-08 11.44.15.556000 AM
End Time:        08-AUG-08 11.46.50.586000 AM
================================================================

In this sample, the number of rejected records is 2. If the log file shows that any records were rejected by the utility, then see Fixing Exceptions and Reloading Data Records for information about retrying the load operation for these records.

Note:

At the end of each bulk load operation, it is recommended that you create a backup of the exception tables.

13.6.7 Fixing Exceptions and Reloading Data Records

Errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table.

This section describes exception handling for the bulk load operation and how to reload rejected records. It contains the following topics:

13.6.7.1 About Fixing Exceptions

As mentioned earlier, errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table. The following is an example of the query to retrieve error messages:

SELECT MSG FROM OIM_BLKLD_LOG 
WHERE MODULE = 'USER' AND LOG_LEVEL = 'ERROR'
ORDER BY MSG_SEQ_NO;

An exception table OIM_BLKLD_EX_SUFFIX is created for each data table used as the input source during the bulk load operation. Records that do not meet the criteria for the operation are copied into this exception table. The suffix appended to the name of each exception table is the same as suffix appended to the name of the corresponding data table.

13.6.7.2 Reloading Rejected Records

To reload rejected records:

Create a backup of the exception table in which rejected records are stored.

Note:

Although this is an optional step, it is recommended that you create a backup.
Review each record in the exception table, and fix errors in the data based on the message recorded in the EXCEPTION_MSG column.
After you fix errors in all the rejected records in an exception table, rename the table to OIM_BLKLD_TMP_SUFFIX and then use it as the input source.
Load records from the OIM_BLKLD_TMP_SUFFIX table by running the utility. See Running the Utility for more information.
Repeat Steps 1 through 4 until the Number of Records Rejected label in the oim_blkld_user_load_summary.log file shows the value 0.
Restart Oracle Identity Manager if loading was done in offline mode.

Note:

Being a database-intensive operation by design, Bulk Load disables the constraints and indexes on the relevant Oracle Identity Manager entity tables during the start of the operation. Bulk Load operation failure towards the end of the load might at times render the indexes and constraints in disabled state. To identify and fix this issue, manually restore the indexes and constraints as follows:

Identify the unusable indexes and disabled constraints. To do so, the following SQL queries or similar mechanism can be used:

SELECT TABLE_NAME, CONSTRAINT_NAME FROM user_constraints WHERE status = 'DISABLED';
SELECT index_name FROM user_indexes WHERE status = 'UNUSABLE';

Enable the constraints and rebuild the indexes manually, as shown:

ALTER TABLE TABLE_NAME ENABLE CONSTRAINT CONSTRAINT_NAME;
ALTER INDEX INDEX_NAME REBUILD;

13.6.8 Verifying the Outcome of the Bulk Load Operation

Verify the outcome of the Bulk Load operation for one of the OIM Users added by the utility by provisioning a resource for the user and by modifying the account of the user.

To verify the outcome of the bulk load operation, check if you are able to perform the following steps for one of the OIM Users added by the utility:

Note:

These steps leave footprints in the system, and therefore, the bulk load verification must be performed by using a test user. If you do not want to leave the footprints in the system, then revert the changes. For example, if you have provisioned a resource to a OIM User, then deprovision the resource after testing the outcome of the bulk load operation.
If Oracle Identity Manager is synchronized with LDAP, then after running the user data upload, run the Bulk Load Post Process scheduled job with the LdapSync option set to Yes. If you want to generate the random password and send an email to the user, then you must configure the email notification and set the Generate Password and Notification parameters to Yes in the Scheduler. See Predefined Scheduled Tasks in Administering Oracle Identity Governance for information about the Bulk Load Post Process scheduled job.

Log in as the OIM User. The system should prompt you to change the password.
Provision a resource for the OIM User.
Add the OIM User to a role.
Modify the account profile of the OIM User.
Revoked the resource provisioned to the OIM User.
Unassign the OIM User from the role to which the user was added earlier.
Modify the account profile again to restore the profile to its original state.
Check if the User Resource Access report (an operational report) and the User Resource Access History report can be generated for the user.

13.6.9 Generating an Audit Snapshot

If required, you can generate an audit snapshot of Oracle Identity Manager data after a bulk load operation, or at any time during the bulk load operation.

You can also generate audit snapshots by selecting option 7 in the Bulk Load utility. The utility uses the audit engine shipped with Oracle Identity Manager. Internally, the GenerateSnapshot script is called when you run the audit utility. Similarly, the GenerateSnapshot script is called when you select the option to generate an audit snapshot.

Note:

Oracle Identity Manager must be up and running when you run the audit utility.

Before you generate an audit snapshot, for running the GenerateSnapshot script, you must set the following environment variables:

APP_SERVER: weblogic
OIM_ORACLE_HOME: c:\work1\Oracle_IDM1
JAVA_HOME: C:\jdk160
MW_HOME: c:\work1
WL_HOME: c:\work1\wlserver_10.3
DOMAIN_HOME: C:\work1\user_projects\domains\base_domain

Note:

C:\work1\ is a sample directory path of MW_HOME.

See Configuring Auditing in Administering Oracle Identity Governance for information about the procedure to generate audit snapshots.

13.6.10 Running the Bulk Load Post Process Scheduled Task

The Bulk Load Post Process scheduled task does the post processing operations for users loaded via the Bulk Load utility.

There is no post process job for any other entity loaded into Oracle Identity Governance by using the Bulk Load utility. The Bulk Load Post Process job can only perform the following operations:

Password generation: By default, the Bulk Load utility copies the same password of a user created by Oracle Identity Governance, which it prompts while execution. With this, the entire set of users have the same password initially. Then you can generate a random password by running the Bulk Load Post Process job.
Email notification: Oracle Identity Governance notifies all the users with their credentials.
LDAP synchronization: Bulk Load Post Process job takes care of the bulk loaded users to the LDAP configured in LDAP synchronization mode.

See Predefined Scheduled Tasks in Administering Oracle Identity Governance for more information about the Bulk Load Post Process scheduled task.

13.7 Loading Account Data

Loading account data involves creating the input source for the bulk load operation, running the Bulk Load utility, and handling exceptions, if any.

This section provides detailed information about the steps involved in loading account data. It contains the following topics:

13.7.1 Overview of Loading Account Data

Loading account data involves creating the input source for the bulk load operation, running the Bulk Load utility, and handling exceptions, if any.

The following is a summary of the steps involved in loading account data:

Prepare your database for a bulk load operation, if not already done. See Preparing Your Database for a Bulk Load Operation for details.
Create the input source for the bulk load operation.

If you want to use a database table as the input source, then create the table and copy account data into the table.

If you want to use CSV files as the input source, then create the CSV files and copy account data into the files.
Determine values for the input parameters of the utility.
Stop Oracle Identity Manager.
Run the oim_blkld.sh or oim_blkld.bat script.
Monitor the progress of the bulk load operation.
Determine the outcome of the bulk load operation.
If required, reload data that was not loaded during the first run.
Restart Oracle Identity Manager.
Verify the outcome of the bulk load operation.
Gather diagnostic data from the operation.
Remove temporary tables and files created during the operation.

13.7.2 Requirements and Features of the Bulk Load Operation for Account Data

There are certain requirements of loading account data by running the Bulk Load utility, for example reconciliation must be set up and you should be able to test reconciliation by importing a few accounts from the target system.

The following are requirements and features of the bulk load operation for account data:

Reconciliation must be set up and you should be able to test reconciliation by importing a few accounts from the target system.
Only accounts for which there are corresponding OIM Users can be loaded.
A target system that requires multiple IT resources is not supported.
Duplicate accounts cannot be detected during a bulk load operation. If there are multiple entries for the same account in the input source, then multiple accounts are created for the corresponding OIM User.
For a particular target system, if there are multiple provisioning processes/process forms in Oracle Identity Manager, then the utility uses the default provisioning process for the resource object.
Information about the stage up to which earlier bulk load operations progressed is not stored. In other words, the utility cannot resume a bulk load operation. You must backup the Oracle Identity Manager database before a bulk load operation. If you want to retry a bulk load operation, you must first restore the database and then rerun the procedure.
Bulk Load utility takes the corresponding application instance name as input to load account data. If the application instance name is not known to the user, then Bulk Load utility prompts for the resource object name and IT resource name, based on which account data is loaded.
Loading accounts where the target system is Active Directory, make sure that the input source (CSV file or database table) have the following attributes as mandatory in the attribute list along with its values:
- UD_ADUSER_COMMONNAME
- UD_ADUSER_USERPRINCIPALNAME
Failing to load values for these attributes at the time of Bulkload can result into failures in the provisioning-related operations at a later stage for this target.
If you are loading account data with entitlements, then:
1. Ensure that the lookup recon task has been run to populate the appropriate lookup table, and the table is recent.
2. Run the Entitlement List scheduled job and let it complete before loading the data. To verify, ensure that all the entitlements are showing up in the ENT_LIST table. See Predefined Scheduled Tasks in Administering Oracle Identity Governance for information about the Entitlement List scheduled job.
Failure to do these steps before account load may lead to missing entitlements on user pages. Such situation can be corrected by running the Entitlement Assignments scheduled job to completion after bulk load, which is avoidable if steps 1 and 2 are followed. See Predefined Scheduled Tasks in Administering Oracle Identity Governance for information about the Entitlement Assignments scheduled job.

13.7.3 Creating the Input Source for the Bulk Load Operation

Depending on the input source that you want to use, you must apply the guidelines for using CSV files or creating database tables as the input source.

This section contains the following topics:

13.7.3.1 Using CSV Files As the Input Source

If you want to use CSV files as the input source for the bulk load operation, then apply the following guidelines while creating the CSV files:

The CSV files must be placed in the oimbulkload/csv_files directory.
The first line in the CSV file is called the control line. This line must contain a comma-separated list of column names in the account (UD_*) table into which you want to load the account data. To find out the UD_ table, go to the process form in the Design Console. See Developing Provisioning Processes for information about process forms.

Note:

Ensure that the Password column or any other encrypted column is not included in the list of columns.
From the second line onward, the file must contain values for the columns in the control line. The order of columns in the first line and the values in the rest of the lines must be the same.
If the value in any column contains a comma, then that value must be enclosed in double quotation marks (").
The CSV file must contain values for all columns that are designated as mandatory in the account table. The key mandatory columns in the account table must be ignored.
If you want to load account data into parent and child tables, then you must create one parent CSV file and one child CSV file for each child table. For example if you are loading data into one parent table and three child tables, then you must create one parent CSV file and three child CSV files.
If you want to load account data into parent and child tables, then at least one column must be the same in both tables. This column corresponds to the link attribute between the parent and child CSV files. The following example illustrates this:

The following are sample contents of a parent CSV file:
```
UD_ADUSER_UID,,UD_ADUSER_FNAME,UD_ADUSER_LNAME,UD_ADUSER_MNAME,UD_ADUSER_FULLNAME,UD_ADUSER_OBJECTGUID
ADTEST1,"7~CN=ForeignSecurityPrincipals,dc=example,dc=com",adtest1,adtest1,,adtest1,102
```
Note:
- The UD_ADUSER_OBJECTGUID column is mandatory in the parent CSV file for loading accounts by using the bulk load operation. This column must be added to the parent CSV file in spite of nullable column in the database.
- Common key column being defined in parent and child CSV need not be present in the child database table. This column is being used by the BulkUpload utility to identify the record key and make the respective entries in corresponding child tables. For example, UD_ADUSER_UID. This column should be present in parent table but need not be present in any of the child table.
The following are sample contents of a child CSV file:
```
UD_ADUSER_UID,UD_ADUSRC_GROUPNAME
ADTEST1,"7~CN=ForeignSecurityPrincipals,dc=example,dc=com",group2
```
The UD_ADUSER_UID column is common to both the parent file and the child file.
If the CSV file is generated on Microsoft Windows and is to be loaded on Linux environment, then remove the special characters, such as '\n\r', to avoid run-time errors.

13.7.3.2 Creating Database Tables As the Input Source

If you want to use a database table as the input source for loading account data, then apply the following guidelines while creating the database table:

Create the table in the Oracle Identity Manager database.
The table must contain the following primary key column:

OIM_BLKLD_USRSEQ NUMBER(19)

The utility uses this column as the primary key. If required, you can use a database sequence to populate this column.
The rest of the columns must be the same as the ones in the account (UD_) table that you want to use. In other words, ignore optional UD_ columns that you do not want to include in the table that you create.

13.7.3.3 Sample Parent Table Structure

Table 13-2 shows the structure of a sample parent table.

Table 13-2 Structure of a Sample Database Table

Name	Null?	Type
UD_ADUSER_UID		VARCHAR2(20)
UD_ADUSER_ORGNAME		VARCHAR2(256)
UD_ADUSER_FNAME		VARCHAR2(80)
UD_ADUSER_LNAME		VARCHAR2(80)
UD_ADUSER_MNAME		VARCHAR2(80)
UD_ADUSER_FULLNAME		VARCHAR2(240)
OIM_BLKLD_SEQ	NOT NULL	NUMBER(19)

13.7.3.4 Sample Child Table Structure

Table 13-3 shows the structure of a sample child table.

Table 13-3 Structure of a Sample Child Database Table

Name	Null?	Type
UD_ADUSER_UID		VARCHAR2(20)
UD_ADUSER_ORGNAME		VARCHAR2(256)
UD_ADUSRC_GROUPNAME		VARCHAR2(32)
OIM_BLKLD_SEQ	NOT NULL	NUMBER(19)

13.7.4 Determining Values for the Input Parameters of the Utility

You must set correct values for the input parameters of the Bulk Load utility for the type of data being loaded.

The following are input parameters of the utility:

Oracle Home

Value of the ORACLE_HOME environment variable on the host computer for the Oracle Identity Manager database.
Database Connection String

Connection string to connect to the database that must be entered in the following format:

//HOST_IP_ADDRESS:PORT_NUMBER/SERVICE_NAME
OIM DB User

Database login ID of the Oracle Identity Manager database user.
OIM DB Pwd

Password of the Oracle Identity Manager database user. This must be entered twice when prompted.
Application instance name (APP_INSTANCE)

Name of the application instance corresponding to the account data to be loaded. If the user is not aware of the application instance name, then Account Bulkload utility prompts for the resource object name and IT resource name. The prompt is as shown:
```
Do you know the Application Instance name? (Y,y,N,n)
```
If you enter Y or y, then you are prompted for the application instance name. If you enter N or n, then you are prompted for the following:
- Resource Object Name (OBJ_NAME)
  
  If the user is not aware of the application instance name, then Bulk Load utility prompts for the resource object name corresponding to the account data to be loaded.
- IT Resource Name
  
  Name of the IT resource created for the target system. This is required only when the user is not aware of the application instance name. The account bulkload utility first prompts for resource object name, and then prompts for IT resource name.
CSV file names

Names of the CSV files to be used as the input source.

This parameter is used only if the input source is CSV files. See Using CSV Files As the Input Source for more information. If you are loading data from parent and child CSV file, then use a comma-delimited list to enter the names of the files. The name of the parent CSV file must be provided first, and it must be followed by the names of the child CSV files. In addition, enter the column that links the parent and child data.
Tmp table name

Name of the temporary table to be used as the input source.

This parameter is used only if the input source for the bulk load operation is a database table. See Creating Database Tables As the Input Source for more information.
Control Line

Comma-separated list of names of columns to be loaded from the database table into Oracle Identity Manager.

This parameter is used only if the input source for the bulk load operation is a database table.
Tablespace Name

Name of the tablespace in which temporary tables are to be created during the bulk load operation (if end user won't provide the tablespace name then it will pick the default tablespace).

See Preparing Your Database for a Bulk Load Operation for more information.
Date format

Date format used by date columns in the CSV files.

This parameter is used only if the input source is a single or multiple CSV files.

The date format must match the following:
- Oracle supported date formats, such as dd-mm-yyyy or MM-DD-YYYY
- The date format specified in the CSV file
Batch Size

Number of user records that must be processed by the utility as a single transaction.

The batch size can influence the performance of the bulk load operation. The default value of this parameter is 10000.
Debug Flag

You can specify Y or N as the value of this parameter. If this parameter is set to Y, then the utility records detailed information about events that occur during the bulk load operation. See Data Recorded During the Operation for more information.
Application Instance (APP_INSTANCE)

Name of the application instance corresponding to the account data to be loaded.

If the user is not aware of the application instance name, then account bulkload utility prompts for the Object name (OBJ_NAME)
User ID (USR_LOGIN)

The user login ID that is used to determine the user that provisioned Accounts using Bulk Load utility.

Note:

After you apply Bundle Patch 12.2.1.4.201011, the Bulk Load Utility for loading account data prompts for additional information. See Bulk Load Utility for Loading Accounts in Oracle Identity Governance Bundle Patch Readme for information about the additional input.

13.7.5 Monitoring the Progress of the Operation

During the bulk load operation, you can query the OIM_BLKLD_LOG table for information about the progress of the operation.

For example, you can run the following query to see progress messages generated during the bulk load operation to load account data:

SELECT MSG FROM OIM_BLKLD_LOG
WHERE MODULE = 'ACCOUNT' AND LOG_LEVEL = 'PROGRESS_MSG'
ORDER BY MSG_SEQ_NO;

Errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table. The following is an example of the query to retrieve error messages:

SELECT MSG FROM OIM_BLKLD_LOG 
WHERE MODULE = 'ACCOUNT' AND LOG_LEVEL = 'ERROR'
ORDER BY MSG_SEQ_NO;

13.7.6 Handling Exceptions Recorded During the Operation

At the end of a bulk load operation, the utility records statistics related to the operation.

The statistics are recorded in the following file:

oimbulkload/logs_YYYYMMDD_hhmm/oim_blkld_account_load_summary.log

The following is the sample statistics recorded in the log file at the end of a bulk load operation to store account data.

=============================================================
A C C O U N T    L O A D    S T A T I S T I C S
=============================================================
Start Time:   22-JUL-08 03.59.30.206000 PM
End Time:     22-JUL-08 04.03.21.126000 PM
Number of Records Processed:  100026
Number of Records Loaded:     100000
Number of Records Rejected:   26
=============================================================
 
The names of the TMP tables used during the load:
OIM_BLKLD_TMP_P100001
OIM_BLKLD_TMP_C100002
The names of the Exception tables used during the load:
OIM_BLKLD_EX_P100001
OIM_BLKLD_EX_C100002

In this sample, the number of rejected records is 26. If the log file shows that any records were rejected by the utility, then see Fixing Exceptions and Reloading Data Records for information about retrying the load operation for these records.

Note:

At the end of each bulk load operation, it is recommended that you create a backup of the exception tables.

13.7.7 Fixing Exceptions and Reloading Data Records

Errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table.

Note:

If you want to load data from CSV files for multiple target systems, then you can apply one of the following approaches:

Approach 1: Run the utility for all the sets of CSV files, and then perform the procedure described in this section.
Approach 2: Run the utility for one set of CSV files, and perform the procedure described in this section. Then, repeat this procedure for the next set of CSV files.

This section describes exception handling for the bulk load operation and how to reload the rejected records. It contains the following topics:

13.7.7.1 About Fixing Exceptions

Errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table. The following is an example of the query to retrieve error messages:

SELECT MSG FROM OIM_BLKLD_LOG 
WHERE MODULE = 'ACCOUNT' AND LOG_LEVEL = 'ERROR'
ORDER BY MSG_SEQ_NO;

13.7.7.2 Reloading Rejected Records

To reload rejected records:

Create a backup of the exception table in which rejected records are stored.

Note:

Although this is an optional step, it is recommended that you create a backup.
Review each record in the exception table, and fix errors in the data based on the message recorded in the EXCEPTION_MSG column.
After you fix errors in all the rejected records in an exception table, rename the table to OIM_BLKLD_TMP_SUFFIX and then use it as the input source.
Load records from the OIM_BLKLD_TMP_SUFFIX table by running the utility. See Running the Utility for more information about running the Bulk Load utility.
Repeat Steps 1 through 4 until the Number of Records Rejected label in the oim_blkld_account_load_summary.log file shows the value 0.
Restart Oracle Identity Manager if loading was done in offline mode.

Note:

Identify the unusable indexes and disabled constraints. To do so, the following SQL queries or similar mechanism can be used:

SELECT TABLE_NAME, CONSTRAINT_NAME FROM user_constraints WHERE status = 'DISABLED';
SELECT index_name FROM user_indexes WHERE status = 'UNUSABLE';

Enable the constraints and rebuild the indexes manually, as shown:

ALTER TABLE TABLE_NAME ENABLE CONSTRAINT CONSTRAINT_NAME;
ALTER INDEX INDEX_NAME REBUILD;

13.7.8 Verifying the Outcome of the Bulk Load Operation

Verify the outcome of the Bulk Load operation for one of the account added by the utility by verifying if the newly created account is displayed in the UI and by logging in to the target system by using the credentials of the account.

To verify the outcome of the bulk load operation, check if you are able to perform the following steps for one of the OIM Users for whom an account has been added by the utility:

Log in as the OIM User, and check if the newly created account is displayed in the Accounts tab of the User details page or in My Accounts tab of My Access page for the user.
Log in to the target system by using the credentials of the newly created account.

13.8 Loading Role, Role Hierarchy, Role Membership, and Role Category Data

Loading role, role hierarchy, role membership, and role category data involves creating the input source for the bulk load operation, running the Bulk Load utility, and handling exceptions, if any.

This section provides detailed information about the steps involved in loading role, role hierarchy, role membership, and role category data. It contains the following topics:

13.8.1 Overview of Loading Role, Role Hierarchy, Role Membership, and Role Category Data

Loading role, role hierarchy, role membership, and role category data involves creating the input source for the bulk load operation, running the Bulk Load utility, and handling exceptions, if any.

The following is a summary of the steps involved in loading role-related data:

Prepare your database for a bulk load operation, if not already done. See Preparing Your Database for a Bulk Load Operation for details.
Create the input source for the bulk load operation.

If you want to use a database table as the input source, then create the table and copy role-related data into the table.

If you want to use CSV files as the input source, then create the CSV files and copy role-related data into the files. In addition, create a master.txt file containing the names of the files in the sequence in which you want to load data from them.
Determine values for the input parameters of the utility.
Stop Oracle Identity Manager.
Run the oim_blkld.sh or oim_blkld.bat script.
Monitor the progress of the bulk load operation.
Determine the outcome of the bulk load operation.
If required, reload data that is not loaded during the first run.
Restart Oracle Identity Manager.
Verify the outcome of the bulk load operation.
Gather diagnostic data from the operation.
Remove temporary tables and files created during the operation.

13.8.2 Creating the Input Source for the Bulk Load Operation

Depending on the input source that you want to use, you must apply the guidelines for using CSV files or creating database tables as the input source.

This section contains the following topics:

13.8.2.1 Using CSV Files As the Input Source

If you want to use CSV files as the input source for the bulk load operation, then apply the following guidelines while creating the CSV files:

The CSV files must be placed in the oimbulkload/csv_files directory.
The first line in the CSV file is called the control line.
This line must contain a comma-separated list of column names based on the selected role upload (role, role hierarchy, role membership, and role category) in the Oracle Identity Manager database.
From the second line onward, the file must contain values for the columns in the control line. The order of columns in the first line and the values in the rest of the lines must be the same. The following is a sample content of a role (UGP) CSV file:
```
UGP_ROLENAME,UGP_NAMESPACE,USR_LOGIN,ORG_NAME,INCLUDE_HIERARCHY
"Finance Controllers",Default,XELSYSADM,Finance,YES
"Finance Controllers",Default,XELSYSADM,Requests,YES
```
Role load is capable of publishing the roles to organizations to follow the security model in Oracle Identity Manager, with an option to include hierarchy.

As a value of the ORG_NAME parameter, specify the organization name, such as Finance or Requests, to which you want to publish the roles. Specify YES for INCLUDE_HIERARCHY if you want to publish the roles to the specified organization and its suborganizations. Specify NULL or NO for INCLUDE_HIERARCHY if you want to publish the roles only to the specified organization and not its suborganizations. If you do not specify values for the ORG_NAME and INCLUDE_HIERARCHY parameters, then by default, the roles are published to the Top organization with hierarchy.
If the value in any column contains a comma, then that value must be enclosed in double quotation marks (").
The CSV file must contain values for all columns that are designated as mandatory in the respective role tables.
The CSV file must contain values for all columns that are designated as mandatory depending on the upload role data, role hierarchy data, role membership data, and role category data.
- Role UGP): UGP_ROLENAME,UGP_NAMESPACE,USR_LOGIN,ORG_NAME,INCLUDE_HIERARCHY (UGP_NAMESPACE,ORG_NAME)
  
  INCLUDE_HIERARCHY can be left as null when not required.
- Role Hierarchy (GPG): UGP_NAME, GPG_UGP_NAME
- Role Membership (USG): UGP_NAME, USR_LOGIN
- Role Category (ROLE_CATEGORY): ROLE_CATEGORY_NAME
Each row in the CSV file must have a unique value for the combinationation of manadatory columns.
The following default values are inserted into Oracle Identity Manager if the CSV file does not contain values for these columns:
- For Role (UGP)
  
  ROLE_CATEGORY_NAME: Default
  
  UGP_DISPLAY_NAME: Defaults to UGP_NAME
  
  ORG_NAME: TOP
  
  INCLUDE_HIERARCHY: YES
- For Role Hierarchy (GPG)
  
  None
- For Role Membership (USG)
  
  RUL_KEY: RUL_KEY from RUL table with RUL_NAME as 'Default'
  
  USG_PRIORITY: group and rank based on UGP_KEY based on the rows given for upload.
- Role Category (ROLE CATEGORY)
  
  None
Create a master TXT file containing the names of the CSV files containing role data to be loaded. You can specify any name for the file, for example, master.txt. Save the master file in the oimbulkload/csv_files directory.

If you want to load multiple CSV files, then enter the name of each data CSV file on a separate line in the master file. Order the list of CSV file names in the sequence in which you want the utility to load data from the files. For example, suppose you have created three data CSV files, Role1.csv, Role2.csv, and Role3.csv. In the master file, enter the names of the data CSV files in the following order:

Role1.csv

Role2.csv

Role3.csv

When you run the utility, data is loaded in this order.
If the CSV file is generated on Microsoft Windows and is to be loaded on Linux environment, then remove the special characters, such as '\n\r', to avoid run-time errors.

13.8.2.2 Creating Database Tables As the Input Source

If you want to use a database table as the input source for loading OIM User data, then apply the following guidelines while creating the database table:

Create the table in the Oracle Identity Manager database.
The table must contain the following primary key column:

OIM_BLKLD_USRSEQ NUMBER(19)

The utility uses this column as the primary key. If required, you can use a database sequence to populate this column.
The rest of the columns must be the same as the ones in the respective role tables that you want to use.

13.8.2.3 Structure of a Sample Database Table

Table 13-4 shows the structure of a sample database role table.

Table 13-4 Structure of a Sample Database Table

Role	NULL	Type
UGP_ROLENAME	NOT NULL	VARCHAR2(2000)
UGP_NAMESPACE		VARCHAR2(512)
ORG_NAME	NOT NULL	VARCHAR2(256)
INCLUDE_HIERARCHY	NOT NULL	VARCHAR2(256)
...	...	...
OIM_BLKLD_USRSEQ	NOT NULL	NUMBER(19)

Note:

ORG_NAME and INCLUDE_HIERARCHY are required for loading roles only, and not for role hierarchy, role membership, and role category.

13.8.2.4 Determining the UGP_NAME Generated After Role Load

Bulkload utility generates UGP_NAME during role load in the following format:

UGP_NAMESPACE.UGP_ROLENAME

By default, the value of UGP_NAMESPACE is Default, when you do not provide any specific value for UGP_NAMESPACE in the CSV file. To determine the generated UGP_NAME:

If UGP_NAMESPACE is null in the CSV file, then the namespace value is Default, and the generated UGP_NAME is equal to the value of UGP_ROLENAME.
If UGP_NAMESPACE is not null and has a defined value in the CSV file, then the generated UGP_NAME is equal to the value of UGP_NAMESPACE.UGP_ROLENAME.

On the basis of the UGP_NAME generation methodology, you can determine the UGP_NAME values for the next loading of role hierarchy, role membership, and role category, even if you do not have direct access to the database. Otherwise, you can check the generated value of UGP_NAME in the UGP table.

13.8.3 Determining Values for the Input Parameters of the Utility

You must set correct values for the input parameters of the Bulk Load utility for the type of data being loaded.

The following are input parameters of the utility:

Oracle Home

Value of the ORACLE_HOME environment variable on the host computer for the Oracle Identity Manager database
Database Connection String

Connection string to connect to the database that must be entered in the following format:

//HOST_IP_ADDRESS:PORT_NUMBER/SERVICE_NAME
OIM DB User

Database login ID of the Oracle Identity Manager database user
OIM DB Pwd

Password of the Oracle Identity Manager database user. Enter the password twice when prompted.
CSV file names

Names of the CSV files to be used as the input source

This parameter is used only if the input source is CSV files. See Using CSV Files As the Input Source for more information. If you are loading data from parent and child CSV file, then use a comma-delimited list to enter the names of the files. The name of the parent CSV file must be provided first, and it must be followed by the names of the child CSV files.
Tmp table name

Name of the temporary table to be used as the input source

This parameter is used only if the input source for the bulk load operation is a database table. See Creating Database Tables As the Input Source for more information.
Control Line

Comma-separated list of names of columns to be loaded from the database table into Oracle Identity Manager

This parameter is used only if the input source for the bulk load operation is a database table.
Tablespace Name

Name of the tablespace in which temporary tables are to be created during the bulk load operation (if end user won't provide the tablespace name then it will pick the default tablespace)

See Preparing Your Database for a Bulk Load Operation for more information.
Date format

Date format used by date columns in the CSV files. This is prompted only for role load, and not for role hierarchy, role membership, and role category.

This parameter is used only if the input source is a single or multiple CSV files.

The date format must match the following:
- Oracle supported date formats, such as dd-mm-yyyy or MM-DD-YYYY
- The date format specified in the CSV file
Batch Size

Number of user records that must be processed by the utility as a single transaction

The batch size can influence the performance of the bulk load operation. The default value of this parameter is 10000.
Debug Flag

You can specify Y or N as the value of this parameter. If this parameter is set to Y, then the utility records detailed information about events that occur during the bulk load operation. See Data Recorded During the Operation for more information.

13.8.4 Monitoring the Progress of the Operation

During the bulk load operation, you can query the OIM_BLKLD_LOG table for information about the progress of the operation.

For example, you can run the following query to see progress messages generated during the bulk load operation to load OIM Role data:

SELECT MSG FROM OIM_BLKLD_LOG
WHERE MODULE = 'ROLE' AND LOG_LEVEL = 'PROGRESS_MSG'
ORDER BY MSG_SEQ_NO;

Errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table. The following is an example of the query to retrieve error messages:

SELECT MSG FROM OIM_BLKLD_LOG
WHERE MODULE = 'ROLE' AND LOG_LEVEL = 'ERROR'
ORDER BY MSG_SEQ_NO;

13.8.5 Handling Exceptions Recorded During the Operation

At the end of a bulk load operation, the utility records statistics related to the operation.

The statistics are recorded in the following file:

oimbulkload/logs_YYYYMMDD_HHMM/oim_blkld_ENTITY_NAME_load_summary.log

In the log file name, ENTITY_NAME stands for the entity being loaded. For example:

For roles, the log file name is oim_blkld_role_load_summary.log.
For role memberships, the log file name is oim_blkld_rolemem_load_summary.log.

The following is the sample statistics recorded in the log file at the end of a bulk load operation to store OIM Role data.

***************************************************************************************************
Processing File: Role.csv
==========================================================================================
R O L E    L O A D    S T A T I S T I C S    F O R   F I L E : Role.csv
==========================================================================================
Start Time:   17-NOV-09 02.48.18.447767 AM
End Time:     17-NOV-09 02.48.19.228710 AM
Number of Records Processed:  2
Number of Records Loaded:     2
Number of Records Rejected:   0
==========================================================================================

The name of the TMP table used during the load:
OIM_BLKLD_TMP_ROLE1

The name of the Exception table used during the load:
OIM_BLKLD_EX_ROLE1
==========================================================================================
===============================================================================
Time taken in re-building indexes and enabling FK constraints
===============================================================================

Start time:      17-NOV-09 02.48.19.243781 AM

In this sample, the number of rejected loaded is 2. If the log file shows that any records have been rejected by the utility, then see Fixing Exceptions and Reloading Data Records for information about retrying the load operation for these records.

Note:

You cannot use the utility to load data into a remote Oracle Identity Manager database.

13.8.6 Fixing Exceptions and Reloading Data Records

Errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table.

This section describes exception handling and how to reload the rejected records. It contains the following topics:

13.8.6.1 About Fixing Exceptions

Errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table. The following is an example of the query to retrieve error messages:

SELECT MSG FROM OIM_BLKLD_LOG
WHERE MODULE = 'ROLE' AND LOG_LEVEL = 'ERROR'
ORDER BY MSG_SEQ_NO;

13.8.6.2 Reloading Rejected Records

To reload rejected records:

Create a backup of the exception table in which rejected records are stored.

Note:

Although this is an optional step, it is recommended that you create a backup.
Review each record in the exception table, and fix errors in the data based on the message recorded in the EXCEPTION_MSG column.
After you fix errors in all the rejected records in an exception table, rename the table to OIM_BLKLD_TMP_SUFFIX and then use it as the input source.
Load records from the OIM_BLKLD_TMP_SUFFIX table by running the utility. See Running the Utility for more information about running the Bulk Load utility.
Repeat Steps 1 through 4 until the Number of Records Rejected label shows the value 0 in the oim_blkld_role_load_summary.log file or the corresponding log file for role membership, role hierarchy, and role category.
Restart Oracle Identity Manager if loading was done in offline mode.

Note:

Identify the unusable indexes and disabled constraints. To do so, the following SQL queries or similar mechanism can be used:

SELECT TABLE_NAME, CONSTRAINT_NAME FROM user_constraints WHERE status = 'DISABLED';
SELECT index_name FROM user_indexes WHERE status = 'UNUSABLE';

Enable the constraints and rebuild the indexes manually, as shown:

ALTER TABLE TABLE_NAME ENABLE CONSTRAINT CONSTRAINT_NAME;
ALTER INDEX INDEX_NAME REBUILD;

13.8.7 Verifying the Outcome of the Bulk Load Operation

Verify the outcome of the Bulk Load operation for one of the role, role hierarchy, role membership, and role category added by the utility by verifying that the newly created role, role hierarchy, role membership, and role category are displayed in the UI,

To verify the outcome of the bulk load operation, check if you are able to perform the following steps for one of the OIM Role added by the utility:

Log in to Oracle Identity Self Service, and verify that the newly created role is displayed in the search result for roles.
For the newly created role hierarchy and role members, click the Hierarchy and Members tabs respectively on the role details page.
To verify the newly created role category, in the Welcome page of Oracle Identity Administration, click Advanced Search - Role Categories. Then, perform an advanced search to find the newly created role.

Note:
When roles are created using the Bulk Load utility, the catalog data is not generated for the roles. To create the catalog data, run the Catalog Synchronization Job with Process Role = true. See Predefined Scheduled Tasks in Administering Oracle Identity Governance for information about the Catalog Synchronization Job scheduled job.

13.9 Loading Organization Data

Loading organization data involves creating the input source for the bulk load operation, running the Bulk Load utility, and handling exceptions, if any.

This section provides detailed information about the steps involved in loading organization data. It contains the following topics:

13.9.1 Overview of Loading Organization Data

Loading organization data involves creating the input source for the bulk load operation, running the Bulk Load utility, and handling exceptions, if any.

The following is a summary of steps involved in loading organization data:

Prepare your database for a bulk load operation, if not already done. See Preparing Your Database for a Bulk Load Operation.
Create the input source for the bulk load operation.

If you want to use a database table as the input source, then create the table and copy organization-related data into the table.

If you want to use CSV files as the input source, then create the CSV files and copy organization-related data into the files. In addition, create a master.txt file containing the names of the files in the sequence in which you want to load data from them.
Determine values for the input parameters of the utility.
Stop Oracle Identity Manager if you want to run Bulk Load utility in offline mode. For online mode, Oracle Identity Manager server can be running.
Run the oim_blkld.sh (for UNIX) or oim_blkld.bat (for Windows) script.
Monitor the progress of the bulk load operation.
Determine the outcome of the bulk load operation.
If required, reload data that is not loaded during the first run.
Restart Oracle Identity Manager , if it was stopped in step 4.
Verify the outcome of the bulk load operation.
Gather diagnostic data from the operation.
Remove temporary tables and files created during the operation.

13.9.2 Creating the Input Source for the Bulk Load Operation

Depending on the input source that you want to use, you must apply the guidelines for using CSV files or creating database tables as the input source.

This section contains the following sections:

13.9.2.1 Using CSV Files as the Input Source

If you want to use CSV files as the input source for the bulk load operation, then apply the following guidelines while creating the CSV files:

The CSV files must be placed in the oimbulkload/csv_files directory.
The first line in the CSV file is called the control line. This line must contain a comma-separated list of column names of the ACT table in the Oracle Identity Manager database.
From the second line onward, the file must contain values for the columns in the control line. The order of columns in the first line and the values in the rest of the lines must be the same. The following are sample contents of a CSV file:
```
ACT_NAME,ACT_PARENT_NAME,ACT_STATUS,ACT_CUST_TYPE
Org1,Xellerate Users,Active,System
Org2, Org3,Active,Company
Org3,Org4,Active, System
Org4,Top,Active,Company
```
The CSV file can contain hierarchal data as well, such as the example provided in the previous bullet point. Here, Org4 is created first, then Org3, and finally Org2 is created. If the last entry of Org4 is missed from this CSV, then neither Org3 nor Org2 are created because respective parents are not available in Oracle Identity Manager.
ACT_NAME and ACT_PARENT_NAME are mandatory columns .Along with these columns you can also load other columns present in ACT table.
If the value in any column contains a comma, then that value must be enclosed in double quotation marks (").
The CSV file must contain values for all columns that are designated as mandatory in the ACT table.
Each row in the CSV file must have a unique value for the ACT_NAME column in the USR table. If there are multiple files, then ensure that ACT_NAME values are unique across the CSV files. This check for uniqueness of ACT_NAME values must also cover existing organization in Oracle Identity Manager.
Note that the following default values are inserted into Oracle Identity Manager if the CSV file does not contain values for these columns:

ACT_PARENT_NAME : Top

ACT_CUST_TYPE : System

ACT_STATUS: Active
Create a master TXT file containing the names of the CSV files containing organization data to be loaded. You can specify any name for the file, for example, master.txt. Save the master file in the oimbulkload/csv_files directory.
If you want to load multiple CSV files, then enter the name of each data CSV file on a separate line in the master file. Order the list of CSV file names in the sequence in which you want the utility to load data from the files. For example, suppose you have created three data CSV files, London_Orgs.csv, NewYork_Orgs.csv, and Tokyo_Orgs.csv. In the master file, you enter the names of the data CSV files in the following order:
```
Tokyo_Orgs.csv
London_Orgs.csv
NewYork_Orgs.csv
```
When you run the utility, data is loaded in this order. This is because the organization data in London and New York may have a dependency on the Tokyo Orgs.
If the CSV file is generated on Microsoft Windows and is to be loaded on Linux environment, then remove the special characters, such as '\n\r', to avoid run-time errors.

Note:

While copying a CSV file from Windows to UNIX, Solaris, or Linux systems, some special characters, such as ^M, are appended to the file. This is because, the file from Windows is in DOS (ASCII) format and must be converted to ISO format. Solaris preinstalls the dos2unix utility into the system to do this job. But for UNIX/Linux systems, the CSV file must be converted from DOS format to UNIX format to ensure sanity of the input file before being used in the Bulk Load operation. To do this, the syntax is:

# dos2unix CSV_FILE_NAME

If the dos2unix utility does not exist in the UNIX/Linux systems, then the administrator can install the utility for the respective UNIX/Linux versions by using the relevant documentation.

13.9.2.2 Creating Database Tables as the Input Source

If you want to use a database table as the input source for loading organization data, then apply the following guidelines while creating the database table:

Create the table in the Oracle Identity Manager database.
ACT_NAME and ACT_PARENT_NAME are mandatory columns.
Along with mandatory columns, the table must contain the following primary key column:
```
OIM_BLKLD_ACTSEQ NUMBER(19)
```
The utility uses this column as the primary key. If required, you can use a database sequence to populate this column.
The rest of the columns must be the same as the ones in the ACT table that you want to use. In other words, ignore optional ACT columns that you do not want to include in the table that you create.

13.9.3 Determining Values for the Input Parameters of the Utility

You must set correct values for the input parameters of the Bulk Load utility for the type of data being loaded.

The following are input parameters of the utility:

Oracle Home

Value of the ORACLE_HOME environment variable on the host computer for the Oracle Identity Manager database
Database Connection String

Connection string to connect to the database that must be entered in the following format:

//HOST_IP_ADDRESS:PORT_NUMBER/SERVICE_NAME
OIM DB User

Database login ID of the Oracle Identity Manager database user
OIM DB Pwd

Password of the Oracle Identity Manager database user. Enter the password twice when prompted.
CSV file names

Names of the master CSV files to be used as the input source

This parameter is used only if the input source is CSV files. See Using CSV Files as the Input Source. If you are loading data from parent and child CSV file, then use a comma-delimited list to enter the names of the files. The name of the parent CSV file must be provided first, and it must be followed by the names of the child CSV files.
Tmp table name

Name of the temporary table to be used as the input source

This parameter is used only if the input source for the bulk load operation is a database table. See Creating Database Tables as the Input Source.
Control Line

Comma-separated list of names of columns to be loaded from the database table into Oracle Identity Manager

This parameter is used only if the input source for the bulk load operation is a database table.
Tablespace Name

Name of the tablespace in which temporary tables are to be created during the bulk load operation. (If end user does not provide the tablespace name, then it will pick the default tablespace.) See Preparing Your Database for a Bulk Load Operation.
Date format

Date format used by date columns in the CSV files.

This parameter is used only if the input source is CSV file.

The date format must march the following:
- Oracle supported date formats, such as dd-mm-yyyy or MM-DD-YYYY
- The date format specified in the CSV file
Batch Size

Number of user records that must be processed by the utility as a single transaction

The batch size can influence the performance of the bulk load operation. The default value of this parameter is 10000.
Debug Flag

You can specify Y or N as the value of this parameter. If this parameter is set to Y, then the utility records detailed information about events that occur during the bulk load operation. See Data Recorded During the Operation.

13.9.4 Monitoring the Progress of the Operation

During the bulk load operation, you can query the OIM_BLKLD_LOG table for information about the progress of the operation.

For example, you can run the following query to see progress messages generated during the bulk load operation to load OIM Organization data:

SELECT MSG FROM OIM_BLKLD_LOG
WHERE MODULE = 'ORG' AND LOG_LEVEL = 'PROGRESS_MSG'
ORDER BY MSG_SEQ_NO;

Errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table. The following is an example of the query to retrieve error messages:

SELECT MSG FROM OIM_BLKLD_LOG
WHERE MODULE = 'ORG' AND LOG_LEVEL = 'ERROR'
ORDER BY MSG_SEQ_NO;

13.9.5 Handling Exceptions Recorded During the Operation

At the end of a bulk load operation, the utility records statistics related to the operation.

The statistics are recorded in the following file:

oimbulkload/logs_YYYYMMDD_hhmm/oim_blkld_org_load_summary.log

To determine if there were exceptions during the operation, open this log file and look for the number against the Number of Records Rejected label. If the number of rejected records is greater than zero, then exceptions were thrown during the operation. Organization records that are rejected by the utility are recorded in the exception table (OIM_BLKLD_EX_SUFFIX). For each rejected record, the ACT_LOAD_NOTE column in the OIM_BLKLD_EX_SUFFIX table stores information about the reason the record could not be loaded. ACT_LOAD_NOTE column in respective TMP table also shows the status/error of org creation.

The following is the sample statistics recorded in the log file at the end of a bulk load operation to store OIM Organization data.

//Sample log file when source of input is DB table- 
***************************************************************************************************
Bulkload Mode : online
 
 
***************************************************************************************************
Source for Organization bulkload  : DB Table
Processing TMP table              : DB_TBL
successfully loaded  TMP Table    : DB_TBL
successfully loaded  TMP Table    : DB_TBL
TMP       Table                   : DB_TBL
Exception Table                   : DB_TBL_EX1
Log       Table                   : OIM_BLKLD_LOG
 
==========================================================================================
O R G    L O A D    S T A T I S T I C S    F O R   T A B L E : DB_TBL
==========================================================================================
Start Time                     :  19-FEB-16 12.55.35.101932 AM
End Time                       :  19-FEB-16 01.01.38.269610 AM
Number of Records Processed    :  160
Number of Records Loaded       :  160
Number of Records Rejected     :  0
 
//Sample log file when source of input is csv file- 
 
 
***************************************************************************************************
Bulkload Mode : online
 
 
***************************************************************************************************
Source for Organization bulkload  : CSV File
Processing csv File               : org.csv
successfully loaded  File         : org.csv
successfully loaded  TMP Table    : OIM_BLKLD_TMP_ORG1
TMP       Table                   : OIM_BLKLD_TMP_ORG1
Exception Table                   : OIM_BLKLD_EX_ORG1
Log       Table                   : OIM_BLKLD_LOG
 
==========================================================================================
O R G    L O A D    S T A T I S T I C S    F O R   F I L E : org.csv
==========================================================================================
Start Time                     :  19-FEB-16 12.39.27.469318 AM
End Time                       :  19-FEB-16 12.39.27.519390 AM
Number of Records Processed    :  4
Number of Records Loaded       :  0
Number of Records Rejected     :  4
==========================================================================================

In this sample, the number of rejected records is 4. If the log file shows that any records have been rejected by the utility, then see Fixing Exceptions and Reloading Data Recordsfor information about retrying the load operation for these records.

13.9.6 Fixing Exceptions and Reloading Data Records

Errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table.

This section describes exception handling for the bulk load operation and how to reload rejected records. It contains the following topics:

13.9.6.1 About Fixing Exceptions

As mentioned earlier, errors encountered during the bulk load operation can be viewed by querying the OIM_BLKLD_LOG table. The following is an example of the query to retrieve error messages:

SELECT MSG FROM OIM_BLKLD_LOG
WHERE MODULE = 'ORG' AND LOG_LEVEL = 'ERROR'
ORDER BY MSG_SEQ_NO;

13.9.6.2 Reloading Rejected Records

To reload rejected records:

Create a backup of the exception table in which rejected records are stored.

Note:
Although this is an optional step, it is recommended that you create a backup.
Review each record in the exception table, and fix errors in the data based on the message recorded in the EXCEPTION_MSG column.
After you fix errors in all the rejected records in an exception table, rename the table to OIM_BLKLD_TMP_SUFFIX, and run the following update statement:
```
UPDATE OIM_BLKLD_TMP_SUFFIX
                SET   ACT_LOAD_STATUS='P',ACT_ACT_KEY=NULL,
                       ACT_LOAD_NOTE=NULL;
                COMMIT;
```
Now use the table as the input source.
Load records from the OIM_BLKLD_TMP_SUFFIX table by running the utility. SeeRunning the Utilityfor more information about running the Bulk Load utility.
Repeat Steps 1 through 4 until the Number of Records Rejected label shows the value 0 in the oim_blkld_role_load_summary.log file.
Restart Oracle Identity Manager if loading has been in offline mode.
Note:
Being a database-intensive operation by design, Bulk Load disables the constraints and indexes on the relevant Oracle Identity Manager entity tables during the start of the operation. Bulk Load operation failure towards the end of the load might at times render the indexes and constraints in disabled state. To identify and fix this issue, manually restore the indexes and constraints as follows:
1. Identify the unusable indexes and disabled constraints. To do so, the following SQL queries or similar mechanism can be used:
  SELECT TABLE_NAME, CONSTRAINT_NAME FROM user_constraints WHERE status = 'DISABLED'; SELECT index_name FROM user_indexes WHERE status = 'UNUSABLE';
2. Enable the constraints and rebuild the indexes manually, as shown:
  ALTER TABLE TABLE_NAME ENABLE CONSTRAINT CONSTRAINT_NAME; ALTER INDEX INDEX_NAME REBUILD;

13.9.7 Verifying the Outcome of the Bulk Load Operation

Verify the outcome of the Bulk Load operation for one of the organizations added by the utility by verifying that the newly created organization is added in the UI.

To verify the outcome of the bulk load operation, check if you are able to perform the following steps for one of the OIM Organizations added by the utility:

Log in to Oracle Identity Self Service, and verify that the newly created organization is displayed in the search result for Organizations.
Create a user under newly created organization. To do so:
1. Click the Members tab of the newly created organization. This user should be displayed as a member.
2. Click the Organization tab of the user. The newly created organization should be displayed in the results.

13.10 Data Recorded During the Operation

During the bulk load operation, the utility inserts progress and error messages in the OIM_BLKLD_LOG table.

Data in the OIM_BLKLD_LOG table is not deleted at the start of a new bulk load operation. One of the columns in this table holds the time stamp at which messages are recorded in the table.

Table 13-5 describes the structure of the OIM_BLKLD_LOG table.

Table 13-5 Structure of the OIM_BLKLD_LOG Table

Column	NULL	Type	Description
MSG_SEQ_NO	NULL	NUMBER(19)	This column stores the number that denotes the order in which messages are inserted in this table. The column is populated by using the OIM_BLKLD_LOG_SEQ sequence. You can use this column to query for messages in the order in which they are recorded in the table.
MODULE	NOT NULL	VARCHAR2(20)	This column stores one of the following values: ROLE: This value indicates that the message has been recorded while loading OIM Role data. ROLE HIERARCHY: This value indicates that the message has been recorded while loading role hierarchy data. ROLE MEMBERSHIP: This value indicates that the message has been recorded while loading OIM role membership data. ROLE CATEGORY: This value indicates that the message has been recorded while loading OIM role category data.
LOG_LEVEL	NOT NULL	VARCHAR2(20)	This column stores one of the following values: ERROR: Designates fine-grained informational events that are useful to debug. DEBUG: Designates error events that might allow the application to continue running. Error is used to log all unhandled exceptions. PROGRESS_MSG: Designates intermediate progress messages.
LOAD_SOURCE	NOT NULL	VARCHAR2(40)	This column indicates the source of data for the bulk load operation during which the row was inserted. The value can be one of the following:CSV File: FILE_NAME DB Table
MSG	NOT NULL	VARCHAR2(4000)	This column stores a message corresponding to the value stored in the LOG_LEVEL column.
CREATE_DATE		DATE	This column holds the time stamp at which the record was created. The format for entries in this column is as follows: yyyy/mm/dd hh24:mi:ss For example: 2008/06/23 21:49:16:32

13.11 Gathering Diagnostic Data from the Bulk Load Operation

Data recorded in the files can be used to collate performance-related information about the bulk load operation.

The following log files are created during the bulk load operation:

For OIM Users:

oimbulkload/logs_YYYYMMDD_HHMM/oim_blkld_user_load_summary.log
For accounts:

oimbulkload/logs_YYYYMMDD_HHMM/oim_blkld_account_load_summary.log
For roles, role hierarchies, memberships, and role categories:

oimbulkload/logs_YYYYMMDD_HHMM/oim_blkld_ENTITY_NAME_load_summary.log

In the log file name, ENTITY_NAME stands for the entity being loaded. For example:
- For roles, the log file name is oim_blkld_role_load_summary.log.
- For role memberships, the log file name is oim_blkld_rolemem_load_summary.log.

Data recorded in this file can be used to collate performance-related information about the bulk load operation. The following information can be collected after the bulk load operation:

Start time
Input source
Number of records in the system before the load
Number of records successfully loaded
Number of records rejected
Total time taken

You can use this information during future runs of the utility.

13.12 Cleaning Up After a Bulk Load Operation

Log tables, tablespaces, and files for bulk load can be removed after the operation.

If you do not want to save the results of a bulk load operation, then:

Remove the OIM_BLKLD_TMP_SUFFIX, OIM_BLKLD_EX_SUFFIX, and OIM_BLKLD_LOG tables.
Remove any files that you created or used during the operation.
If you created a tablespace for the operation, then remove the tablespace.
See Gathering Diagnostic Data from the Bulk Load Operation before you remove log files created in the logs_timestamp directory.

Note:

At this point, you can restart Oracle Identity Manager if you have not already done so.

13.13 Bulk Load High Volume Strategy and Case Studies

Follow best practices, and understand case studies and strategies for high-volume data load.

For information about general best practices and few case studies about high-volume data load, see the technote titled OIM 11G BulkLoad Utility Strategies & Case Studies (Doc ID 1959363.1) in the My Oracle Support web site at:

https://support.oracle.com