Siebel Data Quality Administration Guide > Configuring Data Quality >

Incremental Data Load

When loading Siebel application data into Informatica Identity Resolution tables, rather than generating keys for the entire data set in one go, the incremental data load feature provides the functionality to divide the total load into smaller batches of configurable size and load them incrementally. Use the following procedure to configure a new field for data matching.

To incrementally load Siebel application data into Informatica Identity Resolution tables

1. Start the Informatica Identity Resolution server in Configure Mode.
2. Start the Informatica Identity Resolution Console client in Admin Mode.
3. Select System (if already created), otherwise create a new system using the required sdf file.
4. Run the IDS_IDT_<ENTITY TO BE LOADED>_STG.sql script to take a snapshot of records in the Siebel application.

   For example, execute the following script from the SQL command line prompt for account initial load using your appropriate login credentials:

   IDS_IDT_ACCOUNT_STG.sql

   The following sample SQL scripts can be used to capture snapshots of the data:

   • IDS_IDT_ACCOUNT_STG.SQL
   • IDS_IDT_CONTACT_STG.SQL
   • IDS_IDT_PROSPECT_STG.SQL

     For more information about these example scripts, see Sample SQL Scripts.

5. While creating a snapshot using the example scripts listed in the previous step, users are prompted to enter a batch size. Depending on the value entered, the entire snapshot is grouped into batches of the specified batch size.

   For example, run the following SQL script to create the database table to store the current batch number being loaded (this value is usually 1 for the first time):

   IDS_IDT_CURRENT_BATCH.sql

6. Run the IDS_IDT_CURRENT_BATCH_<ENTITY TO BE LOADED>.sql script to create the dynamic view to load the snapshot for the staging table created in the previous step.

   For example, execute the following script from the command line:

   IDS_IDT_CURRENT_BATCH_ACCOUNT.sql

   The following sample SQL scripts can be used to create the views to process the records in a given batch:

   • IDS_IDT_CURRENT_BATCH_ACCOUNT.SQL
   • IDS_IDT_CURRENT_BATCH_CONTACT.SQL
   • IDS_IDT_CURRENT_BATCH_PROSPECT.SQL

     For more information about these example scripts, see Sample SQL Scripts.

7. Open the Informatica Identity Resolution client and perform a Load IDT.

   Load the remaining batches of data through the ISS batch Utility. Open a command window and navigate to the directory where the initial scripts for loading have been copied. Execute the initial load process by entering the following command at the command line:

   IDS_IDT_LOAD_ANY_ENTITY.CMD <Entity> <Work Directory>

   For example, execute the following script from the command line for account initial load:

   IDS_IDT_LOAD_ANY_ENTITY.CMD Account c:/initialLoad/logs

8. The following files contain the parameters used by the batch load utility; you must update these files to reflect your installation:
   • idt_Account_load.txt
   • idt_Contact_load.txt
   • idt_Prospect_load.txt

     NOTE:  Certain SQL and shell scripts are required to create materialized views and to load data incrementally. Some sample SQL and shell scripts are provided in Sample Configuration and Script Files. Depending on customer requirements, you can fine tune these sample files during implementation.

9. Incrementally load the failed batches individually using the following script from the command line:

   IDS_IDT_LOADBATCH_ANY_ENTITY <Entity> <Work Directory> <Batch Number>

   For example, execute the following script from the command line to load batch 33 of account:

   IDS_IDT_LOADBATCH_ANY_ENTITY.CMD Account c:/initialLoad/logs 33

   Examine the log files and error files to ensure that all batches have successfully loaded. In the case of errors, use the information in the log and error files to determine the root cause for the failure and fix the underlying issue; repeat the load process as necessary.