Loading the Matched Data Into the Master Index Database

The IBML Tool provides two methods to load the master data images generated by the Bulk Matcher. A command line tool is provided to generate and then run the extract, transform, and load (ETL) collaborations that load the data. Alternatively, you can use SQL*Loader to load the data if the master index database is running on Oracle.

Perform one of the following procedures to load the matched data into your master index database:

• Loading Matched Data Using SQL*Loader

• Loading Matched Data Using the Command-Line Bulk Loader

Loading Matched Data Using SQL*Loader

If the master index database runs on an Oracle platform, you can use either SQL*Loader or the command-line Bulk Loader to load the matched data into the database. SQL*Loader cannot be used for a SQL Server or MySQL database.

To Load Matched Data Using SQL*Loader

1. Complete the steps under Performing the Bulk Match.
2. From the master IBML Tool home directory, run cluster-truncate.sql against the cluster synchronizer database.
3. For each IBML Tool, open loader-config.xml (located in the IBML Tool home directory in the conf subdirectory).
   1. Define the SQL*Loader property as described in SQL*Loader Configuration.
   2. Change the value of the BulkLoad property to true.
   3. Save and close the file.
4. To generate the loader, do one of the following.
   • If the master loader is running on Windows:
     1. Navigate to the master IBML Tool home directory and open generate-sql-loader.bat for editing.
     2. Change the value of the JDBC_JAR_PATH variable in the first line to the location and name of the database driver for the master index database platform; for example, set JDBC_JAR_PATH=C:\oracle\jdbc\lib\ojdbc14.jar.
     3. Close and save the file.
     4. Double-click generate-sql-loader.bat or type generate-sql-loader from a command line.
   • If the master loader is running on UNIX:
     1. Navigate to the master IBML Tool home directory and open sh generate-sql-loader.sh for editing.
     2. Change the value of the JDBC_JAR_PATH variable in the first line to the location and name of the database driver for the master index database platform; for example, export JDBC_JAR_PATH=${oracle_home}/jdbc/lib/ojdbc14.jar.
     3. Close and save the file.
     4. Type sh generate-sql-loader.sh at the command line.

     A new directory named sqlldr is created in the working directory.

5. In the master IBML Tool home directory, run cluster-truncate.sql against the master index database to clear the cluster synchronizer tables.
6. In the sqlldr folder in the working directory, run drop.sql against the master index database to drop constraints and indexes.
7. In the sqlldr directory, do one of the following:
   • On Windows, double-click bulk-loader.bat or type bulk-loader.bat from a command line.
   • On UNIX, type sh bulk-loader.sh at the command line.
8. After the data is loaded, close any command prompts that were left open by the process and examine the SQL*Loader log files located in the sqlldr/log directory to ensure there were no errors during processing.

   ---

   Note - Any records that contained bad data and were not inserted into the master index database are written to the sqlldr/bad directory. Any records that contained bad data and were discarded are written to the sqlldr/discard directory.

   ---

9. In the sqlldr directory, run create.sql against the master index database to reinstate the dropped indexes and constraints.

Loading Matched Data Using the Command-Line Bulk Loader

You can use the command-line Bulk Loader to load data into an Oracle, MySQL, or SQL Server database. Using the command-line tool does not require the use of NetBeans, but it does require that NetBeans be installed on the master loader machine.

To Load Matched Data Using the Command-Line Bulk Loader

1. Complete the steps under Performing the Bulk Match.
2. In the master IBML Tool home directory, open genCollab.bat (or genCollab.sh for UNIX) and configure the properties described in Command-Line Bulk Loader Properties.
3. Save and close the file.
4. In the master IBML Tool home directory, do one of the following:
   • On Windows, double-click genCollab.bat or type genCollab.bat from a command line.
   • On UNIX, type sh genCollab.sh at the command line.

     This generates a zip file in the IBML Tool home directory.

5. Extract the contents of etl-loader.zip to the current directory.

   This generates an ETL collaboration and creates a new directory, ETLloader, in the IBML Tool home directory.

6. In the master IBML Tool home directory, run cluster-truncate.sql against the master index database to clear the cluster synchronizer tables.
7. In the ETLloader/config directory, open logger.properties and modify any logging properties if needed.
8. In the ETLloader directory, do one of the following:
   • On Windows, double-click startLoad.bat or type startLoad.bat from a command line.
   • On UNIX, type sh startLoad.sh at the command line.
9. After the data is loaded, check the log files to ensure there were no errors during processing.

Command–Line Bulk Loader Properties

The ETL collaboration is generated by a file that includes configurable properties you need to define. The file is named genCollab.bat for Windows and genCollab.sh for UNIX. It is located in the directory where you extracted the IBML Tool files on the master processor. The following table lists and describes the default properties for the file.

Table 8 Command–Line Bulk Loader Properties