Loading the Initial Data Set for a Sun Master Index

Loading Matched Data Using the Data Integrator Wizard Bulk Loader


Note –

This process is new for Java CAPS 6 Update 1 and is unavailable in Release 6.


You can use the Data Integrator Wizard to generate the Bulk Loader for a master index application. The wizard generates the collaborations that define the load process. Once you build the Data Integrator project, you can generate a command-line tool that runs the collaborations in the correct order.

ProcedureTo Load Matched Data Using the Data Integrator Wizard Bulk Loader

Before You Begin
  1. On the NetBeans Projects window, expand the new Data Integrator project and right-click Collaborations.

  2. Point to New, and then select ETL.

    The Data Integrator Wizard appears with the Name and Location window displayed.

  3. Enter name for the collaboration.

    Figure shows the Name and Location window of the wizard.
  4. Click Next.

  5. On the Select Type of ETL Loader window, select Bulk Loader.

    Figure shows the Select Type of ETL Loader window of
the wizard.
  6. Click Next.

    The Select or Create Database window appears.

  7. To specify a staging database to use for external data sources (for this project only), do one of the following:

    1. Select an existing database to use from the DB URL field.

    2. Select Create and Use New Database, enter a name for a new database in the DB Name field, and then click Create Database. Select the new database in the DB URL field.

      Figure shows the Select or Create Database window of
the wizard.
      Note –

      This database is required and is used for internal processing only.


  8. Click Next.

    The Select JDBC Target Tables window appears.

  9. To choose the target tables to load the extracted data into, do the following:

    1. Under Available Connections, select the master index database.

    2. Under Schemas, select the schema that contains the tables to load the data into.

    3. Under Schema, select only the tables that correspond to the data files produced by the Bulk Matcher, and then click Select.


      Tip –

      You can use the Shift and Control keys to select multiple tables at once. If you select target tables that do not correspond to the Bulk Matcher files, collaborations without source table are generated and the project fails to build.


      Figure shows the Select Target Tables of the wizard.
    4. Click Next.

      The Choose Bulk Loader Data Source window appears.

  10. To specify the source data for the Bulk Loader, do the following:

    1. In the upper portion of the window, browse to the location of the of the output files from the Bulk Matcher.


      Note –

      These files are located in NetBeansProjects_Home/Project_Name/loader-generated/loader/work/masterindex, where work is the location you specified for the working directory in loader-config.xml.


    2. Select all of the data files in the masterindex directory, and then click Add.

      Figure shows the Choose Bulk Loader Data Source window
of the wizard.
    3. Click Next.

      The Map Selected Collaboration Tables window appears.

  11. To map source and target data, do the following:

    1. To disable constraints on the target tables, select Disable Target Table Constraints.

    2. Select the SQL statement type to use for the transfer. You can select insert, update, or both.

    3. The wizard automatically maps the source and target tables for you. Review the mapping to verify its accuracy.


      Note –

      Not every table on the left will be mapped. For example, system tables such as SBYN_COMMON_HEADER, SBYN_COMMON_DETAIL, SBYN_APPL, and SBYN_SYSTEMS do not need to be mapped.


      The Map Selected Collaboration Tables window of the wizard.
    4. Click Finish.

      An ETL collaboration is created for each target table. This might take a few minutes to generate.

  12. Verify that all the required collaborations were created, and then right-click the Data Integrator project and select Clean and Build.

  13. Right-click the Data Integrator project again, and select Generate Command Line ETL.

  14. To configure the load process, do the following:

    1. On the Files window, expand the Data Integrator project and then expand ETLLoader.

    2. Open startLoad.bat or startLoad.sh for editing.

    3. In the JAVA_HOME variable, enter the path to your JRE installation; for example, C:\Java\jre1.5.0_12.

    4. In the DATABASE_DRIVERS variable, enter the paths to the master index database driver and the Axion database driver.


      Tip –

      The master index driver should be located in app_server/lib and the Axion database driver can be found in ETLLoader/lib.


    5. Save and close the file.

  15. On your computer, navigate to your NetBeans projects home directory and then to bulk_loader_project/ETLLoader.

  16. To perform the load, do one of the following:

    • For Windows, double-click startLoad.bat or type startLoad.bat from a command line.

    • For UNIX, type sh startLoad.sh at the command prompt.

  17. If your master index application is deployed, you can use the Master Index Data Manager to verify that the data was loaded successfully. Also review the log files under ETLLoader/logs.