This chapter describes how to import data from legacy, third party, or other external data sources.
This chapter covers the following topics:
Bulk import is the process of loading customer data in bulk from legacy, third party, or other external sources into the TCA Registry.
In Oracle Customer Data Librarian (CDL), you can use:
File Load: Load files of high volume organization, person, or contact data into the TCA import interface tables. See: File Loads Overview, Oracle Customers Online User Guide.
Batch Import: Transfer batches of data from the interface tables into the TCA Registry. See: Import Batches Overview.
With Batch Import, you can review the batch and control the import process. Alternatively, if the appropriate import profile options are set, the file load data can automatically transfer from the interface tables into the Registry, as soon as the file load completes successfully. See: Setting Up Import, Oracle Customers Online Implementation Guide.
The import is only for information on the party level. To import both party and account information, use Customer Interface. See: Customer Interface, Oracle Trading Community Architecture User Guide.
Related Topics
Introduction to Oracle Customer Data Librarian
After data from a source file is successfully loaded into the import interface tables, that set of data is automatically included in a new import batch. The import batch name is the same as the file load name, and the batch ID is automatically generated if the batch comes from File Load.
Aside from import batches that contain data loaded using the File Load feature, import batches with data from other methods, such as D&B Load and SQL/ETL Load, are also displayed. See: Loading Data into the Interface Tables, Oracle Trading Community Architecture Administration Guide.
Note: Data must be loaded into the interface tables in the context of a batch. If using File Load, a batch is automatically generated. For any other loading tool, create a new batch with a new unique batch ID, which is required for loading into the interface tables. See: Generating Batches.
An import batch contains information about transferring a set of records from the interface tables into the TCA Registry, including settings for optional data quality preimport processes that you can perform on the batch.
Batch de-duplication: Identify and resolve duplicates within the batch that you are importing from the interface tables.
Address validation: Validate addresses within the batch against authorized sources.
Registry de-duplication: Identify and resolve duplicates between the interface tables and the Registry.
If automatic import after file load is implemented, then import batches are created with the import processes already running. If not, you must manually define the import and submit the batch for the import process. See: Setting Up Import, Oracle Customers Online Implementation Guide.
All import batches are considered active, with any import or batch status, until they are purged. See: Purging Import Batches.
You can search for active import batches, and purged batches in the batch import history, by:
Address Validation Status: The status of the address validation process.
Batch De-Duplication Status: The status of the batch de-duplication process.
Note: No status for address validation or batch de-duplication means that the import batch is either rejected or not yet defined, while the Declined status indicates that the preimport process is declined as part of the batch definition.
Batch ID: The import batch ID.
Batch Name: The user-defined import batch name, which is the same as the name of the file load that provided the same set of data into the interface tables.
Batch Status: The status of the import batch itself, including:
Blank: The import batch was generated in this application and does not yet have data loaded for it in the interface tables.
Active: The import batch is new and not yet defined for import.
Action Required: Preimport processes have either the Declined or Completed status, and the import batch was defined with preview before the actual import. You can now update the batch and submit it for the import.
Or, the import status is Completed with Errors or Completed with Error Limit Reached. You must resolve errors and import corrected records.
Pre-processing: Reimport processes are running.
Post-processing: Post-import processes are running. See: Postimport Processes, Oracle Trading Community Architecture User Guide.
Rejected: The import batch was rejected and cannot be defined for import.
Stage: The stage of the import process, including the percentage complete within that stage.
The Pending, Processing, Error, and Completed statuses are the same as for import status.
Created By (for active batches search only): The user who created the file load that corresponds to the import batch.
Import Status: The status of transferring the import batch from the interface tables into the TCA Registry, including:
Blank: Either the import batch is not yet defined, or it is was defined with preview before the actual import.
Action Required: Preimport processes have either the Declined or Completed status, and the import batch was defined with preview before the actual import. You can now update the batch and submit it for the import.
Completed: The data in the batch was successfully imported into the TCA Registry.
Completed with Errors: The import process has completed. There are errors, but successful records might already be imported.
Completed with Error Limit Reached: The import process has stopped after reaching a specified number of errors, as set by your administrator. Successful records might already be imported.
The Rejected status is the same as for batch status. The Pending, Processing, and Error statuses reflect the concurrent program status.
Purged By (for purged batches search only): The user who purged the import batch.
Source System: The legacy, third party, or other external system that the source file data comes from.
In the Batch Import History page, you view the purged batches in the search results. See: Viewing Batch Import History.
In the Import Batches page, after you view the search results, you can:
Generate Batch: Create batches before loading data into the interface tables. See: Generating Batches.
Reject: Reject import batches so that they are not eligible for import into the TCA Registry. The data remains in the interface tables until you purge the import batch. You can reject any batch except those that are currently being imported, already completed the import process, or are already rejected.
Activate Batch: Updates batch status to Active for batches generated in this application. See: Generating Batches.
Define Import: Define imports to submit import batches for preimport and import processes, or only preimport processes. You can define import for any batch except those that are being or were already imported, or are rejected. See: Defining Imports.
Batch Name: View import batch details for any batch except those with preimport or import process currently running. You can also resolve any import errors and resubmit the batch. See: Viewing or Updating Import Batches and Resolving Import Errors.
Update: If only preimport processes were run for the batch, then you can preview results and resubmit for the actual import process. See: Viewing or Updating Import Batches.
Purge: Purge import batches, except those with preimport or import process currently running. See: Purging Import Batches.
Related Topics
Create import batches before loading data into the import interface tables. After a batch is generated, you can use File Load or other loading tools to populate the interface tables. See: Import Overview and Import Batches Overview.
A generated batch initially has no batch status. After data is loaded into the interface tables for this batch, you can activate the batch on the Import Batches page, and then proceed to define the import. See: Managing Import Batches.
This table describes some terms in the pages used for this procedure.
| Term | Description |
|---|---|
| Original System | The source system that the data to be imported in this batch comes from. |
| Estimated Total Records in Batch | The total number of records you estimate to import using this batch. |
To submit import batches for the import process, you define the import settings for the batch:
Batch De-Duplication: Optionally choose to run batch de-duplication and, if so, specify how to identify and resolve duplicates within the import batch. See: Batch De-Duplication.
Address Validation: Optionally choose to run address validation on the import batch, and if so, specify the authorized sources to use for validation. See: Address Validation.
Import to Registry: Optionally choose to preview results before import, or to run Registry de-duplication. You also specify error handling settings. See: Import to Registry.
When you submit a batch for import, you run the Import Batch to TCA Registry program for that batch. The import process is the same, except that you do not need to run the Import Batch De-Duplication report. You can access batch de-duplication preview or results through viewing or updating the batch. See: Import Process, Oracle Trading Community Architecture User Guide.
After the import process successfully completes, postimport processes automatically run. See: Postimport Processes, Oracle Trading Community Architecture User Guide.
You should be familiar with:
How records from the interface tables are determined for either insertion as a new record or for updating an existing record in the TCA Registry. See: Matching Records with Source System Management, Oracle Trading Community Architecture User Guide.
What types of validation the records from the interface tables must pass for successful import. See: Import Validations, Oracle Trading Community Architecture User Guide.
Related Topics
Batch de-duplication identifies and resolves duplicates within the batch that you are importing from the interface tables. You cannot check for duplicates between batches in the interface tables.
If you select to run batch de-duplication, you must:
Enter the match rule to use. Only match rules with Bulk Duplicate Identification purpose are used for batch de-duplication.
Important: Do not select a match rule with user-defined custom attributes.
Specify how to resolve duplicate parties found with batch de-duplication. The specified action would be taken on the batch before the data is imported into the TCA Registry.
Specify how to resolve duplicate addresses, contact points, and contacts of party records that are not duplicates of other parties in the import batch.
For example, Organization 1 and Organization 2 are found to be duplicates. You selected to keep the record with the earliest creation date, which is Organization 1. Organization 1 is now a unique record, but it has five duplicate addresses. Those addresses are resolved based on the selected action to take on duplicate addresses.
The possible actions to take on duplicate parties and child entities include keeping one record based on creation or update date. Entering the most recent update date in the interface tables, however, is not mandatory. If none of the records in a set of duplicates has a date, then the one record to be imported is randomly selected. If only some of the records have a date, then the record to be imported is selected only from records that have a date, based on the specified action.
The action you select for duplicate parties affects address validation. If you select to update the batch and preview preimport process results, then address validation checks all addresses in the import batch, regardless of the selected action for duplicates. When you preview the results, you can still change the action to take.
If you choose to import directly, without a preview, then address validation checks only addresses that are imported, not those that are removed based on the selected action. See: Address Validation and Import to Registry.
Related Topics
If you choose to validate the addresses in the import batch before importing them into the TCA Registry, then you must select the address validation adapters to use. The address validation process is the same as that for the Import Batch to TCA Registry program, except that you can specify different adapters to use for different countries. The default adapter defined for each country is initially displayed. See: Import Address Validation, Oracle Trading Community Architecture User Guide.
If there are addresses with countries that are either not in the FND Territories lookup or are incorrectly formatted, then a row for Others appears at the bottom of the table. The default adapter for those addresses is from the HZ: Default Location Service Adapter profile option.
The addresses available for validation depend on the selected action to take on duplicate parties within the batch. See: Details on Actions to Take on Duplicates.
Related Topics
You can preview import statistics and preimport processes results before the actual import into the TCA Registry. After the preimport processes finish, you update the batch to preview results and submit the batch for import. See: Viewing or Updating Import Batches.
If you selected to run batch de-duplication and address validation, whether you select to update and preview the batch or not can affect which addresses get validated. See: Details on Actions to Take on Duplicates.
Registry de-duplication identifies duplicates between the import batch and the TCA Registry. If you select to run Registry de-duplication, you must enter the match rule to use. Only match rules with Bulk Duplicate Identification purpose are used for Registry de-duplication. You can select the same match rule as for batch de-duplication.
Important: Do not select a match rule with user-defined custom attributes.
If the specified match rule is designated for Automerge, and you select to run Automerge, then potential duplicates that reach or exceed the match rule's:
Match threshold, but not the automatic merge threshold, are by default included in System Duplicate Identification (SDI) batches for merge consideration. See: System Duplicate Identification.
Automatic merge threshold are automatically merged after the import. The master record is the TCA Registry record with the highest match percentage to the incoming record.
If the match rule is not enabled for Automerge or you select not to run Automerge, then all potential duplicates that reach or exceed the match threshold are by default included in SDI batches.
All records that do not reach the match threshold are considered not duplicates and inserted as new parties. All potential duplicates are also inserted into the TCA Registry as new parties, but then accordingly included in SDI batches or automatically merged.
Tip: If you select to update and preview the import batch before the actual import, you can change the default actions to take. For example, you can select to automatically merge potential duplicates that, by default, are designated for manual merge in SDI batches. See: Viewing or Updating Import Batches.
See: Automerge, Oracle Trading Community Architecture Administration Guide.
Specify whether or not to generate fuzzy keys as part of the postimport processes.
Important: Skipping fuzzy key generation helps optimize performance, but select No only if you do not use Oracle E-Business Suite applications that use fuzzy keys.
See: Postimport Processes, Oracle Trading Community Architecture User Guide.
Optionally specify a limit on the number or percentage of errors allowed before canceling the import process. For example, if you enter 50 as the error limit number, then the process stops as soon as error 51 occurs. If you enter 50 as the error limit percentage, then the process stops as soon as 51% of records in the import batch result in error.
If you select to validate flexfields, then flexfield validation applies for inserting and updating records for these tables in the TCA Registry:
HZ_CONTACT_POINTS
HZ_LOCATIONS
HZ_PARTIES
HZ_RELATIONSHIPS
Related Topics
The same type of import batch information is available whether you view or update import batches. You can update only batches that has only run the preimport processes and has an import status of Action Required.
Update: Preview preimport processes results, after they complete. You can also:
Modify settings for preimport processes, for example the actions to take on duplicates.
Submit the batch for actual import, either immediately after your modifications, or at a later time. If you had changed preimport settings, those processes would run, and the actual import would automatically start afterwards. You cannot run only the preimport processes again for this batch.
Important: If you modify preimport settings without clicking Apply, and submit to complete the import, the import still runs with your modified settings.
View: Review import results after the import process completes.
If you are viewing batches with the Completed with Errors import status, you can also view and resolve errors, if any. See: Resolving Import Errors.
You can view batches that are rejected or not yet defined, but those batches have no preimport and import statistics.
The import batch details provide:
General information about the batch itself.
Aside from general statistics, you can access details on the record level for batch and Registry de-duplication results, as well as for import errors. These record level details include:
Source System ID: The source ID of the record.
Party Source System ID: The source ID for the party record that the address, contact, or contact point record belongs to. See: Source Systems Overview, Oracle Trading Community Architecture Administration Guide.
Related Topics
The preimport statistics include the number of records for each supported entity. The total number of records for import is the total number in the batch minus the records that were removed and not to be imported, as a result of actions taken on duplicates from the batch de-duplication process. See: Batch De-Duplication.
The postimport statistics include the number of records that are:
Insert: Imported as new records.
Update: Imported to update existing records.
Merge Request: Included in a System Duplicate Identification (SDI) batch for merge consideration.
Automatic Merge: Automatically merged. The merge occurs after all potential duplicates are inserted, so the total number of records imported includes these records.
Note: The statistics for Insert and Update do not include parties that are also included in a SDI batch or automatically merged.
You can compare, for the entire import batch before and after import, the number of records removed or imported.
Related Topics
Viewing and Updating Import Batch Details
The import details include statistics and settings for the preimport processes, either before or after the actual import process has run. If you are updating the import batch and have run only the preimport processes, you can submit the batch for the actual import process, and rerun the preimport processes if you modified any of the settings.
View the number of duplicate records and duplicate sets that batch de-duplication has identified. If a selected action is to keep one record, based on creation or update date, then only one record from each duplicate set is imported.
You can also view the output from the Import Batch De-Duplication report, which displays all the records that are identified as duplicates, sorted by duplicate sets for parties, addresses, contacts, and contact points.
Each record has a status of either I for insert, or D for duplicate. If the action is Remove All Duplicates, all the records in a duplicate set would have the D status. Likewise, for the Keep All Duplicates action, all records have the I status. For any other action, all records in a duplicate set have the D status except the one record to be imported, with the I status.
If you are updating the import batch and previewing import details after running only preimport processes, you can still modify the actions to take on duplicates. If you change the action from Remove All Duplicates to any other action, then when you submit the batch and rerun batch de-duplication, those records originally marked for removal are:
Sent through Registry de-duplication, if that process is selected.
Checked for other validations and marked for either insert or update.
See:
Matching Records with Source System Management, Oracle Trading Community Architecture User Guide
Import Validations, Oracle Trading Community Architecture User Guide
The address validation statistics include the number of address records that:
Passed or failed validation.
Were sent to be validated by the specified adapter. Not all records in the batch are always validated. See: Details on Actions to Take on Duplicates.
See: Address Validation.
View the number of records from the import batch that Registry de-duplication has categorized, as well as the default action to take on each record type. This table describes the record types and the default actions to take on each type.
| Record Type | Record Type Description | Entities | Default Action | Action Description |
|---|---|---|---|---|
| New and Unique | Does not match any record in the TCA Registry. | All | Insert | Insert the interface table record as a new record in the TCA Registry |
| Existing | Exactly matches a TCA Registry record. | All | Update | Update the existing TCA Registry record with the interface table record |
| Possible Duplicates | Is a potential duplicate of at least one Registry record. The record's match score exceeds the match threshold of the match rule used for the de-duplication, but not the automatic merge threshold. | Parties, party sites, contacts, and contact points | Manual Merge | Insert the record and then include it and its duplicates in a System Duplicate Identification batch for manual merge consideration |
| Duplicates | Is a definite duplicate of at least one Registry record. The record's match score exceeds the automatic merge threshold of the match rule.
Note: This record type is only available if the match rule is enabled for Automerge. |
Parties |
|
Auto Merge: Insert the record and then automatically merge it with its duplicates |
Note: An interface table record with the Existing type has the same source system and source ID as the corresponding TCA Registry record. The two records also match, as determined by the selected match rule. A record that is a duplicate or potential duplicate also matches TCA Registry records, but does not have the same source system and source ID.
If you are updating the import batch and previewing import details after running only preimport processes, you can still modify the actions to take. Not all actions are available for every record type. Aside from the actions described in the table above, the Remove action is also available for all record types, for removing the record from the batch and not importing it at all.
See: Registry De-Duplication.
Related Topics
View the errors of the import batch from the batch details. See: Viewing and Updating Import Batch Details. Only records with errors are displayed.
Correct the data directly in the interface tables. Records with errors are marked with E in the INTERFACE_STATUS column of the interface table and associated with detail records in the import errors table, HZ_IMP_ERRORS.
Change the INTERFACE_STATUS column value to C for each corrected record.
Return to the import batch details to import the corrected records. See: Viewing and Updating Import Batch Details.
Related Topics
Purge import batches to permanently remove all records in the batch from the import interface tables, as well as internal tables such as staging and error tables. You can purge any batch, imported or not, except those that have preimport processes or the import process running.
You usually purge batches:
After a satisfactory number of records are successfully imported into the TCA Registry from that batch.
If the batch is rejected and you do not plan to import that set of data using another import batch.
Purging the interface tables improves import performance. To archive imported data, you should copy the data to a set of custom tables.
Purged batches are no longer active, and you can view them only on the Batch Import History page. To search for purged batches, see: Searching for Import Batches.
You can also view details of purged batches that were not rejected. See: Viewing and Updating Import Batch Details.
Related Topics