Creating Data Loader Tasks

A data loader task lets you take data from a source, and load the data into a target.

In Data Integration, you can use a data loader task to perform 1-to-1 or n-to-n loading of data from one system type into another, with the option of multiple to no data transformations before loading. Data loader tasks are essential for data preparation, data migration, or loading diverse data into data lakes or data warehouses.

When you create a data loader task, Data Integration guides you through the entire process of selecting the source and target entity or entities, applying transformations, and validating the task. For the target, you have the option to create the entity or entities before loading, or you can select from existing entities to load into.

You create a data loader task in a project or folder. Data Integration includes one default project to get you started, but if you want to create your own, see Using Projects and Folders.

Parts of a Data Loader Task

Configuring a task to load data from a source to a target involves several steps.

  • Basic information and Load type: Choose the type of the source data entity and target data entity, and the load type.

    For the types of Database, File storage, and SaaS applications data assets that you can use as the source and target data entities, see Supported Source and Target Types.

    For load type, the source data to be loaded can come from Multiple data entities in a schema, or from a Single data entity. For example, data in two or more entities from an Oracle Database source can be loaded to an Object Storage target.

  • Source: Select the data asset, connection, and schema that has the source data for loading. Then, depending on the load type you specified, select one or more data entities to add to the source for loading. See Selecting the Source.
  • Target: Select the data asset, connection, schema, and data entity to use as the target. By default, the source and target entities are mapped by name. If you don't have an existing entity to load to, you can create a new data entity. See Selecting the Target.
  • Transformation: Use the Data Xplorer to apply transformations on the source attributes. A data loader task supports transformations at the metadata and data levels. See Applying Transformations.
  • Attribute mapping: When loading data to an existing target data entity or multiple entities, by default the source attributes are mapped to the target attributes by attribute name. You can apply more mapping rules to all attributes across all mapped entities. See Mapping Attributes.
  • Review and validate: Review and amend any of the configuration steps, and ensure that the data loader task is valid before you publish. See Reviewing and Validating the Task.

Creating a Data Loader Task

A configuration UI steps you through the process of creating a data loader task. The steps include selecting the data entities to use for the source and target, applying transformations, and mapping attributes when loading data to existing targets.

Important

If you are using hierarchical data entities, see Hierarchical Data Types to understand what is supported.

To create a data loader task:

  1. On the workspace home page, in the Quick actions tile, select Create data loader task.

    Alternatively, you can navigate to a project or folder, and click Tasks on the project or folder details page. Then select Data loader from the Create task menu.

    The Create data loader task page displays. Numbered steps at the top guide you through the configuration. A check mark displays on the step icon after you have configured a step. To move between steps, select Next or Previous. You can also navigate directly to a configured step by selecting the icon.

  2. On the Create data loader task page, Basic information step, select the following:
    • Source type: The data source type to use for the source data entity.

    • Target type: The data source type for use for the target data entity.

    • Load type: Select Multiple data entities to use more than one entity from the same source schema. Select Single data entity to use only one entity for the source.

  3. Enter a Name and Identifier, and an optional Description for the task.

    The Identifier field is a system-generated value based on what you enter for Name. You can change the value, but after you create and save the task, you won't be able to update it again.

  4. For Project or folder, click Select, and select the project or folder to save your data loader task to.

    If you're creating this task from a project or folder details page, this field is auto-populated for you.

  5. To save the task for the first time, you can click:
    • Create: Creates and saves the task. You can continue to create and edit the task.
    • Create and close: Creates and saves the task, closes the page, and returns you to the project or folder details tasks page or workspace home page.
  6. To configure the source and target data entities for this data loader task, follow these steps:
    1. Select the source data entity.
    2. Create or select the existing target data entity.
    3. (Optional) Apply transformations on the source attributes.
    4. For an existing target data entity, map the source attributes to the target attributes.
    5. Review and validate the task configuration.
  7. When you have completed working on your task, click Create and close or Save and close.
Publish the data loader task to an application in Data Integration before you can run the task, schedule the task for running, or use the task in a pipeline.

Selecting the Source

Select the data asset, connection, and schema that has the source data for loading. Then select one data entity or multiple data entities in the schema, depending on the load type you have specified for the data loader task.

When selecting multiple data entities from a file storage source type (such as Object Storage), you can use a file pattern to add entities that match the pattern, and use the logical entity qualifier to group matching entities into one or more pattern groups. Each pattern group is treated as a logical entity during runtime.

Selecting a Source Data Asset, Connection, and Schema

To configure the source data for a data loader task, you begin by selecting a data asset, connection, and schema.

  1. On the Create data loader task page, go to the Source step.
  2. To create a data asset for selecting the source, select Create next to Data asset to create a data asset.
  3. To select a source data entity from an existing data asset, first use the resource menus to select the Data asset, Connection, and Schema (or Bucket) only in the order as displayed. Note the following:
    • The Data asset resources available for selection depend on the type of source you selected on the Basic information step. Click View all next to a resource if you do not find what you want from the resource menu.

    • If applicable for a data asset (for example, Object Storage), select a Compartment, and then select the Bucket that contains data entities.

    • For BICC Oracle Fusion Applications, select the BICC offering as the Schema. The BICC offering that you select provides the BICC view objects (VO) for data extraction.

    • For a database Connection, you can select Add to create a connection for adding.

  4. Depending on the data asset type of your source, you might have other options to select.
    Data asset type Option

    Object Storage, S3, HDFS

    Under File settings, select the File type, Compression type, and Encoding, if applicable. The compression types that are available for selection depends on the selected file type.

    For CSV, the other settings you can configure are:

    • If the first row in the file is a header row, select Yes for Has header.
    • If the values in the data rows span multiple lines, select Yes for Multi-line.
    • Specify the Escape character that escapes other characters found in data values. For example: \
    • Select the Delimiter character that separates data fields. For example: COLON, COMMA, PIPE, SEMICOLON, or TAB
    • If a column delimiter is included at the end of a data row in the file, select Yes for Trailing delimiter.
    • Specify the Quote chracter that treats other characters as literal characters. For example: "

    For EXCEL, the settings you can configure are:

    • By default, Data Integration treats the first row in a file as a header row. If the first row in your file is not a header row, select No for Has header.
    • For Select entity by, you can choose the criteria as Sheet name, Sheet index, or Table name. Then enter a Value for the worksheet name, worksheet index, or table name. Sheet index is zero-based.
    • For Sheet name or Sheet index, enter the area of the file to use as the Data range for selection. If you don't enter a data range value, the default is the data range A1, which corresponds to the entire sheet. If the file has a header row, enter a value that starts from the header row, for example, A1:K56.
    BICC Oracle Fusion Applications

    Select the Full extract strategy to extract and load all data, or extract and load all data from the specified Initial extract date.

    Select the Incremental extract strategy to extract and load only data that is created or modified after the specified Last extract date.

    Specify a different BICC external configuration storage to use for the extracted data. Otherwise, the default storage that is configured within the BICC Oracle Fusion Applications data asset is used.

    Under Settings:

    Select Enable BI broker if you want the BICC extractor to read input rows directly from the Fusion Applications database. The BI Broker extract mode is supported only for some data stores. If you do not select Enable BI broker, the data is extracted through the BI server. For more information, see the BICC documentation.

    Select the type of BICC column properties from the BICC source to include as attributes in the target.

    • All: All columns of the BICC PVO are included. However, it does not display columns with names that start with ExtnAttribute or the columns that have same name and label.
    • Default only: Only the columns enabled for extract by default in the BICC PVO are included. It displays all columns including the Flex columns.
    • Default and primary keys: Default and primary columns of the BICC PVO are included. It displays all columns including the Flex columns.
    • Primary keys only: Only the primary columns of the BICC PVO are included. It displays all columns including the Flex columns.
    BIP Oracle Fusion Applications

    Specify the Staging location.

    Select Enable row limit to specify the maximum number of rows to retrieve.

    In the Row limit field, specify the row limit.

    If you expect huge data volume, specify the size in the Chunk size field under Enable chunking. If the specified row limit is greater than 100000, specifying the chunk size is mandatory.

    For BIP properties, select a report property each in the Number of rows returned and Starting offset fields. You cannot select the same property in both the fields.

After selecting the data asset, connection, schema (or bucket), and applicable source options, the data entities that are available for selection are displayed in the Available data entities table on the Data entities tab.
Selecting a Single Source Data Entity

From the data entities table, select the data entity you want to use as the source. Data from the selected data entity loads to the target when the task is run.

  1. Go to the Source step, Data entities tab.
  2. In the Available data entities table, select one data entity.

    The number of entities that are available for selection is shown in parentheses, for example Available data entities (27).

    Depending on your source type, there might be more than one way to select a data entity.

    • To select a data entity, select the check box that is next to the entity name. Then click Set as source.

    • To filter the list of available entities, enter a name or a pattern in the field, and press Enter.

      You can enter a partial name or a pattern using special characters such as *. For example, you can enter ADDRESS_* to locate ADDRESS_EU1, ADDRESS_EU2, and ADDRESS_EU3.

      From the filtered list, select the check box that is next to an entity name, and then click Set as source to select that entity as the source.

    • For a Database source type, you can use a SQL statement to select a data entity.

      Click Enter custom SQL. In the editor, provide a single SQL statement that defines the data to use as the source. Then click Validate to check your statement for errors. If validation is successful, click Add.

    The name of the data entity you selected is displayed next to Selected data entity.

    (For a Database source type) If you provided a SQL statement, the label SQL_ENTITY<nnnnnnnnn> is displayed, for example: SQL_ENTITY123456789

  3. To further configure your data source and loader task, click the Settings tab, if applicable. Depending on your source type, you can:
    • Allow pushdown or turn off pushdown: By default, some data processing is offloaded to the source system. To apply processing or transformations outside the source system, clear the check box.
    • Allow schema drift or lock the schema definitions: By default, schema definition changes in the specified data entity are automatically detected and picked up (design time and runtime). To use a fixed shape of the specified data entity even when the underlying shape has changed, clear the check box.
    • Fetch file metadata as attributes: By default, the file name, file size, and other file metadata are included as attributes in the source data. Clear the check box if you do not want to use file metadata as attributes.
Selecting Multiple Source Data Entities

From the list of available data entities, select the data entities you want to use as the source. You can select the data entities individually or use a file pattern to select entities as a group. Data from the selected source data entities loads to the mapped targets when the task is run.

Data Integration creates rules for the data entities that you include as the source. Rules are added when you make individual data entity selections or when you use a file pattern (with or without a group name). Grouped data entities are treated as a logical entity during runtime.

  1. Go to the Source step, Data entities tab.

    The Data entities tab has two tables. One table shows the data entities that are available for selection. The other table shows the data entities that are selected for loading to the target.

  2. In the Available data entities table, select the data entities you want to include in the source for this data loader task.

    The number of entities that are available for selection is shown in parentheses, for example Available data entities (27).

    Depending on your source type, there might be more than one way to select the data entities you want. Use the approriate step or steps as needed.

    MethodStep
    Select data entities manually Step 3
    Filter the list and select all data entities Step 4
    Use a file pattern without a group name Step 5
    Use a file pattern with a group name Step 6 (for file storage source types only, for example Object Storage)
    Use a file pattern with a logical entity identifier and prefix group names Step 7 (for file storage source types only, for example Object Storage)
    Revert selections The Selected source data entities table shows the data entities you have selected to be included in the source. To revert selections, see Removing Entities from the Selected Source Data Entities List.
  3. To select multiple data entities manually:
    • Click the add icon (plus sign) for each data entity that you want to include as the source.

    • Select the check box for each data entity, and then click Add as source to include your selections as the source.

  4. To filter the list and select all available data entities:
    • Filter the list of available data entities by entering a name or a pattern in the Filter by name and pattern field and pressing Enter. For example, enter c* to show only the entity names that begin with the letter C, such CUSTOMER_A and CUSTOMERS.

      The matching data entities might span across multiple pages in the table.

    • To select all the available data entities that are shown on the current page:

      Select the topmost check box (next to Name). Then click Add to source.

    • To select all the available data entities across all pages in the table:

      Select the topmost check box (next to Name). Click Select all data entities that match the pattern, then click Add to source.

    To select all the available data entities across all pages in the table without using a filter pattern:

    • Select the topmost check box (next to Name). Then click Select all available data entities, and click Add to source.

  5. To use a file pattern without providing a group name:
    1. In the Filter by name and pattern field, enter a file pattern and press Enter.
    2. Click Add to source.
    3. (Optional) If applicable, in the Add to source panel, you can change the value in the Pattern field.
    4. Click Add.
  6. (File storage source types) To use a file pattern with a group name:
    1. In the Available data entities table, Filter by name and pattern field, enter a file pattern (for example, MYSRC_BANK_C*.csv) and press Enter.
    2. Click Add to source.
    3. (Optional) In the Add to source panel, you can change the value in the Pattern field.
    4. Select Add as a group using a file pattern.
    5. In the Group name field, enter a name for this pattern group.
    6. Click Preview groups.

      Wait for a clickable group name to return in the preview table. This might take some time if there are many files that match.

      1. To verify that you have the files you want in the pattern group, click the group name.
      2. In the View pattern group details panel, verify the details and click Close.
    7. In the Add to source panel, click Add.

      The data entities that match the file pattern are included as one group in the Selected source data entities table. Future incoming data entities that match the pattern are also included in the group.

  7. (File storage source types) To use a file pattern with a logical entity identifier and a prefix for the group names:
    1. In the Available data entities table, Filter by name and pattern field, enter a file pattern using a logical entity identifier (for example, MYSRC_BANK_{logicalentity:B|D}*.csv) and press Enter.
    2. Click Add to source.
    3. (Optional) In the Add to source panel, you can change the value in the Pattern field.
    4. Select Add as a group using a file pattern.
    5. In the Prefix for pattern group name field, enter a name to use as the prefix for the logical groups. For example: MYSRC_
    6. Click Preview groups.

      Wait for clickable group names to return in the preview table. This might take some time if there are many files that match the pattern groups.

      1. To verify that you have the files you want in each pattern group, click a group name. For example: MYSRC_B
      2. In the View pattern group details panel, verify the details and click Close.
    7. In the Add to source panel, click Add.

      The data entities that match the file pattern and logical entity identifier are included as groups in the Selected source data entities table. Future incoming data entities that match the pattern groups are also included in the groups. If the same entity matches multiple groups, it's included in all those groups.

  8. To further configure your data sources and loader task, select the Settings tab, if applicable. Depending on your source type, you can:
    • Allow pushdown or turn off pushdown: By default, some data processing is offloaded to the source system. To apply processing or transformations outside the source system, clear the check box.
    • Allow schema drift or lock the schema definitions: By default, schema definition changes in the specified data entity are automatically detected and picked up (design time and runtime). To use a fixed shape of the specified data entity even when the underlying shape has changed, clear the check box.
    • Fetch file metadata as attributes: By default, the file name, file size, and other file metadata are included as attributes in the source data. Clear the check box if you do not want to use file metadata as attributes.
Removing Entities from the Selected Source Data Entities List

When you remove data entities from the Selected source data entities list, those data entities are no longer included in the source for the data loader task.

  1. Go to the Source step, Data entities tab.
  2. You can remove data entities from the Selected source data entities list in the following ways:
    • Click the remove icon (negative sign) for the data entites that you want to remove from the selected list.

      Data Integration created a rule when you selected multiple data entities by using a pattern without a group. If you remove individual entities that were added to the Selected source data entities list by a pattern rule, exclude rules are created.

    • Select the check boxes for the data entities (that are not groups), and then click Remove to remove those entities from the source.

    • Filter the selected list by entering a name or a pattern in the field and pressing Enter. You can enter a partial name or a pattern using special characters such as *.

      The matching entities might span across multiple pages in the table.

      • Exclude all the selected data entities that are shown on the current page:

        Select the topmost check box (next to Name). Then click Remove.

      • Exclude all the selected data entities across all pages in the table:

        Select the topmost check box (next to Name). Click Select all included data entities, then click Remove.

    • Exclude a group of data entities by group name.

      • Click a group name. In the panel, review the list of data entities that were added in that group.

      • Then click the remove icon (negative sign) for the group that you want to exclude.

      Note

      If you exclude a group that is created by a file pattern, other pattern groups that were created as a result of the file pattern are also excluded.

    • Exclude different types of data entities (for example, by group and by selection).

      • Select the check boxes for group names and data entity names, and then click Remove.

      • In the Remove entity panel, review the list of data entities that are impacted. For example, all pattern groups related to the group you select to remove are impacted. Then click Remove.

    • Remove a rule to exclude those data entities impacted by that rule. See Viewing, Editing, and Removing Rules.

    The data entities you removed are now available for selection again from the Available data entities list.

Using File Patterns and Groups

When selecting multiple data entities from a file storage source type (for example, Object Storage) to use as the source for a data loader task, you can use a file pattern to group and add existing files that match the pattern. Future incoming files that match the pattern are also included in the group.

In the file pattern, you can also use the logicalentity qualifier to group matching entities into one or more pattern groups. Each pattern group is treated as a logical entity during runtime.

Data entities that match multiple pattern groups are included in all those groups.

Consider the following filenames of data entities that are available for selection:

SRC_BANK_A_01.csv
SRC_BANK_B_01.csv
SRC_BANK_C_01.csv
SRC_BANK_C_02.csv
MYSRC_BANK_A_01.csv
MYSRC_BANK_B_01.csv
MYSRC_BANK_C_01.csv
MYSRC_BANK_C_02.csv
MYSRC_BANK_D_01.csv
MYSRC_BANK_D_02.csv

When you use the file pattern SRC*.csv, Data Integration creates a pattern rule and adds the following files to the source:

SRC_BANK_A_01.csv
SRC_BANK_B_01.csv
SRC_BANK_C_01.csv
SRC_BANK_C_02.csv

When you use the file pattern MYSRC_BANK_C*.csv and provide the group name MYSRC, Data Integration creates a group rule. At runtime, the group name consolidates all the files matching the pattern into one source entity named MYSRC. For example, the following files are consolidated:

MYSRC_BANK_C_01.csv
MYSRC_BANK_C_02.csv

Any future incoming files that match the pattern are added to the group. For example:

MYSRC_BANK_C_03.csv
MYSRC_BANK_C_04.csv

When you use the file pattern with the logicalentity qualifier, MYSRC_BANK_{logicalentity:B|D}*.csv, and you provide the group name prefix MYNEWSRC_, Data Integration creates a group rule, and adds two pattern groups that consolidate the following matching files:

For pattern group MYNEWSRC_B:
MYSRC_BANK_B_01.csv

For pattern group MYNEWSRC_D:
MYSRC_BANK_D_01.csv
MYSRC_BANK_D_02.csv
Viewing the List of Files Included in a Group

Data Integration creates groups in the Select source data entities list when you use a file pattern to select multiple files (for example, from Object Storage) as a group for inclusion in the source for a data loader task.

  1. Go to the Source step, Data entities tab.
  2. In the Selected source data entities list, click a group name.
  3. In the View pattern group details panel, you can view the pattern used to create the group, and the list of data entities that match the pattern.
Viewing, Editing, and Removing Rules (Multiple Source Data Entities)

Data Integration adds rules when you select multiple data entities to be included in the source for a data loader task.

A rule is added when you made individual data entity selections or when applicable, you included the entities by a pattern or group. The number of rules is shown above the Selected source data entities table, in parentheses next to View rules. For example, View rules (3).

Before removing a group rule, ensure that you review the list of data entities impacted by the rule removal. See Viewing the List of Files Included in a Group.

  1. Go to the Source step, Data entities tab.
  2. To display the rules created for a data loader task, select View rules above the Selected source data entities table.
  3. In the Rules panel, review the criteria that were created when you added or removed source data entities.
  4. To remove a rule that is not a pattern group, you can use one of two ways:
    • Select the check box for a rule, then click Remove.

    • Select the actions menu for a rule, then click Remove.

    Note

    When you remove the rule, Data Integration removes from the Selected source data entities table those data entities impacted by the original rule (included by selection or by pattern). To add the entities back to the selected list, you have to select and add them again from the Available data entities table.
  5. To edit a pattern group rule:
    1. Select Edit from the actions menu for the pattern group rule.
    2. In the Edit rule panel, make your changes and click Save changes.
  6. To remove a pattern group rule:
    1. Select Remove from the actions menu for the pattern group rule.
    2. In the Remove rule panel, review the impact of removing this named pattern group rule and click Remove.
    Note

    If the named pattern group rule is a logical entity file pattern group, the other pattern groups that were created as a result of the logical entity file pattern are also removed.

Selecting the Target

Select the data asset, connection, and schema to use as the target. Then configure a new target entity or select an existing data entity to load the data into.

Selecting a Target Data Asset, Connection, and Schema

To configure the target data entity for a data loader task, you begin by selecting a data asset, connection, and schema.

  1. On the Create data loader task page, go to the Target step.
  2. To create a data asset for selecting the target, select Create next to Data asset to create a data asset.
  3. To select a target data entity from an existing data asset, first use the resource menus to select the Data asset, Connection, and Schema (or Bucket) only in the order as displayed. Note the following:
    • The Data asset resources available for selection depend on the type of target you selected on the Basic information step. Click View all next to a resource if you do not find what you want from the resource menu.

    • If applicable for a data asset (for example, Object Storage), select a Compartment before you select a Bucket (schema).

    • For a database Connection, you can select Add to create a connection for adding.

  4. Depending on the data asset type of your target, you might have other options to select.
    Data asset type Option

    Object Storage, HDFS

    Under File settings, select the File type and Compression type, and Encoding, if applicable. The compression types that are available for selection depends on the file type.

    For CSV, the other settings you can configure are:

    • If the first row in all the files is a header row, select Data has header.
    • If the values in the data rows span multiple lines, select Multi-line.
    • Specify the Escape character that escapes other characters found in data values. For example: \
    • Select the Delimiter character that separates data fields. For example: COLON, COMMA, PIPE, SEMICOLON, or TAB
    • Specify the Quote character that treats other characters as literal characters. For example: "
    • If a column delimiter is included at the end of a data row in all files, select Trailing delimiter.

    Autonomous Data Warehouse, Autonomous Transaction Processing

    Use the default staging location that is configured for the data asset or specify a different staging location.

    To use a different staging location, under Staging location, clear the check box Use default staging location settings. Then use the resource menus to select a data asset, connection, compartment, and then the bucket (schema) to use for staging.

After selecting the data asset, connection, schema (or bucket), and applicable target options, configure the target load settings for a new data entity or existing data entity.
Using New Target Data Entities

Data Integration can create new target data entities when loading data from the source entities.

  1. Go to the Target step, Data entities tab.
  2. Under Target data entities load settings, select Create new data entity.

    For a new target, the Integration strategy is always Insert.

  3. For single data entity load type and Object Storage target only: Select the Create output as a single file check box if you want to use a single output file. Otherwise, multiple files are created.

    The single output file is overwritten every time the task is run. Creating a single output file might affect the performance of Data Integration. Do not use the single output file option for large datasets.

  4. Specify the Target data entity name option you want for your new target. You can choose from:
    • Use same entity names as source: Select this option to create target entities with the same names as the source entities.

    • Add prefix/suffix: Select this option to add a string at the start (prefix) or at the end (suffix) of the source entity names to create the target entity names.

    • (For single entity load type only) Specify entity name: Enter the new target entity name in the field.

      • For Object Storage: Enter the new data entity name, followed by a forward slash (/). For example, enter newfile/ or newdirectory/newfile/. However, if you select the Create output as a single file check box because you want to create a single file output, enter the new entity name without the forward slash (/) at the end.

      • For a database target: If the entity name that you provide exists, the outcome of the operation depends on the shape of the target from the data loader task and the shape of the existing entity.

  5. If applicable, select the Settings tab and enter a value for Reject limit.

    For Autonomous Data Warehouse or Autonomous Transaction Processing:

    You can specify the maximum number of erroneous rows that can fail to load into the target before the task fails. For example, if your data source has 1,000 rows and you set the reject limit at 200, the task fails immediately after the 200th erroneous row is rejected.

    If you don't specify a value, the default is zero, which means the task fails upon the first erroneous row being rejected.

    If you encounter a task failure, check the logs for the table names where the rows had been rejected. Then query the affected tables and rows in the autonomous database.

Using Existing Target Data Entities

When using existing target data entities, you select the integration strategy that determines how to load data into the existing target.

  1. Go to the Target step, Data entities tab.
  2. Under Target data entities load settings, select Use existing data entity.
  3. To specify how to load data into the target, select one of the available Integration strategy options.
    • Insert: Inserts new records, or appends the records when the data exists on the target.

    • Overwrite: Performs a truncate on the target before inserting new records.

      For single data entity load type, the integration strategy must be Overwrite if you want to create output as a single file for the Object Storage target.

    • Merge: Inserts new records, and merges existing ones. This integration strategy is available for single data entity load type and database targets only.

  4. For Single data entity load type:
    1. In the Available data entities table, select one data entity. You can:
      • Select the check box that is next to an entity name, and then select Set as target.

      • Filter the list by entering a name or a pattern in the field and pressing Enter.

        You can enter a partial name or a pattern using special characters such as *. For example, you can enter ADDRESS_* to locate ADDRESS_EU1, ADDRESS_EU2, and ADDRESS_EU3.

        From the filtered list, select the check box that is next to an entity name, and then select Set as target.

      • For an Object Storage target with the Overwrite integration strategy, select an existing data entity that does not have the forward slash (/) at the end of the entity name.
      The name of the data entity you select is displayed above the table next to the label Selected data entity. To select a different data entity to use as the target, click Remove and then select the entity from the Available data entities table.
    2. (Optional) For a database target with the Merge integration strategy, click Select next to Merge key.
      In the Merge key panel, select a unique key other than the primary key to merge your data.
  5. For Multiple data entities load type: By default, source data entities are automatically mapped to available target data entities by name.
    1. In the Source data entities table, Mapping column, you can place your cursor over a mapping that has a green check mark to see the mapped target entity name.
    2. To create a manual mapping, drag the source data entity to the target data entity. A mapping rule is added when you perform a manual mapping.
    3. To remove an AUTO or a manual mapping, in the Target data entities table, Mapping column, select Clear next to a mapping. Then select Clear mapping to confirm that you want to remove the mapping. A mapping rule is added when you clear a mapping.
    4. In either the Source data entities or Target data entities table, you can:
      • Filter the list by entering a name or a pattern in the field and pressing Enter.

        You can enter a partial name or a pattern using special characters such as *. For example, you can enter ADDRESS_* to locate ADDRESS_EU1, ADDRESS_EU2, and ADDRESS_EU3.

      • View all entities, mapped entities, or unmapped entities by selecting the appropriate option from the menu above the table.

    5. From the Actions menu, you can select:
      • Auto-map by name: Use this action to let Data Integration automatically create mappings between source entities and target entities by name.

      • Map by pattern: Use this action to define a source pattern and a target pattern for mapping source entities to target entities. Pattern matching is case-sensitive.

        See Map by Pattern.

    To manage mapping rules, see See Viewing and Removing Mapping Rules.
Viewing and Removing Mapping Rules (Multiple Target Data Entities)

Data Integration adds mapping rules when you manually map source data entities to existing target data entities for a data loader task.

The number of mapping rules is shown above the Target data entities table, in parentheses next to View mappings. For example, View mappings (3).

Rules are also added when you remove mappings.

  1. Go to the Target step, Data entities tab.
  2. To display the rules created for a data loader task, select View mappings above the Target data entities table.
  3. In the Entity mapping rules panel, review the mappings that you created.
  4. To remove a mapping rule, you can use one of two ways:
    • Select the check box for a rule, and then select Remove.

    • Select the Actions menu for a rule and then select Remove.

Applying Transformations

A data loader task supports transformations at the metadata and data levels.

Applying transformations is optional. After selecting the source data entity, you can access the Data Xplorer to apply transformations on the source attributes.

  1. On the Create data loader task page, go to the Transformation (optional) step.

    The interactive Data Xplorer displays. Learn about Data Xplorer.

  2. On the Attributes tab, filter the data entity attributes by pattern or data type.

    For multiple data entities load type, use the Select source entity menu to choose the data entity you want before viewing and filtering attributes.

    Select an option from Actions to apply transformations to the filtered attributes. You can also use the Actions icon (three dots) next to individual attributes to apply a transformation to a single attribute.

    Learn about the transformations you can apply.

  3. Display the transformation rules that you have applied by selecting the icon next to the data entity.
    Data Loader transformations panel
  4. On the Data tab, view a sampling of data.

    When you apply transformations, the data sampling is updated to reflect the transformations.

Mapping Attributes

The Attribute mapping step is only available for data loader tasks that use existing target data entities.

When you specify existing target data entities for a data loader task, by default Data Integration uses attribute names to automatically map the attributes between the source and target entities.

You can add more mappings to map source entity attributes to target entity attributes. Mapping attributes is supported in data loader tasks that are configured with either the single entity or multiple entities load type.

To view the data entity attributes and their mappings:

  1. On the Create data loader task page, go to the Attribute mapping step.

    If your data loader task is configured for multiple entities load type, you might have to wait for the list of entities to display.

  2. For single data entity load type:

    A source entity table is displayed next to a target entity table.

    The total number of attributes in each entity, the number of mapped attributes, and the number of attributes without mappings, are shown in colored number icons above the tables.

    OptionDescription
    menu to filter attributes

    In both tables, All the attributes are shown by default, as indicated by the menu above the Name column header.

    Use the menu to filter the list of attributes. You can choose to view only the Mapped attributes or Attributes not mapped.

    In the tables, an icon in the Mapping column indicates the type of attribute mapping. For example, Auto or Manual. You can place your cursor over a mapping to see the mapped attribute name.

    Search by name

    In the search field, enter a partial attribute name to filter the list quickly. For example, you can enter name to display the attributes FIRST_NAME and LAST_NAME.

    icon to collapse/expand list

    You can collapse and expand the lists by using the three dots icon above the Mapping column header.

    Map by name

    By default, the option is not enabled because the Map by name mapping is already added. Data Integration automatically adds the default mapping, which maps source entity attributes to target entity attributes by their attribute names.

    The option is enabled when the Map by name mapping is removed.

    Add pattern

    Use the option to add more mappings, either manually or by using a pattern.

    See:

    View mappings (1)

    Use the option to view all the mappings that have been added to the data loader task. The number in parentheses shows the number of mappings.

    See Viewing and Removing Mappings.

  3. For multiple data entities load type:

    A table is displayed showing the source data entities that are mapped to the target data entities. The number of mapped attributes between the two entities is also shown.

    The Number of mapped attributes is shown as n1/n2, where n1 is the number of mapped attributes in the source entity, and n2 is the total number of source attributes. For example, 5/10 means that the source entity has 10 attributes in total, and 5 attributes are mapped to attributes in the target.

    OptionDescription
    Filter by target data entity name or pattern

    Enter a name or pattern in the field and press Enter to filter the data entities list. You can enter a partial name or a pattern using special characters such as *. For example, enter c* to display only the entity names that begin with the letter C, such CUSTOMER_A and CUSTOMERS.

    Map by name

    By default, the option is not enabled because the Map by name mapping is already added. Data Integration automatically adds the default mapping, which maps source entity attributes to target entity attributes by their attribute names.

    The option is enabled when the Map by name mapping is removed.

    Add mapping

    Use the option to add more mappings, either manually or by using a pattern.

    See:

    View mappings (1)

    Use the option to view all the mappings that have been added to the data loader task. The number in parentheses shows the number of mappings.

    See Viewing and Removing Mappings.

Viewing and Removing Mappings

By default, Data Integration automatically maps source entity attributes to target entity attributes by matching attribute names. The default mapping is All attributes, of the type Map by name.

You can view the default mapping, and other mappings that you have added for mapping source entity attributes to target entity attributes.

  1. On the Create data loader task page, go to the Attribute mapping step.

    If your data loader task is configured for multiple entities load type, you might have to wait for the list of entities to display.

  2. Select View mappings (n).

    The number in parentheses shows the number of mappings. For example: View mappings (1)

    In the Mappings panel, the types of mappings are:

    TypeMapping
    Map by name

    All attributes

    Data Integration automatically adds the default mapping for mapping all attributes by attribute names.

    Map by pattern

    For example:

    *_NAME to $1_NAME might map the source attribute LAST_NAME to the target attribute TGT_LAST_NAME, and the source attribute FISRT_NAME to the target attribute TGT_FIRST_NAME.

    Manual

    For example:

    CITY to CITY_NAME maps the source attribute CITY to the target attribute CITY_NAME.

  3. To remove a mapping, you can use one of two ways:
    • Select the check box for a mapping, and then select Remove.

    • Select the Actions menu for a mapping, and then select Remove.

Mapping an Attribute Manually

You create an attribute mapping manually by dragging a source attribute to a target attribute.

  1. On the Create data loader task page, go to the Attribute mapping step.

    If your data loader task is configured for multiple entities load type, you might have to wait for the list of entities to display.

  2. For single data entity load type:

    By default, all attributes are shown in the source entity and target entity tables. The menu above the Name column header indicates All. Attributes that are not mapped do not have an icon in the Mapping column.

    1. In each table, select Attributes not mapped from the menu.
    2. To create a manual mapping, drag a source attribute to a target attribute.

      A Manual icon is added to the Mapping column.

  3. For multiple data entities load type:

    In the data entities table, the Number of mapped attributes column shows the number of source attributes that are mapped. For example, 5/10 means that the source entity has 10 attributes in total, and 5 attributes are already mapped to attributes in the target.

    1. Select the check box next to a source data entity, and select Add mapping.

      The Add mapping page displays, showing the attributes of the selected source entity in one table, and the attributes of the mapped target entity in another table. By default, all attributes are shown in both source and target entities. The menu above the Name column header indicates All.

      In both tables, an icon in the Mapping column indicates the type of attribute mapping. For example, Auto or Manual. You can place your cursor over a mapping to see the mapped attribute name. Attributes that are not yet mapped do not have an icon in the Mapping column.

    2. In each table, select Attributes not mapped from the menu.
    3. To create a manual mapping, drag a source attribute to a target attribute.

      A Manual icon is added to the Mapping column.

    4. Above the target attributes table, you can use the menu to select another source entity and create more manual mappings.
    5. When you have finished creating manual mappings, select Save pattern mapping.
Adding a Mapping by Pattern (Single Entity Load)
  1. On the Create data loader task page, go to the Attribute mapping step.

    By default, all attributes are shown in the source entity and target entity tables. The menu above the Name column header indicates All.

  2. Select Add pattern.
  3. In the Map by pattern panel that displays, define a Source pattern and a Target pattern to map source attributes to target attributes. Pattern matching is case-sensitive.
  4. Select Preview mapping to verify that the identified source attributes and target attributes are the attributes that you want to map.
  5. Modify the Source pattern and Target pattern, if necessary, then select Preview mapping again.
  6. When done, select Map.
Adding a Mapping by Pattern (Multiple Entities Load)
  1. On the Create data loader task page, go to the Attribute mapping step.

    If your data loader task is configured for multiple entities load type, you might have to wait for the list of entities to display.

  2. Select the check box next to a source data entity, and select Add mapping.

    The Add mapping page displays, showing the attributes of the selected source entity in one table, and the attributes of the mapped target entity in another table. By default, all attributes are shown in both source and target entities. The menu above the Name column header indicates All. Attributes that are not mapped do not have an icon in the Mapping column.

  3. In each table, you can select Attributes not mapped from the menu to list unmapped attributes only.
  4. Select Add pattern.
  5. In the Map by pattern panel that displays, define a Source pattern and a Target pattern to map source attributes to target attributes. Pattern matching is case-sensitive.
  6. Select Preview mapping to verify that the identified source attributes and target attributes are the attributes that you want to map.
  7. Modify the Source pattern and Target pattern, if necessary, then select Preview mapping again.
  8. When done, select Map.
  9. In the Add mapping page, above the target attributes table, select View all or use the menu to select another source entity. Then select Add pattern to create more mappings as described from step 5 through step 8.
  10. When you have finished creating mappings, select Save pattern mapping.

Reviewing and Validating the Task

Review the data loader task configuration before you publish the task to an application in Data Integration.

Validation of the task begins automatically when you navigate to the Review and validate step.

  1. On the Create data loader task page, go to the Review and validate step.

    A summary of the configuration details for each step is presented in a block. For example, Source, Target, and Transformation.

    The result of the task validation is shown in the last block, Validation.

  2. In the Validation block, select View messages to review validation errors or warnings, if any.

    Resolve any errors before you publish the data loader task.

  3. To modify a step's configuration, you can select Edit in the block of that step. You can also navigate directly to a configured step by selecting the step icon.

    The display changes to the full configuration details for that step when you select Edit or navigate directly to the step.

  4. After reviewing the steps and making the necessary changes, navigate to the Review and validate step to validate the task again.
    The result of the task validation is shown in the last block, Validation.

Editing a Data Loader Task

You can edit and modify a data loader task from the project or folder details page where you saved the task.

To edit a data loader task:

  1. Click Open tab (plus icon) in the tab bar, and select Projects.
  2. On the Projects page, select the project containing the data loader task you want to edit.
  3. If the task is saved to a folder in this project, click Folders, then click Tasks. Otherwise, on the project details page, click Tasks.
  4. From the tasks list, select View details from the Actions menu of the task that you want to edit.
  5. In the Edit data loader task page that displays, use the configuration UI to edit the step or steps that you want to modify.

    To move between steps, you can select Next or Previous. You can also navigate directly to a step by selecting the icon.

  6. Save periodically while you work. You can click:
    • Save: Commits changes since your last save. You can continue editing after saving.
    • Save and close: Commits changes, closes the page, and returns you to the project or folder details tasks page.

Deleting a Data Loader Task

If you want to delete a data loader task, you can do so from the tasks list on a project or folder details page. After a task is deleted, it cannot be restored.

To delete a data loader task:

  1. Click Open tab tab (plus icon) in the tab bar, and select Projects.
  2. On the Projects page, select the project containing the data loader task you want to delete.
  3. If the task is saved to a folder in this project, click Folders, then click Tasks. Otherwise, on the project details page, click Tasks.
  4. In the tasks list, for the task you want to delete, select Delete from the Actions menu.
  5. In the Delete task dialog, confirm that you want to delete the task, and then click Delete.