Load Data into Autonomous Data Warehouse Using a Data Loader Task

A data loader task helps you transform and load data into Data Lakes or other targets such as, Autonomous Data Warehouse.

In this tutorial, you:

  1. Create a data loader task to transform and load data into Autonomous Data Warehouse.
  2. Publish the data loader task to an Application.
  3. Run the data loader task.

1. Creating a Data Loader Task

  1. In the tab bar, click Open tab (plus icon), and then select Projects.
  2. On the Projects page, select DI_Lab.
  3. On the DI_Lab Project Details page, from the submenu, click Tasks.
  4. Click Create Task, and then select Data Loader.

    The Create Data Loader Task page opens in a new tab.

  5. On the Create Data Loader Task page, for Name, enter Load Revenue Data into Data Warehouse.
  6. In the Source section, click Select.
  7. In the Select Source page, select the following values:
    • Data Asset: Data_Lake
    • Connection: Default Connection
    • Compartment: The Compartment in which you have uploaded your REVENUE.CSV file
    • Schema: Your Object Storage bucket
    • Data Entity: Click Browse by Name, then select REVENUE.csv

      If you don't see the data entity you want in the initial list, click View More at the end of the list, and use the search field to narrow your search for a data entity. The search field is case-sensitive, and allows for full text and partial text search using wildcards.

    • File Type: Select csv. Then leave the default settings as-is in the remaining fields, and click Select.
  8. In the Configure Transformations section, click Configure.

    The Configure Transformations panel opens, showing the metadata information of the data entity and its attributes. You can also view sample data in the Data tab.

    Learn more about Data Xplorer.

  9. Click Data to navigate to the Data tab, then locate and select SRC_ORDER_NUMBER.

    A panel displays, showing the Data Profile and the Attribute Profile for SRC_ORDER_NUMBER. Null Data Percent for SRC_ORDER_NUMBER is at 100%.

  10. From the transformations icon (three dots) for SRC_ORDER_NUMBER, select Null Fill Up.

    Transformation menu

  11. In the Null Fill Up dialog, do the following:
    • Enter Not Available in the Replace by String field.
    • Do not select Keep Source Attributes.
    • Leave the Name as-is.
  12. Click Apply.

    After the Data tab refreshes, use the horizontal scrollbar to scroll to the end of the dataset where the updated SRC_ORDER_NUMBER column is. Notice the values for SRC_ORDER_NUMBER have been replaced by the string you entered.

  13. Click the Transformations icon next to the data entity name.

    Transformation panel expansion button

    You can review the list of transformations that are applied to the source dataset. To remove a transformation, click the X icon next to a transformed attribute name.

  14. To close the Configure Transformations panel, click OK.

    The number of transformation rules applied is shown in the Configure Transformations section.

  15. In the Target section, select the Create New Entity check box, and then click Select.
  16. In the Select Target page, select the following values:
    • Data Asset: Data Warehouse
    • Connection: Default Connection
    • Schema: BETA
    • New data entity: Enter REVENUE_TARGET for the new entity you're going to create.
  17. Under Staging Location, select the Data Lake data asset, its default connection, and your compartment name. Then for Schema, select a different bucket.
  18. Click Select to complete selecting the target.
  19. Click Create and Close.

2. Publishing a Data Loader Task

  1. On the DI Lab Project Details page, select Tasks in the submenu.
  2. In the Tasks list, locate Load Revenue Data into Data Warehouse.
  3. Click the Actions icon (three dots), and then select Publish to Application.
  4. In the Publish to Application dialog, select Lab Application.
  5. Click Publish.

3. Running a Data Loader Task

  1. In the tab bar, click Open tab (plus icon), and then select Applications.
  2. Select Lab Application to open its details page.
  3. On the Application Details page, in the Tasks list, select Run from the Actions icon (three dots) for Load Revenue Data into Data Warehouse.

    A success message appears. Running a task creates a task run.

  4. In the submenu, click Runs.

    The Runs list lets you view all task runs and their statuses. You can also check in Object Storage to verify if a new file is created in the correct compartment and bucket.

  5. Click Refresh to get the latest task run status updates.

    Learn more about task runs

  6. In the Application Details page, click Patches, and review details of the task patch.

    A patch contains updates to published tasks in an Application. You can find a list of patches on the Application Details page. When you publish a task, a Publish patch is created. When you unpublish a task, an Unpublish patch is created. Learn more about Patches.