9 Data Ingestion Errors Browser

DFCS is introducing an interface for monitoring data ingestion jobs. It provides a summary, like, number of records processed, number of records successfully loaded, and number of records with error. It also provides a browser for errors identified in data ingestion jobs – both errors in reading the source files as well as errors in populating respective target entities.

Data Flow

The ingestion process follows this flow: Source File → EDD → Connector → Target Table

Stages
  • EDD:
    • Reads data from the source file
    • Performs initial validation and transformation
  • Application Connector:
    • Loads processed data into the target database table
    • Applies database constraints

How Data Ingestion Works

Data moves in three stages:
Stage What Happens Where Errors Can Occur
1. Source File User uploads a CSV file to the system. No errors here
2. EDD The system reads the file and moves data into the application connector. Data type mismatch, column size mismatch, and similar issues
3. Application Connector to Target Table The system moves data from the connector into the final database table. Database violations such as duplicate primary key errors

If a source file has 100 records:

  • 80 records pass through EDD successfully
  • 20 records fail at the EDD stage

Out of the 80 records that go to the connector:

  • 60 records are loaded into the final table
  • 20 records fail at the connector stage

The failed records are stored separately:

  • 20 EDD failures → stored in EDD error table
  • 20 connector failures → stored in connector error table

3.2 Application Connector-Level Errors

Errors occurring while inserting data into the database.

Examples:
  • Primary key violation
  • Constraint violations
  • Duplicate records
These records:
  • Fail at connector stage
  • Stored in Application Connector error table

Important Execution Scenarios

Pipeline Stopped and Restarted

If a pipeline execution is stopped before it finishes and then resumed, the system creates a second instance of the same execution. Both instances share the same pipeline name, as-of date, and Run SKey, but they have different internal instance IDs.

In this case, the page automatically shows the errors from the latest (most recent) instance only. You do not need to do anything extra.

Note:

This means the page may not show errors from the earlier stopped instance. If you need to review both, contact your system administrator.

Threshold Value Set to Zero

Each application connector has a threshold setting. This setting controls how many record failures are allowed before the pipeline stops.
Threshold Value What Happens
0 (zero) The pipeline stops as soon as the very first record fails. No error records are saved to the database.
5% (example) The pipeline allows a small number of failures before stopping. Those failed records are saved and visible on this page.
If the threshold is set to zero, the pipeline will stop as soon as the first error occurs. In this case, no records are saved in the error table. So, even if the pipeline status shows as Failed, you will not see any records on the error page.

Note:

The threshold value is set by the user when configuring the connector. If you are seeing unexpected behavior, check the threshold value for that connector.