Incremental Data Quality Remediation

Allows user to run DQ checks on multiple files incrementally throughout the day. By introducing a new parameter called "Watermark" which keeps increasing with each data load. The DQ service will be enhanced to track the last processed "Watermark" and use it as a filter for subsequent incremental runs.

Provides a mechanism to run DQ checks only on newly ingested data since the last execution thereby improving performance when there are multiple transaction files arriving throughout the day.

Steps to enable and configure

You don't need to do anything to enable this feature.