Defining Data Quality Rules on Partitioned Tables

Data correction on partitioned table is accomplished by overwriting the particular partition specified. At run time, DQ engine look for partition information from OFSAAI object registration table REV_TAB_PARTITION. If base table is partitioned, REV_TAB_PARTITIONS table will have partition column, value, and sequence registered in it.

If PARTITION_VALUE does not present in REV_TAB_PARTITIONS table for a TABLE_NAME.COLUMN_NAME, it is considered as a dynamic partition.

Hive allows operations on dynamic partition only in non-strict mode. Non-strict mode is set by DQ engine while it identifies REV_TAB_PARTITION.V_PARTITION_VALUE as null.

Static partition value can also be set with placeholders. For example, $PARAM1, $PARAM2 and the same can be mentioned as ‘Additional Parameters’ while DQ batch execution. Value for the placeholders/ additional parameters will be substituted as the static partition values during DQ run time.