Data set whitelists and blacklists

BDD whitelists and blacklists control which Hive tables are processed by the DP CLI. Whitelists specify tables that should be processed and blacklists specify tables that should be ignored. Only tables that have been processed are available to BDD and its users.

BDD provides default lists in $BDD_HOME/dataprocessing/edp_cli/config/. By default, the blacklist excludes all tables and the whitelist is empty, so you don't need to worry about accidentally processing sensitive data immediately after installation. No data will be processed until you modify these files, even if the Hive Table Detector is configured to run automatically.

The blacklist overrides the whitelist—if a file is specified in both lists, it won't be processed. Additionally, the lists only work when they're used with the DP CLI.

To keep your sensitive data secure, be sure to specify any tables you don't want processed in the blacklist, and always include this file when launching Data Processing jobs. If the Hive Table Detector is scheduled to run automatically, verify that the blacklist is specified in the crontab file (if you enabled the Hive Table Detector at install time, this should already be set). Also, as previously mentioned, access to the DP CLI and its whitelist and blacklist should be restricted.

For more information on using whitelists and blacklists, see the Data Processing Guide.