The DP CLI (Command Line Interface) shell utility is used to
launch Data Processing workflows, either manually or via a cron job.
The Data
Processing workflow can be run on an individual Hive table, all tables within a
Hive database, or all tables within Hive. The tables must be of the
auto-provisioned type (as explained further in this topic).
The DP CLI starts workflows that are run by Spark workers. The results
of the DP CLI workflow are the same as if the tables were processed by a
Studio-generated Data Processing workflow.
Two important use cases for the DP CLI are:
- Ingesting data from your
Hive tables immediately after installing the Big Data Discovery (BDD) product.
When you first install BDD, your existing Hive tables are not processed.
Therefore, you must use the DP CLI to launch a first-time Data Processing
operation on your tables.
- Invoking the BDD Hive Table
Detector, which in turn can start Data Processing workflows for new or deleted
Hive tables.
The DP CLI can be run either manually or from a cron job. The BDD
installer creates a cron job as part of the installation procedure if the
ENABLE_HIVE_TABLE_DETECTOR property is set to TRUE in
the
bdd.conf file.
Skipped and auto-provisioned Hive tables
From the point of view of Data Processing, there are two types of Hive
tables: skipped tables and auto-provisioned tables. The table type depends on
the presence of a special table property,
skipAutoProvisioning. The
skipAutoProvisioning property (when set to
true) tells the BDD Hive Table Detector to skip the
table for processing.
Skipped tables are Hive tables that have the
skipAutoProvisioning table property present and set to
true. Thus, a Data Processing workflow will never be
launched for a skipped table (unless the DP CLI is run manually with the
--table flag set to the table). This property is set
in two instances:
- The table was created from
Studio, in which case the
skipAutoProvisioning property is always set at table
creation time.
- The table was created by a
Hive administrator and a corresponding BDD data set was provisioned from that
table. Later, that data set was deleted from Studio. When a data set (from an
admin-created table) is deleted, Studio modifies the underlying Hive table by
adding the
skipAutoProvisioning table property.
For information on changing the value of the
skipAutoProvisioning property, see
Changing Hive table properties.
Auto-provisioned tables are Hive tables that were created by
the Hive administrator and do not have a
skipAutoProvisioning property. These tables can be
provisioned by a Data Processing workflow that is launched by the BDD Hive
Table Detector.
Note: Keep in mind that when a BDD data set is deleted, its source Hive
table is not deleted from the Hive database. This applies to data sets that
were generated from either Studio-created tables or admin-created tables. The
skipAutoProvisioning property ensures that the table
will not be re-provisioned when its corresponding data set is deleted
(otherwise, the deleted data set would re-appear when the table was
re-processed).
BDD Hive Table Detector
The BDD Hive Table Detector is a process that automatically keeps a
Hive database in sync with the BDD data sets. The BDD Hive Table Detector has
two major functions:
- Automatically checks all
Hive tables within a Hive database:
- For each
auto-provisioned table that does not have a corresponding BDD data set, the BDD
Hive Table Detector launches a new data provisioning workflow (unless the table
is skipped via the blacklist).
- For all skipped
tables, such as, Studio-created tables, the BDD Hive Table Detector never
provisions them, even if they do not have a corresponding BDD data set.
- Automatically launches the
data set clean-up process if it detects that a BDD data set does not have an
associated Hive table. (That is, an orphaned BDD data set is automatically
deleted if its source Hive table no longer exists.) Typically, this scenario
occurs when a Hive table (either admin-created or Studio-created) has been
deleted by a Hive administrator.
The BDD Hive Table Detector detects empty tables, and does not launch
workflows for those tables.
The BDD Hive Table Detector is invoked with the DP CLI, which has
command flags to control the behavior of the script. For example, you can
select the Hive tables you want to be processed. The
--whitelist flag of the CLI specifies a file
listing the Hive tables that should be processed, while the
--blacklist flag controls a file with Hive
tables that should be filtered out during processing.
