Harvesting Object Storage Files as Logical Data Entities
Your data lake typically has a large number of files that represent a single data set. The files naming conversions indicate that multiple files are part of a single logical data entity.
You can group multiple Object Storage files into logical data entities in data catalog using filename patterns . A logical data entity is like any other data entity and can be used for search and discovery. Using logical data entities, you can organize your data lake content meaningfully and prevent the explosion of data entities and attributes in your data catalog.
Typical tasks you perform while harvesting Object Storage files as logical data entities:
Understanding Logical Data Entities
Consider the following set of files:
myserv/20191205_yny_myIOTSensor.json
myserv/20191105_yny_myIOTSensor.json
myserv/20191005_yny_myIOTSensor.json
myserv/20190905_yny_myIOTSensor.json
myserv/20191005_hyd_my2ndIOTSensor.json
myserv/20190905_hyd_my2ndIOTSensor.json
myserv/20191005_bom_my3rdIOTSensor.json
myserv/20190905_bom_my3rdIOTSensor.json
myserv/somerandomfile_2019AUG05.json
If you harvest these files in your Oracle Object Storage data source without creating filename patterns , Data Catalog creates nine individual data entities in your data catalog. Imagine this situation with hundreds of files in your data source resulting in hundreds of data entities in your data catalog.
myserv/20191205_yny_myIOTSensor.json
myserv/20191105_yny_myIOTSensor.json
myserv/20191005_yny_myIOTSensor.json
myserv/20190905_yny_myIOTSensor.json
myserv/20191005_hyd_my2ndIOTSensor.json
myserv/20190905_hyd_my2ndIOTSensor.json
myserv/20191005_bom_my3rdIOTSensor.json
myserv/20190905_bom_my3rdIOTSensor.json
myserv/somerandomfile_2019AUG05.json
Understanding Expressions
In Data Catalog, a filename pattern is defined using expressions.
An expression can have one or more components that you separate using a delimiter. Each component specifies a matching rule for the pattern. Filename patterns are created using Java regular expressions. You specify the regular expression that should be used to group your files into required logical data entities.
You can specify qualifiers that are used when parsing the expression. You can use the following qualifiers:
bucketName
: Use this qualifier to specify that the bucket name should be derived from the path that matches the given expression. ThebucketName
qualifier is used only once in the expression and always as the first component of the expression. ThebucketName
qualifier value can be a static text or an expression.logicalEntity
: Use this qualifier to specify that the logical data entity name should be derived from the path that matches the given expression. You can uselogicalEntity
multiple times in an expression. ThelogicalEntity
qualifier values can consist of static text or expressions.
Consider the following filenames:
bling_metering/1970120520_yny_hourly_region_res_delayed.json
bling_metering/1973110523_yny_hourly_region_res_delayed.json
bling_metering/1988101605_hyd_daily_region_res_delayed.json
bling_metering/1991042302_yny_hourly_region_res_delayed.json
bling_metering/2019073019_zrh_daily_region_res_delayed.json
bling_metering/2019073020_zrh_monthly_region_res_delayed.json
bling_metering/some_random_file_123.json
To derive logical data entities based on frequency (hourly, daily, monthly) mentioned in the filename, you can use the following pattern expression:
{bucketName:bling_metering}/[0-9]*_[a-z]*_{logicalEntity:[a-z]*}_.*.json
This expression uses the bucketName
and logicalEntity
qualifiers. In this example, [0-9]*
matches any number;
[a-z]*
matches any lowercase alphabet; and .*
matches any character. The expression results in the following logical data
entities:
- bling_metering_monthly
bling_metering/2019073020_zrh_monthly_region_res_delayed.json
- bling_metering_hourly
bling_metering/1970120520_yny_hourly_region_res_delayed.json bling_metering/1973110523_yny_hourly_region_res_delayed.json bling_metering/1991042302_yny_hourly_region_res_delayed.json
- bling_metering_daily
bling_metering/1988101605_hyd_daily_region_res_delayed.json bling_metering/2019073019_zrh_daily_region_res_delayed.json
Unmatched
bling_metering/some_random_file_123.json
To derive logical data entities based on regions (yny, hyd, zrh) mentioned in the filename, you can use either of the following pattern expression:
{bucketName:bling_metering}/[0-9]*_{logicalEntity:yny|hyd|zrh}_[a-z]*_region_res_delayed.json
{bucketName:bling_metering}/[0-9]*_{logicalEntity:[a-z]*}_[a-z]*_.*.json
This expression results in the following logical data entities:
- bling_metering_zrh
bling_metering/2019073020_zrh_monthly_region_res_delayed.json bling_metering/2019073019_zrh_daily_region_res_delayed.json
bling_metering_yny
bling_metering/1970120520_yny_hourly_region_res_delayed.json bling_metering/1973110523_yny_hourly_region_res_delayed.json bling_metering/1991042302_yny_hourly_region_res_delayed.json
- bling_metering_hyd
bling_metering/1988101605_hyd_daily_region_res_delayed.json
Unmatched
bling_metering/some_random_file_123.json
To derive logical data entities based on regions and frequency (hourly, daily, monthly) mentioned in the filename, you can use the following pattern expression:
{bucketName:bling_metering}/[0-9]*_{logicalEntity:[a-z]*}_{logicalEntity:[a-z]*}_region_res_delayed.json
The above expression uses the bucketName
and two
logicalEntity
qualifiers. The expression results in the following
logical data entities:
- bling_metering_zrh_monthly
bling_metering/2019073020_zrh_monthly_region_res_delayed.json
- bling_metering_hyd_daily
bling_metering/1988101605_hyd_daily_region_res_delayed.json
- bling_metering_zrh_daily
bling_metering/2019073019_zrh_daily_region_res_delayed.json
- bling_metering_yny_hourly
bling_metering/1970120520_yny_hourly_region_res_delayed.json bling_metering/1973110523_yny_hourly_region_res_delayed.json bling_metering/1991042302_yny_hourly_region_res_delayed.json
Unmatched
bling_metering/some_random_file_123.json
If no logicalEntity
qualifier is specified, the filename pattern name is
used as the logical data entity name. For example, consider the following expression for
the filename pattern bling pattern
:
{bucketName:bling_metering}/[0-9]*_[a-z]*_[a-z]*_.*.json
The above expression uses the bucketName
qualifier, but no
logicalEntity
qualifier. The expression results in the following
logical data entities:
- bling
pattern
bling_metering/2019073020_zrh_monthly_region_res_delayed.json bling_metering/1970120520_yny_hourly_region_res_delayed.json bling_metering/1973110523_yny_hourly_region_res_delayed.json bling_metering/1991042302_yny_hourly_region_res_delayed.json bling_metering/1988101605_hyd_daily_region_res_delayed.json bling_metering/2019073019_zrh_daily_region_res_delayed.json
Unmatched
bling_metering/some_random_file_123.json
When you test this expression with no
logicalEntity
qualifier, in
resulting logical data entity the expression is shown as the logical entity name. But on
harvesting, the name of the filename pattern is used as the logical data entity
name.Creating a Filename Pattern
Filename patterns let you group multiple Object Storage files as logical data entities in data catalog. If an Object Storage file is matched with multiple filename patterns, it can be part of multiple logical data entities.
- On your data catalog instance Home tab, click Manage Filename Patterns from Quick Actions to access the Filename Patterns tab.
- Click Create Filename Patterns.
- In the Create Filename Pattern panel, enter a Name for your filename pattern.
- Optionally, enter a Description for the filename pattern.
- For Expression, enter a regular expression that corresponds to your Oracle Object Storage files. The files matching this regular expression are grouped into logical data entities in the data catalog.
- To test your expression, enter list of filenames in Test filenames and click Test Expression. The test results show the logical data entities that are derived using the specified expression. Any unmatched files are created as separate file type data entities.
- Modify the filename pattern expression and test it until you get the required logical data entities for your file set.
- Click Create.
Viewing All Filename Patterns
On your data catalog instance Home tab, click Filename Patterns to access the Filename Patterns tab. You can also click + from tabs and select Filename Patterns.
All filename patterns created in your data catalog are listed in the Filename Patterns tab. Use the available filters to refine the list to display filename patterns that were updated before or after a specified date.
Additionally, from the Filename Patterns tab, you can create, view, edit, test, or delete a filename pattern.
Viewing a Filename Pattern's Details
- On your data catalog instance Home tab, click Filename Patterns to access the Filename Patterns tab.
- Click the filename pattern name. Alternatively, click the Actions icon (three dots) for the filename pattern and select View Details. The filename pattern details tab displays.
Additionally, you can edit, test, or delete the filename pattern.
Assigning Filename Patterns to Data Assets
- On the Home tab, click Data Assets to access the Data Assets tab.
- In the Data Assets list, select the Object Storage data asset for which you want to assign filename patterns.
- In the Summary tab on the data asset details tab, under Filename Patterns, click Assign Filename Patterns.
- From the Assign Filename Patterns panel, select the filename patterns that you want to assign to this data asset. You can use the filter box to filter the filename patterns by name. You can also deselect already assigned filename patterns to unassign them from this data asset.
- Click Assign.
When you assign a new filename pattern to a data asset, the status of any harvested logical data entities is set to Inactive. You need to harvest the data asset again to derive valid logical data entities again.
Viewing Harvested Logical Data Entities
Editing a Filename Pattern
Testing a Filename Pattern
- On your data catalog instance Home tab, click Filename Patterns to access the Filename Patterns tab.
- In the Filename Patterns list, click the Actions icon (three dots) for the filename pattern you want to test and select Test. The Test Filename Pattern panel opens. You can also test a filename pattern expression by using the Test button on the filename pattern's details page.
- Enter the filenames you want to test against the expression in Test filenames.
- Click Test Expression.
- Verify the resulting logical data entities. If the results are not what you require, edit the filename pattern and enter a different expression.
- Click Close.
Deleting a Filename Pattern
- On your data catalog instance Home tab, click Filename Patterns to access the Filename Patterns tab.
- In the Filename Patterns list, click the Actions icon (three dots) for the filename pattern you want to delete and select Delete. The Delete Filename Pattern panel opens. You can also delete a filename pattern by using the Delete button on the filename pattern's details page.
- Click Delete to confirm delete.
Unassigning a Filename Pattern
When you unassign a filename pattern from a data asset, the status of any harvested logical data entities is set to Inactive. You need to harvest the data asset again to derive valid logical data entities again.
- On the Home tab, click Data Assets to access the Data Assets tab.
- In the Data Assets list, select the Object Storage data asset from which you want to unassign filename patterns.
- In the Summary tab on the data asset details tab, under Filename Patterns, click the actions menu for the filename pattern you want to unassign from the data asset. Then, click Unassign Filename Patterns.
Understanding Logical Data Entity Statuses
When a logical data entity is created during harvest, the status of the logical data
entity is Active
.
Logical data entities are a created from the actual names of the files in the Object Storage buckets, the filename pattern expression, and the filename patterns assigned to the data assets. When there is a change to the Object Storage files, filename pattern expression, or the associated filename patterns, the status of the logical data entity can change.
- Object Storage: If your Object Storage buckets have more or less files since the last harvest, your data catalog reflects the changes on reharvest. New files are added to the logical data entity and deleted files are removed from the logical data entity.
- Filename pattern expression: When you modify an expression in a filename pattern, there is no certainty that the logical data entities created on reharvest are same as the logical data entities created during the previous harvest. To indicate this state of uncertainty, Data Catalog sets the status of the logical data entities harvested using the expression to Inactive. During reharvest, if same logical data entities are derived, Data Catalog sets their status to Active again. Logical data entities from the previous harvest that were not derived during reharvest remain Inactive. You need to explicitly delete them as needed.
- Filename patterns: Consider you assign a new filename pattern to a data asset or unassign any existing filename patterns from a data asset. There is no certainty that the logical data entities created on reharvest are same as the logical data entities created during the previous harvest. To indicate this state of uncertainty, Data Catalog sets the status of the logical data entities harvested using the filename patterns to Inactive. During reharvest, if same logical data entities are derived, Data Catalog sets their status to Active again. Logical data entities from the previous harvest that were not derived during reharvest remain Inactive. You need to explicitly delete them as needed.
Understanding Limitations
- If an Object Storage bucket contains archived files, then you must select the Include unrecognized option to create a logical data entity with only the archived files.
- If you select the Incremental harvest option while harvesting data assets
with associated filename patterns, the following changes in your Object Storage are
reflected in your data catalog:
- When new files are added to your Object Storage bucket that match any existing logical data entity, they are added to the logical data entity.
- When new files are added to your Object Storage bucket that derive new logical data entities, the new logical data entities are created and the files are added to the new logical data entity.
- When files are deleted from your Object Storage bucket that were associated to a logical data entity, they are removed from the logical data entity.
- When all the files associated to a logical data entity are deleted from your Object Storage bucket, the logical data entity itself is removed from your data catalog.
Note
Incremental harvest for logical data entities doesn't track the changes made to your filename pattern expressions, or the filename patterns associated with the data asset. You must perform a full harvest to be in sync with all the updates.