27 Manage with Master Catalog
This chapter helps you use and understand the Master Catalog, standard and external catalogs, schemas, tables, and volumes in Oracle AI Data Platform.
Master Catalog
In Oracle AI Data Platform, the Master Catalog is the top level entity that enables you to manage your data and metadata by providing a centralized view.
The Master Catalog is a container for both standard and external catalogs. You create catalogs with their data assets in Oracle Autonomous AI Lakehouse, OCI Object Storage, and Kafka. The Master Catalog enables you to enforce permissions on its child objects.
Standard and external catalogs have different functions and use cases:
- Standard catalog: A standard catalog is a logical container for schemas (databases). Users can create tables, views, and volumes in a schema. The standard catalog manages the lifecycle of metadata of all child objects.
- External catalog: An external catalog is backed by external data sources such as Oracle Autonomous AI Lakehouse or Kafka. With an external catalog, the metadata is synced from the external source and users can query the data in the external source using the 3-part name, such as catalog_name.schema_name.table_name. The metadata lifecycle is managed by the external source and the Master Catalog keeps a copy of the metadata.
To create, modify, and delete standard and external catalogs, you need certain permissions. See Master Catalog Permissions.
Use Cases for Master Catalog
You can leverage the Master Catalog to help with data preparation and analysis, storing unstructured data, and more.
Query and Analyze Data Using SQL Syntax
Create managed or external tables in a standard catalog to query and analyze data using familiar SQL-like syntax, making it easier to explore and understand the data stored in Oracle AI Data Platform.
Prepare Data for Machine Learning
Leverage the structured format of data stored in managed external tables for preparing data for machine learning models, making it easier to clean, transform, and feature engineer data. This facilitates efficient data access and processing for feature engineering and model training.
Query Historical Data with Time Travel
Open table formats support schema evolution. The structure of the data can change over time without rewriting the entire dataset. Version these tables and run time travel queries to query historical versions of data. This facilitates retrospective analysis and data recovery.
Support ACID Transactions
Open table formats support full Create, Read, Update, and Delete (CRUD) operations, ensuring data consistency and enabling data updates. Use tables to store and manage transactional data so applications can track changes to the data.
Efficiently Read and Write Data
Partition tables in AI Data Platform to enable efficient data access and processing, especially for large datasets.
Store and Process Unstructured Data
Create managed or external volumes to store unstructured data so that they can be processed using Apache Spark.
Cross-Tenancy External Tables and Volumes
Cross-tenancy external tables and volumes allow you to securely access and query data stored in disparate tenancies without the need for complex ETL pipelines or manual data movement.
Oracle AI Data Platform enables you to create cross-tenancy external tables and volumes, a powerful capability designed to eliminate data silos and streamline collaboration.
- Zero Data Duplication: You access live data where it resides, saving on storage costs and ensuring single source of truth integrity.
- Simplified Governance: You manage permissions across boundaries using Identity and Access Management (IAM) policies and AI Data Platform access controls.
Cross-Tenancy Access Requirements
Setting up cross-tenancy access for external tables and volumes requires specific IAM policies configured in a provider tenancy and a consumer tenancy.
In the provider tenancy, you need to create an IAM Dynamic Group in the Oracle Cloud Infrastructure (OCI) Console that includes your specific AI Data Platform resource as a member. For more information, see Managing Dynamic Groups.
- Define resources in IAM for consumer tenancy, user group and dynamic groups
- Write an admit IAM policy for the consumer tenancy resources
define tenancy <consumer_tenancy_name1> as <consumer tenancy OCID>
define group <group_name1> as <consumer user group>
define dynamic-group <dynamic_group_name1> as <consumer dynamic group OCID>
admit dynamic-group <dynamic_group_name1> of tenancy <consumer_tenancy_name1> to manage object-family in tenancy
admit dynamic-group <dynamic_group_name1> of tenancy <consumer_tenancy_name1> to { OBJECTSTORAGE_NAMESPACE_READ } in tenancy
admit group <group_name1> of tenancy <consumer_tenancy_name1> to manage object-family in tenancy- Define the resource in IAM for provider tenancy
- Write an endorse IAM policy for the local consumer tenancy resources
define tenancy <provider_tenancy_name1> as <provider tenancy OCID>
endorse dynamic-group <dynamic_group_name> to manage object-family in tenancy <provider_tenancy_name1>
endorse dynamic-group <dynamic_group_name> to { OBJECTSTORAGE_NAMESPACE_READ } in tenancy <provider_tenancy_name1>
endorse group <group_name> to manage object-family in tenancy <provider_tenancy_name1>Once both the provider and consumer tenancy IAM policies are configured, you can create cross-tenancy external tables and volumes using SQL grammar. For more information, see SQL Grammar.
Example: Create a Cross-Tenancy Table with SQL
CREATE EXTERNAL TABLE [IF NOT EXISTS] <catalog_name>.<schema-name>.<table-name>
[ ( <column1-name><column1-type> [comment <column1-comment>], ... ) ]
USING [HIVE|DELTA, CSV, TXT, ORC, JDBC, PARQUET, etc.]
LOCATION 'oci://my-bucket@mytenancynamespace/my-folder/'
[TBLPROPERTIES ( DESCRIPTION = 'some-description', '<property-name>'='<property-value>'[, ...]) ]Limitation
AI Data Platform doesn't support creating cross-tenancy external tables or external volumes in the UI.