Part II Data Management

Oracle AI Data Platform enables organizations to effectively manage all their data and metadata across OCI Object Storage and other external sources like Autonomous Database, Kafka, etc. AI Data Platform provides centralized metadata across your data estate and enables enterprises to define unified access control for their structured, semi-structured and unstructured data.

AI Data Platform helps enterprises solve their data management use cases seamlessly. Various personas work in tandem in an organization to deliver value to the business leaders.

  • Data stewards need to discover data assets and entities to understand where data is located, how it is structured and used, thus effectively managing the data/metadata life cycle.
  • Data admins organize data in catalogs, schemas, tables, and volumes, to ensure efficient and secure storage, organization, and retrieval of data.
  • Data engineers and analysts need to share data with other analysts or business leaders to unlock the true value of data

Discover Data

All data assets in the AI Data Platform can be discovered seamlessly using:

  • Master Catalog Explorer
  • Catalog Explorer in Workspace while working with notebooks, sql/python files
  • SQL grammar like SHOW, LIST and DESCRIBE
  • APIs

Organize Data

You can organize the data in catalogs, schemas, tables, volumes:

  • Standard Catalog: A standard catalog is a logical container for schemas (databases), users can create tables, views and volumes in a schema. Standard catalogs manage the lifecycle of metadata of all child objects.
  • External Catalog: An external catalog is backed by external data sources like Autonomous Database, Kafka, etc. In case of external catalog, the only metadata is synched from the external source and users can query the data residing in an external source using the 3-part name like: catalog_name.schema.name.table_name. In case of external catalog the metadata lifecycle is managed by the external source and the Master Catalog keeps a copy of the metadata. External Catalog only harvests the metadata from the external source, the data is not copied into your AI Data Platform.

You can choose to let AI Data Platform manage the metadata lifecycle, by creating:

  • External tables, by defining a table, its schema, and referring to a location in OCI Object Storage, or
  • External volume, by defining a volume referring to a location in OCI Object Storage and then further storing files and folder in the volume

You can also choose to let AI Data Platform manage the data and metadata lifecycle, by creating:

  • Managed table and AI Data Platform manages the OCI Object Storage location in customer's tenancy
  • Managed volume and AI Data Platform manages the OCI Object Storage location in customer's tenancy so that users can store files and folders (semi-structured or unstructured data) in the volume

Data Sharing

Data Share in AI Data Platform enables users to share data assets with users in the organization as well as outside the organization. Data Sharing in AI Data Platform is built on top of open source Delta Share protocol and to ensure that Data is shared in a secure manner, you can enforce permissions on who can share and create recipients.

Auto Populate

The Auto Populate feature simplifies metadata management by automatically detecting and creating data entities in a selected standard catalog. This automates the process of manually creating huge number of tables by enabling users to create metadata extractors by pointing to data location in OCI Object Storage.

Topics: