External Catalogs

External catalogs are catalogs where the data is sourced from a location outside AI Data Platform.

External catalogs connect to sources like Autonomous Data Warehouse (ADW), Oracle Database, and Autonomous Transaction Processing to extract data into your AI Data Platform. External catalogs use the credentials provided during the external catalog creation for querying the external source. For more information on data sources, see Internal Sources.

For example, if you create an external catalog for an ADW instance where the ADW user credentials used have access to schema1 but not schema2, only schema1 appears in the external catalog. Users with permissions for the external catalog can only query the schema the ADW user has access to.

Note:

Writing to an ADW based external catalog using Spark's saveAsTable() API is only supported in overwrite mode and the table should already exist in ADW external source.

Limitations

External catalogs support Query and DML. DDL is not supported, even when the credentials used to create the external catalog has permissions to execute DDL statements.

Create an External Catalog

You can connect a catalog from your Intelligent Data Lake to a external source.

1. Click Create in the left navigation pane and select Catalog. You can also navigate to the Master catalog and click Create Catalog in Master Catalog.
2. Fill in the name and description fields.
3. Select External Catalog from the Catalog Type dropdown.
4. Select the external source type.
   • For Autonomous Data Warehouse, provide either a wallet file, or the instance configuration.
   • For Autonomous Transaction Processing, provide either a wallet file, or the instance configuration.
   • For Oracle Database, provide either a wallet file, or the instance configuration.
   • For Kafka, provide the bootstrap server. Separate multiple servers with a comma. (Coming soon)
5. Fill in the user name and password.
6. SSL is enabled by default. Clear the box to disable SSL.
7. Click Create.
   External catalogs that are extracting data from an external source display a spinning circle icon.
   

   

   
   

   You can also monitor progress from Job Runs.

Create an External Catalog for Private Networks

You can create an external catalog that accesses data sources in a private network.

1. Click Create in the left navigation pane and select Catalog. You can also navigate to the Master catalog and click Create Catalog in Master Catalog.
2. Fill in the name and description fields.
3. Select External Catalog from the Catalog Type dropdown.
4. Select the external source type.
   • For Autonomous Data Warehouse, provide either a wallet file, or the instance settings.
   • For Autonomous Transaction Processing, provide either a wallet file, or the instance configuration.
   • For Oracle Database, provide either a wallet file, or the instance configuration.
   • For Kafka, provide the bootstrap server. Separate multiple servers with a comma. (Coming soon)
5. Fill in the user name and password.
6. SSL is enabled by default. Clear the box to disable SSL.
7. Select Enable private network.
8. Select the workspace with the desired private network configuration.
   For information on setting up a workspace configured for private networks, see Create a Workspace with Private Network Access Enabled.
9. Click Create.
   External catalogs that are extracting data from an external source display a spinning circle icon.
   

   

   
   

   You can also monitor progress from Job Runs.

Refresh External Data Catalogs

You can trigger a refresh of all external catalogs to update their contents from the source.

1. On the Home page, click Master Catalog.
2. Select the external catalog you want to refresh.
3. Click Refresh.
   When you click refresh, workflows start in the background to extract and update metadata from external catalogs. Catalogs that are extracting data from an external source display a spinning circle icon.
   

   

   
   

   You can also monitor progress from Job Runs.

Edit an External Catalog Description

You can edit the description for external catalogs after creation if their contents or purpose has changed.

1. On the Home page, click Master catalog.
2. Next to your external catalog, click Actions and click Edit description.
   
   

   

   
   
3. Make your changes to the catalog description and click Save.

Rename an External Catalog

You can rename your external catalogs to provide a descriptive label when the contents or purpose of the catalog has changed.

You can't rename the default catalog.

1. On the Home page, click Master catalog.
2. Next to your catalog, click Actions and click Rename.
   
   

   

   
   
3. Make your changes to the catalog name and click Save.

Edit an External Catalog Configuration

You can edit the configuration of an external catalogs to update the required password.

1. On the Home page, click Master catalog.
2. Next to your catalog, click Actions and click Edit configuration.
3. Enter the new password for the external catalog and click Save.

Delete an External Catalog

You can delete external catalogs to remove all locally stored metadata.

Deleting an external catalog only deletes the locally stored metadata. Data in the data source is not impacted.

1. On the Home page, click Master catalog.
2. Next to your catalog, click Actions then click Delete.
   
   

   

   
   
3. Select Confirm deletion of the catalogs.
4. Click Delete.