Connect to Azure Data Lake Storage

Learn to create a connection to Azure Data Lake Storage to use as a target with OCI GoldenGate.

Before you begin

Before you create the connection, ensure that you review how OCI GoldenGate connects to your source and targets.

Create the connection

To create an Azure Data Lake Storage connection
  1. From the OCI GoldenGate Overview page, click Connections.
    You can also click Create Connection under the Get started section and skip to step 3.
  2. On the Connections page, click Create Connection.
  3. On the Create Connection page, complete the fields as follows:
    1. For Name, enter a name for the connection.
    2. (Optional) For Description, enter a description that helps you distinguish this connection from others.
    3. For Compartment, select the compartment in which to create the connection.
    4. From the Type dropdown, select Azure Data Lake Storage.
    5. For Account name, enter the Azure Cloud Storage Account Name.
    6. Enter an Endpoint.
    7. For Authentication type select from the following:
      • Shared key:
        • Storage Account Key: Select the Storage account key secret. If located in a different compartment, use the dropdown to change compartments. See Manage storage account access keys.
          Note

          • Secrets are credentials such as passwords, certificates, SSH keys, or authentication tokens that you use with OCI services. To create a secret, see Creating a secret. Ensure that you:
            • Select Manual secret generation.
            • Paste the credentials into Secret contents.
          • If you prefer not to use password secrets, ensure that you deselect Use secrets in vault in the Security section under Advanced Options, located at the bottom of this form.
      • Shared access signature:
        • SAS token: Select the SAS token secret. If located in a different compartment, use the dropdown to change compartments. See Create an account SAS.
          Note

          • Secrets are credentials such as passwords, certificates, SSH keys, or authentication tokens that you use with OCI services. To create a secret, see Creating a secret. Ensure that you:
            • Select Manual secret generation.
            • Paste the credentials into Secret contents.
          • If you prefer not to use password secrets, ensure that you deselect Use secrets in vault in the Security section under Advanced Options, located at the bottom of this form.
      • Azure Active Directory:
        Note

        Before you configure the Azure Active Directory authentication type, ensure that you register an application in Azure AD App Registrations and assign the appropriate roles, for example "Storage BLOB Data Owner". See Use the portal to create an Azure AD application and service principal that can access resources.
        • Azure tenant ID, located in Azure Active Directory/ App Registrations, select the application, and enter the Tenant ID.
        • Client ID, located in Azure Active Directory/ App Registrations and select the application.
        • Select the Client secret. If located in a different compartment, use the dropdown to change compartments.
          Note

          • Secrets are credentials such as passwords, certificates, SSH keys, or authentication tokens that you use with OCI services. To create a secret, see Creating a secret. Ensure that you:
            • Select Manual secret generation.
            • Paste the credentials into Secret contents.
          • If you prefer not to use password secrets, ensure that you deselect Use secrets in vault in the Security section under Advanced Options, located at the bottom of this form.
    8. Expand Show advanced options. You can configure the following options:
      • Security
        • Select Use Oracle-managed encryption key to leave all encryption key management to Oracle.
        • Select Use customer-managed encryption key to select a specific encryption key stored in your OCI Vault to encrypt your connection credentials.
      • Network connectivity
        • Shared endpoint, to share an endpoint with the assigned deployment. You must allow connectivity from the deployment's ingress IP.
        • Dedicated endpoint, for network traffic through a dedicated endpoint in the assigned subnet in your VCN. You must allow connectivity from this connection's ingress IPs.
          Note

          • If a dedicated connection remains unassigned for seven days, then the service converts it to a shared connection.
          • Learn more about Oracle GoldenGate connectivity.
      • Settings
        • Deselect Use vault secrets you prefer not to use password secrets for this connection.
      • Tags: Add tags to organize your resources.
  4. Click Create.
After the connection is created, it appears in the Connections list. Ensure that you assign the connection to a deployment to use it as a source or target in a replication.

Troubleshoot connection issues

Most Azure Data Lake Storage connection issues happen because of Azure Data Lake Storage private endpoint configurations.

The following are common connectivity related error messages that you could encounter in the Replicat report file:

  • =ERROR 2023-08-04 07:23:08.000008 [main] - Exception during initialisation of Azure blob service client for account[ociggtest].
    com.azure.storage.blob.models.BlobStorageException: Status code 400, "{"error":{"code":"InvalidUri","message":"The request URI is invalid.
  • =ERROR 2023-08-01 20:23:24.000861 [main] - The Event Handler Framework failed to initialise.
  • =ERROR 2023-08-04 08:13:30.000477 [main] - Exception during initialization of Azure blob service client for account[ociggtest].
    com.azure.storage.blob.models.BlobStorageException:Status code 403, "<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.

If you're using Azure Data Lake Storage private endpoints and having issues with connection and/or replication, ensure that you: