10.4.3 Databricks

Parent topic: Lakehouse and Open Table Formats

10.4.3.1 About Databricks Connections

A Databricks connection stores the information that Oracle GoldenGate DAA uses to connect to a Databricks SQL Warehouse or Databricks workspace. After the connection is created and validated, you can reuse it when configuring Replicat processes that deliver transactional data to Databricks.

The connection stores the Databricks endpoint, authentication information, and optional runtime settings independently of the Replicat configuration. This enables multiple Replicat processes to reuse the same validated connection.

Parent topic: Databricks

10.4.3.2 Before You Begin

Before creating a Databricks connection, ensure that you have the following:

  • An Oracle GoldenGate for Distributed Applications and Analytics deployment.
  • Network connectivity between the Oracle GoldenGate for Distributed Applications and Analytics deployment and the Databricks SQL Warehouse or workspace.
  • A valid Databricks JDBC connection URL.
  • One of the following authentication methods:
    • Personal Access Token (PAT)
    • OAuth Machine-to-Machine (OAuth M2M)
  • An Azure Data Lake Storage (ADLS), Google Cloud Storage (GCS), or Amazon S3 connection. Databricks Stage and Merge Replicat requires one of these cloud storage connections to stage data before loading it into Databricks. For more information, see: Create and Validate an Azure Data Lake Storage Connection, Create and Validate a Google Cloud Storage Connection, and Create and .Validate an Amazon S3 Connection
  • A Databricks storage credential, if required by your deployment.
  • The Databricks JDBC driver and any required client libraries.

To download the required dependencies, use the Oracle GoldenGate for Distributed Applications and Analytics Dependency Downloader: 

{GGforDAA install}/DependencyDownloader/databricks.sh 2.6.36

Download the supported Databricks JDBC driver from:

https://mvnrepository.com/artifact/com.databricks/databricks-jdbc/2.6.36

After downloading the dependencies, configure the gg.classpath property in the Java Adapter properties file to reference the downloaded libraries before starting the Replicat process.

  • Configure common runtime settings, such as the runtime classpath, if additional libraries are required.
  • Proxy information if the Oracle GoldenGate for Distributed Applications and Analytics deployment accesses Databricks through an HTTP proxy.

Parent topic: Databricks

10.4.3.3 Create and Validate a Databricks Connection

Use the Add Connection page to create a Databricks connection.

Procedure

  1. In the navigation menu, select DB Connections.
  2. Click Add Connection.
  3. In Connection Alias, enter a unique name for the connection.
  4. From Technology Type, select Databricks.
  5. In the Technology Settings section, provide the Databricks connection information.

    Configure the following technology settings:

    • Connection URL – Specifies the JDBC connection URL for the Databricks SQL Warehouse or workspace. This field is required.
    • Authentication Type – Specifies the authentication method. Supported values are Personal Access Token and OAuth M2M.
    • Storage Credential Name – Specifies the Databricks storage credential used by the deployment, when required.
  6. Configure the authentication settings based on the selected authentication type.

    Configure the following settings when Authentication Type is Personal Access Token:

    • Username – Specifies the Databricks user name associated with the personal access token. This field is required.
    • Password – Specifies the Databricks personal access token. This field is treated as sensitive information and is displayed as a password field.
  7. Configure the following settings when Authentication Type is OAuth M2M:
    • Client ID – Specifies the OAuth client ID registered with Databricks. This field is required.
    • Client Secret – Specifies the OAuth client secret associated with the client ID. This field is treated as sensitive information and is displayed as a password field.
  8. Configure Common Settings, if additional runtime configuration is required.

    Configure the following common settings:

    Classpath: (Required) Specify the full path to client libraries are required by the replicat type. You can use Dependency Downloader to download client libraries.

    Note:

    Download the supported Databricks JDBC driver and required client libraries by using the Dependency Downloader for Databricks. Configure the runtime classpath to reference the downloaded libraries before starting the Replicat process.
  9. (Optional) Configure Proxy Settings if the Oracle GoldenGate for Distributed Applications and Analytics deployment accesses Databricks through an HTTP proxy.

    Configure the following proxy settings:

    • Proxy URL – Specifies the URL or host name of the proxy server used to access Databricks.
    • Proxy Port – Specifies the port number of the proxy server.
    • Proxy Username – Specifies the user name for proxy authentication, if required.
    • Proxy Password – Specifies the password for proxy authentication. This field is treated as sensitive information.
    • Truststore – Specifies the Java truststore file that contains the certificate authority (CA) certificates used to validate the proxy server certificate, when required.
    • Truststore Password – Specifies the password associated with the truststore.
    • Keystore – Specifies the Java keystore file containing the client certificate and private key used for client authentication, when required.
    • Keystore Password – Specifies the password associated with the keystore.
    • SSL Key Password – Specifies the password protecting the private key stored in the keystore when it differs from the keystore password.

    Note:

    If the Oracle GoldenGate for Distributed Applications and Analytics deployment has direct network connectivity to Databricks, leave the proxy settings blank.
  10. Click Submit.
  11. On the DB Connections page, click DB Login to validate the connection.

    Note:

    A successful DB Login validates only the saved connection information. It does not validate the Replicat configuration or Databricks Handler configuration. After successful validation, the connection is available for selection during Replicat creation.

Parent topic: Databricks

10.4.3.4 Use a Databricks Connection in Replicat

After the Databricks connection has been successfully validated:

  1. Create or edit a Replicat.
  2. Select Databricks as the target technology.
  3. Select the validated Databricks connection.
  4. Select the storage connection to use for staging data. Supported storage connections include:
    • Azure Data Lake Storage (ADLS)
    • Google Cloud Storage (GCS)
    • Amazon S3
  5. Complete the remaining Replicat configuration.
  6. Create the Replicat.
  7. Configure any remaining Databricks-specific handler properties before starting the Replicat.

Note:

Databricks Stage and Merge Replicat requires a valid storage connection (Azure Data Lake Storage, Google Cloud Storage, or Amazon S3) for staging data before loading it into Databricks. Ensure that the required storage connection has been created and validated before configuring the Replicat. For storage connection setup, see, Create and Validate an Azure Data Lake Storage Connection, Create and Validate a Google Cloud Storage Connection, and Create and Validate an Amazon S3 Connection.

Related Topics

Parent topic: Databricks