1. Oracle GoldenGate for Distributed Applications and Analytics
  2. Overview
  3. Dependency Downloader

1.3 Dependency Downloader

Utility scripts are located in the {GGforDAA install}/DependencyDownloader directory to download client dependency jars for the various supported Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) integrations.

Topics:

Parent topic: Overview

1.3.1 Dependency Downloader Setup

To complete the Dependency Downloader setup:
  1. To verify that Java is installed, execute the following from the command line: java -version.

    Note:

    The Dependency Downloader utility scripts require Java to run. Ensure that Oracle Java is downloaded and is available in the PATH on the machine where the scripts are installed.
  2. Configure the proxy settings in the following script: {GGforDAA install}/DependencyDownloader/config_proxy.sh. Following are the 2 entries in this file:
    • #export PROXY_SERVER_HOST=www-proxy-hqdc.us.oracle.com
    • #export PROXY_SERVER_PORT=80
    To configure the proxy settings:
    1. Uncomment the configuration settings. (remove the # at beginning of the lines).
    2. Change the host name and port number to your correct proxy server settings.

    Note:

    Most companies maintain a private network which in turn has a network firewall to shield it from the public Internet. Additionally, most companies maintain a forwarding proxy server which serves as a gateway between the customer’s private network and the public Internet. The Dependency Downloader utilities must access Maven repositories, which are available on the Internet. Therefore, you need to supply configuration for HTTP proxy settings in order to download dependency libraries. Proxy servers are identified by host name and port. If you do not know whether your company employs a proxy server or the settings, then contact your IT or network administrators.

The Dependency Downloader uses Bash scripts in order to invoke Maven and download dependencies. The Bash shell is not supported natively from the Windows Command Prompt. You can run the Dependency Downloader scripts on Windows, but it requires the installation of a Unix emulator. A Unix emulator provides a Unix style command line on Windows and supports various flavors of the Unix shells including Bash. An option for Unix emulators is Cygwin, which is available free of charge. After Cygwin is installed, the setup process is the same. Setup and running of the scripts should be done through the Cygwin64 Terminal. See https://www.cygwin.com/.

Parent topic: Dependency Downloader

1.3.2 Running the Dependency Downloader Scripts

To run the dependency downloader scripts:
  1. Use a Unix terminal interface navigate to the following directory: {GGforDAA install}/DependencyDownloader.
  2. Execute the following to run the scripts: ./{the dependency script} {version of the dependencies to download}

    For example: ./aws.sh 1.11.893

    Dependency libraries get downloaded to the following directory:

    {GGforDAA install}/DependencyDownloader/dependencies/{the dependency name}_{the_dependency_version}.

    For example: {GGforDAA install}/DependencyDownloader/dependencies/aws_sdk_1.11.893.

Ensure that the version string exactly matches the version string of the dependency which is being downloaded. If a dependency version doesn't exist in the public Maven repository,then it is not possible to download the dependency and running the script results in an error. Most public Maven repositories support a web-based GUI whereby you can browse the supported versions of various dependencies. The exception is the Confluent Maven repository does not support a web-based GUI. This makes downloading dependencies challenging, because the version string is not independently verifiable through a web interface.

After the dependencies are successfully downloaded, you must configure the gg.classpath variable in the Java Adapter properties file to include the dependencies for the corresponding replicat process.

Note:

Best Practices
  1. Whenever possible, use the exact version of the client libraries to the server/application integration to which you are connecting.
  2. Prior to running the Dependency Downloader scripts, independently verify that the version string exists in the repository through the web GUI.

Parent topic: Dependency Downloader

1.3.3 Dependency Downloader Scripts

Table 1-1 Relevant Handlers/Capture

Client Script Description Relevant Handlers/Capture Versions Supported Dependency Link

Amazon Web Services SDK

 aws.sh This script downloads the Amazon Web Services (AWS) SDK, which provides client libraries for connectivity to the AWS cloud. Kinesis Handler

S3 Event Handler

1.12.x https://search.maven.org/artifact/com.amazonaws/aws-java-sdk
Google BigQuery bigquery.sh This script downloads the required client libraries for Google BigQuery. BigQuery Handler 2.x https://search.maven.org/artifact/com.google.cloud/google-cloud-bigquery
Cassandra DSE (Datastax Enterprise) Client cassandra_dse.sh This script downloads the Cassandra DSE client. Cassandra DSE is the for-purchase version of Cassandra available from Datastax. Cassandra Handler 2.0.0 and higher https://search.maven.org/artifact/com.datastax.dse/dse-java-driver-core
Apache Cassandra Client cassandra.sh This script downloads the Apache Cassandra client. Cassandra Handler 4.0.0 and higher https://search.maven.org/artifact/com.datastax.oss/java-driver-core
Cassandra Capture 3x Client cassandra_capture_3x.sh This script downloads all the client libraries needed for Capture from Cassandra 3.x versions. Cassandra Capture 3x 3.3.1 (used by default) https://mvnrepository.com/artifact/com.datastax.cassandra/cassandra-driver-core/3.3.1
Cassandra Capture 4x Client cassandra_capture_4x.sh This script downloads all the client libraries needed for Capture from Cassandra 4.x versions. Cassandra Capture 4x 4.14.1 (used by default) https://mvnrepository.com/artifact/com.datastax.oss/java-driver-core/4.14.1
Cassandra Capture DSE Client cassandra_capture_dse.sh This script downloads all the client libraries needed for Capture from DSE Cassandra 6.x versions. Cassandra Capture DSE 4.14.1 (used by default) https://mvnrepository.com/artifact/com.datastax.oss/java-driver-core/4.14.1
Elasticsearch Java Client elasticsearch_java.sh This script downloads the Elasticsearch Java Client. Elasticsearch Handler 7.x and 8.x https://search.maven.org/artifact/co.elastic.clients/elasticsearch-java
Hadoop Azure Client from Cloudera hadoop_azure_cloudera.sh This script downloads the Hadoop Azure client libraries provided by Cloudera. The Hadoop Azure client libraries cannot be loaded along with the Hadoop client because in Cloudera, the version numbers between the two components do not line up perfectly.
  • HDFS Handler
  • HDFS Event Handler
  • ORC Event Handler
  • Parquet Event Handler
 3.x https://repository.cloudera.com/service/rest/repository/browse/cloudera-repos/org/apache/hadoop/hadoop-azure/
Hadoop Client from Cloudera hadoop_cloudera.sh This script downloads the Hadoop client libraries provided by Cloudera.
  • HDFS Handler
  • HDFS Event Handler
  • ORC Event Handler
  • Parquet Event Handler
 3.x https://repository.cloudera.com/service/rest/repository/browse/cloudera-repos/org/apache/hadoop/hadoop-client/
Hadoop Client from Hortonworks hadoop_hortonworks.sh The Hadoop client including the libraries for connectivity to Azure Data Lake available from Hortonworks.
  • HDFS Handler
  • HDFS Event Handler
  • ORC Event Handler
  • Parquet Event Handler
 3.x https://repo.hortonworks.com/service/rest/repository/browse/public/org/apache/hadoop/hadoop-client/
Apache Hadoop Client Plus Required Libraries for Azure Connectivity hadoop.sh The Hadoop client including the libraries for connectivity to Azure Data Lake.
  • HDFS Handler
  • HDFS Event Handler
  • ORC Event Handler
  • Parquet Event Handler
 3.x https://search.maven.org/artifact/org.apache.hadoop/hadoop-azure
HBase Client Provided by Cloudera hbase_cloudera.sh The HBase client libraries provided by Cloudera. HBase Handler 2.x https://repository.cloudera.com/service/rest/repository/browse/cloudera-repos/org/apache/hbase/hbase-client/
HBase Client Provided by Hortonworks hbase_hortonworks.sh The HBase client libraries provided by Hortonworks. HBase Handler 2.x https://repo.hortonworks.com/service/rest/repository/browse/public/org/apache/hbase/hbase-client/
Apache HBase Client hbase.sh

The HBase client.

 HBase Handler 2.x https://search.maven.org/artifact/org.apache.hbase/hbase-client
Apache Kafka Client plus Kafka Connect Framework and JSON Converter from Cloudera kafka_cloudera.sh The Kafka Client plus libraries for the Kafka Connect framework and the Kafka Connect JSON Converter provided by Cloudera.
  • Kafka Handler
  • Kafka Connect Handler
  • Kafka Capture
0.9.x to current https://repository.cloudera.com/service/rest/repository/browse/cloudera-repos/org/apache/kafka/kafka-clients/
Apache Kafka Client plus Kafka Connect Framework and JSON Converter from Hortonworks kafka_hortonworks.sh The Kafka Client plus libraries for the Kafka Connect framework and the Kafka Connect JSON Converter provided by Hortonworks.
  • Kafka Handler
  • Kafka Connect Handler
  • Kafka Capture
0.9.x to current https://repo.hortonworks.com/service/rest/repository/browse/public/org/apache/kafka/kafka-clients/
Apache Kafka Client plus Kafka Connect Framework and JSON Converter kafka.sh The Kafka Client plus libraries for the Kafka Connect framework and the Kafka Connect JSON Converter.
  • Kafka Handler
  • Kafka Connect Handler
  • Kafka Capture
0.9.x to current https://search.maven.org/artifact/org.apache.kafka/kafka-clients
Confluent Kafka Client plus Kafka Connect Framework and JSON and Avro Converters kafka_confluent.sh The Kafka Client plus libraries for the Kafka Connect framework and the Kafka Connect JSON Converter and the Kafka Connect Avro Converter available from Confluent.
  • Kafka Handler
  • Kafka Connect Handler
  • Kafka Capture
Confluent platform 4.1.0 and higher. See https://packages.confluent.io/maven/io/confluent/kafka-connect-avro-converter/
MapR Kafka Client kafka_mapr.sh The MapR Kafka Client libraries. Kafka Handler 0.x, 1.x, and 2.x https://repository.mapr.com/nexus/content/groups/mapr-public/org/apache/kafka/kafka-clients/
Confluent Kafka Client plus Kafka Connect Framework and Protobuf Converter kafka_confluent_protobuf.sh The Kafka Client plus libraries for the Kafka Connect framework and the Kafka Connect Protobuf converter available from Confluent.
  • Kafka Handler
  • Kafka Connect Handler
 Confluent 5.x and higher See https://packages.confluent.io/maven/io/confluent/kafka-connect-protobuf-converter/
MongoDB Client mongodb.sh The MongoDB client libraries. MongoDB Handler 5.x https://mvnrepository.com/artifact/org.mongodb/mongodb-driver-legacy
Oracle NoSQL SDK Client oracle_nosql_sdk.sh The Oracle NoSQL client libraries. Oracle NoSQL Handler 5.x https://search.maven.org/artifact/com.oracle.nosql.sdk/nosqldriver
Oracle OCI Client oracle_oci.sh The Oracle OCI client libraries. Oracle OCI Event Handler 3.x https://search.maven.org/artifact/com.oracle.oci.sdk/oci-java-sdk-objectstorage
Apache ORC (Optimized Row Columnar) Client orc.sh The Apache ORC client libraries. ORC is built on top of the Hadoop client so the ORC Event Handler needs the Hadoop client in order to run. The Hadoop client needs to be downloaded separately. ORC Event Handler 1.x https://search.maven.org/artifact/org.apache.orc/orc-core
Apache Parquet Client parquet.sh The Apache Parquet client libraries. Parquet is built on top of the Hadoop client, so the Parquet Event Handler needs the Hadoop client in order to run. The Hadoop client needs to be downloaded separately. Parquet Event Handler 1.x https://search.maven.org/artifact/org.apache.parquet/parquet-hadoop
Apache Velocity velocity.sh The Velocity libraries were removed from the Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) installation starting from the 21.1 release. This script downloads the libraries required for formatting using Velocity. Velocity Formatter 1.x https://search.maven.org/artifact/org.apache.velocity/velocity
Google Cloud Storage Java SDK gcs.sh This script downloads the required client libraries for Google Cloud Storage. GCS Event Handler 2.x https://search.maven.org/artifact/com.google.cloud/google-cloud-storage
MongoDB Capture mongodb_capture.sh This script downloads the required client libraries for MongoDB capture. MongoDB Capture 5.x https://search.maven.org/artifact/org.mongodb/mongodb-driver-reactivestreams
Synapse JDBC Driver synapse.sh This script downloads the Synapse JDBC driver. Additionally, the Hadoop client is also required to stage data to Azure Data Lake. Synapse Stage and Merge 12.6.1jre8 https://mvnrepository.com/artifact/com.microsoft.sqlserver/mssql-jdbc/12.6.1.jre8
Snowflake JDBC Driver snowflake.sh This script downloads the Snowflake JDBC driver. Other client libraries are likely required for staging the data to AWS or Azure cloud. Snowflake Stage and Merge 3.15.1 https://search.maven.org/artifact/net.snowflake/snowflake-jdbc/3.15.1/jar
Jedis client for Redis redis.sh This script downloads Jedis which is a Redis client. Redis Handler 4.x https://search.maven.org/artifact/redis.clients/jedis
Google Pub/Sub Client googlepubsub.sh This script downloads the Java client for Google Pub/Sub Messaging. Google Pub/Sub Handler 1.x https://search.maven.org/artifact/com.google.cloud/google-cloud-pubsub
Databricks JDBC Driver databricks.sh This script downloads the Databricks JDBC driver. Databricks Stage and Merge 2.6.36 https://mvnrepository.com/artifact/com.databricks/databricks-jdbc/2.6.36
Azure Blob Storage Client azure_blob_storage.sh This script downloads the Microsoft Azure Blob Storage Client. Azure Blob Storage Event Handler

Data Warehouse Stage and Merge implementations can use this as well to upload to Azure Data Lake. 		12.25.3 https://search.maven.org/artifact/com.azure/azure-storage-blob
Snowflake Streaming snowflakestreaming.sh This script can be downloaded using the Dependency Downloader script. NA NA The script can be found in following location :<OGGDIR>/DependencyDownloader/snowflakestreaming.sh

Parent topic: Dependency Downloader