7 Dependency Downloader
Utility scripts are located in the {OGGBD
install}/DependencyDownloader
directory to download client
dependency jars for the various supported Oracle GoldenGate for Big Data
integrations.
These scripts use Java and Apache Maven to download the dependency jars from the Maven Central Repository and other publicly available repositories (Hortonworks, Cloudera, Confluent).
Topics:
7.1 Dependency Downloader Setup
The Dependency Downloader uses Bash scripts in order to invoke Maven and download dependencies. The Bash shell is not supported natively from the Windows Command Prompt. You can run the Dependency Downloader scripts on Windows, but it requires the installation of a Unix emulator. A Unix emulator provides a Unix style command line on Windows and supports various flavors of the Unix shells including Bash. An option for Unix emulators is Cygwin, which is available free of charge. After Cygwin is installed, the setup process is the same. Setup and running of the scripts should be done through the Cygwin64 Terminal. See https://www.cygwin.com/.
Parent topic: Dependency Downloader
7.2 Running the Dependency Downloader Scripts
Ensure that the version string exactly matches the version string of the dependency which is being downloaded. If a dependency version doesn't exist in the public Maven repository,then it is not possible to download the dependency and running the script results in an error. Most public Maven repositories support a web-based GUI whereby you can browse the supported versions of various dependencies. The exception is the Confluent Maven repository does not support a web-based GUI. This makes downloading dependencies challenging, because the version string is not independently verifiable through a web interface.
After the dependencies are successfully downloaded, you must configure the
gg.classpath
variable in the Java Adapter properties file to include
the dependencies for the corresponding replicat process.
Note:
Best Practices- Whenever possible, use the exact version of the client libraries to the server/application integration to which you are connecting.
- Prior to running the Dependency Downloader scripts, independently verify that the version string exists in the repository through the web GUI.
Parent topic: Dependency Downloader
7.3 Dependency Downloader Scripts
Table 7-1 Dependency Downloader Scripts
Client | Script | Description | Relevant Handlers | Versions Supported | Dependency Link |
---|---|---|---|---|---|
Amazon Web Services SDK |
aws.sh |
This script downloads the Amazon Web Services (AWS) SDK, which provides client libraries for connectivity to the AWS cloud. | Kinesis Handler
S3 Event Handler |
1.11.x | https://search.maven.org/artifact/com.amazonaws/aws-java-sdk |
Google BigQuery | bigquery.sh |
This script downloads the required client libraries for Google BigQuery. | BigQuery Handler | 1.x | https://search.maven.org/artifact/com.google.cloud/google-cloud-bigquery |
Cassandra DSE (Datastax Enterprise) Client | cassandra_dse.sh |
This script downloads the Cassandra DSE client. Cassandra DSE is the for-purchase version of Cassandra available from Datastax. | Cassandra Handler | 2.0.0 and higher | https://search.maven.org/artifact/com.datastax.dse/dse-java-driver-core |
Apache Cassandra Client | cassandra.sh |
This script downloads the Apache Cassandra client. | Cassandra Handler | 4.0.0 and higher | https://search.maven.org/artifact/com.datastax.oss/java-driver-core |
Elasticsearch REST client | elasticsearch_rest.sh |
This script downloads the Elasticsearch High Level Rest Client. | Elasticsearch Handler | 7.x versions are currently supported | https://search.maven.org/artifact/org.elasticsearch.client/elasticsearch-rest-high-level-client |
Elasticsearch Transport Client | elasticsearch_transport.sh |
This script downloads the Elasticsearch Transport Client. | Elasticsearch Handler | 5.x, 6.x, and 7.x | https://search.maven.org/artifact/org.elasticsearch.client/transport |
Hadoop Azure Client from Cloudera | hadoop_azure_cloudera.sh |
This script downloads the Hadoop Azure client libraries provided by Cloudera. The Hadoop Azure client libraries cannot be loaded along with the Hadoop client because in Cloudera, the version numbers between the two components do not line up perfectly. |
|
2.x and 3.x | https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-azure/ |
Hadoop Client from Cloudera | hadoop_cloudera.sh |
This script downloads the Hadoop client libraries provided by Cloudera. |
|
2.x and 3.x | https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-client/ |
Hadoop Client from Hortonworks | hadoop_hortonworks.sh |
The Hadoop client including the libraries for connectivity to Azure Data Lake available from Hortonworks. |
|
2.x and 3.x | https://repo.hortonworks.com/content/groups/public/org/apache/hadoop/hadoop-azure/ |
Apache Hadoop Client Plus Required Libraries for Azure Connectivity | hadoop.sh |
The Hadoop client including the libraries for connectivity to Azure Data Lake. |
|
2.7.x and higher, and 3.x | https://search.maven.org/artifact/org.apache.hadoop/hadoop-azure |
HBase Client Provided by Cloudera | hbase_cloudera.sh |
The HBase client libraries provided by Cloudera. | HBase Handler | 1.x and 2.x | https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hbase/hbase-client/ |
HBase Client Provided by Hortonworks | hbase_hortonworks.sh |
The HBase client libraries provided by Hortonworks. | HBase Handler | 1.x and 2.x | https://repo.hortonworks.com/content/groups/public/org/apache/hbase/hbase-client/ |
Apache HBase Client | hbase.sh |
The HBase client. |
HBase Handler | 1.x and 2.x | https://search.maven.org/artifact/org.apache.hbase/hbase-client |
Apache Kafka Client plus Kafka Connect Framework and JSON Converter from Cloudera | kafka_cloudera.sh |
The Kafka Client plus libraries for the Kafka Connect framework and the Kafka Connect JSON Converter provided by Cloudera. |
|
0.9.x to current | https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/kafka/kafka-clients/ |
Apache Kafka Client plus Kafka Connect Framework and JSON Converter from Hortonworks | kafka_hortonworks.sh |
The Kafka Client plus libraries for the Kafka Connect framework and the Kafka Connect JSON Converter provided by Hortonworks. |
|
0.9.x to current | https://repo.hortonworks.com/content/groups/public/org/apache/kafka/kafka-clients/ |
Apache Kafka Client plus Kafka Connect Framework and JSON Converter | kafka.sh |
The Kafka Client plus libraries for the Kafka Connect framework and the Kafka Connect JSON Converter. |
|
0.9.x to current | https://search.maven.org/artifact/org.apache.kafka/kafka-clients |
Confluent Kafka Client plus Kafka Connect Framework and JSON and Avro Converters | kafka_confluent.sh |
The Kafka Client plus libraries for the Kafka Connect framework and the Kafka Connect JSON Converter and the Kafka Connect Avro Converter available from Confluent. |
|
Confluent platform 4.1.0 and higher. | The Confluent Maven repository does not provide a web GUI interface. |
MongoDB Client | mongodb.sh |
The MongoDB client libraries. | MongoDB Handler | 3.x | https://search.maven.org/artifact/org.mongodb/mongo-java-driver |
Oracle NoSQL Client | oracle_nosql.sh |
The Oracle NoSQL client libraries. | Oracle NoSQL Handler | 3.x, 4.x, and 18.x | https://search.maven.org/artifact/com.oracle.kv/oracle-nosql-client |
Oracle OCI Client | oracle_oci.sh |
The Oracle OCI client libraries. | Oracle OCI Event Handler | 1.x | https://search.maven.org/artifact/com.oracle.oci.sdk/oci-java-sdk-full |
Apache ORC (Optimized Row Columnar) Client | orc.sh |
The Apache ORC client libraries. ORC is built on top of the Hadoop client so the ORC Event Handler needs the Hadoop client in order to run. The Hadoop client needs to be downloaded separately. | ORC Event Handler | 1.x | https://search.maven.org/artifact/org.apache.orc/orc-core |
Apache Parquet Client | parquet.sh |
The Apache Parquet client libraries. Parquet is built on top of the Hadoop client, so the Parquet Event Handler needs the Hadoop client in order to run. The Hadoop client needs to be downloaded separately. | Parquet Event Handler | 1.x | https://search.maven.org/artifact/org.apache.parquet/parquet-hadoop |
Parent topic: Dependency Downloader