Realtime Message Ingestion to OCI Streaming with Oracle GoldenGate for Distributed Applications and Analytics

Overview

This Quickstart covers a step-by-step process showing how to ingest messages to OCI Streaming in real-time with Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA).

Oracle Cloud Infrastructure (OCI) Streaming provides a fully managed, scalable, and durable messaging solution for ingesting continuous, high-volume streams of data that you can consume and process in real-time. OCI Streaming supports Kafka APIs.

GG for DAA connects to OCI Streaming with Kafka Handler. GG for DAA reads the source operations from the trail file, formats them, maps to OCI Streaming streams and delivers.

Prerequisites

To successfully complete this Quickstart, you must have the following:

Install Dependency Files

GG for DAA uses Java SDK provided by OCI Streaming. You can download the SDKs using Dependency Downloader utility shipped with GG for DAA. Dependency downloader is a set of shell scripts that downloads dependency jar files from Maven and other repositories.

  1. In your GG for DAA VM, go to Dependency Downloader utility. It is located at GG_HOME/opt/DependencyDownloader/ and locate kafka.sh.

  2. Run kafka.sh with the required version. You can check the version and reported vulnerabilities in Maven Central. This document uses 2.7.0. Figure 75: Run kafka.sh

Description of image follows

  1. A directory is created in GG_HOME/opt/DependencyDownloader/dependencies. Note the directories. For example: /u01/app/ogg/opt/DependencyDownloader/dependencies/kafka_2.7.0

Create Kafka Producer Properties

GG for DAA must access a Kafka producer configuration file to publish messages to OCI Streaming. The Kafka producer configuration file contains kafka connection settings provided by OCI Streaming. To get OCI Streaming Kafka connection settings, go to Analytics&AI/Streaming/Stream Pools/Stream Pool Details/Kafka Connection Settings. You also need to create an AUTH_TOKEN.

In your GG for DAA instance, create a Kafka producer config file for OCI Streaming.

Sample Config File

bootstrap.servers=cell-1.streaming.us-phoenix-1.oci.oraclecloud.com:9092
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="paasdevgg/oracleidentitycloudservice/user.name@oracle.com/ocid1.streampool.oc1.phx.amaaaaaa3p5c3vqa4hfyl7uv465pay4audmoajughhxlsgj7afc2an5u3xaq" password="YOUR-AUTH-TOKEN";

Create a Replicat in GG for DAA

To create a replicat in GG for DAA:

  1. In the Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) UI, in the Administration Service tab, click the + sign to add a replicat. Figure 76: Click the Administration Service tab

Description of image follows

  1. Select the Classic Replicat Replicat Type and click Next. There are two different Replicat types available: Classic and Coordinated. Classic Replicat is a single threaded process whereas Coordinated Replicat is a multithreaded one that applies transactions in parallel. Figure 77: Add Replicat

Description of image follows

  1. Enter the Replicat information, and click Next:

    1. Replicat Trail: Name of the required trail file. For sample trail, provide tr.

    2. Subdirectory: Enter GG_HOME/opt/AdapterExamples/trail/ if using the sample trail.

    3. Target: Kafka

    Figure 78: Replicat Options

Description of image follows

  1. Leave Managed Options as is and click Next. Figure 79: Managed Options

Description of image follows

  1. Enter Parameter File details and click Next. In the Parameter File, you can either specify source to target mapping or leave it as-is with a wildcard selection. Figure 80: Parameter File

Description of image follows

  1. In the Properties File, update the properties marked as TODO and click Create and Run.

    #Kafka Handler Template
gg.handlerlist=kafkahandler
gg.handler.kafkahandler.type=kafka
**#TODO**: Set the name of the Kafka producer properties file.
gg.handler.kafkahandler.kafkaProducerConfigFile=/path_to/producer.properties
**#TODO**: Set the template for resolving the topic name.
gg.handler.kafkahandler.topicMappingTemplate=<target_stream_name>
gg.handler.kafkahandler.keyMappingTemplate=${primaryKeys}
gg.handler.kafkahandler.mode=op
gg.handler.kafkahandler.format=json
gg.handler.kafkahandler.format.metaColumnsTemplate=${objectname[table]},${optype[op_type]},${timestamp[op_ts]},${currenttimestamp[current_ts]},${position[pos]}
**#TODO**: Set the location of the Kafka client libraries.
gg.classpath=path_to/dependencies/kafka_2.7.0/*
jvm.bootoptions=-Xmx512m -Xms32m

  2. If replicat starts successfully, then it is in running state. You can go to Replicats/Statistics to see the replication statistics. Figure 81: Replicat Statistics

Description of image follows

  1. Go to the OCI console and check the statistics. Figure 82: OCI Streaming console

Description of image follows

For more details about OCI Streaming replication, see OCI Streaming.

  <div class="infoboxnote" markdown="1">

  **Note:**

  - If target kafka topic does not exist, then it is auto created by GG for DAA if **Auto topic create** is selected in OCI streaming Kafka connection settings. See [Template Keywords](additional-details.html#GUID-742BA6BE-D446-4E21-8E38-7105AC9F5E5E) to dynamically assign the topic names.

  - You can refer to [this blog](https://blogs.oracle.com/dataintegration/post/how-to-improve-kafka-handler-performance-in-oracle-goldengate-for-big-data) for improving the performance of the OCI Streaming replication.

  </div>