7.7 Realtime Message Ingestion to OCI Streaming with Oracle GoldenGate for Distributed Applications and Analytics

Overview

This Quickstart covers a step-by-step process showing how to ingest messages to OCI Streaming in real-time with Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA).

Oracle Cloud Infrastructure (OCI) Streaming provides a fully managed, scalable, and durable messaging solution for ingesting continuous, high-volume streams of data that you can consume and process in real-time. OCI Streaming supports Kafka APIs.

GG for DAA connects to OCI Streaming with Kafka Handler. GG for DAA reads the source operations from the trail file, formats them, maps to OCI Streaming streams and delivers.

• Prerequisites
  
• Install Dependency Files
  
• Create Kafka Producer Properties
  
• Create a Replicat in GG for DAA
  

7.7.1 Prerequisites

To successfully complete this Quickstart, you must have the following:

• OCI Streaming Stream Pools
• OCI Auth Token

In this Quickstart, a sample trail file (named tr) which is shipped with GG for DAA is used. The sample trail file is located at GG_HOME/opt/AdapterExamples/trail/ in your GG for DAA instance.

7.7.2 Install Dependency Files

GG for DAA uses Java SDK provided by OCI Streaming. You can download the SDKs using Dependency Downloader utility shipped with GG for DAA. Dependency downloader is a set of shell scripts that downloads dependency jar files from Maven and other repositories.

1. In your GG for DAA VM, go to Dependency Downloader utility. It is located at GG_HOME/opt/DependencyDownloader/ and locate kafka.sh.
2. Execute kafka.sh with the required version. You can check the version and reported vulnerabilities in Maven Central. This document uses 2.7.0.

   Figure 7-43 Execute kafka.sh

   
3. A directory is created in GG_HOME/opt/DependencyDownloader/dependencies. Note the directories. For example: /u01/app/ogg/opt/DependencyDownloader/dependencies/kafka_2.7.0

7.7.3 Create Kafka Producer Properties

GG for DAA must access a Kafka producer configuration file to publish messages to OCI Streaming. The Kafka producer configuration file contains kafka connection settings provided by OCI Streaming. To get OCI Streaming Kafka connection settings, go to Analytics&AI/Streaming/Stream Pools/Stream Pool Details/Kafka Connection Settings. You also need to create an AUTH_TOKEN.

In your GG for DAA instance, create a Kafka producer config file for OCI Streaming.

Sample Config File

bootstrap.servers=cell-1.streaming.us-phoenix-1.oci.oraclecloud.com:9092
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="paasdevgg/oracleidentitycloudservice/[e-mail]/ocid1.streampool.oc1.phx.amaaaaaa3p5c3vqa4hfyl7uv465pay4audmoajughhxlsgj7afc2an5u3xaq" password="YOUR-AUTH-TOKEN";

7.7.4 Create a Replicat in GG for DAA

To create a replicat in GG for DAA:

1. In the Oracle GoldenGate for Distributed Applications and Analytics (GG for DAA) UI, in the Administration Service tab, click the + sign to add a replicat.

   Figure 7-44 Click the Administration Service tab

   
2. Select the Classic Replicat Replicat Type and click Next. There are two different Replicat types available: Classic and Coordinated. Classic Replicat is a single threaded process whereas Coordinated Replicat is a multithreaded one that applies transactions in parallel.

   Figure 7-45 Add Replicat

   
3. Enter the Replicat information, and click Next:
   1. Replicat Trail: Name of the required trail file. For sample trail, provide tr.
   2. Subdirectory: Enter GG_HOME/opt/AdapterExamples/trail/ if using the sample trail.
   3. Target: Kafka

   Figure 7-46 Replicat Options

   
4. Leave Managed Options as is and click Next.

   Figure 7-47 Managed Options

   
5. Enter Parameter File details and click Next. In the Parameter File, you can either specify source to target mapping or leave it as-is with a wildcard selection.

   Figure 7-48 Parameter File

   
6. In the Properties File, update the properties marked as TODO and click Create and Run.

   #Kafka Handler Template
   gg.handlerlist=kafkahandler
   gg.handler.kafkahandler.type=kafka
   #TODO: Set the name of the Kafka producer properties file.
   gg.handler.kafkahandler.kafkaProducerConfigFile=/path_to/producer.properties
   #TODO: Set the template for resolving the topic name.
   gg.handler.kafkahandler.topicMappingTemplate=<target_stream_name>
   gg.handler.kafkahandler.keyMappingTemplate=${primaryKeys}
   gg.handler.kafkahandler.mode=op
   gg.handler.kafkahandler.format=json
   gg.handler.kafkahandler.format.metaColumnsTemplate=${objectname[table]},${optype[op_type]},${timestamp[op_ts]},${currenttimestamp[current_ts]},${position[pos]}
   #TODO: Set the location of the Kafka client libraries.
   gg.classpath=path_to/dependencies/kafka_2.7.0/*
   jvm.bootoptions=-Xmx512m -Xms32m
   

7. If replicat starts successfully, then it is in running state. You can go to Replicats/Statistics to see the replication statistics.

   Figure 7-49 Replicat Statistics

   
8. Go to the OCI console and check the statistics.

   Figure 7-50 OCI Streaming console

   

For more details about OCI Streaming replication, see OCI Streaming.

Note:

• If target kafka topic does not exist, then it is auto created by GG for DAA if Auto topic create is selected in OCI streaming Kafka connection settings. See Template Keywords to dynamically assign the topic names.
• You can refer to this blog for improving the performance of the OCI Streaming replication.