7.6 Realtime Data Ingestion into Kafka with Oracle GoldenGate for Big Data

This topic covers a step-by-step process on how to ingest messages into Kafka in real-time with GoldenGate for Big Data.

• Install Dependency Files
  
• Create Kafka Producer Properties File
  
• Create a Replicat in Oracle GoldenGate for Big Data
  

7.6.1 Install Dependency Files

Oracle GoldenGate for Big Data uses client libraries in the replication process. You need to download these libraries by using the Dependency Downloader utility available in Oracle GoldenGate for Big Data before setting up the replication process. Dependency downloader is a set of shell scripts that downloads dependency jar files from Maven and other repositories.

To install the required dependency files:

1. Go to installation location of Dependency Downloader: GG_HOME/opt/DependencyDownloader/.
2. Execute kafka.sh with the required version.

   Figure 7-34 Executing kafka.sh with the required versions

   

   A directory is created in GG_HOME/opt/DependencyDownloader/dependencies. For example, /u01/app/ogg/opt/DependencyDownloader/dependencies/kafka_2.7.0.

7.6.2 Create Kafka Producer Properties File

Oracle GoldenGate for Big Data must access a Kafka producer configuration file to publish messages to Kafka. The Kafka producer configuration file contains Kafka proprietary properties.

To create a Kafka producer configuration file:

1. In the Oracle GoldenGate for Big Data, go to GGBD_Deployment_Home/etc/conf/ogg.
2. Create a Kafka producer config file for OCI Streaming. Sample configuration file:

   bootstrap.servers=localhost:9092
   acks = 1
   compression.type = gzip
   reconnect.backoff.ms = 1000
    
   value.serializer = org.apache.kafka.common.serialization.ByteArraySerializer
   key.serializer = org.apache.kafka.common.serialization.ByteArraySerializer

7.6.3 Create a Replicat in Oracle GoldenGate for Big Data

To create a replicat in Oracle GoldenGate for Big Data:

1. In the Oracle GoldenGate for Big Data UI, in the Administration Service tab, click the + sign to add a replicat.

   Figure 7-35 Click + in the Administration Service tab.

   
2. Select the Replicat Type and click Next.

   There are two different Replicat types here: Classic and Coordinated. Classic Replicat is a single threaded process whereas Coordinated Replicat is a multithreaded one that applies transactions in parallel.

   For KafKa, Oracle recommends Classic replicat as sending messages in multiple threats may result in data consistency problems.

   Figure 7-36 Select the Replicat Type and click Next.

   
3. Enter the basic information, and click Next:
   1. Process Name: Name of the Replicat
   2. Trail Name: Name of the required trail file
   3. Target: Kafka

      Figure 7-37 Process Name, Trail Name, and Target Names

      
4. Enter Parameter File details and click Next. In the Parameter File, you can either specify source to target mapping or leave it as is with a wildcard selection.

   Figure 7-38 Provide Parameter File details and click Next.

   
5. Oracle GoldenGate for Big Data populates the properties file automatically. Update the following fields:
   • gg.handler.kafkahandler.kafkaProducerConfigFile=name of the Kafka producer file created in Create Kafka Producer Properties File.
   • gg.handler.kafkahandler.topicMappingTemplate=name of the target topic gg.classpath=path to dependency files downloaded in Install Dependency Files.
6. Click Create and Run.

   Figure 7-39 Properties file updates.

   
7. If replicat starts successfully, then the replicat is in running state. You can go to action/details/statistics to see the replication statistics:

   Figure 7-40 Replication Statistics

   

Note:

• If target Kafka topic does not exist, then it will be auto created by Oracle GoldenGate for Big Data. See Template Keywords to dynamically assign topic names.
• For improving the performance of Kafka replication, refer the blog: How to Improve Kafka Handler Performance in Oracle GoldenGate for Big Data.
• Oracle GoldenGate for Big Data supports SSL and kerberos authentication into Kafka. For more information, see Schema Propagation.
• For Kafka connection issues, see Oracle Support.