7.4 Realtime Replication into Oracle Cloud Infrastructure (OCI) Streaming with GoldenGate for Big Data
This topic covers a step-by-step process on how to ingest messages into OCI Streaming in real-time with Oracle GoldenGate for Big Data.
- Install Dependency Files
- Create Kafka Producer Properties File
- Create a Replicat in Oracle GoldenGate for Big Data
Parent topic: Quickstarts
7.4.1 Install Dependency Files
Oracle GoldenGate for Big Data uses client libraries in the replication process. You need to download these libraries by using the Dependency Downloader utility available in Oracle GoldenGate for Big Data before setting up the replication process. Dependency downloader is a set of shell scripts that downloads dependency jar files from Maven and other repositories.
To install the required dependency files:
- Go to installation location of Dependency Downloader:
GG_HOME/opt/DependencyDownloader/
. - Execute kafka.sh with the required
version.
Figure 7-19 Executing kafka.sh with the required versions
A directory is created in
GG_HOME/opt/DependencyDownloader/dependencies
. For example,/u01/app/ogg/opt/DependencyDownloader/dependencies/kafka_2.7.0
.
7.4.2 Create Kafka Producer Properties File
Oracle GoldenGate for Big Data must access a Kafka producer configuration file to publish messages to OCI Streaming. The Kafka producer configuration file contains kafka connection settings provided by OCI Streaming. To get OCI Streaming Kafka connection settings, go to Analytics&AI/Streaming/Stream Pools/Stream Pool Details/Kafka Connection Settings. You also need to create an AUTH_TOKEN.
To create a Kafka producer configuration file:
- In the Oracle GoldenGate for Big Data, go to
GGBD_Deployment_Home/etc/conf/ogg
. - Create a Kafka producer config file for OCI Streaming. Sample
configuration
file:
bootstrap.servers=cell-1.streaming.us-phoenix-1.oci.oraclecloud.com:9092 security.protocol=SASL_SSL sasl.mechanism=PLAIN value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="paasdevgg/oracleidentitycloudservice/user.name@oracle.com/ocid1.streampool.oc1.phx.amaaaaaa3p5c3vqa4hfyl7uv465pay4audmoajughhxlsgj7afc2an5u3xaq" password="YOUR-AUTH-TOKEN";
7.4.3 Create a Replicat in Oracle GoldenGate for Big Data
To create a replicat in Oracle GoldenGate for Big Data:
- In the Oracle GoldenGate for Big Data UI, in the Administration Service
tab, click the + sign to add a replicat.
Figure 7-20 Click + in the Administration Service tab.
- Select the Replicat Type and click Next.
There are two different Replicat types here: Classic and Coordinated. Classic Replicat is a single threaded process whereas Coordinated Replicat is a multithreaded one that applies transactions in parallel.
For KafKa, Oracle recommends Classic replicat as sending messages in multiple threats may result in data consistency problems.Figure 7-21 Select the Replicat Type and click Next.
- Enter the basic information, and click Next:
- Process Name: Name of the Replicat
- Trail Name: Name of the required trail file
- Target: Kafka
Figure 7-22 Process Name, Trail Name, and Target Names
- Enter Parameter File details and click Next. In the Parameter
File, you can either specify source to target mapping or leave it as is with a
wildcard selection.
Figure 7-23 Provide Parameter File details and click Next.
- Oracle GoldenGate for Big Data populates the properties file
automatically. Update the following fields:
gg.handler.kafkahandler.kafkaProducerConfigFile
=name of the oci streaming producer file created in Create Kafka Producer Properties Filegg.handler.kafkahandler.topicMappingTemplate
=name of the target topic gg.classpath=path to dependency files downloaded in Install Dependency Files
- Click Create and Run.
Figure 7-24 Properties file updates.
- If replicat starts successfully, then the replicat is in running
state. You can go to action/details/statistics to see the replication
statistics.
Figure 7-25 Replication Statistics
Note:
- If target Kafka topic does not exist, then it will be auto created by Oracle GoldenGate for Big Data if you have selected auto topic create in OCI streaming Kafka connection settings. See Template Keywords.
- For improving the performance of the OCI Streaming replication, see the blog: How to Improve Kafka Handler Performance in Oracle GoldenGate for Big Data
- Oracle GoldenGate for Big Data supports SSL and kerberos authentication into Kafka. For more information, see Schema Propagation.
- For Kafka connection issues, Oracle Support.