Capture data from Kafka platforms

Use OCI GoldenGate to extract message from Kafka platform streaming sources.

Overview

You can use OCI GoldenGate to capture messages from the following streaming sources:

Apache Kafka
OCI Streaming
Confluent Kafka, with or without Confluent Schema Registry
Azure Event Hubs
Amazon MSK
Microsoft Fabric Eventstream

OCI GoldenGate reads messages from a Kafka topic or topics, and then converts the data into logical change records written to GoldenGate Trail files. GoldenGate Replicat processes can then use the generate Trail files to propagate data to support RDBMS implementations.

Task 1: Configure Consumer properties

Create a Kafka Consumer properties file with one of the following deserializers or converters. If the source is a topic in Confluent Kafka with Confluent Schema Registry, you can use the Avro converter. For other sources, use the JSON converter or deserializer as needed:
- Kafka Consumer properties for JSON deserializer:
```
key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
```
- Kafka Consumer properties for JSON converter:
```
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
```
- Kafka Consumer properties for Avro converter:
```
key.converter=io.confluent.connect.avro.AvroConverter
value.converter=io.confluent.connect.avro.AvroConverter
```
Save the properties file and note its location.

Task 2: Create OCI GoldenGate resources

This task guides you on how to create new resources if they don’t yet exist. Ensure that the Big Data deployment you’re using is upgraded to the latest version available.

Create an OCI GoldenGate deployment for Big Data.
Create a connection.

Note: When creating any of the following connections, ensure that you, select Show Advanced Options and then upload the Consumer properties file.
- For Apache Kafka or Amazon MSK, create a Kafka connection.
- For Confluent Kafka, create a Confluent Kafka connection.
- For Confluent Schema Registry, create a Confluent Schema Registry connection.
- For Azure Event Hubs, create an Azure Event Hubs connection.
- For OCI Streaming, create an OCI Streaming connection.
- For Microsoft Fabric Eventstream, create a Microsoft Fabric Eventstream connection
Assign the connection to the Big Data deployment.

Task 3: Create a credential

Before you can create the Extract, you must first create a credential.

To create a credential:

Select the Big Data deployment on the Deployments page.
On the deployment details page, select Launch console.
Log in to the Big Data deployment with the user name and password specified when you created the deployment in Task 2 Step 1.
From the navigation menu, select DB Connections.
On the Configuration page, under the Database tab, select Add DB Connection (plus icon), and then complete the form as follows:
1. For Credential Domain enter OracleGoldenGate.
2. For Credential Alias, enter kafka.
3. For User ID, enter kafka://
4. For Password and Verify Password, enter a password.
5. Select Submit.

Task 4: Create the Extract

On the Home page, select Add Extract (plus icon).
Add an Extract with the following values:
1. On the Extract Information page, complete the field as follows, and then select Next:
  - For Extract Type, select Integrated Extract.
  - For Process Name, enter a name for the extract.
2. On the Extract Options page, complete the fields as follows, and then select Next:
  - For Domain, select the domain.
  - For Alias, select the connection assigned to the deployment.
  - For Name, enter a 2-character name.
3. On the Managed Options page, leave the fields as they are, and select Next.
4. On the Parameter File page:
  - Leave the table mapping as TABLE TESTSCHEMA.*; to listen to all topics in the given bootstrap server. You can also set the table mapping as TABLE TESTSCHEMA.<topic-name>; to capture from a designated topic.
  - Update SOURCEDB USERIDALIAS to SOURCEDB USERIDALIAS kafka DOMAIN OracleGoldenGate.
5. Select Create and Run.

You return to the Oracle GoldenGate Home page, where you can observe the Extract process start and review event messages.