Capture data from Kafka platforms

Overview

You can use OCI GoldenGate to capture messages from the following streaming sources:

Apache Kafka
OCI Streaming
Confluent Kafka, with or without Schema Registry
Azure Event Hubs
Amazon MSK

OCI GoldenGate reads messages from a Kafka topic or topics, and then converts the data into logical change records written to GoldenGate Trail files. GoldenGate Replicat processes can then use the generate Trail files to propogate data to support RDBMS implementations.

Task 1: Configure Consumer properties

Create a Kafka Consumer properties file with one of the following deserializers or converters. If the source is a topic in Confluent Kafka with Schema Registry, you can use the Avro converter. For other sources, use the JSON converter or deserializer as needed:
- Kafka Consumer properties for JSON deserializer:
```
key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
```
- Kafka Consumer properties for JSON converter:
```
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
```
- Kafka Consumer properties for Avro converter:
```
key.converter=io.confluent.connect.avro.AvroConverter
value.converter=io.confluent.connect.avro.AvroConverter
```
Save the properties file and note its location.

Task 2: Create OCI GoldenGate resources

This task guides you on how to create new resources if they don't yet exist. Ensure that the Big Data deployment you're using is upgraded to the latest version available.

Create an OCI GoldenGate deployment for Big Data.
Create a connection.

Note:
When creating any of the following connections, ensure that you, click Show Advanced Options and then upload the Consumer properties file.
- For Apache Kafka or Amazon MSK, create a Kafka connection.
- For Confluent Kafka, create a Confluent Kafka connection.
- For Confluent Schema Registry, create a Confluent Schema Registry connection.
- For Azure Event Hubs, create an Azure Event Hubs connection.
- For OCI Streaming, create an OCI Streaming connection.
Assign the connection to the Big Data deployment.

Task 3: Create a credential

Before you can create the Extract, you must first create a credential.

To create a credential:

Select the Big Data deployment on the Deployments page.
On the deployment details page, click Launch console.
Log in to the Big Data deployment with the user name and password specified when you created the deployment in Task 2 Step 1.
From the navigation menu, select DB Connections.
On the Configuration page, under the Database tab, click Add DB Connection (plus icon), and then complete the form as follows:
1. For Credential Domainenter OracleGoldenGate.
2. For Credential Alias, enter kafka.
3. For User ID, enter kafka://
4. For Password and Verify Password, enter a password.
5. Click Submit.

Task 4: Create the Extract

On the Administration Service Overview page, click Add Extract (plus icon).
Add an Extract with the following values:
1. On the Extract Information page, complete the field as follows, and then click Next:
  - For Extract Type, select Integrated Extract.
  - For Process Name, enter a name for the extract.
2. On the Extract Options page, complete the fields as follows, and then click Next:
  - For Domain, select the domain.
  - For Alias, select the connection assigned to the deployment.
  - For Name, enter a 2-character name.
3. On the Managed Options page, leave the fields as they are, and click Next.
4. On the Parameter File page:
  - Leave the table mapping as TABLE TESTSCHEMA.*; to listen to all topics in the given bootstrap server. You can also set the table mapping as TABLE TESTSCHEMA.<topic-name>; to capture from a designated topic.
  - Update SOURCEDB USERIDALIAS to SOURCEDB USERIDALIAS kafka DOMAIN OracleGoldenGate.
5. Click Create and Run.

You return to the Administration Service Overview page, where you can observe the Extract process start and review event messages.