Capture data from Kafka platforms

Use OCI GoldenGate to extract message from Kafka platform streaming sources.

Overview

You can use OCI GoldenGate to capture messages from the following streaming sources:

  • Apache Kafka
  • OCI Streaming
  • Confluent Kafka, with or without Schema Registry
  • Azure Event Hubs
  • Amazon MSK

OCI GoldenGate reads messages from a Kafka topic or topics, and then converts the data into logical change records written to GoldenGate Trail files. GoldenGate Replicat processes can then use the generate Trail files to propogate data to support RDBMS implementations.

Task 1: Configure Consumer properties

  1. Create a Kafka Consumer properties file with one of the following deserializers or converters. If the source is a topic in Confluent Kafka with Schema Registry, you can use the Avro converter. For other sources, use the JSON converter or deserializer as needed:
    • Kafka Consumer properties for JSON deserializer: 
      key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
    • Kafka Consumer properties for JSON converter:
      key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
    • Kafka Consumer properties for Avro converter:
      key.converter=io.confluent.connect.avro.AvroConverter
value.converter=io.confluent.connect.avro.AvroConverter
  2. Save the properties file and note its location.

Task 2: Create OCI GoldenGate resources

This task guides you on how to create new resources if they don't yet exist. Ensure that the Big Data deployment you're using is upgraded to the latest version available.

  1. Create an OCI GoldenGate deployment for Big Data.
  2. Create a connection.

    Note:

    When creating any of the following connections, ensure that you, click Show Advanced Options and then upload the Consumer properties file.
  3. Assign the connection to the Big Data deployment.

Task 3: Create a credential

Before you can create the Extract, you must first create a credential.
To create a credential:
  1. Select the Big Data deployment on the Deployments page.
  2. On the deployment details page, click Launch console.
  3. Log in to the Big Data deployment with the user name and password specified when you created the deployment in Task 2 Step 1.
  4. From the navigation menu, select Configuration.
  5. On the Configuration page, under the Database tab, click Add Credential, and then complete the form as follows:
    1. For Credential Domainenter OracleGoldenGate.
    2. For Credential Alias, enter kafka.
    3. For User ID, enter kafka://
    4. For Password and Verify Password, enter a password.
    5. Click Submit.

Task 4: Create the Extract

  1. On the Administration Service Overview page, click Add Extract (plus icon).
  2. On the Add Extract page, for Extract type, select Change Data Capture, and then click Next.
  3. On the Extract Options page, complete the fields as follows, and then click Next:
    1. For Process Name, enter a name for the extract.
    2. For Alias, select the connection assigned to the deployment.
    3. For Begin, select Now.
    4. For Trail Name, enter a 2-character name.
    5. (Optional) Enable Kafka Connect, if the source is a Kafka Connect framework.
    6. (Optional) Select a Converter. If you select Avro, select Schema Registry.
  4. On the Parameter File page:
    1. Leave the table mapping as TABLE TESTSCHEMA.*; to listen to all topics in the given bootstrap server. You can also set the table mapping as TABLE TESTSCHEMA.<topic-name>; to capture from a designated topic.
    2. Update SOURCEDB USERIDALIAS to SOURCEDB USERIDALIAS kafka DOMAIN OracleGoldenGate
  5. Click Create and Run.
You return to the Administration Service Overview page, where you can observe the Extract process start and review event messages.