Add an Extract for Kafka

Adding an Extract for a Kafka source differs between Big Data deployment versions. Follow the instructions according to your Big Data deployment version.

To check your OCI GoldenGate version, view the Version information on the deployment details page.

You can capture messages from the following streaming sources:

  • Apache Kafka
  • OCI Streaming
  • Confluent Kafka, with or without Schema Registry
  • Azure Event Hubs
  • Amazon MSK

OCI GoldenGate reads messages from a Kafka topic or topics, and then converts the data into logical change records written to GoldenGate Trail files. GoldenGate Replicat processes can then use the generated Trail files to propagate data to support relational database management system (RDBMS) implementations.

An OCI GoldenGate Big Data deployment supports multiple source technologies within the same deployment, so you can configure multiple extracts from different supported source technologies within the same deployment. OCI GoldenGate uses credential store entries to identify the source technology. For example, OCI GoldenGate uses the credential's User ID to determine whether to start a Kafka (kafka:// ) or Mongo (Mongo://) Extract.

Add an Extract (21c)

Before you create an Extract, create a credential in the Big Data deployment console:

  1. On the Deployments page, select the Big Data deployment to view its details.
  2. Log in to the Big Data deployment console using the credentials specified when you created the deployment. You're brought to the Administration Server Overview page.
  3. Add a credential:
    1. Open the navigation menu, and then click Configuration.
    2. On the Credentials page, click Add Credential (plus icon).
    3. Enter the following details in the fields provided, and then click Submit:
      • For Credential Domain, enter OracleGoldenGate.
      • For Credential Alias, enter kafka.
      • For User ID: enter kafka://
      • For Password, enter a password.
      • For Verify Password, enter the password again.
To add an Extract for Kafka in OCI GoldenGate for Big Data deployments version 21.9 or later:
  1. In the navigation menu, click Overview to return to the Administration Service Overview page.
  2. Click Add Extract (plus icon).
  3. On the Add Extract page, select the following, and then click Next:
    • Source: Kafka
    • Extract type: Change Data Capture Extract
  4. On the Extract Options page, complete the fields as follows, and then click Next:
    • For Process Name, enter a name, up to 8 characters.
    • (Optional) Enter a description.
    • For Connection Alias, select the previously assigned connection alias.
    • For Begin, select Now.
    • For Trail Name, enter a 2-character name.
    • (Optional) Enable Kafka Connect, if the source is a Kafka Connect framework.
    • (Optional) If Kafka Connect is selected, select one of the available Converters:
      • JSON
      • AVRO
    • (Optional) If AVRO is selected, select Schema Registry.
    • (Optional) Under Managed Options, you can configure the following:
      • Profile Name
      • Auto Start
      • Auto Restart

      Note:

      Adding a profile and configuring Auto Start and Auto Restart options enables your deployment to restart automatically after a network disruption. See Configure managed processes for more information.
  5. On the Parameter file page, update the following:
    1. Update SOURCEDB USERIDALIAS to SOURCEDB USERIDALIAS kafka DOMAIN OracleGoldenGate
    2. Leave the table mapping as is (TABLE source.*;), to listen to all topics in the given bootstrap server. To capture from a designated topic, change the mapping to TABLE source.<topic-name>; where <topic-name> is the name of the topic to capture from.

      Note:

      The Extract mapping format is Table SourceSchema.Table. In Kafka, there are no source schemas/tables. OCI GoldenGate writes the first part ("source" in above step) as the schema name and the second part as table name to trail file. This way, you can replicate the captured kafka message into other GoldenGate supported targets.
  6. Click Create and Run.
You return to the Administration Service Overview page where you can observe the Extract status. You can then select the Extract to view its details, statistics, and reports.

Add an Extract (23ai)

Before you add and run an Extract, ensure that you:
  • Review details specific to this source connection type.
  • Create a credential in the Big Data deployment console:
    1. Launch the Big Data deployment console from the deployment details page, and then log in.
    2. In the left navigation menu of the Administration Service, click DB Connections.
    3. On the DB Connections page, click Add Credential (plus icon).
    4. Enter the following details in the fields provided, and then click Submit:
      • For Credential Domain, enter OracleGoldenGate.
      • For Credential Alias, enter kafka.
      • For User ID: enter kafka://
      • For Password, enter a password.
      • For Verify Password, enter the password again.
To add an Extract in Oracle GoldenGate 23ai:
  1. In the OCI GoldenGate deployment console, on the Administration Service Home page, click Add Extract.
  2. In the Add Extract panel, on the Extract Information page, complete the following fields as needed, and then click Next:
    • Enter a Process Name, no more than 8 characters long.
    • Enter a Description (optional) to help you distinguish this process from others.
    • Select a Source from the dropdown.
  3. On the Extract Options page, complete the following fields as needed, and then click Next:
    1. For Extract Trail:
      1. Enter a Name for the Extract process.
      2. Enter a Subdirectory name to set a custom location for the generated Trail file.
      3. Enter Trail Sequence to set the starting number for Trail files.
      4. Enter a Trail Size to set the max size for the generated trail file.
      5. Select an Encryption Profile. The Local Wallet profile is selected by default if an encryption profile wasn't created.
      6. Select an Encryption Algorithm:
        • NONE
        • AES256
        • AES192
        • AES128
    2. For Source Credentials, select the:
      1. Domain
      2. Alias
      3. When/where to Begin
  4. On the Managed Options page, complete the following optional fields as needed, and then click Next:
    • Profile Name
    • Critical to deployment health
    • Auto Start
    • Auto Restart

    Note:

    Adding a profile and configuring Auto Start and Auto Restart options enables your deployment to restart automatically after a network disruption. See Configure managed processes to learn more.

  5. On the Parameter file page, update the following:
    1. Update SOURCEDB USERIDALIAS to SOURCEDB USERIDALIAS kafka DOMAIN OracleGoldenGate
    2. Leave the table mapping as is (TABLE source.*;), to listen to all topics in the given bootstrap server. To capture from a designated topic, change the mapping to TABLE source.<topic-name>; where <topic-name> is the name of the topic to capture from.

      Note:

      The Extract mapping format is Table SourceSchema.Table. In Kafka, there are no source schemas/tables. OCI GoldenGate writes the first part ("source" in above step) as the schema name and the second part as table name to trail file. This way, you can replicate the captured kafka message into other GoldenGate supported targets.
  6. Click Create and Run to create and start the Extract. If you click Create, then you can manually start the Extract later from the Extracts page.