Connect to Apache Kafka

Learn to create a connection to Apache Kafka to use as an OCI GoldenGate source or target.

Before you begin

Before you create the connection, ensure that you:

Review how OCI GoldenGate connects to your source and targets.
Configure the required policies to enable secure Vault and Secrets access, such as use secrets, use vaults, and read secret-bundles. For more information, see Minimum recommended policies.

Create a source connection

To create a source Apache Kafka connection:

From the OCI GoldenGate Overview page, click Connections.
You can also click Create Connection under the Get started section and skip to step 3.
On the Connections page, click Create Connection.
On the Create Connection page, complete the fields as follows:
1. For Name, enter a name for the connection.
2. (Optional) For Description, enter a description that helps you distinguish this connection from others.
3. For Compartment, select the compartment in which to create the connection.
4. From the Type dropdown, select Apache Kafka.
5. Under Bootstrap servers:
  1. Enter the Host and Port number for the Bootstrap server. Enter the Private IP only if the hostname is not resolvable from your subnet or if it uses SSL/TLS.
    Note
    
    If you enter a private IP, then OCI GoldenGate rewrites the private IP in the format, ip-10-0-0-0.ociggsvc.oracle.vcn.com.
    
    Tip:
    All nodes in the cluster must have FQDNs to allow for traversal over private endpoints.
  2. (Optional) Click + Bootstrap server to add another bootstrap server.
6. For Security protocol, select one of the following and then complete the corresponding fields:
  - Plaintext
  - SASL over plaintext
  - SASL over SSL
  - SSL
7. Expand Show advanced options. You can configure the following options:
  - Security
    - Deselect Use vault secrets you prefer not to use password secrets for this connection. If not selected:
      
      Select Use Oracle-managed encryption key to leave all encryption key management to Oracle.
      
      Select Use customer-managed encryption key to select a specific encryption key stored in your OCI Vault to encrypt your connection credentials.
  - Network connectivity
    - Shared endpoint, to share an endpoint with the assigned deployment. You must allow connectivity from the deployment's ingress IP.
    - Dedicated endpoint, for network traffic through a dedicated endpoint in the assigned subnet in your VCN. You must allow connectivity from this connection's ingress IPs.
      Note
      
      If a dedicated connection remains unassigned for seven days, then the service converts it to a shared connection.
      
      Learn more about Oracle GoldenGate connectivity.
  - Settings
    - Deselect Use vault secrets you prefer not to use password secrets for this connection. If not selected:
      
      Select Use Oracle-managed encryption key to leave all encryption key management to Oracle.
      
      Select Use customer-managed encryption key to select a specific encryption key stored in your OCI Vault to encrypt your connection credentials.
    - Kafka Consumer properties for JSON deserializer:
      key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
    - Kafka Consumer properties for JSON converter:
      key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter
    - Kafka Consumer properties for Avro converter:
      key.converter=io.confluent.connect.avro.AvroConverter value.converter=io.confluent.connect.avro.AvroConverter
  - Tags: Add tags to organize your resources.
Click Create.

After the connection is created, it appears in the Connections list. Ensure that you assign the connection to a deployment to use it as a source or target in a replication.

Create the target connection

To create a target Apache Kafka connection:

From the OCI GoldenGate Overview page, click Connections.
You can also click Create Connection under the Get started section and skip to step 3.
On the Connections page, click Create Connection.
On the Create Connection page, complete the fields as follows:
1. For Name, enter a name for the connection.
2. (Optional) For Description, enter a description that helps you distinguish this connection from others.
3. For Compartment, select the compartment in which to create the connection.
4. From the Type dropdown, select Apache Kafka.
5. Under Bootstrap servers:
  1. Enter the Host and Port number for the Bootstrap server. Enter the Private IP only if the hostname is not resolvable from your subnet or if it uses SSL/TLS.
    Note
    
    If you enter a private IP, then OCI GoldenGate rewrites the private IP in the format, ip-10-0-0-0.ociggsvc.oracle.vcn.com.
    
    Tip:
    All nodes in the cluster must have FQDNs to allow for traversal over private endpoints.
  2. (Optional) Click + Bootstrap server to add another bootstrap server.
6. For Security protocol, select one of the following and then complete the corresponding fields:
  - Plaintext
  - SASL over plaintext
  - SASL over SSL
  - SSL
7. Expand Show advanced options. You can configure the following options:
  - Security
    - Deselect Use vault secrets you prefer not to use password secrets for this connection. If not selected:
      
      Select Use Oracle-managed encryption key to leave all encryption key management to Oracle.
      
      Select Use customer-managed encryption key to select a specific encryption key stored in your OCI Vault to encrypt your connection credentials.
  - Network connectivity
    - Shared endpoint, to share an endpoint with the assigned deployment. You must allow connectivity from the deployment's ingress IP.
    - Dedicated endpoint, for network traffic through a dedicated endpoint in the assigned subnet in your VCN. You must allow connectivity from this connection's ingress IPs.
      Note
      
      If a dedicated connection remains unassigned for seven days, then the service converts it to a shared connection.
      
      Learn more about Oracle GoldenGate connectivity.
  - Settings
    - Deselect Use vault secrets you prefer not to use password secrets for this connection. If not selected:
      
      Select Use Oracle-managed encryption key to leave all encryption key management to Oracle.
      
      Select Use customer-managed encryption key to select a specific encryption key stored in your OCI Vault to encrypt your connection credentials.
    To use Snappy compression in Kafka replication, drag and drop or select Producer properties, and change replication settings as discussed in Using Compression OCI GoldenGate (Confluent) Kafka Replication.
  - Tags: Add tags to organize your resources.
Click Create.

After the connection is created, it appears in the Connections list. Ensure that you assign the connection to a deployment to use it as a source or target in a replication.

Troubleshoot Kafka connection errors

Most connection issues result in TimeoutException errors. For example:

A failure occurred sending a message to Kafka to topic [ggstest] org.apache.kafka.common.errors.TimeoutException: Topic ggstest not  present in metadata after 60000/120000 ms.

If you encounter this message in your Replicat report file, you can:

Ensure the target topic is present or check that auto topic creation is enabled within the target Kafka settings.
Ensure that there are no firewall rules blocking traffic.
If you're running Kafka on OCI with a private endpoint, then ensure that you use the Internal FQDN as the bootstrap server in server.properties and in the Kafka connection.
If you're connecting to a Confluent Cloud with private endpoints:
- Ensure that the DNS zones and DNS records are configured properly in both OCI and the target third party cloud.
- Ensure that the network connection between OCI and the target cloud work fine.
- Test that you can connect to the target Confluent Cloud with OpenSSL (openssl s_client -connect <bootstrap>) from an OCI VM running in the same subnet connected to the third party cloud.
- Test that you can publish or consume messages from a Kafka client running on OCI within the same subnet connected to the third party cloud. If it fails, then check your network settings on both OCI and the third party cloud.

Oracle Cloud Infrastructure Documentation