Connect to Confluent Kafka
Learn to create a connection to Confluent Kafka, which you use with the Confluent Schema Registry connection, to serve as a source or target in an OCI GoldenGate Big Data deployment.
Before you begin
Before you create the connection, ensure that you:
- Review how OCI GoldenGate connects to your source and targets.
- Configure the required policies to enable secure Vault and Secrets access, such as use secrets, use vaults, and read secret-bundles. For more information, see Minimum recommended policies.
Create a source connection
To create a source Confluent Kafka connection:
The connection appears in the Connections list, where you can select it
to view its details. Ensure that you also create a connection to Confluent Schema Registry, and then assign both
connections to a Big Data deployment.
Create a target connection
To create a target Confluent Kafka connection:
The connection appears in the Connections list, where you can select it
to view its details. Ensure that you also create a connection to Confluent Schema Registry, and then assign both
connections to a Big Data deployment.
Create a connection to Confluent Cloud with Private Links
Private Link lets you access your Confluent Cloud cluster running on a third party cloud through a private endpoint that exists in your virtual network.
Before you create the connection, ensure you have the following:
- Create private network connectivity between Oracle Cloud Infrastructure (OCI) and the target third party cloud.
- While adding network configuration for private link in Confluent Cloud, ensure that you select Private DNS Resolution.
- Configure DNS zones and set up DNS records in the third party cloud where you configured Confluent Cloud and in OCI. In OCI, you can create zones within your VCN's private views. Within zones, you can add the required DNS records.
You can use the instructions above to create the connection, but in place of Steps 5 and 6, do the following:
- For Step 3e:
- Provide the Bootstrap servers host and port details.
- You can add multiple Bootstrap servers.
- For 3f:
- For Security protocol, select SASL over Plaintext.
- Enter the username and password.
Troubleshoot Kafka connection errors
Most connection issues result in TimeoutException errors. For example:
A failure occurred sending a message to Kafka to topic [ggstest] org.apache.kafka.common.errors.TimeoutException: Topic ggstest not present in metadata after 60000/120000 ms.
If you encounter this message in your Replicat report file, you can:
- Ensure the target topic is present or check that auto topic creation is enabled within the target Kafka settings.
- Ensure that there are no firewall rules blocking traffic.
- If you're running Kafka on OCI with a private endpoint, then ensure that you use the Internal FQDN as the bootstrap server in server.properties and in the Kafka connection.
- If you're connecting to a Confluent Cloud with private endpoints:
- Ensure that the DNS zones and DNS records are configured properly in both OCI and the target third party cloud.
- Ensure that the network connection between OCI and the target cloud work fine.
- Test that you can connect to the target Confluent Cloud with OpenSSL
(
openssl s_client -connect <bootstrap>
) from an OCI VM running in the same subnet connected to the third party cloud. - Test that you can publish or consume messages from a Kafka client running on OCI within the same subnet connected to the third party cloud. If it fails, then check your network settings on both OCI and the third party cloud.