Using Kafka APIs

This topic describes how to configure Apache Kafka for API compatibility with Oracle Cloud Infrastructure Streaming. When your producers use Kafka APIs to interact with Streaming the decision of which partition to publish a unique message to is handled client-side by Kafka.

Please refer to Kafka API Support for additional information.

Endpoints

For bootstrap servers, use your region endpoint on port 9092. For example:

streaming.us-phoenix-1.oci.oraclecloud.com:9092

Authentication

Authentication with the Kafka protocol uses auth tokens and the SASL/PLAIN mechanism. You can generate tokens in the Console user details page. See Working with Auth Tokens for more information.

Tip

It's a good idea to create a dedicated group/user and grant that group the permission to manage streams in the appropriate compartment or tenancy. You then can generate an auth token for the user you created and use it in your Kafka client configuration.

Your username should be in the format:

tenancyName/username/streamPoolId

Kafka Configuration

You need to set up the following properties for your Kafka client.

For the Java SDK

Recommended settings for Java SDK:

Properties properties = new Properties();
	properties.put("bootstrap.servers", "streaming.{region}.oci.oraclecloud.com:9092");
	properties.put("security.protocol", "SASL_SSL");
	properties.put("sasl.mechanism", "PLAIN");
	properties.put("sasl.jaas.config", "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"{tenancyName}/{username}/{streamPoolId}\" password=\"{authToken}\";");

Recommended settings for Java SDK producers:

properties.put("retries", 5); // retries on transient errors and load balancing disconnection
		properties.put("max.request.size", 1024 * 1024); // limit request size to 1MB

Recommended settings for Java SDK consumers:

properties.put("max.partition.fetch.bytes", 1024 * 1024); // limit request size to 1MB per partition
For the Librdkafka SDK

Recommended settings for Librdkafka SDK:

'metadata.broker.list': 'streaming.{region}.oci.oraclecloud.com:9092',
			 'security.protocol': 'SASL_SSL',
		 	 'sasl.mechanisms': 'PLAIN',
			 'sasl.username': '{tenancyName}/{username}/{streamPoolID}',
		         'sasl.password': '{authToken}'

Recommended settings for Librdkafka SDK producers:

'message.send.max.retries': 5 // retries on transient errors and load balancing disconnection
		'max.request.size': 1024 * 1024 // limit request size to 1 MB

Recommended settings for Librdkafka SDK consumers:

'max.partition.fetch.bytes': 1024 * 1024 // limit request size to 1 MB per partition

For More Information