Oracle Cloud Infrastructure Documentation

Managing Cluster Configurations

Each Kafka cluster  in Streaming with Apache Kafka has a cluster configuration file that contains all the server configuration properties.

You must also manage client configurations.

Configurable Broker Properties

With Streaming with Apache Kafka, you can customize the Kafka configuration by updating specific broker properties. The service streamlines Kafka configuration by limiting the properties that can be updated. Validate the cluster configuration file against the supported properties to avoid unexpected behavior.

If invalid or incorrect configurations are provided, they're ignored and default values as defined by Apache Kafka are used. This ensures cluster stability and prevents misconfiguration from affecting service availability.

For the default configurations set by OCI Streaming with Apache Kafka, see Cluster Default Options.

You can update the following broker properties:

Property Sample Value Purpose
client.quota.callback.static.storage.soft 97% of totalStorageBytes Sets a soft limit on the broker disk storage capacity.
client.quota.callback.static.storage.hard 98% of totalStorageBytes Sets a hard limit on the broker disk storage capacity.
default.replication.factor 1 Sets the default replication factor for new topics.
log.retention.hours 168 Sets the minimum time to retain log segments.
log.segment.bytes 1073741824 Sets the maximum size of a single log segment file.
log.cleanup.policy delete Sets the log retention strategy (delete or compact).
message.max.bytes 1048588 Sets the maximum size of a message that the broker accepts.
min.insync.replicas 1 Sets the minimum number of in-sync replicas required for a write to be acknowledged.
num.io.threads 8 Sets the number of threads for disk I/O operations.
num.replica.fetchers 1 Sets the number of threads used to replicate data from other brokers.
unclean.leader.election.enable false Allows out-of-sync replicas to be elected as leaders.

Define custom broker disk quotas

You can define custom broker disk quotas.

  1. Create or update the cluster configuration JSON file with the required quota configuration. For example, update file kafka-config.json with the following details about the disk quotas:
    {
"properties": {
 "client.quota.callback.class": "io.strimzi.kafka.quotas.StaticQuotaCallback",
 "client.quota.callback.static.storage.soft": "1000000000",
 "client.quota.callback.static.storage.hard": "2000000000",
 "client.quota.callback.static.storage.check-interval": "30"
 }
}
  2. Specify the configuration file created or updated in the previous step as the configuration file for the cluster. For example, specify --latest-config file://kafka-config.json in the create or update cluster-config CLI command.
  3. Update the Kafka cluster using API or CLI and specify the new or updated configuration file kafka-config.json.
Note

You can't set disk quota values higher than the default limits.

Non Configurable Broker Properties

Streaming with Apache Kafka sets some broker properties as non editable and if you configure these properties they're ignored by the service. You must validate the configuration file against the support properties to avoid unexpected behavior.

You can't update the following broker properties:

  • advertised.
  • authorizer.
  • broker.
  • controller
  • cruise.control.metrics.reporter.bootstrap.
  • cruise.control.metrics.topic
  • host.name
  • inter.broker.listener.name
  • listener.
  • listeners.
  • log.dir
  • password.
  • port
  • process.roles
  • sasl.
  • security.
  • servers.node.id
  • ssl.
  • super.user

For example, Streaming with Apache Kafka doesn't support plaintext listeners and hence configuring the setting "listeners": "PLAINTEXT://:9092" in the cluster configuration file isn't supported too. If you apply this setting to configure a plaintext listener, the configuration is silently ignored and the plaintext listener is not made active and a warning is logged.

Following is an example of invalid custom configurations. Here, all settings are ignored and defaults applied.

{
  "listeners": "PROTOCOL://:9092",                  // Ignored: Protocol not supported.
  "num.network.threads": -5,                        // Ignored: Cannot be negative.
  "num.io.threads": "eight",                        // Ignored: Should be an integer (like, 8).
  "socket.send.buffer.bytes": 0,                    // Ignored: Value is too low.
  "socket.receive.buffer.bytes": 9999999999,        // Ignored: Value is too high.
  "socket.request.max.bytes": -104857600,           // Ignored: Negative value.
  "log.retention.hours": "one hundred",             // Ignored: Should be an integer.
  "log.segment.bytes": 123.45,                      // Ignored: Should be an integer, not a decimal.
  "log.retention.check.interval.ms": "five minutes",// Ignored: Should be an integer in ms.
  "num.partitions": 0,                              // Ignored: Should be at least 1.
  "default.replication.factor": 5,                  // Ignored: May fail if fewer than 5 brokers.
  "min.insync.replicas": 10,                        // Ignored: Higher than replication factor.
  "message.max.bytes": -1000012,                    // Ignored: Negative value.
  "replica.fetch.max.bytes": "1MB",                 // Ignored: Should be an integer.
  "offsets.topic.replication.factor": "three",      // Ignored: Should be an integer.
  "transaction.state.log.replication.factor": -1,   // Ignored: Negative value.
  "transaction.state.log.min.isr": 0                // Ignored: Should be at least 1.
}