2.2 Kafka Setup

This topic provides the systematic instruction to install and setup kafka.

1. Extract the kafka installation file in /tools/kafka on both the servers.
2. Navigate to config folder in Apache Kafka (/tools/kafka/config).
3. Duplicate the server.properties from config folder and rename it to server1.properties.
4. Open server1.properties and modify the following properties.

   broker.id= (Unique Integer which identifies the kafka broker in the cluster.
   listeners=PLAINTEXT://<hostname>:<Kafka broker listen port(9092)>
   log.dirs=<Kafka home directory>/logs
   log.retention.hours= <The number of hours to keep a log file 
   before deleting it (in hours), tertiary to log.retention.ms property>
   log.retention.bytes= <The maximum size of the log before deleting it>
   log.segement.bytes= <The maximum size of a single log file>
   log.retention.check.interval.ms= <The frequency in milliseconds that
   the log cleaner checks whether any log is eligible for deletion>
   zookeeper.connect=<zookeeper_hostname_1>:<zookeeper_client_port>,
   <zookeeper_hostname_2>:<zookeeper_client_port>,<zookeeper_hostname_3>:<zookeeper_client_port>

   Example:

   
                        broker.id=0
                     

   
                        port=9092
                     

   
                        log.dirs=/tmp/kafka-oblm/logs-node1
                     

   
                        zookeeper.connect=server1-IP:2181,server2-IP:2182
                     

   
                        num.partitions=2
                     

   
                        min.insync.replicas=1
                     

   
                        default.replication.factor=2
                     

   
                        offsets.topic.replication.factor=2
                     

   
                        transaction.state.log.replication.factor=2
                     

   
                        transaction.state.log.min.isr=1
                     

   Note:

   If the Apache Zookeeper is on different server, then change the zookeeper.connect property. i.e., update the highlighted value for the respective server IPs. min.insync.replicas: A typical configuration is replication-factor minus 1.
5. Duplicate the server.properties into the same directory and rename it to server2.properties on server 2.
6. Open server2.properties and modify the following properties.

   
                     broker.id=1
                 

   
                     broker.id=1
                 

   
                     log.dirs=/tmp/kafka-oblm/logs-node2
                 

   Note:

   By default, Apache Kafka will run on port 9092 and Apache Zookeeper will run on port 2181.
7. Copy the server1.properties, server2.properties, server3.properties and paste it in local.
8. To run Kafka brokers, change path to /tools/kafka directory and run the following command in separate terminals.
   On Server 1:

   
                        nohup ./bin/kafka-server-start.sh config/server1.properties
                     

   On Server 2:

   
                        nohup ./bin/kafka-server-start.sh config/server2.properties
                     

9. The values set for Logs is under the segment: “Log Retention Policy” in server*.properties file attached in the document. The values set under this segment are defaults from Apache.
10. At present, kafka takes the default value for message size as: message.max.bytes=1000012.
11. Add and update this field in server*.properties for increasing based on requirement.
12. To add compression type for all data generated by the producer, add the following property in server*.properties file.
    compression.type=none

    Note:

    The default is none (i.e. no compression). Valid values are none, gzip, snappy, lz4, or zstd.