5.16 Kafka Installation

Windows

Apache Zookeeper (Download the bin tar):

https://shaaslam.medium.com/installing-apache-zookeeper-on-windows-45eda303e835#.fgofwm6n6

Apache Kafka:

https://shaaslam.medium.com/installing-apache-kafka-on-windows-495f6f2fd3c8

Linux

Method 1(Using separate zookeeper)

Apache Zookeeper

1. Download Apache Zookeeper bin files from https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.7.1/apache-zookeeper-3.7.1-bin.tar.gz
2. Untar apache-zookeeper-3.7.1-bin.tar.gz tar -zxvf apache-zookeeper-3.7.1-bin.tar.gz
3. In conf folder, rename zoo-sample.cfg to zoo.cfg
4. Make a folder named data in /scratch/users/ofsaa/zookeeper/apache-zookeeper-3.7.1-bin
5. Change the value of dataDir in zoo.cfg to dataDir=/scratch/users/ofsaa/zookeeper/apache-zookeeper-3.7.1-bin/data
6. Open a terminal and set the below values:

   export ZOOKEEPER_HOME=/scratch/users/ofsaa/zookeeper/apache-zookeeper-3.7.1-bin

   export PATH=/scratch/users/ofsaa/zookeeper/apache-zookeeper-3.7.1-bin/bin:$PATH

7. Start the zookeeper server:cd /scratch/users/ofsaa/zookeeper/apache-zookeeper-3.7.1-bin/bin ./zkServer.sh start

A few commands for zookeeper server:

• ./zkServer.sh start
• ./zkServer.sh start-foreground
• ./zkServer.sh stop
• ./zkServer.sh status

Apache Kafka

• Download Apache Kafka bin files

  wget https://downloads.apache.org/kafka/3.3.1/kafka_2.13-3.2.0.tgz

• Untar kafka_2.13-3.2.0.tgz

  tar -zxvf kafka_2.13-3.2.0.tgz

• Make a folder "logs" in the directory /scratch/users/ofsaa/kafka/kafka_2.13-3.2.0/.
• Go to config folder in Apache Kafka and edit server.properties using any text editor. Change value of logs.dir to log.dirs=/scratch/users/ofsaa/kafka/kafka_2.13-3.2.0/logs in server.properties
• Go to config folder in Apache Kafka and edit zookeeper.properties using any text editor. Change value of dataDir dataDir=/scratch/users/ofsaa/zookeeper/apache-zookeeper-3.7.1-bin/data
• Start Apache Kafka broker

  cd /scratch/users/ofsaa/kafka/kafka_2.13-3.2.0/bin/

  ./kafka-server-start.sh ../config/server.properties

Method 2(Using zookeeper shipped with Apache Kafka)

Apache Zookeeper and Kafka

• Download Apache Kafka bin files

  wget https://downloads.apache.org/kafka/3.3.1/kafka_2.13-3.2.0.tgz

• Untar kafka_2.13-3.2.0.tgz

  tar -zxvf kafka_2.13-3.2.0.tgz

• Go to config folder in Apache Kafka and edit zookeeper.properties using any text editor. Change value of dataDir to dataDir=/scratch/users/ofsaa/zookeeper/apache-zookeeper-3.7.1-bin/data or any folder where the snapshot of zookeeper metadata can be stored.
• Make a folder "logs" in the directory /scratch/users/ofsaa/kafka/kafka_2.13-3.2.0/.
• Go to config folder in Apache Kafka and edit server.properties using any text editor. Change value of logs.dir to log.dirs=/scratch/users/ofsaa/kafka/kafka_2.13-3.2.0/logs in server.properties.

• Start Apache Zookeeper

  cd /scratch/users/ofsaa/kafka/kafka/kafka_2.13-3.2.0

  bin/zookeeper-server-start.sh config/zookeeper.properties

• Start Apache Kafka Broker

  cd /scratch/users/ofsaa/kafka/kafka_2.13-3.2.0

  bin/kafka-server-start.sh /config/server.properties