Kafka Installation

Windows

Apache Zookeeper (Download the bin tar):

https://shaaslam.medium.com/installing-apache-zookeeper-on-windows-45eda303e835#.fgofwm6n6

Apache Kafka:

https://shaaslam.medium.com/installing-apache-kafka-on-windows-495f6f2fd3c8

Linux

Method 1(Using separate zookeeper)

Apache Zookeeper
  1. Download Apache Zookeeper bin files from https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.7.1/apache-zookeeper-3.7.1-bin.tar.gz
  2. Untar apache-zookeeper-3.7.1-bin.tar.gz tar -zxvf apache-zookeeper-3.7.1-bin.tar.gz
  3. In conf folder, rename zoo-sample.cfg to zoo.cfg
  4. Make a folder named data in /scratch/users/ofsaa/zookeeper/apache-zookeeper-3.7.1-bin
  5. Change the value of dataDir in zoo.cfg to dataDir=/scratch/users/ofsaa/zookeeper/apache-zookeeper-3.7.1-bin/data
  6. Open a terminal and set the below values:

    export ZOOKEEPER_HOME=/scratch/users/ofsaa/zookeeper/apache-zookeeper-3.7.1-bin

    export PATH=/scratch/users/ofsaa/zookeeper/apache-zookeeper-3.7.1-bin/bin:$PATH

  7. Start the zookeeper server:cd /scratch/users/ofsaa/zookeeper/apache-zookeeper-3.7.1-bin/bin ./zkServer.sh start
A few commands for zookeeper server:
  • ./zkServer.sh start
  • ./zkServer.sh start-foreground
  • ./zkServer.sh stop
  • ./zkServer.sh status

Apache Kafka

  • Download Apache Kafka bin files

    wget https://downloads.apache.org/kafka/3.3.1/kafka_2.13-3.2.0.tgz

  • Untar kafka_2.13-3.2.0.tgz

    tar -zxvf kafka_2.13-3.2.0.tgz

  • Make a folder "logs" in the directory /scratch/users/ofsaa/kafka/kafka_2.13-3.2.0/.
  • Go to config folder in Apache Kafka and edit server.properties using any text editor. Change value of logs.dir to log.dirs=/scratch/users/ofsaa/kafka/kafka_2.13-3.2.0/logs in server.properties
  • Go to config folder in Apache Kafka and edit zookeeper.properties using any text editor. Change value of dataDir dataDir=/scratch/users/ofsaa/zookeeper/apache-zookeeper-3.7.1-bin/data
  • Start Apache Kafka broker

    cd /scratch/users/ofsaa/kafka/kafka_2.13-3.2.0/bin/

    ./kafka-server-start.sh ../config/server.properties

Method 2(Using zookeeper shipped with Apache Kafka)

Apache Zookeeper and Kafka
  • Download Apache Kafka bin files

    wget https://downloads.apache.org/kafka/3.3.1/kafka_2.13-3.2.0.tgz

  • Untar kafka_2.13-3.2.0.tgz

    tar -zxvf kafka_2.13-3.2.0.tgz

  • Go to config folder in Apache Kafka and edit zookeeper.properties using any text editor. Change value of dataDir to dataDir=/scratch/users/ofsaa/zookeeper/apache-zookeeper-3.7.1-bin/data or any folder where the snapshot of zookeeper metadata can be stored.
  • Make a folder "logs" in the directory /scratch/users/ofsaa/kafka/kafka_2.13-3.2.0/.
  • Go to config folder in Apache Kafka and edit server.properties using any text editor. Change value of logs.dir to log.dirs=/scratch/users/ofsaa/kafka/kafka_2.13-3.2.0/logs in server.properties.
  • Start Apache Zookeeper

    cd /scratch/users/ofsaa/kafka/kafka/kafka_2.13-3.2.0

    bin/zookeeper-server-start.sh config/zookeeper.properties

  • Start Apache Kafka Broker

    cd /scratch/users/ofsaa/kafka/kafka_2.13-3.2.0

    bin/kafka-server-start.sh /config/server.properties

Create Keys and Certificates
  • keytool -keystore kafka.server.keystore.jks -alias localhost -keyalg RSA -validity 365 -genkey -storepass oracle123 -keypass oracle123 -ext SAN=DNS:ofss-mum-1035.snbomprshared1.gbucdsint02bom.oraclevcn.com
  • openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
  • keytool -keystore kafka.client.truststore.jks -alias CARoot -storepass oracle123 -keypass oracle123 -importcert -file ca-cert keytool -keystore kafka.server.truststore.jks -alias CARoot -storepass oracle123 -keypass oracle123 -importcert -file ca-cert
  • keytool -keystore kafka.server.keystore.jks -alias localhost -storepass oracle123 -keypass oracle123 -certreq -file cert-file
  • openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:oracle123
  • keytool -keystore kafka.server.keystore.jks -alias CARoot -storepass oracle123 -keypass oracle123 -importcert -file ca-cert keytool -keystore kafka.server.keystore.jks -alias localhost -storepass oracle123 -keypass oracle123 -importcert -file cert-signed
Converting Keystore file from jks to pem
  • keytool -exportcert -alias localhost -keystore kafka.server.keystore.jks -rfc -file certificate.pem -storepass oracle123
  • keytool -v -importkeystore -srckeystore kafka.server.keystore.jks -srcalias localhost -keystore cert_and_key.p12 -storetype PKCS12 -storepass oracle123 -srcstorepass oracle123
  • openssl pkcs12 -in cert_and_key.p12 -nodes -nocerts -out key.pem -passin pass:oracle123
  • keytool -exportcert -alias CARoot -keystore kafka.server.keystore.jks -rfc -file CARoot.pem -storepass oracle123
Inside the Kafka/config path we have to create client-ssl.properties and server-ssl.properties.
The client-ssl.properties file contains the following:
  • security.protocol=SSL
  • ssl.truststore.location=/scratch/mmg8131/kafka/ssl/kafka.server.truststore.jks
  • ssl.truststore.password=oracle123
  • ssl.keystore.location=/scratch/mmg8131/kafka/ssl/kafka.server.keystore.jks
  • ssl.keystore.password=oracle123
  • ssl.key.password=oracle123
server-ssl.properties contains the same content as server.properties, but we have to update the ssl parameters in it.