Kafka SSL Setup

A Kafka SSL Setup

To configure Kafka SSL, follow below steps:

Execute below commands to create certificate on local machine.
```
"<keytoolPath>" -genkeypair -alias OBCRcert -keyalg RSA -keysize 1024 -sigalg SHA512withRSA -validity 365 -keystore "<sslPath>/KafkaServerKeystore.jks"  -ext "SAN=IP:<machineIp>"
```
```
"<keytoolPath>" -export -alias OBCRcert -file <sslPath>/KafkaCert.crt -keystore <sslPath>/KafkaServerKeystore.jks -keypass <Password> -storepass <Password>
```
```
"<keytoolPath>" -import -alias OBCRcert -file <sslPath>/KafkaCert.crt -keystore <sslPath>/KafkaServerTrustStore.jks -storepass <Password>
```
```
"<keytoolPath>" -import -alias OBCRcert -file <sslPath>/KafkaCert.crt -keystore <sslPath>/KafkaClientTrustStore.jks -storepass <Password>
```
First command will prompt for the following attributes of the certificate and keystore:
1. Keystore Password: <Password>
2. Key Password: <Password>
3. First and Last Name (CN):<machineName>
  
  e.g. First and Last Name (CN):ofss-mum-xxxx.snbomprshared1.gbucdsint02bom.oraclevcn.com
4. Name of your Organizational Unit: obcr
5. Name of your Organization : Oracle Financial Services
6. Name of your City or Locality: Mumbai
7. Name of your State or Province: Maharastra
8. Two-letter CountryCode for this Unit: IN
9. Please verify provided information is correct or not as below:
  OU=obcr, O=Oracle Financial Services, L=Mumbai, ST=Maharastra, C=IN correct? [no ] : yes
Below files will be generated in ssl folder:
- KafkaServerKeystore.jks
- KafkaCert.crt
- KafkaServerTrustStore.jks
- KafkaClientTrustStore.jks
Copy generated files on env ssl folder (/scratch/ssl/kafka_cert).
Verify /scratch/obma/kafka/kafka_2.13-2.8.1/config/server.properties, password, and ssl location is correct.
Description of the illustration kafka_step3.png
Verify /scratch/obma/kafka/kafka_2.13-2.8.1/config/ssl.properties, ssl.truststore.password, username, password is correct.
Description of the illustration kafka_step4.png

Update kafka properties. Execute below queries on PLATO schema (replace machine name as per env).

update properties set value='ofss-mum-xxxx.snbomprshared1.gbucdsint02bom.oraclevcn.com' where key like '%plato.eventhub.kafka.brokers%';

update properties set value='ofss-mum-xxxx.snbomprshared1.gbucdsint02bom.oraclevcn.com' where key like '%plato.eventhub.zk.nodes%';

update properties set value='ofss-mum-xxxx.snbomprshared1.gbucdsint02bom.oraclevcn.com:9092' where key like '%spring.cloud.stream.kafka.binder.brokers%';

update properties set value='ofss-mum-xxxx.snbomprshared1.gbucdsint02bom.oraclevcn.com:2181' where key like '%spring.cloud.stream.kafka.binder.zknodes%';

update properties set value='obcr' where key like '%spring.cloud.stream.kafka.binder.jaas.options.username%';

update properties set value='obcr-secret' where key like '%spring.cloud.stream.kafka.binder.jaas.options.password%';

update properties set value='<Password>' where key like '%spring.cloud.stream.kafka.binder.configuration.ssl.truststore.password%';

Stop zookeeper and kafka.
- In putty, go to this location /scratch/obma/kafka/kafka_2.13-2.8.1/bin and run below command.
  To stop zookeeper use command:
```
./zookeeper-server-stop.sh
```
  - Verify: No zookeeper server to stop (run above command 2 times then this message is displayed)
  To Stop Kafka use command
```
./kafka-server-stop.sh
```
  - Verify: No Kafka server to stop (run above command 2 times then this message is displayed)
- If still the kafka/zoopkeeper does not stop, use the below command to stop kafka:
  - ```
  ps -ef|grep zookeeper
```
  For specific process ID use the below command:
```
  ps aux | grep "org.apache.zookeeper.server.quorum.QuorumPeerMain" | grep -v grep | awk '{print $2}'
```
  Once you run the command, kill the process ID with below command:
```
  kill -9 <process ID>
```
- ```
ps -ef|grep kafka_2.13-2.8.1
```
    For specific process ID use the below command:
```
jps | grep Kafka | awk '{print $1}'
```
    Once you run the command, kill the process ID with below command:
```
kill -9 <process ID>
```
Delete kafka logs from this location (/scratch/obma/kafka/logs and /scratch/obma/kafka/kafka_2.13-2.8.1/logs) and zookeeper logs (/tmp/zookeeper).
In Putty, go to this location /scratch/obma/kafka/kafka_2.13-2.8.1/bin and start zookeeper using command.
```
nohup ./zookeeper-server-start.sh ../config/zookeeper.properties &
```

In Putty, go to this location /scratch/obma/kafka/kafka_2.13-2.8.1/bin and execute below commands (replace machine name in command).

 ./kafka-configs.sh --zookeeper ofss-mum-xxxx.snbomprshared1.gbucdsint02bom.oraclevcn.com:2181 --alter --add-config "SCRAM-SHA-256=[password=admin-secret],SCRAM-SHA-512=[password=admin-secret]" --entity-type users --entity-name admin

 ./kafka-configs.sh --zookeeper ofss-mum-xxxx.snbomprshared1.gbucdsint02bom.oraclevcn.com:2181 --alter --add-config "SCRAM-SHA-256=[password=obcr-secret],SCRAM-SHA-512= [password=obcr-secret]" --entity-type users --entity-name obcr

In Putty, go to this location /scratch/obma/kafka/kafka_2.13-2.8.1/bin and start kafka using command:
```
nohup /scratch/obma/kafka/kafka_2.13-2.8.1/bin/kafka-server-start.sh /scratch/obma/kafka/kafka_2.13-2.8.1/config/server.properties > /scratch/obma/kafka/kafka_2.13-2.8.1/bin/start_server.log &
```
Verify: see the start log on location /scratch/obma/kafka/kafka_2.13-2.8.1/bin/start_server.log (if there is SSL handshake error is present the go for the step 11, other than any logs are present then follow the Verify Kafka is Up section below.
Login in weblogic, restart server Plato_Others_Server1 and CMC_Server4 which contains below services.
- plato-alerts-management-services
- plato-batch-servers
- cmc-advice-services
Confirm kafka error is gone in logs (/scratch/work_area/logs).

Verify Kafka is Up

Execute below command:
```
ps -ef|grep kafka_2.13-2.8.1
```
It should show pid running.
Check logs at /scratch/obma/kafka/kafka_2.13-2.8.1/bin/server.log → no ssl error should present on this file.
Check logs here /scratch/obma/kafka/kafka_2.13-2.8.1/logs → it should display topic names.

Alternatively, you can check using below commands:

netstat -tlnp | grep :9092

Verify Kafka Health

Run the below command and verify:

$ netstat -tlnp | grep 9092

Note:

9092 is default port of kafka

Verify Zookeeper Health

Kafka instance will not start if Zookeeper is not yet started.

Run the below command and verify.

$ netstat tlnp | grep :2181 (2181 is default port of zookeeper)
top6 0 0 :::2181  :::* LISTEN 19936/java

To debug, check if the permissions of Kafka log folder are correct.
The log folder path can be found by looking at the value of the property log-dirs in the server.properties file of Kafka installation.