4 Troubleshooting Kafka Issues

This topic describes the troubleshooting procedures for the Kafka issues.

4.1 Kafka Health

This topic describes the troubleshooting procedures for the Kafka Health.

4.1.1 Verify Kafka Health

Run the below command and verify: $ netstat -tlnp | grep :9092

Note:

9092 is default port of kafka

4.1.2 Verify Zookeeper Health

Kafka instance will not start if Zookeeper is not yet started.

1. Run the below command and verify.

   $ netstat -tlnp | grep :2181 (2181 is default port of zookeeper)

   tcp6 0 0 :::2181 :::* LISTEN 19936/java

2. To debug, check if the permissions of Kafka log folder are correct.

   The log folder path can be found by looking at the value of the property log.dirs in the server.properties file of Kafka installation.

4.2 Prometheus and Grafana

This topic describes about the Troubleshooting Kafka issues using Prometheus and Grafana.

4.2.1 Prometheus Setup

Prometheus is an open-source project, which helps monitoring of the applications metrics. It is widely used for the monitoring of Kafka and its metrics. The installer for Prometheus can be downloaded Prometheus from https://prometheus.io/download/.

4.2.2 JMX-Exporter Setup

A JMX-Exporter application is used to integrate with the Kafka broker as a Java agent to expose the values of JMX MBeans as an API. The JMX-Exporter is used by the Prometheus to fetch the values of the JMX metrics.

Perform the following steps:

1. Download the latest jmx_prometheus_javaagent jar file from the maven repository in the Kafka directory along with the bin, config directories.

   Note:

   This can be used to monitor consumer_laghttps://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.15.0/jmx_prometheus_javaagent-0.15.0.jar
2. Set the KAFKA_OPTS variable to the desired value to execute the jar as a java agent export KAFKA_OPTS="$KAFKA_OPTS -javaagent:$PWD/jmx_prometheus_javaagent-0.15.0.jar=7071:$PWD/kafka-0-8-2.yml"

   Note:

   You can choose the port according to our preference.
3. Restart Kafka Broker.

4.2.3 Grafana Setup

Perform the following steps:

1. Download Grafana from https://grafana.com/grafana/download in the stand-alone application mode, and extract its contents.
2. Go to the bin folder in the extracted contents, and start the Grafana server.

   Note:

   Grafana should start on the default port 3000 (HOST: 3000). The default user and password for Grafana are admin/admin.

Perform the following steps to integrate Grafana with the Prometheus instance installed:

3. Click on the Grafana logo to open the sidebar.
4. Click Data Sources in the sidebar.
5. Choose Add New.
6. Select Prometheus as the data source.
7. Click Add to test the connection and to save the new data source.

4.2.4 Prometheus Metrics

The Prometheus Metrics are as follows:

• process_cpu_seconds_total.
• http_request_duration_seconds.
• node_memory_usage_bytes.
• http_requests_total.
• process_cpu_seconds_total.