Changing Kafka Retention Policies
Redundancy of the metric database is provided by linking Kafka message buses to the ingestion pipeline. The Telegraf instance on each server is configured to send metrics to the Kafka message bus instead of directly to InfluxDB. Telegraf will failover to the redundant Kafka server as needed. An additional Telegraf instance is then installed on each metric server to read from both primary and redundant Kafka message buses and batch insert those metrics into the local InfluxDB server. This redundancy will handle failovers and split-brain scenarios to provide eventual consistency of data between both InfluxDB servers. This document will provide the steps that are needed to change how many days of data is stored in Kafka. By default, the Kafka retention policy is 3 days, but can be extended if you need more time to complete extended maintenance of your primary or backup influx database.
Steps
-
Check the existing retention period in your configuration as shown below.
Note:
In this example, the Configs field has no value listed so the retention period is 3 days until you change it.
# $A1BASEDIR/vendor/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic influxdb Topic:influxdb PartitionCount:1 ReplicationFactor:1 Configs: Topic: influxdb Partition: 0 Leader: 0 Replicas: 0 Isr: 0 -
Update the existing retention period in the configuration file:
# $A1BASEDIR/vendor/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic influxdb --config retention.ms=259200000 WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases. Going forward, please use kafka-configs.sh for this functionality Updated config for topic "influxdb". -
Next, check the configuration again. Notice that the Configs field has changed in this example and now shows a retention value.
# $A1BASEDIR/vendor/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic influxdb Topic:influxdb PartitionCount:1 ReplicationFactor:1 Configs:retention.ms=259200000 Topic: influxdb Partition: 0 Leader: 0 Replicas: 0 Isr: 0