Recovering Metric Data from Kafka Logs

Use this optional manual recovery procedure only when the normal Kafka and Telegraf recovery flow does not restore metric data after an outage and retained Kafka backup log files are available. For information about the normal automatic recovery flow, see Metric Database Scalability and Redundancy.

KafkaLogsToCuratedMessages is a Unified Assurance command-line utility that prepares retained Kafka backup log files for manual import. Use it when metric messages cannot be restored through the normal recovery flow, such as after a prolonged outage or a failed automatic replay. The utility extracts the Kafka message payloads into InfluxDB line-protocol files. Telegraf then imports the files into the raw retention policy in the Metric database and the MetricConsolidator generates the hourly and daily data for the recovered time range.

To recover metric data manually:

Curate Kafka backup log files
Update the InfluxDB Raw Retention Period, if necessary.
Configure Telegraf
Import the recovered data
Run MetricConsolidator

Prerequisites

Before you begin, ensure that the following requirements are met:

Unified Assurance is installed on the server where you run the recovery. The utilities-app package includes KafkaLogsToCuratedMessages.pl, which uses Unified Assurance configuration and logging libraries.
The Unified Assurance Kafka and Telegraf packages are installed on the recovery server.
You have the Kafka backup log files for the metric data that you need to recover.
Writable locations and absolute paths for the following directories:
- <kafka-backup-logs-directory>: Contains the Kafka backup log files to recover.
- <processed-directory>: Receives files that Telegraf imports successfully.
- <error-directory>: Receives files that Telegraf cannot import.

Note:

The Kafka backup log directory does not need to be under $A1BASEDIR.

Curating Kafka Backup Log Files

To create InfluxDB line-protocol files from the Kafka backup logs:

As root user, run the following command:
```
source /opt/assure1/.assure1_bashrc
```
Create the directory for the Kafka backup logs.
```
mkdir -p <kafka-backup-logs-directory>
```
Copy the Kafka backup .log files to <kafka-backup-logs-directory>.
Switch to the utility directory:
```
cd $A1BASEDIR/bin/util/app/kafka
```
Run the following command to create InfluxDB line-protocol files from the Kafka backup log files:
```
./KafkaLogsToCuratedMessages.pl \
  --LogsDir <kafka-backup-logs-directory> \
  --KafkaDir $A1BASEDIR/vendor/kafka
```
The --LogsDir and --KafkaDir options are required. You can optionally use --Silent to suppress normal console output, or --Help to display usage information without processing files.

The utility creates a readable raw-data file for each Kafka log file, extracts the metric payload messages, and creates a curated .txt file in InfluxDB line-protocol format for Telegraf to import.
Verify that <kafka-backup-logs-directory>/curated contains the expected .txt files.

If the utility fails, review $A1BASEDIR/logs/KafkaLogsToCuratedMessages.log.

Updating the InfluxDB Raw Retention Period

Before importing recovered data, compare the timestamp of the oldest Kafka backup log with the current raw retention period on each destination InfluxDB instance. If the recovered data is older than the retention period, temporarily extend the raw retention period. This prevents InfluxDB from removing the recovered raw metrics immediately after Telegraf imports them.

To check and extend the raw retention period:

Open the InfluxDB shell as root:
```
a1influxroot
```
Check the current retention policies:
```
SHOW RETENTION POLICIES ON "Metric";
```
If necessary, extend the raw retention period before you start Telegraf.

For example, run the following command to retain raw metrics for 30 days:
```
ALTER RETENTION POLICY "raw" ON "Metric" DURATION 30d;
```
After recovery and metric consolidation complete, restore the previous raw retention period similarly. See Changing Metric InfluxDB Retention Policies for more information.

Configuring Telegraf

To configure Telegraf to import the curated files:

On the recovery server, open the following Telegraf recovery configuration file in a text editor:
```
$A1BASEDIR/bin/util/app/kafka/telegraf-kafka-data-recovery.conf
```
In the [agent] section, set hostname to the hostname of the server where Telegraf runs:
In the [[outputs.influxdb]] section, update the following values as needed:
- urls: Specify the URL of each destination InfluxDB instance. For example: https://<influxdb-hostname>:8086.
- database: Set to Metric.
- retention_policy: Set to raw.
- tls_ca: Specify the path to the certificate authority (CA) certificate that validates the destination InfluxDB server certificate.
- tls_cert: Specify the path to the client certificate that Telegraf uses to authenticate to InfluxDB.
- tls_key: Specify the path to the private key for the client certificate.
Note:

To import recovered data to a redundant InfluxDB server, copy the complete [[outputs.influxdb]] section and paste it immediately after the existing section in telegraf-kafka-data-recovery.conf. Update the copied section with the redundant server’s URLs and TLS certificate paths. Skip this step if you only need to import data to one InfluxDB instance.

In the [[inputs.directory_monitor]] section, update the directory paths:

directory = "<kafka-backup-logs-directory>/curated"
finished_directory = "<processed-directory>"
error_directory = "<error-directory>"

Create the processed and error directories:
```
mkdir -p <processed-directory>
mkdir -p <error-directory>
```
Telegraf moves successfully imported files to <processed-directory> and files that fail import to <error-directory>.

Importing the Recovered Data

Run Telegraf as a Broker service or from the command line.

To run Telegraf as a Broker service on a server where Unified Assurance is installed:

From the main navigation menu, select Configuration, then Broker Control, and then Services.
Clone the Telegraf-Kafka service.

Open the cloned service and set its arguments to the following value:

--config /opt/assure1/bin/util/app/kafka/telegraf-kafka-data-recovery.conf

Set the service Status to Enabled.
Start the service.
Review $A1BASEDIR/logs/VendorTelegraf-Kafka.log to verify that Telegraf starts, processes the curated files, and connects to InfluxDB without errors.

To run Telegraf manually from the command line, run the following command:

$A1BASEDIR/vendor/telegraf/telegraf --config $A1BASEDIR/bin/util/app/kafka/telegraf-kafka-data-recovery.conf

Note:

If telegraf-kafka-data-recovery.conf contains $BASEDIR values, set BASEDIR to the Unified Assurance installation directory before running Telegraf.

Verifying the Recovery

After Telegraf starts, verify the following results:

Telegraf reads the curated .txt files from <kafka-backup-logs-directory>/curated.
Telegraf moves successfully imported files to <processed-directory>.
Telegraf moves files that fail import to <error-directory>.
$A1BASEDIR/logs/VendorTelegraf-Kafka.log contains no processing or InfluxDB connection errors.

Running MetricConsolidator

After the recovered raw data is written to InfluxDB, run MetricConsolidator for the recovered time range to generate hourly and daily metric data. See MetricConsolidator for more information.

Title and Copyright Information

Implementation Guide

G49441-03