13.3 Install and Configure Alertmanager

Prometheus scrapes data from the MicroTx coordinator at regular intervals. Prometheus Alertmanager sends notifications based on the specified thresholds and rules. If you have configured an email ID to receive notifications, Alertmanager sends notifications to the configured email ID.

Alerts are sent based on the thresholds or rules mentioned in the prometheus.yaml file. For example, users are notified when the MicroTx service or database service is not available. You can customize the alerts. To add a custom alert, see https://prometheus.io/docs/alerting/latest/notifications/.

1. Install Prometheus. See Set Up Prometheus and Grafana.
2. Open prometheus.yaml, which is located at installation_directory/otmm-RELEASE/otmm/visualization/alert/kubernetes, and provide values for the following properties.
   • Enter the scrape_interval and evaluation_interval to specify the time interval at which Prometheus scrapes the metrics from the coordinator.

     server:
         global:
             scrape_interval: 1m
             evaluation_interval: 1m

   • Enter a unique value for job_name and specify the instance that you want to connect to. Prometheus scrapes the metrics logs from the specified instance.

     scrape_configs:
       - job_name: prometheus
         static_configs:
           - targets:
             - localhost:9090
        - job_name: "tcs"
          static_configs:
            - targets: 
              - otmm-tcs.otmm.svc.cluster.local:9000
        - job_name: kube-state-metrics
          metrics_path: /metrics
          static_configs:
            - targets:
              - prometheus-kube-state-metrics.otmm.svc.cluster.local:8080

     Where,

     • scrape_config is the name of the job in Prometheus.
     • otmm-tcs.otmm.svc.cluster.local:9000 is the name of the MicroTx coordinator service and the port number at which the service is running. Alert messages are generated for this service.
     • prometheus-kube-state-metrics.otmm.svc.cluster.local:8080 is the target from which you want to scrape the metrics. Prometheus exposes a set of metrics by default. When you install Prometheus in a cluster, a set of services are deployed which includesprometheus-kube-state-metrics. Use this service to query the state, health, or performance of any Kubernetes resource.

     This file contains details about the instance from which Prometheus scrapes the metrics and it defines the threshold for which alert messages are sent. For description about each property and the permitted values, see https://prometheus.io/docs/prometheus/latest/configuration/configuration/.

3. Run the following commands to download the Helm Charts from the prometheus-community repo.

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update

4. Reinstall Prometheus using the prometheus.yaml file that you have downloaded.

   helm install -f prometheus.yaml prometheus prometheus-community/prometheus

5. Download the Prometheus Alertmanager from https://prometheus.io/download/#alertmanager.
6. Run the following command to install Prometheus Alertmanager.

   helm install [RELEASE_NAME] prometheus-community/alertmanager

   Where, [RELEASE_NAME] is the release number of Prometheus Alertmanager that you want to install.

You can configure to receive alerts on Slack or email. Refer to the Prometheus Alertmanager documentation.