Managing Grafana Alerts
Private Cloud Appliance includes a set of predefined Grafana alerts. You can also add your own alerts.
You can show only alerts that are in a specified state, such as Alerting. You can display detailed information about the alert, including the values that trigger the alert and that trigger a state change. In many cases, you can change these values. If an alert is in the Alerting state, view the alert definition to determine what caused the alert to go to that state, and then use this information to evaluate the component that the alert is monitoring and determine what action might be needed.
Browsing Grafana Alerts
To view all alerts, click the bell icon in the vertical menu bar on the left side the page. An icon shows the status of each alert, and text below the alert name shows how long the alert has been in that status.
Enter text in the search field at the top of the list to show only alerts with that text in their names. Use the States list to show only alerts that are in the selected state: OK, Not OK, Alerting, No Data, Paused, Pending.
Use the "How to add an alert" button above the alerts list to create a new alert, or use information referenced on the Grafana home page to add or modify an alert, add a notification channel, and add a notification for a particular alert.
Click an alert name to see detailed information about the alert. This is the same page you see if you go to the dashboard, scroll to the metric, click the metric name, and select Edit.
Hover over the graph to list all data that is being monitored, for example each host, switch, device, or endpoint.
On the Alert tab below the graph, you can view and edit the rule. An alert rule consists of one or more queries and expressions, a condition, the frequency of evaluation, and optionally, the duration over which the condition is met. You can see how the alert state is set for various error conditions. You can send a notification message for this alert.
The state history button shows the last 50 state changes for this alert. Another button enables you to test the alert.
Adding or Configuring Notification Channels
To add or configure notification channels, click the bell icon in the menu bar on the left side the page and then select the Notification channels option, or select the Notification channels tab at the top of the list of alerts.
To change the configuration of an existing notification channel, click the name of the channel. When you are finished making changes, click the Save button. Click the Test button to send a test notification.
To add a notification channel, go to the Notification channels tab, click the New channel button, and fill out the page. Click the Save button. Click the Test button to send a test notification. Click the Back button to cancel and not create a new notification channel.
Configuring Custom External Email Notifications
To configure email notification, open a service request (SR) for Oracle Support to do the initial configuration. When the initial configuration is complete, go to the Grafana alerts page, click the Notification channels tab, click the New channel button, and fill out the page, selecting Email in the Type field.
Configuring Custom External HTTP/HTTPS Notifications
To configure external HTTP or HTTPS based custom alerts, you must first configure the proxy for Grafana as shown in the following example.
Log in to the management node that owns the management virtual IP, and run the following command:
$ sudo curl -u admin_user_name -XPUT \
'https://api.PCA_system_name.your_domain/v1/grafana/proxy/config?http-proxy=proxy_fqdn:proxy_port&https-proxy=proxy_fqdn:proxy_port'
Enter host password for user 'admin_user_name':
Grafana proxy config successfully updated!Run the following command so that Grafana can still contact the internal Loki and Prometheus services:
$ sudo curl -u admin_user_name -XPUT \
'https://api.PCA_system_name.your_domain/v1/grafana/proxy/config?no-proxy="sauron-sauron-prometheus,sauron-sauron-alertmanager,grafana-loki.loki.svc.cluster.local"'The Grafana pod is restarted. Run the following command until you see that the Grafana pod (sauron-sauron-grafana-unique_ID) is running:
$ kubectl get pods -n sauron