Monitoring and Diagnostics

The following section explains how to setup and use Grafana dashboards and event notifications for the virtualization environment, and about Performance Co-Pilot metrics.

Using Event Notifications

The following section explains how to set up event notifications to monitor events in the virtualization environment. You can configure the Manager to send event notifications in email to alert specified users when certain events occur or enable Simple Network Management Protocol (SNMP) traps to monitor the virtualization environment.

For more information, see Event Logging and Notifications in the Oracle Linux Virtualization Manager: Architecture and Planning Guide.

Configuring Event Notification Services on the Engine

For event notifications to be sent properly to email recipients, you must configure the mail server on the Engine and enable ovirt-engine-notifier service. For more information about creating event notifications in the Administration portal, see Creating Event Notifications in the Administration Portal.

To configure notification services on the Engine:

Copy the ovirt-engine-notifier.conf to a new file named 90-email-notify.conf.

cp /usr/share/ovirt-engine/services/ovirt-engine-notifier/ovirt-engine-notifier.conf/ etc/ovirt-engine/notifier/notifier.conf.d/90-email-notify.conf

Edit the 90-email-notify.conf file by deleting everything except the EMAIL Notifications section.

Note:

If you plan to also configure SNMP traps in the virtualization environment, you can also copy the values from the SNMP_TRAP Notifications section of the ovirt-notifier.conf file to a file named 20-snmp.conf. For more information, see Configuring the Engine to Send SNMP Traps.

Enter the correct email variables. This file overrides the values in the original ovirt-engine-notifier.conf file.

---------------------
# EMAIL Notifications #
---------------------

# The SMTP mail server address. Required.
MAIL_SERVER=myemailserver.mycompany.com

# The SMTP port (usually 25 for plain SMTP, 465 for SMTP with SSL, 587 for SMTP with TLS)
MAIL_PORT=25

# Required if SSL or TLS enabled to authenticate the user. Used also to specify 'from' 
  user address if mail server
# supports, when MAIL_FROM is not set. Address is in RFC822 format
MAIL_USER=email.example.com

# Required to authenticate the user if mail server requires authentication or if SSL or 
  TLS is enabled
SENSITIVE_KEYS="${SENSITIVE_KEYS},MAIL_PASSWORD"
MAIL_PASSWORD=

# Indicates type of encryption (none, ssl or tls) should be used to communicate with 
  mail server.
MAIL_SMTP_ENCRYPTION=none

# If set to true, sends a message in HTML format.
HTML_MESSAGE_FORMAT=false

# Specifies 'from' address on sent mail in RFC822 format, if supported by mail server.
MAIL_FROM=myovirtengine@mycompany.com

# Specifies 'reply-to' address on sent mail in RFC822 format.
MAIL_REPLY_TO=myusername@mycompany.com

# Interval to send smtp messages per # of IDLE_INTERVAL
MAIL_SEND_INTERVAL=1

# Amount of times to attempt sending an email before failing.
MAIL_RETRIES=4

Note:

For information about the other parameters available for event notification in the ovirt-engine-notifier.conf file, see oVirt Documentation.

Enable and restart the ovirt-engine-notifier service to activate the changes.

systemctl daemon-reload

systemctl enable ovirt-engine-notifier.service

systemctl restart ovirt-engine-notifier.service

Creating Event Notifications in the Administration Portal

Before creating event notifications, you must have access to an email server that can handle incoming automated messages and deliver these messages to a distribution list. Also, configure event notification services on the Engine. For more information, see Configuring Event Notification Services on the Engine.

To create event notifications in the Administration Portal:

Go to Administration and then select Users.

The Users pane opens.
Under the User Name column, select the name of the user to display the detailed view for the user.

Note:

A user doesn't appear in the Administration Portal until the user is created and assigned appropriate permissions. For more information, see Creating a New User Account.
Select the Event Notifier tab.
Select Manage Events.

The Add Event Notification dialog box opens.
Select the events for which you want to create notifications by selecting the checkbox next to individual events or event topic areas for notification.

The events available for notification are grouped under topic areas. By default, selecting the checkbox for a top-level topic area, such as General Host Events, selects all events under that topic area. You can optionally expand or collapse all the event topic areas by selecting Expand All or Collapse All. You can also select the arrow icon next to a specific top-level topic area to expand or collapse the events associated with a specific topic area.
For the Mail Recipient field, enter an email address.
Select OK to save the changes.

Canceling Event Notifications in the Administration Portal

To cancel event notifications in the Administration Portal:

Go to Administration and then select Users.

The Users pane opens.
Under the User Name column, select the name of the user to display the detailed view for the user.
Select the Event Notifier tab.
Select Manage Events.

The Add Event Notification dialog box opens.
Select Expand All, or the topic-specific expansion options, to display the events.
Clear the appropriate check boxes to cancel the notification for that event.
Select OK to save the changes.

Configuring the Engine to Send SNMP Traps

You can configure the Manager to send SNMP traps to one or more external SNMP managers. SNMP traps contain system event information that are used to monitor the virtualization environment. The number and type of traps sent to the SNMP manager can be defined within the Engine.

Before performing this task, you must have configured one or more external SNMP managers to receive traps, and know the following details:

The IP addresses or fully-qualified domain names of machines that act as SNMP managers. Optionally, determine the port through which the SNMP manager receives trap notifications; the default UDP port is 162.
The SNMP community. Several SNMP managers can belong to a single community. Management systems and agents can communicate only if they're within the same community. The default community is public.
The trap object identifier for alerts. The Engine provides a default OID of 1.3.6.1.4.1.2312.13.1.1. All trap types are sent, appended with event information, to the SNMP manager when this OID is defined.
Note:
- Changing the default trap prevents generated traps from complying with the Engine's management information base.
- The Engine provides management information bases at /usr/share/doc/ovirt-engine/mibs/OVIRT-MIB.txt and /usr/share/doc/ovirt-engine/mibs/REDHAT-MIB.txt. Load the MIBs in the SNMP manager before proceeding.

To configure SNMP traps on the Engine:

Sign in to the host that's running the Manager.
On the Engine, create the SNMP configuration file:
```
vi /etc/ovirt-engine/notifier/notifier.conf.d/20-snmp.conf
```
Default SNMP configuration values exist on the Engine in the events notifications configuration file (ovirt-engine-notifier.conf), which is available at the following directory path: /usr/share/ovirt-engine/services/ovirt-engine-notifier/ovirt-engine-notifier.conf. The values provided in this step are based on the default or example values provided in that file. To persist configuration settings across reboots, define an override file for the SNMP configuration (20-snmp.conf), rather than edit the ovirt-engine-notifier.conf file, For more information, see Configuring Event Notification Services on the Engine.

Specify the SNMP manager, the SNMP community, and the OID in the following format:

SNMP_MANAGERS="manager1.example.com manager2.example.com:162"
SNMP_COMMUNITY=public
SNMP_OID=1.3.6.1.4.1.2312.13.1.1

The following values can be configured in the 20-snmp.conf file.

#-------------------------#
# SNMP_TRAP Notifications #
#-------------------------#
# Send v2c snmp notifications

# Minimum SNMP configuration
#
# Create /etc/ovirt-engine/notifier/notifier.conf.d/20-snmp.conf with:
# SNMP_MANAGERS="host"
# FILTER="include:*(snmp:) ${FILTER}"

# Default whitespace separated IPv4/[IPv6]/DNS list with optional port, default is 162.
# SNMP_MANAGERS="manager1.example.com manager2.example.com:164"
SNMP_MANAGERS=

# Default SNMP Community String.
SNMP_COMMUNITY=public

# SNMP Trap Object Identifier for outgoing notifications.
# { iso(1) org(3) dod(6) internet(1) private(4) enterprises(1) redhat(2312) ovirt(13) 
  engine(1) notifier(1) }
#
# Note: changing the default will prevent generated traps from complying with 
  OVIRT-MIB.txt.
SNMP_OID=1.3.6.1.4.1.2312.13.1.1

# Default SNMP Version. SNMP version 2 and version 3 traps are supported
# 2 = SNMPv2
# 3 = SNMPv3
SNMP_VERSION=2

# The engine id used for SNMPv3 traps
SNMP_ENGINE_ID=

# The user name used for SNMPv3 traps
SNMP_USERNAME=

# The SNMPv3 auth protocol. Supported values are MD5 and SHA.
SNMP_AUTH_PROTOCOL=

# The SNMPv3 auth passphrase, used when SNMP_SECURITY_LEVEL is set to AUTH_NOPRIV 
  and AUTH_PRIV
SNMP_AUTH_PASSPHRASE=

# The SNMPv3 privacy protocol. Supported values are AES128, AES192 and AES256.
# Be aware that AES192 and AES256 are not defined in RFC3826, so please verify
# that your SNMP server supports those protocols before enabling them.
SNMP_PRIVACY_PROTOCOL=

# The SNMPv3 privacy passphrase, used when SNMP_SECURITY_LEVEL is set to AUTH_PRIV
SNMP_PRIVACY_PASSPHRASE=

# The SNMPv3 security level.
# 1 = NOAUTH_NOPRIV
# 2 = AUTH_NOPRIV
# 3 = AUTH_PRIV
SNMP_SECURITY_LEVEL=1

# SNMP profile support
#
# Multiple SNMP profiles are supported.
# Specify profile settings by using _profile suffix,
# for example, to define a profile to sent specific
# message to host3, specify:
# SNMP_MANAGERS_profile1=host3
# FILTER="include:VDC_START(snmp:profile1) ${FILTER}"

Define which events to send to the SNMP Manager.
By default, the following default filter is defined in the ovirt-engine-notifier.conf file; if you don't override this filter or apply overriding filters, no notifications are sent.
```
FILTER="exclude:\*"
```
The following are other common examples of event filters.
- Send all events to the default SNMP profile.
```
FILTER="include:\*(snmp:) ${FILTER}"
```
- Send all events with the severity ERROR or ALERT to the default SNMP profile:
```
FILTER="include:\*:ERROR(snmp:) ${FILTER}"
 FILTER="include:\*:ALERT(snmp:) ${FILTER}"
```
Save the file.

Start the ovirt-engine-notifier service, and ensure that this service starts on boot.

 # systemctl start ovirt-engine-notifier.service

systemctl enable ovirt-engine-notifier.service

(Optional) Validate that traps are being sent to the SNMP Manager.

Using Grafana

The following section explains how to setup and use Grafana dashboards in the virtualization environment.

Caution:

You must install the Data Warehouse database, the Data Warehouse service and Grafana all on the same machine, even though you can install each of these components on separate machines from each other.

For more information, see Monitoring with Grafana in the Oracle Linux Virtualization Manager: Architecture and Planning Guide.

For more information on using Grafana, see the Grafana website.

Installing Grafana

Grafana integration is enabled and installed by default when you run engine-setup in a standard engine or a self-hosted engine installation. In some scenarios, you might need to install Grafana manually, such as performing an upgrade, restoring a backup, or after migrating the data warehouse to a separate machine.

To install Grafana manually:

(Self-hosted engine only) Put the environment in global maintenance mode:
```
hosted-engine --set-maintenance --mode=global
```
Sign in to the machine where you want to install Grafana. This must be the same machine where the data warehouse is configured. This is usually the engine machine.
Run the engine-setup command as follows to start the reconfiguration for Grafana:
```
engine-setup --reconfigure-optional-components
```
Press Enter to answer Yes to install Grafana on this machine.
```
Configure Grafana on this host (Yes, No) [Yes]:
```
(Self-hosted engine only) Disable global maintenance mode.
```
hosted-engine --set-maintenance --mode=none
```

After installation, you can access the Grafana dashboards in one of the following ways:

Go to https://<engine FQDN or IP address>/ovirt-engine-grafana.
Click Monitoring Portal in the web administration welcome page.

For more information, see Default Grafana Dashboards in the Oracle Linux Virtualization Manager: Architecture and Planning Guide.

Configuring Users for Single Sign-On with Grafana

Even though engine-setup automatically configures Grafana to let existing users sign in from the Administration Portal, it doesn't automatically create these users within Grafana.

To configure a user for single sign-on with Grafana:

Edit the user account to add an email address if not already defined, for example:

ovirt-aaa-jdbc-tool user edit test1 --attribute=email=jdoe@example.com

updating user test1...
user updated successfully

Sign in to Grafana with an existing admin user (the initially configured admin).
Navigate to Configuration and then Users and select Invite.
Enter the email address and name of the user account and select a Role.
Send the invitation using one of these options:
- Select Send invite mail and select Submit. For this option, you need an operational local mail server configured on the Grafana machine.
- Select Pending Invites
  - Find the entry you want
  - Select Copy invite
  - Use this link to create the account by pasting it directly into a browser address bar, or by sending it to another user.
  Note:
  If you use the Pending Invites option, no email is sent, and the email address isn't important. Any valid-looking address works, if it's configured as the email address of a Manager user.

To sign in with this account:

Sign in to the Administration Portal using the account that has the email address configured in Grafana.
Select Monitoring Portal to open the Grafana dashboard.
Select Sign in with oVirt Engine Auth.

Using Performance Co-Pilot

Performance Co-Pilot (PCP) is enabled by default for all Oracle Linux KVM installs and collects OS and network metrics that you can use to diagnose performance issues and that can help the support team to resolve issues. The logs generated and archived by the pmlogger service are stored in the /var/log/pcp/pmlogger/<hostname> directory and are collected using sosreport.

By default, PCP keeps logs for 7 days and compresses them after 1 day as defined in the pmlogger_timers file. For example:

grep PMLOGGER_DAILY_PARAMS /etc/sysconfig/pmlogger_timers |grep -v '#'

PMLOGGER_DAILY_PARAMS="-x 1 -k 7 -R"

The archive logs are rotated after 100MB and can be controlled using the following parameter:

grep LOCALHOSTNAME /etc/pcp/pmlogger/control.d/local

LOCALHOSTNAME y n PCP_ARCHIVE_DIR/LOCALHOSTNAME -r -T24h10m -c config.ora -v 100Mb

When the pcp-oracle-conf package is installed, only metrics collected by the pmlogger service are those listed in the /var/lib/pcp/config/pmlogger/config.ora configuration file in which the default metrics collection time is 10 seconds.

For more information, see: