Monitoring and Diagnostics

The following section explains how to setup and use Grafana dashboards and event notifications for your virtualization environment, and about Performance Co-Pilot metrics.

Using Event Notifications

The following section explains how to set up event notifications to monitor events in your virtualization environment. You can configure the Manager to send event notifications in email to alert designated users when certain events occur or enable Simple Network Management Protocol (SNMP) traps to monitor your virtualization environment.

For more information, see Event Logging and Notifications in the Oracle Linux Virtualization Manager: Architecture and Planning Guide.

Configuring Event Notification Services on the Engine

For event notifications to be sent properly to email recipients, you must configure the mail server on the Engine and enable ovirt-engine-notifier service. For more information about creating event notifications in the Administration portal, see Creating Event Notifications in the Administration Portal.

To configure notification services on the Engine:

  1. Log in to the host that is running the Manager.

  2. Copy the ovirt-engine-notifier.conf to a new file named 90-email-notify.conf. 

    # cp /usr/share/ovirt-engine/services/ovirt-engine-notifier/ovirt-engine-notifier.conf/ \
etc/ovirt-engine/notifier/notifier.conf.d/90-email-notify.conf
  3. Edit the 90-email-notify.conf file by deleting everything except the EMAIL Notifications section.

    Note:

    If you plan to also configure SNMP traps in your virtualization environment, you can also copy the values from the SNMP_TRAP Notifications section of the ovirt-notifier.conf file to a file named 20-snmp.conf. For more information, see Configuring the Engine to Send SNMP Traps.

  4. Enter the correct email variables. This file overrides the values in the original ovirt-engine-notifier.conf file.
    ---------------------
# EMAIL Notifications #
---------------------

# The SMTP mail server address. Required.
MAIL_SERVER=myemailserver.mycompany.com

# The SMTP port (usually 25 for plain SMTP, 465 for SMTP with SSL, 587 for SMTP with TLS)
MAIL_PORT=25

# Required if SSL or TLS enabled to authenticate the user. Used also to specify 'from' 
  user address if mail server
# supports, when MAIL_FROM is not set. Address is in RFC822 format
MAIL_USER=email.example.com

# Required to authenticate the user if mail server requires authentication or if SSL or 
  TLS is enabled
SENSITIVE_KEYS="${SENSITIVE_KEYS},MAIL_PASSWORD"
MAIL_PASSWORD=

# Indicates type of encryption (none, ssl or tls) should be used to communicate with 
  mail server.
MAIL_SMTP_ENCRYPTION=none

# If set to true, sends a message in HTML format.
HTML_MESSAGE_FORMAT=false

# Specifies 'from' address on sent mail in RFC822 format, if supported by mail server.
MAIL_FROM=myovirtengine@mycompany.com

# Specifies 'reply-to' address on sent mail in RFC822 format.
MAIL_REPLY_TO=myusername@mycompany.com

# Interval to send smtp messages per # of IDLE_INTERVAL
MAIL_SEND_INTERVAL=1

# Amount of times to attempt sending an email before failing.
MAIL_RETRIES=4

    Note:

    For information about the other parameters available for event notification in the ovirt-engine-notifier.conf file, refer to oVirt Documentation.

  5. Enable and restart the ovirt-engine-notifier service to activate your changes. 

    # systemctl daemon-reload
# systemctl enable ovirt-engine-notifier.service
# systemctl restart ovirt-engine-notifier.service

Creating Event Notifications in the Administration Portal

Before creating event notifications, you must have access to an email server that can handle incoming automated messages and deliver these messages to a distribution list. You should also configure event notification services on the Engine. For more information, see Configuring Event Notification Services on the Engine.

To create event notifications in the Administration Portal:

  1. Go to Administration and then click Users.

    The Users pane opens.

  2. Under the User Name column, click the name of the user to display the detailed view for the user.

    Note:

    A user does not appear in the Administration Portal until the user is created and assigned appropriate permissions. For more information, refer to Creating a New User Account.

  3. Click the Event Notifier tab.

  4. Click Manage Events.

    The Add Event Notification dialog box opens.

  5. Select the events for which you want to create notifications by selecting the check box next to individual events or event topic areas for notification.

    The events available for notification are grouped under topic areas. By default, selecting the check box for a top-level topic area, such as General Host Events, selects all events under that topic area. You can optionally expand or collapse all the event topic areas by clicking Expand All or Collapse All. Additionally, you can click the arrow icon next to a specific top-level topic area to expand or collapse the events associated with a specific topic area.

  6. For the Mail Recipient field, enter an email address.

  7. Click OK to save the changes.

Canceling Event Notifications in the Administration Portal

To cancel event notifications in the Administration Portal:

  1. Go to Administration and then click Users.

    The Users pane opens.

  2. Under the User Name column, click the name of the user to display the detailed view for the user.

  3. Click the Event Notifier tab.

  4. Click Manage Events.

    The Add Event Notification dialog box opens.

  5. Click Expand All, or the topic-specific expansion options, to display the events.

  6. Clear the appropriate check boxes to cancel the notification for that event.

  7. Click OK to save your changes.

Configuring the Engine to Send SNMP Traps

You can configure the Manager to send SNMP traps to one or more external SNMP managers. SNMP traps contain system event information that are used to monitor your virtualization environment. The number and type of traps sent to the SNMP manager can be defined within the Engine.

Before performing this task, you must have configured one or more external SNMP managers to receive traps, and know the following details:

  • The IP addresses or fully-qualified domain names of machines that act as SNMP managers. Optionally, determine the port through which the SNMP manager receives trap notifications; the default UDP port is 162.

  • The SNMP community. Multiple SNMP managers can belong to a single community. Management systems and agents can communicate only if they are within the same community. The default community is public.

  • The trap object identifier for alerts. The Engine provides a default OID of 1.3.6.1.4.1.2312.13.1.1. All trap types are sent, appended with event information, to the SNMP manager when this OID is defined.

    Note:

    • Changing the default trap prevents generated traps from complying with the Engine’s management information base.

    • The Engine provides management information bases at /usr/share/doc/ovirt-engine/mibs/OVIRT-MIB.txt and /usr/share/doc/ovirt-engine/mibs/REDHAT-MIB.txt. Load the MIBs in your SNMP manager before proceeding.

To configure SNMP traps on the Engine:

  1. Log in to the host that is running the Manager.

  2. On the Engine, create the SNMP configuration file:
    # vi /etc/ovirt-engine/notifier/notifier.conf.d/20-snmp.conf

    Default SNMP configuration values exist on the Engine in the events notifications configuration file (ovirt-engine-notifier.conf), which is available at the following directory path: /usr/share/ovirt-engine/services/ovirt-engine-notifier/ovirt-engine-notifier.conf. The values provided in this step are based on the default or example values provided in that file. To persist that your configuration settings persist across reboots, define an override file for your SNMP configuration (20-snmp.conf), rather than edit the ovirt-engine-notifier.conf file, For more information, see Configuring Event Notification Services on the Engine.

  3. Specify the SNMP manager, the SNMP community, and the OID in the following format:
    SNMP_MANAGERS="manager1.example.com manager2.example.com:162"
SNMP_COMMUNITY=public
SNMP_OID=1.3.6.1.4.1.2312.13.1.1
    The following values can be configured in the 20-snmp.conf file.
    #-------------------------#
# SNMP_TRAP Notifications #
#-------------------------#
# Send v2c snmp notifications

# Minimum SNMP configuration
#
# Create /etc/ovirt-engine/notifier/notifier.conf.d/20-snmp.conf with:
# SNMP_MANAGERS="host"
# FILTER="include:*(snmp:) ${FILTER}"

# Default whitespace separated IPv4/[IPv6]/DNS list with optional port, default is 162.
# SNMP_MANAGERS="manager1.example.com manager2.example.com:164"
SNMP_MANAGERS=

# Default SNMP Community String.
SNMP_COMMUNITY=public

# SNMP Trap Object Identifier for outgoing notifications.
# { iso(1) org(3) dod(6) internet(1) private(4) enterprises(1) redhat(2312) ovirt(13) 
  engine(1) notifier(1) }
#
# Note: changing the default will prevent generated traps from complying with 
  OVIRT-MIB.txt.
SNMP_OID=1.3.6.1.4.1.2312.13.1.1

# Default SNMP Version. SNMP version 2 and version 3 traps are supported
# 2 = SNMPv2
# 3 = SNMPv3
SNMP_VERSION=2

# The engine id used for SNMPv3 traps
SNMP_ENGINE_ID=

# The user name used for SNMPv3 traps
SNMP_USERNAME=

# The SNMPv3 auth protocol. Supported values are MD5 and SHA.
SNMP_AUTH_PROTOCOL=

# The SNMPv3 auth passphrase, used when SNMP_SECURITY_LEVEL is set to AUTH_NOPRIV 
  and AUTH_PRIV
SNMP_AUTH_PASSPHRASE=

# The SNMPv3 privacy protocol. Supported values are AES128, AES192 and AES256.
# Be aware that AES192 and AES256 are not defined in RFC3826, so please verify
# that your SNMP server supports those protocols before enabling them.
SNMP_PRIVACY_PROTOCOL=

# The SNMPv3 privacy passphrase, used when SNMP_SECURITY_LEVEL is set to AUTH_PRIV
SNMP_PRIVACY_PASSPHRASE=

# The SNMPv3 security level.
# 1 = NOAUTH_NOPRIV
# 2 = AUTH_NOPRIV
# 3 = AUTH_PRIV
SNMP_SECURITY_LEVEL=1

# SNMP profile support
#
# Multiple SNMP profiles are supported.
# Specify profile settings by using _profile suffix,
# for example, to define a profile to sent specific
# message to host3, specify:
# SNMP_MANAGERS_profile1=host3
# FILTER="include:VDC_START(snmp:profile1) ${FILTER}"

  4. Define which events to send to the SNMP Manager.

    By default, the following default filter is defined in the ovirt-engine-notifier.conf file; if you do not override this filter or apply overriding filters, no notifications are sent.
    FILTER="exclude:\*"
    The following are other common examples of event filters.
    • Send all events to the default SNMP profile.
      FILTER="include:\*(snmp:) ${FILTER}"
    • Send all events with the severity ERROR or ALERT to the default SNMP profile:
      FILTER="include:\*:ERROR(snmp:) ${FILTER}"
 FILTER="include:\*:ALERT(snmp:) ${FILTER}"

  5. Save the file.

  6. Start the ovirt-engine-notifier service, and ensure that this service starts on boot.
     # systemctl start ovirt-engine-notifier.service
 # systemctl enable ovirt-engine-notifier.service

  7. (Optional) Validate that traps are being sent to the SNMP Manager.

Using Grafana

The following section explains how to setup and use Grafana dashboards in your virtualization environment.

Important:

You must install the Data Warehouse database, the Data Warehouse service and Grafana all on the same machine, even though you can install each of these components on separate machines from each other.

For more information, see Monitoring with Grafana in the Oracle Linux Virtualization Manager: Architecture and Planning Guide.

For more information on using Grafana, see the Grafana website.

Installing Grafana

Grafana integration is enabled and installed by default when you run engine-setup in a standard engine or a self-hosted engine installation. In some scenarios, you might need to install Grafana manually, such as performing an upgrade, restoring a backup, or after migrating the data warehouse to a separate machine.

To install Grafana manually:

  1. (Self-hosted engine only) Put the environment in global maintenance mode: 

              # hosted-engine --set-maintenance --mode=global

  2. Log in to the machine where you want to install Grafana. This should be the same machine where the data warehouse is configured; usually the engine machine.

  3. Run the engine-setup command as follows to initiate the reconfiguration for Grafana: 

    # engine-setup --reconfigure-optional-components

  4. Press Enter to answer Yes to install Grafana on this machine. 

    Configure Grafana on this host (Yes, No) [Yes]:

  5. (Self-hosted engine only) Disable global maintenance mode. 

              # hosted-engine --set-maintenance --mode=none

Once installed, you can access the Grafana dashboards in one of the following ways:

  • Go to https://<engine FQDN or IP address>/ovirt-engine-grafana.

  • Click Monitoring Portal in the web administration welcome page.

For more information, see Default Grafana Dashboards in the Oracle Linux Virtualization Manager: Architecture and Planning Guide.

Configuring Users for Single Sign-On with Grafana

Even though engine-setup automatically configures Grafana to allow existing users to log in from the Administration Portal, it does not automatically create these users within Grafana.

To configure a user for single sign-on with Grafana:

  1. Log in to the host that is running the Manager.

  2. Edit the user account to add an email address if not already defined, for example:

    # ovirt-aaa-jdbc-tool user edit test1 --attribute=email=jdoe@example.com
updating user test1...
user updated successfully
  3. Log in to Grafana with an existing admin user (the initially configured admin).
  4. Navigate to Configuration and then Users and select Invite.
  5. Enter the email address and name of the user account and select a Role.

  6. Send the invitation using one of these options:

    • Select Send invite mail and click Submit. For this option, you need an operational local mail server configured on the Grafana machine.

    • Select Pending Invites

      • Locate the entry you want
      • Select Copy invite
      • Use this link to create the account by pasting it directly into a browser address bar, or by sending it to another user.

      Note:

      If you use the Pending Invites option, no email is sent, and the email address does not really need to exist; any valid-looking address will work, as long as it’s configured as the email address of a Manager user.

To log in with this account:

  1. Log in to the Administration Portal using the account that has the email address configured in Grafana.
  2. Select Monitoring Portal to open the Grafana dashboard.
  3. Select Sign in with oVirt Engine Auth.

Using Performance Co-Pilot

Performance Co-Pilot (PCP) is enabled by default for all Oracle Linux KVM installs and collects OS and network metrics that you can use to diagnose performance issues and that can help the support team to resolve issues. The logs generated and archived by the pmlogger service are stored in the /var/log/pcp/pmlogger/<hostname> directory and are collected using sosreport.

By default, PCP keeps logs for 7 days and compresses them after 1 day as defined in the pmlogger_timers file. For example: 

# grep PMLOGGER_DAILY_PARAMS /etc/sysconfig/pmlogger_timers |grep -v '#'
PMLOGGER_DAILY_PARAMS="-x 1 -k 7 -R"

The archive logs are rotated after 100MB and can be controlled using the following parameter:

# grep LOCALHOSTNAME /etc/pcp/pmlogger/control.d/local
LOCALHOSTNAME y n PCP_ARCHIVE_DIR/LOCALHOSTNAME -r -T24h10m -c config.ora -v 100Mb

When the pcp-oracle-conf package is installed, only metrics collected by the pmlogger service are those listed in the /var/lib/pcp/config/pmlogger/config.ora configuration file in which the default metrics collection time is 10 seconds.

For more information, see Collecting and Analyzing Metrics With Performance Co-Pilot.