5 Setting Up Performance Monitoring

This chapter describes how to define the KPIs and SLAs used to monitor your network's performance, and which you can review via dashboards and reports. This includes controlling how the SLAs used to track service levels should apply. The management of the alerts used to notify staff members about incidents that impact service levels, such as who should be notified and when, is also highlighted.

5.1 Introduction

A Service Level Agreement (SLA) is an agreement between a provider and a customer that explains the terms of the provider's responsibility to the customer, and the level of service that the customer can expect. Typically, this agreement is expressed in terms of a number of Key Performance Indicators (KPIs). These are a way of measuring and benchmarking specific aspects of an organization's performance.

For example, an SLA for a given service might promise that it will be up and running 99.999 percent of the time. Because this is a commitment given to customers, the organization could make this a KPI. As such, service availability would be monitored, and whenever it fell below this level, the appropriate staff would be notified, and corrective action taken.

It is important to understand that an organization may also set KPIs for its own performance monitoring, independently of an SLA. Because KPIs provide insight into an organization's performance, they may also be tracked as part of a management dashboard.

The creation and modification of KPIs can only be undertaken by users with Analytical level access.

5.1.1 Filtering KPIs

KPIs are grouped into categories, which can be customized to contain related performance indicators. For example, separate categories could be defined for business and IT-related issues, such as transaction completion, visitor traffic, Web site availability, and so on.

Because you may need to handle large number of KPIs, you can use list shown in Figure 5-1 to filter the currently defined KPIs.

Figure 5-1 Filter KPIs

Description of "Figure 5-1 Filter KPIs"

If you select "Service Levels", the left-hand side KPIs listing is updated to show only those KPIs that have service levels associated with them. Folders that do not contain such KPIs are not shown. Similarly, you can select "Alerts" to filter the listing to show only those KPIs that have alerts associated with them. The "All" option shows all KPIs.

5.2 Defining KPIs and SLAs

To create a KPI and, optionally, use it as the basis for alerts and service levels, do the following:

Select Configuration, then Service level management, then select KPIs, and click the New KPI button. The dialog shown in Figure 5-2 appears.

Figure 5-2 Metric Selection Dialog

Description of "Figure 5-2 Metric Selection Dialog"
Use the list to select the metric to be used as the basis for monitoring. When ready, click Next. If the metric you selected requires a filter, the dialog shown in Figure 5-3 appears. Otherwise, the dialog shown in Figure 5-4 appears.

Figure 5-3 Required Filter Dialog

Description of "Figure 5-3 Required Filter Dialog"
Use the list to specify a filter for the selected metric. For example, if you selected the transaction-load-time(sec) metric, you need to specify the transaction to which it refers. For information on defining transactions, see Section 6.1, "Naming Pages". When ready, click Next. The dialog shown in Figure 5-4 appears.

Figure 5-4 KPI Attributes

Description of "Figure 5-4 KPI Attributes"
Use the check boxes to specify the following:
- Filters: specifies whether you want to add filters to the selected metric at this time. For example, you could define that a metric should apply to a particular domain.
- Requirements: specifies any additional requirements for the selected metric. Using this facility, you can build compound KPIs.
- Targets: specifies whether targets are associated with the KPI. If so, you can define a minimum and maximum range for the KPI, and how they should be calculated.
- Service Level Agreement: specifies whether the KPI should be incorporated into an SLA. If so, you can configure the level of your committed agreement (in percentage terms) for specific time periods.
- Alerting: specifies whether an alert should be associated with the KPI. If so, you define the duration the KPI must be up (or down) before an alert is issued, the severity of the incident, and whether additional notification should be created when the KPI has returned to its set target range.
When ready, click Next. The dialog shown in Figure 5-5 appears.

Figure 5-5 Filters Dialog

Description of "Figure 5-5 Filters Dialog"
Use this dialog to define a filter to tighten the conditions for the KPI. For example, you might specify a KPI that concerns transaction load time. Using the Dimension level list, you can specify that you only want the KPI to apply to a particular transaction step, or only to users coming from a particular location. Click Add filter for each filter that you want to apply. Note that you see the history of your filter selections in the lower part of the dialog. If you define multiple filters, all the conditions must be met for a match to be made. Note that this dialog only appears if you checked the Filters check box in Figure 5-4. When ready, click Next. The dialog shown in Figure 5-6 appears.

Figure 5-6 Requirements Dialog

Description of "Figure 5-6 Requirements Dialog"
Use this dialog to specify additional requirements for the KPI. In this way, you can build compound metric conditions. For example, the monitored service should provide an end-to-end page time of between 3 and 5 seconds for 98% of requested pages, but this requirement should only apply when page views per minute are between 5 and 10. Click Add requirement to specify compound metrics.

Note:
Any filter you specified in Figure 5-1 will also apply to any additional metrics. Therefore, you should ensure that the filter is relevant to the additional metrics. Also, if you require additional (compound) metrics, all the defined requirements must be met for the KPI to yield a result that can be monitored.

Note that this dialog only appears if you checked the Requirements check box in Figure 5-4. When ready, click Next. The dialog shown in Figure 5-7 appears.

Figure 5-7 Targets Dialog

Description of "Figure 5-7 Targets Dialog"
Use this dialog to set a range for the KPI. You can define it in terms of a fixed range. For example, between 80 and 100. Alternatively, you can specify if the KPI should be sampled for small, medium, or large deviations from its auto-learnt target. For more information on the use of this facility, see Section 5.3.1, "Automatic and Fixed Targets". Note that this dialog only appears if you checked the Targets check box in Figure 5-4. When ready, click Next. The dialog shown in Figure 5-8 appears.

Figure 5-8 Service Level Agreement Dialog

Description of "Figure 5-8 Service Level Agreement Dialog"
Use this dialog to specify the level of your service agreement. For example, you undertake that the service will meet its specified objectives throughout 98% of the year. However, on an hourly basis, the commitment is 80%, and on a daily basis, 90%. All the period fields are mandatory.

Note that this dialog only appears if you checked the Service Level Agreement check box in Figure 5-4. When ready, click Next. The dialog shown in Figure 5-9 appears.

Figure 5-9 Alerting Dialog

Description of "Figure 5-9 Alerting Dialog"
Use this dialog to specify the alert schedule that should be used (business, technical, or both), and the duration that the KPI must be down (or up) before an alert is generated. You can also specify the severity (Harmless, Warning, Minor, Critical, or Fatal) of the incident, and whether an additional notification should be generated when the KPI returns to its set target range. It is recommended that you carefully review these settings to prevent excessive notifications.

This is dialog only appears if you checked the Alerting check box in Figure 5-4. When ready, click Next. The dialog shown in Figure 5-10 appears.

Figure 5-10 Save As Dialog

Description of "Figure 5-10 Save As Dialog"
Use this dialog to specify a name, category, and brief description for the monitored KPI. If you specify a new category name, this category will be automatically created. When ready, click Finish to complete your KPI definition. Note that monitoring of the new KPI starts immediately.

5.2.1 Renaming, Moving, and Deleting KPIs

You can modify, rename (or move), or delete KPIs by right clicking them and selecting the Rename or Remove options from the menu. Select the Edit option to modify the KPI. The procedure to do this is described in Section 5.3, "Modifying Existing KPIs".

5.2.2 Copying Existing KPIs

In addition to creating new KPIs from scratch, as explained in Section 5.2, "Defining KPIs and SLAs", you can also create a copy of an existing KPI and use it as the basis for your new KPI. This is particularly useful when the new KPI is very similar to an existing one. For example, you already have an existing KPI that monitors transaction availability in the USA, but now want to create a new one for Canada. To use an existing KPI as the basis for a new one, do the following:

Select Configuration, then Service level management, then KPIs, and select the required KPI from the displayed listing. Click the Copy KPI button. The dialog shown in Figure 5-11 appears.

Figure 5-11 Copy KPI Dialog

Description of "Figure 5-11 Copy KPI Dialog"
Specify a new name or location for the new KPI. Optionally, click Add category to create a new category. When ready, click Save.
Use the facilities described in Section 5.3, "Modifying Existing KPIs" to modify the new KPI to your requirements.

5.3 Modifying Existing KPIs

You can review and modify the definitions of existing KPIs by selecting Configuration, then Service level management, then KPIs, and selecting the required KPI from the displayed listing. A screen similar to the one shown in Figure 5-12 appears:

Figure 5-12 KPI Definition

Description of "Figure 5-12 KPI Definition"

You can use the tabs to locate particular aspects to the selected KPI, and review and modify their definition. Their associated settings are equivalent to those described in Section 5.2, "Defining KPIs and SLAs".

5.3.1 Automatic and Fixed Targets

As mentioned earlier, you can specify a KPI should use automatic (or auto-learnt) targets. Because visitor traffic and usage patterns can differ widely during the course of a day, these auto-learnt minimum and maximum targets are calculated as moving averages for the current 5-minute period, based on the sampled metric value for that 5-minute period over the last 30 days. For example, when a KPI metric is sampled at 10.45 AM, the average against which it is compared is calculated from the last 30 days of samples at 10.45 AM. You can specify the minimum and maximum targets in terms of small, medium, or large deviations from these moving averages.

In contrast, a fixed KPI target essentially represents, either minimum or maximum, a straight line. This is shown in Figure 5-13.

Figure 5-13 Automatic and Fixed KPI Targets Contrasted

Description of "Figure 5-13 Automatic and Fixed KPI Targets Contrasted"

When using auto-learnt targets, be aware of the following:

Auto-learnt targets assume that a KPI has approximately the same value at the same time of day during each of the last 30 days. If this is not the case, it is recommended you use fixed targets.
It requires a full day before the auto-learnt targets become available. Clearly, the more days of historical data that are available, the more reliable the automatic targets can be calculated. During the first day that a KPI is created with auto-learnt targets, these targets are automatically set to slightly above and below the actual recorded values in order to prevent the generation of alerts.
Although auto-learnt targets can signal a problem if the metric value is too high or too low, if the problem persists over a long period, these abnormal values will become part of the auto-learnt targets and will, eventually, be assumed to be normal behavior.
Auto-learnt targets can drop dramatically if the KPI value is unavailable every day at about the same time. For example, in the case of no network traffic after 18:00.

If you define a KPI to use automatic targets (see Figure 5-7), and later modify the KPI to use fixed targets, the previously calculated targets (derived by monitoring the KPI over time) are set as the new fixed targets. If you are in doubt about the fixed targets that should be set for a KPI, you can use this facility to obtain realistic initial values. Of course, you are free to modify these at any time.

5.4 Defining Service Level Schedules

In addition to defining the KPIs that will be used to track the service levels achieved by your organization, you also need to specify when these service levels should apply. Typically, an organization has a core time (for example, 9 am - 5 pm, Monday - Friday) when the committed service level should be achieved. However, you may need to define exceptions to this, such as for public holidays. For example, a limited service between 10 am and 4 pm may be required on Easter Monday. Finally, you will also need to take account of planned maintenance periods.

The scheduling of planned service levels is maintained through the Service level schedule (shown in Figure 5-14). To open it, select Configuration, then Service level management, and then select Service level schedule.

Figure 5-14 Service Level Schedule

Description of "Figure 5-14 Service Level Schedule"

You can mark a period within the Service level schedule by clicking and dragging over the required period of the week. Assign the selected period a status by clicking the Active or Non-active modes.

You can define exceptions by clicking the Plus (+) icon, and selecting the day, month, and year from the Exceptions list. You can remove exceptions by clicking the Minus (-) icon to the right of an exception.

Note that any changes you make are not put into effect until you click Save. On exit, any unsaved changes you made are discarded.

5.5 Defining Alert Schedules

If your organization uses alerts to notify staff members about incidents that impact service levels, you will need to specify who should be notified and when. Within RUEI, two types of alert schedule are available: business and technical.

When you define a KPI, you specify (in Figure 5-9) whether the KPI is a business or technical (or both) KPI. These two schedules enable you to extend this distinction, and specify groups of users, notification details, and the operative time frame. Exceptions to standard operating times can also be defined.

To open these schedules, select Configuration, then Service level management, then select Alert schedule, and then select Business or Technical from View the list. Figure 5-15 shows an example of the Business alert schedule.

Figure 5-15 Business Alert Schedule

Description of "Figure 5-15 Business Alert Schedule"

You can mark a period within the Business or Technical level schedule by clicking and dragging over the required period of the week. Assign the selected period by clicking one of the Alert profiles.

Note that any changes you make are not put into effect until you click Save. On exit, any unsaved changes you made are discarded.

5.5.1 Alert Profiles

These define the users who will be notified if a business or technical KPI has been down (or up) for the specified duration required to generate an alert. Depending on how the KPI has been defined, these users will also be notified when the KPI returns to within its set target range.

For example, you might have defined a KPI for transaction-success-rate, and have specified that a success rate of least 70% is required for normal operation. If the KPI falls below this level within core business hours (9 am - 5 pm, Monday - Friday), all Web application Business Managers should be notified. If the failure occurs outside these hours, the Helpdesk should be notified.

Each profile can be customized by right clicking it, and selecting Edit from the menu. This is shown in Figure 5-16:

Figure 5-16 Alert Profile Menu

Description of "Figure 5-16 Alert Profile Menu"

The dialog shown in Figure 5-17 appears.

Figure 5-17 Alert Profile Dialog

Description of "Figure 5-17 Alert Profile Dialog"

Use this dialog to specify the name and a brief description of the users to be notified. Use the other tabs in this dialog to specify the recipients of E-mail, SNMP, and text message notification. Use the Enabled check box for each method to activate notification.

Note:

When receiving text message-based alerts, the timestamp of the message shown within your mobile telephone may not match that recorded within your RUEI installation. This is due to time zone differences on your mobile telephone.

5.5.2 Escalation Procedures

Within the Escalation tab, shown in Figure 5-18, you can set reminders to be sent to the alert's recipients if the KPI remains down. In addition, you can define an escalation procedure if the KPI is still down after a defined period. For example, if the KPI is still down after three hours, notify another group. This escalation group can be customized by right clicking it, and selecting Edit from the menu.

Figure 5-18 Escalation Tab

Description of "Figure 5-18 Escalation Tab"

5.5.3 Sampling and Notification Intervals

It is important to understand that there are two states associated with a KPI: the KPI state, and the alert state. The KPI state can change at each sampling interval. The alert state is controlled by the properties you define for the alert. For example, consider the case in which a KPI starts to fail, and you have defined a sample interval of 5 minutes (the default), and a DOWN duration of 15 minutes. Although after 5 minutes the KPI is considered to be failing, you will not be notified about it unless it has been continually down for 15 minutes.

Similarly, the reminder and escalation durations you specify in Figure 5-18 refer to the alert. Hence, specifying a reminder duration of every hour would generate a reminder notification every 60 minutes after the original alert was sent while the KPI is still failing. It is recommended that you carefully review the values you specify for these settings.

5.5.4 Testing Alert Messages

If you have enabled e-mail, SNMP, or text message notification, you can use the Test profile option in the menu shown in Figure 5-16 to send a test alert to all specified recipients in an alert or escalation profile. This is useful for testing that the contact information has been entered correctly. You are prompted to confirm the test notification.

5.5.5 Using Mail Notifications

To define E-mail alert recipients, click the E-mail tab to open the E-mail dialog (shown in Figure 5-19) and do the following:

Figure 5-19 E-mail Dialog

Description of "Figure 5-19 E-mail Dialog"

Use the Recipients fields to specify the e-mail addresses of the users to be notified. Click Add to include a user in the notification list. Note that you can remove a user from the list by clicking the icon to the right of the user.
Check the Enable check box to activate e-mail notification. When ready, click Save.

5.5.6 Using SNMP Not ifications

To define SNMP alert recipients, click the SNMP tab to open the SNMP dialog (shown in Figure 5-20) and do the following:

Figure 5-20 SNMP Dialog

Description of "Figure 5-20 SNMP Dialog"

Use the Version list to specify which version of the SNMP protocol is being used. The default is version 2c.
Use the Manager address field to specify the client software address. This must be a valid network address, and can either IP address or a host name.
Use the Community field to specify the group to which information is sent. This string acts as a password to control the clients' access to the server.
Check the Enable check box to activate SNMP notification.
Download the Management Information Base (MIB) definition and incorporate it into your address book of managed objects. It contains necessary information about how the received SNMP messages should be interpreted. The structure of the MIB file is shown in Figure 5-21 ^Foot 1.

Figure 5-21 SNMP MIB Structure

Description of "Figure 5-21 SNMP MIB Structure"

The available KPI information and metrics in the MIB represent the most important properties of every KPI configured within the system, and can be used as the basis for filtering and alerting. They are explained in Table 5-1.

Table 5-1 KPI Information and Metrics Structure

Object	Type
KPI Duration	Value
KPI Severity	Text
KPI Maximum	Value
KPI Minimum	Value
KPI Value	Value
KPI Category	Text
KPI Name	Text

Note KPI names in SNMP alerts are sent in UTF-8 format. Any characters in the KPI name not in ISO-Latin-1 format will be replaced by a question mark (?) character. Also, be aware not all SNMP managers fully support UTF-8. For further information, refer to your SNMP manager product documentation.

5.5.7 Using Text Message Notifications

To define text message notifications, click the Text message tab to open the Text message dialog (shown in Figure 5-22), and do the following:

Figure 5-22 Text Message Dialog

Description of "Figure 5-22 Text Message Dialog"

Use the Recipients field to specify the telephone numbers of the users to be notified. Click Add to include a user in the notification list. Note that you can remove a user from the list by clicking the icon to the right of the user.
Check the Enable check box to activate text message notification.
If you have not already done so, you will need to configure an text message provider. If you are warned that one has not already been configured, click the warning link, and follow the instructions described in Section 9.11, "Configuring Text Message Providers".

Footnote Legend

Footnote 1: This screen features the iReasoning MIB Browser (http://www.ireasoning.com). This utility is not distributed as part of RUEI, and requires a separate license. It is intended only to illustrate the structure of the provided MIB file.