It is particularly important that Notifications be set for the Heat Chart group of rules. This is easily done from the Current Schedule page by clicking the + button beside a rule and then clicking a server.

Doing this opens a window with three tabs—Overview, Settings, and Advanced.

The Overview tab shows which advisor group a rule belongs to, a description of its purpose, and a link to the history of this alert.

In the Settings tab you can adjust the frequency of this rule and also specify a notification group. To select more than one contiguous group press the Shift key and click the desired groups. (Some web browsers may require that you drag your selection.) Noncontiguous selections are made by holding down the Control key and clicking the desired groups.

If you haven't set up global SNMP traps and would like your Network Management System (NMS) to handle events related to a specific rule then check the Use SNMP Traps checkbox. For more information about Simple Network Management Protocol (SNMP) see Simple Network Management Protocol (SNMP) Traps.

The Advanced tab gives detailed information about how this rule is implemented.

Similar existing rules are grouped together in advisor groups.

The built-in advisors are:

The ability to create your own advisor group allows you to create groupings suitable to your circumstances.

You can create your own grouping by simply clicking the create advisor button. Enter an appropriate name and click the add button. The newly created group will appear in the Advisor column.

The newly created advisor is added to the list box of advisors shown in Figure 5.2, “MySQL Enterprise Dashboard: Editing Rules”. You can now use this category of advisors when you create a new rule.

Rules are created using the same screen seen in Figure 5.2, “MySQL Enterprise Dashboard: Editing Rules”. To begin creating a rule from scratch, click the create rule button. However, the simplest way to create a new rule is to copy an existing one. Unlike editing an existing rule, when you copy a rule, every element of that rule is editable.

You can change the rule name, the advisor group that a rule belongs to and you can set your own version number. In Figure 5.2, “MySQL Enterprise Dashboard: Editing Rules”, you have already seen how the threshold and frequency of a rule may be altered.

Most importantly you can alter a rule's expression. Expressions are the core of a MySQL Enterprise Advisor and are used to define the scenario being monitored. An expression can be as simple as a single server parameter or can be quite complex, combining multiple parameters with various mathematical operations.

An expression has two main characteristics:

If an expression evaluates to true for a specific server, an alarm is raised, indicating that a best practice is not being followed. If an expression evaluates to false no alarm is raised because the best practice is indeed being followed.

For example, if having binary logging enabled is considered a best practice for a production server (which we believe it is), then this best practice is being violated if log_bin is OFF. Consequently, the expression for the “Binary Logging Not Enabled” rule is “%log_bin% == OFF”. If this evaluates to 1, an alarm is raised because the best practice is not being followed.

An expression is made up of one or more variables and zero or more mathematical operators. The MySQL Enterprise Monitor uses the MySQL database server's expression parser and evaluator For a complete list of operators and functions see Functions and Operators. For a complete list of the built-in variables used when creating rules see Server Option and Variable Reference.

Creating an expression is dependent on variables defined in the Variable Assignment frame. This frame links variables used in the expression field with data gathered from the target MySQL server instance—server status variables, operating system status information, and table information. Variable names are associated with elements in the Data Item drop-down list. If you need to define more than one variable simply click the add row button. For a complete listing of the data collection items used in creating rules see Appendix G, Data Collection Items.

The remaining fields determine the information that displays in a notification email or the informational pop-up window associated with each advisor.

When an expression is evaluated variables get replaced by values. For example, part of the expression for the “MyISAM Key Cache Has Sub-Optimal Hit Rate” rule calculates the hit rate as follows:

100-((%Key_reads% / %Key_read_requests%)*100)

If the current value of %Key_reads% is 4522 and the current value of %Key_read_requests% is 125989, the hit ratio assesses to 96.4%:

100 -((4522 / 125989) * 100)

By convention, the Advisors supplied by MySQL use ‘%’ as the delimiter, for example, %Key_reads%. This makes variables more readily identifiable.

In addition to being used in an expression, variables may also be used in the Description, Advice, Action, and Links attributes of a rule. This allows you to report the current value of an expression.

For instance, you can add the message, “The current value of Key_reads is %Key_reads%.” to the Advice text box. When this is displayed on the screen, the value of %Key_reads% is substituted into the text. Supposing %Key_reads% has a value of 4522, the message becomes “The current value of Key_reads is 4522.”

Each expression has a threshold value that triggers an alert. The THRESHOLD keyword is used to associate that value with an alert level—either an Info, Warning, or Critical alert.

For example, the expression for the performance advisor, “Thread Cache Size May Not Be Optimal”, is:

100-((%Threads_created% / %Connections%) * 100) < THRESHOLD

The THRESHOLD is set at 95% for an Info level alert, 85% for a Warning alert, and 75% for a Critical alert; producing alerts of three different levels.

Expressions can be quite simple. The expression for “Binary Logging Not Enabled” (one of the Administration alerts) is:

%log_bin% == THRESHOLD

When the result is OFF, only one alert is triggered—a Warning level alert. In this situation you might think we could just use the expression %log_bin% == "OFF". However, doing this would not test binary logging against a threshold so would not result in an alert.

When you create an expression, think carefully about the conditions under which it should be evaluated and the conditions under which it should not. For example, the expression for the “MyISAM Key Cache Has Sub-Optimal Hit Rate” rule is:

(%Uptime% > 10800) && (%Key_read_requests% > 10000) »
 && (100-((%Key_reads% / %Key_read_requests%) * 100) < THRESHOLD)

The essence of the rule is really: (100-((%Key_reads% / %Key_read_requests% ) * 100) < THRESHOLD). However, when a server is first starting up, it may take a while to reach a state that is representative of normal operations. For example, the key cache and the query cache may need some period of time before they have cached typical application data as opposed to start-up and initialization data. In this case, the first part of the expression, (%Uptime% > 10800), holds off evaluating this expression until the system has been running for 10800 seconds (3 hours).

In addition, if some part of the system is not heavily used an alert may be triggered based on limited data. For example, if your application does not use the MyISAM storage engine, the “MyISAM Key Cache Has Sub-Optimal Hit Rate” rule may be triggered based on very limited use of other MyISAM tables such as the mysql.user table. For this reason, this advisor has a second part— (%Key_read_requests% > 10000)—meaning the rule won't be evaluated unless there is plenty of activity associated with the key cache.

In other circumstances, there may be periods of time during which you don't want a rule to be evaluated—a blackout period. For example, the expression for the “Slave Too Far Behind Master” rule is: %Seconds_Behind_Master% > THRESHOLD. However, suppose you run a backup process between 6 and 7 pm on a replication slave, and it's normal for that slave to get behind the master by an amount more than the THRESHOLD during that time. In that case you don't want to receive an alert because the rule violation is expected. You can achieve this by adding the following to the expression: && CURTIME() NOT BETWEEN '18:00:00' AND '19:00:00' In essence, this means “don't trigger an alert between 18:00:00 and 19:00:00 (6 pm and 7 pm)”.

String values may appear in the Expression or the Thresholds text boxes. In both cases, they must be enclosed within quotation marks. For example, the expression for the “Slave I/O Thread Not Running” rule is:

(%Slave_running% == "ON") && (%Slave_IO_Running% != THRESHOLD)

In similar fashion the Critical Alerts threshold text box is set to a value of "Yes".

When the expression is evaluated, either "OFF" or "ON" will be substituted for %Slave_running%, and "Yes" or "No" for %Slave_IO_Running%, depending on the state of your system. If the slave is running but the I/O thread is not, the expression then becomes:

("ON" == "ON") && ("No" != "Yes")

Without quotation marks this expression would not evaluate to TRUE as it should.

When editing or defining a rule, the text entered in the Problem Description, Advice, Recommended Action, and Links and Further Reading text boxes may be formatted in Wiki format. This allows you to format text and add hyperlinks when creating or editing your own rules.

Find a brief introduction to using Wiki formatting in the following table.

So the following Wiki text:

Replication is a __very nice feature__ of MySQL.  Replication can be very
useful for solving problems in the following areas:
* Data Distribution
* Load Balancing
* Backup and Recovery
You can check replication status and start a slave using the following
commands: SHOW SLAVE STATUS \\\\G\\START SLAVE;
{moreInfo:MySQL Manual: Replication
      FAQ|http://dev.mysql.com/doc/refman/5.0/en/replication-faq.html}

Would be translated into the following HTML markup:

Replication is a <b>very nice feature</b> of MySQL.  Replication can be very
useful for solving problems in the following areas:
<ul>
  <li>Data distribution</li>
  <li>Load Balancing</li>
  <li>Backup and recovery</li>
</ul>You can check replication status and start a slave with the following
commands: SHOW SLAVE STATUS \G;<br/>START SLAVE;
<a href="http://dev.mysql.com/doc/refman/5.0/en/replication-faq.html"
  target="_blank" >MySQL Manual: Replication FAQ</a>

To find out more about this format go to the wikipedia.org web site.

This section documents the steps required to create a rule. Before attempting to create a rule, please review the preceding sections of this chapter.

This example creates a rule that checks the number of rows in a table. Having 50,000 rows in this table is deemed to warrant a critical alert. Lesser numbers are assigned to informational and warning level alerts.

Begin by navigating to the Advisors tab and clicking the manage rules link. Then choose the create rule button.

Create your custom rule by following these steps:

Once the rule is created it needs to be scheduled against the server that contains the database table you wish to monitor. For instructions on scheduling rules see Section 5.2, “Scheduling Rules”.

Section 5.4.7, “Creating a New Rule: An Example” shows how to create a custom rule and Appendix G, Data Collection Items shows the data items that can be used in rule creation. However, in some circumstances you may want to create a rule that uses a custom data collection item.

This section describes how to create a custom data collection item. The steps are as follows:

<?xml version="1.0" encoding="utf-8"?>
<classes>
  <class>
    <namespace>mysql</namespace>
    <classname>innodb_min_free</classname>
    <query><![CDATA[SELECT MIN(substring_index(substring_index(table_comment," ",3)," ",-1)/1024/1024)
      as Free FROM INFORMATION_SCHEMA.TABLES WHERE engine = 'InnoDB']]></query>
  </class>
</classes>

<?xml version="1.0" encoding="utf-8"?>
<classes>
  <class>
    <namespace>mysql</namespace>
    <classname>innodb_min_free</classname>
    <query><![CDATA[SELECT MIN(data_free/1024/1024)
      as Free FROM INFORMATION_SCHEMA.TABLES WHERE engine = 'InnoDB']]></query>
  </class>
</classes>

[mysql-proxy]
 ...
item-files = items-mysql-monitor.xml,innodb_min_free.xml
 ...

Rather than opening your web browser and blacking out a server by typing entries into the address bar, you can write a script to achieve the same effect. This section documents a sample blackout script that can be run from the command line.

Create the following file and save it as blackout.pl.

  #!/usr/bin/perl

  use LWP 5.64;

  # USAGE: blackout.pl servicemanager:18080 admin password servername:3306 true

  # $ARGV[0] = management server hostname:port
  # $ARGV[1] = management server username
  # $ARGV[2] = management server password
  # $ARGV[3] = mysqld managed instance server name and port
  # $ARGV[4] = blackout state (true/false)


  my $browser = LWP::UserAgent->new;
  $browser->credentials(
    $ARGV[0],
    '',
    $ARGV[1],
    $ARGV[2]
  );

  my $url = URI->new('http://'.$ARGV[0].'/rest');

  $url->query_form( # And here the form data pairs:
    'command' => 'blackout',
    'server_name' => $ARGV[3],
    'blackout_state' => $ARGV[4]
  );

  my $response = $browser->post( $url );

  if (!$response->is_success) {
    die $response->status_line . "\n";
  }

On Unix systems use the chmod +x blackout.pl command to make the file executable.

At the command line enter blackout.pl servicemanager:18080 admin password servername:3306 true.

If you are unsure of the host name and port to use, check the configuration_report.txt file. Be sure to specify the correct port for the Tomcat server. Specify the server you wish to blackout using the name that appears in the Server Tree, being sure to include a colon and port number as shown in the preceding example. Make sure that the user you specify is a "manager". Specifying a user with "dba" rights only will not black out a server and no error will be displayed.

You can confirm that a server is blacked out by looking at the server name in the Dashboard; the name of a blacked out server is greyed. To end the blackout, run the same script, changing the final argument to false.