17.5.5 Performing a Rolling Restart of a MySQL Cluster

This section discusses how to perform a rolling restart of a MySQL Cluster installation, so called because it involves stopping and starting (or restarting) each node in turn, so that the cluster itself remains operational. This is often done as part of a rolling upgrade or rolling downgrade, where high availability of the cluster is mandatory and no downtime of the cluster as a whole is permissible. Where we refer to upgrades, the information provided here also generally applies to downgrades as well.

There are a number of reasons why a rolling restart might be desirable. These are described in the next few paragraphs.

Configuration change.  To make a change in the cluster's configuration, such as adding an SQL node to the cluster, or setting a configuration parameter to a new value.

MySQL Cluster software upgrade or downgrade.  To upgrade the cluster to a newer version of the MySQL Cluster software (or to downgrade it to an older version). This is usually referred to as a rolling upgrade (or rolling downgrade, when reverting to an older version of MySQL Cluster).

Change on node host.  To make changes in the hardware or operating system on which one or more MySQL Cluster node processes are running.

System reset (cluster reset).  To reset the cluster because it has reached an undesirable state. In such cases it is often desirable to reload the data and metadata of one or more data nodes. This can be done in any of three ways:

Resource Recovery.  To free memory previously allocated to a table by successive INSERT and DELETE operations, for re-use by other MySQL Cluster tables.

The process for performing a rolling restart may be generalized as follows:

  1. Stop all cluster management nodes (ndb_mgmd processes), reconfigure them, then restart them.

    (If there are multiple management servers and you are using MySQL Cluster NDB 7.0.8 or later, see Rolling restarts with multiple management servers (MySQL Cluster NDB 7.0.8 and later).)

  2. Stop, reconfigure, then restart each cluster data node (ndbd process) in turn.

  3. Stop, reconfigure, then restart each cluster SQL node (mysqld process) in turn.

The specifics for implementing a given rolling upgrade depend upon the changes being made. A more detailed view of the process is presented here:

MySQL Cluster Rolling Restarts (By Type)

In the previous diagram, the Stop and Start steps indicate that the process must be stopped completely using a shell command (such as kill on most Unix systems) or the management client STOP command, then started again from a system shell by invoking the ndbd or ndb_mgmd executable as appropriate. On Windows (MySQL Cluster NDB 7.1.3 and later), you can also use the system NET START and NET STOP commands or the Windows Service Manager to start and stop nodes which have been installed as Windows services (see Section 17.2.2.4, “Installing MySQL Cluster Processes as Windows Services”).

Restart indicates that the process may be restarted using the ndb_mgm management client RESTART command (see Section 17.5.2, “Commands in the MySQL Cluster Management Client”).

Prior to MySQL Cluster NDB 6.3.29 and MySQL Cluster NDB 7.0.10.  When performing an upgrade or downgrade of the cluster software, you must upgrade or downgrade the management nodes first, then the data nodes, and finally the SQL nodes. Doing so in any other order may leave the cluster in an unusable state.

MySQL Cluster NDB 6.3.29 and later; MySQL Cluster NDB 7.0.10 and later.  MySQL Cluster supports a more flexible order for upgrading the cluster nodes. When upgrading a cluster running MySQL Cluster NDB 6.3.29 or later, or a cluster that is running MySQL Cluster NDB 7.0.10 or later, you may upgrade API nodes (including SQL nodes) before upgrading the management nodes, data nodes, or both. In other words, you are permitted to upgrade the API and SQL nodes in any order. This is subject to the following provisions:

See also Bug #48528 and Bug #49163.

Rolling restarts with multiple management servers (MySQL Cluster NDB 7.0.8 and later).  A change in MySQL Cluster NDB 7.0.8 in the way in which management nodes obtain configuration data makes it necessary to alter the procedure previously outlined, when performing a rolling restart of a MySQL Cluster with multiple management nodes. (Previously, a management node checked only its own configuration cache, but in MySQL Cluster NDB 7.0.8 and later, ndb_mgmd also checks to see if any other management node is running, and, if so, tries to use that node's configuration data.) To keep this from occurring, and to force ndb_mgmd to reread its configuration file, perform the following steps:

  1. Stop all MySQL Cluster ndb_mgmd processes.

  2. Update all config.ini files.

  3. Start a single ndb_mgmd with --reload, --initial, or both options as desired.

  4. If you started the first ndb_mgmd with the --initial option, you must also start any remaining ndb_mgmd processes using --initial.

    Regardless of any other options used when starting the first ndb_mgmd, you should not start any remaining ndb_mgmd processes after the first one using --reload.

  5. Complete the rolling restarts of the data nodes and API nodes as normal.

See also Bug #45495 and Bug #46488.

MySQL Cluster NDB 7.0.24 and later; MySQL Cluster NDB 7.1.13 and later.  When performing a rolling restart to update the cluster's configuration, you can use the config_generation column of the ndbinfo.nodes table to keep track of which data nodes have been successfully restarted with the new configuration. See Section 17.5.10.13, “The ndbinfo nodes Table”.