Chapter 17. MySQL Cluster NDB 6.X/7.X

Table of Contents

17.1. MySQL Cluster Overview
17.1.1. MySQL Cluster Core Concepts
17.1.2. MySQL Cluster Nodes, Node Groups, Replicas, and Partitions
17.1.3. MySQL Cluster Hardware, Software, and Networking Requirements
17.1.4. MySQL Cluster Development History
17.1.5. Known Limitations of MySQL Cluster
17.2. MySQL Cluster Multi-Computer How-To
17.2.1. Installing MySQL Cluster on Linux
17.2.2. Installing MySQL Cluster on Windows
17.2.3. Loading Sample Data into MySQL Cluster and Performing Queries
17.2.4. Safe Shutdown and Restart of MySQL Cluster
17.2.5. Upgrading and Downgrading MySQL Cluster
17.3. MySQL Cluster Configuration
17.3.1. Quick Test Setup of MySQL Cluster
17.3.2. MySQL Cluster Configuration Files
17.3.3. Overview of MySQL Cluster Configuration Parameters
17.3.4. MySQL Server Options and Variables for MySQL Cluster
17.3.5. Using High-Speed Interconnects with MySQL Cluster
17.4. MySQL Cluster Programs
17.4.1. MySQL Server Usage for MySQL Cluster
17.4.2. ndbd — The MySQL Cluster Data Node Daemon
17.4.3. ndbmtd — The MySQL Cluster Data Node Daemon (Multi-Threaded)
17.4.4. ndb_mgmd — The MySQL Cluster Management Server Daemon
17.4.5. ndb_mgm — The MySQL Cluster Management Client
17.4.6. ndb_config — Extract MySQL Cluster Configuration Information
17.4.7. ndb_cpcd — Automate Testing for NDB Development
17.4.8. ndb_delete_all — Delete All Rows from an NDB Table
17.4.9. ndb_desc — Describe NDB Tables
17.4.10. ndb_drop_index — Drop Index from an NDB Table
17.4.11. ndb_drop_table — Drop an NDB Table
17.4.12. ndb_error_reporter — NDB Error-Reporting Utility
17.4.13. ndb_print_backup_file — Print NDB Backup File Contents
17.4.14. ndb_print_schema_file — Print NDB Schema File Contents
17.4.15. ndb_print_sys_file — Print NDB System File Contents
17.4.16. ndbd_redo_log_reader — Check and Print Content of Cluster Redo Log
17.4.17. ndb_restore — Restore a MySQL Cluster Backup
17.4.18. ndb_select_all — Print Rows from an NDB Table
17.4.19. ndb_select_count — Print Row Counts for NDB Tables
17.4.20. ndb_show_tables — Display List of NDB Tables
17.4.21. ndb_size.pl — NDBCLUSTER Size Requirement Estimator
17.4.22. ndb_waiter — Wait for MySQL Cluster to Reach a Given Status
17.4.23. Options Common to MySQL Cluster Programs
17.5. Management of MySQL Cluster
17.5.1. Summary of MySQL Cluster Start Phases
17.5.2. Commands in the MySQL Cluster Management Client
17.5.3. Online Backup of MySQL Cluster
17.5.4. Event Reports Generated in MySQL Cluster
17.5.5. MySQL Cluster Log Messages
17.5.6. MySQL Cluster Single User Mode
17.5.7. Quick Reference: MySQL Cluster SQL Statements
17.5.8. The ndbinfo MySQL Cluster Information Database
17.5.9. MySQL Cluster Security Issues
17.5.10. MySQL Cluster Disk Data Tables
17.5.11. Adding MySQL Cluster Data Nodes Online
17.6. MySQL Cluster Replication
17.6.1. MySQL Cluster Replication: Abbreviations and Symbols
17.6.2. MySQL Cluster Replication: Assumptions and General Requirements
17.6.3. Known Issues in MySQL Cluster Replication
17.6.4. MySQL Cluster Replication Schema and Tables
17.6.5. Preparing the MySQL Cluster for Replication
17.6.6. Starting MySQL Cluster Replication (Single Replication Channel)
17.6.7. Using Two Replication Channels for MySQL Cluster Replication
17.6.8. Implementing Failover with MySQL Cluster Replication
17.6.9. MySQL Cluster Backups With MySQL Cluster Replication
17.6.10. MySQL Cluster Replication: Multi-Master and Circular Replication
17.6.11. MySQL Cluster Replication Conflict Resolution
17.7. Changes in MySQL Cluster NDB 6.X and 7.X
17.7.1. Changes in MySQL Cluster NDB 7.1
17.7.2. Changes in MySQL Cluster NDB 7.0
17.7.3. Changes in MySQL Cluster NDB 6.3
17.7.4. Changes in MySQL Cluster NDB 6.2
17.7.5. Changes in MySQL Cluster NDB 6.1
17.7.6. Release Series Changelogs: MySQL Cluster NDB 6.X and 7.X

This chapter contains information about MySQL Cluster, which is a high-availability, high-redundancy version of MySQL adapted for the distributed computing environment. Current releases of MySQL Cluster use versions 6 and 7 of the NDBCLUSTER storage engine (also known as NDB) to enable running several computers with MySQL servers and other software in a cluster.

Beginning with MySQL 5.1.24, support for the NDBCLUSTER storage engine was removed from the standard MySQL server binaries built by MySQL. Instead, users of MySQL Cluster binaries built by MySQL should upgrade to the most recent binary release of MySQL Cluster NDB 7.0 or MySQL Cluster 7.1 for supported platforms—these include RPMs that should work with most Linux distributions. MySQL Cluster users who build from source should be aware that, also beginning with MySQL 5.1.24, NDBCLUSTER sources in the standard MySQL 5.1 tree are no longer maintained; these users should use the sources provided for MySQL Cluster NDB 7.0 or later. (Locations where the sources can be obtained are listed later in this section.)

Note

MySQL Cluster NDB 6.1, 6.2, and 6.3 were formerly known as “MySQL Cluster Carrier Grade Edition”. Beginning with MySQL Cluster NDB 6.2.15 and MySQL Cluster NDB 6.3.14, this term is no longer applied to the MySQL Cluster software—which is now known simply as “MySQL Cluster”—but rather to a commercial licensing and support package. You can learn more about available options for commercial licensing of MySQL Cluster from http://mysql.com/products/database/cluster/features.html, on the MySQL web site.

This chapter contains information about MySQL Cluster in MySQL 5.1 mainline releases through MySQL 5.1.23, MySQL Cluster NDB 6.2 releases through 5.1.51-ndb-6.2.19, MySQL Cluster NDB 6.3 releases through 5.1.51-ndb-6.3.39, MySQL Cluster NDB 7.0 releases through 5.1.51-ndb-7.0.20 and MySQL Cluster NDB 7.1 releases through 5.1.51-ndb-7.1.9. Currently, the MySQL Cluster NDB 7.0 (formerly known as “MySQL Cluster NDB 6.4”) and MySQL Cluster NDB 7.1 release series are Generally Available (GA). MySQL Cluster NDB 6.2 and MySQL Cluster NDB 6.3 are previous GA release series; although these are still supported, we recommend that new deployments use MySQL Cluster NDB 7.0 or MySQL Cluster NDB 7.1.

This chapter also contains historical information about MySQL Cluster NDB 6.1, although this release series is no longer in active development, and no longer supported for new deployments. You should upgrade to MySQL Cluster 6.3 or a MySQL Cluster NDB 7.x release series as soon as possible.

Platforms supported. MySQL Cluster is currently available and supported on a number of platforms, including Linux, Solaris, Mac OS X, and other Unix-style operating systems on a variety of hardware. Beginning with MySQL Cluster NDB 7.1.3, MySQL Cluster is also available for production use on Microsoft Windows platforms. For exact levels of support available for MySQL Cluster on specific combinations of operating system versions, operating system distributions, and hardware platforms, please refer to http://www.mysql.com/support/supportedplatforms/cluster.html, maintained by the MySQL Support Team on the MySQL web site.

Availability. MySQL Cluster NDB 6.3, MySQL Cluster NDB 7.0, and MySQL Cluster NDB 7.1 binary and source packages are available for supported platforms from http://dev.mysql.com/downloads/cluster/.

Note

Binary releases and RPMs were not available for MySQL Cluster NDB 6.2 prior to MySQL Cluster NDB 6.2.15.

MySQL Cluster release numbers. Starting with MySQL Cluster NDB 6.1 and MySQL Cluster NDB 6.2, MySQL Cluster follows a somewhat different release pattern from the mainline MySQL 5.1 Cluster series of releases. In this Manual and other MySQL documentation, we identify these and later MySQL Cluster releases employing a version number that begins with “NDB”. This version number is that of the NDBCLUSTER storage engine used in the release, and not of the MySQL server version on which the MySQL Cluster release is based.

Version strings used in MySQL Cluster NDB 6.x and 7.x software. The version string displayed by MySQL Cluster NDB 6.x and 7.x software uses this format:

mysql-mysql_server_version-ndb-ndbcluster_engine_version

mysql_server_version represents the version of the MySQL Server on which the MySQL Cluster release is based. For all MySQL Cluster NDB 6.x and 7.x releases, this is “5.1”. ndbcluster_engine_version is the version of the NDBCLUSTER storage engine used by this release of the MySQL Cluster software. You can see this format used in the mysql client, as shown here:

shell> mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.51-ndb-7.1.9 Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> SELECT VERSION()\G
*************************** 1. row ***************************
VERSION(): 5.1.51-ndb-7.1.9
1 row in set (0.00 sec)

This version string is also displayed in the output of the SHOW command in the ndb_mgm client:

ndb_mgm> SHOW
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.0.10.6  (5.1.51-ndb-7.1.9, Nodegroup: 0, Master)
id=2    @10.0.10.8  (5.1.51-ndb-7.1.9, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=3    @10.0.10.2  (5.1.51-ndb-7.1.9)

[mysqld(API)]   2 node(s)
id=4    @10.0.10.10  (5.1.51-ndb-7.1.9)
id=5 (not connected, accepting connect from any host)

The version string identifies the mainline MySQL version from which the MySQL Cluster release was branched and the version of the NDBCLUSTER storage engine used. For example, the full version string for MySQL Cluster NDB 7.0.5 (the first GA MySQL Cluster NDB 7.0 binary release) was mysql-5.1.32-ndb-7.0.5. From this we can determine the following:

New MySQL Cluster releases are numbered according to updates in the NDB storage engine, and do not necessarily correspond in a linear fashion with mainline MySQL Server releases. For example, MySQL Cluster NDB 7.0.5 (as previously noted) is based on MySQL 5.1.32, and MySQL Cluster NDB 7.0.6 is based on MySQL 5.1.34 (version string: mysql-5.1.34-ndb-7.0.6).

Compatibility with standard MySQL 5.1 releases. While many standard MySQL schemas and applications can work using MySQL Cluster, it is also true that unmodified applications and database schemas may be slightly incompatible or have suboptimal performance when run using MySQL Cluster (see Section 17.1.5, “Known Limitations of MySQL Cluster”). Most of these issues can be overcome, but this also means that you are very unlikely to be able to switch an existing application datastore—that currently uses, for example, MyISAM or InnoDB—to use the NDB storage engine without allowing for the possibility of changes in schemas, queries, and applications. Moreover, from MySQL 5.1.24 onwards, the MySQL Server and MySQL Cluster codebases diverge considerably (and NDB storage engine support dropped from subsequent MySQL Server releases), so that the standard mysqld cannot function as a dropin replacement for the version of mysqld that is supplied with MySQL Cluster.

MySQL Cluster development source trees. MySQL Cluster development trees can also be accessed from https://code.launchpad.net/~mysql/:

The MySQL Cluster development sources maintained at https://code.launchpad.net/~mysql/ are licensed under the GPL. For information about obtaining MySQL sources using Bazaar and building them yourself, see Section 2.11.4, “Installing from the Development Source Tree”.

Currently, MySQL Cluster NDB 6.3, MySQL Cluster NDB 7.0, and MySQL Cluster NDB 7.1 releases are all Generally Available (GA), although we recommend that new deployments use MySQL Cluster NDB 7.1. MySQL Cluster NDB 6.1 is no longer in active development. For an overview of major features added in MySQL Cluster NDB 6.x and 7.x releases, see Section 17.1.4, “MySQL Cluster Development History”.

This chapter represents a work in progress, and its contents are subject to revision as MySQL Cluster continues to evolve. Additional information regarding MySQL Cluster can be found on the MySQL Web site at http://www.mysql.com/products/cluster/.

Additional Resources. More information may be found in the following places:

17.1. MySQL Cluster Overview

MySQL Cluster is a technology that enables clustering of in-memory databases in a shared-nothing system. The shared-nothing architecture enables the system to work with very inexpensive hardware, and with a minimum of specific requirements for hardware or software.

MySQL Cluster is designed not to have any single point of failure. In a shared-nothing system, each component is expected to have its own memory and disk, and the use of shared storage mechanisms such as network shares, network file systems, and SANs is not recommended or supported.

MySQL Cluster integrates the standard MySQL server with an in-memory clustered storage engine called NDB (which stands for “Network DataBase”). In our documentation, the term NDB refers to the part of the setup that is specific to the storage engine, whereas “MySQL Cluster” refers to the combination of one or more MySQL servers with the NDB storage engine.

A MySQL Cluster consists of a set of computers, known as hosts, each running one or more processes. These processes, known as nodes, may include MySQL servers (for access to NDB data), data nodes (for storage of the data), one or more management servers, and possibly other specialized data access programs. The relationship of these components in a MySQL Cluster is shown here:

MySQL Cluster Components

All these programs work together to form a MySQL Cluster (see Section 17.4, “MySQL Cluster Programs”. When data is stored by the NDB storage engine, the tables (and table data) are stored in the data nodes. Such tables are directly accessible from all other MySQL servers (SQL nodes) in the cluster. Thus, in a payroll application storing data in a cluster, if one application updates the salary of an employee, all other MySQL servers that query this data can see this change immediately.

Although a MySQL Cluster SQL node uses the mysqld server damon, it differs in a number of critical respects from the mysqld binary supplied with the MySQL 5.1 distributions, and the two versions of mysqld are not interchangeable.

In addition, a MySQL server that is not connected to a MySQL Cluster cannot use the NDB storage engine and cannot access any MySQL Cluster data.

The data stored in the data nodes for MySQL Cluster can be mirrored; the cluster can handle failures of individual data nodes with no other impact than that a small number of transactions are aborted due to losing the transaction state. Because transactional applications are expected to handle transaction failure, this should not be a source of problems.

Individual nodes can be stopped and restarted, and can then rejoin the system (cluster). Rolling restarts (in which all nodes are restarted in turn) are used in making configuration changes and software upgrades (see Section 17.2.5.1, “Performing a Rolling Restart of a MySQL Cluster”). In MySQL Cluster NDB 7.0 and later, rolling restarts are also used as part of the process of adding new data nodes online (see Section 17.5.11, “Adding MySQL Cluster Data Nodes Online”). For more information about data nodes, how they are organized in a MySQL Cluster, and how they handle and store MySQL Cluster data, see Section 17.1.2, “MySQL Cluster Nodes, Node Groups, Replicas, and Partitions”.

Backing up and restoring MySQL Cluster databases can be done using the NDB native functionality found in the MySQL Cluster management client and the ndb_restore program included in the MySQL Cluster distribution. For more information, see Section 17.5.3, “Online Backup of MySQL Cluster”, and Section 17.4.17, “ndb_restore — Restore a MySQL Cluster Backup”. You can also use the standard MySQL functionality provided for this purpose in mysqldump and the MySQL server. See Section 4.5.4, “mysqldump — A Database Backup Program”, for more information.

MySQL Cluster nodes can use a number of different transport mechanisms for inter-node communications, including TCP/IP using standard 100 Mbps or faster Ethernet hardware. It is also possible to use the high-speed Scalable Coherent Interface (SCI) protocol with MySQL Cluster, although this is not required to use MySQL Cluster. SCI requires special hardware and software; see Section 17.3.5, “Using High-Speed Interconnects with MySQL Cluster”, for more about SCI and using it with MySQL Cluster.

17.1.1. MySQL Cluster Core Concepts

NDBCLUSTER (also known as NDB) is an in-memory storage engine offering high-availability and data-persistence features.

The NDBCLUSTER storage engine can be configured with a range of failover and load-balancing options, but it is easiest to start with the storage engine at the cluster level. MySQL Cluster's NDB storage engine contains a complete set of data, dependent only on other data within the cluster itself.

The “Cluster” portion of MySQL Cluster is configured independently of the MySQL servers. In a MySQL Cluster, each part of the cluster is considered to be a node.

Note

In many contexts, the term “node” is used to indicate a computer, but when discussing MySQL Cluster it means a process. It is possible to run multiple nodes on a single computer; for a computer on which one or more cluster nodes are being run we use the term cluster host.

There are three types of cluster nodes, and in a minimal MySQL Cluster configuration, there will be at least three nodes, one of each of these types:

  • Management node: The role of this type of node is to manage the other nodes within the MySQL Cluster, performing such functions as providing configuration data, starting and stopping nodes, running backup, and so forth. Because this node type manages the configuration of the other nodes, a node of this type should be started first, before any other node. An MGM node is started with the command ndb_mgmd.

  • Data node: This type of node stores cluster data. There are as many data nodes as there are replicas, times the number of fragments (see Section 17.1.2, “MySQL Cluster Nodes, Node Groups, Replicas, and Partitions”). For example, with two replicas, each having two fragments, you need four data nodes. One replica is sufficient for data storage, but provides no redundancy; therefore, it is recommended to have 2 (or more) replicas to provide redundancy, and thus high availability. A data node is started with the command ndbd (see Section 17.4.2, “ndbd — The MySQL Cluster Data Node Daemon”). In MySQL Cluster NDB 7.0 and later, ndbmtd can also be used for the data node process; see Section 17.4.3, “ndbmtd — The MySQL Cluster Data Node Daemon (Multi-Threaded)”, for more information.

    MySQL Cluster tables in MySQL 5.1 are normally stored completely in memory rather than on disk (this is why we refer to MySQL cluster as an in-memory database). In MySQL 5.1, MySQL Cluster NDB 6.X, and later, some MySQL Cluster data can be stored on disk; see Section 17.5.10, “MySQL Cluster Disk Data Tables”, for more information.

  • SQL node: This is a node that accesses the cluster data. In the case of MySQL Cluster, an SQL node is a traditional MySQL server that uses the NDBCLUSTER storage engine. An SQL node is a mysqld process started with the --ndbcluster and --ndb-connectstring options, which are explained elsewhere in this chapter, possibly with additional MySQL server options as well.

    An SQL node is actually just a specialized type of API node, which designates any application which accesses MySQL Cluster data. Another example of an API node is the ndb_restore utility that is used to restore a cluster backup. It is possible to write such applications using the NDB API. For basic information about the NDB API, see Getting Started with the NDB API.

Important

It is not realistic to expect to employ a three-node setup in a production environment. Such a configuration provides no redundancy; to benefit from MySQL Cluster's high-availability features, you must use multiple data and SQL nodes. The use of multiple management nodes is also highly recommended.

For a brief introduction to the relationships between nodes, node groups, replicas, and partitions in MySQL Cluster, see Section 17.1.2, “MySQL Cluster Nodes, Node Groups, Replicas, and Partitions”.

Configuration of a cluster involves configuring each individual node in the cluster and setting up individual communication links between nodes. MySQL Cluster is currently designed with the intention that data nodes are homogeneous in terms of processor power, memory space, and bandwidth. In addition, to provide a single point of configuration, all configuration data for the cluster as a whole is located in one configuration file.

The management server manages the cluster configuration file and the cluster log. Each node in the cluster retrieves the configuration data from the management server, and so requires a way to determine where the management server resides. When interesting events occur in the data nodes, the nodes transfer information about these events to the management server, which then writes the information to the cluster log.

In addition, there can be any number of cluster client processes or applications. These include standard MySQL clients, NDB-specific API programs, and management clients. These are described in the next few paragraphs.

Standard MySQL clients. MySQL Cluster can be used with existing MySQL applications written in PHP, Perl, C, C++, Java, Python, Ruby, and so on. Such client applications send SQL statements to and receive responses from MySQL servers acting as MySQL Cluster SQL nodes in much the same way that they interact with standalone MySQL servers.

MySQL clients using a MySQL Cluster as a data source can be modified to take advantage of the ability to connect with multiple MySQL servers to achieve load balancing and failover. For example, Java clients using Connector/J 5.0.6 and later can use jdbc:mysql:loadbalance:// URLs (improved in Connector/J 5.1.7) to achieve load balancing transparently; for more information about using Connector/J with MySQL Cluster, see Using Connector/J with MySQL Cluster.

NDB client programs. Client programs can be written that access MySQL Cluster data directly from the NDBCLUSTER storage engine, bypassing any MySQL Servers that may connected to the cluster, using the NDB API, a high-level C++ API. Such applications may be useful for specialized purposes where an SQL interface to the data is not needed. For more information, see The NDB API.

Beginning with MySQL Cluster NDB 7.1, NDB-specific Java applications can also be written for MySQL Cluster, using the MySQL Cluster Connector for Java. This MySQL Cluster Connector includes ClusterJ, a high-level database API similar to object-relational mapping persistence frameworks such as Hibernate and JPA that connect directly to NDBCLUSTER, and so does not require access to a MySQL Server. Support is also provided in MySQL Cluster NDB 7.1 and later for ClusterJPA, an OpenJPA implementation for MySQL Cluster that leverages the strengths of ClusterJ and JDBC; ID lookups and other fast operations are performed using ClusterJ (bypassing the MySQL Server), while more complex queries that can benefit from MySQL's query optimizer are sent through the MySQL Server, using JDBC. See Java and MySQL Cluster, and The ClusterJ API and Data Object Model, for more information.

Management clients. These clients connect to the management server and provide commands for starting and stopping nodes gracefully, starting and stopping message tracing (debug versions only), showing node versions and status, starting and stopping backups, and so on. An example of this type of program is the ndb_mgm management client supplied with MySQL Cluster (see Section 17.4.5, “ndb_mgm — The MySQL Cluster Management Client”). Such applications can be written using the MGM API, a C-language API that communicates directly with one or more MySQL Cluster management servers. For more information, see The MGM API.

Event logs. MySQL Cluster logs events by category (startup, shutdown, errors, checkpoints, and so on), priority, and severity. A complete listing of all reportable events may be found in Section 17.5.4, “Event Reports Generated in MySQL Cluster”. Event logs are of the two types listed here:

  • Cluster log: Keeps a record of all desired reportable events for the cluster as a whole.

  • Node log: A separate log which is also kept for each individual node.

Note

Under normal circumstances, it is necessary and sufficient to keep and examine only the cluster log. The node logs need be consulted only for application development and debugging purposes.

Checkpoint. Generally speaking, when data is saved to disk, it is said that a checkpoint has been reached. More specific to MySQL Cluster, a checkpoint is a point in time where all committed transactions are stored on disk. With regard to the NDB storage engine, there are two types of checkpoints which work together to ensure that a consistent view of the cluster's data is maintained. These are shown in the following list:

  • Local Checkpoint (LCP): This is a checkpoint that is specific to a single node; however, LCP's take place for all nodes in the cluster more or less concurrently. An LCP involves saving all of a node's data to disk, and so usually occurs every few minutes. The precise interval varies, and depends upon the amount of data stored by the node, the level of cluster activity, and other factors.

  • Global Checkpoint (GCP): A GCP occurs every few seconds, when transactions for all nodes are synchronized and the redo-log is flushed to disk.

17.1.2. MySQL Cluster Nodes, Node Groups, Replicas, and Partitions

This section discusses the manner in which MySQL Cluster divides and duplicates data for storage.

A number of concepts central to an understanding of this topic are discussed in the next few paragraphs.

(Data) Node. An ndbd process, which stores a replica —that is, a copy of the partition (see below) assigned to the node group of which the node is a member.

Each data node should be located on a separate computer. While it is also possible to host multiple ndbd processes on a single computer, such a configuration is not supported.

It is common for the terms “node” and “data node” to be used interchangeably when referring to an ndbd process; where mentioned, management nodes (ndb_mgmd processes) and SQL nodes (mysqld processes) are specified as such in this discussion.

Node Group. A node group consists of one or more nodes, and stores partitions, or sets of replicas (see next item).

The number of node groups in a MySQL Cluster is not directly configurable; it is a function of the number of data nodes and of the number of replicas (NoOfReplicas configuration parameter), as shown here:

[number_of_node_groups] = number_of_data_nodes / NoOfReplicas

Thus, a MySQL Cluster with 4 data nodes has 4 node groups if NoOfReplicas is set to 1 in the config.ini file, 2 node groups if NoOfReplicas is set to 2, and 1 node group if NoOfReplicas is set to 4. Replicas are discussed later in this section; for more information about NoOfReplicas, see Section 17.3.2.6, “Defining MySQL Cluster Data Nodes”.

Note

All node groups in a MySQL Cluster must have the same number of data nodes.

Prior to MySQL Cluster NDB 7.0, it was not possible to add new data nodes to a MySQL Cluster without shutting down the cluster completely and reloading all of its data. In MySQL Cluster NDB 7.0 (beginning with MySQL Cluster version NDB 6.4.0), you can add new node groups (and thus new data nodes) to a running MySQL Cluster—see Section 17.5.11, “Adding MySQL Cluster Data Nodes Online”, for information about how this can be done.

Partition. This is a portion of the data stored by the cluster. There are as many cluster partitions as nodes participating in the cluster. Each node is responsible for keeping at least one copy of any partitions assigned to it (that is, at least one replica) available to the cluster.

A replica belongs entirely to a single node; a node can (and usually does) store several replicas.

NDB and user-defined partitioning. MySQL Cluster normally partitions NDBCLUSTER tables automatically. However, in MySQL 5.1 and later MySQL Cluster releases, it is possible to employ user-defined partitioning with NDBCLUSTER tables. This is subject to the following limitations:

  1. Only KEY and LINEAR KEY partitioning schemes can be used with NDBCLUSTER tables.

  2. When using ndbd, the maximum number of partitions that may be defined explicitly for any NDBCLUSTER table is 4 per node group, times NoOfReplicas. (The number of node groups in a MySQL Cluster is determined as discussed previously in this section.)

    When using ndbmtd, this maximum is also affected by the number of local query handler threads, which is determined by the value of the MaxNoOfExecutionThreads configuration parameter. In such cases, the maxmimum number of partitions that may be defined explicitly for an NDB table is equal to MaxNoOfExecutionThreads * NoOfReplicas * [number of data nodes].

    See Section 17.4.3, “ndbmtd — The MySQL Cluster Data Node Daemon (Multi-Threaded)”, for more information.

For more information relating to MySQL Cluster and user-defined partitioning, see Section 17.1.5, “Known Limitations of MySQL Cluster”, and Section 18.5.2, “Partitioning Limitations Relating to Storage Engines”.

Replica. This is a copy of a cluster partition. Each node in a node group stores a replica. Also sometimes known as a partition replica. The number of replicas is equal to the number of nodes per node group.

The following diagram illustrates a MySQL Cluster with four data nodes, arranged in two node groups of two nodes each; nodes 1 and 2 belong to node group 0, and nodes 3 and 4 belong to node group 1. Note that only data (ndbd) nodes are shown here; although a working cluster requires an ndb_mgm process for cluster management and at least one SQL node to access the data stored by the cluster, these have been omitted in the figure for clarity.

A MySQL Cluster, with 2 node groups having 2
        nodes each

The data stored by the cluster is divided into four partitions, numbered 0, 1, 2, and 3. Each partition is stored—in multiple copies—on the same node group. Partitions are stored on alternate node groups as follows:

  • Partition 0 is stored on node group 0; a primary replica (primary copy) is stored on node 1, and a backup replica (backup copy of the partition) is stored on node 2.

  • Partition 1 is stored on the other node group (node group 1); this partition's primary replica is on node 3, and its backup replica is on node 4.

  • Partition 2 is stored on node group 0. However, the placing of its two replicas is reversed from that of Partition 0; for Partition 2, the primary replica is stored on node 2, and the backup on node 1.

  • Partition 3 is stored on node group 1, and the placement of its two replicas are reversed from those of partition 1. That is, its primary replica is located on node 4, with the backup on node 3.

What this means regarding the continued operation of a MySQL Cluster is this: so long as each node group participating in the cluster has at least one node operating, the cluster has a complete copy of all data and remains viable. This is illustrated in the next diagram.

Nodes required to keep a 2x2 cluster
        viable

In this example, where the cluster consists of two node groups of two nodes each, any combination of at least one node in node group 0 and at least one node in node group 1 is sufficient to keep the cluster “alive” (indicated by arrows in the diagram). However, if both nodes from either node group fail, the remaining two nodes are not sufficient (shown by the arrows marked out with an X); in either case, the cluster has lost an entire partition and so can no longer provide access to a complete set of all cluster data.

17.1.3. MySQL Cluster Hardware, Software, and Networking Requirements

One of the strengths of MySQL Cluster is that it can be run on commodity hardware and has no unusual requirements in this regard, other than for large amounts of RAM, due to the fact that all live data storage is done in memory. (It is possible to reduce this requirement using Disk Data tables—see Section 17.5.10, “MySQL Cluster Disk Data Tables”, for more information about these.) Naturally, multiple and faster CPUs can enhance performance. Memory requirements for other MySQL Cluster processes are relatively small.

The software requirements for MySQL Cluster are also modest. Host operating systems do not require any unusual modules, services, applications, or configuration to support MySQL Cluster. For supported operating systems, a standard installation should be sufficient. The MySQL software requirements are simple: all that is needed is a production release of MySQL 5.1.51-ndb-7.0.20 or 5.1.51-ndb-7.1.9 to have MySQL Cluster support. It is not strictly necessary to compile MySQL yourself merely to be able to use MySQL Cluster. We assume that you are using the binaries appropriate to your platform, available from the MySQL Cluster software downloads page at http://dev.mysql.com/downloads/cluster/.

For communication between nodes, MySQL Cluster supports TCP/IP networking in any standard topology, and the minimum expected for each host is a standard 100 Mbps Ethernet card, plus a switch, hub, or router to provide network connectivity for the cluster as a whole. We strongly recommend that a MySQL Cluster be run on its own subnet which is not shared with machines not forming part of the cluster for the following reasons:

  • Security. Communications between MySQL Cluster nodes are not encrypted or shielded in any way. The only means of protecting transmissions within a MySQL Cluster is to run your MySQL Cluster on a protected network. If you intend to use MySQL Cluster for Web applications, the cluster should definitely reside behind your firewall and not in your network's De-Militarized Zone (DMZ) or elsewhere.

    See Section 17.5.9.1, “MySQL Cluster Security and Networking Issues”, for more information.

  • Efficiency. Setting up a MySQL Cluster on a private or protected network enables the cluster to make exclusive use of bandwidth between cluster hosts. Using a separate switch for your MySQL Cluster not only helps protect against unauthorized access to MySQL Cluster data, it also ensures that MySQL Cluster nodes are shielded from interference caused by transmissions between other computers on the network. For enhanced reliability, you can use dual switches and dual cards to remove the network as a single point of failure; many device drivers support failover for such communication links.

It is also possible to use the high-speed Scalable Coherent Interface (SCI) with MySQL Cluster, but this is not a requirement. See Section 17.3.5, “Using High-Speed Interconnects with MySQL Cluster”, for more about this protocol and its use with MySQL Cluster.

17.1.4. MySQL Cluster Development History

In this section, we discuss changes in the implementation of MySQL Cluster in MySQL 5.1, MySQL Cluster NDB 6.x, and MySQL Cluster NDB 7.x, as compared to earlier MySQL Cluster releases.

There are a number of significant changes in the implementation of the NDBCLUSTER storage engine in mainline MySQL 5.1 releases up to and including MySQL 5.1.23 as compared to that in MySQL 5.0; MySQL Cluster NDB 6.x and 7.x make further changes and improvements in MySQL Cluster in addition to these. The changes and features most likely to be of interest are shown in the following tables:

MySQL Cluster NDB 7.1
Production-level support for MySQL Cluster on Microsoft Windows platforms.
ndbinfo meta-information database
MySQL Cluster Connector for Java, including ClusterJ and OpenJPA (ClusterJPA) support
Native support for default column values
MySQL Cluster NDB 7.0
Multi-threaded data nodes (ndbmtd data node daemon)
Online addition of data nodes; online data redistribution
MySQL on Windows (alpha; source releases only)
Configuration cache
Backup snapshots (START BACKUP ... SNAPSHOTSTART, START BACKUP ... SNAPSHOTEND commands)
IPv6 support for geo-replication
Protected DDL operations
Dynamic buffering for NDB transporters
Increased flexibility in determining arbitration handling, using a new Arbitration data node configuration parameter
MySQL Cluster NDB 6.3
Conflict detection and resolution for multi-master replication
Compressed backups and local checkpoints
Support for OPTIMIZE TABLE
Parallel data node recovery
Enhanced transaction coordinator selection
Improved SQL statement performance metrics
Transaction batching
ndb_restore attribute promotion
Support for epoll (Linux only)
Distribution awareness
NDB thread locks; realtime extensions for multiple CPUs
MySQL Cluster NDB 6.2
Improved backup status reporting (BackupReportFrequency, REPORT BackupStatus)
Multiple connections per SQL node
Data access with NdbRecord (NDB API)
REPORT MemoryUsage command
Memory allocation improvements
Management client connection control
Micro-GCPs
Online ADD COLUMN; improved online index creation
MySQL Cluster NDB 6.1
Greater number of cluster nodes
Disabling of arbitration
Additional DUMP commands
Faster Disk Data backups
Batched slave updates
MySQL 5.1 (through 5.1.23)
MySQL Cluster Replication
Disk Data storage
Variable-size columns
User-defined partitioning
Autodiscovery of table schema changes
Online adding and dropping of indexes

17.1.4.1. MySQL Cluster Development in MySQL Cluster NDB 7.1

The following improvements to MySQL Cluster have been made in MySQL Cluster NDB 7.1.

  • MySQL Cluster information database (ndbinfo). The ndbinfo information database makes it possible to obtain real-time characteristics of a MySQL Cluster by issuing queries from the mysql client or other MySQL client applications. ndbinfo provides metadata specific to MySQL Cluster similarly to how the INFORMATION_SCHEMA database provides metadata for the standard MySQL Server. This eliminates much of the need to read log files, issue REPORT or DUMP commands in the ndb_mgm client, or parse the output of ndb_config in order to get configuration and status information from a running MySQL Cluster.

    For more information, see Section 17.5.8, “The ndbinfo MySQL Cluster Information Database”.

  • Java connectors for MySQL Cluster. The MySQL Cluster distribution now includes 2 new Java user APIs, ClusterJ and ClusterJPA. ClusterJ is an object-relational interface in a manner similar to that of Java persistence frameworks such as Hibernate. Cluster JPA is a reimplementation of OpenJPA. ClusterJ uses a backend library (NdbJTie) that provides access to the NDB storage engine without using a MySQL Server connection or JDBC. ClusterJPA also uses NdbJTie when it improves performance, but can also process complex queries using JDBC and a MySQL Server connection, where it can take advantage of the MySQL query optimizer.

    ClusterJ and Cluster JPA can also be made to work with recent MySQL Cluster NDB 7.0 releases although the necessary library and JAR files are included only in MySQL Cluster NDB 7.1.1 and later.

    For more information about using ClusterJ and ClusterJPA, see MySQL Cluster Connector for Java.

  • New CHANGE MASTER TO option for circular replication. Beginning with MySQL Cluster NDB 7.1.0, the CHANGE MASTER TO statement supports an IGNORE_SERVER_IDS option which takes a comma-separated list of server IDs and causes events originating from the corresponding servers to be ignored. (Log rotation and log deletion events are preserved.)

    See Section 12.5.2.1, “CHANGE MASTER TO Syntax”, as well as Section 12.4.5.36, “SHOW SLAVE STATUS Syntax”, for more information.

  • Native support for default column values. Starting with MySQL Cluster NDB 7.1.4, default values for table columns are stored in the NDB kernel rather than by the MySQL server as was done previously. This means that inserts on tables having column value defaults can be smaller and faster than before, because less data must be sent from SQL nodes to NDBCLUSTER.

    Tables created using previous MySQL Cluster releases can still be used in MySQL Cluster 7.1.4 and later; however, they do not support native default values until they are upgraded. You can upgrade a table with non-native default values to support native default values using an offline ALTER TABLE statement.

  • MySQL Cluster on Windows (Production). Beginning with MySQL Cluster NDB 7.1.3, MySQL Cluster is available for production use on Microsoft Windows operating systems; MySQL Cluster NDB 7.1 binaries for Windows can be obtained from http://dev.mysql.com/downloads/cluster/.

    Features and behavior are generally comparable to those found on previously supported platforms such as Linux and Solaris. However, you must install the binaries manually. In addition, there is currently no support for running MySQL Cluster processes as Windows services.

    If you wish to build MySQL Cluster from source on Windows, you must configure the build using the WITH_NDBCLUSTER_STORAGE_ENGINE option. For more information, see Section 2.3.9, “Installing MySQL from Source on Windows”.

  • --nowait-nodes option for management servers. It is now possible to configure a cluster with two management servers, but to start the cluster using only one of them by starting the management node daemon with the --nowait-nodes option. The other management server can then be started at a later time to join the running MySQL Cluster.

  • Improved lock handling for primary key lookups on BLOB tables. A MySQL Cluster table stores all but the first 256 bytes of any BLOB or TEXT column values in a separate BLOB table; when executing queries against such tables, a shared lock is obtained. Prior to MySQL Cluster NDB 7.1.1, when the query used a primary key lookup and took place within a transaction, the lock was held for the duration of the transaction, even after no more data was being read from the NDB table. Now in such cases, the lock is released when all BLOB data associated with the table has been read. (Bug#49190)

    Note

    A shared lock is also taken for unique key lookups; it is still the case that this lock is held for the duration of the transaction.

  • Heartbeat thread policy and priority. Beginning with MySQL Cluster NDB 7.1.2, a new configuration parameter HeartbeatThreadPriority makes it possible to set the policy and the priority for the heartbeat thread on management and API nodes.

  • Improved access to partitioning information. The ndb_desc utility now provides additional information about the partitioning of data stored in MySQL Cluster. Beginning with MySQL Cluster NDB 7.1.2, the --blob-info option causes this program to include partition information for BLOB tables in its output. Also beginning with MySQL Cluster NDB 7.1.2, the --extra-node-info option causes ndb_desc to include information about data distribution (that is, which table fragments are stored on which data nodes). Each of these options also requires the use of the --extra-partition-info option.

    Information about partition-to-node mappings can also be obtained using the Table::getFragmentNodes() method, also added in MySQL Cluster NDB 7.1.2.

  • Replication attribute promotion and demotion. Beginning with MySQL Cluster NDB 7.1.3, MySQL Cluster Replication supports attribute promotion and demotion when replicating between columns of different but similar types on the master and the slave. For example, it is possible to promote an INT column on the master to a BIGINT column on the slave, and to demote a TEXT column to a VARCHAR column.

    The implementation of type demotion distinguishes between lossy and non-lossy type conversions, and their use on the slave can be controlled by setting the slave_type_conversions global server system variable.

    For more information, see Attribute promotion and demotion (MySQL Cluster).

  • Change in ndbinfo database. The experimental ndbinfo.pools table was removed from ndbinfo in MySQL Cluster NDB 7.1.3. Applications which used this table can and should be rewritten to use other ndbinfo tables. For more information, see Section 17.5.8.8, “The ndbinfo pools Table”.

  • Configuration caching control. Beginning with MySQL Cluster NDB 7.1.4, it is possible to disable the management server's configuration cache using the --config-cache option, which forces ndb_mgmd to read its configuration data from the config.ini configuration file every time it starts. For more information about configuration caching and this option, see Section 17.3.2, “MySQL Cluster Configuration Files”, as well as Section 17.4.4, “ndb_mgmd — The MySQL Cluster Management Server Daemon”.

  • Incompatible change in NDB API event reporting. Beginning with MySQL Cluster NDB 7.1.4, DDL events are no longer reported on Event objects by default. Instead such event reporting must be enabled explicitly using the Event::setReport() method. For more information, see Event::setReport(), and The Event::EventReport Type.

  • Number of table attributes. Beginning with MySQL Cluster NDB 7.1.4, the maximum number of attributes (columns plus indexes) per table has been increased from 128 to 512.

  • InnoDB support in commercial binaries. Beginning with MySQL Cluster NDB 7.1.4b, all commercial binary releases of MySQL Cluster provide support for the InnoDB storage engine.

  • Heartbeat ordering. Beginning with MySQL Cluster NDB 7.1.5, it is possible to set a specific order for transmission of heartbeats between datat nodes, using the HeartbeatOrder data node configuration parameter introduced in this version. This parameter can be useful in situations where multiple data nodes are running on the same host and a temporary disruption in connectivity between hosts would otherwise cause the loss of a node group (and thus failure of the cluster).

  • Relaxed ndb_restore column comparison rules. When restoring data, ndb_restore compares the attributes of a column for equality with the definition of the column in the target table. However, not all of these attributes need to be the same for ndb_restore to be meaningful, safe and useful. Beginning with MySQL Cluster NDB 7.1.5, ndb_restore automatically ignores differences in certain column attributes which do not necessarily have to match between the version of the column in a backup and the version of that column in the MySQL Cluster to which the column data is being restored. These attributes include the following:

    • COLUMN_FORMAT setting (FIXED, DYNAMIC, or DEFAULT)

    • STORAGE setting (MEMORY or DISK)

    • The default value

    • The distribution key

    In such cases, ndb_restore reports any such differences to minimize any chance of user error.

  • Storage of user data in anyValue. When writing NDB events to the binary log, MySQL Cluster uses OperationOptions::anyValue to store the server ID. Beginning with MySQL Cluster NDB 7.1.6, it is possible to store user data from an NDB API application in part of the anyValue when mysqld has been started with the --server-id-bits option set to a non-default value. Also beginning with MySQL Cluster NDB 7.1.6, it is possible to view this data in the output of mysqlbinlog, for which its own --server-id-bits option is added.

  • --add-drop-trigger option for mysqldump. Beginning with MySQL Cluster NDB 7.1.8, this option can be used to force all CREATE TRIGGER statements in mysqldump output to be preceded by a DROP TRIGGER IF EXISTS statement.

  • Forcing node shutdown and restart. In MySQL Cluster NDB 7.1.8 and later, it is possible using the ndb_mgm management client or the MGM API to force a data node shutdown or restart even if this would force the shutdown or restart of the entire cluster. In the management client, this is implemented through the addition of the -f (force) option to the STOP and RESTART commands. For more information, see Section 17.5.2, “Commands in the MySQL Cluster Management Client”. The MGM API also adds two new methods for forcing such a node shutdown or restart; see ndb_mgm_stop4(), and ndb_mgm_restart4(), for more information about these methods.

17.1.4.2. MySQL Cluster Development in MySQL Cluster NDB 7.0

The following list provides an overview of significant feature additions and changes made in MySQL Cluster NDB 7.0. For more detailed information about all feature changes and bugfixes made in MySQL Cluster NDB 7.0, see Section 17.7.2, “Changes in MySQL Cluster NDB 7.0”.

Important

Early development versions of MySQL Cluster NDB 7.0 were known as “MySQL Cluster NDB 6.4”, and the first four releases in this series were identified as MySQL Cluster NDB 6.4.0 through 6.4.3. Any information relating to these MySQL Cluster NDB 6.4.x releases appearing in this documentation apply to MySQL Cluster NDB 7.0.

MySQL Cluster NDB 7.0.4 is the fifth MySQL Cluster NDB 7.0 release; it is the successor to MySQL Cluster NDB 6.4.3.

  • MySQL Cluster on Windows (alpha). MySQL Cluster NDB 7.0 is available on an experimental basis for Windows operating systems (for production use on Windows, you should use MySQL Cluster NDB 7.1.3 or later). Features and behavior comparable to those found on platforms that are already supported—such as Linux and Solaris—are planned for MySQL Cluster on Windows. In MySQL Cluster NDB 7.0, you must build from source (Windows binaries are available for MySQL Cluster NDB 7.1 releases). To enable MySQL Cluster support on Windows when building from source, you must configure the build using the WITH_NDBCLUSTER_STORAGE_ENGINE option. For more information, see Section 2.3.9, “Installing MySQL from Source on Windows”.

  • Ability to add nodes and node groups online. Beginning with MySQL Cluster NDB 6.4.0, it is possible to add new node groups (and thus new data nodes) to a running MySQL Cluster without shutting down and reloading the cluster. As part of enabling this feature, a new command CREATE NODEGROUP has been added to the cluster management client and the functionality of the ALTER ONLINE TABLE ... REORGANIZE PARTITION SQL statement has been extended. For more information, see Section 17.5.11, “Adding MySQL Cluster Data Nodes Online”.

  • Data node multithreading support. Beginning with MySQL Cluster NDB 6.4.0, a multithreaded version of the data node daemon, named ndbmtd, is available for use on data node hosts with multiple CPU cores. This binary is built automatically when compiling with MySQL Cluster support; no additional options other than those needed to provide MySQL Cluster support are needed when configuring the build. In most respects, ndbmtd functions in the same way as ndbd, and can use the same command-line options and configuration parameters. In addition, the new MaxNoOfExecutionThreads configuration parameter can be used to determine the number of data node process threads for ndbmtd. For more information, see Section 17.4.3, “ndbmtd — The MySQL Cluster Data Node Daemon (Multi-Threaded)”.

    Note

    Disk Data tables are not yet supported for use with ndbmtd.

  • Configuration cache. Formerly, MySQL Cluster configuration was stateless—that is, configuration information was reloaded from the cluster's global configuration file (usually config.ini) each time ndb_mgmd was started. Beginning with MySQL Cluster NDB 6.4.0, the cluster's configuration is cached internally, and the global configuration file is no longer automatically re-read when the management server is restarted. This behavior can be controlled using the management server options --configdir, --initial, and --reload. In MySQL Cluster NDB 7.0.15 and later, the configuration cache can be disabled using the --config-cache option. For more information about these changes, see Section 17.3.2, “MySQL Cluster Configuration Files”. For more information about the new management server options, see Section 17.4.4, “ndb_mgmd — The MySQL Cluster Management Server Daemon”.

  • Detection of NDB API client connection errors. In MySQL Cluster NDB 7.0 (6.4.0 and later releases), the NDB API's Ndb_cluster_connection class provides additional methods for catching and diagnosing problems with NDB API client connections. For more information, see Ndb_cluster_connection::get_latest_error(), and Ndb_cluster_connection::get_latest_error_msg().

  • Snapshot options for backups. Beginning with MySQL Cluster NDB 6.4.0, you can determine when performing a cluster backup whether the backup matches the state of the data when the backup was started or when it was completed, using the new options SNAPSHOTSTART and SNAPSHOTEND for the management client's START BACKUP command. See Section 17.5.3.2, “Using The MySQL Cluster Management Client to Create a Backup”, for more information.

  • Dynamic NDB transporter send buffer memory allocation. Previously, the NDB kernel used a fixed-size send buffer for every data node in the cluster, which was allocated when the node started. Because the size of this buffer could not be changed after the cluster was started, it was necessary to make it large enough in advance to accomodate the maximum possible load on any transporter socket. However, this was an inefficient use of memory, since much of it often went unused. Beginning with MySQL Cluster NDB 6.4.0, send buffer memory is allocated dynamically from a memory pool shared between all transporters, which means that the size of the send buffer can be adjusted as necessary. This change is reflected by the addition of the configuration parameters TotalSendBufferMemory, ReservedSendBufferMemory, and OverLoadLimit, as well as a change in how the existing SendBufferMemory configuration parameter is used. For more information, see Section 17.3.2.13, “Configuring MySQL Cluster Send Buffer Parameters”.

  • Robust DDL operations. Beginning with MySQL Cluster NDB 6.4.0, DDL operations (such as CREATE TABLE or ALTER TABLE) are protected from data node failures; in the event of a data node failure, such operations are now rolled back gracefully. Previously, if a data node failed while trying to perform a DDL operation, the MySQL Cluster data dictionary became locked and no further DDL statements could be executed without restarting the cluster.

  • IPv6 support in MySQL Cluster Replication. Beginning with MySQL Cluster NDB 6.4.1, IPv6 networking is supported between MySQL Cluster SQL nodes, which makes it possible to replicate between instances of MySQL Cluster using IPv6 addresses. However, IPv6 is supported only for direct connections between MySQL servers; all connections within an individual MySQL Cluster must use IPv4. For more information, see Section 17.6.3, “Known Issues in MySQL Cluster Replication”.

  • Restoring specific databases, tables, or columns from a MySQL Cluster backup. It is now possible to exercise more fine-grained control when restoring a MySQL Cluster from backup using ndb_restore. Beginning with MySQL Cluster NDB 6.4.3, you can choose to restore only specified tables or databases, or exclude specific tables or databases from being restored, using the new ndb_restore options --include-tables, --include-databases, --exclude-tables, and --exclude-databases. Beginning with MySQL Cluster NDB 7.0.7, it is also possible to restore to a table having fewer columns than the original using the --exclude-missing-columns option. For more information about all of these options, see Section 17.4.17, “ndb_restore — Restore a MySQL Cluster Backup”.

  • Improved Disk Data filesystem configuration. As of MySQL Cluster NDB 6.4.3, you can specify default locations for MySQL Cluster Disk Data data files and undo log files using the data node configuration parameters FileSystemPathDD, FileSystemPathDataFiles, and FileSystemPathUndoFiles. This eliminates the need to use symbolic links to place Disk Data files separately from other files in data node filesystems to improve Disk Data performance. For more information, see Disk Data filesystem parameters.

  • Automatic creation of Disk Data log file groups and tablespaces. Beginning with MySQL Cluster NDB 6.4.3, using the data node configuration parameters InitialLogFileGroup and InitialTablespace, you can cause the creation of a MySQL Cluster Disk Data log file group, tablespace, or both, when the cluster is first started. When using these parameters, no SQL statements are required to create these Disk Data objects. For more information, see Disk Data object creation parameters.

  • Improved internal message passing and record handling. MySQL Cluster NDB 7.0 contains 2 changes that optimize the use of network connections by addressing the size and number of messages passed between data nodes, and between data nodes and API nodes, which can increase MySQL Cluster and application performance:

    • Packed reads. Formerly, each read request signal contained a list of columns to be retrieved, each of these column identifiers using 4 bytes within the message. This meant that the message size increased as the number of columns being fetched increased. In addition, in the response from the data node, each column result was packed to a 4-byte boundary, which resulted in wasted space. In MySQL Cluster NDB 7.0, messaging for read operations is optimized in both directions, using a bitmap in the read request to specify the columns to be fetched. Where many fields are requested, this can result in a significant message size reduction as compared with the old method. In addition, the 4-byte packing in responses is no longer used, which means that smaller fields consume less space.

    • Long signal transactions. This enhancement reduces the number of messages and signals that are sent to data nodes for complex requests. Prior to MySQL Cluster NDB 7.0, there was a 100 byte limit on the size of the request signal, which meant that complex requests had to be split up between multiple messages prior to transmission, then reassembled on the receiving end. In addition to actual payload data, each message required its own operating system and protocol overhead such as header information. This often wasted network bandwidth and data node CPU. The maximum size of the message is now 32 KB, which is sufficient to accommodate most queries.

    Both of these optimizations are internal to the NDB API, and so is transparent to applications; this is true whether an application uses the NDB API directly or does so indirectly through an SQL node.

  • Configuration parameter data dumps. Starting with MySQL Cluster NDB 7.0.6, the ndb_config utility supports a --configinfo option that causes it to dump a list of all configuration parameters supported by the cluster, along with brief descriptions, information about the parameters' default and permitted values, and the sections of the config.ini file in which the parameters apply. An additional --xml switch causes ndb_config to use XML rather than plaintext output. Using ndb_config --configinfo or ndb_config --configinfo --xml requires no access to a running MySQL Cluster, any other programs, or any files. For more information and examples, see Section 17.4.6, “ndb_config — Extract MySQL Cluster Configuration Information”.

  • Per-table reporting of free space on disk. The INFORMATION_SCHEMA.FILES table shows information about used and free space in MySQL Cluster Disk Data data files, but this information is not applicable to individual tables. In MySQL Cluster NDB 7.0.8 and later, the ndb_desc utility provides two additional columns in its output that show the amount of space allocated on disk for a given NDB table as well the amount of space that remains available for additional storage of disk-based column data for that table. For more information, see Section 17.4.9, “ndb_desc — Describe NDB Tables”.

  • Improved restart times. Optimizations in redo log handling and other filesystem operations introduced in MySQL Cluster NDB 7.0.9 have the potential to reduce considerably the time required for restarts. While actual performance benefits observed in production setups will naturally vary depending on database size, hardware, and other conditions, our own preliminary testing has shown that these improvements can yield startup times that are faster than those typical of previous MySQL Cluster NDB 7.0 releases by a factor of 50 or more.

  • Native support for default column values. Starting with MySQL Cluster NDB 7.0.15, default values for table columns are stored in the NDB kernel rather than by the MySQL server as was done previously. This means that inserts on tables having column value defaults can be smaller and faster than before, because less data must be sent from SQL nodes to NDBCLUSTER.

    Tables created using previous MySQL Cluster releases can still be used in MySQL Cluster 7.0.15 and later; however, they do not support native default values until they are upgraded. You can upgrade a table with non-native default values to support native default values using an offline ALTER TABLE statement.

  • --nowait-nodes option for management servers. Starting with MySQL Cluster NDB 7.0.10, it is possible to configure a cluster with two management servers, but to start the cluster using only one of them by starting the management node daemon with the --nowait-nodes option. The other management server can then be started at a later time to join the running MySQL Cluster.

  • Increased flexibility in online upgrade procedure. Previously, when performing an upgrade of a running MySQL cluster, the order in which the types of cluster nodes had to be upgraded was very strict. However, beginning with MySQL Cluster NDB 7.0.10, MySQL Cluster supports online upgrading of API nodes (including MySQL servers running as SQL nodes) online upgrading management nodes, data nodes, or both.

    Important

    Before attempting to use this new upgrade functionality, see Section 17.2.5.1, “Performing a Rolling Restart of a MySQL Cluster”, for additional information, especially if you are planning an online upgrade to MySQL Cluster NDB 7.0 from MySQL Cluster NDB 6.3.

  • New CHANGE MASTER TO option for circular replication. Beginning with MySQL Cluster NDB 7.0.11, the CHANGE MASTER TO statement supports an IGNORE_SERVER_IDS option which takes a comma-separated list of server IDs and causes events originating from the corresponding servers to be ignored. (Log rotation and log deletion events are preserved.)

    See Section 12.5.2.1, “CHANGE MASTER TO Syntax”, as well as Section 12.4.5.36, “SHOW SLAVE STATUS Syntax”, for more information.

  • New replication conflict resolution strategy. Beginning with MySQL Cluster NDB 7.0.11, the function NDB$MAX_DELETE_WIN() is available to implement “greatest timestamp, delete wins” conflict resolution. See NDB$MAX_DELETE_WIN(column_name), for more information.

  • Improved lock handling for primary key lookups on BLOB tables. A MySQL Cluster table stores all but the first 256 bytes of any BLOB or TEXT column values in a separate BLOB table; when executing queries against such tables, a shared lock is obtained. Prior to MySQL Cluster NDB 7.0.12, when the query used a primary key lookup and took place within a transaction, the lock was held for the duration of the transaction, even after no more data was being read from the NDB table. Now in such cases, the lock is released when all BLOB data associated with the table has been read. (Bug#49190)

    Note

    A shared lock is also taken for unique key lookups; it is still the case that this lock is held for the duration of the transaction.

  • Heartbeat thread policy and priority. Beginning with MySQL Cluster NDB 7.0.13, a new configuration parameter HeartbeatThreadPriority makes it possible to set the policy and the priority for the heartbeat thread on management and API nodes.

  • Improved access to partitioning information. The ndb_desc utility now provides additional information about the partitioning of data stored in MySQL Cluster. Beginning with MySQL Cluster NDB 7.0.13, the --blob-info option causes this program to include partition information for BLOB tables in its output. Beginning with MySQL Cluster NDB 7.0.14, the --extra-node-info option causes ndb_desc to include information about data distribution (that is, which table fragments are stored on which data nodes). Each of these options also requires the use of the --extra-partition-info option.

    Information about partition-to-node mappings can also be obtained using the Table::getFragmentNodes() method, also added in MySQL Cluster NDB 7.0.14.

  • Replication attribute promotion and demotion. Beginning with MySQL Cluster NDB 7.0.14, MySQL Cluster Replication supports attribute promotion and demotion when replicating between columns of different but similar types on the master and the slave. For example, it is possible to promote an INT column on the master to a BIGINT column on the slave, and to demote a TEXT column to a VARCHAR column.

    The implementation of type demotion distinguishes between lossy and non-lossy type conversions, and their use on the slave can be controlled by setting the slave_type_conversions global server system variable.

    For more information, see Attribute promotion and demotion (MySQL Cluster).

  • Incompatible change in NDB API event reporting. Beginning with MySQL Cluster NDB 7.0.15, DDL events are no longer reported on Event objects by default. Instead such event reporting must be enabled explicitly using the Event::setReport() method. For more information, see Event::setReport(), and The Event::EventReport Type.

  • Number of table attributes. Beginning with MySQL Cluster NDB 7.0.15, the maximum number of attributes (columns plus indexes) per table has been increased from 128 to 512.

  • Heartbeat ordering. Beginning with MySQL Cluster NDB 7.0.16, it is possible to set a specific order for transmission of heartbeats between data nodes, using the HeartbeatOrder data node configuration parameter introduced in this version. This parameter can be useful in situations where multiple data nodes are running on the same host and a temporary disruption in connectivity between hosts would otherwise cause the loss of a node group (and thus failure of the cluster).

  • Relaxed ndb_restore column comparison rules. When restoring data, ndb_restore compares the attributes of a column for equality with the definition of the column in the target table. However, not all of these attributes need to be the same for ndb_restore to be meaningful, safe and useful. Beginning with MySQL Cluster NDB 7.0.16, ndb_restore automatically ignores differences in certain column attributes which do not necessarily have to match between the version of the column in a backup and the version of that column in the MySQL Cluster to which the column data is being restored. These attributes include the following:

    • COLUMN_FORMAT setting (FIXED, DYNAMIC, or DEFAULT)

    • STORAGE setting (MEMORY or DISK)

    • The default value

    • The distribution key

    In such cases, ndb_restore reports any such differences to minimize any chance of user error.

  • Storage of user data in anyValue. When writing NDB events to the binary log, MySQL Cluster uses OperationOptions::anyValue to store the server ID. Beginning with MySQL Cluster NDB 7.0.17, it is possible to store user data from an NDB API application in part of the anyValue when mysqld has been started with the --server-id-bits option set to a non-default value. Also beginning with MySQL Cluster NDB 7.0.17, it is possible to view this data in the output of mysqlbinlog, for which its own --server-id-bits option is added.

  • --add-drop-trigger option for mysqldump. Beginning with MySQL Cluster NDB 7.0.19, this option can be used to force all CREATE TRIGGER statements in mysqldump output to be preceded by a DROP TRIGGER IF EXISTS statement.

  • Forcing node shutdown and restart. In MySQL Cluster NDB 7.0.19 and later, it is possible using the ndb_mgm management client or the MGM API to force a data node shutdown or restart even if this would force the shutdown or restart of the entire cluster. In the management client, this is implemented through the addition of the -f (force) option to the STOP and RESTART commands. For more information, see Section 17.5.2, “Commands in the MySQL Cluster Management Client”. The MGM API also adds two new methods for forcing such a node shutdown or restart; see ndb_mgm_stop4(), and ndb_mgm_restart4(), for more information about these methods.

17.1.4.3. MySQL Cluster Development in MySQL Cluster NDB 6.3

The following list provides an overview of significant feature additions and changes first made in MySQL Cluster NDB 6.3. For more detailed information about all feature changes and bugfixes made in MySQL Cluster NDB 6.3, see Section 17.7.3, “Changes in MySQL Cluster NDB 6.3”.

  • Conflict detection and resolution. It is now possible to detect and resolve conflicts that arise in multi-master replication scenarios, such as circular replication, when different masters may try to update the same row on the slave with different data. Both “greatest timestamp wins” and “same timestamp wins” scenarios are supported. For more information, see Section 17.6.11, “MySQL Cluster Replication Conflict Resolution”.

  • Recovery of “one master, many slaves” replication setups. Recovery of multi-way replication setups (“one master, many slaves”) is now supported using the --ndb-log-orig server option and changes in the mysql.ndb_binlog_index table. See Section 17.6.4, “MySQL Cluster Replication Schema and Tables”, for more information.

  • Enhanced selection options for transaction coordinator. New values and behaviors are introduced for --ndb_optimized_node_selection, enabling greater flexibility when an SQL node chooses a transaction coordinator. For more information, see the description of ndb_optimized_node_selection in Section 17.3.4.3, “MySQL Cluster System Variables”.

  • Replication heartbeats. Replication heartbeats facilitate the task of monitoring and detecting failures in master-slave connections in real time. This feature is implemented using a new MASTER_HEARTBEAT_PERIOD = value clause for the CHANGE MASTER TO statement and the addition of two status variables Slave_heartbeat_period and Slave_received_heartbeats. For more information, see Section 12.5.2.1, “CHANGE MASTER TO Syntax”.

  • NDB thread locks. It is possible to lock NDB execution threads and maintenance threads (such as file system and other operating system threads) to specific CPUs on multiprocessor data node hosts, and to leverage real-time scheduling.

  • Improved performance of updates using primary keys or unique keys. The number of unnecessary reads when performing a primary key or unique key update has been greatly reduced. Since it is seldom necessary to read a record prior to an update, this can yield a considerable improvement in performance. In addition, primary key columns are no longer written to when not needed during update operations.

  • Batching improvements. Support of batched DELETE and UPDATE operations has been significantly improved. Batching of UPDATE WHERE... and multiple DELETE operations is also now implemented.

  • Improved SQL statement performance metrics. The Ndb_execute_count system status variable measures the number of round trips made by SQL statements to the NDB kernel, providing an improved metric for determining efficiency with which statements are excuted. For more information, see MySQL Cluster Status Variables: Ndb_execute_count.

  • Compressed LCPs and backups. Compressed local checkpoints and backups can save 50% or more of the disk space used by uncompressed LCPs and backups. These can be enabled using the two new data node configuration parameters CompressedLCP and CompressedBackup, respectively.

  • OPTIMIZE TABLE support with NDBCLUSTER tables. OPTIMIZE TABLE is supported for dynamic columns of in-memory NDB tables. In such cases, it is no longer necessary to drop (and possibly to re-create) a table, or to perform a rolling restart, in order to recover memory from deleted rows for general re-use by Cluster. The performance of OPTIMIZE on Cluster tables can be tuned by adjusting the value of the ndb_optimization_delay system variable, which controls the number of milliseconds to wait between processing batches of rows by OPTIMIZE TABLE. In addition, OPTIMIZE TABLE on an NDBCLUSTER table can be interrupted by, for example, killing the SQL thread performing the OPTIMIZE operation.

  • Batching of transactions. It is possible to cause statements occurring within the same transaction to be run as a batch by setting the session variable transaction_allow_batching to 1 or ON. To use this feature, autocommit must be set to 0 or OFF. Batch sizes can be controlled using the --ndb-batch-size option for mysqld. For additional information, see Section 17.3.4.2, “mysqld Command Options for MySQL Cluster”, and Section 17.3.4.3, “MySQL Cluster System Variables”.

  • Attribute promotion with ndb_restore. It is possible using ndb_restore to restore data reliably from a column of a given type to a column that uses a “larger” type. This is sometimes referred to as attribute promotion. For example, MySQL Cluster backup data that originated in a SMALLINT column can be restored to a MEDIUMINT, INT, or BIGINT column. See Section 17.4.17, “ndb_restore — Restore a MySQL Cluster Backup”, for more information.

  • Parallel data node recovery. Recovery of multiple data nodes can now be done in parallel, rather than sequentially. In other words, several data nodes can be restored concurrently, which can often result in much faster recovery times than when they are restored one at a time.

  • Increased local checkpoint efficiency. Only 2 local checkpoints are stored, rather than 3, lowering disk space requirements and the size and number of redo log files.

  • NDBCLUSTER table persistence control. Persistence of NDB tables can be controlled using the session variables ndb_table_temporary and ndb_table_no_logging. ndb_table_no_logging causes NDB tables not to be checkpointed to disk; ndb_table_temporary does the same, and in addition, no schema files are created. See Section 17.3.4.1, “MySQL Cluster Server Option and Variable Reference”.

  • Epoll support (Linux only). Epoll is an improved method for handling file descriptors, which is more efficient than scanning to determine whether a file descriptor has data to be read. (The term epoll is specific to Linux and equivalent functionality is known by other names on other platforms such as Solaris and FreeBSD.) Currently, MySQL Cluster supports this functionality on Linux only.

  • Distribution awareness (SQL nodes). In MySQL Cluster NDB 6.3, SQL nodes can take advantage of distribution awareness. Here we provide a brief example showing how to design a table to make a given class of queries distrubtion-aware. Suppose an NDBCLUSTER table t1 has the following schema:

    CREATE TABLE t1 (
        userid INT NOT NULL,
        serviceid INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
        data VARCHAR(255)
    )   ENGINE=NDBCLUSTER;
    

    Suppose further that most of the queries to be used in our application test values of the userid column of this table. The form of such a query looks something like this:

    SELECT columns FROM t1
        WHERE userid relation value;
    

    In this query, relation represents some relational operator, such as =, <, >, and so on. Queries using IN and a list of values can also be used:

    SELECT columns FROM t1
        WHERE userid IN value_list;
    

    To make use of distribution awareness, we need to make the userid column part of the table's primary key, then explicitly partition the table with this column being used as the partitioning key. (Recall that for a partitioned table having one or more unique keys, all columns of the table's partitioning key must also be part of all of the unique keys—for more information and examples, see Section 18.5.1, “Partitioning Keys, Primary Keys, and Unique Keys”.) In other words, the table schema should be equivalent to the following CREATE TABLE statement:

    CREATE TABLE t1 (
        userid INT NOT NULL,
        serviceid INT NOT NULL AUTO_INCREMENT,
        data VARCHAR(255),
        PRIMARY KEY p (userid,serviceid)
    )   ENGINE=NDBCLUSTER
        PARTITION BY KEY(userid);
    

    When the table is partitioned in this way, all rows having the same userid value are found on the same node group, and the MySQL Server can immediately select the optimal node to use as the transaction coordinator.

  • Realtime extensions for multiple CPUs. When running MySQL Cluster data nodes on hosts with multiple processors, the realtime extensions make it possible to give priority to the data node process and control on which CPU cores it should operate. This can be done using the data node configuration parameters RealtimeScheduler, SchedulerExecutionTimer, and SchedulerSpinTimer. Doing so properly can significantly lower response times and make them much more predictable response. For more information about using these parameters, see Defining Data Nodes: Realtime Performance Parameters

  • Fully automatic database discovery. It is no longer a requirement for database autodiscovery that an SQL node already be connected to the cluster at the time that a database is created on another SQL node, or for a CREATE DATABASE or CREATE SCHEMA statement to be issued on the new SQL node after it joins the cluster.

  • Detection of NDB API client connection errors. Beginning with MySQL Cluster NDB 6.3.20, the NDB API's Ndb_cluster_connection class provides additional methods for catching and diagnosing problems with NDB API client connections. For more information, see Ndb_cluster_connection::get_latest_error(), and Ndb_cluster_connection::get_latest_error_msg().

  • Restoring specific databases, tables, or columns from a MySQL Cluster backup. It is now possible to exercise more fine-grained control when restoring a MySQL Cluster from backup using ndb_restore. Beginning with MySQL Cluster NDB 6.3.22, you can choose to restore only specified tables or databases, or exclude specific tables or databases from being restored, using the new ndb_restore options --include-tables, --include-databases, --exclude-tables, and --exclude-databases. Beginning with MySQL Cluster NDB 6.3.26, it is also possible to restore to a table having fewer columns than the original using the --exclude-missing-columns option. For more information about all of these options, see Section 17.4.17, “ndb_restore — Restore a MySQL Cluster Backup”.

  • Improved Disk Data filesystem configuration. As of MySQL Cluster NDB 6.3.22, you can specify default locations for MySQL Cluster Disk Data data files and undo log files using the data node configuration parameters FileSystemPathDD, FileSystemPathDataFiles, and FileSystemPathUndoFiles. This eliminates the need to use symbolic links to place Disk Data files separately from other files in data node filesystems to improve Disk Data performance. For more information, see Disk Data filesystem parameters.

  • Automatic creation of Disk Data log file groups and tablespaces. Beginning with MySQL Cluster NDB 6.3.22, using the data node configuration parameters InitialLogFileGroup and InitialTablespace, you can cause the creation of a MySQL Cluster Disk Data log file group, tablespace, or both, when the cluster is first started. When using these parameters, no SQL statements are required to create these Disk Data objects. For more information, see Disk Data object creation parameters.

  • Configuration parameter data dumps. Starting with MySQL Cluster NDB 6.3.25, the ndb_config utility supports a --configinfo option that causes it to dump a list of all configuration parameters supported by the cluster, along with brief descriptions, information about the parameters' default and permitted values, and the sections of the config.ini file in which the parameters apply. An additional --xml switch causes ndb_config to use XML rather than plaintext output. Using ndb_config --configinfo or ndb_config --configinfo --xml requires no access to a running MySQL Cluster, any other programs, or any files. For more information and examples, see Section 17.4.6, “ndb_config — Extract MySQL Cluster Configuration Information”.

  • Per-table reporting of free space on disk. The INFORMATION_SCHEMA.FILES table shows information about used and free space in MySQL Cluster Disk Data data files, but this information is not applicable to individual tables. In MySQL Cluster NDB 6.3.27 and later, the ndb_desc utility provides two additional columns in its output that show the amount of space allocated on disk for a given NDB table as well the amount of space that remains available for additional storage of disk-based column data for that table. For more information, see Section 17.4.9, “ndb_desc — Describe NDB Tables”.

  • Improved restart times. Optimizations in redo log handling and other filesystem operations introduced in MySQL Cluster NDB 6.3.28 have the potential to reduce considerably the time required for restarts. While actual performance benefits observed in production setups will naturally vary depending on database size, hardware, and other conditions, our own preliminary testing has shown that these improvements can yield startup times that are faster than those typical of previous MySQL Cluster NDB 6.3 releases by a factor of 50 or more.

  • Increased flexibility in online upgrade procedure. Previously, when performing an upgrade of a running MySQL cluster, the order in which the types of cluster nodes had to be upgraded was very strict. However, beginning with MySQL Cluster NDB 6.3.29, MySQL Cluster supports online upgrading of API nodes (including MySQL servers running as SQL nodes) before upgrading management nodes, data nodes, or both.

    Important

    Before attempting to use this new upgrade functionality, see Section 17.2.5.1, “Performing a Rolling Restart of a MySQL Cluster”, for additional information, especially if you are planning an online upgrade from MySQL Cluster NDB 6.3 to MySQL Cluster NDB 7.0.

  • New replication conflict resolution strategy. Beginning with MySQL Cluster NDB 6.3.31, the function NDB$MAX_DELETE_WIN() is available to implement “greatest timestamp, delete wins” conflict resolution. See NDB$MAX_DELETE_WIN(column_name), for more information.

  • New CHANGE MASTER TO option for circular replication. Beginning with MySQL Cluster NDB 6.3.31, the CHANGE MASTER TO statement supports an IGNORE_SERVER_IDS option which takes a comma-separated list of server IDs and causes events originating from the corresponding servers to be ignored. (Log rotation and log deletion events are preserved.)

    See Section 12.5.2.1, “CHANGE MASTER TO Syntax”, as well as Section 12.4.5.36, “SHOW SLAVE STATUS Syntax”, for more information.

  • Heartbeat thread policy and priority. Beginning with MySQL Cluster NDB 6.3.32, a new configuration parameter HeartbeatThreadPriority makes it possible to set the policy and the priority for the heartbeat thread on management and API nodes.

  • Improved access to partitioning information. The ndb_desc utility now provides additional information about the partitioning of data stored in MySQL Cluster. Beginning with MySQL Cluster NDB 6.3.32, the --blob-info option causes this program to include partition information for BLOB tables in its output. Beginning with MySQL Cluster NDB 6.3.33, the --extra-node-info option causes ndb_desc to include information about data distribution (that is, which table fragments are stored on which data nodes). Each of these options also requires the use of the --extra-partition-info option.

    Information about partition-to-node mappings can also be obtained using the Table::getFragmentNodes() method, also added in MySQL Cluster NDB 6.3.33.

  • Replication attribute promotion and demotion. Beginning with MySQL Cluster NDB 6.3.33, MySQL Cluster Replication supports attribute promotion and demotion when replicating between columns of different but similar types on the master and the slave. For example, it is possible to promote an INT column on the master to a BIGINT column on the slave, and to demote a TEXT column to a VARCHAR column.

    The implementation of type demotion distinguishes between lossy and non-lossy type conversions, and their use on the slave can be controlled by setting the slave_type_conversions global server system variable.

    For more information, see Attribute promotion and demotion (MySQL Cluster).

  • Incompatible change in NDB API event reporting. Beginning with MySQL Cluster NDB 6.3.34, DDL events are no longer reported on Event objects by default. Instead such event reporting must be enabled explicitly using the Event::setReport() method. For more information, see Event::setReport(), and The Event::EventReport Type.

  • Heartbeat ordering. Beginning with MySQL Cluster NDB 6.3.35, it is possible to set a specific order for transmission of heartbeats between datat nodes, using the HeartbeatOrder data node configuration parameter introduced in this version. This parameter can be useful in situations where multiple data nodes are running on the same host and a temporary disruption in connectivity between hosts would otherwise cause the loss of a node group (and thus failure of the cluster).

  • Relaxed ndb_restore column comparison rules. When restoring data, ndb_restore compares the attributes of a column for equality with the definition of the column in the target table. However, not all of these attributes need to be the same for ndb_restore to be meaningful, safe and useful. Beginning with MySQL Cluster NDB 6.3.35, ndb_restore automatically ignores differences in certain column attributes which do not necessarily have to match between the version of the column in a backup and the version of that column in the MySQL Cluster to which the column data is being restored. These attributes include the following:

    • COLUMN_FORMAT setting (FIXED, DYNAMIC, or DEFAULT)

    • STORAGE setting (MEMORY or DISK)

    • The default value

    • The distribution key

    In such cases, ndb_restore reports any such differences to minimize any chance of user error.

  • --add-drop-trigger option for mysqldump. Beginning with MySQL Cluster NDB 6.3.38, this option can be used to force all CREATE TRIGGER statements in mysqldump output to be preceded by a DROP TRIGGER IF EXISTS statement.

17.1.4.4. MySQL Cluster Development in MySQL Cluster NDB 6.2

The following list provides an overview of significant feature additions and changes made in MySQL Cluster NDB 6.2. All of the changes in this list are also available in MySQL Cluster NDB 6.3 . For more detailed information about all feature changes and bugfixes made in MySQL Cluster NDB 6.2, see Section 17.7.4, “Changes in MySQL Cluster NDB 6.2”.

  • Enhanced backup status reporting. Backup status reporting has been improved, aided in part by the introduction of a BackupReportFrequency configuration parameter.

  • Multiple cluster connections per SQL node. A single MySQL server acting as a MySQL Cluster SQL node can employ multiple connections to the cluster using the --ndb-cluster-connection-pool startup option for mysqld. This option is described in MySQL Cluster-Related Command Options for mysqld: --ndb-cluster-connection-pool option.

  • New data access interface. The NdbRecord interface provides a new and simplified data handler for use in NDB API applications. See The NdbRecord Interface, for more information.

  • New reporting commands. The new management client REPORT BackupStatus and REPORT MemoryUsage commands provide better access to information about the status of MySQL Cluster backups and how much memory is being used by MySQL Cluster for data and index storage. See Section 17.5.2, “Commands in the MySQL Cluster Management Client”, for more information about the REPORT commands. In addition, in-progress status reporting is provided by the ndb_restore utility; see Section 17.4.17, “ndb_restore — Restore a MySQL Cluster Backup”.

  • Improved memory allocation and configuration. Memory is now allocated by the NDB kernel to tables on a page-by-page basis, which significantly reduces the memory overhead required for maintaining NDBCLUSTER tables. In addition, the MaxAllocate configuration parameter now makes it possible to set the maximum size of the allocation unit used for table memory.

  • Choice of fixed-width or variable-width columns. You can control whether fixed-width or variable-width storage is used for a given column of an NDB table by employing of the COLUMN_FORMAT specifier as part of the column's definition in a CREATE TABLE or ALTER TABLE statement. In addition, the ability to control whether a given column of an NDB table is stored in memory or on disk, using the STORAGE specifier as part of the column's definition in a CREATE TABLE or ALTER TABLE statement. For more information, see Section 12.1.17, “CREATE TABLE Syntax”, and Section 12.1.7, “ALTER TABLE Syntax”.

  • Controlling management client connections. The --bind-address cluster management server startup option makes it possible to restrict management client connections to ndb_mgmd to a single host (IP address or host name) and port, which can make MySQL Cluster management operations more secure. For more information about this option, see Section 17.4.4, “ndb_mgmd — The MySQL Cluster Management Server Daemon”.

  • Micro-GCPs. Due to a change in the protocol for handling of global checkpoints (GCPs handled in this manner sometimes being referred to as “micro-GCPs”), it is now possible to control how often the GCI number is updated, and how often global checkpoints are written to disk, using the TimeBetweenEpochs and TimeBetweenEpochsTimeout configuration parameters. This improves the reliability and performance of MySQL Cluster Replication.

  • Core online schema change support. Support for the online ALTER TABLE operations ADD COLUMN, ADD INDEX, and DROP INDEX is available. When the ONLINE keyword is used, the ALTER TABLE is noncopying, which means that indexes do not have to be re-created, which provides these benefits:

    • Single user mode is no longer required for ALTER TABLE operations that can be performed online.

    • Transactions can continue during ALTER TABLE operations that can be performed online.

    • Tables being altered online are not locked against access by other SQL nodes.

      However, such tables are locked against other operations on the same SQL node for the duration of the ALTER TABLE. We are working to overcome this limitation in a future MySQL Cluster release.

    Online CREATE INDEX and DROP INDEX statements are also supported. Online changes can be suppressed using the OFFLINE key word. See Section 12.1.7, “ALTER TABLE Syntax”, Section 12.1.13, “CREATE INDEX Syntax”, and Section 12.1.24, “DROP INDEX Syntax”, for more detailed information.

  • mysql.ndb_binlog_index improvements. More information has been added to the mysql.ndb_binlog_index table so that it is possible to determine which originating epochs have been applied inside an epoch. This is particularly useful for 3-way replication. See Section 17.6.4, “MySQL Cluster Replication Schema and Tables”, for more information.

  • Epoch lag control. The MaxBufferedEpochs data node configuration parameter provides a means to control the maximum number of unprocessed epochs by which a subscribing node can lag. Subscribers which exceed this number are disconnected and forced to reconnect.

  • Fully automatic database discovery. It is no longer a requirement for database autodiscovery that an SQL node already be connected to the cluster at the time that a database is created on another SQL node, or for a CREATE DATABASE or CREATE SCHEMA statement to be issued on the new SQL node after it joins the cluster.

  • Multiple data node processes per host. In earlier MySQL Cluster release series, we did not support MySQL Cluster deployments in production where more than one ndbd process was run on a single physical machine. However, beginning with MySQL Cluster NDB 6.2.0, you can use multiple data node processes on a single host.

    Note

    A multi-threaded version of ndbd tailored for use on hosts with multiple CPUs or cores was introduced in MySQL Cluster NDB 7.0. See Section 17.1.4.2, “MySQL Cluster Development in MySQL Cluster NDB 7.0”, and Section 17.4.3, “ndbmtd — The MySQL Cluster Data Node Daemon (Multi-Threaded)”, for more information.

  • Improved Disk Data filesystem configuration. As of MySQL Cluster NDB 6.2.17, you can specify default locations for MySQL Cluster Disk Data data files and undo log files using the data node configuration parameters FileSystemPathDD, FileSystemPathDataFiles, and FileSystemPathUndoFiles. This eliminates the need to use symbolic links to place Disk Data files separately from other files in data node filesystems to improve Disk Data performance. For more information, see Disk Data filesystem parameters.

  • Automatic creation of Disk Data log file groups and tablespaces. Beginning with MySQL Cluster NDB 6.2.17, using the data node configuration parameters InitialLogFileGroup and InitialTablespace, you can cause the creation of a MySQL Cluster Disk Data log file group, tablespace, or both, when the cluster is first started. When using these parameters, no SQL statements are required to create these Disk Data objects.

  • Improved access to partitioning information. The ndb_desc utility now provides additional information about the partitioning of data stored in MySQL Cluster. Beginning with MySQL Cluster NDB 6.2.19, the --extra-node-info option causes ndb_desc to include information about data distribution (that is, which table fragments are stored on which data nodes). This option also requires the use of the --extra-partition-info option.

    Information about partition-to-node mappings can also be obtained using the Table::getFragmentNodes() method, also added in MySQL Cluster NDB 6.2.19.

  • New CHANGE MASTER TO option for circular replication. Beginning with MySQL Cluster NDB 6.2.19, the CHANGE MASTER TO statement supports an IGNORE_SERVER_IDS option which takes a comma-separated list of server IDs and causes events originating from the corresponding servers to be ignored. (Log rotation and log deletion events are preserved.)

    See Section 12.5.2.1, “CHANGE MASTER TO Syntax”, as well as Section 12.4.5.36, “SHOW SLAVE STATUS Syntax”, for more information.

17.1.4.5. MySQL Cluster Development in MySQL Cluster NDB 6.1

The following list provides an overview of significant feature additions and changes made in MySQL Cluster NDB 6.1. All of the changes in this list are also available in MySQL Cluster NDB 6.2 and 6.3 releases. For detailed information about all changes made in MySQL Cluster NDB 6.1, see Section 17.7.5, “Changes in MySQL Cluster NDB 6.1”.

17.1.4.6. Development History of MySQL Cluster in MySQL 5.1

A number of features for MySQL Cluster were implemented in MySQL 5.1 through MySQL 5.1.23, when support for MySQL Cluster was moved to MySQL Cluster NDB. All of the features in the following list are also available in all MySQL Cluster NDB (6.1 and later) releases.

  • Integration of MySQL Cluster into MySQL Replication. MySQL Cluster Replication makes it possible to replicate from one MySQL Cluster to another. Updates on any SQL node (MySQL server) in the cluster acting as the master are replicated to the slave cluster; the state of the slave side remains consistent with the cluster acting as the master. This is sometimes referred to as asynchronous replication between clusters, providing geographic redundancy. It is also possible to replicate from a MySQL Cluster acting as the master to a standalone MySQL server acting as the slave, or from a standalone MySQL master server to to a slave cluster; in either of these cases, the standalone MySQL server uses a storage engine other than NDBCLUSTER. Multi-master replication setups such as circular replication are also supported.

    See Section 17.6, “MySQL Cluster Replication”.

  • Support for storage of rows on disk. Storage of NDBCLUSTER table data on disk is now supported. Indexed columns, including the primary key hash index, must still be stored in RAM; however, all other columns can be stored on disk.

    See Section 17.5.10, “MySQL Cluster Disk Data Tables”.

  • Variable-size columns. In MySQL 5.0, an NDBCLUSTER table column defined as VARCHAR(255) used 260 bytes of storage independent of what was stored in any particular record. In MySQL 5.1 Cluster tables, only the portion of the column actually taken up by the record is stored. This makes possible a significant reduction in space requirements for such columns as compared to previous release series—by a factor of up to 5 in many cases.

  • User-defined partitioning. Users can define partitions based on columns that are part of the primary key. It is possible to partition NDB tables based on KEY and LINEAR KEY schemes. This feature is also available for many other MySQL storage engines, which support additional partitioning types that are not available with NDBCLUSTER tables.

    For additional general information about user-defined partitioning in MySQL 5.1, see Chapter 18, Partitioning. Specifics of partitioning types are discussed in Section 18.2, “Partitioning Types”.

    The MySQL Server can also determine whether it is possible to “prune away” some of the partitions from the WHERE clause, which can greatly speed up some queries. See Section 18.4, “Partition Pruning”, for information about designing tables and queries to take advantage of partition pruning.

  • Autodiscovery of table schema changes. In MySQL 5.0, it was necessary to issue a FLUSH TABLES statement or a “dummySELECT for new NDBCLUSTER tables or changes made to schemas of existing NDBCLUSTER tables on one SQL node to be visible on the cluster's other SQL nodes. In MySQL 5.1, this is no longer necessary; new Cluster tables and changes in the definitions of existing NDBCLUSTER tables made on one SQL node are immediately visible to all SQL nodes connected to the cluster.

    Note

    When creating a new database, it is still necessary in MySQL 5.1 to issue a CREATE DATABASE or CREATE SCHEMA statement on each SQL node in the cluster.

  • Distribution awareness (NDB API). Distribution awareness is a mechanism by which the best data node is automatically selected to be queried for information. (Conceptually, it is similar in some ways to partition pruning (see Section 18.4, “Partition Pruning”). To take advantage of distribution awareness, you should do the following:

    1. Determine which table column is most likely to be used for finding matching records.

    2. Make this column part of the table's primary key.

    3. Explicitly partition the table by KEY, using this column as the table' partitioning key.

    Following these steps causes records with the same value for the partitioning column to be stored on the same partition (that is, in the same node group). When reading data, transactions are begun on the data node actually having the desired rows instead of this node being determined by the usual round-robin mechanism.

    Important

    To see a measureable impact on performance, the cluster must have at least four data nodes, since, with only two data nodes, both data nodes have exactly the same data.

    Using distribution awareness can yield performance increase of as great as 45% when using four data nodes, and possibly more when using a greater number of data nodes.

    Note

    In mainline MySQL 5.1 releases, distribution awareness was supported only when using the NDB API; support was added for SQL and API nodes in MySQL Cluster NDB 6.3 (see Section 17.1.4.3, “MySQL Cluster Development in MySQL Cluster NDB 6.3”, which includes an example showing how to create a table to take advantage of distribution awareness).

See Section 17.1.5.11, “Previous MySQL Cluster Issues Resolved in MySQL 5.1, MySQL Cluster NDB 6.x, and MySQL Cluster NDB 7.x”, for more information.

17.1.5. Known Limitations of MySQL Cluster

In the sections that follow, we discuss known limitations in current releases of MySQL Cluster as compared with the features available when using the MyISAM and InnoDB storage engines. If you check the “Cluster” category in the MySQL bugs database at http://bugs.mysql.com, you can find known bugs in the following categories under “MySQL Server:” in the MySQL bugs database at http://bugs.mysql.com, which we intend to correct in upcoming releases of MySQL Cluster:

  • MySQL Cluster

  • Cluster Direct API (NDBAPI)

  • Cluster Disk Data

  • Cluster Replication

  • ClusterJ

This information is intended to be complete with respect to the conditions just set forth. You can report any discrepancies that you encounter to the MySQL bugs database using the instructions given in Section 1.7, “How to Report Bugs or Problems”. If we do not plan to fix the problem in MySQL Cluster NDB 6.X or 7.X, we will add it to the list.

See Section 17.1.5.11, “Previous MySQL Cluster Issues Resolved in MySQL 5.1, MySQL Cluster NDB 6.x, and MySQL Cluster NDB 7.x” for a list of issues in MySQL Cluster in MySQL 5.0 that have been resolved in the current version.

Note

Limitations and other issues specific to MySQL Cluster Replication are described in Section 17.6.3, “Known Issues in MySQL Cluster Replication”.

17.1.5.1. Noncompliance with SQL Syntax in MySQL Cluster

Some SQL statements relating to certain MySQL features produce errors when used with NDB tables, as described in the following list:

  • Temporary tables. Temporary tables are not supported. Trying either to create a temporary table that uses the NDB storage engine or to alter an existing temporary table to use NDB fails with the error Table storage engine 'ndbcluster' does not support the create option 'TEMPORARY'.

  • Indexes and keys in NDB tables. Keys and indexes on MySQL Cluster tables are subject to the following limitations:

    • Column width. Attempting to create an index on an NDB table column whose width is greater than 3072 bytes succeeds, but only the first 3072 bytes are actually used for the index. In such cases, a warning Specified key was too long; max key length is 3072 bytes is issued, and a SHOW CREATE TABLE statement shows the length of the index as 3072.

    • TEXT and BLOB columns. You cannot create indexes on NDB table columns that use any of the TEXT or BLOB data types.

    • FULLTEXT indexes. The NDB storage engine does not support FULLTEXT indexes, which are possible for MyISAM tables only.

      However, you can create indexes on VARCHAR columns of NDB tables.

    • Prefixes. There are no prefix indexes; only entire columns can be indexed. (The size of an NDB column index is always the same as the width of the column in bytes, up to and including 3072 bytes, as described earlier in this section. Also see Section 17.1.5.6, “Unsupported or Missing Features in MySQL Cluster”, for additional information.)

    • BIT columns. A BIT column cannot be a primary key, unique key, or index, nor can it be part of a composite primary key, unique key, or index.

    • AUTO_INCREMENT columns. Like other MySQL storage engines, the NDB storage engine can handle a maximum of one AUTO_INCREMENT column per table. However, in the case of a Cluster table with no explicit primary key, an AUTO_INCREMENT column is automatically defined and used as a “hidden” primary key. For this reason, you cannot define a table that has an explicit AUTO_INCREMENT column unless that column is also declared using the PRIMARY KEY option. Attempting to create a table with an AUTO_INCREMENT column that is not the table's primary key, and using the NDB storage engine, fails with an error.

  • MySQL Cluster and geometry data types. Geometry data types (WKT and WKB) are supported in NDB tables in MySQL 5.1 (including MySQL Cluster NDB 6.X and 7.X through 7.1). However, spatial indexes are not supported.

  • Character sets and binary log files. Currently, the ndb_apply_status and ndb_binlog_index tables are created using the latin1 (ASCII) character set. Because names of binary logs are recorded in this table, binary log files named using non-Latin characters are not referenced correctly in these tables. This is a known issue, which we are working to fix. (Bug#50226)

    To work around this problem, use only Latin-1 characters when naming binary log files or setting any the --basedir, --log-bin, or --log-bin-index options.

  • Creating NDBCLUSTER tables with user-defined partitioning. Support for user-defined partitioning for MySQL Cluster in MySQL 5.1 (including MySQL Cluster NDB 6.X and 7.X through 7.1) is restricted to [LINEAR] KEY partitioning. Beginning with MySQL 5.1.12, using any other partitioning type with ENGINE=NDB or ENGINE=NDBCLUSTER in a CREATE TABLE statement results in an error.

    Default partitioning scheme. As of MySQL 5.1.6, all MySQL Cluster tables are by default partitioned by KEY using the table's primary key as the partitioning key. If no primary key is explicitly set for the table, the “hidden” primary key automatically created by the NDBCLUSTER storage engine is used instead. For additional discussion of these and related issues, see Section 18.2.4, “KEY Partitioning”.

    Beginning with MySQL Cluster NDB 6.2.18, MySQL Cluster NDB 6.3.25, and MySQL Cluster NDB 7.0.6, CREATE TABLE and ALTER TABLE statements that would cause a user-partitioned NDBCLUSTER table not to meet either or both of the following two requirements are not permitted, and fail with an error (Bug#40709):

    1. The table must have an explicit primary key.

    2. All columns listed in the table's partitioning expression must be part of the primary key.

    Exception. If a user-partitioned NDBCLUSTER table is created using an empty column-list (that is, using PARTITION BY [LINEAR] KEY()), then no explicit primary key is required.

    Maximum number of partitions for NDBCLUSTER tables. The maximum number of partitions that can defined for a NDBCLUSTER table when employing user-defined partitioning is 8 per node group. (See Section 17.1.2, “MySQL Cluster Nodes, Node Groups, Replicas, and Partitions”, for more information about MySQL Cluster node groups.

    DROP PARTITION not supported. It is not possible to drop partitions from NDB tables using ALTER TABLE ... DROP PARTITION. The other partitioning extensions to ALTER TABLEADD PARTITION, REORGANIZE PARTITION, and COALESCE PARTITION—are supported for Cluster tables, but use copying and so are not optimised. See Section 18.3.1, “Management of RANGE and LIST Partitions” and Section 12.1.7, “ALTER TABLE Syntax”.

  • Row-based replication. When using row-based replication with MySQL Cluster, binary logging cannot be disabled. That is, the NDB storage engine ignores the value of sql_log_bin. (Bug#16680)

17.1.5.2. Limits and Differences of MySQL Cluster from Standard MySQL Limits

In this section, we list limits found in MySQL Cluster that either differ from limits found in, or that are not found in, standard MySQL.

Memory usage and recovery. Memory consumed when data is inserted into an NDB table is not automatically recovered when deleted, as it is with other storage engines. Instead, the following rules hold true:

17.1.5.3. Limits Relating to Transaction Handling in MySQL Cluster

A number of limitations exist in MySQL Cluster with regard to the handling of transactions. These include the following:

  • Transaction isolation level. The NDBCLUSTER storage engine supports only the READ COMMITTED transaction isolation level. (InnoDB, for example, supports READ COMMITTED, READ UNCOMMITTED, REPEATABLE READ, and SERIALIZABLE.) See Section 17.5.3.4, “MySQL Cluster Backup Troubleshooting”, for information on how this can affect backing up and restoring Cluster databases.)

  • Transactions and BLOB or TEXT columns. NDBCLUSTER stores only part of a column value that uses any of MySQL's BLOB or TEXT data types in the table visible to MySQL; the remainder of the BLOB or TEXT is stored in a separate internal table that is not accessible to MySQL. This gives rise to two related issues of which you should be aware whenever executing SELECT statements on tables that contain columns of these types:

    1. For any SELECT from a MySQL Cluster table: If the SELECT includes a BLOB or TEXT column, the READ COMMITTED transaction isolation level is converted to a read with read lock. This is done to guarantee consistency.

    2. Prior to MySQL Cluster NDB 7.0.12, for any SELECT which used a primary key lookup or unique key lookup to retrieve any columns that used any of the BLOB or TEXT data types and that was executed within a transaction, a shared read lock was held on the table for the duration of the transaction—that is, until the transaction was either committed or aborted.

      In MySQL Cluster NDB 7.0.12 and later, for primary key lookups, the lock is released as soon as all BLOB or TEXT data has been read. (Bug#49190) However, for unique key lookups, the shared lock continues to be held for the lifetime of the transaction.

      This issue does not occur for queries that use index or table scans, even against NDB tables having BLOB or TEXT columns.

      For example, consider the table t defined by the following CREATE TABLE statement:

      CREATE TABLE t (
          a INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
          b INT NOT NULL,
          c INT NOT NULL,
          d TEXT,
          INDEX i(b),
          UNIQUE KEY u(c)
      ) ENGINE = NDB,
      

      Either of the following queries on t causes a shared read lock, because the first query uses a primary key lookup and the second uses a unique key lookup:

      SELECT * FROM t WHERE a = 1;
      
      SELECT * FROM t WHERE c = 1;
      

      However, none of the four queries shown here causes a shared read lock:

      SELECT * FROM t WHERE b 1;
      
      SELECT * FROM t WHERE d = '1';
      
      SELECT * FROM t;
      
      SELECT b,c WHERE a = 1; 
      

      This is because, of these four queries, the first uses an index scan, the second and third use table scans, and the fourth, while using a primary key lookup, does not retrieve the value of any BLOB or TEXT columns.

      You can help minimize issues with shared read locks by avoiding queries that use unique key lookups (or primary key lookups in MySQL Cluster NDB 7.0.11 and earlier) that retrieve BLOB or TEXT columns, or, in cases where such queries are not avoidable, by committing transactions as soon as possible afterward.

  • Transactions and memory usage. As noted elsewhere in this chapter, MySQL Cluster does not handle large transactions well; it is better to perform a number of small transactions with a few operations each than to attempt a single large transaction containing a great many operations. Among other considerations, large transactions require very large amounts of memory. Because of this, the transactional behavior of a number of MySQL statements is effected as described in the following list:

    • TRUNCATE TABLE is not transactional when used on NDB tables. If a TRUNCATE TABLE fails to empty the table, then it must be re-run until it is successful.

    • DELETE FROM (even with no WHERE clause) is transactional. For tables containing a great many rows, you may find that performance is improved by using several DELETE FROM ... LIMIT ... statements to “chunk” the delete operation. If your objective is to empty the table, then you may wish to use TRUNCATE TABLE instead.

    • LOAD DATA statements. LOAD DATA INFILE is not transactional when used on NDB tables.

      Important

      When executing a LOAD DATA INFILE statement, the NDB engine performs commits at irregular intervals that enable better utilization of the communication network. It is not possible to know ahead of time when such commits take place.

      LOAD DATA FROM MASTER is not supported in MySQL Cluster.

    • ALTER TABLE and transactions. When copying an NDB table as part of an ALTER TABLE, the creation of the copy is nontransactional. (In any case, this operation is rolled back when the copy is deleted.)

  • Transactions and the COUNT() function. When using MySQL Cluster Replication, it is not possible to guarantee the transactional consistency of the COUNT() function on the slave. In other words, when performing on the master a series of statements (INSERT, DELETE, or both) that changes the number of rows in a table within a single transaction, executing SELECT COUNT(*) FROM table queries on the slave may yield intermediate results. This is due to the fact that SELECT COUNT(...) may perform dirty reads, and is not a bug in the NDB storage engine. (See Bug#31321 for more information.)

17.1.5.4. MySQL Cluster Error Handling

Starting, stopping, or restarting a node may give rise to temporary errors causing some transactions to fail. These include the following cases:

  • Temporary errors. When first starting a node, it is possible that you may see Error 1204 Temporary failure, distribution changed and similar temporary errors.

  • Errors due to node failure. The stopping or failure of any data node can result in a number of different node failure errors. (However, there should be no aborted transactions when performing a planned shutdown of the cluster.)

In either of these cases, any errors that are generated must be handled within the application. This should be done by retrying the transaction.

See also Section 17.1.5.2, “Limits and Differences of MySQL Cluster from Standard MySQL Limits”.

17.1.5.5. Limits Associated with Database Objects in MySQL Cluster

Some database objects such as tables and indexes have different limitations when using the NDBCLUSTER storage engine:

  • Table names containing special characters. NDB tables whose names contain characters other than letters, numbers, dashes, and underscores and which are created on one SQL node were not always discovered correctly by other SQL nodes. (Bug#31470)

    Note

    This issue was fixed in MySQL 5.1.23, MySQL Cluster NDB 6.2.7, and MySQL Cluster NDB 6.3.4.

  • Number of database objects. The maximum number of all NDB database objects in a single MySQL Cluster—including databases, tables, and indexes—is limited to 20320.

  • Attributes per table. Prior to MySQL Cluster NDB 7.0.15 and MySQL Cluster NDB 7.1.4, the maximum number of attributes (that is, columns and indexes) per table is limited to 128.

    Beginning with MySQL Cluster NDB 7.0.15 and MySQL Cluster NDB 7.1.4, this limit is increased to 512.

  • Attributes per key. The maximum number of attributes per key is 32.

  • Row size. The maximum permitted size of any one row is 8052 bytes. Each BLOB or TEXT column contributes 256 + 8 = 264 bytes to this total.

  • Number of rows per partition. A single MySQL Cluster partition can hold a maximum of 46137488 rows. Since the number of partitions is the same as the number of data nodes in the cluster (see Section 17.1.2, “MySQL Cluster Nodes, Node Groups, Replicas, and Partitions”), you can increase the available space for data storage by using more data nodes.

    In MySQL Cluster NDB 7.0 and later MySQL Cluster release series, you can increase the number of data nodes in the cluster while the cluster remains in operation. See Section 17.5.11, “Adding MySQL Cluster Data Nodes Online”, for more information.

    It is also possible to increase the number of partitions for NDB tables by using explicit KEY or LINEAR KEY partitioning (see Section 18.2.4, “KEY Partitioning”).

17.1.5.6. Unsupported or Missing Features in MySQL Cluster

A number of features supported by other storage engines are not supported for NDB tables. Trying to use any of these features in MySQL Cluster does not cause errors in or of itself; however, errors may occur in applications that expects the features to be supported or enforced:

Note

See Section 17.1.5.3, “Limits Relating to Transaction Handling in MySQL Cluster”, for more information relating to limitations on transaction handling in NDB.

17.1.5.7. Limitations Relating to Performance in MySQL Cluster

The following performance issues are specific to or especially pronounced in MySQL Cluster:

  • Range scans. There are query performance issues due to sequential access to the NDB storage engine; it is also relatively more expensive to do many range scans than it is with either MyISAM or InnoDB.

  • Reliability of Records in range. The Records in range statistic is available but is not completely tested or officially supported. This may result in nonoptimal query plans in some cases. If necessary, you can employ USE INDEX or FORCE INDEX to alter the execution plan. See Section 12.2.8.2, “Index Hint Syntax”, for more information on how to do this.

  • Unique hash indexes. Unique hash indexes created with USING HASH cannot be used for accessing a table if NULL is given as part of the key.

17.1.5.8. Issues Exclusive to MySQL Cluster

The following are limitations specific to the NDBCLUSTER storage engine:

  • Machine architecture. All machines used in the cluster must have the same architecture. That is, all machines hosting nodes must be either big-endian or little-endian, and you cannot use a mixture of both. For example, you cannot have a management node running on a PowerPC which directs a data node that is running on an x86 machine. This restriction does not apply to machines simply running mysql or other clients that may be accessing the cluster's SQL nodes.

  • Binary logging. MySQL Cluster has the following limitations or restrictions with regard to binary logging:

See also Section 17.1.5.10, “Limitations Relating to Multiple MySQL Cluster Nodes”.

17.1.5.9. Limitations Relating to MySQL Cluster Disk Data Storage

Disk Data object maxmimums and minimums. Disk data objects are subject to the following maximums and minimums:

  • Maximum number of tablespaces: 232 (4294967296)

  • Maximum number of data files per tablespace: 216 (65536)

  • The theoretical maximum number of extents per tablespace data file is 216 (65536); however, for practical purposes, the recommended maximum number of extents per data file is 215 (32768).

  • Maximum data file size: The theoretical limit is 64G; however, in MySQL 5.1 (including MySQL Cluster NDB 6.X and 7.X through 7.1), the practical upper limit is 32G. This is equivalent to 32768 extents of 1M each.

    The minimum and maximum possible sizes of extents for tablespace data files are 32K and 2G, respectively. See Section 12.1.18, “CREATE TABLESPACE Syntax”, for more information.

Disk Data tables and diskless mode. Use of Disk Data tables is not supported when running the cluster in diskless mode. Beginning with MySQL 5.1.12, it is prohibited altogether. (Bug#20008)

17.1.5.10. Limitations Relating to Multiple MySQL Cluster Nodes

Multiple SQL nodes. The following are issues relating to the use of multiple MySQL servers as MySQL Cluster SQL nodes, and are specific to the NDBCLUSTER storage engine:

  • No distributed table locks. A LOCK TABLES works only for the SQL node on which the lock is issued; no other SQL node in the cluster “sees” this lock. This is also true for a lock issued by any statement that locks tables as part of its operations. (See next item for an example.)

  • ALTER TABLE operations. ALTER TABLE is not fully locking when running multiple MySQL servers (SQL nodes). (As discussed in the previous item, MySQL Cluster does not support distributed table locks.)

Multiple management nodes. When using multiple management servers:

  • You must give nodes explicit IDs in connectstrings because automatic allocation of node IDs does not work across multiple management servers.

  • You must take extreme care to have the same configurations for all management servers. No special checks for this are performed by the cluster.

Multiple network addresses. Multiple network addresses per data node are not supported. Use of these is liable to cause problems: In the event of a data node failure, an SQL node waits for confirmation that the data node went down but never receives it because another route to that data node remains open. This can effectively make the cluster inoperable.

Note

It is possible to use multiple network hardware interfaces (such as Ethernet cards) for a single data node, but these must be bound to the same address. This also means that it not possible to use more than one [tcp] section per connection in the config.ini file. See Section 17.3.2.8, “MySQL Cluster TCP/IP Connections”, for more information.

17.1.5.11. Previous MySQL Cluster Issues Resolved in MySQL 5.1, MySQL Cluster NDB 6.x, and MySQL Cluster NDB 7.x

A number of limitations and related issues existing in earlier versions of MySQL Cluster have been resolved:

  • Variable-length column support. The NDBCLUSTER storage engine now supports variable-length column types for in-memory tables.

    Previously, for example, any Cluster table having one or more VARCHAR fields which contained only relatively small values, much more memory and disk space were required when using the NDBCLUSTER storage engine than would have been the case for the same table and data using the MyISAM engine. In other words, in the case of a VARCHAR column, such a column required the same amount of storage as a CHAR column of the same size. In MySQL 5.1, this is no longer the case for in-memory tables, where storage requirements for variable-length column types such as VARCHAR and BINARY are comparable to those for these column types when used in MyISAM tables (see Section 10.5, “Data Type Storage Requirements”).

    Important

    For MySQL Cluster Disk Data tables, the fixed-width limitation continues to apply. See Section 17.5.10, “MySQL Cluster Disk Data Tables”.

  • Replication with MySQL Cluster. It is now possible to use MySQL replication with Cluster databases. For details, see Section 17.6, “MySQL Cluster Replication”.

    Circular Replication. Circular replication is also supported with MySQL Cluster, beginning with MySQL 5.1.18. See Section 17.6.10, “MySQL Cluster Replication: Multi-Master and Circular Replication”.

  • auto_increment_increment and auto_increment_offset. The auto_increment_increment and auto_increment_offset server system variables are supported for Cluster replication beginning with MySQL 5.1.20, MySQL Cluster NDB 6.2.5, and MySQL Cluster 6.3.2.

  • Database autodiscovery and online schema changes. Autodiscovery of databases is now supported for multiple MySQL servers accessing the same MySQL Cluster. Formerly, autodiscovery in MySQL Cluster 5.1 and MySQL Cluster NDB 6.x releases required that a given mysqld was already running and connected to the cluster at the time that the database was created on a different mysqld—in other words, when a mysqld process connected to the cluster after a database named db_name was created, it was necessary to issue a CREATE DATABASE db_name or CREATE SCHEMA db_name statement on the “new” MySQL server when it first accesseed that MySQL Cluster. Beginning with MySQL Cluster NDB 6.2.16 and MySQL Cluster NDB 6.3.18, such a CREATE statement is no longer required. (Bug#39612)

    This also means that online schema changes in NDB tables are now possible. That is, the result of operations such as ALTER TABLE and CREATE INDEX performed on one SQL node in the cluster are now visible to the cluster's other SQL nodes without any additional action being taken.

  • Backup and restore between architectures. Beginning with MySQL 5.1.10, it is possible to perform a Cluster backup and restore between different architectures. Previously—for example—you could not back up a cluster running on a big-endian platform and then restore from that backup to a cluster running on a little-endian system. (Bug#19255)

  • Character set directory. Beginning with MySQL 5.1.10, it is possible to install MySQL with Cluster support to a nondefault location and change the search path for font description files using either the --basedir or --character-sets-dir options. (Previously, ndbd in MySQL 5.1 searched only the default path—typically /usr/local/mysql/share/mysql/charsets—for character sets.)

  • Multiple management servers. In MySQL 5.1 (including all MySQL Cluster NDB 6.x and later versions), it is no longer necessary, when running multiple management servers, to restart all the cluster's data nodes to enable the management nodes to see one another.

    Also, when using multiple management servers and starting concurrently several API nodes (possibly including one or more SQL nodes) whose connectstrings listed the management servers in different order, it was possible for 2 API nodes to be assigned the same node ID. This issue is resolved in MySQL Cluster NDB 6.2.17, 6.3.23, and 6.4.3. (Bug#42973)

  • Multiple data node processes per host. Beginning with MySQL Cluster NDB 6.2.0, you can use multiple data node processes on a single host. (In MySQL Cluster NDB 6.1, MySQL 5.1, and earlier release series, we did not support production MySQL Cluster deployments in which more than one ndbd process was run on a single physical machine.)

    In addition, MySQL Cluster NDB 7.0 introduces support for multi-threaded data nodes (ndbmtd). See Section 17.1.4.2, “MySQL Cluster Development in MySQL Cluster NDB 7.0”, and Section 17.4.3, “ndbmtd — The MySQL Cluster Data Node Daemon (Multi-Threaded)”, for more information.

  • Identifiers. Formerly (in MySQL 5.0 and earlier), database names, table names and attribute names could not be as long for NDB tables as tables using other storage engines, because attribute names were truncated internally. In MySQL 5.1 and later, names of MySQL Cluster databases, tables, and table columns follow the same rules regarding length as they do for any other storage engine.

  • Length of CREATE TABLE statements. CREATE TABLE statements may be no more than 4096 characters in length. This limitation affects MySQL 5.1.6, 5.1.7, and 5.1.8 only. (See Bug#17813)

  • IGNORE and REPLACE functionality. In MySQL 5.1.7 and earlier, INSERT IGNORE, UPDATE IGNORE, and REPLACE were supported only for primary keys, but not for unique keys. It was possible to work around this issue by removing the constraint, then dropping the unique index, performing any inserts, and then adding the unique index again.

    This limitation was removed for INSERT IGNORE and REPLACE in MySQL 5.1.8. (See Bug#17431.)

  • AUTO_INCREMENT columns. In MySQL 5.1.10 and earlier versions, the maximum number of tables having AUTO_INCREMENT columns—including those belonging to hidden primary keys—was 2048.

    This limitation was lifted in MySQL 5.1.11.

  • Maximum number of cluster nodes. Prior to MySQL Cluster NDB 6.1.1, the total maximum number of nodes in a MySQL Cluster was 63, including all SQL nodes (MySQL Servers), API nodes (applications accessing the cluster other than MySQL servers), data nodes, and management servers.

    Starting with MySQL Cluster NDB 6.1.1, the total maximum number of nodes in a MySQL Cluster is 255, including all SQL nodes (MySQL Servers), API nodes (applications accessing the cluster other than MySQL servers), data nodes, and management servers. The total number of data nodes and management nodes beginning with this version is 63, of which up to 48 can be data nodes.

    Note

    The limitation that a data node cannot have a node ID greater than 49 continues to apply.

  • Recovery of memory from deleted rows. Beginning with MySQL Cluster NDB 6.3.7, memory can be reclaimed from an NDB table for reuse with any NDB table by employing OPTIMIZE TABLE, subject to the following limitations:

    You can regulate the effects of OPTIMIZE on performance by adjusting the value of the global system variable ndb_optimization_delay, which sets the number of milliseconds to wait between batches of rows being processed by OPTIMIZE. The default value is 10 milliseconds. It is possible to set a lower value (to a minimum of 0), but not recommended. The maximum is 100000 milliseconds (that is, 100 seconds).

  • Implicit Rollbacks. Prior to MySQL Cluster NDB 6.2.17 and MySQL Cluster NDB 6.3.19, MySQL Cluster did not automtically roll back a transaction that was aborted by a duplicate key or similar error, and subsequent statements raised ERROR 1296 (HY000): Got error 4350 'Transaction already aborted' from NDBCLUSTER. In such cases, it was necessary to issue an explicit ROLLBACK statement first, and then to retry the entire transaction.

    Beginning with MySQL Cluster NDB 6.2.17 and MySQL Cluster NDB 6.3.19, this limitation has been removed; now, an error which causes a transaction to be aborted generates an implicit rollback of the entire transaction. This is logged with the warning Storage engine NDB does not support rollback for this statement. Transaction rolled back and must be restarted. A statement subsequent to this starts a new transaction. (Bug#32656)

    Note

    The NDBCLUSTER storage engine does not support partial transactions or partial rollbacks of transactions in any version of MySQL Cluster.

  • Number of tables. Previously, the maximum number of NDBCLUSTER tables in a single MySQL Cluster was 1792, but this is no longer the case in MySQL 5.1 and later MySQL Cluster releases. However, the number of tables is still included in the total maximum number of NDBCLUSTER database objects (20320). (See Section 17.1.5.5, “Limits Associated with Database Objects in MySQL Cluster”.)

  • DDL operations. Beginning with MySQL Cluster NDB 6.4.0, DDL operations (such as CREATE TABLE or ALTER TABLE) are protected from data node failures. Previously, if a data node failed while trying to perform one of these, the data dictionary became locked and no further DDL statements could be executed without restarting the cluster (Bug#36718).

  • Adding and dropping of data nodes. In MySQL Cluster NDB 6.3 and previous versions of MySQL Cluster, the online adding or dropping of data nodes was not possible; such operations required a complete shutdown and restart of the entire cluster. In MySQL Cluster NDB 7.0 (beginning with MySQL Cluster NDB 6.4.0) and later MySQL Cluster release series, it is possible to add new data nodes to a running MySQL Cluster by performing a rolling restart, so that the cluster and the data stored in it remain available to applications.

    When planning to increase the number of data nodes in the cluster online in MySQL Cluster NDB 7.0 or MySQL Cluster NDB 7.1, you should be aware of and take into account the following issues:

    • New data nodes can be added online to a MySQL Cluster only as part of a new node group.

    • New data nodes can be added online, but cannot yet be dropped online. Reducing the number of data nodes still requires a system restart of the cluster.

    • As in previous MySQL Cluster releases, it is not possible to change online either the number of replicas (NoOfReplicas configuration parameter) or the number of data nodes per node group. These changes require a system restart.

    • Redistribution of existing cluster data using the new data nodes is not automatic; however, this can be accomplished using simple SQL statements in the mysql client or other MySQL client application once the nodes have been added. During this procedure, it is not possible to perform DDL operations, although DML operations can continue as normal.

      The distribution of new cluster data (that is, data stored in the cluster after the new nodes have been added) uses the new nodes without manual intervention.

    For more information, see Section 17.5.11, “Adding MySQL Cluster Data Nodes Online”.

  • Native support for default column values. Starting with MySQL Cluster NDB 7.1.0, default values for table columns are stored by NDBCLUSTER, rather than by the MySQL server as was previously the case. Because less data must be sent from an SQL node to the data nodes, inserts on tables having column value defaults can be performed more efficiently than before.

    Tables created using previous MySQL Cluster releases can still be used in MySQL Cluster 7.1.0 and later, although they do not support native default values and continue to use defaults supplied by the MySQL server until they are upgraded. This can be done by means of an offline ALTER TABLE statement.

    Important

    You cannot set or change a table column's default value using an online ALTER TABLE operation

17.2. MySQL Cluster Multi-Computer How-To

This section is a “How-To” that describes the basics for how to plan, install, configure, and run a MySQL Cluster. Whereas the examples in Section 17.3, “MySQL Cluster Configuration” provide more in-depth information on a variety of clustering options and configuration, the result of following the guidelines and procedures outlined here should be a usable MySQL Cluster which meets the minimum requirements for availability and safeguarding of data.

This section covers hardware and software requirements; networking issues; installation of MySQL Cluster; configuration issues; starting, stopping, and restarting the cluster; loading of a sample database; and performing queries.

Basic assumptions. This How-To makes the following assumptions:

  1. The cluster is to be set up with four nodes, each on a separate host, and each with a fixed network address on a typical Ethernet network as shown here:

    NodeIP Address
    Management (MGMD) node192.168.0.10
    MySQL server (SQL) node192.168.0.20
    Data (NDBD) node "A"192.168.0.30
    Data (NDBD) node "B"192.168.0.40

    This may be made clearer in the following diagram:

    MySQL Cluster Multi-Computer Setup

    In the interest of simplicity (and reliability), this How-To uses only numeric IP addresses. However, if DNS resolution is available on your network, it is possible to use host names in lieu of IP addresses in configuring Cluster. Alternatively, you can use the hosts file (typically /etc/hosts for Linux and other Unix-like operating systems, C:\WINDOWS\system32\drivers\etc\hosts on Windows, or your operating system's equivalent) for providing a means to do host lookup if such is available.

    Note

    A common problem when trying to use host names for Cluster nodes arises because of the way in which some operating systems (including some Linux distributions) set up the system's own host name in the /etc/hosts during installation. Consider two machines with the host names ndb1 and ndb2, both in the cluster network domain. Red Hat Linux (including some derivatives such as CentOS and Fedora) places the following entries in these machines' /etc/hosts files:

    #  ndb1 /etc/hosts:
    127.0.0.1   ndb1.cluster ndb1 localhost.localdomain localhost
    
    #  ndb2 /etc/hosts:
    127.0.0.1   ndb2.cluster ndb2 localhost.localdomain localhost
    

    SUSE Linux (including OpenSUSE) places these entries in the machines' /etc/hosts files:

    #  ndb1 /etc/hosts:
    127.0.0.1       localhost
    127.0.0.2       ndb1.cluster ndb1
    
    #  ndb2 /etc/hosts:
    127.0.0.1       localhost
    127.0.0.2       ndb2.cluster ndb2
    

    In both instances, ndb1 routes ndb1.cluster to a loopback IP address, but gets a public IP address from DNS for ndb2.cluster, while ndb2 routes ndb2.cluster to a loopback address and obtains a public address for ndb1.cluster. The result is that each data node connects to the management server, but cannot tell when any other data nodes have connected, and so the data nodes appear to hang while starting.

    You should also be aware that you cannot mix localhost and other host names or IP addresses in config.ini. For these reasons, the solution in such cases (other than to use IP addresses for all config.ini HostName entries) is to remove the fully qualified host names from /etc/hosts and use these in config.ini for all cluster hosts.

  2. Each host in our scenario is an Intel-based desktop PC running a supported operating system installed to disk in a standard configuration, and running no unnecessary services. The core operating system with standard TCP/IP networking capabilities should be sufficient. Also for the sake of simplicity, we also assume that the file systems on all hosts are set up identically. In the event that they are not, you should adapt these instructions accordingly.

  3. Standard 100 Mbps or 1 gigabit Ethernet cards are installed on each machine, along with the proper drivers for the cards, and that all four hosts are connected through a standard-issue Ethernet networking appliance such as a switch. (All machines should use network cards with the same throughout. That is, all four machines in the cluster should have 100 Mbps cards or all four machines should have 1 Gbps cards.) MySQL Cluster works in a 100 Mbps network; however, gigabit Ethernet provides better performance.

    Note that MySQL Cluster is not intended for use in a network for which throughput is less than 100 Mbps or which experiences a high degree of latency. For this reason (among others), attempting to run a MySQL Cluster over a wide area network such as the Internet is not likely to be successful, and is not supported in production.

  4. For our sample data, we use the world database which is available for download from the MySQL Web site (see http://dev.mysql.com/doc/index-other.html). We assume that each machine has sufficient memory for running the operating system, host NDB process, and (on the data nodes) storing the database.

We assume that you already know how to perform a minimal installation and configuration of the operating system with networking capability, or that you are able to obtain assistance in this elsewhere if needed.

For information relating to installation of MySQL Cluster on Linux and other Unix-like operating systems, see Section 17.2.1, “Installing MySQL Cluster on Linux”. For information relating to installation of MySQL Cluster on Windows operating systems, see Section 17.2.2, “Installing MySQL Cluster on Windows”.

For information about MySQL Cluster hardware, software, and networking requirements, see Section 17.1.3, “MySQL Cluster Hardware, Software, and Networking Requirements”.

17.2.1. Installing MySQL Cluster on Linux

The next few sections refer to a Linux operating system, but the instructions and procedures given there should be easily adaptable to other supported Unix-like operating systems.

Beginning with MySQL Cluster NDB 7.1.3, MySQL Cluster is also supported for production use on Windows operating systems; for basic setup instructions specific to Windows, see Section 17.2.2, “Installing MySQL Cluster on Windows”.

17.2.1.1. MySQL Cluster Multi-Computer Installation

Each MySQL Cluster host computer running an SQL node must have installed on it a MySQL binary. For management nodes and data nodes, it is not necessary to install the MySQL server binary, but management nodes require the management server daemon (ndb_mgmd) and data nodes require the data node daemon (ndbd; in MySQL Cluster NDB 7.0 and later, you can use ndbmtd instead). It is also a good idea to install the management client (ndb_mgm) on the management server host. This section covers the steps necessary to install the correct binaries for each type of Cluster node.

Oracle provides precompiled binaries that support MySQL Cluster. However, we also include information relating to installing a MySQL Cluster after building MySQL from source. For setting up a cluster using MySQL's binaries, the first step in the installation process for each cluster host is to download the latest MySQL Cluster NDB 6.3, MySQL Cluster NDB 7.0, or MySQL Cluster NDB 7.1 binary archive (mysql-cluster-gpl-6.3.39-linux-i686-glibc23.tar.gz, mysql-cluster-gpl-7.0.20-linux-i686-glibc23.tar.gz, or mysql-cluster-gpl-7.1.9-linux-i686-glibc23.tar.gz, respectively) from the MySQL Cluster downloads area. We assume that you have placed this file in each machine's /var/tmp directory. (If you do require a custom binary, see Section 2.11.4, “Installing from the Development Source Tree”.)

When compiling MySQL Cluster NDB 7.0 or later from source, no special options are required for building multi-threaded data node binaries. On Unix platforms, configuring the build with any of the options --plugins=ndbcluster, --plugins=max, or, --plugins=max-no-innodb causes ndbmtd to be built automatically; make install places the ndbmtd binary in the libexec directory along with mysqld, ndbd, and ndb_mgm.

Important

Currently, MySQL Cluster is not compatible with the InnoDB Plugin. You must use the version of InnoDB that is supplied with the MySQL Server. You can build MySQL Cluster with InnoDB storage engine support using the --with-plugins=max or --with-innodb option for configure.

RPMs are also available for both 32-bit and 64-bit Linux platforms. For a MySQL Cluster, three RPMs are required:

  • The Server RPM (for example, MySQL-Cluster-gpl-server-6.3.39-0.sles10.i586.rpm, MySQL-Cluster-gpl-server-7.0.20-0.sles10.i586.rpm, or MySQL-Cluster-gpl-server-7.1.9-0.sles10.i586.rpm), which supplies the core files needed to run a MySQL Server with NDBCLUSTER storage engine support (that is, as a MySQL Cluster SQL node).

    If you do not have your own client application capable of administering a MySQL server, you should also obtain and install the Client RPM (for example, MySQL-Cluster-gpl-client-6.3.39-0.sles10.i586.rpm, MySQL-Cluster-gpl-client-7.0.20-0.sles10.i586.rpm, or MySQL-Cluster-gpl-client-7.1.9-0.sles10.i586.rpm).

  • The Cluster storage engine RPM (for example, MySQL-Cluster-gpl-storage-6.3.39-0.sles10.i586.rpm, MySQL-Cluster-gpl-storage-7.0.20-0.sles10.i586.rpm, or MySQL-Cluster-gpl-storage-7.1.9-0.sles10.i586.rpm), which supplies the MySQL Cluster data node binary (ndbd).

  • The Cluster storage engine management RPM (for example, MySQL-Cluster-gpl-management-6.3.39-0.sles10.i586.rpm, MySQL-Cluster-gpl-management-7.0.20-0.sles10.i586.rpm, or MySQL-Cluster-gpl-management-7.1.9-0.sles10.i586.rpm) which provides the MySQL Cluster management server binary (ndb_mgmd).

In addition, you should also obtain the NDB Cluster - Storage engine basic tools RPM (for example, MySQL-Cluster-gpl-tools-6.3.39-0.sles10.i586.rpm, MySQL-Cluster-gpl-tools-7.0.20-0.sles10.i586.rpm, or MySQL-Cluster-gpl-tools-7.1.9-0.sles10.i586.rpm), which supplies several useful applications for working with a MySQL Cluster. The most important of these is the MySQL Cluster management client (ndb_mgm). The NDB Cluster - Storage engine extra tools RPM (for example, MySQL-Cluster-gpl-extra-6.3.39-0.sles10.i586.rpm, MySQL-Cluster-gpl-extra-7.0.20-0.sles10.i586.rpm, or MySQL-Cluster-gpl-extra-7.1.9-0.sles10.i586.rpm) contains some additional testing and monitoring programs, but is not required to install a MySQL Cluster. (For more information about these additional programs, see Section 17.4, “MySQL Cluster Programs”.)

The MySQL Cluster version number in the RPM file names (shown here as 6.3.39, 7.0.20, or 7.1.9) can vary according to the version which you are actually using. It is very important that all of the Cluster RPMs to be installed have the same version number. The glibc version number (if present), and architecture designation (shown here as i586) should be appropriate to the machine on which the RPM is to be installed.

See Section 2.5.1, “Installing MySQL from RPM Packages on Linux”, for general information about installing MySQL using RPMs supplied by Oracle.

After installing from RPM, you still need to configure the cluster as discussed in Section 17.2.1.2, “MySQL Cluster Multi-Computer Configuration”.

Note

After completing the installation, do not yet start any of the binaries. We show you how to do so following the configuration of all nodes.

Data and SQL Node Installation: .tar.gz Binary. On each of the machines designated to host data or SQL nodes, perform the following steps as the system root user:

  1. Check your /etc/passwd and /etc/group files (or use whatever tools are provided by your operating system for managing users and groups) to see whether there is already a mysql group and mysql user on the system. Some OS distributions create these as part of the operating system installation process. If they are not already present, create a new mysql user group, and then add a mysql user to this group:

    shell> groupadd mysql
    shell> useradd -g mysql mysql
    

    The syntax for useradd and groupadd may differ slightly on different versions of Unix, or they may have different names such as adduser and addgroup.

  2. Change location to the directory containing the downloaded file, unpack the archive, and create a symlink to the mysql directory named mysql. Note that the actual file and directory names vary according to the MySQL Cluster version number.

    shell> cd /var/tmp
    shell> tar -C /usr/local -xzvf mysql-cluster-gpl-7.1.9-linux-i686-glibc23.tar.gz
    shell> ln -s /usr/local/mysql-cluster-gpl-7.1.9-linux-i686-glibc23.tar.gz /usr/local/mysql
    
  3. Change location to the mysql directory and run the supplied script for creating the system databases:

    shell> cd mysql
    shell> scripts/mysql_install_db --user=mysql
    
  4. Set the necessary permissions for the MySQL server and data directories:

    shell> chown -R root .
    shell> chown -R mysql data
    shell> chgrp -R mysql .
    

    Note that the data directory on each machine hosting a data node is /usr/local/mysql/data. This piece of information is essential when configuring the management node. (See Section 17.2.1.2, “MySQL Cluster Multi-Computer Configuration”.)

  5. Copy the MySQL startup script to the appropriate directory, make it executable, and set it to start when the operating system is booted up:

    shell> cp support-files/mysql.server /etc/rc.d/init.d/
    shell> chmod +x /etc/rc.d/init.d/mysql.server
    shell> chkconfig --add mysql.server
    

    (The startup scripts directory may vary depending on your operating system and version—for example, in some Linux distributions, it is /etc/init.d.)

    Here we use Red Hat's chkconfig for creating links to the startup scripts; use whatever means is appropriate for this purpose on your operating system and distribution, such as update-rc.d on Debian.

Remember that the preceding steps must be repeated on each machine where an SQL node is to reside.

SQL node installation: RPM files. On each machine to be used for hosting a cluster SQL node, install the Server RPM by executing the following command as the system root user, replacing the name shown for the RPM as necessary to match the name of the RPM downloaded from the MySQL web site:

shell> rpm -Uhv MySQL-Cluster-gpl-server-7.1.9-0.sles10.i586.rpm

This installs the MySQL server binary (mysqld) in the /usr/sbin directory, as well as all needed MySQL Server support files. It also installs the mysql.server and mysqld_safe startup scripts in /usr/share/mysql and /usr/bin, respectively. The RPM installer should take care of general configuration issues (such as creating the mysql user and group, if needed) automatically.

Note

To administer the SQL node (MySQL server), you should also install the Client RPM, as shown here:

shell> rpm -Uhv MySQL-Cluster-gpl-client-7.1.9-0.sles10.i586.rpm

This installs the mysql client program.

SQL node installation: building from source. If you compile MySQL with clustering support (for example, by using the BUILD/compile-platform_name-max script appropriate to your platform), and perform the default installation (using make install as the root user), mysqld is placed in /usr/local/mysql/bin. Follow the steps given in Section 2.11, “Installing MySQL from a Source Distribution” to make mysqld ready for use. If you want to run multiple SQL nodes, you can use a copy of the same mysqld executable and its associated support files on several machines. The easiest way to do this is to copy the entire /usr/local/mysql directory and all directories and files contained within it to the other SQL node host or hosts, then repeat the steps from Section 2.11, “Installing MySQL from a Source Distribution” on each machine. If you configure the build with a nondefault --prefix, you need to adjust the directory accordingly.

Data node installation: RPM Files. On a computer that is to host a cluster data node it is necessary to install only the NDB Cluster - Storage engine RPM. To do so, copy this RPM to the data node host, and run the following command as the system root user, replacing the name shown for the RPM as necessary to match that of the RPM downloaded from the MySQL web site:

shell> rpm -Uhv MySQL-Cluster-gpl-storage-7.1.9-0.sles10.i586.rpm

The previous command installs the MySQL Cluster data node binary (ndbd) in the /usr/sbin directory.

Data node installation: building from source. The only executable required on a data node host is ndbd or (in MySQL Cluster NDB 7.0 and later) ndbmtd (mysqld, for example, does not have to be present on the host machine). By default when doing a source build, this file is placed in the directory /usr/local/mysql/libexec. For installing on multiple data node hosts, only ndbd need be copied to the other host machine or machines. (This assumes that all data node hosts use the same architecture and operating system; otherwise you may need to compile separately for each different platform.) ndbd need not be in any particular location on the host's file system, as long as the location is known.

Management node installation: .tar.gz binary. Installation of the management node does not require the mysqld binary. Only the MySQL Cluster management server (ndb_mgmd) is required; you most likely want to install the management client (ndb_mgm) as well. Both of these binaries also be found in the .tar.gz archive. Again, we assume that you have placed this archive in /var/tmp.

As system root (that is, after using sudo, su root, or your system's equivalent for temporarily assuming the system administrator account's privileges), perform the following steps to install ndb_mgmd and ndb_mgm on the Cluster management node host:

  1. Change location to the /var/tmp directory, and extract the ndb_mgm and ndb_mgmd from the archive into a suitable directory such as /usr/local/bin:

    shell> cd /var/tmp
    shell> tar -zxvf mysql-5.1.51-ndb-7.1.9-linux-i686-glibc23.tar.gz
    shell> cd mysql-5.1.51-ndb-7.1.9-linux-i686-glibc23
    shell> cp bin/ndb_mgm* /usr/local/bin
    

    (You can safely delete the directory created by unpacking the downloaded archive, and the files it contains, from /var/tmp once ndb_mgm and ndb_mgmd have been copied to the executables directory.)

  2. Change location to the directory into which you copied the files, and then make both of them executable:

    shell> cd /usr/local/bin
    shell> chmod +x ndb_mgm*
    

Management node installation: RPM file. To install the MySQL Cluster management server, it is necessary only to use the NDB Cluster - Storage engine management RPM. Copy this RPM to the computer intended to host the management node, and then install it by running the following command as the system root user (replace the name shown for the RPM as necessary to match that of the Storage engine management RPM downloaded from the MySQL web site):

shell> rpm -Uhv MySQL-Cluster-gpl-management-7.1.9-0.sles10.i586.rpm

This installs the management server binary (ndb_mgmd) to the /usr/sbin directory.

You should also install the NDB management client, which is supplied by the Storage engine basic tools RPM. Copy this RPM to the same computer as the management node, and then install it by running the following command as the system root user (again, replace the name shown for the RPM as necessary to match that of the Storage engine basic tools RPM downloaded from the MySQL web site):

shell> rpm -Uhv MySQL-Cluster-gpl-tools-7.1.9-0.sles10.i586.rpm

The Storage engine basic tools RPM installs the MySQL Cluster management client (ndb_mgm) to the /usr/bin directory.

Note

You can also install the Cluster storage engine extra tools RPM, if you wish, as shown here:

shell> rpm -Uhv MySQL-Cluster-gpl-extra-7.1.9-0.sles10.i586.rpm

You may find the extra tools useful; however the Cluster storage engine extra tools RPM is not required to install a working MySQL Cluster.

Management node installation: building from source. When building from source and running the default make install, the management server binary (ndb_mgmd) is placed in /usr/local/mysql/libexec, while the management client binary (ndb_mgm) can be found in /usr/local/mysql/bin. Only ndb_mgmd is required to be present on a management node host; however, it is also a good idea to have ndb_mgm present on the same host machine. Neither of these executables requires a specific location on the host machine's file system.

In Section 17.2.1.2, “MySQL Cluster Multi-Computer Configuration”, we create configuration files for all of the nodes in our example MySQL Cluster.

MySQL Cluster on Windows. Experimental support for MySQL Cluster on Microsoft Windows operating systems was introduced in MySQL Cluster NDB 7.0. Beginning with MySQL Cluster NDB 7.1.3, production support is provided for MySQL Cluster on Windows, and MySQL Cluster binaries for Windows can be obtained from http://dev.mysql.com/downloads/cluster/. If you wish to compile MySQL Cluster from source on Windows, you must configure the build using the WITH_NDBCLUSTER_STORAGE_ENGINE option before creating the Visual Studio project files and compiling the project. After running make_win_bin_dist, the MySQL Cluster binaries can be found in the bin directory of the resulting archive. For more information, see Section 2.3.9, “Installing MySQL from Source on Windows”.

17.2.1.2. MySQL Cluster Multi-Computer Configuration

For our four-node, four-host MySQL Cluster, it is necessary to write four configuration files, one per node host.

  • Each data node or SQL node requires a my.cnf file that provides two pieces of information: a connectstring that tells the node where to find the management node, and a line telling the MySQL server on this host (the machine hosting the data node) to enable the NDBCLUSTER storage engine.

    For more information on connectstrings, see Section 17.3.2.3, “The MySQL Cluster Connectstring”.

  • The management node needs a config.ini file telling it how many replicas to maintain, how much memory to allocate for data and indexes on each data node, where to find the data nodes, where to save data to disk on each data node, and where to find any SQL nodes.

Configuring the storage and SQL nodes. The my.cnf file needed for the data nodes is fairly simple. The configuration file should be located in the /etc directory and can be edited using any text editor. (Create the file if it does not exist.) For example:

shell> vi /etc/my.cnf
Note

We show vi being used here to create the file, but any text editor should work just as well.

For each data node and SQL node in our example setup, my.cnf should look like this:

[mysqld]
# Options for mysqld process:
ndbcluster                      # run NDB storage engine
ndb-connectstring=192.168.0.10  # location of management server

[mysql_cluster]
# Options for ndbd process:
ndb-connectstring=192.168.0.10  # location of management server

After entering the preceding information, save this file and exit the text editor. Do this for the machines hosting data node “A”, data node “B”, and the SQL node.

Important

Once you have started a mysqld process with the NDBCLUSTER and ndb-connectstring parameters in the [mysqld] in the my.cnf file as shown previously, you cannot execute any CREATE TABLE or ALTER TABLE statements without having actually started the cluster. Otherwise, these statements will fail with an error. This is by design.

Configuring the management node. The first step in configuring the management node is to create the directory in which the configuration file can be found and then to create the file itself. For example (running as root):

shell> mkdir /var/lib/mysql-cluster
shell> cd /var/lib/mysql-cluster
shell> vi config.ini

For our representative setup, the config.ini file should read as follows:

[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2    # Number of replicas
DataMemory=80M    # How much memory to allocate for data storage
IndexMemory=18M   # How much memory to allocate for index storage
                  # For DataMemory and IndexMemory, we have used the
                  # default values. Since the "world" database takes up
                  # only about 500KB, this should be more than enough for
                  # this example Cluster setup.

[tcp default]
# TCP/IP options:
portnumber=2202   # This the default; however, you can use any
                  # port that is free for all the hosts in the cluster
                  # Note: It is recommended that you do not specify the port 
                  # number at all and simply allow the default value to be used 
                  # instead

[ndb_mgmd]
# Management process options:
hostname=192.168.0.10           # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster  # Directory for MGM node log files

[ndbd]
# Options for data node "A":
                                # (one [ndbd] section per data node)
hostname=192.168.0.30           # Hostname or IP address
datadir=/usr/local/mysql/data   # Directory for this data node's data files

[ndbd]
# Options for data node "B":
hostname=192.168.0.40           # Hostname or IP address
datadir=/usr/local/mysql/data   # Directory for this data node's data files

[mysqld]
# SQL node options:
hostname=192.168.0.20           # Hostname or IP address
                                # (additional mysqld connections can be
                                # specified for this node for various
                                # purposes such as running ndb_restore)
Note

The world database can be downloaded from http://dev.mysql.com/doc/, where it can be found listed under “Examples”.

After all the configuration files have been created and these minimal options have been specified, you are ready to proceed with starting the cluster and verifying that all processes are running. We discuss how this is done in Section 17.2.1.3, “Initial Startup of MySQL Cluster”.

For more detailed information about the available MySQL Cluster configuration parameters and their uses, see Section 17.3.2, “MySQL Cluster Configuration Files”, and Section 17.3, “MySQL Cluster Configuration”. For configuration of MySQL Cluster as relates to making backups, see Section 17.5.3.3, “Configuration for MySQL Cluster Backups”.

Note

The default port for Cluster management nodes is 1186; the default port for data nodes is 2202. However, the cluster can automatically allocate ports for data nodes from those that are already free.

17.2.1.3. Initial Startup of MySQL Cluster

Starting the cluster is not very difficult after it has been configured. Each cluster node process must be started separately, and on the host where it resides. The management node should be started first, followed by the data nodes, and then finally by any SQL nodes:

  1. On the management host, issue the following command from the system shell to start the management node process:

    shell> ndb_mgmd -f /var/lib/mysql-cluster/config.ini
    

    Note

    ndb_mgmd must be told where to find its configuration file, using the -f or --config-file option. (See Section 17.4.4, “ndb_mgmd — The MySQL Cluster Management Server Daemon”, for details.)

    For additional options which can be used with ndb_mgmd, see Section 17.4.23, “Options Common to MySQL Cluster Programs”.

  2. On each of the data node hosts, run this command to start the ndbd process:

    shell> ndbd
    
  3. If you used RPM files to install MySQL on the cluster host where the SQL node is to reside, you can (and should) use the supplied startup script to start the MySQL server process on the SQL node.

If all has gone well, and the cluster has been set up correctly, the cluster should now be operational. You can test this by invoking the ndb_mgm management node client. The output should look like that shown here, although you might see some slight differences in the output depending upon the exact version of MySQL that you are using:

shell> ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> SHOW
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=2    @192.168.0.30  (Version: 5.1.51-ndb-6.3.39, Nodegroup: 0, Master)
id=3    @192.168.0.40  (Version: 5.1.51-ndb-6.3.39, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @192.168.0.10  (Version: 5.1.51-ndb-6.3.39)

[mysqld(API)]   1 node(s)
id=4    @192.168.0.20  (Version: 5.1.51-ndb-6.3.39)

The SQL node is referenced here as [mysqld(API)], which reflects the fact that the mysqld process is acting as a MySQL Cluster API node.

Note

The IP address shown for a given MySQL Cluster SQL or other API node in the output of SHOW is the address used by the SQL or API node to connect to the cluster data nodes, and not to any management node.

You should now be ready to work with databases, tables, and data in MySQL Cluster. See Section 17.2.3, “Loading Sample Data into MySQL Cluster and Performing Queries”, for a brief discussion.

17.2.2. Installing MySQL Cluster on Windows

Experimental support for MySQL Cluster on Microsoft Windows operating systems was introduced in MySQL Cluster NDB 7.0. Beginning with MySQL Cluster NDB 7.1.3, production support is provided for MySQL Cluster on Windows, and MySQL Cluster binaries for Windows can be obtained from http://dev.mysql.com/downloads/cluster/. For information about installing MySQL Cluster on Windows from a binary release provided by Oracle, see Section 17.2.2.1, “Installing MySQL Cluster on Windows from a Binary Release”.

It is also possible to compile and install MySQL Cluster from source on Windows using Microsoft Visual Studio. For more information, see Section 17.2.2.2, “Compiling and Installing MySQL Cluster from Source on Windows”.

17.2.2.1. Installing MySQL Cluster on Windows from a Binary Release

This section describes a basic installation of MySQL Cluster on Windows using a binary no-install MySQL Cluster release provided by Oracle, using the same 4-node setup outlined in the beginning of this section (see Section 17.2, “MySQL Cluster Multi-Computer How-To”), as shown in the following table:

NodeIP Address
Management (MGMD) node192.168.0.10
MySQL server (SQL) node192.168.0.20
Data (NDBD) node "A"192.168.0.30
Data (NDBD) node "B"192.168.0.40

As on other platforms, the MySQL Cluster host computer running an SQL node must have installed on it a MySQL Server binary (mysqld.exe). You should also have the MySQL client (mysql.exe) on on this host. For management nodes and data nodes, it is not necessary to install the MySQL Server binary; however, each management node requires the management server daemon (ndb_mgmd.exe); each data node requires the data node daemon (ndbd.exe or ndbmtd.exe). For this example, we refer to ndbd.exe as the data node executable, but you can install ndbmtd.exe, the multi-threaded version of this program, instead, in exactly the same way. You should also install the management client (ndb_mgm.exe) on the management server host. This section covers the steps necessary to install the correct Windows binaries for each type of MySQL Cluster node.

ndbmtd.exe was not included in MySQL Cluster NDB 7.1.3 binary releases for Windows, due to a problem with make_win_bin_dist. This issue was corrected in MySQL Cluster NDB 7.1.5.

Note

As with other Windows programs, MySQL Cluster executables are named with the .exe file extension. However, it is not necessary to include the .exe extension when invoking these programs from the command line. Therefore, we often simply refer to these programs in this documentation as mysqld, mysql, ndb_mgmd, and so on. You should understand that, whether we refer (for example) to mysqld or mysqld.exe, either name means the same thing (the MySQL Server program).

For setting up a MySQL Cluster using Oracles's no-install binaries, the first step in the installation process is to download the latest MySQL Cluster Windows binary archive from http://dev.mysql.com/downloads/cluster/. This archive has a filename of the form mysql-cluster-gpl-noinstall-ver-winarch.zip, where ver is the NDB storage engine version (such as 7.1.3), and arch is the architecture (32 for 32-bit binaries, and 64 for 64-bit binaries). For example, the MySQL Cluster NDB 7.1.3 no-install archive for 32-bit Windows systems is named mysql-cluster-gpl-noinstall-7.1.3-win32.zip.

You can run 32-bit MySQL Cluster binaries on both 32-bit and 64-bit versions of Windows; however, 64-bit MySQL Cluster binaries can be used only on 64-bit versions of Windows. If you are using a 32-bit version of Windows on a computer that has a 64-bit CPU, then you must use the 32-bit MySQL Cluster binaries.

To minimize the number of files that need to be downloaded from the Internet or copied between machines, we start with the computer where you intend to run the SQL node.

SQL node. We assume that you have placed a copy of the no-install archive in the directory C:\Documents and Settings\username\My Documents\Downloads on the computer having the IP address 192.168.0.20, where username is the name of the current user. (You can obtain this name using ECHO %USERNAME% on the command line.) To install and run MySQL Cluster executables as Windows services, this user should be a member of the Adminstrators group.

Extract all the files from the archive. The Extraction Wizard integrated with Windows Explorer is adequate for this task. (If you use a different archive program, be sure that it extracts all files and directories from the archive, and that it preserves the archive's directory structure.) When you are asked for a destination directory, enter C:\, which causes the Extraction Wizard to extract the archive to the directory C:\mysql-cluster-gpl-noinstall-ver-winarch. Rename this directory to C:\mysql.

It is possible to install the MySQL Cluster binaries to directories other than C:\mysql\bin; however, if you do so, you must modify the paths shown in this procedure accordingly. In particular, if the MySQL Server (SQL node) binary is installed to a location other than C:\mysql or C:\Program Files\MySQL\MySQL Server 5.1, or if the SQL node's data directory is in a location other than C:\mysql\data or C:\Program Files\MySQL\MySQL Server 5.1\data, extra configuration options must be used on the command line or added to the my.ini or my.cnf file when starting the SQL node. For more information about configuring a MySQL Server to run in a nonstandard location, see Section 2.3.5, “Installing MySQL on Microsoft Windows from a noinstall Zip Archive”.

For a MySQL Server with MySQL Cluster support to run as part of a MySQL Cluster, it must be started with the options --ndbcluster and --ndb-connectstring. While you can specify these options on the command line, it is usually more convenient to place them in an option file. To do this, create a new text file in Notepad or another text editor. Enter the following configuration information into this file:

[mysqld]
# Options for mysqld process:
ndbcluster                      # run NDB storage engine
ndb-connectstring=192.168.0.10  # location of management server

You can add other options used by this MySQL Server if desired (see Section 2.3.5.2, “Creating an Option File”), but the file must contain the options shown, at a minimum. Save this file as C:\mysql\my.ini. This completes the installation and setup for the SQL node.

Data nodes. A MySQL Cluster data node on a Windows host requires only a single executable, one of either ndbd.exe or ndbmtd.exe. For this example, we assume that you are using ndbd.exe, but the same instructions apply when using ndbmtd.exe. On each computer where you wish to run a data node (the computers having the IP addresses 192.168.0.30 and 192.168.0.40), create the directories C:\mysql, C:\mysql\bin, and C:\mysql\cluster-data; then, on the computer where you downloaded and extracted the no-install archive, locate ndbd.exe in the C:\mysql\bin directory. Copy this file to the C:\mysql\bin directory on each of the two data node hosts.

To function as part of a MySQL Cluster, each data node must be given the address or hostname of the management server. You can supply this information on the command line using the --ndb-connectstring or -c option when starting each data node process. However, it is usually preferable to put this information in an option file. To do this, create a new text file in Notepad or another text editor and enter the following text:

[mysql_cluster]
# Options for data node process:
ndb-connectstring=192.168.0.10  # location of management server

Save this file as C:\mysql\my.ini on the data node host. Create another text file containing the same information and save it on as C:mysql\my.ini on the other data node host, or copy the my.ini file from the first data node host to the second one, making sure to place the copy in the second data node's C:\mysql directory. Both data node hosts are now ready to be used in the MySQL Cluster, which leaves only the management node to be installed and configured.

Management node. The only executable program required on a computer used for hosting a MySQL Cluster management node is the management server program ndb_mgmd.exe. However, in order to administer the MySQL Cluster once it has been started, you should also install the MySQL Cluster management client program ndb_mgm.exe on the same machine as the management server. Locate these two programs on the machine where you downloaded and extracted the no-install archive; this should be the directory C:\mysql\bin on the SQL node host. Create the directory C:\mysql\bin on the computer having the IP address 192.168.0.10, then copy both programs to this directory.

You should now create two configuration files for use by ndb_mgmd.exe:

  1. A local configuration file to supply configuration data spcific to the management node itself. Typically, this file needs only to supply the location of the MySQL Cluster global configuration file (see item 2).

    To create this file, start a new text file in Notepad or another text editor, and enter the following information:

    [mysql_cluster]
    # Options for management node process
    config-file=C:/mysql/bin/config.ini
    

    Save this file as the plaintext file C:\mysql\bin\my.ini.

  2. A global configuration file from which the management node can obtain configuration information governing the MySQL Cluster as a whole. At a minimum, this file must contain a section for each node in the MySQL Cluster, and the IP addresses or hostnames for the management node and all data nodes (HostName configuration parameter). It is also advisable to include the following additional information:

    • The IP address or hostname of any SQL nodes

    • The data memory and index memory allocated to each data node (DataMemory and IndexMemory configuration parameters)

    • The number of replicas, using the NoOfReplicas configuration parameter (see Section 17.1.2, “MySQL Cluster Nodes, Node Groups, Replicas, and Partitions”)

    • The directory where each data node stores it data and log file, and the directory where the management node keeps its log files (in both cases, the DataDir configuration parameter)

    Create a new text file using a text editor such as Notepad, and input the following information:

    [ndbd default]
    # Options affecting ndbd processes on all data nodes:
    NoOfReplicas=2                      # Number of replicas
    DataDir=C:/mysql/bin/cluster-data   # Directory for each data node's data files
                                        # Forward slashes used in directory path, 
                                        # rather than backslashes. This is correct;
                                        # see Important note in text
    DataMemory=80M    # Memory allocated to data storage
    IndexMemory=18M   # Memory allocated to index storage
                      # For DataMemory and IndexMemory, we have used the
                      # default values. Since the "world" database takes up
                      # only about 500KB, this should be more than enough for
                      # this example Cluster setup.
    
    [ndb_mgmd]
    # Management process options:
    HostName=192.168.0.10               # Hostname or IP address of management node
    DataDir=C:/mysql/bin/cluster-logs   # Directory for management node log files
    
    [ndbd]
    # Options for data node "A":
                                    # (one [ndbd] section per data node)
    HostName=192.168.0.30           # Hostname or IP address
    
    [ndbd]
    # Options for data node "B":
    HostName=192.168.0.40           # Hostname or IP address
    
    [mysqld]
    # SQL node options:
    HostName=192.168.0.20           # Hostname or IP address
    

    Save this file as the plaintext file C:\mysql\bin\config.ini.

Important

A single backslash character (\) cannot be used when specifying directory paths in program options or configuration files used by MySQL Cluster on Windows. Instead, you must either escape each backslash character with a second backslash (\\), or replace the backslash with a forward slash character (/). For example, the following line from the [ndb_mgmd] section of a MySQL Cluster config.ini file does not work:

DataDir=C:\mysql\bin\cluster-logs

Instead, you may use either of the following:

DataDir=C:\\mysql\\bin\\cluster-logs  # Escaped backslashes
DataDir=C:/mysql/bin/cluster-logs     # Forward slashes

For reasons of brevity and legibility. we recommend that you use forward slashes in directory paths used in MySQL Cluster program options and configuration files on Windows.

17.2.2.2. Compiling and Installing MySQL Cluster from Source on Windows

Oracle provides precompiled MySQL Cluster binaries for Windows which should be adequate for most users. However, if you wish, it is also possible to compile MySQL Cluster for Windows from source code. The procedure for doing this is almost identical to the procedure used to compile the standard MySQL Server binaries for Windows, and uses the same tools. However, there are two major differences:

  • To build MySQL Cluster, you must use the MySQL Cluster sources, which you can obtain from http://dev.mysql.com/downloads/cluster/.

    Attempting to build MySQL Cluster from the source code for the standard MySQL Server is likely not to be successful, and is not supported by Oracle.

  • You must configure the build using the WITH_NDBCLUSTER_STORAGE_ENGINE option in addition to any other build options you wish to use before creating the Visual Studio project files. Once you have run configure.js with the desired options, you can create the project files and build from them in the same manner as you do when compiling the standard MySQL Server. For more information, see Section 2.3.9, “Installing MySQL from Source on Windows”.

Important

Currently, MySQL Cluster is not compatible with the InnoDB Plugin. You must use the version of InnoDB that is supplied with the MySQL Server. You can also build MySQL Cluster with InnoDB storage engine support on Windows using WITH_INNOBASE_STORAGE_ENGINE with configure.js.

Once the build process is complete, you can create a ZIP archive containing the compiled binaries by running make_win_bin_dist. The MySQL Cluster binaries can be found in the bin directory of the resulting archive, which is equivalent to the no-install archive, and which can be installed and configured in the same manner. For basic information about how to accomplish these tasks, see Section 17.2.2.1, “Installing MySQL Cluster on Windows from a Binary Release”.

On Windows, beginning with MySQL Cluster NDB 7.0.11, using WITH_NDBCLUSTER_STORAGE_ENGINE with configure.js causes ndbmtd.exe to be built automatically, and to be found in the bin directory of the archive created by make_win_bin_dist. (It was not possible to build ndbmtd.exe on Windows prior to MySQL Cluster NDB 7.0.11.)

17.2.2.3. Initial Startup of MySQL Cluster on Windows

Once the MySQL Cluster executables and needed configuration files are in place, performing an initial start of the cluster is simply a matter of starting the MySQL Cluster executables for all nodes in the cluster. Each cluster node process must be started separately, and on the host computer where it resides. The management node should be started first, followed by the data nodes, and then finally by any SQL nodes.

  1. On the management node host, issue the following command from the command line to start the management node process:

    C:\mysql\bin> ndb_mgmd
    2010-06-23 07:53:34 [MgmtSrvr] INFO     -- NDB Cluster Management Server. mysql-5.1.51-ndb-7.1.9
    2010-06-23 07:53:34 [MgmtSrvr] INFO     -- Reading cluster configuration from 'config.ini'
    

    The management node process continues to print logging output to the console. This is normal, because the management node is not running as a Windows service. (If you have used MySQL Cluster on a Unix-like platform such as Linux, you may notice that the management node's default behavior in this regard on Windows is effectively the opposite of its behavior on Unix systems, where it runs by default as a Unix daemon process. This behavior is also true of MySQL Cluster data node processes running on Windows.) For this reason, do not close the window in which ndb_mgmd.exe is running; doing so kills the management node process. (See Section 17.2.2.4, “Installing MySQL Cluster Processes as Windows Services”, where we show how to install and run MySQL Cluster processes as Windows services.)

    The required -f option tells the management node where to find the global configuration file (config.ini). The long form of this option is --config-file.

    Important

    A MySQL Cluster management node caches the configuration data that it reads from config.ini; once it has created a configuration cache, it ignores the config.ini file on subsequent starts unless forced to do otherwise. This means that, if the management node fails to start due to an error in this file, you must make the management node re-read config.ini after you have corrected any errors in it. You can do this by starting ndb_mgmd.exe with the --reload or --initial option on the command line. Either of these options works to refresh the configuration cache.

    It is not necessary or advisable to use either of these options in the management node's my.ini file.

    For additional information about options which can be used with ndb_mgmd, see Section 17.4.4, “ndb_mgmd — The MySQL Cluster Management Server Daemon”, as well as Section 17.4.23, “Options Common to MySQL Cluster Programs”.

  2. On each of the data node hosts, run the command shown here to start the data node processes:

    C:\mysql\bin> ndbd
    2010-06-23 07:53:46 [ndbd] INFO     -- Configuration fetched from 'localhost:1186', generation: 1
    

    In each case, the first line of output from the data node process should resemble what is hsown in the preceding example, and is followed by additional lines of logging output. As with the management node process, this is normal, because the data node is not running as a Windows service. For this reason, do not close the console window in which the data node process is running; doing so kills ndbd.exe. (For more information, see Section 17.2.2.4, “Installing MySQL Cluster Processes as Windows Services”.)

  3. Do not start the SQL node yet; it cannot connect to the cluster until the data nodes have finished starting, which may take some time. Instead, in a new console window on the management node host, start the MySQL Cluster management client ndb_mgm.exe, which should be in C:\mysql\bin on the management node host. (Do not try to re-use the console window where ndb_mgmd.exe is running by typing CTRL+C, as this kills the management node.) The resulting output should look like this:

    C:\mysql\bin> ndb_mgm
    -- NDB Cluster -- Management Client --
    ndb_mgm>                                   
    

    When the prompt ndb_mgm> appears, this indicates that the management client is ready to receive MySQL Cluster management commands. You can observe the status of the data nodes as they start by entering ALL STATUS at the management client prompt. This command causes a running report of the data nodes's startup sequence, which should look something like this:

    ndb_mgm> ALL STATUS 
    Connected to Management Server at: localhost:1186
    Node 2: starting (Last completed phase 3) (mysql-5.1.51-ndb-7.1.9)
    Node 3: starting (Last completed phase 3) (mysql-5.1.51-ndb-7.1.9)
    
    Node 2: starting (Last completed phase 4) (mysql-5.1.51-ndb-7.1.9)
    Node 3: starting (Last completed phase 4) (mysql-5.1.51-ndb-7.1.9)
    
    Node 2: Started (version 7.1.9)
    Node 3: Started (version 7.1.9)
    
    ndb_mgm>
    
    Note

    Commands issued in the management client are not case-sensitive; we use uppercase as the canonical form of these commands, but you are not required to observe this convention when inputting them into the ndb_mgm client. For more information, see Section 17.5.2, “Commands in the MySQL Cluster Management Client”.

    The output produced by ALL STATUS is likely to vary from what is shown here, according to the speed at which the data nodes are able to start, the release version number of the MySQL Cluster software you are using, and other factors. What is significant is that, when you see that both data nodes have started, you are ready to start the SQL node.

    You can leave ndb_mgm.exe running; it has no negative impact on the performance of the MySQL Cluster, and we use it in the next step to verify that the SQL node is connected to the cluster after you have started it.

  4. On the computer designated as the SQL node host, open a console window and navigate to the directory where you unpacked the MySQL Cluster binaries (if you are following our example, this is C:\mysql\bin).

    Start the SQL node by invoking mysqld.exe from the command line, as shown here:

    C:\mysql\bin> mysqld --console
    

    The --console option causes logging information to be written to the console, which can be helpful in the event of problems. (Once you are satisfied that the SQL node is running in a satisfactory manner, you can stop it and restart it out without the --console option, so that logging is performed normally.)

    In the console window where the management client (ndb_mgm.exe) is running on the management node host, enter the SHOW command, which should produce output similar to what is shown here:

    ndb_mgm> SHOW
    Connected to Management Server at: localhost:1186
    Cluster Configuration
    ---------------------
    [ndbd(NDB)]     2 node(s)
    id=2    @192.168.0.30  (Version: 5.1.51-ndb-7.1.9, Nodegroup: 0, Master)
    id=3    @192.168.0.40  (Version: 5.1.51-ndb-7.1.9, Nodegroup: 0)
    
    [ndb_mgmd(MGM)] 1 node(s)
    id=1    @192.168.0.10  (Version: 5.1.51-ndb-7.1.9)
    
    [mysqld(API)]   1 node(s)
    id=4    @192.168.0.20  (Version: 5.1.51-ndb-7.1.9)
    

    You can also verify that the SQL node is connected to the MySQL Cluster in the mysql client (mysql.exe) using the SHOW ENGINE NDB STATUS statement.

You should now be ready to work with database objects and data using MySQL Cluster's NDBCLUSTER storage engine. See Section 17.2.3, “Loading Sample Data into MySQL Cluster and Performing Queries”, for more information and examples.

Beginning with MySQL Cluster NDB 7.0.16 and MySQL Cluster NDB 7.1.5, you can install ndb_mgmd.exe, ndbd.exe, and ndbmtd.exe as Windows services. For information on how to do this, see Section 17.2.2.4, “Installing MySQL Cluster Processes as Windows Services”).

17.2.2.4. Installing MySQL Cluster Processes as Windows Services

Once you are satisfied that MySQL Cluster is running as desired, you can—beginning with MySQL Cluster NDB 7.0.16 and MySQL Cluster NDB 7.1.5—install the management nodes and data nodes as Windows services, so that these processes are started and stopped automatically whenever Windows is started or stopped. This also makes it possible to control these processes from the command line with the appropriate NET START or NET STOP command, or using the Windows graphical Services utility.

Installing programs as Windows services usually must be done using an account that has Administrator rights on the system.

To install the management node as a service on Windows, invoke ndb_mgmd.exe from the command line on the machine hosting the management node, using the --install option, as shown here:

C:\> C:\mysql\bin\ndb_mgmd.exe --install
Installing service 'MySQL Cluster Management Server' as '"C:\mysql\bin\ndbd.exe" "--service=ndb_mgmd"'
Service successfully installed. 
Important

When installing a MySQL Cluster program as a Windows service, you should always specify the complete path; otherwise the service installation may fail with the error The system cannot find the file specified.

The --install option must be used first, ahead of any other options that might be specified for ndb_mgmd.exe. However, it is preferable to specify such options in an options file instead. If your options file is not in one of the default locations as shown in the output of ndb_mgmd.exe --help, you can specify the location using the --config-file option.

Now you should be able to start and stop the management server like this:

C:\> NET START ndb_mgmd
The MySQL Cluster Management Server service is starting.
The MySQL Cluster Management Server service was started successfully.

C:\> NET STOP ndb_mgmd
The MySQL Cluster Management Server service is stopping..
The MySQL Cluster Management Server service was stopped successfully. 

You can also start or stop the management server as a Windows service using the descriptive name, as shown here:

C:\> NET START 'MySQL Cluster Management Server'
The MySQL Cluster Management Server service is starting.
The MySQL Cluster Management Server service was started successfully. 

C:\> NET STOP  'MySQL Cluster Management Server'
The MySQL Cluster Management Server service is stopping..
The MySQL Cluster Management Server service was stopped successfully. 

However, it is usually simpler to specify a short service name or to permit the default service name to be used when installing the service, and then reference that name when starting or stopping the service. To specify a service name other than ndb_mgmd, append it to the --install option, as shown in this example:

C:\> C:\mysql\bin\ndb_mgmd.exe --install=mgmd1
Installing service 'MySQL Cluster Management Server' as '"C:\mysql\bin\ndb_mgmd.exe" "--service=mgmd1"'
Service successfully installed. 

Now you should be able to start or stop the service using the name you have specified, like this:

C:\> NET START mgmd1
The MySQL Cluster Management Server service is starting.
The MySQL Cluster Management Server service was started successfully.

C:\> NET STOP mgmd1
The MySQL Cluster Management Server service is stopping..
The MySQL Cluster Management Server service was stopped successfully. 

To remove the management node service, invoke ndb_mgmd.exe with the --remove option, as shown here:

C:\> C:\mysql\bin\ndb_mgmd.exe --remove
Removing service 'MySQL Cluster Management Server'
Service successfully removed. 

If you installed the service using a service name other than the default, you can remove the service by passing this name as the value of the the --remove option, like this:

C:\> C:\mysql\bin\ndb_mgmd.exe --remove=mgmd1
Removing service 'mgmd1'
Service successfully removed. 

Installation of a MySQL Cluster data node processs as a Windows service can be done in a similar fashion, using the --install option for ndbd.exe (or ndbmtd.exe), as shown here:

C:\> C:\mysql\bin\ndbd.exe --install
Installing service 'MySQL Cluster Data Node Daemon' as '"C:\mysql\bin\ndbd.exe" "--service=ndbd"'
Service successfully installed. 

Now you can start or stop the data node using either the default service name or the descriptive name with net start or net stop, as shown in the following example:

C:\> NET START ndbd
The MySQL Cluster Data Node Daemon service is starting.
The MySQL Cluster Data Node Daemon service was started successfully.

C:\> NET STOP ndbd
The MySQL Cluster Data Node Daemon service is stopping..
The MySQL Cluster Data Node Daemon service was stopped successfully.

C:\> NET START 'MySQL Cluster Data Node Daemon'
The MySQL Cluster Data Node Daemon service is starting.
The MySQL Cluster Data Node Daemon service was started successfully.

C:\> NET STOP 'MySQL Cluster Data Node Daemon'
The MySQL Cluster Data Node Daemon service is stopping..
The MySQL Cluster Data Node Daemon service was stopped successfully.

To remove the data node service, invoke ndbd.exe with the --remove option, as shown here:

C:\> C:\mysql\bin\ndbd.exe --remove
Removing service 'MySQL Cluster Data Node Daemon'
Service successfully removed. 

As with ndb_mgmd.exe (and mysqld.exe), when installing ndbd.exe as a Windows service, you can also specify a name for the service as the value of --install, and then use it when starting or stopping the service, like this:

C:\> C:\mysql\bin\ndbd.exe --install=dnode1
Installing service 'dnode1' as '"C:\mysql\bin\ndbd.exe" "--service=dnode1"'
Service successfully installed.

C:\> NET START dnode1
The MySQL Cluster Data Node Daemon service is starting.
The MySQL Cluster Data Node Daemon service was started successfully.

C:\> NET STOP dnode1
The MySQL Cluster Data Node Daemon service is stopping..
The MySQL Cluster Data Node Daemon service was stopped successfully.

If you specified a service name when installing the data node service, you can use this name when removing it as well, by passing it as the value of the --remove option, as shown here:

C:\> C:\mysql\bin\ndbd.exe --remove=dnode1
Removing service 'dnode1'
Service successfully removed. 

Installation of the SQL node as a Windows service, starting the service, stopping the service, and removing the service are done in a similar fashion, using mysqld --install, NET START, NET STOP, and mysqld --remove. For additional information, see Section 2.3.5.7, “Starting MySQL Server as a Microsoft Windows Service”.

17.2.3. Loading Sample Data into MySQL Cluster and Performing Queries

Note

The information in this section applies to MySQL Cluster running on both Unix and Windows platforms.

Working with data in MySQL Cluster is not much different from doing so in MySQL without clustering support. There are two points to keep in mind:

  • For a table to be replicated in the cluster, it must use the NDBCLUSTER storage engine. To specify this, use the ENGINE=NDBCLUSTER or ENGINE=NDB option when creating the table:

    CREATE TABLE tbl_name (col_name column_definitions) ENGINE=NDBCLUSTER;
    

    Alternatively, for an existing table that uses a different storage engine, use ALTER TABLE to change the table to use NDBCLUSTER:

    ALTER TABLE tbl_name ENGINE=NDBCLUSTER;
    
  • Each NDBCLUSTER table must have a primary key. If no primary key is defined by the user when a table is created, the NDBCLUSTER storage engine automatically generates a hidden one.

    Note

    This hidden key takes up space just as does any other table index. It is not uncommon to encounter problems due to insufficient memory for accommodating these automatically created indexes.)

If you are importing tables from an existing database using the output of mysqldump, you can open the SQL script in a text editor and add the ENGINE option to any table creation statements, or replace any existing ENGINE options. Suppose that you have the world sample database on another MySQL server that does not support MySQL Cluster, and you want to export the City table:

shell> mysqldump --add-drop-table world City > city_table.sql

The resulting city_table.sql file will contain this table creation statement (and the INSERT statements necessary to import the table data):

DROP TABLE IF EXISTS `City`;
CREATE TABLE `City` (
  `ID` int(11) NOT NULL auto_increment,
  `Name` char(35) NOT NULL default '',
  `CountryCode` char(3) NOT NULL default '',
  `District` char(20) NOT NULL default '',
  `Population` int(11) NOT NULL default '0',
  PRIMARY KEY  (`ID`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

INSERT INTO `City` VALUES (1,'Kabul','AFG','Kabol',1780000);
INSERT INTO `City` VALUES (2,'Qandahar','AFG','Qandahar',237500);
INSERT INTO `City` VALUES (3,'Herat','AFG','Herat',186800);
(remaining INSERT statements omitted)

You need to make sure that MySQL uses the NDBCLUSTER storage engine for this table. There are two ways that this can be accomplished. One of these is to modify the table definition before importing it into the Cluster database. Using the City table as an example, modify the ENGINE option of the definition as follows:

DROP TABLE IF EXISTS `City`;
CREATE TABLE `City` (
  `ID` int(11) NOT NULL auto_increment,
  `Name` char(35) NOT NULL default '',
  `CountryCode` char(3) NOT NULL default '',
  `District` char(20) NOT NULL default '',
  `Population` int(11) NOT NULL default '0',
  PRIMARY KEY  (`ID`)
) ENGINE=NDBCLUSTER DEFAULT CHARSET=latin1;

INSERT INTO `City` VALUES (1,'Kabul','AFG','Kabol',1780000);
INSERT INTO `City` VALUES (2,'Qandahar','AFG','Qandahar',237500);
INSERT INTO `City` VALUES (3,'Herat','AFG','Herat',186800);
(remaining INSERT statements omitted)

This must be done for the definition of each table that is to be part of the clustered database. The easiest way to accomplish this is to do a search-and-replace on the file that contains the definitions and replace all instances of TYPE=engine_name or ENGINE=engine_name with ENGINE=NDBCLUSTER. If you do not want to modify the file, you can use the unmodified file to create the tables, and then use ALTER TABLE to change their storage engine. The particulars are given later in this section.

Assuming that you have already created a database named world on the SQL node of the cluster, you can then use the mysql command-line client to read city_table.sql, and create and populate the corresponding table in the usual manner:

shell> mysql world < city_table.sql

It is very important to keep in mind that the preceding command must be executed on the host where the SQL node is running (in this case, on the machine with the IP address 192.168.0.20).

To create a copy of the entire world database on the SQL node, use mysqldump on the noncluster server to export the database to a file named world.sql; for example, in the /tmp directory. Then modify the table definitions as just described and import the file into the SQL node of the cluster like this:

shell> mysql world < /tmp/world.sql

If you save the file to a different location, adjust the preceding instructions accordingly.

Running SELECT queries on the SQL node is no different from running them on any other instance of a MySQL server. To run queries from the command line, you first need to log in to the MySQL Monitor in the usual way (specify the root password at the Enter password: prompt):

shell> mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1 to server version: 5.1.51-ndb-6.2.19

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

We simply use the MySQL server's root account and assume that you have followed the standard security precautions for installing a MySQL server, including setting a strong root password. For more information, see Section 2.12.2, “Securing the Initial MySQL Accounts”.

It is worth taking into account that Cluster nodes do not make use of the MySQL privilege system when accessing one another. Setting or changing MySQL user accounts (including the root account) effects only applications that access the SQL node, not interaction between nodes. See Section 17.5.9.2, “MySQL Cluster and MySQL Privileges”, for more information.

If you did not modify the ENGINE clauses in the table definitions prior to importing the SQL script, you should run the following statements at this point:

mysql> USE world;
mysql> ALTER TABLE City ENGINE=NDBCLUSTER;
mysql> ALTER TABLE Country ENGINE=NDBCLUSTER;
mysql> ALTER TABLE CountryLanguage ENGINE=NDBCLUSTER;

Selecting a database and running a SELECT query against a table in that database is also accomplished in the usual manner, as is exiting the MySQL Monitor:

mysql> USE world;
mysql> SELECT Name, Population FROM City ORDER BY Population DESC LIMIT 5;
+-----------+------------+
| Name      | Population |
+-----------+------------+
| Bombay    |   10500000 |
| Seoul     |    9981619 |
| São Paulo |    9968485 |
| Shanghai  |    9696300 |
| Jakarta   |    9604900 |
+-----------+------------+
5 rows in set (0.34 sec)

mysql> \q
Bye

shell>

Applications that use MySQL can employ standard APIs to access NDB tables. It is important to remember that your application must access the SQL node, and not the management or data nodes. This brief example shows how we might execute the SELECT statement just shown by using the PHP 5.X mysqli extension running on a Web server elsewhere on the network:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
  "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
  <meta http-equiv="Content-Type"
           content="text/html; charset=iso-8859-1">
  <title>SIMPLE mysqli SELECT</title>
</head>
<body>
<?php
  # connect to SQL node:
  $link = new mysqli('192.168.0.20', 'root', 'root_password', 'world');
  # parameters for mysqli constructor are:
  #   host, user, password, database

  if( mysqli_connect_errno() )
    die("Connect failed: " . mysqli_connect_error());

  $query = "SELECT Name, Population
            FROM City
            ORDER BY Population DESC
            LIMIT 5";

  # if no errors...
  if( $result = $link->query($query) )
  {
?>
<table border="1" width="40%" cellpadding="4" cellspacing ="1">
  <tbody>
  <tr>
    <th width="10%">City</th>
    <th>Population</th>
  </tr>
<?
    # then display the results...
    while($row = $result->fetch_object())
      printf("<tr>\n  <td align=\"center\">%s</td><td>%d</td>\n</tr>\n",
              $row->Name, $row->Population);
?>
  </tbody
</table>
<?
  # ...and verify the number of rows that were retrieved
    printf("<p>Affected rows: %d</p>\n", $link->affected_rows);
  }
  else
    # otherwise, tell us what went wrong
    echo mysqli_error();

  # free the result set and the mysqli connection object
  $result->close();
  $link->close();
?>
</body>
</html>

We assume that the process running on the Web server can reach the IP address of the SQL node.

In a similar fashion, you can use the MySQL C API, Perl-DBI, Python-mysql, or MySQL Connectors to perform the tasks of data definition and manipulation just as you would normally with MySQL.

17.2.4. Safe Shutdown and Restart of MySQL Cluster

To shut down the cluster, enter the following command in a shell on the machine hosting the management node:

shell> ndb_mgm -e shutdown

The -e option here is used to pass a command to the ndb_mgm client from the shell. (See Section 17.4.23, “Options Common to MySQL Cluster Programs”, for more information about this option.) The command causes the ndb_mgm, ndb_mgmd, and any ndbd or ndbmtd processes to terminate gracefully. Any SQL nodes can be terminated using mysqladmin shutdown and other means. On Windows platforms, assuming that you have installed the SQL node as a Windows service, you can use NET STOP MYSQL.

To restart the cluster on Unix platforms, run these commands:

  • On the management host (192.168.0.10 in our example setup):

    shell> ndb_mgmd -f /var/lib/mysql-cluster/config.ini
    
  • On each of the data node hosts (192.168.0.30 and 192.168.0.40):

    shell> ndbd
    
  • Use the ndb_mgm client to verify that both data nodes have started successfully.

  • On the SQL host (192.168.0.20):

    shell> mysqld_safe &
    

On Windows platforms, assuming that you have installed all MySQL Cluster processes as Windows services using the default service names (see Section 17.2.2.4, “Installing MySQL Cluster Processes as Windows Services”), you can restart the cluster as follows:

  • On the management host (192.168.0.10 in our example setup), execute the following command:

    C:\> NET START ndb_mgmd
    
  • On each of the data node hosts (192.168.0.30 and 192.168.0.40), execute the following command:

    C:\> NET START ndbd
    
  • On the management node host, use the ndb_mgm client to verify that the management node and both data nodes have started successfully (see Section 17.2.2.3, “Initial Startup of MySQL Cluster on Windows”).

  • On the SQL node host (192.168.0.20), execute the following command:

    C:\> NET START mysql
    

In a production setting, it is usually not desirable to shut down the cluster completely. In many cases, even when making configuration changes, or performing upgrades to the cluster hardware or software (or both), which require shutting down individual host machines, it is possible to do so without shutting down the cluster as a whole by performing a rolling restart of the cluster. For more information about doing this, see Section 17.2.5.1, “Performing a Rolling Restart of a MySQL Cluster”.

17.2.5. Upgrading and Downgrading MySQL Cluster

This portion of the MySQL Cluster chapter covers upgrading and downgrading a MySQL Cluster from one MySQL release to another. It discusses different types of Cluster upgrades and downgrades, and provides a Cluster upgrade/downgrade compatibility matrix (see Section 17.2.5.2, “MySQL Cluster 5.1 and MySQL Cluster NDB 6.x/7.x Upgrade and Downgrade Compatibility”). You are expected already to be familiar with installing and configuring a MySQL Cluster prior to attempting an upgrade or downgrade. See Section 17.3, “MySQL Cluster Configuration”.

For more information about upgrading or downgrading between MySQL Cluster NDB releases, or between MySQL Cluster NDB releases and mainline MySQL releases, see the changelogs relating to the applicable MySQL Cluster versions.

This section remains in development, and continues to be updated and expanded.

17.2.5.1. Performing a Rolling Restart of a MySQL Cluster

This section discusses how to perform a rolling restart of a MySQL Cluster installation, so called because it involves stopping and starting (or restarting) each node in turn, so that the cluster itself remains operational. This is often done as part of a rolling upgrade or rolling downgrade, where high availability of the cluster is mandatory and no downtime of the cluster as a whole is permissible. Where we refer to upgrades, the information provided here also generally applies to downgrades as well.

There are a number of reasons why a rolling restart might be desirable. These are described in the next few paragraphs.

Cluster configuration change. To make a change in the cluster's configuration, such as adding an SQL node to the cluster, or setting a configuration parameter to a new value.

Cluster software upgrade/downgrade. To upgrade the cluster to a newer version of the MySQL Cluster software (or to downgrade it to an older version). This is usually referred to as a “rolling upgrade” (or “rolling downgrade”, when reverting to an older version of MySQL Cluster).

Change on node host. To make changes in the hardware or operating system on which one or more cluster nodes are running.

Cluster reset. To reset the cluster because it has reached an undesirable state. In such cases it is often desirable to reload the data and metadata of one or more data nodes. This can be done 1 of 3 ways:

Freeing of resources. To permit memory allocated to a table by successive INSERT and DELETE operations to be freed for re-use by other MySQL Cluster tables.

The process for performing a rolling restart may be generalized as follows:

  1. Stop all cluster management nodes (ndb_mgmd processes), reconfigure them, then restart them.

  2. Stop, reconfigure, then restart each cluster data node (ndbd process) in turn.

  3. Stop, reconfigure, then restart each cluster SQL node (mysqld process) in turn.

The specifics for implementing a particular rolling upgrade depend upon the actual changes being made. A more detailed view of the process is presented here:

MySQL Cluster Rolling Restarts (By
          Type)

In the previous diagram, the Stop and Start steps indicate that the process must be stopped completely using a shell command (such as kill on most Unix systems) or the management client STOP command, then started again from a system shell by invoking the ndbd or ndb_mgmd executable as appropriate. Restart indicates the process may be restarted using the ndb_mgm management client RESTART command.

Prior to MySQL Cluster NDB 6.3.29 and MySQL Cluster NDB 7.0.10. When performing an upgrade or downgrade of the cluster software, you must upgrade or downgrade the management nodes first, then the data nodes, and finally the SQL nodes. Doing so in any other order may leave the cluster in an unusable state.

MySQL Cluster NDB 6.3.29 and later; MySQL Cluster NDB 7.0.10 and later. MySQL Cluster supports a more flexible order for upgrading the cluster nodes. When upgrading a cluster running MySQL Cluster NDB 6.3.29 or later, or a cluster that is running MySQL Cluster NDB 7.0.10 or later, you may upgrade API nodes (including SQL nodes) before upgrading the management nodes, data nodes, or both. In other words, you are permitted to upgrade the API and SQL nodes in any order. This is subject to the following provisions:

  • This functionality is intended for use as part of an online upgrade only. A mix of node binaries from different MySQL Cluster releases is neither intended nor supported for constant, long-term use in a production setting.

  • All management nodes must be upgraded before any data nodes are upgraded. This remains true regardless of the order in which you upgrade the cluster's API and SQL nodes.

  • For MySQL Cluster NDB 6.3, the ability to upgrade API nodes in any order relative to upgading management nodes and data nodes is supported only for MySQL Cluster NDB 6.3.29 and later; for MySQL Cluster NDB 7.0, it supported only for MySQL Cluster NDB 7.0.10 and later. This means that, if you are upgrading from a MySQL Cluster NDB 6.3 release to a MySQL Cluster NDB 7.0 release, the “oldNDB engine version must be 6.3.29 or later, and the “newNDB engine version must be 7.0.10 or later.

  • When upgrading the cluster from a MySQL Cluster NDB 6.3 release to a MySQL Cluster NDB 7.0 release: Once you have started upgrading the API nodes, you should not perform DDL operations until all management nodes and data nodes have been upgraded. DML operations should be unaffected, and can continue while the upgrade is in progress.

    However, it is possible to perform DDL from an “old” (NDB 6.3 version) API node as long as the master data node is also running the “old” version of MySQL Cluster. You should keep in mind that a data node restart could result in the master node running a “new” (NDB 7.0 version) binary while one or more data nodes are still using the “old” (NDB 6.3) version; in this situation, no DDL can be performed from any API node, because the master data node is no longer using an NDB 6.3 binary, but the cluster still contains nodes which are not yet using NDB 7.0. For this reason, we recommend that you avoid performing DDL at any time while the upgrade is in progress.

  • Features specific to the “new” version must not be used until all management nodes and data nodes have been upgraded.

    This also applies to any MySQL Server version change that may apply, in addition to the NDB engine version change, so do not forget to take this into account when planning the upgrade. (This is true for online upgrades of MySQL Cluster in general.)

See also Bug#48528 and Bug#49163.

17.2.5.2. MySQL Cluster 5.1 and MySQL Cluster NDB 6.x/7.x Upgrade and Downgrade Compatibility

This section provides information about MySQL Cluster software and table file compatibility between MySQL 5.1.23 and earlier, MySQL Cluster NDB 6.x, and MySQL Cluster NDB 7.x releases with regard to performing upgrades and downgrades.

For information about upgrades and downgrades to, from, and between different releases of MySQL Cluster NDB 7.0 and MySQL Cluster NDB 7.1, see Upgrade and downgrade compatibility: MySQL Cluster NDB 7.x.

For information about upgrades and downgrades to, from, and between different releases of MySQL Cluster NDB 6.1, MySQL Cluster NDB 6.2, and MySQL Cluster NDB 6.3, see Upgrade and downgrade compatibility: MySQL Cluster NDB 6.x.

For information about upgrades and downgrades to, from, and between different editions of MySQL Cluster as found in standard MySQL 5.1.23 and earlier releases, see Upgrade and downgrade compatibility: MySQL 5.1.

Important

Only compatibility between MySQL versions with regard to NDBCLUSTER is taken into account in this section, and there are likely other issues to be considered. As with any other MySQL software upgrade or downgrade, you are strongly encouraged to review the relevant portions of the MySQL Manual for the MySQL versions from which and to which you intend to migrate, before attempting an upgrade or downgrade of the MySQL Cluster software. See Section 2.13.1, “Upgrading MySQL”.

Upgrade and downgrade compatibility: MySQL Cluster NDB 7.x. The table shown here provides information on MySQL Cluster upgrade and downgrade compatibility among different releases of MySQL Cluster NDB 7.0 and 7.1. Additional notes about upgrades and downgrades to, from, or within the MySQL Cluster NDB 7.x release series can be found immediately following the table.

MySQL Cluster upgrade/downgrade
          compatibility, MySQL Cluster NDB 7.x
Notes: MySQL Cluster NDB 7.x
  • Online upgrades from any MySQL Cluster NDB 7.0 release up to and including MySQL Cluster NDB 7.0.4 (as well as all early releases numbered NDB 6.4.x) to MySQL Cluster NDB 7.0.5 or later are not possible. Upgrades to MySQL Cluster NDB 7.0.6 or later from MySQL Cluster NDB 6.3.8 or a later MySQL Cluster NDB 6.3 release, or from MySQL Cluster NDB 7.0.5 or later, are supported. (Bug#44294)

    When upgrading online from a MySQL Cluster NDB 6.3 release to a MySQL Cluster NDB 7.0 release, you should not try to upgrade the data nodes from ndbd to ndbmtd at the same time. Instead, perform the upgrade using the new ndbd executable (from the MySQL Cluster NDB 7.0.x distribution to which you are upgrading) to replace the one in use on the data nodes. Once the version upgrade is complete, you can perform a second (online) upgrade to replace the data node executables with ndbmtd from the MySQL Cluster NDB 7.0.x distribution.

  • In MySQL Cluster NDB 7.0.4, the default values for a number of MySQL Cluster configuration parameters relating to memory usage and buffering changed (see Section 17.7.2.26, “Changes in MySQL Cluster NDB 7.0.4 (5.1.32-ndb-7.0.4) (18 March 2008)”, for a list of the parameters whose defaults changed). For this reason, you may encounter issues if you try to use a configuration that does not explicitly define each of these buffers (because it was developed for a previous version of MySQL Cluster, SendBufferMemory and ReceiveBufferMemory in particular.

  • Prior to MySQL Cluster NDB 7.0.7, DML statements failed if executed while performing an online upgrade from a MySQL Cluster NDB 6.3 release. (Bug#45917)

  • Following an upgrade from any MySQL Cluster NDB 6.3.x release to MySQL Cluster NDB 7.0.6, DDL and backup operations failed. This issue was resolved in MySQL Cluster NDB 7.0.7. (Bug#46494, Bug#46563)

  • In some cases, there could be problems with online upgrades from MySQL Cluster NDB 6.3 releases to MySQL Cluster NDB 7.0 releases due to a previous change in the signalling format used between nodes. This issue was corrected in MySQL Cluster NDB 7.0.9.

  • Once an NDB table had an ALTER ONLINE TABLE operation performed on it using a MySQL Cluster NDB 6.3.x release, it could not be upgraded online to MySQL Cluster NDB 7.0. This issue was resolved in MySQL Cluster NDB 7.0.8. (See Bug#47542.)

  • Following an upgrade from MySQL Cluster NDB 6.3 to MySQL Cluster NDB 7.0, if there were any tables having unique indexes prior to the upgrade, attempts to create unique indexes failed. This could also occur when performing offline ALTER TABLE operations on tables having indexes that were not dropped as a result of the ALTER TABLE. This issue was due to a change in the way that NDB tracked unique indexes internally, and was resolved in MySQL Cluster NDB 7.0.9. (Bug#48416)

    Workaround. For upgrades to MySQL Cluster NDB 7.0 releases prior to version 7.0.9, a workaround is available: Following the upgrade, perform a second rolling restart of the cluster before before performing any ALTER TABLE operations involving indexes.

  • Due to an issue discovered after the release of MySQL Cluster NDB 7.0.10 (Bug#50433), it is not possible to perform an online upgrade from MySQL Cluster NDB 7.0.9b and earlier MySQL Cluster NDB 7.0 releases to MySQL Cluster NDB 7.0.10. Instead, you should upgrade your MySQL Cluster NDB 7.0 installation directly to MySQL Cluster NDB 7.0.11 or later.

    This issue did not appear to affect MySQL Cluster NDB 6.3, and it should be possible to upgrade online from MySQL Cluster NDB 6.3 to MySQL Cluster NDB 7.0.10 without any problems other than those noted previously.

  • It was not possible to perform an online upgrade from a MySQL Cluster NDB 6.3 or 7.0 release to MySQL Cluster NDB 7.1.0 or 7.1.1. This issue was fixed in MySQL Cluster NDB 7.1.2 (see Bug#51429).

  • Following an upgrade to MySQL Cluster NDB 7.0.15 or later (MySQL Cluster NDB 7.0), or to MySQL Cluster NDB 7.1.4 or later (MySQL Cluster NDB 7.1), a table created in a previous version of MySQL Cluster does not automatically support NDB-native default values. Such a table continues to use default values supplied by the MySQL server until it is upgraded by performing an offline ALTER TABLE on it.

    When upgrading to a MySQL Cluster release that supports native default values from a MySQL Cluster release that does not, you should not attempt to create any new tables until all data nodes are using the new ndbd or ndbmtd binaries. This is because the older binaries do not provide support for native default values.

    Important

    Tables created with native default value support cannot be used with versions of MySQL Cluster that do not support native default values.

Upgrade and downgrade compatibility: MySQL Cluster NDB 6.x. The table shown here provides information on MySQL Cluster upgrade and downgrade compatibility among different releases of MySQL Cluster NDB 6.1, 6.2, and 6.3. Additional notes about upgrades and downgrades to, from, or within the MySQL Cluster NDB 6.x release series can be found immediately following the table.

MySQL Cluster upgrade/downgrade
          compatibility, MySQL Cluster NDB 6.x
Notes: MySQL Cluster NDB 6.x
  • MySQL Cluster NDB 6.1 is no longer in production; if you are still using a MySQL Cluster NDB 6.1 release, you should upgrade to the most recent MySQL Cluster NDB 7.0 or MySQL Cluster NDB 7.1 release as soon as possible.

  • It is not possible to upgrade from MySQL Cluster NDB 6.1.2 (or an older 6.1 release) directly to 6.1.4 or a newer NDB 6.1 release, or to downgrade from 6.1.4 (or a newer 6.1 release) directly to 6.1.2 or an older NDB 6.1 release; in either case, you must upgrade or downgrade to MySQL Cluster NDB 6.1.3 first.

  • It is not possible to perform an online downgrade from MySQL Cluster NDB 6.1.8 (or a newer 6.1 release) to MySQL Cluster NDB 6.1.7 (or an older 6.1 release).

  • MySQL Cluster NDB 6.1.6 and 6.1.18 were not released.

  • It is not possible to perform an online upgrade or downgrade between MySQL Cluster NDB 6.2 and any previous release series (including mainline MySQL 5.1 and MySQL Cluster NDB 6.1); it is necessary to perform a dump and reload. However, it should be possible to perform online upgrades or downgrades between any MySQL Cluster NDB 6.2 release and any MySQL Cluster NDB 6.3 release up to and including 6.3.7.

  • The internal specifications for columns in NDB tables changed in MySQL Cluster NDB 6.1.17 and 6.2.1 to enable compatibility with future MySQL Cluster releases that are expected to implement online adding and dropping of columns. This change is not backward-compatible with earlier MySQL or MySQL Cluster NDB 6.x versions.

    To make tables created in earlier versions compatible with online adding and dropping of columns in later versions, it is necessary to force MySQL Cluster to convert the tables to the new format by following this procedure following an upgrade:

    1. Upgrade the MySQL Cluster software on all data, management, and SQL nodes

    2. Back up all NDB tables

    3. Shut down the cluster (all data, management, and SQL nodes)

    4. Restart the cluster, starting all data nodes with the --initial option (to clear and rebuild the data node file systems)

    5. Restore the tables from backup

    To minimize possible later difficulties, it is strongly advised that the procedure outlined above be followed as soon as possible after to upgrading between the versions indicated. The procedure is not necessary for NDBCLUSTER tables created in any of the following versions:

    • MySQL Cluster NDB 6.1.8 or a later MySQL Cluster NDB 6.1 release

    • MySQL Cluster 6.2.1 or a later MySQL Cluster NDB 6.2 release

    • Any MySQL Cluster NDB 6.3 release

    Tables created in the listed versions (or later ones, as indicated) are already compatible with online adding and dropping of columns (as implemented beginning with MySQL Cluster NDB 6.2.5 and MySQL Cluster NDB 6.3.2).

  • It was not possible to perform an online upgrade between any MySQL Cluster NDB 6.2 release and MySQL Cluster NDB 6.3.8 and later MySQL Cluster 6.3 releases. This issue was fixed in MySQL Cluster NDB 6.3.21. (Bug#41435)

  • Online downgrades between MySQL Cluster NDB 6.2.5 and earlier releases are not supported.

  • Online downgrades between MySQL Cluster NDB 6.3.8 and earlier releases are not supported.

Upgrade and downgrade compatibility: MySQL 5.1. The table shown here provides information on MySQL Cluster upgrade and downgrade compatibility among different releases of MySQL 5.1 prior to MySQL 5.1.24. Additional notes about upgrades and downgrades to, from, or within the MySQL 5.1 release series can be found immediately following the table.

MySQL Cluster upgrade/downgrade
          compatibility, MySQL 5.1
Notes: MySQL 5.1
  • MySQL 5.1.3 was the first public release in this series.

  • Direct upgrades or downgrades between MySQL Cluster 5.0 and 5.1 are not supported; you must dump all NDBCLUSTER tables using mysqldump, install the new version of the software, and then reload the tables from the dump.

  • You cannot downgrade a MySQL Cluster based on MySQL 5.1.6 or later and using Disk Data tables to MySQL 5.1.5 or earlier unless you convert all such tables to in-memory NDB tables first.

  • MySQL 5.1.8, MySQL 5.1.10, and MySQL 5.1.13 were not released.

  • Online cluster upgrades and downgrades between MySQL 5.1.11 (or an earlier version) and 5.1.12 (or a later version) are not possible due to major changes in the cluster file system. In such cases, you must perform a backup or dump, upgrade (or downgrade) the software, start each data node with --initial, and then restore from the backup or dump. You can use NDB backup/restore or mysqldump for this purpose.

  • Online downgrades from MySQL 5.1.14 or later to versions previous to 5.1.14 are not supported due to incompatible changes in the cluster system tables.

  • Online upgrades from MySQL 5.1.17 and earlier to 5.1.18 and later MySQL 5.1.x releases are not supported for clusters using replication due to incompatible changes in the mysql.ndb_apply_status table. (Online upgrades from MySQL 5.1 to MySQL Cluster NDB 6.2 and later are not supported, as discussed elsewhere in this section.) However, it should not be necessary to shut down the cluster entirely, if you follow this modified rolling restart procedure:

    Note

    You should upgrade the MySQL Cluster software on each node using the same method by which it was originally installed. See Section 17.2.1.1, “MySQL Cluster Multi-Computer Installation”, for more information.

    1. Stop the management server, update the management server software, then start the management server again. For multiple management servers, repeat this step for each management server in turn.

    2. For each data node in turn: Stop the data node, update the data node daemon (in MySQL Cluster NDB 7.0 and later, this can be either ndbd or ndbmtd) with the new version, then restart the data node. It should not be necessary to use --initial when restarting any of the data nodes after updating the software.

    3. Stop all SQL nodes. Upgrade the existing MySQL server installations to the new version on all SQL nodes, then restart them. It is not necessary to start them one at a time after upgrading the MySQL server software, but there must be a time when none of them is running before starting any of them again using the 5.1.18 (or later) mysqld. Otherwise—due to the fact that mysql.ndb_apply_status uses the NDB storage engine and is thus shared between all SQL nodes—there may be conflicts between the old and new versions of the table on different SQL nodes.

      You can find more information about the changes to ndb_apply_status in Section 17.6.4, “MySQL Cluster Replication Schema and Tables”.

    As with any other MySQL Cluster version upgrade, you should also update the MySQL Cluster management client (ndb_mgm) and other MySQL Cluster client programs such as ndb_config and ndb_error_reporter; however, this does not have to be done in any particular order.

  • The internal specifications for columns in NDBCLUSTER tables changed in MySQL 5.1.18 to enable compatibility with later MySQL Cluster releases that permit online adding and dropping of columns. This change is not backward-compatible with earlier MySQL versions.

    To make tables created in MySQL 5.1.17 and earlier compatible with online adding and dropping of columns (available beginning with beginning with MySQL Cluster NDB 6.2.5 and MySQL Cluster NDB 6.3.3—see Section 12.1.7, “ALTER TABLE Syntax”, for more information), it is necessary to force MySQL 5.1.18 and later to convert the tables to the new format by following this procedure:

    1. Back up all NDBCLUSTER tables.

    2. Upgrade the MySQL Cluster software on all data, management, and SQL nodes.

    3. Shut down the cluster completely (this includes all data, management, and API or SQL nodes).

    4. Restart the cluster, starting all data nodes with the --initial option (to clear and rebuild the data node file systems).

    5. Restore the NDBCLUSTER tables from backup.

    This is not necessary for NDBCLUSTER tables created in MySQL 5.1.18 and later; such tables will automatically be compatible with online adding and dropping of columns (as implemented beginning with MySQL Cluster NDB 6.2.5 and MySQL Cluster NDB 6.3.2).

    To minimize possible later difficulties, it is strongly advised that the procedure outlined above be followed as soon as possible after to upgrading from MySQL 5.1.17 or earlier to MySQL 5.1.18 or later.

    Information about how this change effects users of MySQL Cluster NDB 6.x and 7.x is provided later in this section.

  • MySQL Cluster is not supported in standard MySQL 5.1 releases, beginning with MySQL 5.1.24. If you are using MySQL Cluster in a standard MySQL 5.1 release, you should upgrade to the most recent MySQL Cluster NDB 6.2 or 6.3 release.

17.3. MySQL Cluster Configuration

A MySQL server that is part of a MySQL Cluster differs in one chief respect from a normal (nonclustered) MySQL server, in that it employs the NDBCLUSTER storage engine. This engine is also referred to simply as NDB, and the two forms of the name are synonymous.

To avoid unnecessary allocation of resources, the server is configured by default with the NDB storage engine disabled. To enable NDB, you must modify the server's my.cnf configuration file, or start the server with the --ndbcluster option.

For more information about --ndbcluster and other MySQL server options specific to MySQL Cluster, see Section 17.3.4.2, “mysqld Command Options for MySQL Cluster”.

The MySQL server is a part of the cluster, so it also must know how to access an MGM node to obtain the cluster configuration data. The default behavior is to look for the MGM node on localhost. However, should you need to specify that its location is elsewhere, this can be done in my.cnf or on the MySQL server command line. Before the NDB storage engine can be used, at least one MGM node must be operational, as well as any desired data nodes.

NDB, the MySQL Cluster storage engine, is available in binary distributions for Linux, Mac OS X, Solaris. and Windows. We are working to support MySQL Cluster on all operating systems supported by the MySQL Server. For information about installing MySQL Cluster, see Section 17.2.1.1, “MySQL Cluster Multi-Computer Installation”.

17.3.1. Quick Test Setup of MySQL Cluster

To familiarize you with the basics, we will describe the simplest possible configuration for a functional MySQL Cluster. After this, you should be able to design your desired setup from the information provided in the other relevant sections of this chapter.

First, you need to create a configuration directory such as /var/lib/mysql-cluster, by executing the following command as the system root user:

shell> mkdir /var/lib/mysql-cluster

In this directory, create a file named config.ini that contains the following information. Substitute appropriate values for HostName and DataDir as necessary for your system.

# file "config.ini" - showing minimal setup consisting of 1 data node,
# 1 management server, and 3 MySQL servers.
# The empty default sections are not required, and are shown only for
# the sake of completeness.
# Data nodes must provide a hostname but MySQL Servers are not required
# to do so.
# If you don't know the hostname for your machine, use localhost.
# The DataDir parameter also has a default value, but it is recommended to
# set it explicitly.
# Note: [db], [api], and [mgm] are aliases for [ndbd], [mysqld], and [ndb_mgmd],
# respectively. [db] is deprecated and should not be used in new installations.

[ndbd default]
NoOfReplicas= 1

[mysqld  default]
[ndb_mgmd default]
[tcp default]

[ndb_mgmd]
HostName= myhost.example.com

[ndbd]
HostName= myhost.example.com
DataDir= /var/lib/mysql-cluster

[mysqld]
[mysqld]
[mysqld]

You can now start the ndb_mgmd management server. By default, it attempts to read the config.ini file in its current working directory, so change location into the directory where the file is located and then invoke ndb_mgmd:

shell> cd /var/lib/mysql-cluster
shell> ndb_mgmd

Then start a single data node by running ndbd:

shell> ndbd

For command-line options which can be used when starting ndbd, see Section 17.4.23, “Options Common to MySQL Cluster Programs”.

By default, ndbd looks for the management server at localhost on port 1186.

Note

If you have installed MySQL from a binary tarball, you will need to specify the path of the ndb_mgmd and ndbd servers explicitly. (Normally, these will be found in /usr/local/mysql/bin.)

Finally, change location to the MySQL data directory (usually /var/lib/mysql or /usr/local/mysql/data), and make sure that the my.cnf file contains the option necessary to enable the NDB storage engine:

[mysqld]
ndbcluster

You can now start the MySQL server as usual:

shell> mysqld_safe --user=mysql &

Wait a moment to make sure the MySQL server is running properly. If you see the notice mysql ended, check the server's .err file to find out what went wrong.

If all has gone well so far, you now can start using the cluster. Connect to the server and verify that the NDBCLUSTER storage engine is enabled:

shell> mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1 to server version: 5.1.54

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> SHOW ENGINES\G
...
*************************** 12. row ***************************
Engine: NDBCLUSTER
Support: YES
Comment: Clustered, fault-tolerant, memory-based tables
*************************** 13. row ***************************
Engine: NDB
Support: YES
Comment: Alias for NDBCLUSTER
...

The row numbers shown in the preceding example output may be different from those shown on your system, depending upon how your server is configured.

Try to create an NDBCLUSTER table:

shell> mysql
mysql> USE test;
Database changed

mysql> CREATE TABLE ctest (i INT) ENGINE=NDBCLUSTER;
Query OK, 0 rows affected (0.09 sec)

mysql> SHOW CREATE TABLE ctest \G
*************************** 1. row ***************************
       Table: ctest
Create Table: CREATE TABLE `ctest` (
  `i` int(11) default NULL
) ENGINE=ndbcluster DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

To check that your nodes were set up properly, start the management client:

shell> ndb_mgm

Use the SHOW command from within the management client to obtain a report on the cluster's status:

ndb_mgm> SHOW
Cluster Configuration
---------------------
[ndbd(NDB)]     1 node(s)
id=2    @127.0.0.1  (Version: 3.5.3, Nodegroup: 0, Master)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @127.0.0.1  (Version: 3.5.3)

[mysqld(API)]   3 node(s)
id=3    @127.0.0.1  (Version: 3.5.3)
id=4 (not connected, accepting connect from any host)
id=5 (not connected, accepting connect from any host)

At this point, you have successfully set up a working MySQL Cluster. You can now store data in the cluster by using any table created with ENGINE=NDBCLUSTER or its alias ENGINE=NDB.

17.3.2. MySQL Cluster Configuration Files

Configuring MySQL Cluster requires working with two files:

  • my.cnf: Specifies options for all MySQL Cluster executables. This file, with which you should be familiar with from previous work with MySQL, must be accessible by each executable running in the cluster.

  • config.ini: This file, sometimes known as the global configuration file, is read only by the MySQL Cluster management server, which then distributes the information contained therein to all processes participating in the cluster. config.ini contains a description of each node involved in the cluster. This includes configuration parameters for data nodes and configuration parameters for connections between all nodes in the cluster. For a quick reference to the sections that can appear in this file, and what sorts of configuration parameters may be placed in each section, see Sections of the config.ini File.

Caching of configuration data. Beginning with MySQL Cluster NDB 6.4.0, MySQL Cluster uses stateful configuration. The global configuration file is no longer read every time the management server is restarted. Instead, the management server caches the configuration the first time it is started, and thereafter, the global confiuration file is read only when one of the following items is true:

  • The management server is started using the --initial option. In this case, the global configuration file is re-read, any existing cache files are deleted, and the management server creates a new configuration cache.

  • The management server is started using the --reload option. In this case, the management server compares its cache with the global configuration file. If they differ, the management server creates a new configuration cache; any existing configuration cache is preserved, but not used. If the management server's cache and the global configuration file contain the same configuration data, then the existing cache is used, and no new cache is created.

  • The management server is started using a --config-cache option. Beginning with MySQL Cluster NDB 7.0.15 and MySQL Cluster NDB 7.1.4, this option can be used to force the management server to bypass configuration caching altogether. In this case, the management server ignores any configuration files that may be present, always reading its configuration data from the config.ini file instead.

  • No configuration cache is found. In this case, the management server reads the global configuration file and creates a cache containing the same configuration data as found in the file.

Configuration cache files. Beginning with MySQL Cluster 6.4.0, the management server by default creates configuration cache files in a directory named mysql-cluster in the MySQL installation directory. (If you build MySQL Cluster from source on a Unix system, the default location is /usr/local/mysql-cluster.) This can be overridden at run time by starting the management server with the --configdir option. Configuration cache files are binary files named according to the pattern ndb_node_id_config.bin.seq_id, where node_id is the management server's node ID in the cluster, and seq_id is a cache idenitifer. Cache files are numbered sequentially using seq_id, in the order in which they are created. The management server uses the latest cache file as determined by the seq_id.

Note

It is possible to roll back to a previous configuration by deleting later configuration cache files, or by renaming an earlier cache file so that it has a higher seq_id. However, since configuration cache files are written in a binary format, you should not attempt to edit their contents by hand.

For more information about the --configdir, --initial, and --reload options for the MySQL Cluster management server, see Section 17.4.4, “ndb_mgmd — The MySQL Cluster Management Server Daemon”.

We are continuously making improvements in Cluster configuration and attempting to simplify this process. Although we strive to maintain backward compatibility, there may be times when introduce an incompatible change. In such cases we will try to let Cluster users know in advance if a change is not backward compatible. If you find such a change and we have not documented it, please report it in the MySQL bugs database using the instructions given in Section 1.7, “How to Report Bugs or Problems”.

17.3.2.1. MySQL Cluster Configuration: Basic Example

To support MySQL Cluster, you will need to update my.cnf as shown in the following example. You may also specify these parameters on the command line when invoking the executables.

Note

The options shown here should not be confused with those that are used in config.ini global configuration files. Global configuration options are discussed later in this section.

# my.cnf
# example additions to my.cnf for MySQL Cluster
# (valid in MySQL 5.1)

# enable ndbcluster storage engine, and provide connectstring for
# management server host (default port is 1186)
[mysqld]
ndbcluster
ndb-connectstring=ndb_mgmd.mysql.com


# provide connectstring for management server host (default port: 1186)
[ndbd]
connect-string=ndb_mgmd.mysql.com

# provide connectstring for management server host (default port: 1186)
[ndb_mgm]
connect-string=ndb_mgmd.mysql.com

# provide location of cluster configuration file
[ndb_mgmd]
config-file=/etc/config.ini

(For more information on connectstrings, see Section 17.3.2.3, “The MySQL Cluster Connectstring”.)

# my.cnf
# example additions to my.cnf for MySQL Cluster
# (will work on all versions)

# enable ndbcluster storage engine, and provide connectstring for management
# server host to the default port 1186
[mysqld]
ndbcluster
ndb-connectstring=ndb_mgmd.mysql.com:1186
Important

Once you have started a mysqld process with the NDBCLUSTER and ndb-connectstring parameters in the [mysqld] in the my.cnf file as shown previously, you cannot execute any CREATE TABLE or ALTER TABLE statements without having actually started the cluster. Otherwise, these statements will fail with an error. This is by design.

You may also use a separate [mysql_cluster] section in the cluster my.cnf file for settings to be read and used by all executables:

# cluster-specific settings
[mysql_cluster]
ndb-connectstring=ndb_mgmd.mysql.com:1186

For additional NDB variables that can be set in the my.cnf file, see Section 17.3.4.3, “MySQL Cluster System Variables”.

The MySQL Cluster global configuration file is named config.ini by default. It is read by ndb_mgmd at startup and can be placed anywhere. Its location and name are specified by using --config-file=path_name on the ndb_mgmd command line. If the configuration file is not specified, ndb_mgmd by default tries to read a file named config.ini located in the current working directory.

The global configuration file for MySQL Cluster uses INI format, which consists of sections preceded by section headings (surrounded by square brackets), followed by the appropriate parameter names and values. One deviation from the standard INI format is that the parameter name and value can be separated by a colon (“:”) as well as the equal sign (“=”); however, the equal sign is preferred. Another deviation is that sections are not uniquely identified by section name. Instead, unique sections (such as two different nodes of the same type) are identified by a unique ID specified as a parameter within the section.

Default values are defined for most parameters, and can also be specified in config.ini. (Prior to MySQL Cluster NDB 6.3.25 and MySQL Cluster NDB 7.0.6, there was no default value for NoOfReplicas, which always had to be specified explicitly in the [ndbd default] section. Beginning with versions just stated, the default value is 2, which is the recommended setting in most common usage scenarios.) To create a default value section, simply add the word default to the section name. For example, an [ndbd] section contains parameters that apply to a particular data node, whereas an [ndbd default] section contains parameters that apply to all data nodes. Suppose that all data nodes should use the same data memory size. To configure them all, create an [ndbd default] section that contains a DataMemory line to specify the data memory size.

The global configuration file must define the computers and nodes involved in the cluster and on which computers these nodes are located. An example of a simple configuration file for a cluster consisting of one management server, two data nodes and two MySQL servers is shown here:

# file "config.ini" - 2 data nodes and 2 SQL nodes
# This file is placed in the startup directory of ndb_mgmd (the
# management server)
# The first MySQL Server can be started from any host. The second
# can be started only on the host mysqld_5.mysql.com

[ndbd default]
NoOfReplicas= 2
DataDir= /var/lib/mysql-cluster

[ndb_mgmd]
Hostname= ndb_mgmd.mysql.com
DataDir= /var/lib/mysql-cluster

[ndbd]
HostName= ndbd_2.mysql.com

[ndbd]
HostName= ndbd_3.mysql.com

[mysqld]
[mysqld]
HostName= mysqld_5.mysql.com
Note

The preceding example is intended as a minimal starting configuration for purposes of familiarization with MySQL Cluster, and is almost certain not to be sufficient for production settings. See Section 17.3.2.2, “Recommended Starting Configurations for MySQL Cluster NDB 6.2 and Later”, which provides more complete example starting configurations for use with MySQL Cluster NDB 6.2 and newer versions of MySQL Cluster.

Each node has its own section in the config.ini file. For example, this cluster has two data nodes, so the preceding configuration file contains two [ndbd] sections defining these nodes.

Note

Do not place comments on the same line as a section heading in the config.ini file; this causes the management server not to start because it cannot parse the configuration file in such cases.

Sections of the config.ini File

There are six different sections that you can use in the config.ini configuration file, as described in the following list:

You can define default values for each section. All Cluster parameter names are case-insensitive, which differs from parameters specified in my.cnf or my.ini files.

17.3.2.2. Recommended Starting Configurations for MySQL Cluster NDB 6.2 and Later

Achieving the best performance from a MySQL Cluster depends on a number of factors including the following:

  • MySQL Cluster software version

  • Numbers of data nodes and SQL nodes

  • Hardware

  • Operating system

  • Amount of data to be stored

  • Size and type of load under which the cluster is to operate

Therefore, obtaining an optimum configuration is likely to be an iterative process, the outcome of which can vary widely with the specifics of each MySQL Cluster deployment. Changes in configuration are also likely to be indicated when changes are made in the platform on which the cluster is run, or in applications that use the MySQL Cluster's data. For these reasons, it is not possible to offer a single configuration that is ideal for all usage scenarios. However, in this section, we provide recommended base configurations for MySQL Cluster NDB 6.2 and 6.3 that can serve as reasonable starting points.

Starting configuration for MySQL Cluster NDB 6.2. The following is a recommended starting point for configuring a cluster running MySQL Cluster NDB 6.2.

# TCP PARAMETERS

[tcp default]
SendBufferMemory=2M
ReceiveBufferMemory=2M

# Increasing the sizes of these 2 buffers beyond the default values
# helps prevent bottlenecks due to slow disk I/O.

# MANAGEMENT NODE PARAMETERS

[ndb_mgmd default]
DataDir=path/to/management/server/data/directory

# It is possible to use a different data directory for each management
# server, but for ease of administration it is preferable to be
# consistent.

[ndb_mgmd]
HostName=management-server-1-hostname
# Id=management-server-A-id

[ndb_mgmd]
HostName=management-server-2-hostname

# Using 2 management servers helps guarantee that there is always an
# arbitrator in the event of network partitioning, and so is
# recommended for high availability. Each management server must be
# identified by a HostName. You may for the sake of convenience specify
# a node ID for any management server, although one will be allocated
# for it automatically; if you do so, it must be in the range 1-255
# inclusive and must be unique among all IDs specified for cluster
# nodes.

# DATA NODE PARAMETERS

[ndbd default]
NoOfReplicas=2

# Using 2 replicas is recommended to guarantee availability of data; 
# using only 1 replica does not provide any redundancy, which means 
# that the failure of a single data node causes the entire cluster to 
# shut down. We do not recommend using more than 2 replicas, since 2 is 
# sufficient to provide high availability, and we do not currently test 
# with greater values for this parameter.

LockPagesInMainMemory=1

# On Linux and Solaris systems, setting this parameter locks data node
# processes into memory. Doing so prevents them from swapping to disk,
# which can severely degrade cluster performance.

DataMemory=3072M
IndexMemory=384M

# The values provided for DataMemory and IndexMemory assume 4 GB RAM
# per data node. However, for best results, you should first calculate
# the memory that would be used based on the data you actually plan to
# store (you may find the ndb_size.pl utility helpful in estimating
# this), then allow an extra 20% over the calculated values. Naturally,
# you should ensure that each data node host has at least as much
# physical memory as the sum of these two values.

# ODirect=1

# Enabling this parameter causes NDBCLUSTER to try using O_DIRECT
# writes for local checkpoints and redo logs; this can reduce load on
# CPUs. We recommend doing so when using MySQL Cluster NDB 6.2.3 or
# newer on systems running Linux kernel 2.6 or later.

NoOfFragmentLogFiles=300
DataDir=path/to/data/node/data/directory
MaxNoOfConcurrentOperations=100000
TimeBetweenGlobalCheckpoints=1000
TimeBetweenEpochs=200
DiskCheckpointSpeed=10M
DiskCheckpointSpeedInRestart=100M
RedoBuffer=32M
# MaxNoOfLocalScans=64
MaxNoOfTables=1024
MaxNofOfOrderedIndexes=256

[ndbd]
HostName=data-node-A-hostname
# Id=data-node-A-id

[ndbd]
HostName=data-node-B-hostname
# Id=data-node-B-id

# You must have an [ndbd] section for every data node in the cluster;
# each of these sections must include a HostName. Each section may
# optionally include an Id for convenience, but in most cases, it is
# sufficient to allow the cluster to allocate node IDs dynamically. If
# you do specify the node ID for a data node, it must be in the range 1
# to 48 inclusive and must be unique among all IDs specified for
# cluster nodes.

# SQL NODE / API NODE PARAMETERS

[mysqld]
# HostName=SQL-node-1-hostname
# Id=sql-node-A-id

[mysqld]

[mysqld]

# Each API or SQL node that connects to the cluster requires a [mysqld]
# or [api] section of its own. Each such section defines a connection
# “slot”; you should have at least as many of these sections in the
# config.ini file as the total number of API nodes and SQL nodes that
# you wish to have connected to the cluster at any given time. There is
# no performance or other penalty for having extra slots available in
# case you find later that you want or need more API or SQL nodes to
# connect to the cluster at the same time.
# If no HostName is specified for a given [mysqld] or [api] section,
# then any API or SQL node may use that slot to connect to the
# cluster. You may wish to use an explicit HostName for one connection slot
# to guarantee that an API or SQL node from that host can always
# connect to the cluster. If you wish to prevent API or SQL nodes from
# connecting from other than a desired host or hosts, then use a
# HostName for every [mysqld] or [api] section in the config.ini file.
# You can if you wish define a node ID (Id parameter) for any API or
# SQL node, but this is not necessary; if you do so, it must be in the
# range 1 to 255 inclusive and must be unique among all IDs specified
# for cluster nodes.

Starting configuration for MySQL Cluster NDB 6.3. The following is a recommended starting point for configuring a cluster running MySQL Cluster NDB 6.3. It is similar to the recommendation for MySQL Cluster NDB 6.2, with the addition of parameters for better control of NDBCLUSTER process threads.

# TCP PARAMETERS

[tcp default]
SendBufferMemory=2M
ReceiveBufferMemory=2M

# Increasing the sizes of these 2 buffers beyond the default values
# helps prevent bottlenecks due to slow disk I/O.

# MANAGEMENT NODE PARAMETERS

[ndb_mgmd default]
DataDir=path/to/management/server/data/directory

# It is possible to use a different data directory for each management
# server, but for ease of administration it is preferable to be
# consistent.

[ndb_mgmd]
HostName=management-server-1-hostname
# Id=management-server-A-id

[ndb_mgmd]
HostName=management-server-2-hostname

# Using 2 management servers helps guarantee that there is always an
# arbitrator in the event of network partitioning, and so is
# recommended for high availability. Each management server must be
# identified by a HostName. You may for the sake of convenience specify
# a node ID for any management server, although one will be allocated
# for it automatically; if you do so, it must be in the range 1-255
# inclusive and must be unique among all IDs specified for cluster
# nodes.

# DATA NODE PARAMETERS

[ndbd default]
NoOfReplicas=2

# Using 2 replicas is recommended to guarantee availability of data; 
# using only 1 replica does not provide any redundancy, which means 
# that the failure of a single data node causes the entire cluster to 
# shut down. We do not recommend using more than 2 replicas, since 2 is 
# sufficient to provide high availability, and we do not currently test 
# with greater values for this parameter.

LockPagesInMainMemory=1

# On Linux and Solaris systems, setting this parameter locks data node
# processes into memory. Doing so prevents them from swapping to disk,
# which can severely degrade cluster performance.

DataMemory=3072M
IndexMemory=384M

# The values provided for DataMemory and IndexMemory assume 4 GB RAM
# per data node. However, for best results, you should first calculate
# the memory that would be used based on the data you actually plan to
# store (you may find the ndb_size.pl utility helpful in estimating
# this), then allow an extra 20% over the calculated values. Naturally,
# you should ensure that each data node host has at least as much
# physical memory as the sum of these two values.

# ODirect=1

# Enabling this parameter causes NDBCLUSTER to try using O_DIRECT
# writes for local checkpoints and redo logs; this can reduce load on
# CPUs. We recommend doing so when using MySQL Cluster NDB 6.2.3 or
# newer on systems running Linux kernel 2.6 or later.

NoOfFragmentLogFiles=300
DataDir=path/to/data/node/data/directory
MaxNoOfConcurrentOperations=100000

SchedulerSpinTimer=400
SchedulerExecutionTimer=100
RealTimeScheduler=1
# Setting these parameters allows you to take advantage of real-time scheduling
# of NDBCLUSTER threads (introduced in MySQL Cluster NDB 6.3.4) to get higher
# throughput.

TimeBetweenGlobalCheckpoints=1000
TimeBetweenEpochs=200
DiskCheckpointSpeed=10M
DiskCheckpointSpeedInRestart=100M
RedoBuffer=32M

# CompressedLCP=1
# CompressedBackup=1
# Enabling CompressedLCP and CompressedBackup causes, respectively, local
checkpoint files and backup files to be compressed, which can result in a space
savings of up to 50% over noncompressed LCPs and backups.

# MaxNoOfLocalScans=64
MaxNoOfTables=1024
MaxNofOfOrderedIndexes=256

[ndbd]
HostName=data-node-A-hostname
# Id=data-node-A-id

LockExecuteThreadToCPU=1
LockMaintThreadsToCPU=0
# On systems with multiple CPUs, these parameters can be used to lock NDBCLUSTER
# threads to specific CPUs

[ndbd]
HostName=data-node-B-hostname
# Id=data-node-B-id

LockExecuteThreadToCPU=1
LockMaintThreadsToCPU=0

# You must have an [ndbd] section for every data node in the cluster;
# each of these sections must include a HostName. Each section may
# optionally include an Id for convenience, but in most cases, it is
# sufficient to allow the cluster to allocate node IDs dynamically. If
# you do specify the node ID for a data node, it must be in the range 1
# to 48 inclusive and must be unique among all IDs specified for
# cluster nodes.

# SQL NODE / API NODE PARAMETERS

[mysqld]
# HostName=SQL-node-1-hostname
# Id=sql-node-A-id

[mysqld]

[mysqld]

# Each API or SQL node that connects to the cluster requires a [mysqld]
# or [api] section of its own. Each such section defines a connection
# “slot”; you should have at least as many of these sections in the
# config.ini file as the total number of API nodes and SQL nodes that
# you wish to have connected to the cluster at any given time. There is
# no performance or other penalty for having extra slots available in
# case you find later that you want or need more API or SQL nodes to
# connect to the cluster at the same time.
# If no HostName is specified for a given [mysqld] or [api] section,
# then any API or SQL node may use that slot to connect to the
# cluster. You may wish to use an explicit HostName for one connection slot
# to guarantee that an API or SQL node from that host can always
# connect to the cluster. If you wish to prevent API or SQL nodes from
# connecting from other than a desired host or hosts, then use a
# HostName for every [mysqld] or [api] section in the config.ini file.
# You can if you wish define a node ID (Id parameter) for any API or
# SQL node, but this is not necessary; if you do so, it must be in the
# range 1 to 255 inclusive and must be unique among all IDs specified
# for cluster nodes.

Recommended my.cnf options for SQL nodes. MySQL Servers acting as MySQL Cluster SQL nodes must always be started with the --ndbcluster and --ndb-connectstring options, either on the command line or in my.cnf. In addition, set the following options for all mysqld processes in the cluster, unless your setup requires otherwise:

  • --ndb-use-exact-count=0

  • --ndb-index-stat-enable=0

  • --ndb-force-send=1

  • --engine-condition-pushdown=1

17.3.2.3. The MySQL Cluster Connectstring

With the exception of the MySQL Cluster management server (ndb_mgmd), each node that is part of a MySQL Cluster requires a connectstring that points to the management server's location. This connectstring is used in establishing a connection to the management server as well as in performing other tasks depending on the node's role in the cluster. The syntax for a connectstring is as follows:

[nodeid=node_id, ]host-definition[, host-definition[, ...]]

host-definition:
    host_name[:port_number]

node_id is an integer larger than 1 which identifies a node in config.ini. host_name is a string representing a valid Internet host name or IP address. port_number is an integer referring to a TCP/IP port number.

example 1 (long):    "nodeid=2,myhost1:1100,myhost2:1100,192.168.0.3:1200"
example 2 (short):   "myhost1"

localhost:1186 is used as the default connectstring value if none is provided. If port_num is omitted from the connectstring, the default port is 1186. This port should always be available on the network because it has been assigned by IANA for this purpose (see http://www.iana.org/assignments/port-numbers for details).

By listing multiple host definitions, it is possible to designate several redundant management servers. A MySQL Cluster data or API node attempts to contact successive management servers on each host in the order specified, until a successful connection has been established.

Beginning with MySQL Cluster NDB 6.3.19, it is also possible in a connectstring to specify one or more bind addresses to be used by nodes having multiple network interfaces for connecting to management servers. A bind address consists of a hostname or network address and an optional port number. This enhanced syntax for connectstrings is shown here:

[nodeid=node_id, ]
    [bind-address=host-definition, ]
    host-definition[; bind-address=host-definition]
    host-definition[; bind-address=host-definition]
    [, ...]]

host-definition:
    host_name[:port_number]

If a single bind address is used in the connectstring prior to specifying any management hosts, then this address is used as the default for connecting to any of them (unless overridden for a given management server; see later in this section for an example). For example, the following connectstring causes the node to use 192.168.178.242 regardless of the management server to which it connects:

bind-address=192.168.178.242, poseidon:1186, perch:1186

If a bind address is specified following a management host definition, then it is used only for connecting to that management node. Consider the following connectstring:

poseidon:1186;bind-address=localhost, perch:1186;bind-address=192.168.178.242

In this case, the node uses localhost to connect to the management server running on the host named poseidon and 192.168.178.242 to connect to the management server running on the host named perch.

You can specify a default bind address and then override this default for one or more specific management hosts. In the following example, localhost is used for connecting to the management server running on host poseidon; since 192.168.178.242 is specified first (before any management server definitions), it is the default bind address and so is used for connecting to the management servers on hosts perch and orca:

bind-address=192.168.178.242,poseidon:1186;bind-address=localhost,perch:1186,orca:2200

There are a number of different ways to specify the connectstring:

  • Each executable has its own command-line option which enables specifying the management server at startup. (See the documentation for the respective executable.)

  • It is also possible to set the connectstring for all nodes in the cluster at once by placing it in a [mysql_cluster] section in the management server's my.cnf file.

  • For backward compatibility, two other options are available, using the same syntax:

    1. Set the NDB_CONNECTSTRING environment variable to contain the connectstring.

    2. Write the connectstring for each executable into a text file named Ndb.cfg and place this file in the executable's startup directory.

    However, these are now deprecated and should not be used for new installations.

The recommended method for specifying the connectstring is to set it on the command line or in the my.cnf file for each executable.

Previous to MySQL Cluster NDB 7.0.15 and MySQL Cluster NDB 7.1.4, the maximum length of a connectstring was 1024 characters.

17.3.2.4. Defining Computers in a MySQL Cluster

The [computer] section has no real significance other than serving as a way to avoid the need of defining host names for each node in the system. All parameters mentioned here are required.

  • Id

    Restart Typeinitial, node
    Permitted Values
    Typestring
    Default
    Range-

    This is a unique identifier, used to refer to the host computer elsewhere in the configuration file.

    Important

    The computer ID is not the same as the node ID used for a management, API, or data node. Unlike the case with node IDs, you cannot use NodeId in place of Id in the [computer] section of the config.ini file.

  • HostName

    Restart Typesystem
    Permitted Values
    Typestring
    Default
    Range-

    This is the computer's hostname or IP address.

17.3.2.5. Defining a MySQL Cluster Management Server

The [ndb_mgmd] section is used to configure the behavior of the management server. [mgm] can be used as an alias; the two section names are equivalent. All parameters in the following list are optional and assume their default values if omitted.

Note

If neither the ExecuteOnComputer nor the HostName parameter is present, the default value localhost will be assumed for both.

  • Id

    Restart Typenode
    Permitted Values
    Typenumeric
    Default
    Range1-63

    Each node in the cluster has a unique identity. For a management node, this is represented by an integer value in the range 1 to 63 inclusive (previous to MySQL Cluster NDB 6.1.1), or in the range 1 to 255 inclusive (MySQL Cluster NDB 6.1.1 and later). This ID is used by all internal cluster messages for addressing the node, and so must be unique for each MySQL Cluster node, regardless of the type of node.

    Note

    Data node IDs must be less than 49, regardless of the MySQL Cluster version used. If you plan to deploy a large number of data nodes, it is a good idea to limit the node IDs for management nodes (and API nodes) to values greater than 48.

    This parameter can also be written as NodeId, although the short form is sufficient (and preferred for this reason).

  • ExecuteOnComputer

    Restart Typesystem
    Permitted Values
    Typestring
    Default
    Range-

    This refers to the Id set for one of the computers defined in a [computer] section of the config.ini file.

  • PortNumber

    Restart Typenode
    Permitted Values
    Typenumeric
    Default1186
    Range0-64K

    This is the port number on which the management server listens for configuration requests and management commands.

  • HostName

    Restart Typesystem
    Permitted Values
    Typestring
    Default
    Range-

    Specifying this parameter defines the hostname of the computer on which the management node is to reside. To specify a hostname other than localhost, either this parameter or ExecuteOnComputer is required.

  • LogDestination

    Restart Typenode
    Permitted Values
    Typestring
    DefaultFILE:filename=ndb_nodeid_cluster.log,maxsize=1000000,maxfiles=6
    Range-

    This parameter specifies where to send cluster logging information. There are three options in this regard—CONSOLE, SYSLOG, and FILE—with FILE being the default:

    • CONSOLE outputs the log to stdout:

      CONSOLE
      
    • SYSLOG sends the log to a syslog facility, possible values being one of auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6, or local7.

      Note

      Not every facility is necessarily supported by every operating system.

      SYSLOG:facility=syslog
      
    • FILE pipes the cluster log output to a regular file on the same machine. The following values can be specified:

      • filename: The name of the log file.

      • maxsize: The maximum size (in bytes) to which the file can grow before logging rolls over to a new file. When this occurs, the old log file is renamed by appending .N to the file name, where N is the next number not yet used with this name.

      • maxfiles: The maximum number of log files.

      FILE:filename=cluster.log,maxsize=1000000,maxfiles=6
      

      The default value for the FILE parameter is FILE:filename=ndb_node_id_cluster.log,maxsize=1000000,maxfiles=6, where node_id is the ID of the node.

    It is possible to specify multiple log destinations separated by semicolons as shown here:

    CONSOLE;SYSLOG:facility=local0;FILE:filename=/var/log/mgmd
    
  • ArbitrationRank

    Restart Typenode
    Permitted Values
    Typenumeric
    Default1
    Range0-2

    This parameter is used to define which nodes can act as arbitrators. Only management nodes and SQL nodes can be arbitrators. ArbitrationRank can take one of the following values:

    • 0: The node will never be used as an arbitrator.

    • 1: The node has high priority; that is, it will be preferred as an arbitrator over low-priority nodes.

    • 2: Indicates a low-priority node which be used as an arbitrator only if a node with a higher priority is not available for that purpose.

    Normally, the management server should be configured as an arbitrator by setting its ArbitrationRank to 1 (the default for management nodes) and those for all SQL nodes to 0 (the default for SQL nodes).

    Beginning with MySQL 5.1.16 and MySQL Cluster NDB 6.1.3, it is possible to disable arbitration completely by setting ArbitrationRank to 0 on all management and SQL nodes. In MySQL Cluster NDB 7.0.7 and later releases, you can also control arbitration by overriding this parameter; to do this, set the Arbitration parameter in the [ndbd default] section of the config.ini global configuration file.

  • ArbitrationDelay

    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-4G

    An integer value which causes the management server's responses to arbitration requests to be delayed by that number of milliseconds. By default, this value is 0; it is normally not necessary to change it.

  • DataDir

    Restart Typenode
    Permitted Values
    Typestring
    Default.
    Range-

    This specifies the directory where output files from the management server will be placed. These files include cluster log files, process output files, and the daemon's process ID (PID) file. (For log files, this location can be overridden by setting the FILE parameter for LogDestination as discussed previously in this section.)

    The default value for this parameter is the directory in which ndb_mgmd is located.

  • HeartbeatThreadPriority

    Beginning with MySQL Cluster NDB 6.3.32, MySQL Cluster NDB 7.0.13, and MySQL Cluster NDB 7.1.2, it is possible to use this parameter to set the scheduling policy and priority of heartbeat threads for management and API nodes.

    The syntax for setting this parameter is shown here:

    HeartbeatThreadPriority = policy[, priority]
    
    policy:
      {FIFO | RR}
    

    When setting this parameter, you must specify a policy. This is one of FIFO (first in, first in) or RR (round robin). The policy value is followed optionally by the priority (an integer).

  • TotalSendBufferMemory

    This parameter is available beginning with MySQL Cluster NDB 6.4.0. It is used to determine the total amount of memory to allocate on this node for shared send buffer memory among all configured transporters.

    If this parameter is set, its minimum permitted value is 256KB; the maxmimum is 4294967039. For more detailed information about the behavior and use of TotalSendBufferMemory and configuring send buffer memory parameters in MySQL Cluster NDB 6.4.0 and later, see Section 17.3.2.13, “Configuring MySQL Cluster Send Buffer Parameters”.

Note

After making changes in a management node's configuration, it is necessary to perform a rolling restart of the cluster for the new configuration to take effect.

To add new management servers to a running MySQL Cluster, it is also necessary to perform a rolling restart of all cluster nodes after modifying any existing config.ini files. For more information about issues arising when using multiple management nodes, see Section 17.1.5.10, “Limitations Relating to Multiple MySQL Cluster Nodes”.

17.3.2.6. Defining MySQL Cluster Data Nodes

The [ndbd] and [ndbd default] sections are used to configure the behavior of the cluster's data nodes.

[ndbd] and [ndbd default] are always used as the section names whether you are using ndbd or (in MySQL Cluster NDB 6.4.0 and later) ndbmtd binaries for the data node processes.

There are many parameters which control buffer sizes, pool sizes, timeouts, and so forth. The only mandatory parameters are:

  • Either ExecuteOnComputer or HostName, which must be defined in the local [ndbd] section.

  • The parameter NoOfReplicas, which must be defined in the [ndbd default] section, as it is common to all Cluster data nodes.

Note

It is no longer strictly necessary to set NoOfReplicas starting with MySQL Cluster NDB 6.3.25 and MySQL Cluster NDB 7.0.6, where it acquires a default value (2). However, it remains good practice to set it explicitly.

Most data node parameters are set in the [ndbd default] section. Only those parameters explicitly stated as being able to set local values are permitted to be changed in the [ndbd] section. Where present, HostName, Id and ExecuteOnComputer must be defined in the local [ndbd] section, and not in any other section of config.ini. In other words, settings for these parameters are specific to one data node.

For those parameters affecting memory usage or buffer sizes, it is possible to use K, M, or G as a suffix to indicate units of 1024, 1024×1024, or 1024×1024×1024. (For example, 100K means 100 × 1024 = 102400.) Parameter names and values are currently case-sensitive.

Information about configuration parameters specific to MySQL Cluster Disk Data tables can be found later in this section.

Beginning with MySQL Cluster NDB 6.4.0, all of these parameters also apply to ndbmtd (the multi-threaded version of ndbd). An additional data node configuration parameter MaxNoOfExecutionThreads applies to ndbmtd only, and has no effect when used with ndbd. For more information, see Section 17.4.3, “ndbmtd — The MySQL Cluster Data Node Daemon (Multi-Threaded)”.

Identifying data nodes. The Id value (that is, the data node identifier) can be allocated on the command line when the node is started or in the configuration file.

  • Id

    Restart Typenode
    Permitted Values
    Typenumeric
    Default
    Range1-48

    This is the node ID used as the address of the node for all cluster internal messages. For data nodes, this is an integer in the range 1 to 48 inclusive. Each node in the cluster must have a unique identifier.

    This parameter can also be written as NodeId, although the short form is sufficient (and preferred for this reason).

  • ExecuteOnComputer

    Restart Typesystem
    Permitted Values
    Typestring
    Default
    Range-

    This refers to the Id set for one of the computers defined in a [computer] section.

  • HostName

    Restart Typesystem
    Permitted Values
    Typestring
    Defaultlocalhost
    Range-

    Specifying this parameter defines the hostname of the computer on which the data node is to reside. To specify a hostname other than localhost, either this parameter or ExecuteOnComputer is required.

  • ServerPort

    Restart Typenode
    Permitted Values
    Typenumeric
    Default
    Range1-64K

    Each node in the cluster uses a port to connect to other nodes. By default, this port is allocated dynamically in such a way as to ensure that no two nodes on the same host computer receive the same port number, so it should normally not be necessary to specify a value for this parameter.

    However, if you need to be able to open specific ports in a firewall to permit communication between data nodes and API nodes (including SQL nodes), you can set this parameter to the number of the desired port in an [ndbd] section or (if you need to do this for multiple data nodes) the [ndbd default] section of the config.ini file, and then open the port having that number for incoming connections from SQL nodes, API nodes, or both.

    Note

    Connections from data nodes to management nodes is done using the ndb_mgmd management port (the management server's PortNumber; see Section 17.3.2.5, “Defining a MySQL Cluster Management Server”) so outgoing connections to that port from any data nodes should always be permitted.

  • TcpBind_INADDR_ANY

    Setting this parameter to TRUE or 1 binds IP_ADDR_ANY so that connections can be made from anywhere (for autogenerated connections). The default is FALSE (0).

    This parameter was added in MySQL Cluster NDB 6.2.0.

  • NodeGroup

    Version Introduced5.1.30-ndb-6.4.0
    Restart Typeinitial, system
    Permitted Values
    Typenumeric
    Default
    Range0-64K

    This parameter can be used to assign a data node to a specific node group. It is read only when the cluster is started for the first time, and cannot be used to reassign a data node to a different node group online. It is generally not desirable to use this parameter in the [ndbd default] section of the config.ini file, and care must be taken not to assign nodes to node groups in such a way that an invalid numbers of nodes are assigned to any node groups.

    The NodeGroup parameter is chiefly intended for use in adding a new node group to a running MySQL Cluster without having to perform a rolling restart. For this purpose, you should set it to 65535 (the maximum value). You are not required to set a NodeGroup value for all cluster data nodes, only for those nodes which are to be started and added to the cluster as a new node group at a later time. For more information, see Section 17.5.11.3, “Adding MySQL Cluster Data Nodes Online: Detailed Example”.

    This parameter was added in MySQL Cluster NDB 6.4.0.

  • NoOfReplicas

    Restart Typeinitial, system
    Permitted Values
    Typenumeric
    DefaultNone
    Range1-4
    Permitted Values
    Typenumeric
    DefaultNone
    Range1-4
    Permitted Values
    Typenumeric
    Default2
    Range1-4
    Permitted Values
    Typenumeric
    Default2
    Range1-4

    This global parameter can be set only in the [ndbd default] section, and defines the number of replicas for each table stored in the cluster. This parameter also specifies the size of node groups. A node group is a set of nodes all storing the same information.

    Node groups are formed implicitly. The first node group is formed by the set of data nodes with the lowest node IDs, the next node group by the set of the next lowest node identities, and so on. By way of example, assume that we have 4 data nodes and that NoOfReplicas is set to 2. The four data nodes have node IDs 2, 3, 4 and 5. Then the first node group is formed from nodes 2 and 3, and the second node group by nodes 4 and 5. It is important to configure the cluster in such a manner that nodes in the same node groups are not placed on the same computer because a single hardware failure would cause the entire cluster to fail.

    If no node IDs are provided, the order of the data nodes will be the determining factor for the node group. Whether or not explicit assignments are made, they can be viewed in the output of the management client's SHOW command.

    Prior to MySQL Cluster NDB 6.3.25 and MySQL Cluster NDB 7.0.6, there was no default value for NoOfReplicas; beginning with these versions, the default value is 2, which is the recommended setting in most common usage scenarios. (Bug#44746)

    The maximum possible value is 4; currently, only the values 1 and 2 are actually supported.

    Important

    Setting NoOfReplicas to 1 means that there is only a single copy of all Cluster data; in this case, the loss of a single data node causes the cluster to fail because there are no additional copies of the data stored by that node.

    The value for this parameter must divide evenly into the number of data nodes in the cluster. For example, if there are two data nodes, then NoOfReplicas must be equal to either 1 or 2, since 2/3 and 2/4 both yield fractional values; if there are four data nodes, then NoOfReplicas must be equal to 1, 2, or 4.

  • DataDir

    Restart Typeinitial, node
    Permitted Values
    Typestring
    Default.
    Range-

    This parameter specifies the directory where trace files, log files, pid files and error logs are placed.

    The default is the data node process working directory.

  • FileSystemPath

    Version Introduced5.1.15-ndb-6.1.1
    Restart Typeinitial, node
    Permitted Values
    Typestring
    DefaultDataDir
    Range-

    This parameter specifies the directory where all files created for metadata, REDO logs, UNDO logs (for Disk Data tables), and data files are placed. The default is the directory specified by DataDir.

    Note

    This directory must exist before the ndbd process is initiated.

    The recommended directory hierarchy for MySQL Cluster includes /var/lib/mysql-cluster, under which a directory for the node's file system is created. The name of this subdirectory contains the node ID. For example, if the node ID is 2, this subdirectory is named ndb_2_fs.

  • BackupDataDir

    Restart Typeinitial, node
    Permitted Values
    Typestring
    DefaultFileSystemPath
    Range-

    This parameter specifies the directory in which backups are placed.

    Important

    The string '/BACKUP' is always appended to this value. For example, if you set the value of BackupDataDir to /var/lib/cluster-data, then all backups are stored under /var/lib/cluster-data/BACKUP. This also means that the effective default backup location is the directory named BACKUP under the location specified by the FileSystemPath parameter.

Data Memory, Index Memory, and String Memory

DataMemory and IndexMemory are [ndbd] parameters specifying the size of memory segments used to store the actual records and their indexes. In setting values for these, it is important to understand how DataMemory and IndexMemory are used, as they usually need to be updated to reflect actual usage by the cluster:

  • DataMemory

    Restart Typenode
    Permitted Values
    Typenumeric
    Default80M
    Range1M-1024G

    This parameter defines the amount of space (in bytes) available for storing database records. The entire amount specified by this value is allocated in memory, so it is extremely important that the machine has sufficient physical memory to accommodate it.

    The memory allocated by DataMemory is used to store both the actual records and indexes. There is a 16-byte overhead on each record; an additional amount for each record is incurred because it is stored in a 32KB page with 128 byte page overhead (see below). There is also a small amount wasted per page due to the fact that each record is stored in only one page.

    For variable-size table attributes in MySQL 5.1, the data is stored on separate datapages, allocated from DataMemory. Variable-length records use a fixed-size part with an extra overhead of 4 bytes to reference the variable-size part. The variable-size part has 2 bytes overhead plus 2 bytes per attribute.

    The maximum record size is currently 8052 bytes.

    The memory space defined by DataMemory is also used to store ordered indexes, which use about 10 bytes per record. Each table row is represented in the ordered index. A common error among users is to assume that all indexes are stored in the memory allocated by IndexMemory, but this is not the case: Only primary key and unique hash indexes use this memory; ordered indexes use the memory allocated by DataMemory. However, creating a primary key or unique hash index also creates an ordered index on the same keys, unless you specify USING HASH in the index creation statement. This can be verified by running ndb_desc -d db_name table_name in the management client.

    The memory space allocated by DataMemory consists of 32KB pages, which are allocated to table fragments. Each table is normally partitioned into the same number of fragments as there are data nodes in the cluster. Thus, for each node, there are the same number of fragments as are set in NoOfReplicas.

    In addition, due to the way in which new pages are allocated when the capacity of the current page is exhausted, there is an additional overhead of approximately 18.75%. When more DataMemory is required, more than one new page is allocated, according to the following formula:

    number of new pages = FLOOR(number of current pages × 0.1875) + 1
    

    For example, if 15 pages are currently allocated to a given table and an insert to this table requires additional storage space, the number of new pages allocated to the table is FLOOR(15 × 0.1875) + 1 = FLOOR(2.8125) + 1 = 2 + 1 = 3. Now 15 + 3 = 18 memory pages are allocated to the table. When the last of these 18 pages becomes full, FLOOR(18 × 0.1875) + 1 = FLOOR(3.3750) + 1 = 3 + 1 = 4 new pages are allocated, so the total number of pages allocated to the table is now 22.

    Note

    The “18.75% + 1” overhead is no longer required beginning with MySQL Cluster NDB 6.2.3 and MySQL Cluster NDB 6.3.0.

    Once a page has been allocated, it is currently not possible to return it to the pool of free pages, except by deleting the table. (This also means that DataMemory pages, once allocated to a given table, cannot be used by other tables.) Performing a node recovery also compresses the partition because all records are inserted into empty partitions from other live nodes.

    The DataMemory memory space also contains UNDO information: For each update, a copy of the unaltered record is allocated in the DataMemory. There is also a reference to each copy in the ordered table indexes. Unique hash indexes are updated only when the unique index columns are updated, in which case a new entry in the index table is inserted and the old entry is deleted upon commit. For this reason, it is also necessary to allocate enough memory to handle the largest transactions performed by applications using the cluster. In any case, performing a few large transactions holds no advantage over using many smaller ones, for the following reasons:

    • Large transactions are not any faster than smaller ones

    • Large transactions increase the number of operations that are lost and must be repeated in event of transaction failure

    • Large transactions use more memory

    The default value for DataMemory is 80MB; the minimum is 1MB. There is no maximum size, but in reality the maximum size has to be adapted so that the process does not start swapping when the limit is reached. This limit is determined by the amount of physical RAM available on the machine and by the amount of memory that the operating system may commit to any one process. 32-bit operating systems are generally limited to 2–4GB per process; 64-bit operating systems can use more. For large databases, it may be preferable to use a 64-bit operating system for this reason.

  • IndexMemory

    Restart Typenode
    Permitted Values
    Typenumeric
    Default18M
    Range1M-1T

    This parameter controls the amount of storage used for hash indexes in MySQL Cluster. Hash indexes are always used for primary key indexes, unique indexes, and unique constraints. Note that when defining a primary key and a unique index, two indexes will be created, one of which is a hash index used for all tuple accesses as well as lock handling. It is also used to enforce unique constraints.

    The size of the hash index is 25 bytes per record, plus the size of the primary key. For primary keys larger than 32 bytes another 8 bytes is added.

    The default value for IndexMemory is 18MB. The minimum is 1MB.

  • StringMemory

    Restart Typesystem
    Permitted Values (>= 5.1.6)
    Typenumeric
    Default5
    Range0-4G

    This parameter determines how much memory is allocated for strings such as table names, and is specified in an [ndbd] or [ndbd default] section of the config.ini file. A value between 0 and 100 inclusive is interpreted as a percent of the maximum default value, which is calculated based on a number of factors including the number of tables, maximum table name size, maximum size of .FRM files, MaxNoOfTriggers, maximum column name size, and maximum default column value. In general it is safe to assume that the maximum default value is approximately 5 MB for a MySQL Cluster having 1000 tables.

    A value greater than 100 is interpreted as a number of bytes.

    Beginning with MySQL Cluster NDB 6.2.18, MySQL Cluster NDB 6.3.24, and MySQL Cluster NDB 7.0.5, the default value is 25—that is, 25 percent of the default maximum, or approximately 25 KB. (Previously, the default value was 5 beginning with MySQL 5.1.6; prior to MySQL 5.1.6, the default was 0.)

    Under most circumstances, the default value should be sufficient, but when you have a great many Cluster tables (1000 or more), it is possible to get Error 773 Out of string memory, please modify StringMemory config parameter: Permanent error: Schema error, in which case you should increase this value. 25 (25 percent) is not excessive, and should prevent this error from recurring in all but the most extreme conditions.

The following example illustrates how memory is used for a table. Consider this table definition:

CREATE TABLE example (
  a INT NOT NULL,
  b INT NOT NULL,
  c INT NOT NULL,
  PRIMARY KEY(a),
  UNIQUE(b)
) ENGINE=NDBCLUSTER;

For each record, there are 12 bytes of data plus 12 bytes overhead. Having no nullable columns saves 4 bytes of overhead. In addition, we have two ordered indexes on columns a and b consuming roughly 10 bytes each per record. There is a primary key hash index on the base table using roughly 29 bytes per record. The unique constraint is implemented by a separate table with b as primary key and a as a column. This other table consumes an additional 29 bytes of index memory per record in the example table as well 8 bytes of record data plus 12 bytes of overhead.

Thus, for one million records, we need 58MB for index memory to handle the hash indexes for the primary key and the unique constraint. We also need 64MB for the records of the base table and the unique index table, plus the two ordered index tables.

You can see that hash indexes takes up a fair amount of memory space; however, they provide very fast access to the data in return. They are also used in MySQL Cluster to handle uniqueness constraints.

Currently, the only partitioning algorithm is hashing and ordered indexes are local to each node. Thus, ordered indexes cannot be used to handle uniqueness constraints in the general case.

An important point for both IndexMemory and DataMemory is that the total database size is the sum of all data memory and all index memory for each node group. Each node group is used to store replicated information, so if there are four nodes with two replicas, there will be two node groups. Thus, the total data memory available is 2 × DataMemory for each data node.

It is highly recommended that DataMemory and IndexMemory be set to the same values for all nodes. Data distribution is even over all nodes in the cluster, so the maximum amount of space available for any node can be no greater than that of the smallest node in the cluster.

DataMemory and IndexMemory can be changed, but decreasing either of these can be risky; doing so can easily lead to a node or even an entire MySQL Cluster that is unable to restart due to there being insufficient memory space. Increasing these values should be acceptable, but it is recommended that such upgrades are performed in the same manner as a software upgrade, beginning with an update of the configuration file, and then restarting the management server followed by restarting each data node in turn.

Updates do not increase the amount of index memory used. Inserts take effect immediately; however, rows are not actually deleted until the transaction is committed.

Transaction parameters. The next three [ndbd] parameters that we discuss are important because they affect the number of parallel transactions and the sizes of transactions that can be handled by the system. MaxNoOfConcurrentTransactions sets the number of parallel transactions possible in a node. MaxNoOfConcurrentOperations sets the number of records that can be in update phase or locked simultaneously.

Both of these parameters (especially MaxNoOfConcurrentOperations) are likely targets for users setting specific values and not using the default value. The default value is set for systems using small transactions, to ensure that these do not use excessive memory.

  • MaxNoOfConcurrentTransactions

    Restart Typesystem
    Permitted Values
    Typenumeric
    Default4096
    Range32-4G

    Each cluster data node requires a transaction record for each active transaction in the cluster. The task of coordinating transactions is distributed among all of the data nodes. The total number of transaction records in the cluster is the number of transactions in any given node times the number of nodes in the cluster.

    Transaction records are allocated to individual MySQL servers. Each connection to a MySQL server requires at least one transaction record, plus an additional transaction object per table accessed by that connection. This means that a reasonable minimum for this parameter is

    MaxNoOfConcurrentTransactions =
        (maximum number of tables accessed in any single transaction + 1)
        * number of cluster SQL nodes
    

    Suppose that there are 4 SQL nodes using the cluster. A single join involving 5 tables requires 6 transaction records; if there are 5 such joins in a transaction, then 5 * 6 = 30 transaction records are required for this transaction, per MySQL server, or 30 * 4 = 120 transaction records total.

    This parameter must be set to the same value for all cluster data nodes. This is due to the fact that, when a data node fails, the oldest surviving node re-creates the transaction state of all transactions that were ongoing in the failed node.

    Changing the value of MaxNoOfConcurrentTransactions requires a complete shutdown and restart of the cluster.

    The default value is 4096.

  • MaxNoOfConcurrentOperations

    Restart Typenode
    Permitted Values
    Typenumeric
    Default32K
    Range32-4G

    It is a good idea to adjust the value of this parameter according to the size and number of transactions. When performing transactions of only a few operations each and not involving a great many records, there is no need to set this parameter very high. When performing large transactions involving many records need to set this parameter higher.

    Records are kept for each transaction updating cluster data, both in the transaction coordinator and in the nodes where the actual updates are performed. These records contain state information needed to find UNDO records for rollback, lock queues, and other purposes.

    This parameter should be set to the number of records to be updated simultaneously in transactions, divided by the number of cluster data nodes. For example, in a cluster which has four data nodes and which is expected to handle 1,000,000 concurrent updates using transactions, you should set this value to 1000000 / 4 = 250000.

    Read queries which set locks also cause operation records to be created. Some extra space is allocated within individual nodes to accommodate cases where the distribution is not perfect over the nodes.

    When queries make use of the unique hash index, there are actually two operation records used per record in the transaction. The first record represents the read in the index table and the second handles the operation on the base table.

    The default value is 32768.

    This parameter actually handles two values that can be configured separately. The first of these specifies how many operation records are to be placed with the transaction coordinator. The second part specifies how many operation records are to be local to the database.

    A very large transaction performed on an eight-node cluster requires as many operation records in the transaction coordinator as there are reads, updates, and deletes involved in the transaction. However, the operation records of the are spread over all eight nodes. Thus, if it is necessary to configure the system for one very large transaction, it is a good idea to configure the two parts separately. MaxNoOfConcurrentOperations will always be used to calculate the number of operation records in the transaction coordinator portion of the node.

    It is also important to have an idea of the memory requirements for operation records. These consume about 1KB per record.

  • MaxNoOfLocalOperations

    Restart Typenode
    Permitted Values
    Typenumeric
    DefaultUNDEFINED
    Range32-4G

    By default, this parameter is calculated as 1.1 × MaxNoOfConcurrentOperations. This fits systems with many simultaneous transactions, none of them being very large. If there is a need to handle one very large transaction at a time and there are many nodes, it is a good idea to override the default value by explicitly specifying this parameter.

Transaction temporary storage. The next set of [ndbd] parameters is used to determine temporary storage when executing a statement that is part of a Cluster transaction. All records are released when the statement is completed and the cluster is waiting for the commit or rollback.

The default values for these parameters are adequate for most situations. However, users with a need to support transactions involving large numbers of rows or operations may need to increase these values to enable better parallelism in the system, whereas users whose applications require relatively small transactions can decrease the values to save memory.

  • MaxNoOfConcurrentIndexOperations

    Restart Typenode
    Permitted Values
    Typenumeric
    Default8K
    Range0-4G

    For queries using a unique hash index, another temporary set of operation records is used during a query's execution phase. This parameter sets the size of that pool of records. Thus, this record is allocated only while executing a part of a query. As soon as this part has been executed, the record is released. The state needed to handle aborts and commits is handled by the normal operation records, where the pool size is set by the parameter MaxNoOfConcurrentOperations.

    The default value of this parameter is 8192. Only in rare cases of extremely high parallelism using unique hash indexes should it be necessary to increase this value. Using a smaller value is possible and can save memory if the DBA is certain that a high degree of parallelism is not required for the cluster.

  • MaxNoOfFiredTriggers

    Restart Typenode
    Permitted Values
    Typenumeric
    Default4000
    Range0-4G

    The default value of MaxNoOfFiredTriggers is 4000, which is sufficient for most situations. In some cases it can even be decreased if the DBA feels certain the need for parallelism in the cluster is not high.

    A record is created when an operation is performed that affects a unique hash index. Inserting or deleting a record in a table with unique hash indexes or updating a column that is part of a unique hash index fires an insert or a delete in the index table. The resulting record is used to represent this index table operation while waiting for the original operation that fired it to complete. This operation is short-lived but can still require a large number of records in its pool for situations with many parallel write operations on a base table containing a set of unique hash indexes.

  • TransactionBufferMemory

    Restart Typenode
    Permitted Values
    Typenumeric
    Default1M
    Range1K-4G

    The memory affected by this parameter is used for tracking operations fired when updating index tables and reading unique indexes. This memory is used to store the key and column information for these operations. It is only very rarely that the value for this parameter needs to be altered from the default.

    The default value for TransactionBufferMemory is 1MB.

    Normal read and write operations use a similar buffer, whose usage is even more short-lived. The compile-time parameter ZATTRBUF_FILESIZE (found in ndb/src/kernel/blocks/Dbtc/Dbtc.hpp) set to 4000 × 128 bytes (500KB). A similar buffer for key information, ZDATABUF_FILESIZE (also in Dbtc.hpp) contains 4000 × 16 = 62.5KB of buffer space. Dbtc is the module that handles transaction coordination.

Scans and buffering. There are additional [ndbd] parameters in the Dblqh module (in ndb/src/kernel/blocks/Dblqh/Dblqh.hpp) that affect reads and updates. These include ZATTRINBUF_FILESIZE, set by default to 10000 × 128 bytes (1250KB) and ZDATABUF_FILE_SIZE, set by default to 10000*16 bytes (roughly 156KB) of buffer space. To date, there have been neither any reports from users nor any results from our own extensive tests suggesting that either of these compile-time limits should be increased.

  • MaxNoOfConcurrentScans

    Restart Typenode
    Permitted Values
    Typenumeric
    Default256
    Range2-500

    This parameter is used to control the number of parallel scans that can be performed in the cluster. Each transaction coordinator can handle the number of parallel scans defined for this parameter. Each scan query is performed by scanning all partitions in parallel. Each partition scan uses a scan record in the node where the partition is located, the number of records being the value of this parameter times the number of nodes. The cluster should be able to sustain MaxNoOfConcurrentScans scans concurrently from all nodes in the cluster.

    Scans are actually performed in two cases. The first of these cases occurs when no hash or ordered indexes exists to handle the query, in which case the query is executed by performing a full table scan. The second case is encountered when there is no hash index to support the query but there is an ordered index. Using the ordered index means executing a parallel range scan. The order is kept on the local partitions only, so it is necessary to perform the index scan on all partitions.

    The default value of MaxNoOfConcurrentScans is 256. The maximum value is 500.

  • MaxNoOfLocalScans

    Restart Typenode
    Permitted Values
    Typenumeric
    DefaultUNDEFINED
    Range32-4G

    Specifies the number of local scan records if many scans are not fully parallelized. If the number of local scan records is not provided, it is calculated as the product of MaxNoOfConcurrentScans and the number of data nodes in the system. The minimum value is 32.

  • BatchSizePerLocalScan

    Restart Typenode
    Permitted Values
    Typenumeric
    Default64
    Range1-992

    This parameter is used to calculate the number of lock records used to handle concurrent scan operations.

    The default value is 64; this value has a strong connection to the ScanBatchSize defined in the SQL nodes.

  • LongMessageBuffer

    Restart Typenode
    Permitted Values
    Typenumeric
    Default1M
    Range512K-4G
    Permitted Values
    Typenumeric
    Default4M
    Range512K-4G

    This is an internal buffer used for passing messages within individual nodes and between nodes. In MySQL Cluster NDB 6.4.3 and earlier, the default is 1MB; beginning with MySQL Cluster NDB 7.0.4, it is 4MB.

    This parameter seldom needs to be changed from the default. However, when replicating a MySQL Cluster using ndbmtd for the data nodes, you may need to increase this value to 8MB (or possibly more) to prevent data node instability, because ndbmtd uses much more of this resource than ndbd does. Beginning with MySQL Cluster NDB 7.0.13 and MySQL Cluster NDB 7.1.2, this should no longer be necessary when using ndbmtd with MySQL Cluster Replication (Bug#46914).

Memory Allocation

MaxAllocate

This is the maximum size of the memory unit to use when allocating memory for tables. In cases where NDB gives Out of memory errors, but it is evident by examining the cluster logs or the output of DUMP 1000 (see DUMP 1000) that all available memory has not yet been used, you can increase the value of this parameter (or MaxNoOfTables, or both) to cause NDB to make sufficient memory available.

This parameter was introduced in MySQL 5.1.20, MySQL Cluster NDB 6.1.12 and MySQL Cluster NDB 6.2.3.

Logging and checkpointing. The following [ndbd] parameters control log and checkpoint behavior.

  • NoOfFragmentLogFiles

    Restart Typeinitial, node
    Permitted Values (>= 5.1.0)
    Typenumeric
    Default16
    Range3-4G

    This parameter sets the number of REDO log files for the node, and thus the amount of space allocated to REDO logging. Because the REDO log files are organized in a ring, it is extremely important that the first and last log files in the set (sometimes referred to as the “head” and “tail” log files, respectively) do not meet. When these approach one another too closely, the node begins aborting all transactions encompassing updates due to a lack of room for new log records.

    A REDO log record is not removed until the required number of local checkpoints has been completed since that log record was inserted (prior to MySQL Cluster NDB 6.3.8, this was 3 local checkpoints; in later versions of MySQL Cluster, only 2 local checkpoints are necessary). Checkpointing frequency is determined by its own set of configuration parameters discussed elsewhere in this chapter.

    How these parameters interact and proposals for how to configure them are discussed in Section 17.3.2.12, “Configuring MySQL Cluster Parameters for Local Checkpoints”.

    The default parameter value is 16, which by default means 16 sets of 4 16MB files for a total of 1024MB. Beginning with MySQL Cluster NDB 6.1.1, the size of the individual log files is configurable using the FragmentLogFileSize parameter. In scenarios requiring a great many updates, the value for NoOfFragmentLogFiles may need to be set as high as 300 or even higher to provide sufficient space for REDO logs.

    If the checkpointing is slow and there are so many writes to the database that the log files are full and the log tail cannot be cut without jeopardizing recovery, all updating transactions are aborted with internal error code 410 (Out of log file space temporarily). This condition prevails until a checkpoint has completed and the log tail can be moved forward.

    Important

    This parameter cannot be changed “on the fly”; you must restart the node using --initial. If you wish to change this value for all data nodes in a running cluster, you can do so using a rolling node restart (using --initial when starting each data node).

  • FragmentLogFileSize

    Version Introduced5.1.15-ndb-6.1.11
    Restart Typeinitial, node
    Permitted Values
    Typenumeric
    Default16M
    Range4M-1G

    Setting this parameter enables you to control directly the size of redo log files. This can be useful in situations when MySQL Cluster is operating under a high load and it is unable to close fragment log files quickly enough before attempting to open new ones (only 2 fragment log files can be open at one time); increasing the size of the fragment log files gives the cluster more time before having to open each new fragment log file. The default value for this parameter is 16M. FragmentLogFileSize was added in MySQL Cluster NDB 6.1.11.

    For more information about fragment log files, see the description for NoOfFragmentLogFiles.

  • InitFragmentLogFiles

    Version Introduced5.1.29-ndb-6.3.19
    Restart Typeinitial, node
    Permitted Values
    Typestring
    Default
    Range-

    By default, fragment log files are created sparsely when performing an initial start of a data node—that is, depending on the operating system and file system in use, not all bytes are necessarily written to disk. Beginning with MySQL Cluster NDB 6.3.19, it is possible to override this behavior and force all bytes to be written regardless of the platform and file system type being used by mean of this parameter.

    InitFragmentLogFiles takes one of two values:

    • SPARSE. Fragment log files are created sparsely. This is the default value.

    • FULL. Force all bytes of the fragment log file to be written to disk.

    Depending on your operating system and file system, setting InitFragmentLogFiles=FULL may help eliminate I/O errors on writes to the REDO log.

  • MaxNoOfOpenFiles

    Restart Typenode
    Permitted Values (<= 5.1.15)
    Typenumeric
    Default40
    Range20-4G
    Permitted Values (>= 5.1.16)
    Typenumeric
    Default0
    Range20-4G

    This parameter sets a ceiling on how many internal threads to allocate for open files. Any situation requiring a change in this parameter should be reported as a bug.

    The default value is 0. (Prior to MySQL 5.1.16, the default was 40.) However, the minimum value to which this parameter can be set is 20.

  • InitialNoOfOpenFiles

    Version Introduced5.1.9
    Restart Typenode
    Permitted Values (>= 5.1.9)
    Typenumeric
    Default27
    Range20-4G

    This parameter sets the initial number of internal threads to allocate for open files.

    The default value is 27.

  • MaxNoOfSavedMessages

    Restart Typenode
    Permitted Values
    Typenumeric
    Default25
    Range0-4G

    This parameter sets the maximum number of trace files that are kept before overwriting old ones. Trace files are generated when, for whatever reason, the node crashes.

    The default is 25 trace files.

  • MaxLCPStartDelay

    Version Introduced5.1.32-ndb-6.4.3
    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-600
    Permitted Values
    Typenumeric
    Default0
    Range0-600

    In parallel data node recovery (supported in MySQL Cluster NDB 6.3.8 and later), only table data is actually copied and synchronized in parallel; synchronization of metadata such as dictionary and checkpoint information is done in a serial fashion. In addition, recovery of dictionary and checkpoint information cannot be executed in parallel with performing of local checkpoints. This means that, when starting or restarting many data nodes concurrently, data nodes may be forced to wait while a local checkpoint is performed, which can result in longer node recovery times.

    Beginning with MySQL Cluster NDB 6.3.23 and MySQL Cluster NDB 6.4.3, it is possible to force a delay in the local checkpoint to permit more (and possibly all) data nodes to complete metadata synchronization; once each data node's metadata synchronization is complete, all of the data nodes can recover table data in parallel, even while the local checkpoint is being executed.

    To force such a delay, you can set MaxLCPStartDelay, which determines the number of seconds the cluster can wait to begin a local checkpoint while data nodes continue to synchronize metadata. This parameter should be set in the [ndbd default] section of the config.ini file, so that it is the same for all data nodes. The maximum value is 600; the default is 0.

Metadata objects. The next set of [ndbd] parameters defines pool sizes for metadata objects, used to define the maximum number of attributes, tables, indexes, and trigger objects used by indexes, events, and replication between clusters. Note that these act merely as “suggestions” to the cluster, and any that are not specified revert to the default values shown.

  • MaxNoOfAttributes

    Restart Typenode
    Permitted Values
    Typenumeric
    Default1000
    Range32-4G

    Defines the number of attributes that can be defined in the cluster.

    The default value is 1000, with the minimum possible value being 32. The maximum is 4294967039. Each attribute consumes around 200 bytes of storage per node due to the fact that all metadata is fully replicated on the servers.

    When setting MaxNoOfAttributes, it is important to prepare in advance for any ALTER TABLE statements that you might want to perform in the future. This is due to the fact, during the execution of ALTER TABLE on a Cluster table, 3 times the number of attributes as in the original table are used, and a good practice is to permit double this amount. For example, if the MySQL Cluster table having the greatest number of attributes (greatest_number_of_attributes) has 100 attributes, a good starting point for the value of MaxNoOfAttributes would be 6 * greatest_number_of_attributes = 600.

    You should also estimate the average number of attributes per table and multiply this by MaxNoOfTables. If this value is larger than the value obtained in the previous paragraph, you should use the larger value instead.

    Assuming that you can create all desired tables without any problems, you should also verify that this number is sufficient by trying an actual ALTER TABLE after configuring the parameter. If this is not successful, increase MaxNoOfAttributes by another multiple of MaxNoOfTables and test it again.

  • MaxNoOfTables

    Restart Typenode
    Permitted Values
    Typenumeric
    Default128
    Range8-20320

    A table object is allocated for each table and for each unique hash index in the cluster. This parameter sets the maximum number of table objects for the cluster as a whole.

    For each attribute that has a BLOB data type an extra table is used to store most of the BLOB data. These tables also must be taken into account when defining the total number of tables.

    The default value of this parameter is 128. The minimum is 8 and the maximum is 20320. Each table object consumes approximately 20KB per node.

    Note

    The sum of MaxNoOfTables, MaxNoOfOrderedIndexes, and MaxNoOfUniqueHashIndexes must not exceed 232 – 2 (4294967294).

  • MaxNoOfOrderedIndexes

    Restart Typenode
    Permitted Values
    Typenumeric
    Default128
    Range0-4G

    For each ordered index in the cluster, an object is allocated describing what is being indexed and its storage segments. By default, each index so defined also defines an ordered index. Each unique index and primary key has both an ordered index and a hash index. MaxNoOfOrderedIndexes sets the total number of hash indexes that can be in use in the system at any one time.

    The default value of this parameter is 128. Each hash index object consumes approximately 10KB of data per node.

    Note

    The sum of MaxNoOfTables, MaxNoOfOrderedIndexes, and MaxNoOfUniqueHashIndexes must not exceed 232 – 2 (4294967294).

  • MaxNoOfUniqueHashIndexes

    Restart Typenode
    Permitted Values
    Typenumeric
    Default64
    Range0-4G

    For each unique index that is not a primary key, a special table is allocated that maps the unique key to the primary key of the indexed table. By default, an ordered index is also defined for each unique index. To prevent this, you must specify the USING HASH option when defining the unique index.

    The default value is 64. Each index consumes approximately 15KB per node.

    Note

    The sum of MaxNoOfTables, MaxNoOfOrderedIndexes, and MaxNoOfUniqueHashIndexes must not exceed 232 – 2 (4294967294).

  • MaxNoOfTriggers

    Restart Typenode
    Permitted Values
    Typenumeric
    Default768
    Range0-4G

    Internal update, insert, and delete triggers are allocated for each unique hash index. (This means that three triggers are created for each unique hash index.) However, an ordered index requires only a single trigger object. Backups also use three trigger objects for each normal table in the cluster.

    Replication between clusters also makes use of internal triggers.

    This parameter sets the maximum number of trigger objects in the cluster.

    The default value is 768.

  • MaxNoOfIndexes

    This parameter is deprecated in MySQL Cluster 5.1 and later; you should use MaxNoOfOrderedIndexes and MaxNoOfUniqueHashIndexes instead.

    This parameter is used only by unique hash indexes. There needs to be one record in this pool for each unique hash index defined in the cluster.

    The default value of this parameter is 128.

  • MaxNoOfSubscriptions

    Version Introduced5.1.23-ndb-6.3.7
    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-4G
    Permitted Values
    Typenumeric
    Default0
    Range0-4G

    Each NDB table in a MySQL Cluster requires a subscription in the NDB kernel. For some NDB API applications, it may be necessary or desirable to change this parameter, which became available in MySQL Cluster NDB 6.2.10 and MySQL Cluster NDB 6.3.7. However, for normal usage with MySQL servers acting as SQL nodes, there is not any need to do so.

    The default value for MaxNoOfSubscriptions is 0, which is treated as equal to MaxNoOfTables.

  • MaxNoOfSubscribers

    Version Introduced5.1.23-ndb-6.3.7
    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-4G
    Permitted Values
    Typenumeric
    Default0
    Range0-4G

    This parameter, added in MySQL Cluster NDB 6.2.10 and MySQL Cluster NDB 6.3.7, is of interest only when using MySQL Cluster Replication. The default value is 0, which is treated as 2 * MaxNoOfTables; that is, there is one subscription per NDB table for each of two MySQL servers (one acting as the replication master and the other as the slave).

    When using circular replication, multi-master replcation, and other replication setups involving more than 2 MySQL servers, you should increase this parameter to the number of mysqld processes included in replication (this is often, but not always, the same as the number of clusters). For example, if you have a circular replication setup using three MySQL Clusters, with one mysqld attached to each cluster, and each of these mysqld processes acts as a master and as a slave, you should set MaxNoOfSubscribers equal to 3 * MaxNoOfTables.

    For more information, see Section 17.6, “MySQL Cluster Replication”.

  • MaxNoOfConcurrentSubOperations

    Version Introduced5.1.23-ndb-6.3.7
    Restart Typenode
    Permitted Values
    Typenumeric
    Default256
    Range0-4G
    Permitted Values
    Typenumeric
    Default256
    Range0-4G

    This parameter sets a ceiling on the number of operations that can be performed by all API nodes in the cluster at one time. The default value (256) is sufficient for normal operations, and might need to be adjusted only in scenarios where there are a great many API nodes each performing a high volume of operations concurrently.

    This parameter was added in MySQL Cluster NDB 6.2.10 and MySQL Cluster NDB 6.3.7.

Boolean parameters. The behavior of data nodes is also affected by a set of [ndbd] parameters taking on boolean values. These parameters can each be specified as TRUE by setting them equal to 1 or Y, and as FALSE by setting them equal to 0 or N.

  • LockPagesInMainMemory

    Restart Typenode
    Permitted Values (>= 5.1.0, <= 5.1.14)
    Typeboolean
    Default0
    Range0-1
    Permitted Values (>= 5.1.15)
    Typenumeric
    Default0
    Range0-2

    For a number of operating systems, including Solaris and Linux, it is possible to lock a process into memory and so avoid any swapping to disk. This can be used to help guarantee the cluster's real-time characteristics.

    Beginning with MySQL 5.1.15 and MySQL Cluster NDB 6.1.1, this parameter takes one of the integer values 0, 1, or 2, which act as follows:

    • 0: Disables locking. This is the default value.

    • 1: Performs the lock after allocating memory for the process.

    • 2: Performs the lock before memory for the process is allocated.

    Previously, this parameter was a Boolean. 0 or false was the default setting, and disabled locking. 1 or true enabled locking of the process after its memory was allocated.

    Important

    Beginning with MySQL 5.1.15 and MySQL Cluster NDB 6.1.1, it is no longer possible to use true or false for the value of this parameter; when upgrading from a previous version, you must change the value to 0, 1, or 2.

    Prior to MySQL Cluster NDB 6.3.31 and MySQL Cluster NDB 7.0.11, setting this parameter did not cause the stated memory to be allocated when the node was started, but rather only when the memory was used by the data node process for other reasons. (Bug#37430)

    Note

    If the operating system is not configured to permit unprivileged users to lock pages, then the data node process making use of this parameter may have to be run as system root. (LockPagesInMainMemory uses the mlockall function. From Linux kernel 2.6.9, unprivileged users can lock memory as limited by max locked memory. For more information, see ulimit -l and http://linux.die.net/man/2/mlock).

  • StopOnError

    Restart Typenode
    Permitted Values
    Typeboolean
    Defaulttrue
    Range-

    This parameter specifies whether an ndbd process should exit or perform an automatic restart when an error condition is encountered.

    This feature is enabled by default.

  • Diskless

    Restart Typeinitial, system
    Permitted Values
    Typeboolean
    Default0
    Range0-1

    It is possible to specify MySQL Cluster tables as diskless, meaning that tables are not checkpointed to disk and that no logging occurs. Such tables exist only in main memory. A consequence of using diskless tables is that neither the tables nor the records in those tables survive a crash. However, when operating in diskless mode, it is possible to run ndbd on a diskless computer.

    Important

    This feature causes the entire cluster to operate in diskless mode.

    When this feature is enabled, Cluster online backup is disabled. In addition, a partial start of the cluster is not possible.

    Diskless is disabled by default.

  • ODirect

    Version Introduced5.1.19-ndb-6.3.0
    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-1
    Permitted Values
    Typenumeric
    Default0
    Range0-1
    Permitted Values
    Typenumeric
    Default0
    Range0-1

    Enabling this parameter causes NDB to attempt using O_DIRECT writes for LCP, backups, and redo logs, often lowering kswapd and CPU usage. When using MySQL Cluster on Linux, enable ODirect if you are using a 2.6 or later kernel.

    This parameter was added in MySQL 5.1.20, MySQL Cluster NDB 6.1.11, MySQL Cluster NDB 6.2.3, and MySQL Cluster NDB 6.3.0.

    ODirect is disabled by default.

  • RestartOnErrorInsert

    Restart Typenode
    Permitted Values
    Typenumeric
    Default2
    Range0-4

    This feature is accessible only when building the debug version where it is possible to insert errors in the execution of individual blocks of code as part of testing.

    This feature is disabled by default.

  • CompressedBackup

    Version Introduced5.1.23-ndb-6.3.7
    Restart Typenode
    Permitted Values
    Typeboolean
    Defaultfalse
    Range-

    Setting this parameter to 1 causes backup files to be compressed. The compression used is equivalent to gzip --fast, and can save 50% or more of the space required on the data node to store uncompressed backup files. Compressed backups can be enabled for individual data nodes, or for all data nodes (by setting this parameter in the [ndbd default] section of the config.ini file).

    Important

    You cannot restore a compressed backup to a cluster running a MySQL version that does not support this feature.

    The default value is 0 (disabled).

    This parameter was introduced in MySQL Cluster NDB 6.3.7.

  • CompressedLCP

    Version Introduced5.1.23-ndb-6.3.7
    Restart Typenode
    Permitted Values
    Typeboolean
    Defaultfalse
    Range-

    Setting this parameter to 1 causes local checkpoint files to be compressed. The compression used is equivalent to gzip --fast, and can save 50% or more of the space required on the data node to store uncompressed checkpoint files. Compressed LCPs can be enabled for individual data nodes, or for all data nodes (by setting this parameter in the [ndbd default] section of the config.ini file).

    Important

    You cannot restore a compressed local checkpoint to a cluster running a MySQL version that does not support this feature.

    The default value is 0 (disabled).

    This parameter was introduced in MySQL Cluster NDB 6.3.7.

Controlling Timeouts, Intervals, and Disk Paging

There are a number of [ndbd] parameters specifying timeouts and intervals between various actions in Cluster data nodes. Most of the timeout values are specified in milliseconds. Any exceptions to this are mentioned where applicable.

  • TimeBetweenWatchDogCheck

    Restart Typenode
    Permitted Values
    Typenumeric
    Default6000
    Range70-4G

    To prevent the main thread from getting stuck in an endless loop at some point, a “watchdog” thread checks the main thread. This parameter specifies the number of milliseconds between checks. If the process remains in the same state after three checks, the watchdog thread terminates it.

    This parameter can easily be changed for purposes of experimentation or to adapt to local conditions. It can be specified on a per-node basis although there seems to be little reason for doing so.

    The default timeout is 6000 milliseconds (6 seconds).

  • TimeBetweenWatchDogCheckInitial

    Version Introduced5.1.20
    Restart Typenode
    Permitted Values (>= 5.1.20)
    Typenumeric
    Default6000
    Range70-4G

    This is similar to the TimeBetweenWatchDogCheck parameter, except that TimeBetweenWatchDogCheckInitial controls the amount of time that passes between execution checks inside a database node in the early start phases during which memory is allocated.

    The default timeout is 6000 milliseconds (6 seconds).

    This parameter was added in MySQL 5.1.20.

  • StartPartialTimeout

    Restart Typenode
    Permitted Values
    Typenumeric
    Default30000
    Range0-4G

    This parameter specifies how long the Cluster waits for all data nodes to come up before the cluster initialization routine is invoked. This timeout is used to avoid a partial Cluster startup whenever possible.

    This parameter is overridden when performing an initial start or initial restart of the cluster.

    The default value is 30000 milliseconds (30 seconds). 0 disables the timeout, in which case the cluster may start only if all nodes are available.

  • StartPartitionedTimeout

    Restart Typenode
    Permitted Values
    Typenumeric
    Default60000
    Range0-4G

    If the cluster is ready to start after waiting for StartPartialTimeout milliseconds but is still possibly in a partitioned state, the cluster waits until this timeout has also passed. If StartPartitionedTimeout is set to 0, the cluster waits indefinitely.

    This parameter is overridden when performing an initial start or initial restart of the cluster.

    The default timeout is 60000 milliseconds (60 seconds).

  • StartFailureTimeout

    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-4G

    If a data node has not completed its startup sequence within the time specified by this parameter, the node startup fails. Setting this parameter to 0 (the default value) means that no data node timeout is applied.

    For nonzero values, this parameter is measured in milliseconds. For data nodes containing extremely large amounts of data, this parameter should be increased. For example, in the case of a data node containing several gigabytes of data, a period as long as 10–15 minutes (that is, 600000 to 1000000 milliseconds) might be required to perform a node restart.

  • HeartbeatIntervalDbDb

    Restart Typenode
    Permitted Values
    Typenumeric
    Default1500
    Range10-4G

    One of the primary methods of discovering failed nodes is by the use of heartbeats. This parameter states how often heartbeat signals are sent and how often to expect to receive them. After missing three heartbeat intervals in a row, the node is declared dead. Thus, the maximum time for discovering a failure through the heartbeat mechanism is four times the heartbeat interval.

    The default heartbeat interval is 1500 milliseconds (1.5 seconds). This parameter must not be changed drastically and should not vary widely between nodes. If one node uses 5000 milliseconds and the node watching it uses 1000 milliseconds, obviously the node will be declared dead very quickly. This parameter can be changed during an online software upgrade, but only in small increments.

  • HeartbeatIntervalDbApi

    Restart Typenode
    Permitted Values
    Typenumeric
    Default1500
    Range100-4G

    Each data node sends heartbeat signals to each MySQL server (SQL node) to ensure that it remains in contact. If a MySQL server fails to send a heartbeat in time it is declared “dead,” in which case all ongoing transactions are completed and all resources released. The SQL node cannot reconnect until all activities initiated by the previous MySQL instance have been completed. The three-heartbeat criteria for this determination are the same as described for HeartbeatIntervalDbDb.

    The default interval is 1500 milliseconds (1.5 seconds). This interval can vary between individual data nodes because each data node watches the MySQL servers connected to it, independently of all other data nodes.

  • HeartbeatOrder

    Version Introduced5.1.47-ndb-7.1.5
    Restart Typesystem
    Permitted Values
    Typenumeric
    Default0
    Range0-65535

    Data nodes send heartbeats to one another in a circular fashion whereby each data node monitors the previous one. If a heartbeat is not detected by a given data node, this node declares the previous data node in the circle “dead” (that is, no longer accessible by the cluster). The determination that a data node is dead is done globally; in other words; once a data node is declared dead, it is regarded as such by all nodes in the cluster.

    It is possible for heartbeats between data nodes residing on different hosts to be too slow compared to heartbeats between other pairs of nodes (for example, due to a very low heartbeat interval or temporary connection problem), such that a data node is declared dead, even though the node can still function as part of the cluster. .

    In this type of situation, it may be that the order in which heartbeats are transmitted between data nodes makes a difference as to whether or not a particular data node is declared dead. If this declaration occurs unnecessarily, this can in turn lead to the unnecessary loss of a node group and as thus to a failure of the cluster.

    Consider a setup where there are 4 data nodes A, B, C, and D running on 2 host computers host1 and host2, and that these data ndoes make up 2 node groups, as shown in the following table:

    host1host2
    Node Group 0:Node ANode B
    Node Group 1:Node CNode D

    Suppose the heartbeats are transmitted in the order A->B->C->D->A. In this case, the loss of the heartbeat between the hosts causes node B to declare node A dead and node C to declare node B dead. This results in loss of Node Group 0, and so the cluster fails. On the other hand, if the order of transmission is A->B->D->C->A (and all other conditions remain as previously stated), the loss of the heartbeat causes nodes A and D to be declared dead; in this case, each node group has one surviving node, and the cluster survives.

    Priot to MySQL Cluster NDB 6.3.35, MySQL Cluster NDB 7.0.16, and MySQL Cluster NDB 7.1.5, the order of of heartbeat transmission between data nodes was always automatically determined by NDB. However, beginning with these versions, the HeartbeatOrder configuration parameter makes the order of heartbeat transmission user-configurable.

    The default value for HeartbeatOrder is zero; allowing the default value to be used on all data nodes causes the order of heartbeat transmission to be determined by NDB. If this parameter is used, it must be set to a nonzero value (maximum 65535) for every data node in the cluster, and this value must be unique for each data node; this causes the heartbeat transmission to proceed from data node to data node in the order of their HeartbeatOrder values from lowest to highest (and then directly from the data node having the highest HeartbeatOrder to the data node having the lowest value, to complete the circle). The values need not be consecutive; for example, to force the heartbeat transmission order A->B->D->C->A in the scenario outlined previously, you could set the HeartbeatOrder values as shown here:

    NodeHeartbeatOrder
    A10
    B20
    C30
    D25

    To use this parameter to change the heartbeat transmission order in a running MySQL Cluster, you must first set HeartbeatOrder for each data node in the cluster in the global configuration (config.ini) file (or files). To cause the change to take effect, you must perform either of the following:

    • A complete shutdown and restart of the entire cluster.

    • 2 rolling restarts of the cluster in succession. All nodes must be restarted in the same order in both rolling restarts.

    You can use DUMP 908 to observe the effect of this parameter in the data node logs.

  • TimeBetweenLocalCheckpoints

    Restart Typenode
    Permitted Values
    Typenumeric
    Default20
    Range0-31

    This parameter is an exception in that it does not specify a time to wait before starting a new local checkpoint; rather, it is used to ensure that local checkpoints are not performed in a cluster where relatively few updates are taking place. In most clusters with high update rates, it is likely that a new local checkpoint is started immediately after the previous one has been completed.

    The size of all write operations executed since the start of the previous local checkpoints is added. This parameter is also exceptional in that it is specified as the base-2 logarithm of the number of 4-byte words, so that the default value 20 means 4MB (4 × 220) of write operations, 21 would mean 8MB, and so on up to a maximum value of 31, which equates to 8GB of write operations.

    All the write operations in the cluster are added together. Setting TimeBetweenLocalCheckpoints to 6 or less means that local checkpoints will be executed continuously without pause, independent of the cluster's workload.

  • TimeBetweenGlobalCheckpoints

    Restart Typenode
    Permitted Values
    Typenumeric
    Default2000
    Range10-32000

    When a transaction is committed, it is committed in main memory in all nodes on which the data is mirrored. However, transaction log records are not flushed to disk as part of the commit. The reasoning behind this behavior is that having the transaction safely committed on at least two autonomous host machines should meet reasonable standards for durability.

    It is also important to ensure that even the worst of cases—a complete crash of the cluster—is handled properly. To guarantee that this happens, all transactions taking place within a given interval are put into a global checkpoint, which can be thought of as a set of committed transactions that has been flushed to disk. In other words, as part of the commit process, a transaction is placed in a global checkpoint group. Later, this group's log records are flushed to disk, and then the entire group of transactions is safely committed to disk on all computers in the cluster.

    This parameter defines the interval between global checkpoints. The default is 2000 milliseconds.

  • TimeBetweenEpochs

    Version Introduced5.1.22-ndb-6.3.2
    Restart Typenode
    Permitted Values
    Typenumeric
    Default100
    Range0-32000
    Permitted Values
    Typenumeric
    Default100
    Range0-32000

    This parameter defines the interval between synchronisation epochs for MySQL Cluster Replication. The default value is 100 milliseconds.

    TimeBetweenEpochs is part of the implementation of “micro-GCPs”, which can be used to improve the performance of MySQL Cluster Replication. This parameter was introduced in MySQL Cluster NDB 6.2.5 and MySQL Cluster NDB 6.3.2.

  • TimeBetweenEpochsTimeout

    Version Introduced5.1.22-ndb-6.3.4
    Restart Typenode
    Permitted Values
    Typenumeric
    Default4000
    Range0-32000
    Permitted Values
    Typenumeric
    Default4000
    Range0-32000

    This parameter defines a timeout for synchronization epochs for MySQL Cluster Replication. If a node fails to participate in a global checkpoint within the time determined by this parameter, the node is shut down. The default value is 4000 milliseconds.

    TimeBetweenEpochsTimeout is part of the implementation of “micro-GCPs”, which can be used to improve the performance of MySQL Cluster Replication. This parameter was introduced in MySQL Cluster NDB 6.2.7 and MySQL Cluster NDB 6.3.4.

  • MaxBufferedEpochs

    Version Introduced5.1.23-ndb-6.2.14
    Restart Typenode
    Permitted Values
    Typenumeric
    Default100
    Range0-100000

    The number of unprocessed epochs by which a subscribing node can lag behind. Exceeding this number causes a lagging subscriber to be disconnected.

    The default value of 100 is sufficient for most normal operations. If a subscribing node does lag enough to cause disconnections, it is usually due to network or scheduling issues with regard to processes or threads. (In rare circumstances, the problem may be due to a bug in the NDB client.) It may be desirable to set the value lower than the default when epochs are longer.

    Disconnection prevents client issues from affecting the data node service, running out of memory to buffer data, and eventually shutting down. Instead, only the client is affected as a result of the disconnect (by, for example gap events in the binlog), forcing the client to reconnect or restart the process.

  • TimeBetweenInactiveTransactionAbortCheck

    Restart Typenode
    Permitted Values
    Typenumeric
    Default1000
    Range1000-4G

    Timeout handling is performed by checking a timer on each transaction once for every interval specified by this parameter. Thus, if this parameter is set to 1000 milliseconds, every transaction will be checked for timing out once per second.

    The default value is 1000 milliseconds (1 second).

  • TransactionInactiveTimeout

    Restart Typenode
    Permitted Values
    Typenumeric
    Default4G
    Range0-4G

    This parameter states the maximum time that is permitted to lapse between operations in the same transaction before the transaction is aborted.

    The default for this parameter is zero (no timeout). For a real-time database that needs to ensure that no transaction keeps locks for too long, this parameter should be set to a relatively small value. The unit is milliseconds.

  • TransactionDeadlockDetectionTimeout

    Restart Typenode
    Permitted Values
    Typenumeric
    Default1200
    Range50-4G

    When a node executes a query involving a transaction, the node waits for the other nodes in the cluster to respond before continuing. A failure to respond can occur for any of the following reasons:

    • The node is “dead

    • The operation has entered a lock queue

    • The node requested to perform the action could be heavily overloaded.

    This timeout parameter states how long the transaction coordinator waits for query execution by another node before aborting the transaction, and is important for both node failure handling and deadlock detection. In MySQL 5.1.10 and earlier versions, setting it too high could cause undesirable behavior in situations involving deadlocks and node failure. Beginning with MySQL 5.1.11, active transactions occurring during node failures are actively aborted by the MySQL Cluster Transaction Coordinator, and so high settings are no longer an issue with this parameter.

    The default timeout value is 1200 milliseconds (1.2 seconds).

    Prior to MySQL Cluster NDB versions 6.2.18, 6.3.24, and 7.0.5, the effective minimum for this parameter was 100 milliseconds. (Bug#44099) Beginning with these versions, the actual minimum is 50 milliseconds.

  • DiskSyncSize

    Version Introduced5.1.12
    Restart Typenode
    Permitted Values
    Typenumeric
    Default4M
    Range32K-4G

    This is the maximum number of bytes to store before flushing data to a local checkpoint file. This is done to prevent write buffering, which can impede performance significantly. This parameter is not intended to take the place of TimeBetweenLocalCheckpoints.

    Note

    When ODirect is enabled, it is not necessary to set DiskSyncSize; in fact, in such cases its value is simply ignored.

    The default value is 4M (4 megabytes).

    This parameter was added in MySQL 5.1.12.

  • DiskCheckpointSpeed

    Version Introduced5.1.12
    Restart Typenode
    Permitted Values
    Typenumeric
    Default10M
    Range1M-4G

    The amount of data,in bytes per second, that is sent to disk during a local checkpoint. This allocation is shared by DML operations and backups (but not backup logging), which means that backups started during times of intensive DML may be impaired by flooding of the redo log buffer and may fail altogether if the contention is sufficiently severe.

    The default value is 10M (10 megabytes per second).

    This parameter was added in MySQL 5.1.12.

  • DiskCheckpointSpeedInRestart

    Version Introduced5.1.12
    Restart Typenode
    Permitted Values
    Typenumeric
    Default100M
    Range1M-4G

    The amount of data,in bytes per second, that is sent to disk during a local checkpoint as part of a restart operation.

    The default value is 100M (100 megabytes per second).

    This parameter was added in MySQL 5.1.12.

  • NoOfDiskPagesToDiskAfterRestartTUP

    Version Removed5.1.6
    Restart Typenode
    Permitted Values (<= 5.1.6)
    Typenumeric
    Default40
    Range1-4G

    When executing a local checkpoint, the algorithm flushes all data pages to disk. Merely doing so as quickly as possible without any moderation is likely to impose excessive loads on processors, networks, and disks. To control the write speed, this parameter specifies how many pages per 100 milliseconds are to be written. In this context, a “page” is defined as 8KB. This parameter is specified in units of 80KB per second, so setting NoOfDiskPagesToDiskAfterRestartTUP to a value of 20 entails writing 1.6MB in data pages to disk each second during a local checkpoint. This value includes the writing of UNDO log records for data pages. That is, this parameter handles the limitation of writes from data memory. (See the entry for IndexMemory for information about index pages.)

    In short, this parameter specifies how quickly to execute local checkpoints. It operates in conjunction with NoOfFragmentLogFiles, DataMemory, and IndexMemory.

    For more information about the interaction between these parameters and possible strategies for choosing appropriate values for them, see Section 17.3.2.12, “Configuring MySQL Cluster Parameters for Local Checkpoints”.

    The default value is 40 (3.2MB of data pages per second).

    Note

    This parameter is deprecated as of MySQL 5.1.6. For MySQL 5.1.12 and later versions, use DiskCheckpointSpeed and DiskSyncSize instead.

  • NoOfDiskPagesToDiskAfterRestartACC

    Version Removed5.1.6
    Restart Typenode
    Permitted Values (<= 5.1.6)
    Typenumeric
    Default20
    Range1-4G

    This parameter uses the same units as NoOfDiskPagesToDiskAfterRestartTUP and acts in a similar fashion, but limits the speed of writing index pages from index memory.

    The default value of this parameter is 20 (1.6MB of index memory pages per second).

    Note

    This parameter is deprecated as of MySQL 5.1.6. For MySQL 5.1.12 and later versions, use DiskCheckpointSpeed and DiskSyncSize.

  • NoOfDiskPagesToDiskDuringRestartTUP (DEPRECATED)

    Version Removed5.1.6
    Restart Typenode
    Permitted Values (<= 5.1.6)
    Typenumeric
    Default40
    Range1-4G

    This parameter is used in a fashion similar to NoOfDiskPagesToDiskAfterRestartTUP and NoOfDiskPagesToDiskAfterRestartACC, only it does so with regard to local checkpoints executed in the node when a node is restarting. A local checkpoint is always performed as part of all node restarts. During a node restart it is possible to write to disk at a higher speed than at other times, because fewer activities are being performed in the node.

    This parameter covers pages written from data memory.

    The default value is 40 (3.2MB per second).

    Note

    This parameter is deprecated as of MySQL 5.1.6. For MySQL 5.1.12 and later versions, use DiskCheckpointSpeedInRestart and DiskSyncSize.

  • NoOfDiskPagesToDiskDuringRestartACC (DEPRECATED)

    Version Removed5.1.6
    Restart Typenode
    Permitted Values (<= 5.1.6)
    Typenumeric
    Default20
    Range1-4G

    Controls the number of index memory pages that can be written to disk during the local checkpoint phase of a node restart.

    As with NoOfDiskPagesToDiskAfterRestartTUP and NoOfDiskPagesToDiskAfterRestartACC, values for this parameter are expressed in terms of 8KB pages written per 100 milliseconds (80KB/second).

    The default value is 20 (1.6MB per second).

    Note

    This parameter is deprecated as of MySQL 5.1.6. For MySQL 5.1.12 and later versions, use DiskCheckpointSpeedInRestart and DiskSyncSize.

  • ArbitrationTimeout (DEPRECATED)

    Restart Typenode
    Permitted Values
    Typenumeric
    Default1000
    Range10-4G

    This parameter specifies how long data nodes wait for a response from the arbitrator to an arbitration message. If this is exceeded, the network is assumed to have split.

    The default value is 1000 milliseconds (1 second).

  • Arbitration

    Version Introduced5.1.35-ndb-7.0.7
    Restart Typenode
    Permitted Values
    Typeenumeration
    DefaultDefault
    Valid ValuesDefault, Disabled, WaitExternal

    The Arbitration parameter, added in MySQL Cluster NDB 7.0.7, enables a choice of arbitration schemes, corresponding to one of 3 possible values for this parameter:

    • Default. This enables arbitration to proceed normally, as determined by the ArbitrationRank settings for the management and API nodes. This is the default value.

    • Disabled. Previously, it was possible to disable arbitration only by setting ArbitrationRank to 0 on all management and API nodes. Now, you can now use Arbitration = Disabled in the [ndbd default] section of the config.ini file to accomplish this task. In this case, any ArbitrationRank settings are ignored.

    • WaitExternal. The Arbitration parameter also makes it possible to configure arbitration in such a way that the cluster waits until after the time determined by ArbitrationTimeout has passed for an external cluster manager application to perform arbitration instead of handling arbitration internally. This can be done by setting Arbitration = WaitExternal in the [ndbd default] section of the config.ini file. For best results with the WaitExternal setting, it is recommended that ArbitrationTimeout be 2 times as long as the interval required by the external cluster manager to perform arbitration.

    Important

    This parameter should be used only in the [ndbd default] section of the cluster configuration file. The behavior of the cluster is unspecified when Arbitration is set to different values for individual data nodes.

Buffering and logging. Several [ndbd] configuration parameters enable the advanced user to have more control over the resources used by node processes and to adjust various buffer sizes at need.

These buffers are used as front ends to the file system when writing log records to disk. If the node is running in diskless mode, these parameters can be set to their minimum values without penalty due to the fact that disk writes are “faked” by the NDB storage engine's file system abstraction layer.

  • UndoIndexBuffer

    Restart Typenode
    Permitted Values
    Typenumeric
    Default2M
    Range1M-4G

    The UNDO index buffer, whose size is set by this parameter, is used during local checkpoints. The NDB storage engine uses a recovery scheme based on checkpoint consistency in conjunction with an operational REDO log. To produce a consistent checkpoint without blocking the entire system for writes, UNDO logging is done while performing the local checkpoint. UNDO logging is activated on a single table fragment at a time. This optimization is possible because tables are stored entirely in main memory.

    The UNDO index buffer is used for the updates on the primary key hash index. Inserts and deletes rearrange the hash index; the NDB storage engine writes UNDO log records that map all physical changes to an index page so that they can be undone at system restart. It also logs all active insert operations for each fragment at the start of a local checkpoint.

    Reads and updates set lock bits and update a header in the hash index entry. These changes are handled by the page-writing algorithm to ensure that these operations need no UNDO logging.

    This buffer is 2MB by default. The minimum value is 1MB, which is sufficient for most applications. For applications doing extremely large or numerous inserts and deletes together with large transactions and large primary keys, it may be necessary to increase the size of this buffer. If this buffer is too small, the NDB storage engine issues internal error code 677 (Index UNDO buffers overloaded).

    Important

    It is not safe to decrease the value of this parameter during a rolling restart.

  • UndoDataBuffer

    Restart Typenode
    Permitted Values
    Typenumeric
    Default16M
    Range1M-4G

    This parameter sets the size of the UNDO data buffer, which performs a function similar to that of the UNDO index buffer, except the UNDO data buffer is used with regard to data memory rather than index memory. This buffer is used during the local checkpoint phase of a fragment for inserts, deletes, and updates.

    Because UNDO log entries tend to grow larger as more operations are logged, this buffer is also larger than its index memory counterpart, with a default value of 16MB.

    This amount of memory may be unnecessarily large for some applications. In such cases, it is possible to decrease this size to a minimum of 1MB.

    It is rarely necessary to increase the size of this buffer. If there is such a need, it is a good idea to check whether the disks can actually handle the load caused by database update activity. A lack of sufficient disk space cannot be overcome by increasing the size of this buffer.

    If this buffer is too small and gets congested, the NDB storage engine issues internal error code 891 (Data UNDO buffers overloaded).

    Important

    It is not safe to decrease the value of this parameter during a rolling restart.

  • RedoBuffer

    Restart Typenode
    Permitted Values
    Typenumeric
    Default8M
    Range1M-4G

    All update activities also need to be logged. The REDO log makes it possible to replay these updates whenever the system is restarted. The NDB recovery algorithm uses a “fuzzy” checkpoint of the data together with the UNDO log, and then applies the REDO log to play back all changes up to the restoration point.

    RedoBuffer sets the size of the buffer in which the REDO log is written. In MySQL Cluster NDB 6.4.3 and earlier, the default value is 8MB; beginning with MySQL Cluster NDB 7.0.4, the default is 32MB. The minimum value is 1MB.

    If this buffer is too small, the NDB storage engine issues error code 1221 (REDO log buffers overloaded). For this reason, you should exercise care if you attempt to decrease the value of RedoBuffer as part of an online change in the cluster's configuration.

Controlling log messages. In managing the cluster, it is very important to be able to control the number of log messages sent for various event types to stdout. For each event category, there are 16 possible event levels (numbered 0 through 15). Setting event reporting for a given event category to level 15 means all event reports in that category are sent to stdout; setting it to 0 means that there will be no event reports made in that category.

By default, only the startup message is sent to stdout, with the remaining event reporting level defaults being set to 0. The reason for this is that these messages are also sent to the management server's cluster log.

An analogous set of levels can be set for the management client to determine which event levels to record in the cluster log.

  • LogLevelStartup

    Restart Typenode
    Permitted Values
    Typenumeric
    Default1
    Range0-15

    The reporting level for events generated during startup of the process.

    The default level is 1.

  • LogLevelShutdown

    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-15

    The reporting level for events generated as part of graceful shutdown of a node.

    The default level is 0.

  • LogLevelStatistic

    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-15

    The reporting level for statistical events such as number of primary key reads, number of updates, number of inserts, information relating to buffer usage, and so on.

    The default level is 0.

  • LogLevelCheckpoint

    Restart Typeinitial, node
    Permitted Values
    Typenumeric
    Default0
    Range0-15

    The reporting level for events generated by local and global checkpoints.

    The default level is 0.

  • LogLevelNodeRestart

    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-15

    The reporting level for events generated during node restart.

    The default level is 0.

  • LogLevelConnection

    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-15

    The reporting level for events generated by connections between cluster nodes.

    The default level is 0.

  • LogLevelError

    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-15

    The reporting level for events generated by errors and warnings by the cluster as a whole. These errors do not cause any node failure but are still considered worth reporting.

    The default level is 0.

  • LogLevelCongestion

    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-15

    The reporting level for events generated by congestion. These errors do not cause node failure but are still considered worth reporting.

    The default level is 0.

  • LogLevelInfo

    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-15

    The reporting level for events generated for information about the general state of the cluster.

    The default level is 0.

  • MemReportFrequency

    Version Introduced5.1.16
    Restart Typenode
    Permitted Values (>= 5.1.16)
    Typenumeric
    Default0
    Range0-4G

    This parameter controls how often data node memory usage reports are recorded in the cluster log; it is an integer value representing the number of seconds between reports.

    Each data node's data memory and index memory usage is logged as both a percentage and a number of 32 KB pages of the DataMemory and IndexMemory, respectively, set in the config.ini file. For example, if DataMemory is equal to 100 MB, and a given data node is using 50 MB for data memory storage, the corresponding line in the cluster log might look like this:

    2006-12-24 01:18:16 [MgmSrvr] INFO -- Node 2: Data usage is 50%(1280 32K pages of total 2560)
    

    MemReportFrequency is not a required parameter. If used, it can be set for all cluster data nodes in the [ndbd default] section of config.ini, and can also be set or overridden for individual data nodes in the corresponding [ndbd] sections of the configuration file. The minimum value—which is also the default value—is 0, in which case memory reports are logged only when memory usage reaches certain percentages (80%, 90%, and 100%), as mentioned in the discussion of statistics events in Section 17.5.4.2, “MySQL Cluster Log Events”.

    This parameter was added in MySQL Cluster 5.1.16 and MySQL Cluster NDB 6.1.0.

  • StartupStatusReportFrequency

    Version Introduced5.1.30-ndb-6.4.0
    Restart Typenode
    Permitted Values
    Typenumeric

    When a data node is started with the --initial, it initializes the redo log file during Start Phase 4 (see Section 17.5.1, “Summary of MySQL Cluster Start Phases”). When very large values are set for NoOfFragmentLogFiles, FragmentLogFileSize, or both, this initialization can take a long time. Previous to MySQL Cluster NDB 6.4.0, only the beginning and end of the redo log file initialization process were logged. Beginning with this version, it is possible to force reports on the progress of this process to be logged periodically, by means of the StartupStatusReportFrequency configuration parameter. In this case, progress is reported in the cluster log, in terms of both the number of files and the amount of space that have been initialized, as shown here:

    2009-06-20 16:39:23 [MgmSrvr] INFO -- Node 1: Local redo log file initialization status:
    #Total files: 80, Completed: 60
    #Total MBytes: 20480, Completed: 15557
    2009-06-20 16:39:23 [MgmSrvr] INFO -- Node 2: Local redo log file initialization status:
    #Total files: 80, Completed: 60
    #Total MBytes: 20480, Completed: 15570
    

    These reports are logged each StartupStatusReportFrequency seconds during Start Pahe 4. If StartupStatusReportFrequency is 0 (the default), then reports are written to the cluster log only when at the beginning and at the completion of the redo log file initialization process.

Debugging Parameters. Beginning with MySQL Cluster NDB 6.3.36, MySQL Cluster NDB 7.0.17, and MySQL Cluster NDB 7.1.6, it is possible to cause logging of traces for events generated by creating and dropping tables using DictTrace. This parameter is useful only in debugging NDB kernel code. DictTrace takes an integer value; currently, 0 (default - no logging) and 1 (logging enabled) are the only supported values.

Backup parameters. The [ndbd] parameters discussed in this section define memory buffers set aside for execution of online backups.

  • BackupDataBufferSize

    Restart Typenode
    Permitted Values
    Typenumeric
    Default2M
    Range0-4G

    In creating a backup, there are two buffers used for sending data to the disk. The backup data buffer is used to fill in data recorded by scanning a node's tables. Once this buffer has been filled to the level specified as BackupWriteSize, the pages are sent to disk. While flushing data to disk, the backup process can continue filling this buffer until it runs out of space. When this happens, the backup process pauses the scan and waits until some disk writes have completed freed up memory so that scanning may continue.

    In MySQL Cluster NDB 6.4.3 and earlier, the default value is 2MB; in MySQL Cluster NDB 7.0.4 and later, it is 16MB.

  • BackupLogBufferSize

    Restart Typenode
    Permitted Values
    Typenumeric
    Default2M
    Range0-4G

    The backup log buffer fulfills a role similar to that played by the backup data buffer, except that it is used for generating a log of all table writes made during execution of the backup. The same principles apply for writing these pages as with the backup data buffer, except that when there is no more space in the backup log buffer, the backup fails. For that reason, the size of the backup log buffer must be large enough to handle the load caused by write activities while the backup is being made. See Section 17.5.3.3, “Configuration for MySQL Cluster Backups”.

    The default value for this parameter should be sufficient for most applications. In fact, it is more likely for a backup failure to be caused by insufficient disk write speed than it is for the backup log buffer to become full. If the disk subsystem is not configured for the write load caused by applications, the cluster is unlikely to be able to perform the desired operations.

    It is preferable to configure cluster nodes in such a manner that the processor becomes the bottleneck rather than the disks or the network connections.

    In MySQL Cluster NDB 6.4.3 and earlier, the default value is 2MB; in MySQL Cluster NDB 7.0.4 and later, it is 16MB.

  • BackupMemory

    Restart Typenode
    Permitted Values
    Typenumeric
    Default4M
    Range0-4G

    This parameter is simply the sum of BackupDataBufferSize and BackupLogBufferSize.

    In MySQL Cluster NDB 6.4.3 and earlier, the default value was 2MB + 2MB = 4MB; in MySQL Cluster NDB 7.0.4 and later, it is 16MB + 16MB = 32MB.

    Important

    If BackupDataBufferSize and BackupLogBufferSize taken together exceed the default value for BackupMemory, then this parameter must be set explicitly in the config.ini file to their sum.

  • BackupReportFrequency

    Version Introduced5.1.19-ndb-6.2.3
    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-4G

    This parameter controls how often backup status reports are issued in the management client during a backup, as well as how often such reports are written to the cluster log (provided cluster event logging is configured to permit it—see Logging and checkpointing). BackupReportFrequency represents the time in seconds between backup status reports.

    The default value is 0.

    This parameter was added in MySQL Cluster NDB 6.2.3.

  • BackupWriteSize

    Restart Typenode
    Permitted Values
    Typenumeric
    Default32K
    Range2K-4G

    This parameter specifies the default size of messages written to disk by the backup log and backup data buffers.

    In MySQL Cluster 6.4.3 and earlier, the default value for this parameter was 32KB; beginning with MySQL Cluster NDB 7.0.4, it is 256KB.

  • BackupMaxWriteSize

    Restart Typenode
    Permitted Values
    Typenumeric
    Default256K
    Range2K-4G

    This parameter specifies the maximum size of messages written to disk by the backup log and backup data buffers.

    In MySQL Cluster 6.4.3 and earlier, the default value for this parameter was 256KB; beginning with MySQL Cluster NDB 7.0.4, it is 1MB.

Important

When specifying these parameters, the following relationships must hold true. Otherwise, the data node will be unable to start.

  • BackupDataBufferSize >= BackupWriteSize + 188KB

  • BackupLogBufferSize >= BackupWriteSize + 16KB

  • BackupMaxWriteSize >= BackupWriteSize

MySQL Cluster Realtime Performance Parameters

The [ndbd] parameters discussed in this section are used in scheduling and locking of threads to specific CPUs on multiprocessor data node hosts. They were introduced in MySQL Cluster NDB 6.3.4.

Note

To make use of these parameters, the data node process must be run as system root.

  • LockExecuteThreadToCPU

    Version Introduced5.1.19-ndb-6.3.4
    Restart Typenode
    Permitted Values
    Typenumeric
    Default64K
    Range0-64K

    Previous to MySQL Cluster NDB 7.0. This parameter specifies the ID of the CPU assigned to handle the NDBCLUSTER execution thread. The value of this parameter is an integer in the range 0 to 65535 (inclusive). The default is 65535.

    MySQL Cluster NDB 7.0 and later (beginning with MySQL Cluster NDB 6.4.0). When used with ndbd, this parameter (now a string) specifies the ID of the CPU assigned to handle the NDBCLUSTER execution thread. When used with ndbmtd, the value of this parameter is a comma-separated list of CPU IDs assigned to handle execution threads. Each CPU ID in the list should be an integer in the range 0 to 65535 (inclusive). The number of IDs specified should match the number of execution threads determined by MaxNoOfExecutionThreads. There is no default value.

    Prior to MySQL Cluster NDB 7.0.18 and MySQL Cluster NDB 7.1.7, the effective maximum value recognized by this parameter as a valid CPU ID was 255; using a greater value caused thread locking to be disabled. (Bug#56185)

  • LockMaintThreadsToCPU

    Version Introduced5.1.19-ndb-6.3.4
    Restart Typenode
    Permitted Values
    Typenumeric
    Default64K
    Range0-64K

    This parameter specifies the ID of the CPU assigned to handle NDBCLUSTER maintenance threads.

    The value of this parameter is an integer in the range 0 to 65535 (inclusive). This parameter was added in MySQL Cluster NDB 6.3.4. Prior to MySQL Cluster NDB 6.4.0, the default is 65535; in MySQL Cluster NDB 7.0 and later MySQL Cluster release series, there is no default value.

  • RealtimeScheduler

    Version Introduced5.1.19-ndb-6.3.4
    Restart Typenode
    Permitted Values
    Typeboolean
    Defaultfalse
    Range-

    Setting this parameter to 1 enables real-time scheduling of NDBCLUSTER threads.

    The default is 0 (scheduling disabled).

  • SchedulerExecutionTimer

    Version Introduced5.1.22-ndb-6.3.4
    Restart Typenode
    Permitted Values
    Typenumeric
    Default50
    Range0-11000

    This parameter specifies the time in microseconds for threads to be executed in the scheduler before being sent. Setting it to 0 minimizes the response time; to achieve higher throughput, you can increase the value at the expense of longer response times.

    The default is 50 μsec, which our testing shows to increase throughput slightly in high-load cases without materially delaying requests.

    This parameter was added in MySQL Cluster NDB 6.3.4.

  • SchedulerSpinTimer

    Version Introduced5.1.22-ndb-6.3.4
    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-500

    This parameter specifies the time in microseconds for threads to be executed in the scheduler before sleeping.

    The default value is 0.

  • BuildIndexThreads

    Version Introduced5.1.39-ndb-7.0.11
    Restart Type
    Permitted Values
    Typenumeric
    Default0
    Range0-128

    This parameter determines the number of threads to create when rebuilding indexes during a system or node start. It is supported only when there is more than one fragment for the table per data node (for example, when the MAX_ROWS option has been used with CREATE TABLE).

    Setting this parameter to 0 (which is also the default value) disables multi-threaded building of ordered indexes. The maximum allowed value is 128.

    This parameter was added in MySQL Cluster NDB 6.3.30 and MySQL Cluster NDB 7.0.11. Prior to MySQL Cluster NDB 7.0.16 and MySQL Cluster NDB 7.1.5, it was supported only when using ndbd; in these and later MySQL Cluster releases, BuildIndexThreads is also supported for data nodes running ndbmtd (see Bug#54521).

Numa

Version Introduced5.1.51-ndb-7.1.9
Restart Typenode
Permitted Values
Type (linux)boolean
Default0

NDB is extremely sensitive to Non-Uniform Memory Access settings and multi-CPU systems due to timeouts that it can cause. Due to this fact, and because most MySQL Cluster users do not employ numactl, support for NUMA is ignored by default by ndbd when running on a Linux system, beginning with MySQL Cluster NDB 7.0.20 and MySQL Cluster 7.1.9. If your Linux system provides NUMA support and you wish for data node memory to be subject to NUMA control, you can set this parameter equal to 0.

The Numa configuration parameter is supported only on Linux systems where libnuma.so is installed.

Disk Data Configuration Parameters. Configuration parameters affecting Disk Data behavior include the following:

  • DiskPageBufferMemory

    This determines the amount of space used for caching pages on disk, and is set in the [ndbd] or [ndbd default] section of the config.ini file. It is measured in bytes. Each page takes up 32 KB. This means that Cluster Disk Data storage always uses N * 32 KB memory where N is some nonnegative integer.

    The default value for this parameter is 64M (2000 pages of 32 KB each).

    This parameter was added in MySQL 5.1.6.

  • SharedGlobalMemory

    This determines the amount of memory that is used for log buffers, disk operations (such as page requests and wait queues), and metadata for tablespaces, log file groups, UNDO files, and data files. It can be set in the [ndbd] or [ndbd default] section of the config.ini configuration file, and is measured in bytes.

    The default value is 20M.

    This parameter was added in MySQL 5.1.6.

  • DiskIOThreadPool

    This parameter determines the number of unbound threads used for Disk Data file access. Before DiskIOThreadPool was introduced, exactly one thread was spawned for each Disk Data file, which could lead to performance issues, particularly when using very large data files. With DiskIOThreadPool, you can—for example—access a single large data file using several threads working in parallel.

    Currently, this parameter applies to Disk Data I/O threads only, but we plan in the future to make the number of such threads configurable for in-memory data as well.

    The optimum value for this parameter depends on your hardware and configuration, and includes these factors:

    • Physical distribution of Disk Data files. You can obtain better performance by placing data files, undo log files, and the data node filesystem on separate physical disks. If you do this with some or all of these sets of files, then you can set DiskIOThreadPool higher to enable separate threads to handle the files on each disk.

    • Disk performance and types. The number of threads that can be accommodated for Disk Data file handling is also dependent on the speed and throughput of the disks. Faster disks and higher throughput allow for more disk I/O threads. Our test results indicate that solid-state disk drives can handle many more disk I/O threads than conventional disks, and thus higher values for DiskIOThreadPool.

    This parameter was added in MySQL Cluster NDB 6.4.0. Previous to MySQL Cluster NDB 6.4.3, it was named ThreadPool. Previous to MySQL Cluster NDB 7.0.7, the default value was 8. Beginning with MySQL Cluster NDB 7.0.7 and MySQL Cluster NDB 7.1.0, the default is 2.

  • Disk Data filesystem parameters. The parameters in the following list were added in MySQL Cluster NDB 6.2.17, 6.3.22, and 6.4.3 to make it possible to place MySQL Cluster Disk Data files in specific directories without the need for using symbolic links.

    For more information, see Section 17.5.10.1, “MySQL Cluster Disk Data Objects”.

  • Disk Data object creation parameters. The next two parameters enable you—when starting the cluster for the first time—to cause a Disk Data log file group, tablespace, or both, to be created without the use of SQL statements.

    • InitialLogFileGroup

      This parameter can be used to specify a log file group that is created when performing an initial start of the cluster. InitialLogFileGroup is specified as shown here:

      InitialLogFileGroup = [name=name;] [undo_buffer_size=size;] file-specification-list
      
      file-specification-list:
          file-specification[; file-specification[; ...]]
      
      file-specification:
          filename:size
      

      The name of the log file group is optional and defaults to DEFAULT-LG. The undo_buffer_size is also optional; if omitted, it defaults to 64M. Each file-specification corresponds to an undo log file, and at least one must be specified in the file-specification-list. Undo log files are placed according to any values that have been set for FileSystemPath, FileSystemPathDD, and FileSystemPathUndoFiles, just as if they had been created as the result of a CREATE LOGFILE GROUP or ALTER LOGFILE GROUP statement.

      Consider the following:

      InitialLogFileGroup = name=LG1; undo_buffer_size=128M; undo1.log:250M; undo2.log:150M
      

      This is equivalent to the following SQL statements:

      CREATE LOGFILE GROUP LG1
          ADD UNDOFILE 'undo1.log'
          INITIAL_SIZE 250M
          UNDO_BUFFER_SIZE 128M
          ENGINE NDBCLUSTER;
      
      ALTER LOGFILE GROUP LG1
          ADD UNDOFILE 'undo2.log'
          INITIAL_SIZE 150M
          ENGINE NDBCLUSTER;
      

      This logfile group is created when the data nodes are started with --initial.

      This parameter, if used, should always be set in the [ndbd default] section of the config.ini file. The behavior of a MySQL Cluster when different values are set on different data nodes is not defined.

    • InitialTablespace

      This parameter can be used to specify a MySQL Cluster Disk Data tablespace that is created when performing an initial start of the cluster. InitialTablespace is specified as shown here:

      InitialTablespace = [name=name;] [extent_size=size;] file-specification-list
      

      The name of the tablespace is optional and defaults to DEFAULT-TS. The extent_size is also optional; it defaults to 1M. The file-specification-list uses the same syntax as shown with the InitialLogfileGroup parameter, the only difference being that each file-specification used with InitialTablespace corresponds to a data file. At least one must be specified in the file-specification-list. Data files are placed according to any values that have been set for FileSystemPath, FileSystemPathDD, and FileSystemPathDataFiles, just as if they had been created as the result of a CREATE TABLESPACE or ALTER TABLESPACE statement.

      For example, consider the following line specifying InitialTablespace in the [ndbd default] section of the config.ini file (as with InitialLogfileGroup, this parameter should always be set in the [ndbd default] section, as the behavior of a MySQL Cluster when different values are set on different data nodes is not defined):

      InitialTablespace = name=TS1; extent_size=8M; data1.dat:2G; data2.dat:4G
      

      This is equivalent to the following SQL statements:

      CREATE TABLESPACE TS1
          ADD DATAFILE 'data1.dat'
          EXTENT_SIZE 8M
          INITIAL_SIZE 2G
          ENGINE NDBCLUSTER;
      
      ALTER TABLESPACE TS1
          ADD UNDOFILE 'data2.dat'
          INITIAL_SIZE 4G
          ENGINE NDBCLUSTER;
      

      This tablespace is created when the data nodes are started with --initial, and can be used whenever creating MySQL Cluster Disk Data tables thereafter.

Disk Data and GCP Stop errors. Errors encountered when using Disk Data tables such as Node nodeid killed this node because GCP stop was detected (error 2303) are often referred to as “GCP stop errors”. Such errors occur when the redo log is not flushed to disk quickly enough; this is usually due to slow disks and insufficient disk throughput.

You can help prevent these errors from occurring by using faster disks, and by placing Disk Data files on a separate disk from the data node filesystem. Reducing the value of TimeBetweenGlobalCheckpoints tends to decrease the amount of data to be written for each global checkpoint, and so may provide some protection against redo log buffer overflows when trying to write a global checkpoint; however, reducing this value also permits less time in which to write the GCP, so this must be done with caution.

In addition, adjusting the cluster configuration as discussed here can also help:

  • MySQL Cluster NDB 6.2 and 6.3. When working with large amounts of data on disk under high load, the default value for DiskPageBufferMemory may not be large enough. In such cases, you should increase its value to include most of the memory available to the data nodes after accounting for index memory, data memory, internal buffers, and memory needed by the data node host operating system.

    You can use this formula as a guide:

    DiskPageBufferMemory
      = 0.8
        x (
            [total memory]
              - ([operating system memory] + [buffer memory] + DataMemory + IndexMemory)
          )
    

    Once you have established that sufficient memory is reserved for DataMemory, IndexMemory, NDB internal buffers, and operating system overhead, it is possible (and sometimes desirable) to allocate more than the above amount of the remainder to DiskPageBufferMemory.

  • MySQL Cluster NDB 7.X. In addition to the considerations given for DiskPageBufferMemory as explained previously, it is also very important that the DiskIOThreadPool configuration parameter be set correctly; having DiskIOThreadPool set too high is very likely to cause GCP stop errors (Bug#37227).

Parameters for configuring send buffer memory allocation (MySQL Cluster NDB 7.0). Beginning with MySQL Cluster NDB 6.4.0, send buffer memory is allocated dynamically from a memory pool shared between all transporters, which means that the size of the send buffer can be adjusted as necessary. (Previously, the NDB kernel used a fixed-size send buffer for every node in the cluster, which was allocated when the node started and could not be changed while the node was running.) The following data node configuration parameters were added in MySQL Cluster NDB 6.4.0 to permit the setting of limits on this memory allocation; this change is reflected by the addition of the configuration parameters TotalSendBufferMemory and OverLoadLimit, as well as a change in how the existing SendBufferMemory configuration parameter is used. For more information, see Section 17.3.2.13, “Configuring MySQL Cluster Send Buffer Parameters”.

  • TotalSendBufferMemory

    This parameter is available beginning with MySQL Cluster NDB 6.4.0. It is used to determine the total amount of memory to allocate on this node for shared send buffer memory among all configured transporters.

    If this parameter is set, its minimum permitted value is 256KB; the maxmimum is 4294967039.

  • ReservedSendBufferMemory

    This parameter is present in NDBCLUSTER source code beginning with MySQL Cluster NDB 6.4.0. However, it is not currently enabled.

For more detailed information about the behavior and use of TotalSendBufferMemory and about configuring send buffer memory parameters in MySQL Cluster NDB 6.4.0 and later, see Section 17.3.2.13, “Configuring MySQL Cluster Send Buffer Parameters”.

Note

Previous to MySQL Cluster NDB 7.0, to add new data nodes to a MySQL Cluster, it is necessary to shut down the cluster completely, update the config.ini file, and then restart the cluster (that is, you must perform a system restart). All data node processes must be started with the --initial option.

Beginning with MySQL Cluster NDB 7.0, it is possible to add new data node groups to a running cluster online. See Section 17.5.11, “Adding MySQL Cluster Data Nodes Online”, for more information.

Controlling restart attempts. Beginning in MySQL Cluster NDB 6.2.19, MySQL Cluster NDB 6.3.37, MySQL Cluster NDB 7.0.18, and MySQL Cluster NDB 7.1.7, it is possible to exercise more finely-grained control over restart attempts by data nodes when they fail to start using two data node configuration parameters added in these releases. MaxStartFailRetries limits the total number of retries made before giving up on starting the data node; StartFailRetryDelay sets the number of seconds between retry attempts, as described in the following list:

  • StartFailRetryDelay

    Version Introduced5.1.47-ndb-7.1.7
    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-4G
    Permitted Values
    Typenumeric
    Default0
    Range0-4G
    Permitted Values
    Typenumeric
    Default0
    Range0-4G
    Permitted Values
    Typenumeric
    Default0
    Range0-4G

    Beginning in MySQL Cluster NDB 6.2.19, MySQL Cluster NDB 6.3.37, MySQL Cluster NDB 7.0.18, and MySQL Cluster NDB 7.1.7, it is possible to set the number of seconds between restart attempts by the data node in the event on failure on startup. The default is 0 (no delay).

    Note

    This parameter is ignored unless StartOnError is equal to 0.

  • MaxStartFailRetries

    Version Introduced5.1.47-ndb-7.1.7
    Restart Typenode
    Permitted Values
    Typenumeric
    Default3
    Range0-4G
    Permitted Values
    Typenumeric
    Default3
    Range0-4G
    Permitted Values
    Typenumeric
    Default3
    Range0-4G
    Permitted Values
    Typenumeric
    Default3
    Range0-4G

    Beginning in MySQL Cluster NDB 6.2.19, MySQL Cluster NDB 6.3.37, MySQL Cluster NDB 7.0.18, and MySQL Cluster NDB 7.1.7, it is possible to limit the number restart attempts made by the data node in the event that it fails on startup. The default is 3 attempts.

    Note

    This parameter is ignored unless StartOnError is equal to 0.

17.3.2.7. Defining SQL and Other API Nodes in a MySQL Cluster

The [mysqld] and [api] sections in the config.ini file define the behavior of the MySQL servers (SQL nodes) and other applications (API nodes) used to access cluster data. None of the parameters shown is required. If no computer or host name is provided, any host can use this SQL or API node.

Generally speaking, a [mysqld] section is used to indicate a MySQL server providing an SQL interface to the cluster, and an [api] section is used for applications other than mysqld processes accessing cluster data, but the two designations are actually synonomous; you can, for instance, list parameters for a MySQL server acting as an SQL node in an [api] section.

Note

For a discussion of MySQL server options for MySQL Cluster, see Section 17.3.4.2, “mysqld Command Options for MySQL Cluster”; for information about MySQL server system variables relating to MySQL Cluster, see Section 17.3.4.3, “MySQL Cluster System Variables”.

  • Id

    Restart Typenode
    Permitted Values
    Typenumeric
    Default
    Range1-63

    The Id is an integer value used to identify the node in all cluster internal messages. Prior to MySQL Cluster NDB 6.1.1, the permitted range of values for this parameter was 1 to 63 inclusive. Beginning with MySQL Cluster NDB 6.1.1, the permitted range is 1 to 255 inclusive. This value must be unique for each node in the cluster, regardless of the type of node.

    Note

    Data node IDs must be less than 49, regardless of the MySQL Cluster version used. If you plan to deploy a large number of data nodes, it is a good idea to limit the node IDs for API nodes (and management nodes) to values greater than 48.

    This parameter can also be written as NodeId, although the short form is sufficient (and preferred for this reason).

  • ExecuteOnComputer

    Restart Typesystem
    Permitted Values
    Typestring
    Default
    Range-

    This refers to the Id set for one of the computers (hosts) defined in a [computer] section of the configuration file.

  • HostName

    Restart Typesystem
    Permitted Values
    Typestring
    Default
    Range-

    Specifying this parameter defines the hostname of the computer on which the SQL node (API node) is to reside. To specify a hostname, either this parameter or ExecuteOnComputer is required.

    If no HostName or ExecuteOnComputer is specified in a given [mysql] or [api] section of the config.ini file, then an SQL or API node may connect using the corresponding “slot” from any host which can establish a network connection to the management server host machine. This differs from the default behavior for data nodes, where localhost is assumed for HostName unless otherwise specified.

  • ArbitrationRank

    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-2

    This parameter defines which nodes can act as arbitrators. Both MGM nodes and SQL nodes can be arbitrators. A value of 0 means that the given node is never used as an arbitrator, a value of 1 gives the node high priority as an arbitrator, and a value of 2 gives it low priority. A normal configuration uses the management server as arbitrator, setting its ArbitrationRank to 1 (the default for management nodes) and those for all SQL nodes to 0 (the default for SQL nodes).

    Beginning with MySQL 5.1.16 and MySQL Cluster NDB 6.1.3, it is possible to disable arbitration completely by setting ArbitrationRank to 0 on all management and SQL nodes. In MySQL Cluster NDB 7.0.7 and later releases, you can also control arbitration by overriding this parameter; to do this, set the Arbitration parameter in the [ndbd default] section of the config.ini global configuration file.

  • ArbitrationDelay

    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-4G

    Setting this parameter to any other value than 0 (the default) means that responses by the arbitrator to arbitration requests will be delayed by the stated number of milliseconds. It is usually not necessary to change this value.

  • BatchByteSize

    Restart Typenode
    Permitted Values
    Typenumeric
    Default32K
    Range1024-1M

    For queries that are translated into full table scans or range scans on indexes, it is important for best performance to fetch records in properly sized batches. It is possible to set the proper size both in terms of number of records (BatchSize) and in terms of bytes (BatchByteSize). The actual batch size is limited by both parameters.

    The speed at which queries are performed can vary by more than 40% depending upon how this parameter is set. In future releases, MySQL Server will make educated guesses on how to set parameters relating to batch size, based on the query type.

    This parameter is measured in bytes and by default is equal to 32KB.

  • BatchSize

    Restart Typenode
    Permitted Values
    Typenumeric
    Default64
    Range1-992

    This parameter is measured in number of records and is by default set to 64. The maximum size is 992.

  • HeartbeatThreadPriority

    Beginning with MySQL Cluster NDB 6.3.32, MySQL Cluster NDB 7.0.13, and MySQL Cluster NDB 7.1.2, it is possible to use this parameter to set the scheduling policy and priority of heartbeat threads for management and API nodes.

    The syntax for setting this parameter is shown here:

    HeartbeatThreadPriority = policy[, priority]
    
    policy:
      {FIFO | RR}
    

    When setting this parameter, you must specify a policy. This is one of FIFO (first in, first in) or RR (round robin). This followed optionally by the priority (an integer).

  • MaxScanBatchSize

    Restart Typenode
    Permitted Values
    Typenumeric
    Default256K
    Range32K-16M

    The batch size is the size of each batch sent from each data node. Most scans are performed in parallel to protect the MySQL Server from receiving too much data from many nodes in parallel; this parameter sets a limit to the total batch size over all nodes.

    The default value of this parameter is set to 256KB. Its maximum size is 16MB.

  • TotalSendBufferMemory

    Version Introduced5.1.30-ndb-6.4.0
    Restart Typenode
    Permitted Values
    Typenumeric
    Default256K
    Range0-4G

    This parameter is available beginning with MySQL Cluster NDB 6.4.0. It is used to determine the total amount of memory to allocate on this node for shared send buffer memory among all configured transporters.

    If this parameter is set, its minimum permitted value is 256KB; the maxmimum is 4294967039. For more detailed information about the behavior and use of TotalSendBufferMemory and configuring send buffer memory parameters in MySQL Cluster NDB 6.4.0 and later, see Section 17.3.2.13, “Configuring MySQL Cluster Send Buffer Parameters”.

  • AutoReconnect

    Version Introduced5.1.35-ndb-7.0.7
    Restart Typenode
    Permitted Values
    Typeboolean
    Defaultfalse
    Rangefalse-true

    This parameter is available beginning with MySQL Cluster NDB 6.3.26 and MySQL Cluster NDB 7.0.7. By default, its value is false, which forces disconnected API nodes (including MySQL Servers acting as SQL nodes) the use a new connection to the cluster rather than attempting to re-use an existing one, which can cause problems when using dynamically-allocated node IDs. (Bug#45921)

    Note

    This parameter can be overridden using the NDB API. For more information, see Ndb_cluster_connection::set_auto_reconnect(), and Ndb_cluster_connection::get_auto_reconnect().

You can obtain some information from a MySQL server running as a Cluster SQL node using SHOW STATUS in the mysql client, as shown here:

mysql> SHOW STATUS LIKE 'ndb%';
+-----------------------------+---------------+
| Variable_name               | Value         |
+-----------------------------+---------------+
| Ndb_cluster_node_id         | 5             |
| Ndb_config_from_host        | 192.168.0.112 |
| Ndb_config_from_port        | 1186          |
| Ndb_number_of_storage_nodes | 4             |
+-----------------------------+---------------+
4 rows in set (0.02 sec)

For information about these Cluster system status variables, see Section 5.1.6, “Server Status Variables”.

Note

To add new SQL or API nodes to the configuration of a running MySQL Cluster, it is necessary to perform a rolling restart of all cluster nodes after adding new [mysqld] or [api] sections to the config.ini file (or files, if you are using more than one management server). This must be done before the new SQL or API nodes can connect to the cluster.

It is not necessary to perform any restart of the cluster if new SQL or API nodes can employ previously unused API slots in the cluster configuration to connect to the cluster.

17.3.2.8. MySQL Cluster TCP/IP Connections

TCP/IP is the default transport mechanism for all connections between nodes in a MySQL Cluster. Normally it is not necessary to define TCP/IP connections; MySQL Cluster automatically sets up such connections for all data nodes, management nodes, and SQL or API nodes.

To override the default connection parameters, it is necessary to define a connection using one or more [tcp] sections in the config.ini file. Each [tcp] section explicitly defines a TCP/IP connection between two MySQL Cluster nodes, and must contain at a minimum the parameters NodeId1 and NodeId2, as well as any connection parameters to override.

It is also possible to change the default values for these parameters by setting them in the [tcp default] section.

Important

Any [tcp] sections in the config.ini file should be listed last, following all other sections in the file. However, this is not required for a [tcp default] section. This requirement is a known issue with the way in which the config.ini file is read by the MySQL Cluster management server.

Connection parameters which can be set in [tcp] and [tcp default] sections of the config.ini file are listed here:

  • NodeId1

    NodeId2

    To identify a connection between two nodes it is necessary to provide their node IDs in the [tcp] section of the configuration file. These are the same unique Id values for each of these nodes as described in Section 17.3.2.7, “Defining SQL and Other API Nodes in a MySQL Cluster”.

  • OverloadLimit

    Beginning in MySQL Cluster NDB 6.4.0, this parameter can be used to determine the amount of unsent data that must be present in the send buffer before the connection is considered overloaded. See Section 17.3.2.13, “Configuring MySQL Cluster Send Buffer Parameters”, for more information.

  • SendBufferMemory

    Restart Typenode
    Permitted Values
    Typenumeric
    Default256K
    Range64K-4G

    TCP transporters use a buffer to store all messages before performing the send call to the operating system. When this buffer reaches 64KB its contents are sent; these are also sent when a round of messages have been executed. To handle temporary overload situations it is also possible to define a bigger send buffer.

    Prior to MySQL Cluster NDB 7.0, this parameter determines a fixed amount of memory allocated at startup for each configured TCP connection. Beginning with MySQL Cluster NDB 6.4.0, if this parameter is set explicitly, then the memory is not dedicated to each transporter; instead, the value used denotes the hard limit for how much memory (out of the total available memory—that is, TotalSendBufferMemory) that may be used by a single transporter. For more information about configuring dynamic transporter send buffer memory allocation in MySQL Cluster NDB 7.0 and later, see Section 17.3.2.13, “Configuring MySQL Cluster Send Buffer Parameters”.

    In MySQL Cluster NDB 6.4.3 and earlier, the default size of the send buffer was 256 KB; in MySQL Cluster NDB 7.0.4 and later, it is 2MB, which is the size recommended in most situations. The minimum size is 64 KB; the theoretical maximum is 4 GB.

  • SendSignalId

    Restart Typenode
    Permitted Values
    Typeboolean
    Defaultfalse (debug builds: true)
    Range-

    To be able to retrace a distributed message datagram, it is necessary to identify each message. When this parameter is set to Y, message IDs are transported over the network. This feature is disabled by default in production builds, and enabled in -debug builds.

  • Checksum

    Restart Typenode
    Permitted Values
    Typeboolean
    Defaultfalse
    Range-

    This parameter is a boolean parameter (enabled by setting it to Y or 1, disabled by setting it to N or 0). It is disabled by default. When it is enabled, checksums for all messages are calculated before they placed in the send buffer. This feature ensures that messages are not corrupted while waiting in the send buffer, or by the transport mechanism.

  • PortNumber (OBSOLETE)

    This formerly specified the port number to be used for listening for connections from other nodes. This parameter should no longer be used.

  • ReceiveBufferMemory

    Restart Typenode
    Permitted Values
    Typenumeric
    Default64K
    Range16K-4G

    Specifies the size of the buffer used when receiving data from the TCP/IP socket.

    In MySQL Cluster NDB 6.4.3 and earlier, the default value of this parameter was 64 KB; beginning with MySQL Cluster NDB 7.0.4, 2MB is the default. The minimum possible value is 16KB; the theoretical maximum is 4GB.

17.3.2.9. MySQL Cluster TCP/IP Connections Using Direct Connections

Setting up a cluster using direct connections between data nodes requires specifying explicitly the crossover IP addresses of the data nodes so connected in the [tcp] section of the cluster config.ini file.

In the following example, we envision a cluster with at least four hosts, one each for a management server, an SQL node, and two data nodes. The cluster as a whole resides on the 172.23.72.* subnet of a LAN. In addition to the usual network connections, the two data nodes are connected directly using a standard crossover cable, and communicate with one another directly using IP addresses in the 1.1.0.* address range as shown:

# Management Server
[ndb_mgmd]
Id=1
HostName=172.23.72.20

# SQL Node
[mysqld]
Id=2
HostName=172.23.72.21

# Data Nodes
[ndbd]
Id=3
HostName=172.23.72.22

[ndbd]
Id=4
HostName=172.23.72.23

# TCP/IP Connections
[tcp]
NodeId1=3
NodeId2=4
HostName1=1.1.0.1
HostName2=1.1.0.2

The HostNameN parameter, where N is an integer, is used only when specifying direct TCP/IP connections.

The use of direct connections between data nodes can improve the cluster's overall efficiency by enabling the data nodes to bypass an Ethernet device such as a switch, hub, or router, thus cutting down on the cluster's latency. It is important to note that to take the best advantage of direct connections in this fashion with more than two data nodes, you must have a direct connection between each data node and every other data node in the same node group.

17.3.2.10. MySQL Cluster Shared-Memory Connections

MySQL Cluster attempts to use the shared memory transporter and configure it automatically where possible. [shm] sections in the config.ini file explicitly define shared-memory connections between nodes in the cluster. When explicitly defining shared memory as the connection method, it is necessary to define at least NodeId1, NodeId2, and ShmKey. All other parameters have default values that should work well in most cases.

Important

SHM functionality is considered experimental only. It is not officially supported in any current MySQL Cluster release, and testing results indicate that SHM performance is not appreciably greater than when using TCP/IP for the transporter.

For these reasons, you must determine for yourself or by using our free resources (forums, mailing lists) whether SHM can be made to work correctly in your specific case.

  • NodeId1

    NodeId2

    To identify a connection between two nodes it is necessary to provide node identifiers for each of them, as NodeId1 and NodeId2.

  • ShmKey

    Restart Typenode
    Permitted Values
    Typenumeric
    Default
    Range0-4G

    When setting up shared memory segments, a node ID, expressed as an integer, is used to identify uniquely the shared memory segment to use for the communication. There is no default value.

  • ShmSize

    Restart Typenode
    Permitted Values
    Typenumeric
    Default1M
    Range64K-4G

    Each SHM connection has a shared memory segment where messages between nodes are placed by the sender and read by the reader. The size of this segment is defined by ShmSize. The default value is 1MB.

  • SendSignalId

    Restart Typenode
    Permitted Values
    Typeboolean
    Defaultfalse
    Range-

    To retrace the path of a distributed message, it is necessary to provide each message with a unique identifier. Setting this parameter to Y causes these message IDs to be transported over the network as well. This feature is disabled by default in production builds, and enabled in -debug builds.

  • Checksum

    Restart Typenode
    Permitted Values
    Typeboolean
    Defaulttrue
    Range-

    This parameter is a boolean (Y/N) parameter which is disabled by default. When it is enabled, checksums for all messages are calculated before being placed in the send buffer.

    This feature prevents messages from being corrupted while waiting in the send buffer. It also serves as a check against data being corrupted during transport.

  • SigNum

    Restart Typenode
    Permitted Values
    Typenumeric
    Default
    Range0-4G

    When using the shared memory transporter, a process sends an operating system signal to the other process when there is new data available in the shared memory. Should that signal conflict with with an existing signal, this parameter can be used to change it. This is a possibility when using SHM due to the fact that different operating systems use different signal numbers.

    The default value of SigNum is 0; therefore, it must be set to avoid errors in the cluster log when using the shared memory transporter. Typically, this parameter is set to 10 in the [shm default] section of the config.ini file.

17.3.2.11. SCI Transport Connections in MySQL Cluster

[sci] sections in the config.ini file explicitly define SCI (Scalable Coherent Interface) connections between cluster nodes. Using SCI transporters in MySQL Cluster is supported only when the MySQL binaries are built using --with-ndb-sci=/your/path/to/SCI. The path should point to a directory that contains at a minimum lib and include directories containing SISCI libraries and header files. (See Section 17.3.5, “Using High-Speed Interconnects with MySQL Cluster” for more information about SCI.)

In addition, SCI requires specialized hardware.

It is strongly recommended to use SCI Transporters only for communication between ndbd processes. Note also that using SCI Transporters means that the ndbd processes never sleep. For this reason, SCI Transporters should be used only on machines having at least two CPUs dedicated for use by ndbd processes. There should be at least one CPU per ndbd process, with at least one CPU left in reserve to handle operating system activities.

  • NodeId1

    NodeId2

    To identify a connection between two nodes it is necessary to provide node identifiers for each of them, as NodeId1 and NodeId2.

  • Host1SciId0

    Restart Typenode
    Permitted Values
    Typenumeric
    Default
    Range0-4G

    This identifies the SCI node ID on the first Cluster node (identified by NodeId1).

  • Host1SciId1

    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-4G

    It is possible to set up SCI Transporters for failover between two SCI cards which then should use separate networks between the nodes. This identifies the node ID and the second SCI card to be used on the first node.

  • Host2SciId0

    Restart Typenode
    Permitted Values
    Typenumeric
    Default
    Range0-4G

    This identifies the SCI node ID on the second Cluster node (identified by NodeId2).

  • Host2SciId1

    Restart Typenode
    Permitted Values
    Typenumeric
    Default0
    Range0-4G

    When using two SCI cards to provide failover, this parameter identifies the second SCI card to be used on the second node.

  • SharedBufferSize

    Restart Typenode
    Permitted Values
    Typenumeric
    Default10M
    Range64K-4G

    Each SCI transporter has a shared memory segment used for communication between the two nodes. Setting the size of this segment to the default value of 1MB should be sufficient for most applications. Using a smaller value can lead to problems when performing many parallel inserts; if the shared buffer is too small, this can also result in a crash of the ndbd process.

  • SendLimit

    Restart Typenode
    Permitted Values
    Typenumeric
    Default8K
    Range128-32K

    A small buffer in front of the SCI media stores messages before transmitting them over the SCI network. By default, this is set to 8KB. Our benchmarks show that performance is best at 64KB but 16KB reaches within a few percent of this, and there was little if any advantage to increasing it beyond 8KB.

  • SendSignalId

    Restart Typenode
    Permitted Values
    Typeboolean
    Defaulttrue
    Range-

    To trace a distributed message it is necessary to identify each message uniquely. When this parameter is set to Y, message IDs are transported over the network. This feature is disabled by default in production builds, and enabled in -debug builds.

  • Checksum

    Restart Typenode
    Permitted Values
    Typeboolean
    Defaultfalse
    Range-

    This parameter is a boolean value, and is disabled by default. When Checksum is enabled, checksums are calculated for all messages before they are placed in the send buffer. This feature prevents messages from being corrupted while waiting in the send buffer. It also serves as a check against data being corrupted during transport.

17.3.2.12. Configuring MySQL Cluster Parameters for Local Checkpoints

The parameters discussed in Logging and Checkpointing and in Data Memory, Index Memory, and String Memory that are used to configure local checkpoints for a MySQL Cluster do not exist in isolation, but rather are very much interdepedent on each other. In this section, we illustrate how these parameters—including DataMemory, IndexMemory, NoOfDiskPagesToDiskAfterRestartTUP, NoOfDiskPagesToDiskAfterRestartACC, and NoOfFragmentLogFiles—relate to one another in a working Cluster.

Important

The parameters NoOfDiskPagesToDiskAfterRestartTUP and NoOfDiskPagesToDiskAfterRestartACC were deprecated in MySQL 5.1.6. From MySQL 5.1.6 through 5.1.11, disk writes during LCPs took place at the maximum speed possible. Beginning with MySQL 5.1.12, the speed and throughput for LCPs are controlled using the parameters DiskSyncSize, DiskCheckpointSpeed, and DiskCheckpointSpeedInRestart. See Section 17.3.2.6, “Defining MySQL Cluster Data Nodes”.

In this example, we assume that our application performs the following numbers of types of operations per hour:

  • 50000 selects

  • 15000 inserts

  • 15000 updates

  • 15000 deletes

We also make the following assumptions about the data used in the application:

  • We are working with a single table having 40 columns.

  • Each column can hold up to 32 bytes of data.

  • A typical UPDATE run by the application affects the values of 5 columns.

  • No NULL values are inserted by the application.

A good starting point is to determine the amount of time that should elapse between local checkpoints (LCPs). It is worth noting that, in the event of a system restart, it takes 40-60 percent of this interval to execute the REDO log—for example, if the time between LCPs is 5 minutes (300 seconds), then it should take 2 to 3 minutes (120 to 180 seconds) for the REDO log to be read.

The maximum amount of data per node can be assumed to be the size of the DataMemory parameter. In this example, we assume that this is 2 GB. The NoOfDiskPagesToDiskAfterRestartTUP parameter represents the amount of data to be checkpointed per unit time—however, this parameter is actually expressed as the number of 8K memory pages to be checkpointed per 100 milliseconds. 2 GB per 300 seconds is approximately 6.8 MB per second, or 700 KB per 100 milliseconds, which works out to roughly 85 pages per 100 milliseconds.

Similarly, we can calculate NoOfDiskPagesToDiskAfterRestartACC in terms of the time for local checkpoints and the amount of memory required for indexes—that is, the IndexMemory. Assuming that we permit 512 MB for indexes, this works out to approximately 20 8-KB pages per 100 milliseconds for this parameter.

Next, we need to determine the number of REDO log files required—that is, fragment log files—the corresponding parameter being NoOfFragmentLogFiles. We need to make sure that there are sufficient REDO log files for keeping records for at least 3 local checkpoints (in MySQL Cluster NDB 6.3.8 and later, we need only allow for 2 local checkpoints). In a production setting, there are always uncertainties—for instance, we cannot be sure that disks always operate at top speed or with maximum throughput. For this reason, it is best to err on the side of caution, so we double our requirement and calculate a number of fragment log files which should be enough to keep records covering 6 local checkpoints (in MySQL Cluster NDB 6.3.8 and later, a number of fragment log files accommodating 4 local checkpoints should be sufficient).

It is also important to remember that the disk also handles writes to the REDO log, so if you find that the amount of data being written to disk as determined by the values of NoOfDiskPagesToDiskAfterRestartACC and NoOfDiskPagesToDiskAfterRestartTUP is approaching the amount of disk bandwidth available, you may wish to increase the time between local checkpoints.

Given 5 minutes (300 seconds) per local checkpoint, this means that we need to support writing log records at maximum speed for 6 * 300 = 1800 seconds (MySQL Cluster NDB 6.3.8 and later: 4 * 300 = 1200 seconds). The size of a REDO log record is 72 bytes plus 4 bytes per updated column value plus the maximum size of the updated column, and there is one REDO log record for each table record updated in a transaction, on each node where the data reside. Using the numbers of operations set out previously in this section, we derive the following:

  • 50000 select operations per hour yields 0 log records (and thus 0 bytes), since SELECT statements are not recorded in the REDO log.

  • 15000 DELETE statements per hour is approximately 5 delete operations per second. (Since we wish to be conservative in our estimate, we round up here and in the following calculations.) No columns are updated by deletes, so these statements consume only 5 operations * 72 bytes per operation = 360 bytes per second.

  • 15000 UPDATE statements per hour is roughly the same as 5 updates per second. Each update uses 72 bytes, plus 4 bytes per column * 5 columns updated, plus 32 bytes per column * 5 columns—this works out to 72 + 20 + 160 = 252 bytes per operation, and multiplying this by 5 operation per second yields 1260 bytes per second.

  • 15000 INSERT statements per hour is equivalent to 5 insert operations per second. Each insert requires REDO log space of 72 bytes, plus 4 bytes per record * 40 columns, plus 32 bytes per column * 40 columns, which is 72 + 160 + 1280 = 1512 bytes per operation. This times 5 operations per second yields 7560 bytes per second.

So the total number of REDO log bytes being written per second is approximately 0 + 360 + 1260 + 7560 = 9180 bytes. Multiplied by 1800 seconds, this yields 16524000 bytes required for REDO logging, or approximately 15.75 MB. The unit used for NoOfFragmentLogFiles represents a set of 4 16-MB log files—that is, 64 MB. Thus, the minimum value (3) for this parameter is sufficient for the scenario envisioned in this example, since 3 times 64 = 192 MB, or about 12 times what is required; the default value of 8 (or 512 MB) is more than ample in this case.

17.3.2.13. Configuring MySQL Cluster Send Buffer Parameters

Formerly, the NDB kernel used a send buffer whose size was fixed at 2MB for every node in the cluster, which was allocated when the node started. Because the size of this buffer could not be changed after the cluster was started, it was necessary to make it large enough in advance to accomodate the maximum possible load on any transporter socket. However, this was an inefficient use of memory, since much of it often went unused, and could result in large amounts of resources being wasted when scaling up to many API nodes.

Beginning with MySQL Cluster NDB 7.0, this problem is solved by employing a unified send buffer whose memory is allocated dynamically from a pool shared by all transporters. This means that the size of the send buffer can be adjusted as necessary. Configuration of the unified send buffer can accomplished by setting the following parameters:

  • TotalSendBufferMemory. This parameter can be set for all types of MySQL Cluster nodes—that is, it can be set in the [ndbd], [mgm], and [api] (or [mysql]) sections of the config.ini file. It represents the total amount of memory (in bytes) to be allocated by each node for which it is set for use among all configured transporters. If set, its minimum is 256KB; the maximum is 4294967039.

    To be backward-compatible with existing configurations, this parameter takes as its default value the sum of the maximum send buffer sizes of all configured transporters, plus an additional 32KB (one page) per transporter. The maximum depends on the type of transporter, as shown in the following table:

    TransporterMaxmimum Send Buffer Size (bytes)
    TCPSendBufferMemory (default = 2M)
    SCISendLimit (default = 8K) plus 16K
    SHM20K

    This enables existing configurations to function in close to the same way as they did with MySQL Cluster NDB 6.3 and earlier, with the same amount of memory and send buffer space available to each transporter. However, memory that is unused by one transporter is not available to other transporters.

  • OverloadLimit. This parameter is used in the config.ini file [tcp] section, and denotes the amount of unsent data (in bytes) that must be present in the send buffer before the connection is considered overloaded. When such an overload condition occurs, transactions that affect the overloaded connection fail with NDB API Error 1218 (Send Buffers overloaded in NDB kernel) until the overload status passes. The default value is 0, in which case the effective overload limit is calculated as SendBufferMemory * 0.8 for a given connection. The maximum value for this parameter is 4G.

  • SendBufferMemory. In MySQL Cluster NDB 6.3 and earlier, this TCP configuration parameter represented the amount of memory allocated at startup for each configured TCP connection. Beginning with MySQL Cluster NDB 7.0, this value denotes a hard limit for the amount of memory that may be used by a single transporter out of the entire pool specified by TotalSendBufferMemory. However, the sum of SendBufferMemory for all configured transporters may be greater than the TotalSendBufferMemory that is set for a given node. This is a way to save memory when many nodes are in use, as long as the maximum amount of memory is never required by all transporters at the same time.

17.3.3. Overview of MySQL Cluster Configuration Parameters

The next four sections provide summary tables of MySQL Cluster configuration parameters used in the config.ini file to govern the cluster's functioning. Each table lists the parameters for one of the Cluster node process types (ndbd, ndb_mgmd, and mysqld), and includes the parameter's type as well as its default, mimimum, and maximum values as applicable.

These tables also indicate what type of restart is required (node restart or system restart)—and whether the restart must be done with --initial—to change the value of a given configuration parameter.

When performing a node restart or an initial node restart, all of the cluster's data nodes must be restarted in turn (also referred to as a rolling restart). It is possible to update cluster configuration parameters marked as node online—that is, without shutting down the cluster—in this fashion. An initial node restart requires restarting each ndbd process with the --initial option.

A system restart requires a complete shutdown and restart of the entire cluster. An initial system restart requires taking a backup of the cluster, wiping the cluster file system after shutdown, and then restoring from the backup following the restart.

In any cluster restart, all of the cluster's management servers must be restarted for them to read the updated configuration parameter values.

Important

Values for numeric cluster parameters can generally be increased without any problems, although it is advisable to do so progressively, making such adjustments in relatively small increments. Many of these can be increased online, using a rolling restart.

However, decreasing the values of such parameters—whether this is done using a node restart, node initial restart, or even a complete system restart of the cluster—is not to be undertaken lightly; it is recommended that you do so only after careful planning and testing. This is especially true with regard to those parameters that relate to memory usage and disk space, such as MaxNoOfTables, MaxNoOfOrderedIndexes, and MaxNoOfUniqueHashIndexes. In addition, it is the generally the case that configuration parameters relating to memory and disk usage can be raised using a simple node restart, but they require an initial node restart to be lowered.

Because some of these parameters can be used for configuring more than one type of cluster node, they may appear in more than one of the tables.

Note

4294967039 often appears as a maximum value in these tables. This value is defined in the NDBCLUSTER sources as MAX_INT_RNIL and is equal to 0xFFFFFEFF, or 232 – 28 – 1.

17.3.3.1. MySQL Cluster Data Node Configuration Parameters

The summary table in this section provides information about parameters used in the [ndbd] or [ndbd default] sections of a config.ini file for configuring MySQL Cluster data nodes. For detailed descriptions and other additional information about each of these parameters, see Section 17.3.2.6, “Defining MySQL Cluster Data Nodes”.

Beginning with MySQL Cluster NDB 6.4.0, these parameters also apply to ndbmtd, which is a multi-threaded version of ndbd. For more information, see Section 17.4.3, “ndbmtd — The MySQL Cluster Data Node Daemon (Multi-Threaded)”.

Restart types. Changes in MySQL Cluster configuration parameters do not take effect until the cluster is restarted. The type of restart required to change a given parameter is indicated in the summary table as follows:

For more information about restart types, see Section 17.3.3, “Overview of MySQL Cluster Configuration Parameters”.

Table 17.1. Data Node Configuration Parameters

NameType/UnitsDefaultMin ValueMax ValueRestart Type
Arbitration{Disabled|Default|WaitExternal}Default N
ArbitrationTimeoutmilliseconds1000104GN
BackupDataBufferSizebytes2M 4GN
BackupDataDirpathFileSystemPath IN
BackupLogBufferSizebytes2M 4GN
BackupMaxWriteSizebytes256K2K4GN
BackupMemorybytes4M 4GN
BackupReportFrequencyseconds 4GN
BackupWriteSizebytes32K2K4GN
BatchSizePerLocalScaninteger641992N
BuildIndexThreads 128
CompressedBackup false N
CompressedLCP false N
DataDirpath. IN
DataMemorybytes80M1M1024GN
DictTracebytesundefined 100N
DiskCheckpointSpeedbytes10M1M4GN
DiskCheckpointSpeedInRestartbytes100M1M4GN
DiskIOThreadPoolthreads8 4GN
Disklesstrue|false (1|0) 1IS
DiskPageBufferMemorybytes64M4M1TN
DiskSyncSizebytes4M32K4GN
ExecuteOnComputername S
FileSystemPathpathDataDir IN
FileSystemPathDataFiles FileSystemPathDD IN
FileSystemPathDD FileSystemPath IN
FileSystemPathUndoFiles FileSystemPathDD IN
FragmentLogFileSizebytes16M4M1GIN
HeartbeatIntervalDbApimilliseconds15001004GN
HeartbeatIntervalDbDbmilliseconds1500104GN
HostNamename or IPlocalhost S
Idunsigned 148N
IndexMemorybytes18M1M1TN
InitFragmentLogFilessparse|full IN
InitialLogFileGroup [see documentation] S
InitialNoOfOpenFilesfiles27204GN
InitialTablespace [see documentation] S
IOThreadPoolthreads8 4GN
LockExecuteThreadToCPUCPU ID64K 64KN
LockMaintThreadsToCPUCPU ID64K 64KN
LockPagesInMainMemorytrue|false (1|0) 1N
LogLevelCheckpointlog level 15IN
LogLevelCongestionlevelr 15N
LogLevelConnectioninteger 15N
LogLevelErrorinteger 15N
LogLevelInfointeger 15N
LogLevelNodeRestartinteger 15N
LogLevelShutdowninteger 15N
LogLevelStartupinteger1 15N
LogLevelStatisticinteger 15N
LongMessageBufferbytes1M512K4GN
MaxAllocateunsigned32M1M1GN
MaxBufferedEpochsepochs100 100000N
MaxLCPStartDelayseconds 600N
MaxNoOfAttributesinteger1000324GN
MaxNoOfConcurrentIndexOperationsinteger8K 4GN
MaxNoOfConcurrentOperationsinteger32K324GN
MaxNoOfConcurrentScansinteger2562500N
MaxNoOfConcurrentSubOperationsunsigned256 4GN
MaxNoOfConcurrentTransactionsinteger4096324GS
MaxNoOfFiredTriggersinteger4000 4GN
MaxNoOfLocalOperationsintegerUNDEFINED324GN
MaxNoOfLocalScansintegerUNDEFINED324GN
MaxNoOfOpenFilesinteger40204GN
MaxNoOfOrderedIndexesinteger128 4GN
MaxNoOfSavedMessagesinteger25 4GN
MaxNoOfSubscribersunsigned 4GN
MaxNoOfSubscriptionsunsigned 4GN
MaxNoOfTablesinteger128820320N
MaxNoOfTriggersinteger768 4GN
MaxNoOfUniqueHashIndexesinteger64 4GN
MaxStartFailRetriesunsigned3 4GN
MemReportFrequencyunsigned 4GN
NodeGroup 64KIS
NoOfDiskPagesToDiskAfterRestartACC8K pages/100 milliseconds2014GN
NoOfDiskPagesToDiskAfterRestartTUP8K pages/100 milliseconds4014GN
NoOfDiskPagesToDiskDuringRestartACC8K pages/100 milliseconds2014GN
NoOfDiskPagesToDiskDuringRestartTUP8K pages/100 milliseconds4014GN
NoOfFragmentLogFilesinteger1634GIN
NoOfReplicasintegerNone14IS
Numa N
ODirectboolean 1N
RealtimeScheduler false N
RedoBufferbytes8M1M4GN
ReservedSendBufferMemorybytes256K 4GN
RestartOnErrorInserterror code2 4N
SchedulerExecutionTimerµsec50 11000N
SchedulerSpinTimerµsec 500N
ServerPortunsigned 164KN
SharedGlobalMemorybytes20M 64TN
StartFailRetryDelayunsigned 4GN
StartFailureTimeoutmilliseconds 4GN
StartPartialTimeoutmilliseconds30000 4GN
StartPartitionedTimeoutmilliseconds60000 4GN
StartupStatusReportFrequencyseconds N
StopOnErrortrue|false (1|0)true N
StringMemory% or bytes5 4GS
TcpBind_INADDR_ANY false N
TimeBetweenEpochsmilliseconds100 32000N
TimeBetweenEpochsTimeoutmilliseconds4000 32000N
TimeBetweenGlobalCheckpointsmilliseconds20001032000N
TimeBetweenInactiveTransactionAbortCheckmilliseconds100010004GN
TimeBetweenLocalCheckpointsnumber of 4-byte words, as a base-2 logarithm20 31N
TimeBetweenWatchDogCheckmilliseconds6000704GN
TimeBetweenWatchDogCheckInitialmilliseconds6000704GN
TotalSendBufferMemorybytes256K 4GN
TransactionBufferMemorybytes1M1K4GN
TransactionDeadlockDetectionTimeoutmilliseconds1200504GN
TransactionInactiveTimeoutmilliseconds4G 4GN
UndoDataBufferunsigned16M1M4GN
UndoIndexBufferunsigned2M1M4GN
Note

Prior to MySQL Cluster NDB 7.0, to add new data nodes to a MySQL Cluster, it is necessary to shut down the cluster completely, update the config.ini file, and then restart the cluster, starting all data node processes using the --initial option—that is, you must perform a system restart.

Beginning in MySQL Cluster NDB 7.0, it is possible to add new data node groups to a running cluster online. For more information, see Section 17.5.11, “Adding MySQL Cluster Data Nodes Online”.

17.3.3.2. MySQL Cluster Management Node Configuration Parameters

The summary table in this section provides information about parameters used in the [ndb_mgmd] or [mgm] sections of a config.ini file for configuring MySQL Cluster management nodes. For detailed descriptions and other additional information about each of these parameters, see Section 17.3.2.5, “Defining a MySQL Cluster Management Server”.

Restart types. Changes in MySQL Cluster configuration parameters do not take effect until the cluster is restarted. The type of restart required to change a given parameter is indicated in the summary table as follows:

For more information about restart types, see Section 17.3.3, “Overview of MySQL Cluster Configuration Parameters”.

Table 17.2. Management Node Configuration Parameters

NameType/UnitsDefaultMin ValueMax ValueRestart Type
ArbitrationDelaymilliseconds 4GN
ArbitrationRank0-21 2N
DataDirpath. N
ExecuteOnComputername S
HeartbeatThreadPriority none
HostNamename or IP S
Idunsigned 163N
LogDestination{CONSOLE|SYSLOG|FILE}FILE:filename=ndb_nodeid_cluster.log,maxsize=1000000,maxfiles=6 N
MaxNoOfSavedEventsunsigned100 4GN
PortNumberunsigned1186 64KN
PortNumberStatsunsigned 64KN
wan false N
Note

After making changes in a management node's configuration, it is necessary to perform a rolling restart of the cluster for the new configuration to take effect. See Section 17.3.2.5, “Defining a MySQL Cluster Management Server”, for more information.

To add new management servers to a running MySQL Cluster, it is also necessary perform a rolling restart of all cluster nodes after modifying any existing config.ini files. For more information about issues arising when using multiple management nodes, see Section 17.1.5.10, “Limitations Relating to Multiple MySQL Cluster Nodes”.

17.3.3.3. MySQL Cluster SQL Node and API Node Configuration Parameters

The summary table in this section provides information about parameters used in the [SQL] and [api] sections of a config.ini file for configuring MySQL Cluster SQL nodes and API nodes. For detailed descriptions and other additional information about each of these parameters, see Section 17.3.2.7, “Defining SQL and Other API Nodes in a MySQL Cluster”.

Note

For a discussion of MySQL server options for MySQL Cluster, see Section 17.3.4.2, “mysqld Command Options for MySQL Cluster”; for information about MySQL server system variables relating to MySQL Cluster, see Section 17.3.4.3, “MySQL Cluster System Variables”.

Restart types. Changes in MySQL Cluster configuration parameters do not take effect until the cluster is restarted. The type of restart required to change a given parameter is indicated in the summary table as follows:

For more information about restart types, see Section 17.3.3, “Overview of MySQL Cluster Configuration Parameters”.

Table 17.3. SQL Node/API Node Configuration Parameters

NameType/UnitsDefaultMin ValueMax ValueRestart Type
ArbitrationDelaymilliseconds 4GN
ArbitrationRank0-2 2N
AutoReconnect falsefalsetrueN
BatchByteSizebytes32K10241MN
BatchSizerecords641992N
ConnectionMap N
ExecuteOnComputername S
HeartbeatThreadPriority none
HostNamename or IP S
Idunsigned 163N
MaxScanBatchSizebytes256K32K16MN
TotalSendBufferMemorybytes256K 4GN
wan false N
Note

To add new SQL or API nodes to the configuration of a running MySQL Cluster, it is necessary to perform a rolling restart of all cluster nodes after adding new [mysqld] or [api] sections to the config.ini file (or files, if you are using more than one management server). This must be done before the new SQL or API nodes can connect to the cluster.

It is not necessary to perform any restart of the cluster if new SQL or API nodes can employ previously unused API slots in the cluster configuration to connect to the cluster.

17.3.3.4. Other MySQL Cluster Configuration Parameters

The summary tables in this section provide information about parameters used in the [computer], [tcp], [shm], and [sci] sections of a config.ini file for configuring MySQL Cluster management nodes. For detailed descriptions and other additional information about individual parameters, see Section 17.3.2.8, “MySQL Cluster TCP/IP Connections”, Section 17.3.2.10, “MySQL Cluster Shared-Memory Connections”, or Section 17.3.2.11, “SCI Transport Connections in MySQL Cluster”, as appropriate.

Restart types. Changes in MySQL Cluster configuration parameters do not take effect until the cluster is restarted. The type of restart required to change a given parameter is indicated in the summary tables as follows:

For more information about restart types, see Section 17.3.3, “Overview of MySQL Cluster Configuration Parameters”.

Table 17.4. COMPUTER Configuration Parameters

NameType/UnitsDefaultMin ValueMax ValueRestart Type
HostNamename or IP S
Idstring IN

Table 17.5. TCP Configuration Parameters

NameType/UnitsDefaultMin ValueMax ValueRestart Type
Checksum false N
Groupunsigned55 200N
NodeId1 N
NodeId2 N
NodeIdServer N
OverloadLimitbytes 4GN
PortNumberunsigned 64KN
Proxy N
ReceiveBufferMemorybytes64K16K4GN
SendBufferMemoryunsigned256K64K4GN
SendSignalId false (debug builds: true) N
TCP_MAXSEG_SIZEunsigned 2GN
TCP_RCV_BUF_SIZEunsigned7008012GN
TCP_SND_BUF_SIZEunsigned7154012GN
TcpBind_INADDR_ANY false N

Table 17.6. SHM Configuration Parameters

NameType/UnitsDefaultMin ValueMax ValueRestart Type
Checksum true N
Groupunsigned35 200N
NodeId1 N
NodeId2 N
NodeIdServer N
OverloadLimitbytes 4GN
PortNumberunsigned 64KN
SendSignalId false N
ShmKeyunsigned 4GN
ShmSizebytes1M64K4GN
Signumunsigned 4GN

Table 17.7. SCI Configuration Parameters

NameType/UnitsDefaultMin ValueMax ValueRestart Type
Checksum false N
Groupunsigned15 200N
Host1SciId0unsigned 4GN
Host1SciId1unsigned 4GN
Host2SciId0unsigned 4GN
Host2SciId1unsigned 4GN
NodeId1 N
NodeId2 N
NodeIdServer N
OverloadLimitbytes 4GN
PortNumberunsigned 64KN
SendLimitunsigned8K12832KN
SendSignalId true N
SharedBufferSizeunsigned10M64K4GN

17.3.4. MySQL Server Options and Variables for MySQL Cluster

This section provides information about MySQL server options, server and status variables that are specific to MySQL Cluster. For general information on using these, and for other options and variables not specific to MySQL Cluster, see Section 5.1, “The MySQL Server”.

For MySQL Cluster configuration parameters used in the cluster confiuration file (usually named config.ini), see Section 17.3, “MySQL Cluster Configuration”.

17.3.4.1. MySQL Cluster Server Option and Variable Reference

The following table provides a list of the command-line options, server and status variables applicable within mysqld when it is running as an SQL node in a MySQL Cluster. For a table showing all command-line options, server and status variables available for use with mysqld, see Section 5.1.1, “Server Option and Variable Reference”.

Table 17.8. Command Options for MySQL Cluster

NameCmd-LineOption fileSystem VarStatus VarVar ScopeDynamic
Handler_discover YesBothNo
have_ndbcluster Yes GlobalNo
ndb_autoincrement_prefetch_szYesYesYes BothYes
ndb-batch-sizeYesYesYes GlobalNo
ndb_cache_check_timeYesYesYes GlobalYes
ndb-cluster-connection-poolYesYes YesGlobalNo
Ndb_cluster_node_id YesBothNo
Ndb_config_from_host YesBothNo
Ndb_config_from_port YesBothNo
Ndb_conflict_fn_max YesGlobalNo
Ndb_conflict_fn_old YesGlobalNo
ndb-connectstringYesYes
ndb_execute_count YesGlobalNo
ndb_extra_loggingYesYesYes GlobalYes
ndb_force_sendYesYesYes BothYes
ndb_index_stat_cache_entriesYesYes
ndb_index_stat_enableYesYes
ndb_index_stat_update_freqYesYes
ndb_log_binYes Yes BothYes
ndb_log_binlog_indexYes Yes GlobalYes
ndb_log_empty_epochsYesYesYes GlobalYes
ndb_log_orig Yes GlobalNo
ndb-log-update-as-writeYesYesYes GlobalYes
ndb_log_updated_onlyYesYesYes GlobalYes
ndb-mgmd-hostYesYes
ndb-nodeidYesYes YesGlobalNo
Ndb_number_of_data_nodes YesGlobalNo
ndb_optimization_delay Yes GlobalYes
ndb_optimized_node_selectionYesYes
ndb_pruned_scan_count YesGlobalNo
ndb_report_thresh_binlog_epoch_slipYesYes
ndb_report_thresh_binlog_mem_usageYesYes
ndb_scan_count YesGlobalNo
ndb_table_no_logging Yes SessionYes
ndb_table_temporary Yes SessionYes
ndb_use_copying_alter_table Yes BothNo
ndb_use_exact_count Yes BothYes
ndb_use_transactionsYesYesYes BothYes
ndb-wait-connectedYesYesYes No
ndb-wait-setupYesYesYes No
ndbclusterYesYes
- Variable: have_ndbcluster
ndbinfo_database Yes No
ndbinfo_max_bytesYes Yes BothYes
ndbinfo_max_rowsYes Yes BothYes
ndbinfo_show_hiddenYes Yes BothYes
ndbinfo_table_prefixYes Yes BothYes
ndbinfo_version Yes GlobalNo
server-id-bitsYesYes GlobalNo
- Variable: server_id_bits Yes GlobalNo
skip-ndbclusterYesYes
slave_allow_batchingYesYesYes GlobalYes

17.3.4.2. mysqld Command Options for MySQL Cluster

This section provides descriptions of mysqld server options relating to MySQL Cluster. For information about mysqld options not specific to MySQL Cluster, and for general information about the use of options with mysqld, see Section 5.1.2, “Server Command Options”.

For information about command-line options used with other MySQL Cluster processes (ndbd, ndb_mgmd, and ndb_mgm), see Section 17.4.23, “Options Common to MySQL Cluster Programs”. For information about command-line options used with NDB utility programs (such as ndb_desc, ndb_size.pl, and ndb_show_tables), see Section 17.4, “MySQL Cluster Programs”.

  • --ndb-batch-size=#

    Version Introduced5.1.23-ndb-6.3.8
    Command-Line Format--ndb-batch-size
    Option-File Formatndb-batch-size
    Variable Namendb_batch_size
    Variable ScopeGlobal
    Dynamic VariableNo
    Permitted Values
    Typenumeric
    Default32768
    Range0-31536000

    This sets the size in bytes that is used for NDB transaction batches.

  • --ndb-cluster-connection-pool=#

    Version Introduced5.1.19-ndb-6.2.2
    Command-Line Format--ndb-cluster-connection-pool
    Option-File Formatndb-cluster-connection-pool
    Variable NameNdb_cluster_connection_pool
    Variable ScopeGlobal
    Dynamic VariableNo
    Permitted Values
    Typenumeric
    Default1
    Range1-63

    By setting this option to a value greater than 1 (the default), a mysqld process can use multiple connections to the cluster, effectively mimicking several SQL nodes. Each connection requires its own [api] or [mysqld] section in the cluster configuration (config.ini) file, and counts against the maximum number of API connections supported by the cluster.

    Suppose that you have 2 cluster host computers, each running an SQL node whose mysqld process was started with --ndb-cluster-connection-pool=4; this means that the cluster must have 8 API slots available for these connections (instead of 2). All of these connections are set up when the SQL node connects to the cluster, and are allocated to threads in a round-robin fashion.

    This option is useful only when running mysqld on host machines having multiple CPUs, multiple cores, or both. For best results, the value should be smaller than the total number of cores available on the host machine. Setting it to a value greater than this is likely to degrade performance severely.

    Important

    Because each SQL node using connection pooling occupies multiple API node slots—each slot having its own node ID in the cluster—you must not use a node ID as part of the cluster connectstring when starting any mysqld process that employs connection pooling.

    Setting a node ID in the connectstring when using the --ndb-cluster-connection-pool option causes node ID allocation errors when the SQL node attempts to connect to the cluster.

    This option was introduced in MySQL Cluster NDB 6.2.2. Beginning with MySQL Cluster NDB 6.2.16 and MySQL Cluster NDB 6.3.13, the value used for this option is available as a global status variable (Bug#35573).

  • --ndb-connectstring=connect_string

    Command-Line Format--ndb-connectstring
    Option-File Formatndb-connectstring
    Permitted Values
    Typestring

    When using the NDBCLUSTER storage engine, this option specifies the management server that distributes cluster configuration data. See Section 17.3.2.3, “The MySQL Cluster Connectstring”, for syntax.

  • --ndbcluster

    Command-Line Format--ndbcluster
    Option-File Formatndbcluster
    Option Sets VariableYes, have_ndbcluster
    Disabled byskip-ndbcluster
    Permitted Values
    Typeboolean
    DefaultFALSE

    The NDBCLUSTER storage engine is necessary for using MySQL Cluster. If a mysqld binary includes support for the NDBCLUSTER storage engine, the engine is disabled by default. Use the --ndbcluster option to enable it. Use --skip-ndbcluster to explicitly disable the engine.

  • --ndb-nodeid=#

    Version Introduced5.1.15
    Command-Line Format--ndb-nodeid=#
    Option-File Formatndb-nodeid
    Variable NameNdb_cluster_node_id
    Variable ScopeGlobal
    Dynamic VariableNo
    Permitted Values (>= 5.1.5)
    Typenumeric
    Range1-255

    Set this MySQL server's node ID in a MySQL Cluster.

    Prior to MySQL Cluster NDB 7.0.15 and MySQL Cluster NDB 7.1.4, this option can be used instead of specifying the node ID as part of the connectstring or in the config.ini file, or permitting the cluster to determine an arbitrary node ID. If you use this option, then --ndb-nodeid must be specified before --ndb-connectstring. If --ndb-nodeid is used and a node ID is specified in the connectstring, then the MySQL server cannot connect to the cluster.

    Beginning with MySQL Cluster NDB 7.0.15 and MySQL Cluster NDB 7.1.4, the --ndb-nodeid option overrides any node ID set with --ndb-connectstring, regardless of the order in which the two options are used.

    In addition (regardless of the MySQL Cluster version), if --ndb-nodeid is used, then either a matching node ID must be found in a [mysqld] or [api] section of config.ini, or there must be an “open[mysqld] or [api] section in the file (that is, a section without an Id parameter specified). This is also true if the node ID is specified as part of the connectstring.

    Regardless of how the node ID is determined, its is shown as the value of the global status variable Ndb_cluster_node_id in the output of SHOW STATUS, and as cluster_node_id in the connection row of the output of SHOW ENGINE NDBCLUSTER STATUS.

    For more information about node IDs for MySQL Cluster SQL nodes, see Section 17.3.2.7, “Defining SQL and Other API Nodes in a MySQL Cluster”.

  • --server-id-bits=#

    Version Introduced5.1.47-ndb-7.1.6
    Command-Line Format--server-id-bits=#
    Option-File Formatserver-id-bits
    Option Sets VariableYes, server_id_bits
    Variable Nameserver_id_bits
    Variable ScopeGlobal
    Dynamic VariableNo
    Permitted Values
    Typenumeric
    Default32
    Range7-32

    This option indicates the number of least significant bits within the 32-bit server_id which actually identify the server. Indicating that the server is actually identified by fewer than 32 bits makes it possible for some of the remaining bits to be used for other purposes, such as storing user data generated by applications using the NDB API's Event API within the AnyValue of an OperationOptions structure (MySQL Cluster uses the AnyValue to store the server ID).

    When extracting the effective server ID from server_id for purposes such as detection of replication loops, the server ignores the remaining bits. The --server-id-bits option is used to mask out any irrelevant bits of server_id in the IO and SQL threads when deciding whether an event should be ignored based on the server ID.

    This data can be read from the binary log by mysqlbinlog, provided that it is run with its own --server-id-bits option set to 32 (the default).

    The value of server_id must be less than 2 ^ server_id_bits; otherwise, mysqld refuses to start.

    This system variable was added in MySQL Cluster NDB 7.0.17 and MySQL Cluster NDB 7.1.6; it is supported only in these and later releases of MySQL Cluster. It is not supported by the standard MySQL Server.

  • --skip-ndbcluster

    Command-Line Format--skip-ndbcluster
    Option-File Formatskip-ndbcluster

    Disable the NDBCLUSTER storage engine. This is the default for binaries that were built with NDBCLUSTER storage engine support; the server allocates memory and other resources for this storage engine only if the --ndbcluster option is given explicitly. See Section 17.3.1, “Quick Test Setup of MySQL Cluster”, for an example.

  • --ndb-wait-connected=seconds

    Version Introduced5.1.16-ndb-6.2.0
    Command-Line Format--ndb-wait-connected=#
    Option-File Formatndb-wait-connected
    Variable Namendb-wait-connected
    Variable Scope
    Dynamic VariableNo
    Permitted Values
    Typenumeric
    Default0
    Range0-31536000

    This option sets the period of time that the MySQL server waits for connections to MySQL Cluster management and data nodes to be established before accepting MySQL client connections. The time is specified in seconds. The default value is 0.

  • --ndb-wait-setup=seconds

    Version Introduced5.1.39-ndb-7.0.9
    Command-Line Format--ndb-wait-setup=#
    Option-File Formatndb-wait-setup
    Variable Namendb-wait-setup
    Variable Scope
    Dynamic VariableNo
    Permitted Values
    Typenumeric
    Default15
    Range0-31536000

    This variable shows the period of time that the MySQL server waits for the NDB storage engine to complete setup before timing out and treating NDB as unavailable. The time is specified in seconds. The default value is 15.

17.3.4.3. MySQL Cluster System Variables

This section provides detailed information about MySQL server system variables that are specific to MySQL Cluster and the NDB storage engine. For system variables not specific to MySQL Cluster, see Section 5.1.4, “Server System Variables”. For general information on using system variables, see Section 5.1.5, “Using System Variables”.

  • have_ndbcluster

    Variable Namehave_ndbcluster
    Variable ScopeGlobal
    Dynamic VariableNo
    Permitted Values
    Typeboolean

    YES if mysqld supports NDBCLUSTER tables. DISABLED if --skip-ndbcluster is used.

  • multi_range_count

    Command-Line Format--multi_range_count=#
    Option-File Formatmulti_range_count
    Option Sets VariableYes, multi_range_count
    Variable Namemulti_range_count
    Variable ScopeBoth
    Dynamic VariableYes
    Permitted Values
    Typenumeric
    Default256
    Range1-4294967295

    The maximum number of ranges to send to a table handler at once during range selects. The default value is 256. Sending multiple ranges to a handler at once can improve the performance of certain selects dramatically. This is especially true for the NDBCLUSTER table handler, which needs to send the range requests to all nodes. Sending a batch of those requests at once reduces communication costs significantly.

    This variable is deprecated in MySQL 5.1, and is no longer supported in MySQL 5.5, in which arbitrarily long lists of ranges can be processed.

  • ndb_autoincrement_prefetch_sz

    Command-Line Format--ndb_autoincrement_prefetch_sz
    Option-File Formatndb_autoincrement_prefetch_sz
    Option Sets VariableYes, ndb_autoincrement_prefetch_sz
    Variable Namendb_autoincrement_prefetch_sz
    Variable ScopeBoth
    Dynamic VariableYes
    Permitted Values (<= 5.1.22)
    Typenumeric
    Default32
    Range1-256
    Permitted Values (>= 5.1.23)
    Typenumeric
    Default1
    Range1-256

    Determines the probability of gaps in an autoincremented column. Set it to 1 to minimize this. Setting it to a high value for optimization—makes inserts faster, but decreases the likelihood that consecutive autoincrement numbers will be used in a batch of inserts. Default value: 32. Minimum value: 1.

    Beginning with MySQL Cluster NDB 6.2.10, MySQL Cluster NDB 6.3.7, and MySQL 5.1.23, this variable affects the number of AUTO_INCREMENT IDs that are fetched between statements only. Within a statement, at least 32 IDs are now obtained at a time. The default value for ndb_autoincrement_prefetch_sz is now 1, to increase the speed of statements inserting single rows. (Bug#31956)

    Beginning with MySQL Cluster NDB 6.3.31 and MySQL CLuster NDB 7.0.11, the maximum value for ndb_autoincrement_prefetch_sz is increased, from 256 to 65536. (Bug#50621)

  • ndb_cache_check_time

    Command-Line Format--ndb_cache_check_time
    Option-File Formatndb_cache_check_time
    Option Sets VariableYes, ndb_cache_check_time
    Variable Namendb_cache_check_time
    Variable ScopeGlobal
    Dynamic VariableYes
    Permitted Values
    Typenumeric
    Default0

    The number of milliseconds that elapse between checks of MySQL Cluster SQL nodes by the MySQL query cache. Setting this to 0 (the default and minimum value) means that the query cache checks for validation on every query.

    The recommended maximum value for this variable is 1000, which means that the check is performed once per second. A larger value means that the check is performed and possibly invalidated due to updates on different SQL nodes less often. It is generally not desirable to set this to a value greater than 2000.

  • ndb_extra_logging

    Version Introduced5.1.6
    Command-Line Formatndb_extra_logging=#
    Option-File Formatndb_extra_logging
    Variable Namendb_extra_logging
    Variable ScopeGlobal
    Dynamic VariableYes
    Permitted Values
    Typenumeric
    Default0

    This variable can be used to enable recording in the MySQL error log of information specific to the NDB storage engine. It is normally of interest only when debugging NDB storage engine code.

    The default value is 0, which means that the only NDB-specific information written to the MySQL error log relates to transaction handling. If the value is greater than 0 but less than 10, NDB table schema and connection events are also logged, as well as whether or not conflict resolution is in use, and other NDB errors and information. If the value is set to 10 or more, information about NDB internals, such as the progress of data distribution among cluster nodes, is also written to the MySQL error log.

    This variable was added in MySQL 5.1.6.

  • ndb_force_send

    Command-Line Format--ndb-force-send
    Option-File Formatndb_force_send
    Option Sets VariableYes, ndb_force_send
    Variable Namendb_force_send
    Variable ScopeBoth
    Dynamic VariableYes
    Permitted Values
    Typeboolean
    DefaultTRUE

    Forces sending of buffers to NDB immediately, without waiting for other threads. Defaults to ON.

  • ndb_index_stat_cache_entries

    Version Removed5.1.14
    Command-Line Format--ndb_index_stat_cache_entries
    Option-File Formatndb_index_stat_cache_entries
    Permitted Values
    Typenumeric
    Default32
    Range0-4294967295

    Sets the granularity of the statistics by determining the number of starting and ending keys to store in the statistics memory cache. Zero means no caching takes place; in this case, the data nodes are always queried directly. Default value: 32.

    Note

    If ndb_index_stat_enable is OFF, then setting this variable has no effect.

  • ndb_index_stat_enable

    Version Removed5.1.19
    Command-Line Format--ndb_index_stat_enable
    Option-File Formatndb_index_stat_enable
    Permitted Values
    Typeboolean
    DefaultOFF

    Use NDB index statistics in query optimization. Defaults to ON.

  • ndb_index_stat_update_freq

    Version Removed5.1.14
    Command-Line Format--ndb_index_stat_update_freq
    Option-File Formatndb_index_stat_update_freq
    Permitted Values
    Typenumeric
    Default20
    Range0-4294967295

    How often to query data nodes instead of the statistics cache. For example, a value of 20 (the default) means to direct every 20th query to the data nodes.

    Note

    If ndb_index_stat_cache_entries is 0, then setting this variable has no effect; in this case, every query is sent directly to the data nodes.

  • ndb_log_bin

    Version Introduced5.1.6
    Command-Line Format--ndb-log-bin={1|0}
    Option Sets VariableYes, ndb_log_bin
    Variable Namendb_log_bin
    Variable ScopeBoth
    Dynamic VariableYes
    Permitted Values
    Typeboolean
    DefaultON

    Causes updates to NDB tables to be written to the binary log. Setting this variable has no effect if binary logging is not already enabled for the server using log_bin. ndb_log_bin defaults to 1 (ON); normally, there is never any need to change this value in a production environment.

    This variable was added in MySQL 5.1.6.

  • ndb_log_binlog_index

    Version Introduced5.1.6
    Command-Line Format--ndb-log-binlog-index={1|0}
    Option Sets VariableYes, ndb_log_binlog_index
    Variable Namendb_log_binlog_index
    Variable ScopeGlobal
    Dynamic VariableYes
    Permitted Values
    Typeboolean
    DefaultON

    Causes a mapping of epochs to positions in the binary log to be inserted into the ndb_binlog_index table. Setting this variable has no effect if binary logging is not already enabled for the server using log_bin. (In addition, ndb_log_bin must not be disabled.) ndb_log_binlog_index defaults to 1 (ON); normally, there is never any need to change this value in a production environment.

    This variable was added in MySQL 5.1.6.

  • ndb_optimized_node_selection

    Command-Line Format--ndb-optimized-node-selection4.1.9-5.1.22-ndb-6.33
    --ndb-optimized-node-selection=#
    Option-File Formatndb_optimized_node_selection
    Permitted Values (<= 5.1.22-ndb-6.33)
    Typeboolean
    DefaultON

    Prior to MySQL Cluster NDB 6.3.4. Causes an SQL node to use the “closest” data node as transaction coordinator. Enabled by default. Set to 0 or OFF to disable, in which case the SQL node uses each data node in the cluster in succession. When this option is disabled each SQL thread attempts to use a given data node 8 times before proceeding to the next one.

    Beginning with MySQL Cluster NDB 6.3.4. There are two forms of optimized node selection, described here:

    1. The SQL node uses promixity to determine the transaction coordinator; that is, the “closest” data node to the SQL node is chosen as the transaction coordinator. For this purpose, a data node having a shared memory connection with the SQL node is considered to be “closest” to the SQL node; the next closest (in order of decreasing proximity) are: TCP connection to localhost; SCI connection; TCP connection from a host other than localhost.

    2. The SQL thread uses distribution awareness to select the data node. That is, the data node housing the cluster partition accessed by the first statement of a given transaction is used as the transaction coordinator for the entire transaction. (This is effective only if the first statement of the transaction accesses no more than one cluster partition.)

    This option takes one of the integer values 0, 1, 2, or 3. 3 is the default. These values affect node selection as follows:

    • 0: Node selection is not optimized. Each data node is employed as the transaction coordinator 8 times before the SQL thread proceeds to the next data node. (This is the same “round-robin” behavior as caused by setting this option to 0 or OFF in previous versions of MySQL Cluster.)

    • 1: Proximity to the SQL node is used to determine the transaction coordinator. (This is the same behavior as caused by setting this option to 1 or ON in previous MySQL versions.)

    • 2: Distribution awareness is used to select the transaction coordinator. However, if the first statement of the transaction accesses more than one cluster partition, the SQL node reverts to the round-robin behavior seen when this option is set to 0.

    • 3: If distribution awareness can be employed to determine the transaction coordinator, then it is used; otherwise proximity is used to select the transaction coordinator. (This is the default behavior in MySQL Cluster NDB 6.3.4 and later.)

    Important

    Beginning with MySQL Cluster NDB 6.3.4, it is no longer possible to set --ndb_optimized_node_selection to ON or OFF; attempting to do so causes mysqld to abort with an error.

  • ndb_report_thresh_binlog_epoch_slip

    Command-Line Format--ndb_report_thresh_binlog_epoch_slip
    Option-File Formatndb_report_thresh_binlog_epoch_slip
    Permitted Values
    Typenumeric
    Default3
    Range0-256

    This is a threshold on the number of epochs to be behind before reporting binlog status. For example, a value of 3 (the default) means that if the difference between which epoch has been received from the storage nodes and which epoch has been applied to the binlog is 3 or more, a status message will be sent to the cluster log.

  • ndb_report_thresh_binlog_mem_usage

    Command-Line Format--ndb_report_thresh_binlog_mem_usage
    Option-File Formatndb_report_thresh_binlog_mem_usage
    Permitted Values
    Typenumeric
    Default10
    Range0-10

    This is a threshold on the percentage of free memory remaining before reporting binlog status. For example, a value of 10 (the default) means that if the amount of available memory for receiving binlog data from the data nodes falls below 10%, a status message will be sent to the cluster log.

  • slave_allow_batching

    Version Introduced5.1.19-ndb-6.2.3
    Command-Line Format--slave-allow-batching
    Option-File Formatslave_allow_batching
    Option Sets VariableYes, slave_allow_batching
    Variable Nameslave_allow_batching
    Variable ScopeGlobal
    Dynamic VariableYes
    Permitted Values
    Typeboolean
    Defaultoff

    Whether or not batched updates are enabled on MySQL Cluster replication slaves.

    This variable is available beginning with MySQL Cluster NDB 6.2.3. Currently, it is available for mysqld only as supplied with MySQL Cluster or built from the MySQL Cluster sources. For more information, see Section 17.6.6, “Starting MySQL Cluster Replication (Single Replication Channel)”.

  • ndb_table_no_logging

    Version Introduced5.1.23-ndb-6.3.8
    Variable Namendb_table_no_logging
    Variable ScopeSession
    Dynamic VariableYes
    Permitted Values
    Typeboolean
    DefaultFALSE

    When this variable is set to ON or 1, it causes NDB tables not to be checkpointed to disk. More specifically, this setting applies to tables which are created or altered using ENGINE NDB when ndb_table_no_logging is enabled, and continues to apply for the lifetime of the table, even if ndb_table_no_logging is later changed. Suppose that A, B, C, and D are tables that we create (and perhaps also alter), and that we also change the setting for ndb_table_no_logging as shown here:

    SET @@ndb_table_no_logging = 1;
    
    CREATE TABLE A ... ENGINE NDB;
    
    CREATE TABLE B ... ENGINE MYISAM;
    CREATE TABLE C ... ENGINE MYISAM;
    
    ALTER TABLE B ENGINE NDB;
    
    SET @@ndb_table_no_logging = 0;
    
    CREATE TABLE D ... ENGINE NDB;
    ALTER TABLE C ENGINE NDB;
    
    SET @@ndb_table_no_logging = 1;
    

    After the previous sequence of events, tables A and B are not checkpointed; A was created with ENGINE NDB and B was altered to use NDB, both while ndb_table_no_logging was enabled. However, tables C and D are logged; C was altered to use NDB and D was created using ENGINE NDB, both while ndb_table_no_logging was disabled. Setting ndb_table_no_logging back to 1 or ON does not cause table C or D to be checkpointed.

    Note

    ndb_table_no_logging has no effect on the creation of NDB table schema files; to suppress these, use ndb_table_temporary instead.

  • ndb_table_temporary

    Version Introduced5.1.23-ndb-6.3.8
    Variable Namendb_table_temporary
    Variable ScopeSession
    Dynamic VariableYes
    Permitted Values
    Typeboolean
    DefaultFALSE

    When set to ON or 1, this variable causes NDB tables not to be written to disk: This means that no table schema files are created, and that the tables are not logged.

    Note

    Setting this variable currently has no effect in MySQL Cluster NDB 7.0 and later. This is a known issue; see Bug#34036.

  • ndb_use_copying_alter_table

    Version Introduced5.1.12
    Variable Namendb_use_copying_alter_table
    Variable ScopeBoth
    Dynamic VariableNo

    Forces NDB to use copying of tables in the event of problems with online ALTER TABLE operations. The default value is OFF.

    This variable was added in MySQL 5.1.12.

  • ndb_use_exact_count

    Variable Namendb_use_exact_count
    Variable ScopeBoth
    Dynamic VariableYes
    Permitted Values
    Typeboolean
    DefaultON

    Forces NDB to use a count of records during SELECT COUNT(*) query planning to speed up this type of query. The default value is ON. For faster queries overall, disable this feature by setting the value of ndb_use_exact_count to OFF.

  • ndb_use_transactions

    Command-Line Format--ndb_use_transactions
    Option-File Formatndb_use_transactions
    Variable Namendb_use_transactions
    Variable ScopeBoth
    Dynamic VariableYes
    Permitted Values
    Typeboolean
    DefaultON

    You can disable NDB transaction support by setting this variable's values to OFF (not recommended). The default is ON.

    Note

    The setting for this variable was not honored in MySQL Cluster NDB 6.4.3 and MySQL Cluster NDB 7.0.4. (Bug#43236)

The system variables in the following list all relate to the ndbinfo information database (see Section 17.5.8, “The ndbinfo MySQL Cluster Information Database”). They were added in MySQL Cluster NDB 7.1.1.

  • ndbinfo_database

    Version Introduced5.1.41-ndb-7.1.1
    Variable Namendbinfo_database
    Variable Scope
    Dynamic VariableNo
    Permitted Values
    Typestring
    Defaultndbinfo

    Shows the name used for the NDB information database; the default is ndbinfo. This is a read-only variable whose value is determined at compile time; you can set it by starting the server using --ndbinfo-database=name, which sets the value shown for this variable but does not actually change the name used for the NDB information database.

  • ndbinfo_max_bytes

    Version Introduced5.1.41-ndb-7.1.1
    Command-Line Format--ndbinfo-max-bytes=#
    Option Sets VariableYes, ndbinfo_max_bytes
    Variable Namendbinfo_max_bytes
    Variable ScopeBoth
    Dynamic VariableYes
    Permitted Values
    Typenumeric
    Default0

    Used in testing and debugging only.

  • ndbinfo_max_rows

    Version Introduced5.1.41-ndb-7.1.1
    Command-Line Format--ndbinfo-max-rows=#
    Option Sets VariableYes, ndbinfo_max_rows
    Variable Namendbinfo_max_rows
    Variable ScopeBoth
    Dynamic VariableYes
    Permitted Values
    Typenumeric
    Default10

    Used in testing and debugging only.

  • ndbinfo_show_hidden

    Version Introduced5.1.41-ndb-7.1.1
    Command-Line Format--ndbinfo-show-hidden={0|1}
    Option Sets VariableYes, ndbinfo_show_hidden
    Variable Namendbinfo_show_hidden
    Variable ScopeBoth
    Dynamic VariableYes
    Permitted Values
    Typeboolean
    Defaultno

    Whether or not the ndbinfo database's underlying internal tables are shown in the mysql client. The default is OFF.

  • ndbinfo_table_prefix

    Version Introduced5.1.41-ndb-7.1.1
    Command-Line Format--ndbinfo-table-prefix=name
    Option Sets VariableYes, ndbinfo_table_prefix
    Variable Namendbinfo_table_prefix
    Variable ScopeBoth
    Dynamic VariableYes
    Permitted Values
    Typestring
    Defaultndb$

    The prefix used in naming the ndbinfo database's base tables (normally hidden, unless exposed by setting ndbinfo_show_hidden). This is a read-only variable whose default value is “ndb$”. You can start the server with the --ndbinfo-table-prefix option, but this merely sets the variable and does not change the actual prefix used to name the hidden base tables; the prefix itself is determined at compile time.

  • ndbinfo_version

    Version Introduced5.1.41-ndb-7.1.1
    Variable Namendbinfo_version
    Variable ScopeGlobal
    Dynamic VariableNo
    Permitted Values
    Typestring
    Default

    Shows the version of the ndbinfo engine in use; read-only.

  • server_id_bits

    Version Introduced5.1.47-ndb-7.1.6
    Command-Line Format--server-id-bits=#
    Option-File Formatserver-id-bits