6.4 NDB Engine Configuration

NDB memcache engine configuration options.  The NDB engine supports the following configuration options for use with memcache -e (see Section 6.3, “memcached command line options”):

Initial Configuration.  When a the NDB engine starts up, its most important command-line arguments are the cluster connection string and server role. The connection string is used to connect to a particular cluster, called the primary cluster, which contains a configuration schema. The tables in the configuration schema are read to retrieve a set of key-prefix mappings for the given server role (see the ndbmemcache configuration schema). Those mappings instruct the server how to respond to memcache operations on particular keys, based on the leftmost part of the key. For instance, they may specify that data is stored in particular columns of a certain table. This table may be stored in the same cluster as the configuration schema, or in a different cluster. A memcache server may have connections to several different clusters, and many memcache servers may connect to a single cluster but with a variety of roles.

The ndbmemcache configuration schema.  When the memcache NDB engine starts up, it connects to a cluster, and looks for the ndbmemcache configuration schema there. If the schema is not found, it shuts down.

The schema is described (with full comments) in the file ndb_memcache_metadata.sql

The main concept of the schema is a key-prefix mapping. This takes a prefix of a memcache key and maps it to a specific container table, on a particular cluster, with a particular cache policy.

A server role is defined as a set of key-prefix mappings that a memcached server will implement.

Whenever a memcached server is started with a particular server role (from the command-line arguments), that server role must exist in the ndbmemcache.server_roles table.

The following table lists table names and descriptions for tables that belong to the ndbmemcache configuration schema.

Table NameDescription
metaThe meta table describes the version number of the ndbmemcache tables. It should be considered as a read-only table.
ndb_clusters

For each cluster, this table holds a numeric cluster-id and a connection string. The microsec_rtt column is used for performance tuning. It is recommended to use the default value of this column. See Autotuning.

cache_policies

This table maps a policy name to a set of get, set, delete, and flush policies. The policy_name column is used as the key (there is no numeric policy id).

Additional information about cache policies can found in the text following the table.

containers

The containers table describes how the memcached server can use a database table to store data.

Additional information about containers can found in the text following the table.

memcache_server_roles

The memcache_server_roles table maps a role name to a numeric ID and a max_tps specifier, which is used for performance tuning. See Autotuning. It is recommended to use the default value.

This table also has an update_timestamp column. This column can be updated to enable online reconfiguration. See Online reconfiguration.

Additional information about server roles can found in the text following the table.

key_prefixes

In this table, the leftmost part of a memcache key is paired with a cluster ID, container, and cache policy to make a key prefix mapping.

Additional information about key prefix mappings can found in the text following the table.

Cache policies.  There are four policy types: get_policy, set_policy, delete_policy, and flush_from_db. These are described in the following paragraphs.

get_policy determines how the memcached server interprets GET commands. Possible values and their meanings are shown in the following list:

The set_policy determines how the memcached server interprets SET, INSERT, and REPLACE commands. Possible set_policy values and their meanings are listed here:

delete_policy describes how the memcached server interprets DELETE commands. It can take on the values shown and described in the following list:

flush_from_db determines how the memcached server interprets a FLUSH_ALL command with regard to data stored in the MySQL Cluster database, as shown here:

containers table columns.  The columns in the containers table are described in the following list:

Key mappings. 

The following table lists table names and descriptions for non-configuration ndbmemcache logging and container tables.

Table NameDescription
last_memcached_signon

This table is not part of the configuration schema, but is an informative logging table. It records the most recent login time of each memcached server using the configuration.

  • ndb_node_id is an int recording the API node id of the server

  • hostname is the hostname of the memcached server

  • server_role is the server role of the server at signon time

  • signon_time is a timestamp recording the memcached startup time

    In the case of online reconfiguration, signon_time records the time of the latest reconfiguration, not the time of startup. This is an unintended consequence and might be considered a bug.

demo_tabledemo_table is the container table used with default key prefix in the default server role. It is used to demonstrate SET and GET operations as well as INCR, DECR, and CAS, with one key column and one value column.
demo_table_tabsdemo_table_tabs is the container table for the "demo_tabs" container, which is used with the key prefix "t:" in the default server role. It is used to demonstrate one key column with multiple value columns. In memcache operations, the value columns are represented as a tab-separated list of values.

Predefined configuration objects

Predefined clusters.  A single ndb_cluster record is predefined, referring to the primary cluster (the one where configuration data is stored) as cluster id 0. Id 0 should always be reserved for the primary cluster.

Predefined cache policies

Predefined containers

Predefined memcache server roles and their key prefixes

Configuration versioning and upgrade.  The configuration schema is versioned, and the version number is stored in the ndbmemcache.meta table. The NDB Engine begins the configuration process by reading the schema version number from this table. As a rule, newer versions of the NDB engine will remain compatible with older versions of the configuration schema.

STABILITY NOTE: consider this section "unstable" & subject to change

Performance Tuning.  Two parameters are used to tune performance of the NDB memcache engine. The parameters are stored in the configuration schema: the "usec_rtt" value of a particular cluster, and the "max_tps" value of a memcache server role. These values are currently used in two ways: to configure the number of connections to each cluster, and to configure a particular fixed number of concurrent operations supported from each connection.

Autotuning.  Autotuning uses an estimated round trip time between cluster data nodes and a target rate of throughput to determine the ideal number of cluster connections and transactions per connection for a given workload. Autotuning parameters are described in the next few paragraphs.

These values are used, as described in the next few paragraphs, to calculate an optimum number of cluster connections with a given transactions-per-second capacity..

Number of cluster connections.  The NDB Engine scheduler attempts to open 1 cluster connection per 50000 transactions per second (TPS). This behavior can be overridden by using a scheduler configuration string (see Section 6.4, “NDB Engine Configuration”.) If the scheduler fails to open a second or subsequent connection to a cluster—for example, because a node id is not available—this is not a fatal error; it will run with only the connections actually opened.

Number of transactions per connection.  We assume that a transaction takes 5 times the cluster round trip time to complete. We can obtain the total number of in-flight transactions by dividing the server's max_tps by 5 * rtt (in seconds). These in-flight transaction objects are evenly distributed among the cluster connections.

Tuning example.  The following example starts with the default values usec_rtt = 250 and max_tps = 100000, and assumes a memcached server with 4 worker threads.

Online reconfiguration.  It is possible to reconfigure the key-prefix mappings of a running NDB engine without restarting it. This is done by committing a change to the configuration schema, and then updating the update_timestamp column of a particular server role in the memcache server roles table. The updating of the timestamp causes an event trigger to fire, so that the memcache server receives notification of the event.

Online reconfiguration can be disabled by using the -e reconf=false option on the command line.

Online reconfiguration can be used to connect to new clusters and to create new key-prefix mappings. However, it cannot be used to reset autotuning values on existing connections.

Online reconfiguration is a risky operation that could result in memcache server crashes or data corruption, and is used extensively in the mysql test suite. However, it is not recommended for reconfiguring a production server under load.

The stats reconf command can be run before and after online reconfiguration to verify that the version number of the running configuration has increased. Verification of reconfiguration is also written into the memcached log file.