About Deploying a Grid

About Planning a Grid

One of the features of the TimesTen Operator is the ability for it to provision a TimesTen Scaleout grid and its database in the Kubernetes cluster. Just like in any other TimesTen Scaleout environment, you must do some planning for your grid.

Here are some considerations:

K-Safety (represented by k): How many copies of your TimesTen database? The K-Safety factor determines the number of data spaces in your grid. For example, if you set k to 2, there are two copies of your database: one copy resides in data space one and the second copy in data space two.
Replica sets: How many replica sets in your grid? A replica set contains k elements, where each element in the replica set is an exact copy of the other elements in the replica set. The value of k, combined with the number of replica sets, determines the number of data instances in the grid. For example, if you set k to 2, and replica sets to 3, then there are six data instances in the grid.
ZooKeeper instances: How many ZooKeeper instances for the grid?
Database definition file (DbDef): What data store and first connection attributes are needed for the database in the grid?
Direct connectable: What general connection attributes are needed for the database when using direct access?
Client/server connectable: What general connection attributes are needed for the database when using client/server access?

After you define your configuration, you provide that information to the TimesTen Operator. The Operator takes over from there. It automatically does the provisioning and the deployment of the grid and database for you.

About Configuring a Grid

The TimesTen Operator provides the TimesTenScaleout object type and configuration metadata so that you can define and then deploy your TimesTen Scaleout grid and database. The TimesTenScaleout object type provides the syntax for configuring your grid. The configuration metadata lets you define the connection attributes for your database. Taken together, the Operator has the necessary information to provision your grid and database in the Kubernetes cluster.

Based on your planned configuration in About Planning a Grid, you can apply that configuration to the Kubernetes environment:

K-Safety (k): The Operator provides the .spec.ttspec.k element as part of the syntax for the TimesTenScaleout object type. You specify k in the YAML manifest file for your TimesTenScaleout object.
Replica sets: The Operator provides the .spec.ttspec.nReplicaSets element as part of the syntax for the TimesTenScaleout object type. You specify nReplicaSets in the YAML manifest file for your TimesTenScaleout object.
ZooKeeper instances: The Operator provides the .spec.ttspec.nZookeeper element as part of the syntax for the TimesTenScaleout object type. You specify nZookeeper in the YAML manifest file for your TimesTenScaleout object.
Database definition file (DbDef): The Operator creates a DbDef from the contents of the db.ini metadata file. You create this metadata file and then use a Kubernetes facility (or some other means) to place this file in the /ttconfig directory of the tt containers.
One or more direct connectables: The Operator creates one or more direct connectables from the contents of the *.connect metadata files. You create one or more of these *.connect files and then use a Kubernetes facility (or some other means) to place the files in the /ttconfig directory of the tt containers.
One or more client/server connectables: The Operator creates one or more client/server connectables from the contents of the *.csconnect metadata files. You create one or more of these *.csconnect files and then use a Kubernetes facility (or some other means) to place the files in the /ttconfig directory of the tt containers.

See TimesTenScaleoutSpecSpec, List of Configuration Metadata, and Populate the /ttconfig Directory.

After the metadata files are placed in the /ttconfig directory of the tt containers, and you configure and then deploy your TimesTenScaleout object in the Kubernetes cluster, Kubernetes informs the Operator that a TimesTenScaleout object has been created. The Operator begins the process of creating additional Kubernetes objects in order to implement your grid.

About Provisioning a Grid

The TimesTen Operator gathers the information from the TimesTenScaleout object and begins instantiating the TimesTen Scaleout grid and database. It creates the following StatefulSets:

One StatefulSet that provides the management instance for the grid. The Operator supports one management instance. The underlying Pod for this management instance is also created.
One StatefulSet that provides one or more ZooKeeper instances. The Operator determines the number of ZooKeeper instances by the value you specified for the nZookeeper field in your TimesTenScaleout object definition. For example, if you specified a value of 3 for nZookeeper, the Operator creates one StatefulSet with three replicas. The underlying Pods for these ZooKeeper instances are also created. There is one Pod per ZooKeeper instance.
Additional StatefulSets, the number of which is determined by the value you specified for the k field in your TimesTenScaleout object definition. For example, if you specified a value of 2 for k, the Operator creates two StatefulSets. These StatefulSets provide data instances for the grid. Each of the k StatefulSets provides Pods to implement a single data space in the resultant grid. The StatefulSet has M replicas, the number of which is determined by the value you specified for the nReplicaSets field in your TimesTenScaleout object definition. For example, if you set nReplicaSets to 3, each StatefulSet has three replicas. This number of replicas determines the number of replica sets in the resultant grid.

In the preceding example, one StatefulSet has three replicas. This one StatefulSet contains three data instances in data space one. A Pod is created for each data instance, so there are three Pods created. The second StatefulSet contains three data instances in data space two. A Pod is created for each data instance, so there are three Pods created. There are a total of six total Pods created for the six data instances.

In addition, the Operator creates the following Kubernetes headless Services:

One headless Service that provides DNS names for the Pods that contain the management and data instances. This service enables client/server access to the Pods using the TimesTen client/server port 6625.
One headless Service that provides DNS names for the Pods that contain the ZooKeeper instances. This service enables access to the ZooKeeper internal ports 2888 and 3888, as well as the external port 2181.

There is an example of these StatefulSets and headless Services in Verify Underlying Objects.

The Operator also creates Persistent Volume Claims (PVCs) for the Pods. These PVCs cause persistent volumes to be allocated by Kubernetes and attached to the TimesTen Pods and ZooKeeper Pods. See Persistent Storage.

Pods that run ZooKeeper consists of a single container called zookeeper. Each Pod that is running TimesTen consists of at least two containers. The tt container runs the TimesTen agent and TimesTen. The daemonlog container writes the TimesTen daemon log to stdout.

As the tt containers in the TimesTen Pods start, the Operator assembles them into a working grid. The grid's model is configured with several objects, including:

Hosts for the data instances in each of the data space groups
Each host configured with an installation of TimesTen
Each host configured with a single TimesTen instance
A DbDef (the contents of which are from the db.ini file)
Direct mode connectables, if any, (the contents of which are from the *.connect files)
Client/server connectables, if any, (the contents of which are from the *.csconnect files)

About ssh

A TimesTen Scaleout grid relies on password-less ssh among the instances of the grid. The TimesTen Operator automatically configures password-less ssh among the tt containers in the grid in your Kubernetes environment. There is no intervention that you need to do.

About Creating a Grid

When creating the grid, the TimeTen Operator transitions to and from various High Level states. Here is an explanation of these states:

After you create your TimesTenScaleout object in the Kubernetes cluster, the TimesTen Operator creates the StatefulSets and Services that are required to deploy your TimesTenScaleout grid and database. The Operator assigns a High Level state of Initializing to the TimesTenScaleout object.

The Operator periodically polls the status of the StatefulSets' objects and their underlying Pods. When the ZooKeeper Pods are ready, the TimesTenScaleout object transitions from the Initializing state to the ZooKeeperReady state.

When the TimesTen agent in the management Pod starts up, the Operator instructs the agent to create a TimesTen instance and grid. The TimesTenScaleout object transitions to the GridCreated state.

The Operator waits for all of the TimesTen agents in all of the Pods to start. Once all have started, the Operator instructs the TimesTen agent in the management instance to create the hosts, the installations, and the instances in the grid's model for the data instances in the grid.

The DbDef is then created from the contents of the db.ini file. The direct connectables are created from the contents of the *.connect files. The client/server connectables are created from the contents of the *.csconnect files.

The model is applied and the data instances of the grid are created. The TimesTenScaleout object transitions to the InstanceCreated state.

The Operator then instructs the management instance to create the database (by using the TimesTen ttGridAdmin utility with the dbCreate option) and to create the initial distribution map (by using the TimesTen ttGridAdmin utility with the dbDistribute -add all -apply options). The TimesTenScaleout object then transitions to the DbCreated state.

The Operator then instructs the TimesTen agent in one data instance to run the TimesTen ttIsql utility to create the user in the adminUser file and run the schema.sql file (if you provided these files). The TimesTenScaleout object transitions to the Normal state.

The grid and databases are created. The TimesTen agent then opens the database for connections.

The Operator manages and monitors the TimesTenScaleout objects after they are deployed in your Kubernetes cluster. The Operator also detects, repairs, and recovers from failures in your grid and associated databases. See Manage TimesTen Scaleout.

There is an example showing the Operator transitioning to and from these High Level states in Monitor the High Level State of a TimesTenScaleout Object.