1. Kubernetes Operator User's Guide
  2. Overview of the Oracle TimesTen Kubernetes Operator

1 Overview of the Oracle TimesTen Kubernetes Operator

This chapter provides an overview of containers and Kubernetes. It also gives an overview of the TimesTen Kubernetes Operator. It discusses the TimesTenClassic and TimesTenScaleout Custom Resource Definitions (CRDs) and the TimesTen Operator. The chapter details the role the TimesTen Operator plays in deploying, managing, and monitoring active standby pairs of TimesTen Classic databases as well as in deploying TimesTen Scaleout grids and their associated databases in your Kubernetes cluster.

Topics:

Overview of containers and Kubernetes

A container is a lightweight virtual machine, running the Linux operating system. A container usually runs one application that is started from an image. Files that are created and modified are usually not persistent. However, persistent storage is available. Containers are a key component of cloud computing environments.

Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. Kubernetes has the ability to manage the resources of multiple hosts (called Nodes) in a cluster, and to run containers as required across these nodes. It can automatically spawn containers to react to various failures. Kubernetes also manages the networking among the containers and to the outside world. Kubernetes is portable across many cloud and on-premises environments.

Key Kubernetes concepts include:

Kubernetes provides the facilities for the provisioning of Pods and other Kubernetes resources that are required to deploy applications. Once deployed, the objects must be monitored and managed.

Kubernetes does some monitoring and managing of applications, but not all. It does handle problems at the Pod level automatically. For example, if a container fails, Kubernetes restarts it automatically. If an entire Node fails, Kubernetes starts replacement Pods on the other Nodes. However, Kubernetes has no knowledge about problems inside a container. This is not problematic for stateless applications, but for databases (which are stateful), Kubernetes needs help managing what is inside the containers.

This help comes in the form of:

Custom Resource Definition

A Custom Resource Definition (commonly known as a CRD) extends the Kubernetes' object model. It adds a new object type to the Kubernetes cluster, similar to the Pod, the StatefulSet, and the Service object types that it natively supports.

Kubernetes Operator

A Kubernetes Operator (also called Operator) is the brains behind a CRD. An Operator is an application that performs the functions of a human computer operator. It starts, stops, monitors, and manages other applications.

An Operator runs in one or more Pods, one active and the others idle. The active Operator performs the work. The remaining Operators are idle and remain idle until the active Operator fails. The active Operator manages all objects of a particular type and when combined with a CRD enables you to add custom facilities to Kubernetes.

About the TimesTen Kubernetes Operator

The TimesTen Kubernetes Operator provides the ability for you to deploy both active standby pairs of TimesTen Classic databases as well as TimesTen Scaleout grids and their associated databases in your Kubernetes cluster.

The TimesTen Kubernetes Operator consists of these interrelated components:

  • Custom Resource Definitions (CRDs): There are two CRDs. The TimesTenClassic CRD defines an object of type TimesTenClassic to Kubernetes. This TimesTenClassic object type provides the metadata for deploying active standby pairs of TimesTen Classic databases. The TimesTenScalout CRD defines an object of type TimesTenScaleout to Kubernetes. This TimesTenScaleout object type provides the metadata for deploying TimesTen Scaleout grids and their associated databases.

  • TimesTen Operator: There is one TimesTen Operator. The Operator monitors the TimesTenClassic and TimesTenScaleout objects and properly handles both. It deploys, manages, and monitors active standby pairs of TimesTen Classic database. This same Operator deploys TimesTen Scaleout grids and their associated databases.

  • TimesTen Agent: There is one TimesTen agent. This agent runs inside each container that runs TimesTen. The TimesTen Operator communicates with these agents both to determine the state of TimesTen insides of the container as well as to create, start, stop, and control TimesTen instances. The agent does not know how to manage TimesTen. It gives information to the Operator and the Operator provides the instructions for the agent. This agent knows how to work with TimesTen Classic and TimesTen Scaleout.

About TimesTenClassic and TimesTenScaleout objects

The TimesTen Operator distribution provides the file you need to deploy the TimesTenClassic and TimesTenScaleout CRDs in the Kubernetes cluster. Once deployed, Kubernetes understands the TimesTenClassic and TimesTenScaleout object types, just as it understands Pods, Secrets, and Services.

You can define objects of type TimesTenClassic or of type TimesTenScaleout or both. This lets you define the specific attributes for your TimesTen configuration and TimesTen database.

Objects in Kubernetes are named and typed. You can define a TimesTenClassic object named sample and another TimesTenClassic object named sample2. Similarly, you can define a TimesTenScaleout object named sample and another TimesTenScaleout object named sample2. You can have as many of these Kubernetes objects as you want, limited only by the available resources in your Kubernetes cluster.

Objects of different types have different meanings. For example, an object of type TimesTenClassic has a different meaning than an object of type ConfigMap. Therefore, you can define a sample TimesTenClassic object and a sample ConfigMap. The same is true for TimesTenScaleout objects.

Kubernetes supports namespaces. Namespaces split a cluster into multiple independent ones. Each namespace has a completely independent set of names. There can be an object of type a called x in namespace1 and a different object of type a called x in namespace2. For example, you can define an object of type TimesTenClassic called sample in the namespace1 namespace, and a different object of type TimesTenClassic called sample in the namespace2 namespace. The same is true for TimesTenScaleout objects.

Note:

CRDs are cluster-scoped, not namespace-scoped.

Kubernetes object definitions are expressed in JSON or YAML. The examples in this book use YAML.

About provisioning active standby pairs

TimesTen Classic databases almost always run in active standby pairs. Figure 1-1 illustrates an active standby pair replication scheme. There are two databases. One database is the active, and the second database is the standby. Applications update the active database. The standby database is read-only and receives replicated updates from the active database. Only one of the two databases function as the active database at any one time. If the active database fails, the standby database is promoted to be the active. After the failed (active) database is recovered, it becomes the standby database. See "Types of replication schemes" in the Oracle TimesTen In-Memory Database Replication Guide for more information on the active standby pair replication scheme.

Figure 1-1 Active standby pair of TimesTen databases

Description of Figure 1-1 follows
Description of "Figure 1-1 Active standby pair of TimesTen databases"

An active standby pair replication scheme is a good fit for Kubernetes. Specifically, consider a pair of Pods, each with persistent storage, that are running an active standby pair of TimesTen databases. If the Pod containing the active database fails, Kubernetes automatically spawns another Pod to take its place, and attaches the appropriate storage to it.

But, since Kubernetes doesn't know anything about TimesTen, it will not automatically perform the necessary operations to cause the standby database on the surviving Pod to become the active database. This is where the TimesTen Operator comes in.

TimesTen provides a CRD that adds the TimesTenClassic object type to Kubernetes as well as an Operator for managing TimesTen databases. The Operator automates setup and initial configuration, manages databases and replication, and automates failover and recovery.

When you define a TimesTenClassic object, you can specify the configuration of your TimesTen deployment using Kubernetes facilities. When you create a TimesTenClassic object in a Kubernetes cluster, a pair of Pods are created, each running TimesTen. Each Pod contains a TimesTen instance. Each instance provides one TimesTen database. Database replication, through active standby pairs, is automatically configured. In short, you can deploy highly available replicated pairs of TimesTen databases and manage them by issuing a small number of commands.

A Kubernetes Operator manages objects of a particular type. TimesTen provides an Operator that manages Kubernetes objects of type TimesTenClassic. In so doing, TimesTen can be deployed, monitored, managed, and controlled in an automated manner with no required human intervention.

About deploying a TimesTenClassic object

When you create a TimesTenClassic object in the Kubernetes cluster, the process to create and configure your active standby pair of databases begins. The Operator is invoked and creates several Kubernetes objects that are required to run TimesTen. After the objects are created and linked together, the TimesTen containers run a script to configure and start the TimesTen agent. The Operator communicates with the TimesTen agent that is running in each Pod in order to monitor and control TimesTen. The Operator configures one database as the active database, copies the active database to the standby, and then configures the active standby pair replication scheme. The process is described in detail in these sections:

About objects created by the TimesTen Operator

The Operator creates a number of Kubernetes objects that are required to run TimesTen, including a StatefulSet, a Service, and a Secret. These objects in turn create other objects. All of these objects are linked together by Kubernetes and are associated with the TimesTenClassic object you created. Figure 1-2 shows the objects that are created and how they are linked together.

Figure 1-2 Creating the TimesTenClassic object

Description of Figure 1-2 follows
Description of "Figure 1-2 Creating the TimesTenClassic object"

The objects that are created are described in the following sections:

StatefulSet

The Operator creates a StatefulSet consisting of two Pods to run TimesTen. Each Pod has one or more PersistentVolumes (persistent storage), that are mounted in the TimesTen containers. This is where your TimesTen databases are stored. Applications running in the containers with PersistentVolumes mounted can create files that live beyond the lifetime of the container. (By default, all files that containers create and modify automatically vanish when the container exits. Containers are ephemeral.)

One attribute of a StatefulSet is the number of replicas that can be provisioned. Each TimesTenClassic object has an associated StatefulSet with two replicas. If one Pod fails, Kubernetes automatically creates a new one to replace it, and automatically mounts the appropriate PersistentVolume(s) to it.

For example, for a TimesTenClassic object called sample, the Operator creates a StatefulSet called sample, in the same Kubernetes namespace. The StatefulSet, in turn, create two Pods in the namespace, called sample-0 and sample-1.

Service

A Kubernetes Service defines the set of network addresses and ports that should be exposed to applications in the cluster.

The Operator automatically creates a headless Service when you create the TimesTenClassic object. It automatically associates this Service with the StatefulSet. This causes Kubernetes to define entries in the Kubernetes cluster's DNS for the Pods in the StatefulSet, and to keep those DNS entries up to date.

A headless Service is used such that the DNS name/address entry for the active database is different than the DNS name/address entry for the standby database. This enables incoming client connections to be routed to the database that is active. For more information on a headless Service, see:

https://kubernetes.io/docs/concepts/services-networking/service/#headless-services/

For a TimesTenClassic object called sample, a headless Service called sample is also created in the same Kubernetes namespace. This results in entries in the cluster's DNS for sample-0.sample.namespace.svc.cluster.local and sample-1.sample.namespace.svc.cluster.local.

Secret

The TimesTen Operator creates a Secret to inject an SSL certificate into the TimesTen containers. This secures the communication between the TimesTen Operator and the TimesTen Agent.

Pods

The StatefulSet creates two pods. Each Pod contains two containers:

  • The tt container. This TimesTen container is always present in the Pods. It executes the TimesTen agent and runs TimesTen.

  • The daemonlog container: This container copies the contents of the TimesTen ttmesg.log file to stdout, resulting in Kubernetes logging the file. This enables the daemon log of the TimesTen instances to be recorded by the Kubernetes infrastructure.

Events

The Operator creates a Kubernetes event whenever important changes occur.

About the TimesTen containers and the TimesTen agent

After the objects are created, the TimesTen containers run a script that configures and starts the TimesTen agent. The Operator communicates with the TimesTen agent running in each Pod, in order to configure, manage, and monitor TimesTen in that Pod. The agent provides an HTTPS endpoint in the Pod that the Operator uses to query and control the tt container in the Pod. If the TimesTen agent fails, the tt container automatically terminates and is re-spawned by Kubernetes. Figure 1-3 illustrates the two way communication between the Operator and the TimesTen agent.

Figure 1-3 The Operator and the TimesTen agent

Description of Figure 1-3 follows
Description of "Figure 1-3 The Operator and the TimesTen agent"

The TimesTen agent starts TimesTen and thus runs as the instance administrator user. It has full control over TimesTen.

Simple deployment

The TimesTen Operator is designed for simple deployment of your active standby pairs of TimesTen Classic databases and for automated failure detection and recovery. For example,

  • You decide you want to deploy a new replicated pair of TimesTen databases.

  • You decide the attributes of those databases.

  • You create the configuration files for those attributes.

  • You use the kubectl create command to create a TimesTenClassic object to represent the replicated pair.

  • You use the kubectl get and kubectl describe commands to observe the provisioning of the active standby pair.

  • Applications that run in other Pods use TimesTen's standard client/server drivers to access TimesTen databases.

You do not have to monitor the TimesTen databases continually, configure replication, perform failover, or re-duplicate a database after failure. The TimesTen Operator performs all these functions and works to keep the databases up and running with minimal effort on your part.

About deploying a TimesTen Scaleout grid and its database

When you create a TimesTenScaleout object in the Kubernetes cluster, the process to create and configure the TimesTen Scaleout grid and its associated database begins. Kubernetes informs the TimesTen Operator that a TimesTenScaleout object has been created. The Operator begins the process of creating several Kubernetes objects that are required to deploy the grid.

Topics:

StatefulSets

The TimesTen Operator creates the following StatefulSets:

  • One StatefulSet that provides the management instance for the grid. The underlying Pod for this management instance is also created.

  • One StatefulSet that provides one or more ZooKeeper instances. There is one StatefulSet replica for each Zookeeper instance. For example, if there are three ZooKeeper instances, there is one StatefulSet and this one StatefulSet has three replicas.

  • One or more additional StatefulSets, the number of which is determined by your K-Safety (k) value. For example, a k value of 2 means that there are two copies of your TimesTen database. One copy of your database resides in data space one, and the second copy in data space two. In the Kubernetes environment, the TimesTen Operator creates k StatefulSets. Using the previous example, if k is set to 2, the Operator creates two StatefulSets. Each of the k StatefulSets provides Pods to implement a single data space in the resultant grid.

    The number of replicas for each StatefulSet is determined by the number of replica sets you want in your grid. A replica set contains k elements, where each element in the replica set is an exact copy of the other elements in the replica set. For example, if you want three replica sets in your grid, each StatefulSet has three replicas. Each replica contains a Pod for one data instance. Therefore, in this example, one StatefulSet has three Pods each of which contain one data instance. The second StatefulSet (assuming k=2) also has three replicas and each replica also contains a Pod for one data instance. Therefore, the second StatefulSet has three Pods each of which contain one data instance.

    In summary, in a case where k is set to 2 and the number of replica sets is 3, the Operator creates two StatefulSets, each with three replicas. Each StatefulSet provides the Pods to implement a single data space. There are a total of six Pods created for the six data instances.

Services

The TimesTen Operator creates the following headless Services:

  • One headless Service that provides DNS names for the Pods that contain the management and data instances. This Service enables client/server access to the Pods using the TimesTen client/server port 6625.

  • One headless Service that provides DNS names for the Pods that contain the ZooKeeper instances. This Service enables access to the Zookeeper internal ports 2888 and 3888, as well as the external port 2181.

Secret

The TimesTen Operator creates a Secret to inject an SSL certificate into the TimesTen containers. This secures the communication between the TimesTen Operator and the TimesTen Agent.

Persistent Volume Claims and Pods

The TimesTen Operator creates Persistent Volume Claims (PVCs) for the TimesTen and ZooKeeper Pods. These PVCs cause persistent volumes to be allocated by Kubernetes and attached to the TimesTen and ZooKeeper Pods. See Persistent storage.

Each Pod that runs a ZooKeeper instance consists of a single container called zookeeper.

Each Pod that runs TimesTen consists of at least two containers:

  • tt container: Runs the TimesTen Agent and TimesTen.

  • daemonlog container: Writes the TimesTen daemonlog to stdout. This enables the daemon log of the TimesTen instances to be recorded by the Kubernetes infrastructure.

As the tt containers in the TimesTen Pods start, the Operator assembles them into a working grid.

Password-less ssh

A TimesTen Scaleout grid relies on password-less ssh among the instances of the grid. The TimesTen Operator automatically configures password-less ssh among the tt containers in the grid in your Kubernetes environment. There is no intervention that you need to do.

Quick rollout

The TimesTen Kubernetes Operator provides the functionality for deploying TimesTen Scaleout grids and their associated databases. The TimesTen Operator rolls out the grid quickly and proficiently.

The Operator creates the StatefulSets and Services that are required to deploy the grid. It creates the ZooKeeper Pods. When the TimesTen Agent in the management Pod starts up, the Operator instructs the agent to create a TimesTen instance and grid. The Operator waits for all of the TimesTen agents in all of the Pods to start. Once all of the agents have started, the Operator instructs the agent in the management instance to create the hosts, the installations, and the instances in the grid's model for the data instances in the grid. The DbDef is created as are any direct and client/server connectables. The model is applied and the data instances of the grid are created.

The Operator instructs the management instance to create the database and to create the initial distribution map. The Operator then instructs the TimesTen agent in one data instance to run the TimesTen ttIsql utility to create initial database users and objects.

The grid and databases are created. The TimesTen agent opens the database for connections.

The grid rollout is quick and proficient with little or no intervention from you. Once the grid is rolled out, the TimesTen Operator does not manage the grid or database or perform maintenance operations.

Here is additional information: