1 Overview of the Oracle TimesTen Kubernetes Operator

This chapter provides an overview of containers and Kubernetes. It also discusses the TimesTen Operator.

Overview of containers and Kubernetes

A container is a lightweight virtual machine, running the Linux operating system. A container usually runs one application that is started from an image. Files that are created and modified are usually not persistent. However, persistent storage is available. Containers are a key component of cloud computing environments.

Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. Kubernetes has the ability to manage the resources of multiple hosts (called Nodes) in a cluster, and to run containers as required across these nodes. It can automatically spawn containers to react to various failures. Kubernetes also manages the networking among the containers and to the outside world. Kubernetes is portable across many cloud and on-premises environments.

Key Kubernetes concepts include:

Kubernetes provides the facilities for the provisioning of Pods and other Kubernetes resources that are required to deploy applications. Once deployed, the objects must be monitored and managed.

Kubernetes does some monitoring and managing of applications, but not all. It does handle problems at the Pod level automatically. For example, if a container fails, Kubernetes restarts it automatically. If an entire Node fails, Kubernetes starts replacement Pods on the other Nodes. However, Kubernetes has no knowledge about problems inside a container. This is not problematic for stateless applications, but for databases (which are stateful), Kubernetes needs help managing what is inside the containers.

This help comes in the form of:

Custom Resource Definition

A Custom Resource Definition (commonly known as a CRD) extends the Kubernetes' object model. It adds a new object type to the Kubernetes cluster, similar to the Pod, the StatefulSet, and the Service object types that it natively supports.

Kubernetes Operator

A Kubernetes Operator (also called Operator) is the brains behind a CRD. An Operator is an application that performs the functions of a human computer operator. It starts, stops, monitors, and manages other applications.

An Operator runs in one or more Pods, one active and the others idle. The active Operator performs the work. The remaining Operators are idle and remain idle until the active Operator fails. The active Operator manages all objects of a particular type and when combined with a CRD enables you to add custom facilities to Kubernetes.

Introducing the TimesTen Operator

TimesTen Classic databases almost always run in active standby pairs. Figure 1-1, "Active standby pair of TimesTen databases" illustrates an active standby pair replication scheme. There are two databases. One database is the active, and the second database is the standby. Applications update the active database. The standby database is read-only and receives replicated updates from the active database. Only one of the two databases function as the active database at any one time. If the active database fails, the standby database is promoted to be the active. After the failed (active) database is recovered, it becomes the standby database. See "Types of replication schemes" in the Oracle TimesTen In-Memory Database Replication Guide for more information on the active standby pair replication scheme.

Figure 1-1 Active standby pair of TimesTen databases

Description of Figure 1-1 follows
Description of ''Figure 1-1 Active standby pair of TimesTen databases''

An active standby pair replication scheme is a good fit for Kubernetes. Specifically, consider a pair of Pods, each with persistent storage, that are running an active standby pair of TimesTen databases. If the Pod containing the active database fails, Kubernetes automatically spawns another Pod to take its place, and attaches the appropriate storage to it.

But, since Kubernetes doesn't know anything about TimesTen, it will not automatically perform the necessary operations to cause the standby database on the surviving Pod to become the active database. This is where the TimesTen Operator comes in.

TimesTen provides a CRD that adds the TimesTenClassic object type to Kubernetes as well as an Operator for managing TimesTen databases. The Operator automates setup and initial configuration, manages databases and replication, and automates failover and recovery.

When you define a TimesTenClassic object, you can specify the configuration of your TimesTen deployment using Kubernetes facilities. When you create a TimesTenClassic object in a Kubernetes cluster, a pair of Pods are created, each running TimesTen. Each Pod contains a TimesTen instance. Each instance provides one TimesTen database. Database replication, through active standby pairs, is automatically configured. In short, you can deploy highly available replicated pairs of TimesTen databases and manage them by issuing a small number of commands.

A Kubernetes Operator manages objects of a particular type. TimesTen provides an Operator that manages Kubernetes objects of type TimesTenClassic. In so doing, TimesTen can be deployed, monitored, managed, and controlled in an automated manner with no required human intervention.

These sections describe the TimesTenClassic object type and the Operator:

The TimesTenClassic object type

The TimesTen Operator provides an implementation of the TimesTenClassic CRD, which you install in your Kubernetes cluster. After installation, Kubernetes understands the TimesTenClassic object type, just as it understands Pods, Secrets, and Services.

To create an active standby pair of TimesTen databases in your cluster, you use the kubectl create command to create an object of type TimesTenClassic. You define the desired attributes of your TimesTen configuration and your TimesTen database as attributes of this TimesTenClassic object.

Kubernetes objects: named and typed

Objects in Kubernetes are named and typed, so you can define a TimesTenClassic object named sample and another TimesTenClassic object named sample2. You can have many such Kubernetes objects in a cluster, limited only by the available resources in a Kubernetes cluster.

Objects of different types have different meanings. There can be an object of type a called x and an object of type b called x simultaneously. For example, you can define an object of type TimesTenClassic called sample and an object of type ConfigMap called sample simultaneously. There is no relationship between these two objects.

Kubernetes supports namespaces. Namespaces split a cluster into multiple independent ones. Each namespace has a completely independent set of names. There can be an object of type a called x in namespace1 and a different object of type a called x in namespace2. For example, you can define an object of type TimesTenClassic called sample in the namespace1 namespace and a different object of type TimesTenClassic called sample in the namespace2 namespace.


CRDs are cluster-scoped, not namespace-scoped. There can be different Operators in each namespace, but there can be only one CRD definition for the entire cluster.

Kubernetes object definitions are expressed in JSON or YAML. The examples in this book use YAML.

The Operator

The Operator automatically provisions and configures Pods, configures TimesTen in them, and creates and configures a pair of databases. The Operator monitors the Pods, the TimesTen instances, and the TimesTen databases and keeps them running. For example, in an active standby pair configuration, if the Pod containing the active database fails, the database in the standby Pod is automatically promoted to be the active by the Operator.

This Operator is configured through a Deployment. The replicas attribute of the Deployment specifies the number of replicas of the Operator that is desired. When you create the Deployment, it causes Kubernetes to create one or more Pods (depending on the number of replicas), each of which runs the Operator.

  • If you specify replicas: 1 in the Operator deployment, and the Operator fails, Kubernetes automatically spawns another Operator. When that new Operator starts up, it continues to manage the TimesTenClassic objects within the Deployment's namespace.

  • If you specify more than one replica in the Operator deployment, multiple Pods run the Operator. One of these is the active Operator and manages the TimesTenClassic objects in the namespace. The remaining Pods monitor the health of the active Operator. If this active Operator fails, one of the other replicas becomes the active and manages the TimesTenClassic objects within the Deployment's namespace.

Understanding how the Operator functions

When you create a TimesTenClassic object in the Kubernetes cluster, the process to create and configure your active standby pair of databases begins. The Operator is invoked and creates several Kubernetes objects that are required to run TimesTen. After the objects are created and linked together, the TimesTen containers run a script to configure and start the TimesTen agent. The Operator communicates with the TimesTen agent that is running in each Pod in order to monitor and control TimesTen. The Operator configures one database as the active database, copies the active database to the standby, and then configures the active standby pair replication scheme. The process is described in detail in these sections:

Objects created by the Operator

The Operator creates a number of Kubernetes objects that are required to run TimesTen, including a StatefulSet, a Service, and a Secret. These objects in turn create other objects. All of these objects are linked together by Kubernetes and are associated with the TimesTenClassic object you created. Figure 1-2, "Creating the TimesTenClassic object" shows the objects that are created and how they are linked together.

Figure 1-2 Creating the TimesTenClassic object

Description of Figure 1-2 follows
Description of ''Figure 1-2 Creating the TimesTenClassic object''

The objects that are created are described in the following sections:


The Operator creates a StatefulSet consisting of two Pods to run TimesTen. Each Pod has one or more PersistentVolumes (persistent storage), that are mounted in the TimesTen containers. This is where your TimesTen databases are stored. Applications running in the containers with PersistentVolumes mounted can create files that live beyond the lifetime of the container. (By default, all files that containers create and modify automatically vanish when the container exits. Containers are ephemeral.)

One attribute of a StatefulSet is the number of replicas that can be provisioned. Each TimesTenClassic object has an associated StatefulSet with two replicas. If one Pod fails, Kubernetes automatically creates a new one to replace it, and automatically mounts the appropriate PersistentVolume(s) to it.

For example, for a TimesTenClassic object called sample, the Operator creates a StatefulSet called sample, in the same Kubernetes namespace. The StatefulSet, in turn, create two Pods in the namespace, called sample-0 and sample-1.


A Kubernetes Service defines the set of network addresses and ports that should be exposed to applications in the cluster.

The Operator automatically creates a headless Service when you create the TimesTenClassic object. It automatically associates this Service with the StatefulSet. This causes Kubernetes to define entries in the Kubernetes cluster's DNS for the Pods in the StatefulSet, and to keep those DNS entries up to date.

A headless Service is used such that the DNS name/address entry for the active database is different than the DNS name/address entry for the standby database. This enables incoming client connections to be routed to the database that is active. For more information on a headless Service, see:


For a TimesTenClassic object called sample, a headless Service called sample is also created in the same Kubernetes namespace. This results in entries in the cluster's DNS for sample-0.sample.namespace.svc.cluster.local and sample-1.sample.namespace.svc.cluster.local.


The Operator creates a Secret to inject an SSL certificate into the TimesTen containers. This secures the communication between the Operator and the TimesTen agent.


The Stateful set creates two pods. Each Pod contains two containers:

  • The tt container. This TimesTen container is always present in the Pods. It executes the TimesTen agent and runs TimesTen.

  • The daemonlog container: This container copies the contents of the TimesTen ttmesg.log file to stdout, resulting in Kubernetes logging the file. This enables the daemon log of the TimesTen instances to be recorded by the Kubernetes infrastructure.


The Operator creates a Kubernetes event whenever important changes occur.

The TimesTen containers and the TimesTen agent

After the objects are created, the TimesTen containers run a script that configures and starts the TimesTen agent. The Operator communicates with the TimesTen agent running in each Pod, in order to configure, manage, and monitor TimesTen in that Pod. The agent provides an HTTPS endpoint in the Pod that the Operator uses to query and control the tt container in the Pod. If the TimesTen agent fails, the tt container automatically terminates and is re-spawned by Kubernetes. Figure 1-3, "The Operator and the TimesTen agent" illustrates the two way communication between the Operator and the TimesTen agent.

Figure 1-3 The Operator and the TimesTen agent

Description of Figure 1-3 follows
Description of ''Figure 1-3 The Operator and the TimesTen agent''

The TimesTen agent starts TimesTen and thus runs as the instance administrator user. It has full control over TimesTen.

Simple deployment of the TimesTen Operator

The TimesTen Operator is designed for simple deployment and automated failure detection and recovery. For example,

  • You decide you want to deploy a new replicated pair of TimesTen databases.

  • You decide the attributes of those databases.

  • You create the configuration files for those attributes.

  • You use the kubectl create command to create a TimesTenClassic object to represent the replicated pair.

  • You use the kubectl get and kubectl describe commands to observe the provisioning of the active standby pair.

  • Applications that run in other Pods use TimesTen's standard client/server drivers to access TimesTen databases.

You do not have to monitor the TimesTen databases continually, configure replication, perform failover, or re-duplicate a database after failure. The TimesTen Operator performs all these functions and works to keep the databases up and running with minimal effort on your part.