Deploy a Grid

Create Configuration Metadata and a Kubernetes ConfigMap for a Grid

The following example creates the db.ini, the adminUser, and the schema.sql metadata files for the TimesTen grid and the database. The example also creates a direct and a client/server connectable. The example creates a Kubernetes ConfigMap to place these metadata files into the /ttConfig directory of the TimesTen containers.

Note:

You can use any Kubernetes mechanism to place these metadata files into the /ttConfig directory of the TimesTen containers. See Populate the /ttconfig Directory.

On your development host, complete the following steps:

From the directory of your choice, create an empty subdirectory for the metadata files. This example creates the cm_scaleout subdirectory. (The cm_scaleout directory is used in the remainder of this example to denote this directory.)
```
mkdir -p cm_scaleout
```
Change to this ConfigMap directory.
```
cd cm_scaleout
```
Create the db.ini file in this ConfigMap directory. In this example, define the PermSize and the DatabaseCharacterSet connection attributes.
```
vi db.ini

PermSize=200
DatabaseCharacterSet=AL32UTF8
```
Create the adminUser file.
```
vi adminUser

adminuser/adminuserpwd
```
Create the schema.sql file. In this schema.sql file, create the sampleuser user, create the s sequence for the sampleuser user, and the emp table for the sampleuser user. The Operator automatically initializes each element of the TimesTen database with these object definitions.
```
vi schema.sql

create user sampleuser identified by sampleuserpwd;
grant admin to sampleuser;
create sequence sampleuser.s;
create table sampleuser.emp (id number not null primary key, name char (32));
```

Create the sampledirect direct connectable.

vi sampledirect.connect

ConnectionCharacterSet=AL32UTF8

Create the sampleclient client/server connectable.

vi sampleclient.csconnect

ConnectionCharacterSet=AL32UTF8
TTC_Timeout=30

Optional: Verify the metadata files are present in the cm_scaleout ConfigMap directory.
```
ls
```
The output is the following:
```
adminUser
db.ini
sampleclient.csconnect
sampledirect.connect
schema.sql
```
From the cm_scaleout directory, create the samplescaleout ConfigMap. The files in the cm_scaleout directory are included in the ConfigMap. These files are later available in the TimesTen containers. Replace samplescaleout with a name of your choosing.
```
kubectl create configmap samplescaleout --from-file .
```
The output is the following:
```
configmap/samplescaleout created
```

Use the kubectl describe command to verify the contents of the samplescaleout ConfigMap.

kubectl describe configmap samplescaleout

The output is the following:

Name:         samplescaleout
Namespace:    mynamespace
Labels:       <none>
Annotations:  <none>

Data
====
adminUser:
----
adminuser/adminuserpwd

db.ini:
----
PermSize=200
DatabaseCharacterSet=AL32UTF8

sampleclient.csconnect:
----
ConnectionCharacterSet=AL32UTF8
TTC_Timeout=30

sampledirect.connect:
----
ConnectionCharacterSet=AL32UTF8

schema.sql:
----
create user sampleuser identified by sampleuserpwd;
grant admin to sampleuser;
create sequence sampleuser.s;
create table sampleuser.emp (id number not null primary key, name char (32));

Events:  <none>

You successfully created the metadata files and the ConfigMap.

Define and Deploy a TimesTenScaleout Object

Defining your environment involves creating your TimesTenScaleout object with attributes customized for your environment. These attributes are described in the steps below. For additional information on defining objects of type TimesTenScaleout, see TimesTen Scaleout.

To define and create the TimesTenScaleout object, complete the following steps:

Create a YAML file. You can choose any name for this YAML file, but you may want to use the same name you used for the name of the TimesTenScaleout object. (This example uses samplescaleout.) The YAML file contains the definitions for the TimesTenScaleout object. In this example, the fields that are specific to a TimesTenScaleout object are as follows:
- k: Set the value of k to the number of copies of data for your TimesTen database. This value determines the number of StatefulSets that the TimesTen Operator creates. A StatefulSet provides the Pods that are used to implement a single data space in the grid. For example, if you set k to 2, the Operator creates two StatefulSets. One StatefulSet provides the Pods for the data instances in data space one. The second StatefulSet provides the Pods for the data instances in data space two.
  
  This example sets k to 2.
  
  For information on K-safety and determining an appropriate value for k, see K-Safety in the Oracle TimesTen In-Memory Database Scaleout User's Guide.
- nReplicaSets: Set the value to the number of replica sets in the grid. A replica set contains k elements (where each element is an exact copy of the other elements in the replica set). The nReplicaSets value determines the number of replicas for each StatefulSet. For example, if you set k to 2, the TimesTen Operator creates two StatefulSets for the data instances. If you set nReplicaSets to 3, each StatefulSet contains three replicas, and the total number of replica sets in the database is three.
  
  This example sets nReplicaSets to 3.
  
  For information on replica sets, see Understanding Replica Sets in the Oracle TimesTen In-Memory Database Scaleout User's Guide.
- nZookeeper: Set the value to the number of ZooKeeper Pods to provision in a StatefulSet. Your options are 1 or 3 or 5.
  
  This example sets nZookeeper to 3.
You also need to specify the following fields:
- name: Replace samplescaleout with the name of your TimesTenScaleout object.
- storageClassName: Replace oci-bv with the name of the storage class in your Kubernetes cluster that is used to allocate Persistent Volumes to hold the TimesTen database.
- storageSize: Replace 250Gi with the amount of storage that should be requested for each Pod to hold TimesTen. (This example assumes a production environment and uses 250Gi for storage. For demonstration purposes, you can use 50Gi of storage.) See the storageSize and the logStorageSize entries in Table 17-7.
- image: Replace container-registry.oracle.com/timesten/timesten:22.1.1.19.0 with the location of the image registry and the name of the image. If you are using the Oracle container-registry.oracle.com/timesten registry as the image registry and the timesten:22.1.1.19.0 image as the container image, no replacement is necessary.
- imagePullSecret: Replace sekret with the image pull secret that Kubernetes should use to fetch the TimesTen image.
- dbConfigMap: This example uses one ConfigMap (called samplescaleout) for the db.ini, the adminUser, the schema.sql, the sampledirect.connect, and the sampleclient.csconnect metadata files. This ConfigMap is included in the Projected Volume. The volume is mounted as /ttconfig in the TimesTen containers. See Using ConfigMaps and Secrets.
```
vi samplescaleout.yaml

kind: TimesTenScaleout
metadata:
  name: samplescaleout
spec:
  ttspec:
    storageClassName: oci-bv
    storageSize: 250Gi
    image: container-registry.oracle.com/timesten/timesten:22.1.1.19.0
    imagePullSecret: sekret
    dbConfigMap:
    - samplescaleout
    k: 2
    nReplicaSets: 3
    nZookeeper: 3
```
Create the TimesTenScaleout object from the contents of the YAML file. Doing so begins the process of creating and provisioning a TimesTen grid and database in your Kubernetes cluster.
```
kubectl create -f samplescaleout.yaml
```
The output is the following:
```
timestenscaleout.timesten.oracle.com/samplescaleout created
```

You successfully created the TimesTenScaleout object in the Kubernetes cluster. The process of provisioning your TimesTen grid and database begins, but is not yet complete.

Monitor the High Level State of a TimesTenScaleout Object

Use the kubectl get and the kubectl describe commands to monitor the progress of the grid and database creation.

Note:

For the kubectl get timestenscaleout command, you can alternatively specify kubectl get tts. When used in the kubectl get command, timestenscaleout and tts are synonymous, and return the same results. The first kubectl get examples in this chapter use timestenscaleout. For simplicity, the remaining examples in this book use tts.

Review the High Level state of the TimesTenScaleout object. Use the kubectl get command and observe the STATE field. Notice that the value is Initializing. The Operator has created the Kubernetes StatefulSets and Services. The process to deploy and provision your grid and database has begun, but is not yet complete. As you issue additional kubectl get commands, observe how the TimesTenScaleout object transitions to different states. For more information on these states, see About Creating a Grid.

kubectl get tts samplescaleout

The output is similar to the following:

NAME             OVERALL        MGMT   CREATE   LOAD   OPEN   AGE
samplescaleout   Initializing                                 20s

kubectl get tts samplescaleout

The output is similar to the following:

NAME             OVERALL          MGMT   CREATE   LOAD   OPEN   AGE
samplescaleout   ZookeeperReady                                 2m48s

kubectl get tts samplescaleout

The output is similar to the following:

NAME             OVERALL       MGMT   CREATE   LOAD   OPEN   AGE
samplescaleout   GridCreated                                 3m58s

kubectl get tts samplescaleout

The output is similar to the following:

NAME             OVERALL            MGMT   CREATE   LOAD   OPEN   AGE
samplescaleout   InstancesCreated                                 6m4s

kubectl get tts samplescaleout

The output is similar to the following:

NAME             OVERALL   MGMT     CREATE    LOAD              OPEN   AGE
samplescaleout   DatabaseCreated                                       6m10s

Use the kubectl get command again to observe if the High Level state has transitioned from DatabaseCreated to Normal. A Normal state indicates the grid and database are provisioned and the process is complete.
```
kubectl get tts samplescaleout
```
The output is similar to the following:
```
NAME             OVERALL   MGMT     CREATE    LOAD              OPEN   AGE
samplescaleout   Normal    Normal   created   loaded-complete   open   6m52s
```

Your TimesTen Scaleout grid and database are successfully created and provisioned in your Kubernetes cluster. The database is open for connections.

Verify Underlying Objects

The TimesTen Operator creates the following objects for the samplescaleout TimesTenScaleout object.

StatefulSets:
```
kubectl get statefulsets
```
The output is the following:
```
NAME                    READY   AGE
samplescaleout-data-1   3/3     26m
samplescaleout-data-2   3/3     26m
samplescaleout-mgmt     1/1     26m
samplescaleout-zk       3/3     26m
```
The Operator creates the samplescaleout-data-1 and the samplescaleout-data-2 StatefulSets. Two StatefulSets are created because the value of k is set to 2. Each StatefulSet provides the Pods for a single data space. There are two data spaces in the grid. Each of the StatefulSets has three replicas (nReplicaSets is set to 3). Therefore, the number of replica sets in the grid is three. There are three data instances in data space one and three data instances in data space two.

The samplescaleout-mgmt StatefulSet provides the Pod for the management instance. There is one management instance.

The samplscaleout-zk StatefulSet provides the Pods for the ZooKeeper instances. There are three ZooKeeper instances (nZookeeper is set to 3).
Pods:
```
kubectl get pods
```
The output is the following:
```
NAME                                 READY   STATUS    RESTARTS   AGE
samplescaleout-data-1-0              2/2     Running   0          27m
samplescaleout-data-1-1              2/2     Running   0          27m
samplescaleout-data-1-2              2/2     Running   0          27m
samplescaleout-data-2-0              2/2     Running   0          27m
samplescaleout-data-2-1              2/2     Running   0          27m
samplescaleout-data-2-2              2/2     Running   0          27m
samplescaleout-mgmt-0                2/2     Running   0          27m
samplescaleout-zk-0                  1/1     Running   0          27m
samplescaleout-zk-1                  1/1     Running   0          26m
samplescaleout-zk-2                  1/1     Running   0          25m
timesten-operator-7677964df9-sp2zp   1/1     Running   0          3d20h
```
The samplescaleout-data-1 StatefulSet contains the data instances in data space one. This StatefulSet creates the samplescaleout-data-1-0, samplescaleout-data-1-1, and samplescaleout-data-1-2 Pods. The samplescaleout-data-2 contains the data instances in data space two. This StatefulSet creates the samplescaleout-data-2-0, samplescaleout-data-2-1, and samplescaleout-data-2-2 Pods.

The samplescaleout-mgmt-0 StatefulSet contains the management instance for the grid. This StatefulSet creates the samplescaleout-mgmt-0 Pod.

Each of the Pods previously mentioned run TimesTen. Each Pod contains at least two containers. In this example, each Pod contains two containers. The tt container runs the TimesTen agent and TimesTen. The daemonlog container writes the TimesTen daemon log to stdout.

The samplescaleout-zk StatefulSet contains the three ZooKeepers instances. This StatefulSet creates the samplescaleout-zk-1, samplescaleout-zk-2, and samplescaleout-zk-3 Pods. Each of these Pods contain a single container called zookeeper.

The TimesTen Operator is running in the timesten-operator-554887b4c-48zwk Pod.

Headless Services:

kubectl get services

The output is the following:

NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
samplescaleout      ClusterIP   None            <none>        6625/TCP                        28m
samplescaleout-zk   ClusterIP   None            <none>        2181/TCP,2888/TCP,3888/TCP      28m

The Operator creates the samplescaleout Headless Service. This Service provides the DNS names for the Pods that contain the TimesTen management and data instances. The Service allows client/server access to the Pods that use the client/server port (6625). The DNS names are in the format: podname.samplescaleout.namespace.svc.cluster.local.

The following example shows the DNS names for the samplescaleout Headless Service:

samplescaleout-mgmt-0.samplescaleout.mynamespace.svc.cluster.local
samplescaleout-data-1-0.samplescaleout.mynamespace.svc.cluster.local
samplescaleout-data-1-1.samplescaleout.mynamespace.svc.cluster.local
samplescaleout-data-1-2.samplescaleout.mynamespace.svc.cluster.local
samplescaleout-data-2-0.samplescaleout.mynamespace.svc.cluster.local
samplescaleout-data-2-1.samplescaleout.mynamespace.svc.cluster.local
samplescaleout-data-2-2.samplescaleout.mynamespace.svc.cluster.local

The Operator creates a second Headless Service called samplescaleout-zk. This Service allows access to the ZooKeeper internal ports (2888 and 3888) as well as the external port (2181). The DNS names are in the format: samplescaleout-zk-n.samplescaleout-zk.namespace.svc.cluster.local.

The following example shows the DNS names for the samplescaleout-zk Headless Service:

samplescaleout-zk-0.samplescaleout-zk.mynamespace.svc.cluster.local
samplescaleout-zk-1.samplescaleout-zk.mynamespace.svc.cluster.local
samplescaleout-zk-2.samplescaleout-zk.mynamespace.svc.cluster.local

Persistent Volume Claims (PVCs):

kubectl get pvc

The output is the following:


NAME                                    STATUS   VOLUME                                                                              CAPACITY      ACCESS MODES   STORAGECLASS   AGE
tt-persistent-samplescaleout-data-1-0   Bound    ocid1.volume.oc1.phx.abyhqljtl63wgaxd3nvengilkaxs5h2b23kmtinpzmmqt7bkzjdpnfu2c2fq   53687091200   RWO            oci-bv            14m
tt-persistent-samplescaleout-data-1-1   Bound    ocid1.volume.oc1.phx.abyhqljthxuwtpqx7rjtwvwsxjjkbgr25amjk7wtk26untbqrealcqhe324q   53687091200   RWO            oci-bv            14m
tt-persistent-samplescaleout-data-1-2   Bound    ocid1.volume.oc1.phx.abyhqljthtm6frcjtttp6ye7hq4w5vm3jxyay54f4xtcbolv2ercjeca5khq   53687091200   RWO            oci-bv            14m
tt-persistent-samplescaleout-data-2-0   Bound    ocid1.volume.oc1.phx.abyhqljtxgzm3raxj5sfoe56aonlh2mqqjre4quva4k3q3zbbe7lftwqk3xa   53687091200   RWO            oci-bv            14m
tt-persistent-samplescaleout-data-2-1   Bound    ocid1.volume.oc1.phx.abyhqljtk3htdair4akll5dfrwpkipv3acjtww5hx3x2fz46af7zdew7gsiq   53687091200   RWO            oci-bv            14m
tt-persistent-samplescaleout-data-2-2   Bound    ocid1.volume.oc1.phx.abyhqljtiekgulpbadtwigehsuml75sngcuqjgqmx77lfpi3wdeecbecavfa   53687091200   RWO            oci-bv            14m
tt-persistent-samplescaleout-mgmt-0     Bound    ocid1.volume.oc1.phx.abyhqljtacxwop7r2wqx6hsvun4haaydnco3y6g3rkgbwp5hq35alay7uwaq   53687091200   RWO            oci-bv            14m
tt-persistent-samplescaleout-zk-0       Bound    ocid1.volume.oc1.phx.abyhqljtyoa5hdchax4sus652jtp665ckaef2cq3lakac2lq52vfbls6kkcq   53687091200   RWO            oci-bv            14m
tt-persistent-samplescaleout-zk-1       Bound    ocid1.volume.oc1.phx.abyhqljtongpcoggzpg2is25vmumijmah5gustwc3avgnijrjigtqphtrana   53687091200   RWO            oci-bv            13m
tt-persistent-samplescaleout-zk-2       Bound    ocid1.volume.oc1.phx.abyhqljttmgoljskb2ruawzv365uit7lsln2sbfno5e4vhh6plbgh4tiblfq   53687091200   RWO            oci-bv            10m

The Operator automatically creates one or two Persistent Volume Claims (PVCs) per Pod. These PVCs cause Persistent Volumes (PVs) to be allocated by Kubernetes and attached to the TimesTen Pods. TimesTen uses these PVs to hold the TimesTen instance and database. If you specify two PVCs, one PV holds the TimesTen instance and the checkpoint files and the second PV holds the TimesTen transaction logs. In this example, the Operator creates one PVC for each Pod for a total of six PVCs. Each of the six PVs hold the TimesTen instance and an element of the database.

The Operator creates one PVC for the Pod that contains the management instance.

The Operator creates one PVC for each Pod that runs a ZooKeeper instance. This PVC causes a PV to be allocated by Kubernetes and attached to the ZooKeeper Pod. Each PV holds ZooKeeper's persistent data. In this example, since there are three ZooKeeper instances, the Operator created three PVCs.

Connect to the Database

You can establish a shell in a TimesTen Pod and connect to the TimesTen database in the grid. You can then run operations in this TimesTen database.

Establish a shell in the TimesTen samplescaleout-data-1-0 Pod.
```
kubectl exec -it samplescaleout-data-1-0 -c tt -- /bin/bash
```

Connect to the samplescaleout database. Verify the information from the metadata files is in the database. This example connects to the database as the sampleuser user. (This user was created in the schema.sql file). The example then calls the ttConfiguration built-in procedure to check that the PermSize connection attribute has a value of 200 for this element of the database. The example then verifies the sampleuser.emp table exists.

ttisql -connstr "DSN=samplescaleout;uid=sampleuser;pwd=sampleuserpwd";

The output is similar to the following:

Copyright (c) 1996, 2023, Oracle and/or its affiliates. All rights reserved.
Type ? or "help" for help, type "exit" to quit ttIsql.



connect "DSN=samplescaleout;uid=sampleuser;pwd=********";
Connection successful: DSN=samplescaleout;Durability=0;UID=sampleuser;DataStore=/tt/home/timesten/datastore/samplescaleout;
DatabaseCharacterSet=AL32UTF8;ConnectionCharacterSet=AL32UTF8;PermSize=200;
Connections=100;
(Default setting AutoCommit=1)
Command> call ttConfiguration ('PermSize');
< PermSize, 200 >
1 row found.
Command> tables;
  SAMPLEUSER.EMP
1 table found.

Manage a Grid and Its Database

You can establish a shell in the TimesTen Pod of the management instance. You can then use the ttGridAdmin utility to manage and monitor the grid, including the health of the TimesTen database and its elements.

Establish a shell in the TimesTen samplescaleout-management-0 Pod.
```
kubectl exec -it samplescaleout-mgmt-0 -c tt -- /bin/bash
```

Run the ttGridAdmin dbStatus -all to check the status of the TimesTen database.

ttGridAdmin dbStatus -all

The output is the following:

Database samplescaleout summary status as of Sun Jan 15 17:00:11 UTC 2023

created,loaded-complete,open
Completely created elements: 6 (of 6)
Completely loaded elements: 6 (of 6)
Completely created replica sets: 3 (of 3)
Completely loaded replica sets: 3 (of 3)

Open elements: 6 (of 6)

Database samplescaleout element level status as of Sun Jan 15 17:00:11 UTC 2023

Host                    Instance  Elem Status Cache Agent Date/Time of Event  Message
----------------------- --------- ---- ------ ----------- ------------------- -------
samplescaleout-data-1-0 instance1    1 opened stopped     2023-01-15 16:28:57
samplescaleout-data-1-1 instance1    2 opened stopped     2023-01-15 16:28:57
samplescaleout-data-1-2 instance1    3 opened stopped     2023-01-15 16:28:56
samplescaleout-data-2-0 instance1    4 opened stopped     2023-01-15 16:28:57
samplescaleout-data-2-1 instance1    5 opened stopped     2023-01-15 16:28:56
samplescaleout-data-2-2 instance1    6 opened stopped     2023-01-15 16:28:56

Database samplescaleout Replica Set status as of Sun Jan 15 17:00:11 UTC 2023

RS DS Elem Host                    Instance  Status Cache Agent Date/Time of Event  Message
-- -- ---- ----------------------- --------- ------ ----------- ------------------- -------
 1  1    1 samplescaleout-data-1-0 instance1 opened stopped     2023-01-15 16:28:57       
 1  2    4 samplescaleout-data-2-0 instance1 opened stopped     2023-01-15 16:28:57       
 2  1    2 samplescaleout-data-1-1 instance1 opened stopped     2023-01-15 16:28:57       
 2  2    5 samplescaleout-data-2-1 instance1 opened stopped     2023-01-15 16:28:56       
 3  1    3 samplescaleout-data-1-2 instance1 opened stopped     2023-01-15 16:28:56       
 3  2    6 samplescaleout-data-2-2 instance1 opened stopped     2023-01-15 16:28:56       

Database samplescaleout Data Space Group status as of Sun Jan 15 17:00:11 UTC 2023

DS RS Elem Host                    Instance  Status Cache Agent Date/Time of Event  Message
-- -- ---- ----------------------- --------- ------ ----------- ------------------- -------
 1  1    1 samplescaleout-data-1-0 instance1 opened stopped     2023-01-15 16:28:57       
 1  2    2 samplescaleout-data-1-1 instance1 opened stopped     2023-01-15 16:28:57       
 1  3    3 samplescaleout-data-1-2 instance1 opened stopped     2023-01-15 16:28:56       
 2  1    4 samplescaleout-data-2-0 instance1 opened stopped     2023-01-15 16:28:57       
 2  2    5 samplescaleout-data-2-1 instance1 opened stopped     2023-01-15 16:28:56       
 2  3    6 samplescaleout-data-2-2 instance1 opened stopped     2023-01-15 16:28:56