Simulate Single Data Instance Failure

Let's simulate a single data instance failure and observe how a TimesTenScaleout object transitions through various state changes.

Note:

This example is for demonstration purposes only. Do not attempt this example in a production environment.

In the example, there is a deployed TimesTenScaleout object that is functioning properly.

kubectl get tts samplescaleout
NAME             OVERALL   MGMT     CREATE    LOAD              OPEN   AGE
samplescaleout   Normal    Normal   created   loaded-complete   open   10m

Note the High Level state is Normal, the management state is Normal, and the database state is created,loaded-complete,open.

To simulate a single data instance failure, let's delete a Pod that contains a data instance. Here are the Pods:

kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
samplescaleout-data-1-0              2/2     Running   0          11m
samplescaleout-data-1-1              2/2     Running   0          11m
samplescaleout-data-1-2              2/2     Running   0          11m
samplescaleout-data-2-0              2/2     Running   0          11m
samplescaleout-data-2-1              2/2     Running   0          11m
samplescaleout-data-2-2              2/2     Running   0          11m
samplescaleout-mgmt-0                2/2     Running   0          11m
samplescaleout-zk-0                  1/1     Running   0          11m
samplescaleout-zk-1                  1/1     Running   0          10m
samplescaleout-zk-2                  1/1     Running   0          9m35s
timesten-operator-7677964df9-sp2zp   1/1     Running   0          7d3h

Let's delete the samplescaleout-data-1-0 Pod and observe the behavior.

Delete the Pod.

kubectl delete pod samplescaleout-data-1-0
pod "samplescaleout-data-1-0" deleted

Use the kubectl get command to observe state transitions.
```
kubectl get tts samplescaleout
NAME             OVERALL           MGMT     CREATE    LOAD                OPEN   AGE
samplescaleout   DatabaseImpeded   Normal   created   loaded-functional   open   16m
```
The High Level state is DatabaseImpeded, indicating that the database within the grid is fully operational, but one or more elements is not functional. The database loaded state is loaded-functional, indicating loading is in progress and at least one element from each replica set is loaded.
```
kubectl get tts samplescaleout
NAME             OVERALL   MGMT     CREATE    LOAD              OPEN   AGE
samplescaleout   Normal    Normal   created   loaded-complete   open   18m
```
The object transitioned to the Normal High Level state, indicating the grid and database are functioning normally. The database state is loaded-complete, indicating the element loaded successfully.

Even though there was a single data instance failure, TimesTen Scaleout fully recovered. There was no manual intervention required.