Simulate Management Instance Failure

Let's simulate a management instance failure. Let's observe how a TimesTenScaleout object transitions through various state changes.

Note:

This example is for demonstration purposes only. Do not attempt this example in a production environment.

In the example, there is a deployed TimesTenScaleout object that is functioning properly.

kubectl get tts samplescaleout
NAME             OVERALL   MGMT     CREATE    LOAD              OPEN   AGE
samplescaleout   Normal    Normal   created   loaded-complete   open   68m

Note the High Level state is Normal, the management state is Normal, and the database state is created,loaded-complete,open.

To simulate a management instance failure, let's delete the Pod that contains the management instance. Here are the Pods:

kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
samplescaleout-data-1-0              2/2     Running   0          57m
samplescaleout-data-1-1              2/2     Running   0          73m
samplescaleout-data-1-2              2/2     Running   0          73m
samplescaleout-data-2-0              2/2     Running   0          73m
samplescaleout-data-2-1              2/2     Running   0          73m
samplescaleout-data-2-2              2/2     Running   0          73m
samplescaleout-mgmt-0                2/2     Running   0          73m
samplescaleout-zk-0                  1/1     Running   0          73m
samplescaleout-zk-1                  1/1     Running   0          72m
samplescaleout-zk-2                  1/1     Running   0          71m
timesten-operator-7677964df9-sp2zp   1/1     Running   0          7d4h

Let's delete the samplescaleout-mgmt-0 Pod and observe the behavior.

Delete the Pod.

kubectl delete pod samplescaleout-mgmt-0
pod "samplescaleout-mgmt-0" deleted

Use the kubectl get command to observe state transitions.
```
kubectl get tts samplescaleout
NAME             OVERALL     MGMT         CREATE    LOAD              OPEN   AGE
samplescaleout   Unmanaged   ActiveDown   created   loaded-complete   open   79m
```
The High Level state is Unmanaged, indicating that the grid has no functional management instance. As a result, the grid cannot be further managed, monitored, or controlled. The management instance state is ActiveDown, indicating the management instance is down. Note that since the management instance is down, the Operator ignores the state of the data instances until the management instance is back up.
```
kubectl get tts samplescaleout
NAME             OVERALL     MGMT             CREATE    LOAD              OPEN   AGE
samplescaleout   Unmanaged   ActiveDaemonUp   created   loaded-complete   open   81m
```
The object remains in the Unmanaged High Level state. The management state transitions to ActiveDaemonUp, indicating the management instance has been started, but is not yet functional.
```
kubectl get tts samplescaleout
NAME             OVERALL   MGMT     CREATE    LOAD              OPEN   AGE
samplescaleout   Normal    Normal   created   loaded-complete   open   81m
```
The object transitioned to the Normal High Level state, indicating the grid and database are functioning normally. The management state is Normal, indicating that management instance is functioning normally.

Even though there was a management instance failure, TimesTen Scaleout fully recovered. There was no manual intervention required.