Simulate Replica Set Failure with Manual

Let's simulate a total replica set failure where a TimesTenScaleout object's .spec.ttspec.replicaSetRecovery datum has a value of Manual.

Note:

This example is for demonstration purposes only. Do not attempt this example in a production environment.

Let's take a look at the TimesTenScaleout object definition.

cat samplescaleout2.yaml
apiVersion: timesten.oracle.com/v1
kind: TimesTenScaleout
metadata:
  name: samplescaleout2
spec:
  ttspec:
    storageClassName: oci-bv
    storageSize: 250G
    image: container-registry.oracle.com/timesten/timesten:22.1.1.19.0
    imagePullSecret: sekret
    replicaSetRecovery: Manual
    dbConfigMap:
    - samplescaleout2
    k: 2
    nReplicaSets: 3
    nMgmt: 1
    nZookeeper: 3

Note the .spec.ttspec.replicaSetRecovery datum for the object has a value of Manual, indicating that the Operator will set this TimesTenScaleout object's High Level state to ManualInterventionRequired when a total replica set failure occurs.

Let's simulate a total replica set failure with this object. Before we begin, let's do a quick check of the state of the object.

kubectl get tts samplescaleout2
NAME              OVERALL   MGMT     CREATE    LOAD              OPEN   AGE
samplescaleout2   Normal    Normal   created   loaded-complete   open   11m

Note the High Level state is Normal, the management state is Normal, and the database state is created,loaded-complete,open.

In this example, there are three replica sets.

kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
samplescaleout2-data-1-0             2/2     Running   0          12m
samplescaleout2-data-1-1             2/2     Running   0          12m
samplescaleout2-data-1-2             2/2     Running   0          12m
samplescaleout2-data-2-0             2/2     Running   0          12m
samplescaleout2-data-2-1             2/2     Running   0          12m
samplescaleout2-data-2-2             2/2     Running   0          12m
samplescaleout2-mgmt-0               2/2     Running   0          13m
samplescaleout2-zk-0                 1/1     Running   0          13m
samplescaleout2-zk-1                 1/1     Running   0          12m
samplescaleout2-zk-2                 1/1     Running   0          11m
timesten-operator-7677964df9-sp2zp   1/1     Running   0          7d6h

Let's delete the samplescaleout2-data-1-0 and samplescaleout2-data-2-0 Pods that belong to one of the replica sets.

Delete the Pods.

kubectl delete pod samplescaleout2-data-1-0;kubectl delete pod samplescaleout2-data-2-0
pod "samplescaleout2-data-1-0" deleted
pod "samplescaleout2-data-2-0" deleted

Use the kubectl get command to observe state transitions.
```
kubectl get tts samplescaleout2
NAME              OVERALL           MGMT     CREATE    LOAD                OPEN   AGE
samplescaleout2   DatabasePartial   Normal   created   loaded-incomplete   open   18m
```
The High Level state is DatabasePartial indicating that the database is up, but some data is not available. One or more replica sets have failed completely. The database loaded state is loaded-incomplete, indicating that at least one replica set has no elements that finished loading successfully.
```
kubectl get tts samplescaleout2
NAME              OVERALL                      MGMT     CREATE    LOAD                 OPEN   AGE
samplescaleout2   ManualInterventionRequired   Normal   created   loading-incomplete   open   20m
```
The object transitions to the ManualInterventionRequired High Level state. The Operator takes no further action to fix the object. The database loaded state remains loading-incomplete. Recall that you can set the .spec.ttspec.reexamine datum to cause the Operator to re-engage with the object. See Set reexamine Datum for details.

There was a total replica set failure. Because the TimesTen Scaleout object's .spec.ttspec.replicaSetRecovery datum had a value of Manual, the Operator set the object to the ManualInterventionRequired state. Review the information in Recovering from a Down Replica Set in the Oracle TimesTen In-Memory Database Scaleout User's Guide for details about how to fix this failure. Next, review Set reexamine Datum in this book to give control back to the Operator.