Illustrate the Failover and Recovery Process

This example simulates a failure of the active TimesTen database. This is for demonstration purposes only. Do not do this in a production environment.

1. Use the kubectl delete pod command to delete the active database (sample-0 in this case)

   % kubectl delete pod sample-0
   

2. Use the kubectl describe command to observe how the Operator recovers from the failure. The Operator promotes the standby database (sample-1) to be active. Any applications that were connected to the sample-0 database are automatically reconnected to the sample-1 database by TimesTen. After a brief outage, the applications can continue to use the database. See "About Monitoring the Health of an Active Standby Pair of Databases" for information on the health and states of the active standby pair.

   Note: In this example, the text for the Message column displays on two lines for three state changes. However, the actual output displays on one line for each of these three state changes.

   % kubectl describe ttc sample
   Name:         sample
   ...
   Events:
     Type  Reason       Age    From       Message
     ----  ------       ----   ----       -------
     -     StateChange  2m1s   ttclassic  TimesTenClassic sample: was Normal, now ActiveDown
     -     StateChange  115s   ttclassic  Pod sample-1 Database Updatable: Yes
     -     StateChange  115s   ttclassic  TimesTenClassic sample:was ActiveDown, now StandbyDown
     -     StateChange  115s   ttclassic  Pod sample-1 RepState ACTIVE
     -     StateChange  110s   ttclassic  Pod sample-0 High Level State Unknown
     -     StateChange  63s    ttclassic  Pod sample-0 Pod Phase Running
     -     StateChange  63s    ttclassic  Pod sample-0 Agent Up
     -     StateChange  63s    ttclassic  Pod sample-0 Instance Exists
     -     StateChange  63s    ttclassic  Pod sample-0 Daemon Up
     -     StateChange  63s    ttclassic  Pod sample-0 Database None
     -     StateChange  42s    ttclassic  Pod sample-0 Database Loaded
     -     StateChange  42s    ttclassic  Pod sample-0 Database Updatable: No
     -     StateChange  42s    ttclassic  Pod sample-0 RepAgent Running
     -     StateChange  42s    ttclassic  Pod sample-0 CacheAgent Not Running
     -     StateChange  42s    ttclassic  Pod sample-0 RepScheme Exists
     -     StateChange  42s    ttclassic  Pod sample-0 RepState IDLE
     -     StateChange  36s    ttclassic  Pod sample-0 High Level State Healthy
     -     StateChange  36s    ttclassic  Pod sample-0 RepState STANDBY
     -     StateChange  36s    ttclassic  TimesTenClassic sample:was StandbyDown,now Normal
   

   Kubernetes has automatically respawned a new sample-0 Pod to replace the Pod you deleted. The Operator configured TimesTen within that Pod, bringing the database in the Pod up as the new standby database. The replicated pair of databases are once again functioning normally.