This chapter illustrates how the Operator recovers from failure.
The Operator automatically detects failures of the active TimesTen database and the standby TimesTen database and works to fix any failures. When the Operator detects a failure of the active database, it promotes the standby TimesTen database to be the active. Client/server applications that are using the database are automatically reconnected to the new active database. Transactions in flight are rolled back. Prepared statements need to be re-prepared by the applications. The Operator will configure a new standby database.
This example simulates a failure of the active TimesTen database. This is for demonstration purposes only. Do not do this in a production environment.
Use the kubectl
delete
pod
command to delete the active database (sample-0
in this case)
% kubectl delete pod sample-0
Use the kubectl
describe
command to observe how the Operator recovers from the failure. The Operator promotes the standby database (sample-1
) to be active. Any applications that were connected to the sample-0
database are automatically reconnected to the sample-1
database by TimesTen. After a brief outage, the applications can continue to use the database. See "Monitoring the health of the active standby pair of databases" for information on the health and states of the active standby pair.
Note: In this example, the text for the Message
column displays on two lines for three state changes. However, the actual output displays on one line for each of these three state changes.
% kubectl describe ttc sample Name: sample ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- - StateChange 2m1s ttclassic TimesTenClassic sample: was Normal, now ActiveDown - StateChange 115s ttclassic Pod sample-1 Database Updatable: Yes - StateChange 115s ttclassic TimesTenClassic sample:was ActiveDown, now StandbyDown - StateChange 115s ttclassic Pod sample-1 RepState ACTIVE - StateChange 110s ttclassic Pod sample-0 High Level State Unknown - StateChange 63s ttclassic Pod sample-0 Pod Phase Running - StateChange 63s ttclassic Pod sample-0 Agent Up - StateChange 63s ttclassic Pod sample-0 Instance Exists - StateChange 63s ttclassic Pod sample-0 Daemon Up - StateChange 63s ttclassic Pod sample-0 Database None - StateChange 42s ttclassic Pod sample-0 Database Loaded - StateChange 42s ttclassic Pod sample-0 Database Updatable: No - StateChange 42s ttclassic Pod sample-0 RepAgent Running - StateChange 42s ttclassic Pod sample-0 CacheAgent Not Running - StateChange 42s ttclassic Pod sample-0 RepScheme Exists - StateChange 42s ttclassic Pod sample-0 RepState IDLE - StateChange 36s ttclassic Pod sample-0 High Level State Healthy - StateChange 36s ttclassic Pod sample-0 RepState STANDBY - StateChange 36s ttclassic TimesTenClassic sample:was StandbyDown, now Normal
Kubernetes has automatically respawned a new sample-0
Pod to replace the Pod you deleted. The Operator configured TimesTen within that Pod, bringing the database in the Pod up as the new standby database. The replicated pair of databases are once again functioning normally.