Fail Over

You are now ready to initiate the fail over from the active database to the standby.

Note:

During fail over, the TimesTen Operator takes down your active database, and immediately fails over to the standby. Do not perform this procedure at the busiest time of your production day. It's best to consider performing this operation during a scheduled production outage.

Before failing over, quiesce your applications on the active database. You can use the TimesTen ttAdmin -close and the ttAdmin -disconnect commands. See Opening and Closing the Database for User Connections and Disconnecting from a Database in the Oracle TimesTen In-Memory Database Operations Guide.

To avoid potential data loss, use the TimesTen ttRepAdmin -wait command to wait until replication is caught up, ensuring that transactions that were executed on the active database are replicated to the standby database. See ttRepAdmin in the Oracle TimesTen In-Memory Database Reference.

Once the standby is caught up, fail over from the active database to the standby by deleting the active Pod. When you delete the active Pod, the TimesTen Operator automatically detects the failure and promotes the standby database to be the active. Client/server applications that are using the active database are automatically reconnected to the new active database. Transactions in flight are rolled back. Prepared SQL statements do need to be re-prepared by the applications. See About Handling Failover and Recovery for more information.

Let's initiate the fail over.

  1. Delete the active Pod.
    kubectl delete pod manrepsample-0

    The output is the following.

    pod "manrepsample-0" deleted

    Kubernetes automatically creates a new manrepsample-0 Pod to replace the deleted Pod. The TimesTen Operator configures the new manresample-0 Pod as the standby Pod. This new Pod runs the upgraded TimesTen image.

  2. Monitor the progress of the upgrade, observing the state transitions.
     kubectl get events -w
    The output is similar to the following:
    LAST SEEN   TYPE      REASON                   OBJECT                         MESSAGE
...
37m         Warning   Upgrade                  timestenclassic/manrepsample   Image updated, automatic upgrade disabled
26m         Warning   Failed                   timestenclassic/manrepsample   Pod manrepsample-1 was replaced
26m         Normal    StateChange              timestenclassic/manrepsample   Pod manrepsample-1 is Not Ready
26m         Warning   StateChange              timestenclassic/manrepsample   TimesTenClassic was Normal, now ActiveTakeover
26m         Normal    StateChange              timestenclassic/manrepsample   TimesTenClassic was ActiveTakeover, now StandbyDown
24m         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-1 Agent Up
24m         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-1 Instance Exists
24m         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-1 Daemon Down
24m         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-1 Daemon Up
24m         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-1 Database Unloaded
24m         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-1 Database None
24m         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-1 Database Loaded
24m         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-1 RepAgent Not Running
24m         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-1 RepScheme Exists
24m         Normal    StateChange              timestenclassic/manrepsample   Pod manrepsample-1 RepState IDLE
23m         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-1 Database Loaded
23m         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-1 RepAgent Running
23m         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-1 RepScheme Exists
23m         Normal    StateChange              timestenclassic/manrepsample   Pod manrepsample-1 RepState STANDBY
23m         Normal    StateChange              timestenclassic/manrepsample   Pod manrepsample-1 is Ready
23m         Normal    StateChange              timestenclassic/manrepsample   TimesTenClassic was StandbyDown, now Normal
3m50s       Warning   Failed                   timestenclassic/manrepsample   Pod manrepsample-0 was replaced
3m44s       Normal    StateChange              timestenclassic/manrepsample   Pod manrepsample-0 is Not Ready
3m44s       Normal    StateChange              timestenclassic/manrepsample   Pod manrepsample-0 is Not Active Ready
3m44s       Warning   StateChange              timestenclassic/manrepsample   TimesTenClassic was Normal, now ActiveDown
3m43s       Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-1 Database Updatable
3m43s       Normal    StateChange              timestenclassic/manrepsample   Pod manrepsample-1 RepState ACTIVE
3m43s       Normal    StateChange              timestenclassic/manrepsample   Pod manrepsample-1 is Not Ready
3m43s       Normal    StateChange              timestenclassic/manrepsample   TimesTenClassic was ActiveDown, now ActiveTakeover
3m38s       Normal    StateChange              timestenclassic/manrepsample   Pod manrepsample-1 is Ready
3m38s       Normal    StateChange              timestenclassic/manrepsample   Pod manrepsample-1 is Active Ready
3m38s       Normal    StateChange              timestenclassic/manrepsample   TimesTenClassic was ActiveTakeover, now StandbyDown
119s        Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-0 Agent Up
119s        Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-0 Instance Exists
119s        Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-0 Daemon Down
118s        Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-0 Daemon Up
118s        Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-0 Database Unloaded
111s        Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-0 Database None
90s         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-0 Database Loaded
90s         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-0 RepAgent Not Running
90s         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-0 RepScheme Exists
90s         Normal    StateChange              timestenclassic/manrepsample   Pod manrepsample-0 RepState IDLE
84s         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-0 Database Loaded
84s         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-0 RepAgent Running
84s         Normal    Info                     timestenclassic/manrepsample   Pod manrepsample-0 RepScheme Exists
84s         Normal    StateChange              timestenclassic/manrepsample   Pod manrepsample-0 RepState STANDBY
84s         Normal    StateChange              timestenclassic/manrepsample   Pod manrepsample-0 is Ready
84s         Normal    StateChange              timestenclassic/manrepsample   TimesTenClassic was StandbyDown, now Normal

    The upgrade process completes.

  3. Confirm which Pod is the active.
    kubectl get ttc manrepsample

    The output is similar to the following:

    NAME           STATE    ACTIVE           AGE
manrepsample   Normal   manrepsample-1   3h21m

    The TimesTen Operator promoted the standby manrepsample-1 database to be the active. Applications that were connected to the manrepsample-0 database are automatically reconnected to the manrepsample-1 database by TimesTen. After a brief outage, the applications can continue to use the database.

  4. Verify the active and standby databases are running the correct release.
    1. Establish a shell in the active Pod.
      kubectl exec -it manrepsample-1 -c tt -- /bin/bash
    2. Run the TimesTen ttVersion utility.
      ttVersion
      The output is similar to the following:
      TimesTen Release 22.1.1.35.0 (64 bit Linux/x86_64) (instance1:6624) 2025-01-16T15:16:01Z
  Instance admin: timesten
  Instance home directory: /tt/home/timesten/instances/instance1
  Group owner: timesten
  Daemon home directory: /tt/home/timesten/instances/instance1/info
  PL/SQL enabled.
    3. Exit from the shell.
    4. Establish a shell in the standby Pod.
      kubectl exec -it manrepsample-0 -c tt -- /bin/bash
    5. Run the TimesTen ttVersion utility.
      ttVersion
      The output is similar to the following:
      TimesTen Release 22.1.1.35.0 (64 bit Linux/x86_64) (instance1:6624) 2025-01-16T15:16:01Z
  Instance admin: timesten
  Instance home directory: /tt/home/timesten/instances/instance1
  Group owner: timesten
  Daemon home directory: /tt/home/timesten/instances/instance1/info
  PL/SQL enabled.
    6. Exit from the shell.
    The active and standby databases are running the correct release of TimesTen.

Congratulations! You successfully performed a manual upgrade for a replicated TimesTenClassic object. The active and standby databases are upgraded, running, and fully operational.