1. Kubernetes Operator User's Guide
  2. Perform Upgrades
  3. Perform a Manual Upgrade

Perform a Manual Upgrade

This section describes the process for performing a manual upgrade of your TimesTenClassic objects and includes the following subsections:

Modify a TimesTenClassic Object: Manual Upgrade

The manual upgrade process requires you to modify the .spec.ttspec.image datum of a TimesTenClassic object to reference the new TimesTen container image. After you modify the TimesTenClassic object to reference the new TimesTen image, the Operator notices the change and modifies the StatefulSet that it created.

Let's modify the TimesTenClassic sample2 object to reference the new TimesTen image. Let's assume the sample2 object's .spec.ttspec.imageUpgradeStrategy is Manual.

  1. On your development host, edit the sample TimesTenClassic object, changing the .spec.ttspec.image datum to reference the new TimesTen container image. In this example, the location and name of the new container image is container-registry.oracle.com/timesten/timesten:22.1.1.19.0.

    Note: Not all output is shown.

    kubectl edit timestenclassic sample2

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: timesten.oracle.com/v1
kind: TimesTenClassic
metadata:
  ...
  name: sample2
  namespace: mynamespace
  resourceVersion: "74928067"
  uid: e6d49f75-05ed-4b4b-9412-4a577ad19bbe
spec:
...
  ttspec:
    additionalMemoryRequest: 1Gi
    automaticMemoryRequests: true
    daemonLogCPURequest: 200m
    daemonLogMemoryRequest: 20Mi
    dbConfigMap:
    - sample2
    exporterCPURequest: 200m
    exporterMemoryRequest: 200Mi
    image: container-registry.oracle.com/timesten/timesten:22.1.1.19.0
    imagePullPolicy: Always
    imagePullSecret: sekret
    imageUpgradeStrategy: Manual
    memoryWarningPercent: 90
    storageClassName: oci-bv
    storageSize: 250G
...
    The output is the following.
    timestenclassic.timesten.oracle.com/sample2 edited
  2. Verify that the Operator has modified the sample2 StatefulSet and replaced the image with the new image. 
    kubectl describe statefulset sample2

    The output is similar to the following.

    Name:               sample2
Namespace:          mynamespace
...
Replicas:           2 desired | 2 total
Update Strategy:    OnDelete
Pods Status:        2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:       app=sample2
                database.timesten.oracle.com=sample2
  Annotations:  TTC: e6d49f75-05ed-4b4b-9412-4a577ad19bbe
  Init Containers:
   ttinit:
    Image:       container-registry.oracle.com/timesten/timesten:22.1.1.19.0
    Ports:       8443/TCP, 6624/TCP, 6625/TCP, 4444/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP
    Environment:
      TT_OPERATOR_MANAGED:      1
      TIMESTEN_HOME:            /tt/home/timesten/instances/instance1
      LD_LIBRARY_PATH:          /tt/home/timesten/instances/instance1/ttclasses/lib:/tt/home/timesten/instances/instance1/install/lib:/tt/home/timesten/instances/instance1/install/ttoracle_home/instantclient_11_2:/tt/home/timesten/instances/instance1/install/ttoracle_home/instantclient
      TT_REPLICATION_TOPOLOGY:  activeStandbyPair
      TT_INIT_CONTAINER:        1
      TTC_UID:                  e6d49f75-05ed-4b4b-9412-4a577ad19bbe
    Mounts:
      /tt from tt-persistent (rw)
      /ttagent from tt-agent (rw)
      /ttconfig from tt-config (rw)
  Containers:
   tt:
    Image:       container-registry.oracle.com/timesten/timesten:22.1.1.19.0
    Ports:       8443/TCP, 6624/TCP, 6625/TCP, 4444/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP
    Environment:
      TT_OPERATOR_MANAGED:      1
      TIMESTEN_HOME:            /tt/home/timesten/instances/instance1
      LD_LIBRARY_PATH:          /tt/home/timesten/instances/instance1/ttclasses/lib:/tt/home/timesten/instances/instance1/install/lib:/tt/home/timesten/instances/instance1/install/ttoracle_home/instantclient_11_2:/tt/home/timesten/instances/instance1/install/ttoracle_home/instantclient
      TT_REPLICATION_TOPOLOGY:  activeStandbyPair
    Mounts:
      /tt from tt-persistent (rw)
      /ttagent from tt-agent (rw)
      /ttconfig from tt-config (rw)
   daemonlog:
    Image:      container-registry.oracle.com/timesten/timesten:22.1.1.9.0
    Port:       <none>
    Host Port:  <none>
    Command:
      sh
      -c
      /bin/bash <<'EOF'
      while [ 1 ] ; do tail --follow=name /tt/home/timesten/instances/instance1/diag/ttmesg.log --max-unchanged-stats=5; sleep 1; done
      exit 0
      EOF
    Requests:
      cpu:     100m
      memory:  20Mi
    Environment:
      TIMESTEN_HOME:        /tt/home/timesten/instances/instance1
      TT_OPERATOR_MANAGED:  1
      LD_LIBRARY_PATH:      /tt/home/timesten/instances/instance1/ttclasses/lib:/tt/home/timesten/instances/instance1/install/lib:/tt/home/timesten/instances/instance1/install/ttoracle_home/instantclient_11_2:/tt/home/timesten/instances/instance1/install/ttoracle_home/instantclient
    Mounts:
      /tt from tt-persistent (rw)
  Volumes:
   tt-agent:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  tte6d49f75-05ed-4b4b-9412-4a577ad19bbe
    Optional:    false
   tt-config:
    Type:               Projected (a volume that contains injected data from multiple sources)
    ConfigMapName:      sample2
    ConfigMapOptional:  <nil>
Volume Claims:
  Name:          tt-persistent
  StorageClass:  oci-bv
  Labels:        <none>
  Annotations:   <none>
  Capacity:      50G
  Access Modes:  [ReadWriteOnce]
Events:          <none>

You successfully modified the sample2 TimesTenClassic object to use the new TimesTen container image. Let's continue the manual upgrade by upgrading the standby database.

Upgrade the Standby Database

Perform these steps to upgrade the standby database.

Note:

Even though you are upgrading the standby database, depending on your replication configuration, this may result in disruption on your active database. This may impact your applications. Perform the upgrade at the appropriate time.

  1. Use the kubectl get ttc command to:

    • Determine which Pod is the standby. The active Pod is the Pod represented in the ACTIVE column. The standby Pod is the other Pod (not represented in the ACTIVE column). Therefore, for the sample2 TimesTenClassic object, the active Pod is sample2-0 and the standby Pod is sample2-1.

    • Ensure the state for the TimesTenClassic object (sample2, in this example) is Normal. 

    kubectl get ttc sample2

    The output is the following.

    NAME      STATE    ACTIVE      AGE
sample2   Normal   sample2-0   19h
  2. To upgrade the standby to the new TimesTen image, delete the standby Pod (sample2-1, in this example).
    kubectl delete pod sample2-1

    The output is the following.

    pod "sample2-1" deleted

    Kubernetes automatically creates a new sample2-1 Pod to replace the deleted Pod. The Operator configures the new sample2-1 Pod as the standby Pod. This new Pod will now run the newly created TimesTen image.

  3. Verify the standby is up and running and the state is Normal.
    kubectl get ttc sample2

    Note that the state is initially StandbyDown. 

    NAME      STATE         ACTIVE      AGE
sample2   StandbyDown   sample2-0   19h

    Wait a few minutes, then run the command again.

    kubectl get ttc sample2

    Note that the state has changed to Normal. 

    NAME      STATE    ACTIVE      AGE
sample2   Normal   sample2-0   19h
  4. Verify that the standby is up and running again and that the active standby pair health is Normal. During the upgrade of the standby, your applications are not disrupted. Your applications can continue to use the active database. 
    kubectl describe ttc sample2

    The output is similar to the following.

    Name:         sample2
Namespace:    mynamespace
...
Kind:         TimesTenClassic
...
Spec:
  ...
  Ttspec:
    Additional Memory Request:  1Gi
    Automatic Memory Requests:  true
    Daemon Log CPU Request:     200m
    Daemon Log Memory Request:  20Mi
    Db Config Map:
      sample2
    Exporter CPU Request:     200m
    Exporter Memory Request:  200Mi
    Image:                    container-registry.oracle.com/timesten/timesten:22.1.1.19.0
    Image Pull Policy:        Always
    Image Pull Secret:        sekret
    Image Upgrade Strategy:   Manual
    Memory Warning Percent:   90
    Storage Class Name:       oci-bv
    Storage Size:             250G
Status:
  ...
  Classic Upgrade Status:
   ...
  High Level State:              Normal
  ...
  Pod Status:
    Active:           true
    Admin User File:  true
    Cache Status:
      Cache Agent:        Not Running
      Cache UID Pwd Set:  true
      N Cache Groups:     0
    Cache User File:      false
    Cg File:              false
    Db Status:
      ...
    Name:                                           sample2-0
    Pod Status:
      Agent:                Up
      Last Time Reachable:  1673213068
      Pod IP:               192.0.2.3
      Pod Phase:            Running
    Prev Active:            true
    Prev High Level State:  Healthy
    Prev Image:             container-registry.oracle.com/timesten/timesten:22.1.1.9.0
    Prev Intended State:
    Prev Ready:             true
    Ready:                  true
    Replication Status:
      Last Time Rep State Changed:  0
      Rep Agent:                    Running
      Rep Peer P State:             start
      Rep Scheme:                   Exists
      Rep State:                    ACTIVE
    Scaleout Status:
      Instance Type:  classic
    Schema File:      true
    Timesten Status:
      Daemon:         Up
      Instance:       Exists
      Release:        22.1.1.9.0
    Tt Pod Type:      Database
    Using Twosafe:    false
    Active:           false
    Admin User File:  true
    Cache Status:
      Cache Agent:        Not Running
      Cache UID Pwd Set:  true
      N Cache Groups:     0
    Cache User File:      false
    Cg File:              false
    Db Status:
      ...
    Name:                                                 sample2-1
    Pod Status:
      Agent:                Up
      Last Time Reachable:  1673213068
      Pod IP:               192.0.2.4
      Pod Phase:            Running
    Prev Active:            false
    Prev High Level State:  Healthy
    Prev Image:             container-registry.oracle.com/timesten/timesten:22.1.1.9.0
    Prev Intended State:
    Prev Ready:             true
    Ready:                  true
    Replication Status:
      Last Time Rep State Changed:  1673212993
      Rep Agent:                    Running
      Rep Peer P State:             start
      Rep Scheme:                   Exists
      Rep State:                    STANDBY
    Scaleout Status:
      Instance Type:  classic
    Schema File:      true
    Timesten Status:
      Daemon:             Up
      Instance:           Exists
      Release:            22.1.1.19.0
    Tt Pod Type:          Database
    Using Twosafe:        false
  Prev High Level State:  StandbyCatchup
  Prev Reexamine:
  Prev Stop Managing:
  Rep Create Statement:   create active standby pair "sample2" on "sample2-0.sample2.mynamespace.svc.cluster.local", "sample2" on "sample2-1.sample2.mynamesoace.svc.cluster.local" NO RETURN store "sample2" on "sample2-0.sample2.mynamespace.svc.cluster.local" PORT 4444 FAILTHRESHOLD 0 store "sample2" on "sample2-1.sample2.mynamespace.svc.cluster.local" PORT 4444 FAILTHRESHOLD 0
  Rep Port:               4444
  Rep Start Fail Count:   0
  Standby Cache Agent:    Not Running
  Standby Down Standby AS:
    Async Id:           c3cf41af-61c2-438c-a4d0-f3dfbdb6043d
    Destroy Db:         true
    Id:                 8a9a22ee-deaa-4eb6-8e00-c0d8932af623
    Pod Name:           sample2-1
    Rep Duplicate:      true
    Start Rep Agent:    true
    Status:             complete
  Standby Perm In Use:  15291
  Standby Perm Size:    200
  Standby Rep Agent:    Running
  Status Version:       1.0
  Using Twosafe:        false
Events:
  Type     Reason       Age    From      Message
  ----     ------       ----   ----      -------
  Normal   Upgrade      4m15s  timesten  Image updated, automatic upgrade disabled
  Normal   Info         3m24s  timesten  Pod sample2-1 Agent Down
  Normal   StateChange  3m24s  timesten  Pod sample2-1 is Not Ready
  Warning  StateChange  3m19s  timesten  TimesTenClassic was Normal, now ActiveTakeover
  Normal   StateChange  3m14s  timesten  TimesTenClassic was ActiveTakeover, now StandbyDown
  Normal   Info         2m19s  timesten  Pod sample2-1 Agent Up
  Normal   Info         2m19s  timesten  Pod sample2-1 Instance Exists
  Normal   Info         2m19s  timesten  Pod sample2-1 Daemon Down
  Normal   Info         2m19s  timesten  Pod sample2-1 Daemon Up
  Normal   Info         2m19s  timesten  Pod sample2-1 Database Unloaded
  Normal   Info         2m12s  timesten  Pod sample2-1 Database None
  Normal   Info         116s   timesten  Pod sample2-1 Database Loaded
  Normal   Info         116s   timesten  Pod sample2-1 RepAgent Not Running
  Normal   Info         116s   timesten  Pod sample2-1 RepScheme Exists
  Normal   StateChange  116s   timesten  Pod sample2-1 RepState IDLE
  Normal   Info         111s   timesten  Pod sample2-1 RepAgent Running
  Normal   StateChange  111s   timesten  TimesTenClassic was StandbyDown, now StandbyStarting
  Normal   StateChange  110s   timesten  TimesTenClassic was StandbyStarting, now StandbyCatchup
  Normal   StateChange  75s    timesten  Pod sample2-1 RepState STANDBY
  Normal   StateChange  75s    timesten  TimesTenClassic was StandbyCatchup, now Normal
  Normal   StateChange  70s    timesten  Pod sample2-1 is Ready

    Note the following:

    • The image is upgraded to the new release.

    • The active database (sample2-0) is not upgraded to the new release.

    • The standby database (sample2-1) is upgraded to the new release.

You have successfully upgraded the standby database. You are now ready to fail over from the active database to the standby.

Fail Over

You must now fail over from the active database to the standby.

Note:

When you fail over, your active database will be taken down, and failed over immediately. Do not perform this procedure at the busiest time of your production day.

Before failing over, quiesce your applications on the active database. (You can also use the ttAdmin -close and the ttAdmin -disconnect commands. See Opening and Closing the Database for User Connections and Disconnecting from a Database in the Oracle TimesTen In-Memory Database Operations Guide.

To avoid potential data loss, use the ttRepAdmin -wait command to wait until replication is caught up, such that all transactions that were executed on the active database have been replicated to the standby database. See ttRepAdmin in the Oracle TimesTen In-Memory Database Reference.

Once the standby is caught up, fail over from the active database to the standby by deleting the active Pod. When you delete the active Pod, the Operator automatically detects the failure and promotes the standby database to be the active. Client/server applications that are using the active database (sample2-0, in this example) are automatically reconnected to the new active database (sample2-1, in this example). Transactions in flight are rolled back. Prepared SQL statements will need to be re-prepared by the applications. See About Handling Failover and Recovery for more information about client/server failover.

Kubernetes automatically creates a new sample2-0 Pod to replace the deleted Pod. The Operator will configure the new Pod as the standby Pod. This new Pod will run the newly created TimesTen image.

Note:

You may want to perform this operation during a scheduled production outage.

  1. Use the kubectl delete command to delete the active Pod (sample2-0, in this example).
    kubectl delete pod sample2-0

    The output is the following.

    pod "sample2-0" deleted
  2. Use the kubectl get command to assess the state of the sample2 TimesTenClassic object.
    kubectl get ttc sample2

    Note that the state is initially ActiveDown. 

    NAME      STATE        ACTIVE   AGE
sample2   ActiveDown   None     19h

    Wait a few minutes, then run the command again.

    kubectl get ttc sample2

    Note that the state has changed to Normal. 

    NAME     STATE    ACTIVE     AGE
sample2   Normal   sample2-1   19h

    The Operator promoted sample-1 to be the active.

  3. Use the kubectl describe command to observe how the Operator recovers from the failure. The Operator promotes the standby database (sample2-1) to be active. Any applications that were connected to the sample2-0 database are automatically reconnected to the sample2-1 database by TimesTen. After a brief outage, the applications can continue to use the database. See About Monitoring the Health of an Active Standby Pair of Databases for information on the health and states of the active standby pair.
    kubectl describe ttc sample2

    The output is similar to the following.

    Name:         sample2
Namespace:    mynamespace
...
Kind:         TimesTenClassic
...
Spec:
  ...
  Ttspec:
    Additional Memory Request:  1Gi
    Automatic Memory Requests:  true
    Daemon Log CPU Request:     200m
    Daemon Log Memory Request:  20Mi
    Db Config Map:
      sample2
    Exporter CPU Request:     200m
    Exporter Memory Request:  200Mi
    Image:                    container-registry.oracle.com/timesten/timesten:22.1.1.19.0
    Image Pull Policy:        Always
    Image Pull Secret:        sekret
    Image Upgrade Strategy:   Manual
    Memory Warning Percent:   90
    Storage Class Name:       oci-bv
    Storage Size:             250G
Status:
  ...
  Classic Upgrade Status:
    ...
  High Level State:              Normal
  Last Event:                    0
  Last High Level State Switch:  1673214034
  Last Reconcile:                1673214897250
  Last Reconciling Operator:     timesten-operator-6f9d96bdfc-h22lm
  Observed Generation:           2
  Pod Status:
    Active:           false
    Admin User File:  true
    Cache Status:
      Cache Agent:        Not Running
      Cache UID Pwd Set:  true
      N Cache Groups:     0
    Cache User File:      false
    Cg File:              false
    Db Status:
      Db:  Loaded
      ...
    Name:                                                 sample2-0
    Pod Status:
      Agent:                Up
      Last Time Reachable:  1673214897
      Pod IP:               192.0.2.3
      Pod Phase:            Running
    Prev Active:            false
    Prev High Level State:  Healthy
    Prev Image:             container-registry.oracle.com/timesten/timesten:22.1.1.9.0
    Prev Intended State:    Active
    Prev Ready:             true
    Ready:                  true
    Replication Status:
      Last Time Rep State Changed:  1673213965
      Rep Agent:                    Running
      Rep Peer P State:             start
      Rep Scheme:                   Exists
      Rep State:                    STANDBY
    Scaleout Status:
      Instance Type:  classic
    Schema File:      true
    Timesten Status:
      Daemon:         Up
      Instance:       Exists
      Release:        22.1.1.19.0
    Tt Pod Type:      Database
    Using Twosafe:    false
    Active:           true
    Admin User File:  true
    Cache Status:
      Cache Agent:        Not Running
      Cache UID Pwd Set:  true
      N Cache Groups:     0
    Cache User File:      false
    Cg File:              false
    Db Status:
      Db:  Loaded
      ...
    High Level State:                                     Healthy
    Initialized:                                          true
    Intended State:                                       Active
    Last High Level State Switch:                         1673214897
    Local Commit:                                         false
    Name:                                                 sample2-1
    Pod Status:
      Agent:                Up
      Last Time Reachable:  1673214897
      Pod IP:               192.0.2.4
      Pod Phase:            Running
    Prev Active:            true
    Prev High Level State:  Healthy
    Prev Image:             container-registry.oracle.com/timesten/timesten:22.1.1.9.0
    Prev Intended State:    Standby
    Prev Ready:             true
    Ready:                  true
    Replication Status:
      Last Time Rep State Changed:  1673212993
      Rep Agent:                    Running
      Rep Peer P State:             start
      Rep Scheme:                   Exists
      Rep State:                    ACTIVE
    Scaleout Status:
      Instance Type:  classic
    Schema File:      true
    Timesten Status:
      Daemon:             Up
      Instance:           Exists
      Release:            22.1.1.19.0
    Tt Pod Type:          Database
    Using Twosafe:        false
  Prev High Level State:  StandbyDown
  Prev Reexamine:
  Prev Stop Managing:
  ...
Events:
  Type     Reason       Age   From      Message
  ----     ------       ----  ----      -------
  ...
  Warning  StateChange  15m   timesten  TimesTenClassic was Normal, now ActiveDown
  Normal   Info         15m   timesten  Pod sample2-0 Agent Down
  Normal   StateChange  15m   timesten  Pod sample2-0 is Not Ready
  Normal   StateChange  15m   timesten  Pod sample2-0 is Not Active Ready
  Normal   Info         15m   timesten  Pod sample2-1 Database Updatable
  Normal   StateChange  15m   timesten  Pod sample2-1 RepState ACTIVE
  Normal   StateChange  15m   timesten  Pod sample2-1 is Not Ready
  Normal   StateChange  15m   timesten  TimesTenClassic was ActiveDown, now ActiveTakeover
  Normal   StateChange  15m   timesten  Pod sample2-1 is Ready
  Normal   StateChange  15m   timesten  TimesTenClassic was ActiveTakeover, now StandbyDown
  Normal   StateChange  15m   timesten  Pod sample2-1 is Active Ready
  Normal   Info         14m   timesten  Pod sample2-0 Agent Up
  Normal   Info         14m   timesten  Pod sample2-0 Instance Exists
  Normal   Info         14m   timesten  Pod sample2-0 Daemon Down
  Normal   Info         14m   timesten  Pod sample2-0 Daemon Up
  Normal   Info         14m   timesten  Pod sample2-0 Database Unloaded
  Normal   Info         14m   timesten  Pod sample2-0 Database None
  Normal   Info         14m   timesten  Pod sample2-0 Database Loaded
  Normal   Info         14m   timesten  Pod sample2-0 RepAgent Not Running
  Normal   Info         14m   timesten  Pod sample2-0 RepScheme Exists
  Normal   StateChange  14m   timesten  Pod sample2-0 RepState IDLE
  Normal   Info         14m   timesten  Pod sample2-0 RepAgent Running
  Normal   StateChange  14m   timesten  Pod sample2-0 RepState STANDBY
  Normal   StateChange  14m   timesten  Pod sample2-0 is Ready
  Normal   StateChange  14m   timesten  TimesTenClassic was StandbyDown, now Normal

You successfully upgraded to a new release of TimesTen. The active and the standby Pods are running the new TimesTen image, which contains the new TimesTen release.

In this example, the sample and the sample2 TimesTenClassic objects are now upgraded. The upgrade process is complete. Let's verify the active and the standby databases are running the new release of TimesTen.