About Bringing Up One Database

This section assumes you have manually repaired or have manually performed maintenance on one of the databases associated with the TimesTenClassic object. The TimesTenClassic object is currently in the ManualInterventionRequired state. You now want to direct the Operator to treat the repaired database as the active, to perform the necessary steps to duplicate this database to the standby, and to bring up both databases, such that both are running and operating successfully.

Recall that all of these conditions must be met for the database:

• TimesTen agent in the container is running.

• TimesTen daemon (the instance) in the container is running.

• TimesTen database is loaded.

• There is no replication scheme in the database.

• The replication agent is not running.

• The replication state is IDLE.

These sections show you how to verify the conditions are met for the database and how to set the reexamine value:

• Verify Conditions Are Met for the Database

• Set the reexamine Value

Verify Conditions Are Met for the Database

Perform these steps to ensure the conditions are met for the database (the database to be the active). In this example, sample-1 will be the new active.

Note: These steps require you to use TimesTen utilities and TimesTen built-in procedures. See Utilities and Built-In Procedures in the Oracle TimesTen In-Memory Database Reference for details.

1. Confirm the TimesTenClassic object (sample, in this example) is in the ManualInterventionRequired state (represented in bold).

   % kubectl get ttc sample
   NAME     STATE                        ACTIVE     AGE
   sample   ManualInterventionRequired   sample-0   12h
   

2. Use the kubectl exec -it command to invoke the shell within the sample-1 Pod that contains the TimesTen database. (This database will be the new active.)

   The remaining procedures take place within this shell.

   % kubectl exec -it sample-1  -c tt -- /bin/bash

3. Use the ttDaemonAdmin utility to start TimesTen daemon (if not already started). Then use the ttAdmin utility to load the TimesTen database into memory (if not already loaded).

   % ttDaemonAdmin -start
   TimesTen Daemon (PID: 5948, port: 6624) startup OK.
   % ttAdmin -ramLoad sample
   RAM Residence Policy            : manual
   Manually Loaded In RAM          : True
   Replication Agent Policy        : manual
   Replication Manually Started    : False
   Cache Agent Policy              : manual
   Cache Agent Manually Started    : False
   Database State                  : Open
   

4. Use the ttIsql utility to connect to the sample database. Then, call the ttRepStop built-in procedure to stop the replication agent.

   % ttIsql sample
    
   Copyright (c) 1996, 2023, Oracle and/or its affiliates. All rights reserved.
   Type ? or "help" for help, type "exit" to quit ttIsql.
    
    
    
   connect "DSN=sample";
   Connection successful: DSN=sample;UID=timesten;DataStore=/tt/home/timesten/datastore/sample;
   DatabaseCharacterSet=AL32UTF8;ConnectionCharacterSet=AL32UTF8;AutoCreate=0;
   PermSize=200;DDLReplicationLevel=3;ForceDisconnectEnabled=1;
   (Default setting AutoCommit=1)
   Command> call ttRepStop;
   

5. From within ttIsql, use the SQL DROP ACTIVE STANDBY PAIR statement to drop the active standby pair replication scheme. Then use the ttIsql repschemes command to verify there are no replication schemes in the database. Exit from ttIsql.

   Command> DROP ACTIVE STANDBY PAIR;
   Command> repschemes;
    
   0 replication schemes found.
   

6. Use the ttStatus utility to verify the TimesTen daemon is running and the replication agent is not running.

   % ttStatus
   TimesTen status report as of Sat Apr 24 02:14:15 2023
    
   Daemon pid 5948 port 6624 instance instance1
   TimesTen server pid 5955 started on port 6625
   ------------------------------------------------------------------------
   ------------------------------------------------------------------------
   Data store /tt/home/timesten/datastore/sample
   Daemon pid 5948 port 6624 instance instance1
   TimesTen server pid 5955 started on port 6625
   There are 15 connections to the data store
   Shared Memory KEY 0x0a100c60 ID 196609
   PL/SQL Memory Key 0x0b100c60 ID 229378 Address 0x5000000000
   Type            PID     Context             Connection Name              ConnID
   Process         10418   0x000000000218a6e0  sample                            2
   Process         8338    0x0000000001cbb6e0  sample                            1
   Subdaemon       5953    0x00000000015075f0  Manager                        2047
   Subdaemon       5953    0x0000000001588540  Rollback                       2046
   Subdaemon       5953    0x0000000001607210  Checkpoint                     2041
   Subdaemon       5953    0x00007f132c0008c0  Flusher                        2045
   Subdaemon       5953    0x00007f132c080370  Log Marker                     2040
   Subdaemon       5953    0x00007f13340008c0  Monitor                        2044
   Subdaemon       5953    0x00007f133407f330  HistGC                         2037
   Subdaemon       5953    0x00007f13380008c0  Aging                          2042
   Subdaemon       5953    0x00007f133807f330  AsyncMV                        2039
   Subdaemon       5953    0x00007f133c0008c0  Deadlock Detector              2043
   Subdaemon       5953    0x00007f133c07f330  IndexGC                        2038
   Subdaemon       5953    0x00007f135c0008c0  Garbage Collector              2035
   Subdaemon       5953    0x00007f13600e8e20  XactId Rollback                2036
   Open for user connections
   RAM residence policy: Manual
   Data store is manually loaded into RAM
   Replication policy  : Manual
   Cache Agent policy  : Manual
   PL/SQL enabled.
   ------------------------------------------------------------------------
   Accessible by group timesten
   End of report
   

You have successfully verified the conditions for the database. The database is up and running. The Operator will treat this database as the active. You are now ready to set the value for the .spec.ttspec.reexamine datum.

Set the reexamine Value

This example shows you how to set the reexamine value in the TimesTenClassic object definition (sample, in this example). The example also illustrates the action the Operator takes after the reexamine value has been changed.

1. Set the reexamine value. The value must be different than the current value for the TimesTenClassic object. When the Operator examines this value and notices it has changed since the last iteration, it will take appropriate action.

   Use the kubectl edit command to edit the TimesTenClassic object.

   • If there is a line for reexamine in the file, then modify its value. It must be different than the current value.

   • If there is no line for reexamine in the file, then add a line and specify a value.

   In this example, there is no reexamine line. This example adds the reexamine line and sets the value for reexamine to April22reexamine1 (represented in bold).

   Note: Not all output is shown.

   % kubectl edit timestenclassic sample
   # Please edit the object below. Lines beginning with a '#' will be ignored,
   # and an empty file will abort the edit. If an error occurs while saving this 
   # file will be reopened with the relevant failures.
   #
   apiVersion: timesten.oracle.com/v1
   kind: TimesTenClassic
   metadata:
   ...
     name: sample
     namespace: mynamespace
   ...
   repCreateStatement: |
     create active standby pair
       "{{tt-name}}" on "{{tt-node-0}}",
       "{{tt-name}}" on "{{tt-node-1}}"
     RETURN TWOSAFE
     store "{{tt-name}}" on "{{tt-node-0}}"
       PORT {{tt-rep-port}} FAILTHRESHOLD 0 TIMEOUT 999
     store "{{tt-name}}" on "{{tt-node-1}}"
       PORT {{tt-rep-port}} FAILTHRESHOLD 0 TIMEOUT 999
   spec:
     ttspec:
       bothDownBehavior: Best
       dbConfigMap:
       - sample
       image: container-registry.oracle.com/timesten/timesten:22.1.1.19.0
       imagePullSecret: sekret
       storageClassName: oci-bv
       storageSize: 250Gi
       reexamine: April22reexamine1
   ...
   timestenclassic.timesten.oracle.com/sample edited
   

2. Use the kubectl get command to assess the state of the sample TimesTenClassic object. Observe how the state changes as you issue multiple kubectl get commands. Also note that the Operator has successfully configured sample-1 to be the active.

   % kubectl get ttc sample 
   NAME     STATE       ACTIVE   AGE 
   sample   Reexamine   None     68m 
   % kubectl get ttc sample 
   NAME     STATE               ACTIVE   AGE 
   sample   ConfiguringActive   None     68m 
   % kubectl get ttc sample 
   NAME     STATE         ACTIVE     AGE 
   sample   StandbyDown   sample-1   68m 
   % kubectl get ttc sample 
   NAME     STATE    ACTIVE     AGE
   sample   Normal   sample-1   71m
   

3. Use the kubectl describe command to further review the actions of the Operator (represented in bold).

   Not all output is shown:

   % kubectl describe ttc sample
   Name:         sample
   Namespace:    mynamespace
   ...
   Kind:         TimesTenClassic
   ...
   Rep Create Statement:  create active standby pair
     "{{tt-name}}" on "{{tt-node-0}}",
     "{{tt-name}}" on "{{tt-node-1}}"
   RETURN TWOSAFE
   store "{{tt-name}}" on "{{tt-node-0}}"
     PORT {{tt-rep-port}} FAILTHRESHOLD 0 TIMEOUT 999
   store "{{tt-name}}" on "{{tt-node-1}}"
     PORT {{tt-rep-port}} FAILTHRESHOLD 0 TIMEOUT 999
    
   Spec:
     Ttspec:
       Both Down Behavior:  Best
       Db Config Map:
         sample
       Image:               container-registry.oracle.com/timesten/timesten:22.1.1.19.0
       Image Pull Policy:   Always
       Image Pull Secret:   sekret
       Reexamine:           April22reexamine1
       Stop Managing:       April21Stop1
       Storage Class Name:  oci-bv
       Storage Size:        250Gi
   Status:
     Classic Upgrade Status:
       Active Start Time:           0
       Active Status:
       Image Update Pending:        false
       Last Upgrade State Switch:   0
       Prev Reset Upgrade State:
       Prev Upgrade State:
       Standby Start Time:          0
       Standby Status:
       Upgrade Start Time:          0
       Upgrade State:
     Active Pods:                   sample-1
     High Level State:              Normal
     Last Event:                    54
     Last High Level State Switch:  1619230912
     Pod Status:
       Cache Status:
         Cache Agent:        Not Running
         Cache UID Pwd Set:  true
         N Cache Groups:     0
       Db Status:
         Db:                          Loaded
         Db Id:                       475
         Db Updatable:                No
       Initialized:                   true
       Last High Level State Switch:  ?
       Pod Status:
         Agent:                Up
         Last Time Reachable:  1619231126
         Pod IP:               10.244.7.89
         Pod Phase:            Running
       Prev High Level State:  Healthy
       Prev Image:
       Replication Status:
         Last Time Rep State Changed:  0
         Rep Agent:                    Running
         Rep Peer P State:             start
         Rep Scheme:                   Exists
         Rep State:                    STANDBY
       Times Ten Status:
         Daemon:          Up
         Instance:        Exists
         Release:         22.1.1.19.0
       Admin User File:   false
       Cache User File:   false
       Cg File:           false
       Disable Return:    false
       High Level State:  Healthy
       Intended State:    Standby
       Local Commit:      false
       Name:              sample-0
       Schema File:       false
       Using Twosafe:     false
       Cache Status:
         Cache Agent:        Not Running
         Cache UID Pwd Set:  true
         N Cache Groups:     0
       Db Status:
         Db:                          Loaded
         Db Id:                       476
         Db Updatable:                Yes
       Initialized:                   true
       Last High Level State Switch:  ?
       Pod Status:
         Agent:                Up
         Last Time Reachable:  1619231126
         Pod IP:               10.244.6.149
         Pod Phase:            Running
       Prev High Level State:  Healthy
       Prev Image:
       Replication Status:
         Last Time Rep State Changed:  1619228670
         Rep Agent:                    Running
         Rep Peer P State:             start
         Rep Scheme:                   Exists
         Rep State:                    ACTIVE
       Times Ten Status:
         Daemon:             Up
         Instance:           Exists
         Release:            22.1.1.19.0
       Admin User File:      false
       Cache User File:      false
       Cg File:              false
       Disable Return:       false
       High Level State:     Healthy
       Intended State:       Active
       Local Commit:         false
       Name:                 sample-1
       Schema File:          false
       Using Twosafe:        false
     Prev High Level State:  StandbyDown
     Prev Reexamine:         April22reexamine1
     Prev Stop Managing:     April21Stop1
     Rep Create Statement:   create active standby pair "sample" on 
   "sample-0.sample.mynamespace.svc.cluster.local", "sample" on 
   "sample-1.sample.mynamespace.svc.cluster.local" NO RETURN store "sample" on 
   "sample-0.sample.mynamespace.svc.cluster.local" PORT 4444 FAILTHRESHOLD 0 store 
   "sample" on "sample-1.sample.mynamespace.svc.cluster.local" PORT 4444 FAILTHRESHOLD 0
     Rep Port:               4444
     Status Version:         1.0
   Events:
     Type  Reason       Age    From       Message
     ----  ------       ----   ----       -------
     -     StateChange  58m    ttclassic  TimesTenClassic was Normal, now ManualInterventionRequired
     -     StateChange  46m    ttclassic  Pod sample-0 Daemon Down
     -     StateChange  41m    ttclassic  Pod sample-1 Daemon Down
     -     StateChange  41m    ttclassic  Pod sample-1 Daemon Up
     -     StateChange  41m    ttclassic  Pod sample-1 Database Unloaded
     -     StateChange  40m    ttclassic  Pod sample-1 Database Loaded
     -     StateChange  40m    ttclassic  Pod sample-1 RepState IDLE
     -     StateChange  40m    ttclassic  Pod sample-1 RepAgent Not Running
     -     StateChange  17m    ttclassic  Pod sample-1 Database Updatable
     -     StateChange  17m    ttclassic  Pod sample-1 RepScheme None
     -     StateChange  4m21s  ttclassic  TimesTenClassic was ManualInterventionRequired, now Reexamine
     -     Error        4m16s  ttclassic  Active error: Daemon Down
     -     StateChange  4m16s  ttclassic  TimesTenClassic was Reexamine, now ConfiguringActive
     -     StateChange  4m10s  ttclassic  Pod sample-1 RepState ACTIVE
     -     StateChange  4m10s  ttclassic  Pod sample-1 RepScheme Exists
     -     StateChange  4m10s  ttclassic  Pod sample-1 RepAgent Running
     -     StateChange  4m8s   ttclassic  TimesTenClassic was ConfiguringActive, now StandbyDown
     -     StateChange  4m3s   ttclassic  Pod sample-0 Daemon Up
     -     StateChange  4m3s   ttclassic  Pod sample-0 Database Unloaded
     -     StateChange  3m56s  ttclassic  Pod sample-0 Database None
     -     StateChange  3m42s  ttclassic  Pod sample-0 Database Loaded
     -     StateChange  3m42s  ttclassic  Pod sample-0 Database Not Updatable
     -     StateChange  3m42s  ttclassic  Pod sample-0 RepAgent Not Running
     -     StateChange  3m42s  ttclassic  Pod sample-0 RepState IDLE
     -     StateChange  3m36s  ttclassic  Pod sample-0 RepAgent Running
     -     StateChange  3m36s  ttclassic  Pod sample-0 RepState STANDBY
     -     StateChange  3m36s  ttclassic  TimesTenClassic was StandbyDown, now Normal
   

4. Use the kubectl exec -it command to invoke the shell within the sample-1 Pod that contains the TimesTen database. Then, verify you can connect to the active database.

   % kubectl exec -it sample-1 -c tt -- /bin/bash
   $ ttIsql sample
    
   Copyright (c) 1996, 2023, Oracle and/or its affiliates. All rights reserved.
   Type ? or "help" for help, type "exit" to quit ttIsql.
    
    
    
   connect "DSN=sample";
   Connection successful: DSN=sample;UID=timesten;DataStore=/tt/home/timesten/datastore/sample;
   DatabaseCharacterSet=AL32UTF8;ConnectionCharacterSet=AL32UTF8;AutoCreate=0;PermSize=200;DDLReplicationLevel=3;ForceDisconnectEnabled=1;
   (Default setting AutoCommit=1)
   Command> call ttRepStateGet;
   < ACTIVE >
   1 row found.
   

5. Use the kubectl exec -it command to invoke the shell within the sample-0 Pod that contains the TimesTen database. Then, verify you can connect to the standby database.

   % kubectl exec -it sample-0 -c tt -- /bin/bash
   % ttIsql sample
    
   Copyright (c) 1996, 2023, Oracle and/or its affiliates. All rights reserved.
   Type ? or "help" for help, type "exit" to quit ttIsql.
    
    
    
   connect "DSN=sample";
   Connection successful: DSN=sample;UID=timesten;DataStore=/tt/home/timesten/datastore/sample;
   DatabaseCharacterSet=AL32UTF8;ConnectionCharacterSet=AL32UTF8;AutoCreate=0;
   PermSize=200;DDLReplicationLevel=3;ForceDisconnectEnabled=1;
   (Default setting AutoCommit=1)
   Command> call ttRepStateGet;
   < STANDBY >
   1 row found.
   

The Operator is now managing and monitoring your TimesTenClassic object. The TimesTenClassic object is in the Normal state. Both databases are up and running and ready for use.