7 Cache Performance

Enable the cache connection pool

You can specify that TimesTen creates a cache connection pool on the TimesTen server when it starts up. All TimesTen servers share parameters that are stored on the Oracle database.

A dynamic load request from a client/server connection acquires a connection from the cache connection pool, performs the dynamic load, then returns the connection to the cache connection pool after the dynamic load request completes. The cache connection pool is destroyed when the TimesTen server shuts down.

To enable client/server connection requests to use the cache connection pool, your application must specify the following connection attributes when connecting.

• MaxConnsPerServer connection attribute: Value must be set to > 1 to use the cache connection pool.

  A TimesTen server can create and assign work to multiple TimesTen child server processes (in multithreaded mode) when this connection attribute is greater than 1. This connection attribute sets the maximum number of client connections that can be created in a connection pool for each child server process.

• ServersPerDSN connection attribute: Value designates the number of child server processes to spawn for the TimesTen server. Default is 1.

  Each new incoming connection spawns a new child server process up to the value specified by the ServersPerDSN connection attribute. When the maximum number of child server processes is reached, the existing child server processes handle multiple connections (up to the number specified in MaxConnsPerServer) in a round-robin method. That is, if you specify ServersPerDSN = 2 and MaxConnsPerServer = 3, then the first two connections would spawn two child server processes. The third through the sixth connections would be handled by these child server processes, where each child server process would service every other connection.

  Once all of the child server processes have the maximum allowed number of connections, the next incoming connection starts a new set of child server processes.

  The ServersPerDSN and MaxConnsPerServer connection attributes are used to designate how to distribute connections across multiple child server processes.

• UseCacheConnPool connection attribute: Set the value = 2 to enable the use of the cache connection pool. See "UseCacheConnPool" in the Oracle TimesTen In-Memory Database Reference for more details.

The following describes the expected behavior when a dynamic load is initiated from different types of connections:

• Direct connection: The cache connection pool is not used. The dynamic load performs using the existing behavior.

• Single threaded client/server connection (when MaxConnsPerServer=1): The cache connection pool is not used. The dynamic load performs using the existing behavior.

• Multithreaded client/server connection (when MaxConnsPerServer>1): Uses the cache connection pool for dynamic load if it is created. Otherwise, an error is returned.

You can set connection attributes for the connection either within a DSN definition or on a connect string.

[cache1]
DataStore=/users/OracleCache/database1
PermSize=64
OracleNetServiceName=oracledb
DatabaseCharacterSet=AL32UTF8
UseCacheConnPool=2
ServersPerDSN=2
MaxConnsPerServer=3

ttIsql "DSN=cache1; OracleNetServiceName=oracledb; UseCacheConnPool=2; ServersPerDSN=2; MaxConnsPerServer=3"

Size the cache connection pool

You can appropriately size the cache connection pool to avoid contention for connections with the ttCacheConnPoolSet built-in procedure. The ttCacheConnPoolSet built-in procedure saves the values of these parameters on the Oracle database, which are then used as the default values when restarting the TimesTen server.

Once applied to each TimesTen server, the values specified are used for the cache connection pool across all client/server applications for a TimesTen database.

You can execute the ttCacheConnPoolSet built-in procedure from a direct connection, a single-threaded client/server connection or a multithreaded client/server connection.

For example, the following initiates the minimum and maximum number of pooled connections to be between 10 and 32 connections and the increment is 1. The maximum idle time by the client is set to 10 seconds. And all dynamic load operations will wait for an available connection from the cache connection pool.

Command> call ttCacheConnPoolSet(10, 32, 1, 10, 0);

Set the minimum and maximum size of the cache connection pool to levels where connections are available when needed. If no connections are available in the pool, then TimesTen performs the following depending on the setting for ConnNoWait parameter in the ttCacheConnPoolSet built-in procedure:

• ConnNoWait=0: TimesTen stalls until a connection from the pool is available or until a timeout occurs. If the Oracle database is down, applications wait until the Oracle database comes back up, or a timeout occurs.

  If a connection to the Oracle database times out, you receive an error denoting a loss of the connection, sometimes requiring a rollback on TimesTen.

• ConnNoWait=1: Any dynamic load operations fail with an error if there are no available connections in the cache connection pool.

You can query what the cache connection pool parameters are with the ttCacheConnPoolGet built-in procedure.

See "Example demonstrating management of the cache connection pool" for a full example of how to use this built-in procedure.

Use the ChildServer connection attribute to identify a child server process

In a client/server environment, TimesTen can have multiple TimesTen child server processes to handle incoming requests from clients. You provide the ChildServer connection attribute to identify a specific child server process for certain cache connection pool built-in procedures. Each child server process is identified by a number assigned with the ChildServer=n connection attribute, where n is a number ranging from 1 to the number of running child server processes. Once connected to the child server process, you can execute either the ttCacheConnPoolGet('current') or ttCacheConnPoolApply built-in procedures that are meant for a specific child server process.

See "ttCacheConnPoolApply" and "ttCacheConnPoolGet" sections in the Oracle TimesTen In-Memory Database Reference for details on the built-in procedures that require this connection attribute. See "Example demonstrating management of the cache connection pool" for an example of how to use this connection attribute.

Apply cache connection pool sizing to currently executing database

Since the cache connection pool parameters are saved on the Oracle database, these parameters are used to initialize the cache connection pool for the TimesTen database every time that the TimesTen server restarts.

However, if you set the cache connection parameters while the database is already running, then you can dynamically resize the cache connection pool parameters on each child server process with the ttCacheConnPoolApply built-in procedure. After which, the cache connection pool parameters are associated with the child server process.

For example, the following connects to the child server process identified as 1 and applies the saved cache connection pool configuration to this child server process. It does the same process for child server process 2 (given that ServersPerDSN=2).

Command> connect "DSN=cache1;ChildServer=1;";
Command> call ttCacheConnPoolApply;
Command> disconnect;

Command> connect "DSN=cache1;ChildServer=2;";
Command> call ttCacheConnPoolApply;
Command> disconnect;

You can execute the ttCacheConnPoolApply built-in procedure only from a multithreaded client/server connection.

If the cache connection pool fails, you can recreate the pool by executing the ttCacheConnPoolApply built-in procedure from any child server process.

See "Example demonstrating management of the cache connection pool" for a full example of how to use this built-in procedure.

Example demonstrating management of the cache connection pool

Using the cache1 DSN as shown in Example 7-1 that enables the cache connection pool and assuming that you have set the cache administrator and password as described in "Set the cache administration user name and password", the following example sets new values for the cache connection pool and applies them to two separate child server processes.

/* Since ServerPerDSN is set to two and MaxConnsPerServer is set to 3, the first 
 and second connections spawn off both child server processes. And then you can
 create four more connections to reach the MaxConnsPerServer maximum, which are
 routed by the TimesTen server to the appropriate child server process (using a
 round robin method).*/
Command> connect "DSN=cache1;" as conn1;
Command> connect "DSN=cache1;" as conn2;
Command> connect "DSN=cache1;" as conn3;
Command> connect "DSN=cache1;" as conn4;
Command> connect "DSN=cache1;" as conn5;
Command> connect "DSN=cache1;" as conn6;
 
Command> use conn1;
 
/* Query the values for the cache connection pool that are saved on the Oracle database*/
Command> call ttCacheConnPoolGet('saved');
< 1, 10, 1, 10, 0, -1, -1, -1>
 
/* Change the configuration of the cache connection pool */
Command> call ttCacheConnPoolSet(1, 20, 1, 10, 0);
 
/* Query existing values for cache connection pool saved on the Oracle data base. 
 Since these are the saved values, this returns -1 for OpenCount, BusyCount
 and LastOraErr. */
Command> call ttCacheConnPoolGet('saved');
< 1, 20, 1, 10, 0, -1, -1, -1 >
 
/* Query existing values for the current cache connection pool on this TimesTen database */
Command> call ttCacheConnPoolGet('current');
< 1, 10, 1, 10, 0, 1, 0, 0 >
 
/* Connect to the child server process 1 using the ChildServer=1 connection
 attribute. Apply the saved values as the current values to the cache connection
 pool for child server process identified as ChildServer 1. */
Command> connect "DSN=cache1;ChildServer=1;";
Command> call ttCacheConnPoolApply;
Command> disconnect;
 
/* Connect to the child server process 1 using the ChildServer=1 connection
 attribute. Apply the saved values as the current values to the cache connection
 pool for child server process identified as ChildServer 2. */
Command> connect "DSN=cache1;ChildServer=2;";
Command> call ttCacheConnPoolApply;
Command> disconnect;
 
/* Query values for the cache connection pool in ChildServer 1 */
Command> use conn1;
Command> call ttCacheConnPoolGet('current');
< 1, 20, 1, 10, 0, 1, 0, 0 >
 
/* Query values for the cache connection pool in ChildServer 2 */
Command> use conn2;
Command> call ttCacheConnPoolGet('current');
< 1, 20, 1, 10, 0, 1, 0, 0 >

Limit the number of connections to the Oracle database

You can tune your performance while ensuring a limit to the number of connections to the Oracle database. Tuning the number of total connections depends on the following:

• N: The number of connections to the Oracle database.

• P: The limit on the number of connections for each cache connection pool, where each TimesTen child server process has a cache connection pool, which you can set with the MaxSize cache connection pool parameter in the ttCacheConnPoolSet built-in procedure.

• S: The number of child server processes that can be spawned for new connections. Currently, there is no direct way to limit the number of child server processes. Indirectly, you can limit the number of child server processes by setting the MaxSize parameter, MaxConnsPerServer connection attribute, and Connections connection attributes, as shown in the formula below.

• M: The maximum number of connections for each child server process, which you can set with the MaxConnsPerServer connection attribute.

• D: The maximum number of connections to a DSN, which is set with the Connections connection attribute.

The number of connections (N) to the Oracle database is equal to the number of TimesTen child server processes (S) times the number of connections for each cache connection pool (M).

N=S*P

While the maximum number of connections (D) to the DSN is equal to the maximum number of connections for each child server process (M) times the number of TimesTen child server processes (S).

D=M*S

With the above calculation, you can also state:

S=D/M

Since there is no hard limit that we can configure for the number of TimesTen child server processes, we merging the two equations together to eliminate S to get the following equation:

N=(D*P)/M

Thus, the number of connections to the Oracle database is set to the maximum number of connections to the DSN (set by the Connections connection attribute) times the number of connections for each cache connection pool (set by the MaxSize cache connection pool parameter), which is then divided by the maximum number of connections for each child server process (set by the MaxConnsPerServer connection attribute).

Restrictions for the cache connection pool

Restrictions when using the cache connection pool:

• You cannot use this in conjunction with the Oracle Database Resident Connection Pooling feature.

• This is only supported for multithreaded client/server connections, where the MaxConnsPerServer connection attribute must be greater than 1.

• This can only be used for dynamic load operations for dynamic read-only cache groups.

Requirements for setting DynamicLoadReduceContention

The DynamicLoadReduceContention database system parameter is supported on TimesTen Release 11.2.2.8.39 and following. In addition, the following are requirements when enabling the DynamicLoadReduceContention database system parameter:

• Required Oracle Database privileges

• Unsupported Oracle Database feature

• Required settings for active standby pair replication scheme

Using ttCacheAutorefreshXactLimit

For the month end processing, there can be a large number updates in a single transaction for the Oracle tables that are cached in autorefresh cache groups. In order to ensure that the large transaction does not fill up permanent memory, you can enable autorefresh to commit after every 256 (or any other user specified number) operations with the ttCacheAutorefreshXactLimit built-in procedure.

Turn on an autorefresh transaction limit for incremental autorefresh read-only cache groups before a large transaction with the ttCacheAutorefreshXactLimit built-in procedure where the value is set to ON or to a specific number of operations. Then, when autorefresh finishes updating the cached tables in TimesTen, turn off the autorefresh transaction limit for incremental autorefresh read-only cache groups with the ttCacheAutorefreshXactLimit built-in procedure.

The following example sets up the transaction limit to commit after every 256 operations for all incremental autorefresh read-only cache groups that are defined with an interval value of 10 seconds.

call ttCacheAutorefreshXactLimit('10000', 'ON');

After the month end process has completed and the incremental autorefresh read-only cache groups are refreshed, disable the transaction limit for incremental autorefresh read-only cache groups that are defined with the interval value of 10 seconds.

call ttCacheAutorefreshXactLimit('10000', 'OFF');

To enable the transaction limit for incremental autorefresh read-only cache groups to commit after every 1024 operations, provide 1024 as the value as follows:

call ttCacheAutorefreshXactLimit('10000', '1024');

Example of potential transactional inconsistency

The following example uses the employee and departments table, where the department id of the department table is a foreign key that points to the department id of the employee table.

The following example creates two incremental autorefresh read-only cache groups, where each is in its own cache group. The autorefresh transaction limit is enabled with ttCacheAutorefreshXactLimit before a large transaction and is disabled after it completes.

1. Before you initiate the large transaction, invoke ttCacheAutorefreshXactLimit to set the interval value and the number of operations after which to automatically commit. The following sets the number of operations to three (which is intentionally low to show a brief example) for all incremental autorefresh read-only cache groups with a two second interval.

   CALL ttCacheAutorefreshXactLimit('2000', '3');
   < 2000, 3 >
   1 row found.
   

2. Create the incremental autorefresh read-only cache groups with interval of two seconds. This example creates two static (non-dynamic) read-only cache groups, where each contains a single table.

   CREATE READONLY CACHE GROUP cgDepts AUTOREFRESH MODE INCREMENTAL 
    INTERVAL 2 SECONDS 
   FROM departments
       ( department_id    NUMBER(4) PRIMARY KEY
       , department_name  VARCHAR2(30) NOT NULL
       , manager_id       NUMBER(6)
       , location_id      NUMBER(4)
       );
    
   CREATE READONLY CACHE GROUP cgEmpls AUTOREFRESH MODE INCREMENTAL 
    INTERVAL 2 SECONDS 
   FROM employees
       ( employee_id    NUMBER(6) PRIMARY KEY
       , first_name     VARCHAR2(20)
       , last_name      VARCHAR2(25) NOT NULL
       , email          VARCHAR2(25) NOT NULL UNIQUE
       , phone_number   VARCHAR2(20)
       , hire_date      DATE NOT NULL
       , job_id         VARCHAR2(10) NOT NULL
       , salary         NUMBER(8,2)
       , commission_pct NUMBER(2,2)
       , manager_id     NUMBER(6)
       , department_id  NUMBER(4)
       );
   

3. Perform a manual LOAD CACHE GROUP for both autorefresh cache groups.

   LOAD CACHE GROUP cgDepts COMMIT EVERY 256 ROWS;
   27 cache instances affected.
    
   LOAD CACHE GROUP cgEmpls COMMIT EVERY 256 ROWS;
   107 cache instances affected.
   

You can have inconsistency within the table during an autorefresh as shown with the employees table.

1. On TimesTen, select the minimum and maximum salary of all employees.

   SELECT MIN(salary), MAX(salary) FROM employees;
   < 2100, 24000 >
   1 row found.
   

2. On the Oracle database, add 100,000 to everyone's salary.

   UPDATE employees SET salary = salary + 100000;
   107 rows updated.
   

3. On TimesTen, when you perform the SELECT again (while the autorefresh transactions are commmitted after every 3 records), it shows that while the maximum salary has updated, the minimum salary is still the old value.

   SELECT MIN(salary), MAX(salary) FROM employees;
   < 2100, 124000 >
   1 row found.
   

4. However, once the autorefresh completes, transactional consistency is maintained. For this example, once the autorefresh process completes, all salaries have increased by 100,000.

   SELECT MIN(salary), MAX(salary) FROM employees;
   < 102100, 124000 >
   1 row found.
   

5. The large transaction is complete, so disable the transaction limit for autorefresh cache groups with a 2 second interval.

   call ttCacheAutorefreshXactLimit('2000', 'OFF');
   

You can have transactional inconsistency between cache groups if you perform a SQL statement while the autorefresh process is progressing. The following SELECT statement example executes against the employees and department table in the cgDepts autorefresh cache group. With this example, since the foreign key is not enforced on TimesTen and the autorefresh process applies several transactions, the employee table updates may be inserted before the department updates.

In addition, all of the updates for both tables in the cache group are not applied until the autorefresh cycle has completed. In the following example, the SELECT statement is executed before the autorefresh process is complete. Thus, the results do not show all of the expected data, such as the department name and several employees (some of the lawyers in the legal department 1000) are missing.

SELECT e.department_id, d.DEPARTMENT_NAME, e.FIRST_NAME, e.LAST_NAME  
       FROM employees e, departments d         
       WHERE e.DEPARTMENT_ID  = d.DEPARTMENT_ID (+) 
       AND e.department_id >= 1000 ORDER BY 1,2,3,4;
< 1000, <NULL>, Alan, Dershowitz >
< 1000, <NULL>, F. Lee, Bailey >
< 1000, <NULL>, Johnnie, Cochran >
3 rows found.

However, after the autorefresh process completes, transactional consistency is maintained. The following shows the same SELECT statement executed after the autorefresh is complete. All expected data, the department information and all of the new lawyers, are updated.

SELECT e.department_id, d.DEPARTMENT_NAME, e.FIRST_NAME, e.LAST_NAME  
       FROM employees e, departments d         
       WHERE e.DEPARTMENT_ID  = d.DEPARTMENT_ID (+) 
       AND e.department_id >= 1000 ORDER BY 1,2,3,4;
< 1000, Legal, Alan, Dershowitz >
< 1000, Legal, Barry, Scheck >
< 1000, Legal, F. Lee, Bailey >
< 1000, Legal, Johnnie, Cochran >
< 1000, Legal, Robert, Kardashian >
< 1000, Legal, Robert, Shapiro >
6 rows found.

For autorefresh cache groups that have more than one table, you can also experience transactional inconsistency if you execute SQL statements while the autorefresh process is in progress.

1. Initiate the transaction limit for incremental autorefresh cache groups of 2 seconds with the ttCacheAutorefreshXactLimit built-in procedure and create a single autorefresh cache group with two tables: the employees and departments tables.

   CALL ttCacheAutorefreshXactLimit('2000', '3');
   < 2000, 3 >
   1 row found.
    
   CREATE READONLY CACHE GROUP cgDeptEmpls AUTOREFRESH MODE INCREMENTAL
    INTERVAL 2 SECONDS 
   FROM departments
        ( department_id    NUMBER(4) PRIMARY KEY
        , department_name  VARCHAR2(30) NOT NULL
        , manager_id       NUMBER(6)
        , location_id      NUMBER(4)
        )
      , employees
        ( employee_id    NUMBER(6) PRIMARY KEY
        , first_name     VARCHAR2(20)
        , last_name      VARCHAR2(25) NOT NULL
        , email          VARCHAR2(25) NOT NULL UNIQUE
        , phone_number   VARCHAR2(20)
        , hire_date      DATE NOT NULL
        , job_id         VARCHAR2(10) NOT NULL
        , salary         NUMBER(8,2)
        , commission_pct NUMBER(2,2)
        , manager_id     NUMBER(6)
        , department_id  NUMBER(4)
        , foreign key(department_id) references departments(department_id)
        );
   

2. Manually load the cache group.

   LOAD CACHE GROUP cgDeptEmpls COMMIT EVERY 256 ROWS;
   27 cache instances affected.
   

3. Perform a SELECT statement on TimesTen that uploads all of the legal department data.

   SELECT e.department_id, d.department_name, count(*)
          FROM employees e, departments d         
          WHERE e.department_id  = d.department_id (+) 
          GROUP BY e.department_id, d.department_name
          ORDER BY 1 desc;
   < 110, Accounting, 2 >
   < 100, Finance, 6 >
   < 90, Executive, 3 >
   < 80, Sales, 34 >
   < 70, Public Relations, 1 >
   < 60, IT, 5 >
   < 50, Shipping, 45 >
   < 40, Human Resources, 1 >
   < 30, Purchasing, 6 >
   < 20, Marketing, 2 >
   < 10, Administration, 1 >
   11 rows found.
   

4. On Oracle, insert a new legal department, numbered 1000, with 6 new lawyers in both the employee and department tables.

5. When performing a SELECT statement on TimesTen during the autorefresh process, only data on two of the lawyers in department 1000 have been uploaded into TimesTen.

   SELECT e.department_id, d.department_name, count(*)
          FROM employees e, departments d         
          WHERE e.department_id  = d.department_id (+) 
          GROUP BY e.department_id, d.department_name
          ORDER BY 1 desc;
   < 1000, Legal, 2 >
   < 110, Accounting, 2 >
   < 100, Finance, 6 >
   < 90, Executive, 3 >
   < 80, Sales, 34 >
   < 70, Public Relations, 1 >
   < 60, IT, 5 >
   < 50, Shipping, 45 >
   < 40, Human Resources, 1 >
   < 30, Purchasing, 6 >
   < 20, Marketing, 2 >
   < 10, Administration, 1 >
   12 rows found.
   

6. However, after the autorefresh process completes, all 6 employees (lawyers) in the legal department have been uploaded to TimesTen. Now, it is transactionally consistent.

   SELECT e.department_id, d.department_name, COUNT(*)
          FROM employees e, departments d         
          WHERE e.department_id  = d.department_id (+) 
          GROUP BY e.department_id, d.department_name
          ORDER BY 1 desc;
   < 1000, Legal, 6 >
   < 110, Accounting, 2 >
   < 100, Finance, 6 >
   < 90, Executive, 3 >
   < 80, Sales, 34 >
   < 70, Public Relations, 1 >
   < 60, IT, 5 >
   < 50, Shipping, 45 >
   < 40, Human Resources, 1 >
   < 30, Purchasing, 6 >
   < 20, Marketing, 2 >
   < 10, Administration, 1 >
   12 rows found.
   

7. The large transaction is complete, so disable the transaction limit for autorefresh cache groups with a 2 second interval.

   call ttCacheAutorefreshXactLimit('2000', 'OFF');
   

Retrieving statistics to evaluate performance when a transaction limit is set

To see how a autorefresh transaction limit for a particular autorefresh interval is performing, you can retrieve statistics for the last 10 incremental autorefresh transactions for this autorefresh interval with the ttCacheAutorefIntervalStatsGet built-in procedure. See "Retrieving statistics on autorefresh transactions" for more information.

How to determine the cache group name for a particular select limit

To determine the interval for a cache group, use ttIsql and run the cachegroups command:

> cachegroups cgowner.cgname;

This returns all attributes for the cgowner.cgname cache group including the interval.

To determine which intervals have a select limit, you can run the following query on the Oracle database where <cacheAdminUser> is the cache administrator, <hostName> is the host name of the machine where the TimesTen database is located, <databaseFileName> is the database path taken from the DataStore attribute, and substitute the version number (such as 06) for the xx.

SELECT * FROM <cacheAdminUser>.tt_xx_arinterval_params
 WHERE param='AutorefreshSelectEveryN'
   AND host='<hostName>'
   AND database like '%<databaseFileName>%'
 ORDER BY arinterval;
 

For example, if the cache administrator user name is pat, the host name is myhost, the database file name is myTtDb, and 06 is substituted for xx that is the TimesTen Classic minor release number then:

SELECT * FROM pat.tt_06_arinterval_params
 WHERE param='AutorefreshSelectEveryN'
   AND host='myhost'
   AND database like '%myTtDb%'
 ORDER BY arinterval;

The interval is stored in milliseconds.

Retrieving statistics to evaluate performance when using a select limit

To see how a select limit for a particular autorefresh interval is performing, you can retrieve statistics for incremental autorefresh transactions for this autorefresh interval with the ttCacheAutorefIntervalStatsGet built-in procedure. See "Retrieving statistics on autorefresh transactions" for more information.