1. Monitoring and Troubleshooting Guide
  2. Troubleshooting TimesTen Applications and Databases

3 Troubleshooting TimesTen Applications and Databases

The following sections provide information to help you diagnose and remedy some problems encountered while using a TimesTen database:

Note:

If you are still having problems with your database after following the troubleshooting recommendations in this chapter, contact TimesTen Customer Support.

Unable to start or stop TimesTen daemon

This section describes what to check if you cannot start or stop the TimesTen main daemon.

Possible cause What to do

Privilege is incorrect.

You need the ADMIN privilege to start or stop the TimesTen daemon. Ensure that you are using the ttDaemonAdmin utility to start the daemon. The output from ttDaemonAdmin shows whether you have the correct privilege.

Another process is using the TimesTen daemon port.

Use the ttVersion utility to verify what port number the TimesTen daemon is expected to use. Use an operating system command like netstat to check whether another process is listening on the port. If there is a conflict, either change the port number used by the other process.

TimesTen daemon is already running.

Ensure that you are using the ttDaemonAdmin utility to start the daemon. The output from ttDaemonAdmin shows whether the daemon is already running.

There are other problems.

Inspect the user error log produced by the daemon.

No response from TimesTen daemon or subdaemon

The following sections describe what to do if one or more of the TimesTen processes appears to be unavailable:

Check the TimesTen user error log

If you receive an error that indicates the TimesTen subdaemon has stopped and inspect the user error log.

If the TimesTen daemon crashes, it cannot send anything to the user error log, but the subdaemons send a 'main daemon vanished' message to the log before exiting:

09:24:13 Err : 4375 ------------------: Main daemon has vanished

Restart the daemon. The next connection to each database causes TimesTen to recover from the checkpoint and transaction log files. See "Working with the TimesTen Daemon" in the Oracle TimesTen In-Memory Database Operations Guide.

Extract a stack trace from the core file

If you experience a crash by one of the TimesTen processes on a Linux or UNIX system and have exhausted all of the diagnostic options, check to see if TimesTen has generated a core file. Use the ttVersion utility to find the core file. Look for a line in the output that shows a path for the daemon home directory: 

TimesTen Release (ttuser:40732)
2011-04-04T17:53:04Z
   Instance admin: ttuser
   Instance home directory:
/node1/ttuser/ttcur/TTBuild/linux8664_dbg/install
   Daemon home directory:
/node1/ttuser/ttcur/TTBuild/linux8664_dbg/install/info

After locating the core file, attach to the debugger on the system and extract the stack trace from the core file and send the trace results to TimesTen Customer Support.

Unable to create shared segment

You may receive an error that indicates that a shared segment could not be created:

4671: TT14000: TimesTen daemon internal error: Error 28 creating shared segment,
KEY 0x0201f7eb
4671:  -- OS reports too many shared segments in use
4671:  -- Confirm using 'ipcs' and take appropriate action
4671: 18538 ------------------: subdaemon process exited

Using the Linux or UNIX ipcs command may display information like this: 

------ Shared Memory Segments --------
 key        shmid      owner     perms      bytes      nattch     status
 0x00000000 1098350592 user1     777        10624      2          dest
 0x00000000 1084817409 user1     777        2439680    2          dest
 0x911fc211 1098383362 user2     666        67108864   1
 0x2814afba 170721285  root      666        1048576    1

A status of dest means the memory segment is marked to be destroyed. nattch shows the number of processes still attached to the memory segment. The ipcrm command cannot free the shared memory until the processes detach from the segment or exit. If an application connects to TimesTen and then becomes inactive, nothing can free the shared memory until the user exits or stops the application.

Application unable to connect to database in direct mode

This section describes what to check if your application cannot connect to a database in direct mode.

Possible cause See...

There is a mismatch between the release of TimesTen and database.

"Upgrading your database" for information.

User does not have CREATE SESSION privilege.

"Privileges to connect to database" for information.

File permissions are incorrect.

"Check file system permissions to access database" for information.

TimesTen daemon or Data Manager service is not running.

"Check that the TimesTen daemon is running" for information.

Incompatible connection attributes or incorrect path name for database set in the DSN.

"Check DSN definition" for information.

There is no available shared memory segment or maximum size of shared memory segment is too small.

"Manage semaphores and shared memory segments" for information.

There is not enough swap space.

"Check available swap space (virtual memory)" for information.

Number of file descriptors is inadequate.

"Increase the number of available file descriptors" for information.

There are other possible causes.

"Using the logs generated by the TimesTen daemon" for information.

Upgrading your database

A database is only guaranteed to be accessible by the same minor release of TimesTen that was used to create the database. When you upgrade the TimesTen software and you would like to use the new release to access a database that was previously created, create a database with the new release. Then use the ttMigrate utility to copy the tables, indexes, and table data from the old database to the new one.

See "Upgrades in TimesTen Classic" in the Oracle TimesTen In-Memory Database Installation, Migration, and Upgrade Guide for details.

Privileges to connect to database

The user must have the CREATE SESSION privilege to connect to the database. If you do not have access, the administrator must use the GRANT statement to grant you the CREATE SESSION privilege. See "Privileges to connect to the database" in the Oracle TimesTen In-Memory Database Security Guide.

Check file system permissions to access database

A permission denied error is generated if you attempt to connect to a database and you do not have the proper permissions to access the checkpoint or transaction log files or the directory where those files reside. Check the file system permissions on the files located in the directory specified in the DataStore attribute in your DSN.

Check that the TimesTen daemon is running

If the TimesTen daemon or Data Manager service is not running, an attempt to connect to a database generates TimesTen error 799 "Unable to connect to daemon; check daemon status."

Use the ttStatus utility as described in "Check the TimesTen user error log" to check the status of the TimesTen daemon.

Check DSN definition

In your DSN description, perform the following:

Check DSN attributes

Certain connection options or DSN attribute settings combinations are not compatible. In cases where incompatible settings are used, an error is returned to the application when it attempts to connect to a database.

Check path name to database and transaction log directories

Confirm that you have specified the correct path names in the DataStore and LogDir attributes in your DSN. Also confirm that the path names are absolute path names, rather than relative. Otherwise, the path name will be relative to the directory where the application was started.

Manage semaphores and shared memory segments

An error is generated if you attempt to connect to or create a shared database whose size is larger than the maximum size of shared memory segments configured on your system. Also, an error is generated if the system cannot allocate any more shared memory segments.

On Linux and UNIX systems, use commands similar to the following:

  • ipcs -ma to check if you have other shared memory segments using up memory, such as Oracle Database instances or other instances of TimesTen.

  • ipcrm to remove a message queue, semaphore set or shared memory segment identifier. Use ipcrm to clean up semaphores or shared memory segments after a faulty TimesTen shutdown, instance crash, daemon crash or other application issues that use shared memory segments and semaphores. Use -m to remove a shared memory segment. Use -s to remove a semaphore.

  • ps -eafl to see how much memory is being used by running processes.

  • ulimit -a to see if there are any limits on the maximum amount of memory one process can address, maximum file size, and the maximum number of open files.

If a shared memory segment is available but is too small to hold your database, use the ttSize utility to estimate the amount of memory required for your tables and then check the values of the PermSize and TempSize attributes to verify the amount of memory established for your database. "Monitoring PermSize and TempSize attributes" in the Oracle TimesTen In-Memory Database Operations Guide describes guidelines for setting the size of your permanent and temporary memory regions. If the amount of memory established for your database is too large, reset PermSize and TempSize to smaller values. See "Check the amount of memory allocated to the database" for more information. Another option is to increase the maximum size of the shared memory segment, as described below.

If your application is connected to your database and your database is invalidated because of a system or application failure, the database shared segment is not automatically detached from your application. The next time your application tries to use the database, the application will be disconnected from the database and the shared segment will be detached. If your application remains connected to the invalidated database and does not try to use the database, the application will not disconnect from the database and the shared segment will not be detached. To free memory and swap space, make sure you disconnect or terminate all applications that are connected to the invalidated database.

If a database becomes invalidated because of a system or application failure, and you have disconnected all applications from the database, a subsequent connection recovers the database. If recovery fails because you have run out of database space, then reconnect to the database with a larger PermSize and TempSize value than the ones that are currently in effect.

See "Operating system prerequisites" in the Oracle TimesTen In-Memory Database Installation, Migration, and Upgrade Guide for more information on how to configure shared memory.

Check available swap space (virtual memory)

There must be enough swap space to back up shared memory.

On Linux and UNIX systems, use the swap command to check and add virtual memory to your system.

Increase the number of available file descriptors

Each process connected to a TimesTen database keeps at least one operating system file descriptor open. Additional file descriptors may be opened for a connection when the connection commits or rolls back a transaction. If you receive an error that all file descriptors are in use when attempting to connect to a database, increase the allowable number of file descriptors. See your operating system documentation for limits on file descriptors and information about changing the number of file descriptors. Also see "Limits on number of open files" in the Oracle TimesTen In-Memory Database Reference.

Troubleshooting Client/Server problems

This section includes the following topics:

Cannot connect to the TimesTen Server

You have not correctly identified the system where the TimesTen Server is running.

On a Windows client, select the TimesTen Server in the TimesTen Data Source Setup dialog that is displayed as part of the ODBC Data Source Administrator. To verify the TimesTen Server:

  1. On the Windows Desktop, choose Start > Settings > Control Panel.
  2. Double click the ODBC icon. This opens the ODBC Data Source Administrator.
  3. Click the System DSN tab. This displays the System Data Sources list.
  4. Select the TimesTen Client data source. This opens the TimesTen Client DSN Setup dialog.
  5. Click Servers. This opens the TimesTen Logical Server List.
  6. Select the TimesTen Server from the list. This opens the TimesTen Logical Server Name Setup dialog.
  7. Verify that the values for the Network Address and Port Number are correct. If necessary, change the values.

    Note:

    If you typed the host name or network address directly into the Server Name field of the TimesTen Client DSN Setup, the Client tries to connect to the TimesTen Server using the default port.

If the Network Address and Port Number values are correct, the TimesTen Server may not be running. In the Oracle TimesTen In-Memory Database Operations Guide, see "Testing connections" for more information about identifying this problem.

On Linux and UNIX, specify the TimesTen Server with the TTC_Server connection attribute in the odbc.ini file on the client. If the value specified for TTC_Server is an actual host name or IP address, the client tries to connect to the TimesTen Server using the default port. In TimesTen, the default port is associated with the TimesTen release number. If the value specified for TTC_Server is a logical Server Name, this logical Server Name must be defined in the ttconnect.ini file. The ttconnect.ini entry for this Server Name must correctly define the host name/IP address and port number on which the TimesTen Server is listening.

If the Network Address and Port Number values are correct, the TimesTen Server may not be running or did not start. See "Testing connections" in the Oracle TimesTen In-Memory Database Operations Guide for more information about identifying this problem.

TimesTen Server failed

Check the server's log file. See "Creating and configuring Client DSNs on Linux and UNIX" and "Managing TimesTen daemon attributes" in the Oracle TimesTen In-Memory Database Operations Guide for information on the TimesTen daemon configuration file.

The maximum number of concurrent IPC connections to the Server of a particular TimesTen instance is 24,999. However, TimesTen has a limit of 2043 connections (direct or client/server) to a single DSN.

Client/server users can change the file descriptor limit to support a large number of connections.

Cannot find Server DSN

On Linux and UNIX, verify that the Server DSN is defined in the sys.odbc.ini file on the system running the TimesTen Server.

On Windows, verify that the Server DSN is defined as a System DSN in the ODBC Data Source Administrator on the system running the TimesTen Server. See "Creating and configuring a logical server name on Windows" in the Oracle TimesTen In-Memory Database Operations Guide.

TimesTen Server failed to load DRIVER

This error only occurs on Linux or UNIX systems. Open the sys.odbc.ini file on the system running the TimesTen Server and locate the Server DSN you are trying to connect. Verify that the dynamic library specified in the DRIVER attribute for the Server DSN exists and is executable.

Application times out when accessing TimesTen Server

The default TimeOut interval is 60 seconds.

To increase this interval on Linux or UNIX, change the value of the TTC_Timeout attribute in the odbc.ini file.

TimesTen Client loses connection with TimesTen Server

Check to see if the error was due to the Client timing out. Check the TimesTen Server's log to see why the Server may have severed connection with the Client. Use ping to determine if your network is up or try using telnet to connect to the TimesTen Server port number.

Failed to attach to shared memory segment for IPC

While using shared memory segment (SHM) as IPC, the application may see the following error message from the TimesTen Client ODBC Driver if the application reaches the system-defined per-process file-descriptor-limit. 

SQLState    = S1000,
Native Error = 0,
Message     = [TimesTen][TimesTen 22.1 CLIENT]Failed to attach to shared memory 
segment for IPC. System error: 24

This may happen during a connect operation to the Client DSN when the shmat system call fails because the application has more open file descriptors than the system-defined per-process file descriptor limit. To correct this problem, you must increase your system-defined per-process file descriptor limit. For more information about file descriptor limits, see "System Limits" in the Oracle TimesTen In-Memory Database Reference.

Thread stack overflow when using multiple client connections

You may receive messages about a segmentation fault that mention a possible thread stack overflow.

If these messages occur, increase the server stack size by one of the following methods:

  • Specify the server_stack_size option in the /conf/timesten.conf file.

  • Specify the ServerStackSize connection attribute for a specific DSN. This takes precedence over the value in the /conf/timesten.conf file.

Increasing the server stack size decreases the number of concurrent connections that can be made before running out of swap space.

See "Working with the TimesTen Client and Server" in the Oracle TimesTen In-Memory Database Operations Guide.

Out of space when DSN specifies new database

You may receive "out of space" messages if you change a DSN to specify a new database while there are existing connections to the original database in a system with multiple client connections.

Close all connections to the original database. This causes a new server process to be created for connections to the database that is now specified in the DSN. Use the ttStatus utility to list the connections for the old database. Alternatively, you can restart the server by using the ttDaemonAdmin utility with the -restartServer option, which resets all client connections on all DSNs in the instance.

Application connects or disconnects are slow

This section describes what to check if you encounter slow connects and disconnects to a database. First check your ramPolicy setting. If the ramPolicy setting is inUse, and you are the only user, then at first connect, the entire database is loaded into memory. Consider changing ramPolicy to either always or manual. For more information, see "ttAdmin" in the Oracle TimesTen In-Memory Database Reference.

Possible cause See...

Database is being recovered.

"Check if database is being recovered" for information.

ODBC tracing is enabled.

"Check ODBC tracing" for information.

There are other possible causes.

"API tracing" for information.

Check if database is being recovered

A slow connect may indicate that a TimesTen database is being recovered. This happens only for a first connect.

Check ODBC tracing

On Windows platforms, if ODBC tracing is enabled, it can slow connect and disconnect speeds. Double-click ODBC in the Control Panel to open the ODBC Data Source Administrator. Select the Tracing tab and confirm tracing is disabled. See "Using ODBC tracing" for details.

Application is disconnected unexpectedly

If an application becomes disconnected from a TimesTen database, one of the following events occurs:

  • If there was no outstanding transaction, the connection is cleanly removed by the TimesTen daemon. Other existing connections continue processing as if no problem had occurred.

  • If there was an outstanding transaction, the transaction is rolled back and the connection is cleanly removed by the TimesTen daemon. Other existing connections continue processing as if no problem had occurred.

This section describes what to check if your application unexpectedly disconnects from the database.

Possible cause See...

There is an internal application error.

"Check for ODBC, JDBC, OCI, Pro*C, and PL/SQL application errors" for information.

There is failure of a concurrent application thread.

"Check for ODBC, JDBC, OCI, Pro*C, and PL/SQL application errors" for information.

"Check the user error log" for information.

If using a client/server connection, the client may have disconnected from the application.

"Troubleshooting Client/Server problems" for information.

There is an error in the TimesTen library.

Contact TimesTen Customer Support.

Check for ODBC, JDBC, OCI, Pro*C, and PL/SQL application errors

Check for the following types of errors:

  • ODBC errors returned by the SQLError function

  • JDBC errors returned by the SQLException class

TimesTen OCI and Pro*C applications, and those that use PL/SQL, report errors using Oracle database error codes instead of TimesTen error codes. The error messages that accompany the error codes may come from the TimesTen error catalog or the Oracle database error catalog.

By default, TimesTen messages and diagnostic information are stored in:

  • A user error log that contains error message information. Generally, these messages contain information on actions you may need to take. The default file is timesten_home/diag/tterrors.log. For more information on modifying the location of the user error log, see "Error, warning, and informational messages" in the Oracle TimesTen In-Memory Database Operations Guide.

  • A support log containing everything in the user error log plus information of use by TimesTen Customer Support. The default file is timesten_home/diag/ttmesg.log. For more information on modifying the location of the user error log, see "Error, warning, and informational messages" in the Oracle TimesTen In-Memory Database Operations Guide.

  • An invalidation file containing diagnostic information when TimesTen invalidates a database. This file provides useful troubleshooting information for TimesTen Customer Support. The invalidation file is created and named based on the value specified by the DataStore connection attribute. This connection attribute is not a file name. For example on Linux and UNIX systems, if the DataStore connection attribute is /home/ttuser/AdminData, the actual invalidation file name has a suffix, .inval, /home/ttuser/AdminData.inval. For more information on the DataStore connection attribute, see "DataStore" in the Oracle TimesTen In-Memory Database Reference.

It may be helpful to use ttTraceMon to generate a level 4 ERR trace for the application and review all of the errors messages that are pushed in the TimesTen direct driver. See "ERR tracing" for details.

Check the user error log

If a TimesTen application disconnects without returning an ODBC error or any other warning, look through the user error log. See "Using the logs generated by the TimesTen daemon" for information.

Application is slow

For details on how to maximize the performance of your application and TimesTen database, see:

This section describes some issues that impair performance.

Possible cause See...

Using client/server mode.

"Consider connection mode" for information.

Database statistics are outdated.

"Update statistics for your tables" for information.

Committing transactions too frequently.

"Turn off autocommit mode" in the Oracle TimesTen In-Memory Database Operations Guide.

DurableCommits attribute is enabled.

"Use durable commits appropriately" in the Oracle TimesTen In-Memory Database Operations Guide.

SQL statements used more than once are not prepared.

"Prepare statements in advance" in the Oracle TimesTen In-Memory Database Operations Guide.

There is a wrong kind of index, too many indexes, or a wrong size for hash index.

"Select the appropriate type of index" in the Oracle TimesTen In-Memory Database Operations Guide.

"Size hash indexes appropriately" in the Oracle TimesTen In-Memory Database Operations Guide.

Use of locks is inefficient.

"Verify lock and isolation levels" for information.

Operations are slightly slower when performed on out-of-line columns versus inline.

"Inline and out-of-line columns" in the Oracle TimesTen In-Memory Database Operations Guide.

Materialized view is improperly configured.

"Performance implications of materialized views" and "Materialized view tuning" in the Oracle TimesTen In-Memory Database Operations Guide.

If replication is used, configuration of replication scheme or network environment may be impacting application.

"Poor replication or XLA performance" for information.

If TimesTen Cache is used, TimesTen Cache configuration or environment may be impacting application.

"Poor autorefresh performance" for information.

There are too many table partitions.

"Check partition counts for the tables" for information.

Tracing is unnecessarily enabled for one or more TimesTen components.

"Check trace settings" for information.

Consider connection mode

Client/server connections are slower than direct connections to TimesTen databases. Driver manager connections can also moderately impact performance. The performance overhead imposed by client/server connections can be significant because of the network latencies involved in all communication with the database.

If your application must run on a different system from the one hosting the database, see "Client/Server tuning" in the Oracle TimesTen In-Memory Database Operations Guide.

Update statistics for your tables

The TimesTen query optimizer in general is very good at choosing the most efficient query plan. However, it needs additional information about the tables involved in complex queries in order to choose the best plan. By knowing the number of rows and data distributions of column values for a table, the optimizer has a much better chance of choosing an efficient query plan to access that table.

Before preparing queries that will access a TimesTen table, use the ttOptUpdateStats procedure to update the statistics for that table. When updating the statistics for a table, you get the best results if you update statistics on your tables after loading them with data, but before preparing your queries. For example, if you update statistics on a table before populating it with data, then your queries are optimized with the assumption that the tables contain no rows (or very few). If you later populate your tables with millions of rows and then execute the queries, the plans that worked well for the situation where your tables contained few rows may now be very slow.

For more information about updating statistics, see "The TimesTen Query Optimizer" in the Oracle TimesTen In-Memory Database Operations Guide.

Verify lock and isolation levels

The manner in which multiple applications concurrently access the database can have a major impact on performance.

An application can acquire locks on the entire database, individual tables, and individual rows. Additionally, applications can set an isolation level that determines whether they hold read and update locks until their transactions commit or roll back.

Check the SYS.SYSTEMSTATS table, the SYS.MONITOR table, or use the ttXactAdmin utility to detect whether an application is spending time waiting for locks. See "Check for deadlocks and timeouts" and "Using the ttXactAdmin utility" for information.

If lock contention is high, you may be able to improve the overall performance of your system by implementing the following:

  • Set the LockLevel configuration attribute or use the ttLockLevel procedure to place locks on rows, rather than on the entire database. Row locking is the default.

  • Use the ttOptSetFlag procedure to prevent the query optimizer from placing locks on tables. Table locks are sometimes the default, particularly for updates that affect many rows.

  • Use read-committed isolation level (Isolation=1, the default) for those applications do not require serializable access to the transaction data.

If you see a lot of lock contention, but the above settings are all set to minimize contention, then the contention may be related to the application itself. For example, concurrent threads may be repeatedly accessing the same row. The ttXactAdmin utility can sometimes help you detect this sort of contention. Tracing can also be useful in this situation.

For more information about locks and isolation levels, see "Concurrency control through isolation and locking" in the Oracle TimesTen In-Memory Database Operations Guide.

Check trace settings

Use ttTraceMon -e show as described in "Using the ttTraceMon utility" to confirm tracing is off on all TimesTen components. ERR should be set to 1; all other components should be set to 0. Trace levels are preserved when a database is reloaded.

On Windows platforms, confirm that ODBC tracing is disabled. Double-click ODBC in the Control Panel to open the ODBC Data Source Administrator. Select the Tracing tab and confirm tracing is disabled. See "Using ODBC tracing" for information.

Check partition counts for the tables

When a table is created, it has one partition. When you use ALTER TABLE ... ADD COLUMN to add new columns, a new partition is added to the table. Adding multiple columns with a single ALTER TABLE ... ADD COLUMN statement only adds one partition.

There is a limit of 999 partitions per table. Exceeding this number generates error 8204. An extra read for each new partition slightly degrades performance for each of the new partitions. A high partition count should be avoided. On replicated tables that have multiple partitions, additional space is used for each update on the subscriber side, proportional to the number of partitions. This can result in the subscribers using slightly more perm space than the master.

The partition value for each table is tracked in the SYS16 column of the system table, SYS.TABLES. Obtain the partition counts for tables by using the following query: 

SELECT tblname, sys16 FROM SYS.TABLES;

If you discover that a table has too many partitions, do one of the following:

  • Recreate the table.

  • Save and restore the table. Use ttMigrate -c to create a migration file. Then restore the table without additional partitions by using ttMigrate -r -relaxedUpgrade.

ALTER TABLE ... DROP COLUMN does not remove partitions from a table. On replicated systems, all master and subscriber databases must be migrated using the -relaxedUpgrade option. Replication does not occur for tables that have different partition structures.

Application unresponsive, appears hung

This section describes what to check if your application is unresponsive and appears to be hung. First use pstack or equivalent to determine where your application is spending its time.

Possible cause See...

There is an internal application error.

"Check for ODBC errors" for information.

Connection attributes set in DSN are inconsistent.

"Consider connection mode" for information.

There is excessive lock contention.

"Check for deadlocks and timeouts" for information.

For any possible cause, see "Check logs and gather trace information" for details.

Check logs and gather trace information

If your application hangs, check the transaction log by using the ttXactAdmin utility. See "Using the ttXactAdmin utility" for information.

You can also check the user error log for errors. See "Using the logs generated by the TimesTen daemon" for details.

You can also generate a trace log to detect the activities on various TimesTen components. See "Using the ttTraceMon utility" for information.

Check for ODBC errors

Check the ODBC errors returned by the SQLError function in all applications to determine whether one of them has encountered a problem that caused it to hang. Call SQLError after each ODBC call to identify error or warning conditions when they first happen. Examples of SQLError usage can be found in the demo programs and in "Retrieving errors and warnings" in the Oracle TimesTen In-Memory Database Error Messages and SNMP Traps.

If the problem is repeatable, use ttTraceMon to generate a SQL trace to determine where the application is hanging. See "SQL tracing" for details. In more extreme cases, it may be helpful to generate a level 4 ERR trace for the application and review all of the errors messages that are pushed in the TimesTen direct driver. See "ERR tracing" for details.

Check for deadlocks and timeouts

If there is no connect problem, a deadlock or timeout may be the problem. The SYS.SYSTEMSTATS table and the SYS.MONITOR table records information about deadlocks and timeouts. See "Monitoring the TimesTen system tables" for information on how view the contents of this table. You can also use the ttXactAdmin utility to detect the types of locks currently held by uncommitted transactions and the resources on which they are being held.

If a deadlock occurs, the TimesTen subdaemon negotiates the problem by having an application involved in the deadlock generate TimesTen error 6002, "Lock request denied because of deadlock." The error message contains the SQL that the lock holder is running, which can help you diagnose the cause of the deadlock. If your application encounters this error, it should roll back the transaction and then reissue the statements for that transaction. Deadlocks can be caused if your application issues statements in a particular order that results in a circular wait, and can sometimes be prevented by changing the order in which the statements are issued.

An application encounters TimesTen error 6003, "Lock request denied because of timeout," if it cannot acquire a lock within the time period defined by the lock timeout interval set by the LockWait attribute in the DSN or by the ttLockWait procedure in your application. Upon encountering a timeout error, your application can reissue the statement. Keeping transactions short reduces the possibility of lock timeout errors.

In multithreaded applications, a thread that issues requests on different connection handles to the same database may encounter lock conflict with itself. TimesTen resolves these conflicts with lock timeouts.

Application unable to find previously created objects

This section describes what to check if your application cannot locate previously created tables, indexes, sequences or views in the database.

Possible cause See...

Database is temporary.

"Check temporary DSN attribute" for information.

Overwrite attribute is enabled.

"Check Overwrite DSN attribute" for information.

Path name specified in DSN is relative.

"Check path name to database" for information.

Check temporary DSN attribute

Temporary databases (DSN attribute: Temporary=1) persist until all connections to the database have been removed. When attempting to access a table in a temporary database and the table does not exist, it is possible that the database in which the table resided in has been dropped.

Check Overwrite DSN attribute

If the Overwrite and AutoCreate DSN attributes are enabled and the database already exists, TimesTen drops that database and creates a new one. Any tables that were created in the old database are dropped.

Check path name to database

To ensure that you are always accessing the same database when connecting to a particular DSN, use an absolute database path name instead of a relative one. For example, if the demo database is in the datastore directory, specify: 

DataStore=/datastore/demo

rather than:

DataStore=demo

In the latter case, the database path name is relative to the directory where the application was started. If you cannot find a table and you are using a relative database path name, it is possible that the database in which the table resides in does exist but the database (checkpoint and log) files are in a different directory than the one that you are accessing.

See "Specifying Data Source Names to identify TimesTen databases" in the Oracle TimesTen In-Memory Database Operations Guide.

Troubleshooting OCI and Pro*C/C++ applications

On Windows, if NLS_LANG is not set in the environment, the NLS_LANG setting is taken from the registry, HKEY_LOCAL_MACHINE\Software\ORACLE\NLS_LANG. If NLS_LANG is set to an invalid value or if NLS_LANG indicates an unsupported character set, either an OCI connection failed error or an ORA-12705 error is thrown. For more information on supported character sets, see "Supported character sets" in the Oracle TimesTen In-Memory Database Reference.

See "Globalization support in TimesTen OCI" in the Oracle TimesTen In-Memory Database C Developer's Guide for more information on NLS_LANG.

Running out of a resource

This section describes what to check if TimesTen runs out of resources such as memory space, file system space, file descriptors, and semaphores.

Symptom See...

Memory consumption seems high.

"Operating system tools and shared memory" for information.

Running out of memory space.

Running out of file system space.

"Check transaction log file use of file system space" for information.

"Check if tracing is enabled" for information.

Running out of transaction log space.

"Check transaction log file use of file system space" for information.

Running out of file descriptors.

"Increase the number of available file descriptors" for information.

Running out of semaphores.

"Check the semaphore limit" for information.

Running out of CPU.

Obtain a stack trace and contact TimesTen Customer Support.

Operating system tools and shared memory

Operating system tools such as top, vmstat, and sar provide statistics about processes and memory usage. The output from these tools can be misleading as an indicator of TimesTen memory consumption because they report shared memory usage for each process but do not report total shared memory usage. Adding together various memory statistics for TimesTen processes overestimates the amount of memory used by TimesTen because shared memory is by definition shared.

Check the amount of memory allocated to the database

TimesTen uses both permanent and temporary memory regions. The amount of memory allocated for these regions is set by the PermSize and TempSize attributes in the DSN definition for the database.

When the TimesTen database fills up, it is important to determine whether it is the permanent or the temporary memory region that is filling up. Use the ttIsql dssize command to list allocated, in-use, and high water mark sizes for the permanent and temporary memory regions. The dssize command selects the following values from SYS.MONITOR:

  • PERM_ALLOCATED_SIZE

  • PERM_IN_USE_SIZE

  • PERM_IN_USE_HIGH_WATER

  • TEMP_ALLOCATED_SIZE

  • TEMP_IN_USE_SIZE

  • TEMP_IN_USE_HIGH_WATER

The permanent memory region consists of table and index data, while the temporary memory region consists of internal structures, such as locks, sorting areas, and compiled commands.

Keeping transactions short and making sure there is enough temporary space in the database prevents locks from occupying all of the remaining temporary space. You can also use table locks if transactions are acquiring tens of thousands of row locks.

For tips on how to estimate the size of your database, see "Size your database correctly" in the Oracle TimesTen In-Memory Database Operations Guide.

Permanent memory region filling up

Consider whether you can drop any indexes. You may want to look at query plans to see which indexes are actually used. See "Viewing and changing query optimizer plans" in the Oracle TimesTen In-Memory Database Operations Guide. You can also use the ttRedundantIndexCheck procedure to discover redundant indexes. The procedure returns suggestions about which indexes to drop.

Use the ttSize utility to estimate the amount of memory used by each table in the database. If the amount of data you need to store is too big, you may need to reset the PermSize attribute for the database to increase the size of the permanent memory region. Alternatively, you may need to partition your data into several different databases if, for example, you cannot shrink the temporary memory region or create a bigger database because of limits on the memory.

Sometimes when the permanent memory region fills up, copying the data out of the database, deleting all the data, and copying it back in frees up space. This can be done more efficiently by using the ttMigrate utility with the -relaxedUpgrade option to migrate the data out, destroy and recreate the database, and migrate the data back in.

Finally, you may have to configure the operating system to allow a larger amount of shared memory to be allocated to a process. You may also have to allocate more swap space for virtual memory.

Temporary memory region filling up

Some commands may be allocating too much space because of out-of-date statistics. See "Update query optimizer statistics" for information.

If updating the statistics does not reduce the temporary memory region, disconnect all connections and then reconnect them. Verify that all connections have been disconnected by using the ttStatus utility. That frees up all temporary space, but you must reprepare commands.

Diagnose memory usage by queries. See "Check memory used by queries" for information.

Update query optimizer statistics

Make sure you have updated the optimizer statistics with the ttOptUpdateStats or ttOptEstimateStats procedure. To execute some queries, TimesTen must allocate temporary space. The amount of temporary space required is estimated from statistics about the tables used by the query. Without correct statistics, the temporary space required may be underestimated. See "Using the query optimizer" for information.

Check memory used by queries

You can check the memory that a query uses by observing the high water mark for temporary memory usage. The high water mark represents the largest amount of in-use temporary space used since the high water mark was initialized or reset.

Complete the following tasks:

  1. Use the ttIsql dssize command to check TEMP_IN_USE_SIZE and TEMP_IN_USE_HIGH_WATER. Alternatively, you can query the SYS.SYSTEMSTATS or the SYS.MONITOR table for these values.
  2. Call the ttMonitorHighWaterReset procedure to reset the TEMP_IN_USE_HIGH_WATER to the current value for TEMP_IN_USE_SIZE.
  3. Execute a query.
  4. Use dssize to check TEMP_IN_USE_HIGH_WATER for peak memory usage for the query.

Out of memory after fatal crash of the database

Irrecoverable errors, such as errors 846 and 994, invalidate a TimesTen database. However, the database remains in memory, which is only freed after all users have disconnected from the database. If the database is restarted while users are connected to the invalidated database, both old and new instances exist in memory at the same time. In this case, users could receive out-of-memory conditions. To prevent an "Out of memory" error, disconnect all active connections at the time of the irrecoverable error before reconnecting. These irrecoverable errors are rare.

Check transaction log file use of file system space

TimesTen saves a copy of the database in two checkpoint files, which are stored in the directory specified by the DataStore attribute. Each checkpoint file can grow on the file system to be equivalent to the size of the database in shared memory. For each permanent database, you must have enough file system space for the two checkpoint files and for transaction log files.

Transaction log files accumulate in the directory specified by the LogDir attribute and are only deleted when checkpoints are performed. If the LogDir attribute is not specified in the DSN, transaction log files accumulate in the directory specified by the DataStore attribute. The maximum size of your transaction log files is set by the LogFileSize attribute.

When a file system fills up with TimesTen data, it is most often due to a build-up of transaction log files. Transaction log files are used for numerous purposes in TimesTen, including transaction rollbacks, backups, and replication. It is important to determine which operation is putting a "hold" on the transaction log files, so that appropriate action can be taken to enable the transaction log files to be purged. This can be done by using the ttLogHolds built-in procedure. There are six types of log holds. They are discussed in detail below.

  • Long-running transactions - TimesTen uses the transaction log to roll back transactions. A log hold is placed for the duration of a transaction. Transactions that are active for a long time result in log file building up if the transaction has written at least one log record. (That is, it is not a read-only transaction.) Commit write transactions with reasonable frequency to avoid significant log file build-up. See "Size transactions appropriately" in the Oracle TimesTen In-Memory Database Operations Guide for more information on transaction length.

  • Checkpoint - If a TimesTen application crashes and the database must be recovered, the checkpoint files and transaction log files are used to recover the data. The "most recent" transaction log files are used -- those written since the checkpoint was done. Transaction log files accumulate during the interval between checkpoints. If checkpoints are done very infrequently, a large number of transaction log files may accumulate, particularly if many changes are made to the database during that interval. See "Checkpoint operations" in the Oracle TimesTen In-Memory Database Operations Guide.

  • Replication -TimesTen replication transmits changes from one database to one or more other databases. It does this by reading the transaction log and sending any relevant changes. If replication goes down, the transaction log files build up. See "Set the replication state of subscribers" in Oracle TimesTen In-Memory Database Replication Guide for more information on pausing and restarting or resetting replication.

  • Backup - TimesTen supports an incremental backup facility that uses transaction log files to augment a backup with changes made since the last backup. Transaction log files accumulate during the interval between incremental backups. To avoid a large log build-up, do incremental backups at relatively frequent intervals. If desired, disable incremental backups and do full backups instead.

    See "Backup, Restore, and Migrate Data in TimesTen Classic" in the Oracle TimesTen In-Memory Database Installation, Migration, and Upgrade Guide for more information.

  • XLA - TimesTen's persistent XLA facility reports changes to the database by using transaction log files. Transaction log files are kept until the corresponding transactions have been acknowledged using the ttXlaAcknowledge C function. Call ttXlaAcknowledge frequently enough to prevent transaction log files building up. See "Retrieving update records from the transaction log" in the Oracle TimesTen In-Memory Database C Developer's Guide.

  • XA - TimesTen's XA support uses transaction log files to resolve distributed transactions. If these transactions are not resolved in a timely manner, transaction log files build up. See "Distributed Transaction Processing: XA" in the Oracle TimesTen In-Memory Database C Developer's Guide.

The following attributes are related to file system use.

  • The LogPurge attribute indicates whether transaction log files that no longer have a hold on them are purged (removed from the file system) or simply archived (renamed). If the LogPurge attribute is set to the default value of 0, TimesTen renames transaction log files that it no longer needs by appending the string .arch to the name. Once renamed, you must delete the transaction log files manually when they are no longer needed. If transaction log files are not purged, they continue to accumulate space, even when no longer needed by TimesTen.

  • The Preallocate attribute indicates whether file system space should be reserved for checkpoint files at connect time. This is useful for big databases, to ensure that the file system always has room for the checkpoint files as data is added to the database.

Check if tracing is enabled

When tracing to a file has been enabled, the file may grow so large that a process attempting an operation may exceed the file limits. Tracing always appends to an existing file.

Check the semaphore limit

When creating multiple client/server connections to a TimesTen database configured to allow shared memory segment as IPC, you may encounter errors that indicate TimesTen could not create a semaphore. Semaphore limits are platform-dependent.

Duplicate results from a SELECT statement

Using read-committed isolation level can lead to duplicates in a result set. A SELECT statement selects more or fewer rows than the total number of rows in the table if some rows are added or removed and committed in the range in which the SELECT scan is occurring. This may happen when an UPDATE, INSERT or DELETE statement adds or deletes a value from an index and the SELECT scan is using this index. This can also happen when an INSERT or DELETE adds or deletes rows from the table and the SELECT operation is using an all-table scan.

Index values are ordered. An UPDATE of an index value may delete the old value and insert the new value into a different place. In other words it moves a row from one position in the index to another position. If an index scan sees the same row in both positions, it returns the row twice. This does not happen with a serial scan because table pages are unordered and rows do not need to be moved around for an UPDATE. Hence once a scan passes a row, it will not see that same row again.

The only general way to avoid this problem is for the SELECT statement to use Serializable isolation. This prevents a concurrent INSERT, DELETE or UPDATE operation. There is no reliable way to avoid this problem with INSERT or DELETE by forcing the use of an index because these operations affect all indexes. With UPDATE, this problem can be avoided by forcing the SELECT statement to use an index that is not being updated.

For more information about serializable isolation, see "Concurrency control through isolation and locking" in the Oracle TimesTen In-Memory Database Operations Guide.

Cannot attach PL/SQL shared memory

The PLSQL_MEMORY_ADDRESS first connection attribute determines the virtual address at which the PL/SQL shared memory segment is loaded into each process that uses the TimesTen direct drivers. Since each operating system platform has different mappings for its address space, the default values for the PL/SQL address space defined in the PLSQL_MEMORY_ADDRESS connection attribute are different for each platform, which avoids conflict with operating system mapped address space.

However, if your application overlaps with the PL/SQL mapped address space, you may receive error 8517 "Cannot attach PL/SQL shared memory; PLSQL_MEMORY_ADDRESS not valid or already in use." In this case, modify the setting for the PLSQL_MEMORY_ADDRESS connection attribute to eliminate the overlap. The reasons for receiving error 8517 can be one of the following:

  • User allocated memory already uses that address.

  • Some shared memory already uses that address.

  • A shared library already uses that address.

If an application accesses two or more TimesTen databases at the same time, you must modify the default setting for the PLSQL_MEMORY_ADDRESS attribute in all but one of the TimesTen databases, since the default settings would map the PL/SQL memory address to the same address for all TimesTen databases.