1. Error Messages and SNMP Traps
  2. Diagnostics Through SNMP Traps
  3. TimesTen and SNMP

TimesTen and SNMP

TimesTen cannot be queried nor controlled through SNMP. TimesTen only sends SNMP traps for certain critical events, to possibly facilitate some user recovery mechanisms. TimesTen can send traps for various events, including:

  • Assertion failure

  • Death of daemons

  • Database invalid

  • Replicated transaction failure

  • Database out of space

  • Autorefresh transaction failure

  • Replication conflict resolution

  • File write errors

These events also cause log entries to be written by the TimesTen daemon, but exposing them through SNMP traps allows for the possibility of having some network management software take immediate action.

Topics include:

TimesTen MIB

A Management Information Base (MIB) is similar to a database schema. It describes the structure of the SNMP data.

The MIB extension file, installation_dir/network/admin/samples/TimesTen-MIB.txt, describes the structure of the TimesTen SNMP information.

The TimesTen OID is rooted at Private Enterprise 5549. The complete path to root is iso.org.dod.internet.private.enterprise.TimesTen.* or numerically, 1.3.6.1.4.1.5549.*.

Data Types in TimesTen SNMP Traps

The TimesTen MIB defines more than 60 variables that are either INTEGER or TEXT data type.

Table 3-1 describes the variables that are ASN_INTEGER type:

Table 3-1 ASN_INTEGER Variables in the TimesTen MIB

Variable Description

ttPid

Process ID

ttTrapTruncated

1 if the trap is too large and has been truncated

ttDSNConn

Connection number, starting with 0

ttDSMaxSize

Maximum size of database, in KB

ttDSCurSize

Current size of database, in KB

ttDSReqSize

Requested increase of database size, in KB

ttDSError

Error number

ttTxnId

Transaction ID

ttTxnCounter

Transaction counter

ttStmId

Statement ID

ttQueryThreshold

Query threshold time, in seconds

ttClientPid

Process ID of the client that generated the query

ttViewId

View identifier

ttFileErrorNum

Error number when file access encounters an error

ttDaePid

Process ID of the daemon

ttDaeInst

Daemon number, starting with 0

ttRepPid

Process ID of replication agent

ttRepMasterPort

Port number of master

ttRepSubscriberPort

Port number of subscriber

ttRepConflictCount

Replication conflict counter

ttCacheAgentPid

Process ID of cache agent

The rest of the variables are ASN_OCTET_STRING type.

TimesTen SNMP Trap Names and Severity Levels

TimesTen SNMP traps can be categorized by severity level. The information in the trap can be of the type:

  • Error

  • Informational

  • Warning

Table 3-2 describes each trap and its severity level.

Table 3-2 TimesTen SNMP Trap Description and Severity Levels

Trap Name Severity Level Description

ttAssertFailTrap

Error

TimesTen Assertion Failure.

ttAsyncMVFailed

Warning

A refresh of Asynchronous materialized view failed. The SNMP trap includes dsname, daemon PID and viewid. If the error is due to a transient error, such as locking, the refresh may succeed in the next refresh.

ttCacheAgentDiedTrap

Error

The cache agent died.

ttCacheAgentFailoverTrap

Warning

The cache agent detected that a connection to the Oracle database had been lost and has begun to recover the connection.

ttCacheAutoRefFailedTrap

Error

TimesTen Cache incremental autorefresh failed.

ttCacheAutoRefQueFullTrap

Warning

TimesTen Cache incremental autorefresh queue is full.

ttCacheAutorefreshDsMarkedDeadTrap

Warning

TimesTen Cache incremental autorefresh failed. The cache agent for a remote datastore has stopped or is no longer responding. Autorefresh for the remote datastore has been disabled.

ttCacheAutorefreshLogSpaceDeFragDetectedTrap

Warning

Some of the autorefresh change log tables on the Oracle database are fragmented. See Fragmented Autorefresh Change Log Table Space in Oracle TimesTen In-Memory Database Monitoring and Troubleshooting Guide.

ttCacheAwtRtReadFailedTrap

Error

For asynchronous writethrough cache groups, runtime information is stored on the Oracle database instance. While reading this information from the Oracle database, replication either could not find the runtime data table (tt_version_reppeers) or could not find the information within the table.

ttCacheAwtRtUpdateFailedTrap

Error

For asynchronous writethrough cache groups, runtime information is stored on the Oracle database instance. While updating this information replication either could not find the runtime data table (tt_version_reppeers) or could not find the information within the table.

ttCacheCgNotAutorefreshedTrap

Warning

The cache group will not be autorefreshed. Instead, it must be manually recovered by performing manual load or refresh cache group.

ttCacheLowOracleTblSpace

Warning

The tablespace the cache admin user is using is below the minimum threshold.

ttCacheRecoveryAutorefreshTrap

Warning

The cache agent is performing a full autorefresh. This may be needed when a change log table on the Oracle database was truncated because of lack of tablespace for the cache administration user.

ttCacheValidationAbortedTrap

Error

The cache agent terminated cache group validation because of an irrecoverable error. Please refer to the user error log for details.

ttCacheValidationErrorTrap

Error

The cache agent has detected irrecoverable anomalies with cache group cache-group-name that will prevent it from properly refreshing the cache group, or it has detected irrecoverable anomalies within the refresh interval time-in-ms. Please refer to the user error log for details.

ttCacheValidationWarningTrap

Warning

The cache agent has detected anomalies with cache group cache-group-name that may prevent it from properly refreshing the cache group. Please refer to the user error log for details.

ttDaemonOutOfMemoryTrap

Error

Call to malloc failed in TimesTen daemon.

ttDSCkptFailedTrap

Error

A checkpoint has failed. Check the user error log and get view the checkpoint history using either the SYS.GV$CKPT_HISTORY or SYS.V$CKPT_HISTORY system views or the ttCkptHistory built-in procedure.

ttDSDataCorruptionTrap

Error

Database corruption error has occurred.

ttDSGoingInvalidTrap

Error

Setting database to invalid state. Database invalidation usually happens when an application that is connected to the database is terminated or exits abruptly without first disconnecting from the database. If TimesTen encounters an irrecoverable internal error during a database operation, it may also invalidate the database. You must commit or rollback and recover the database.

ttDSThreadCreateFailedTrap

Error

A process (typically multithreaded) having multiple connections to a database exits anomalously. The subdaemon assigned to clean up the connections creates a separate thread for each connection. If creation of one of these threads fails, this trap is thrown. Thread creation may fail due to memory limitations or having too many threads in the system. After the trap is thrown, the thread creation is attempted four more times, with an increasingly longer pause between each attempt. The total time between the first and last attempt is approximately 30 seconds. If the fifth attempt fails, the database is invalidated.

ttFileWriteErrorTrap

Error

Error encountered during file I/O write.

ttMainDaemonDiedTrap

Error

Main or sub daemons died anomalously. This message is sent by a subdaemon when it notices that the main daemon has died. It suggests that the main daemon has been terminated or has crashed. You must restart the main daemon.

ttMainDaemonExitingTrap

Informational

Main or sub daemons exiting ordinarily.

ttMainDaemonReadyTrap

Informational

Main daemon has started.

ttMsgLogOpenFailedTrap

Error

The message log could not be opened, possibly because of a lack of privileges on the file. Check the file location and privileges.

ttPartitionSpaceExhaustedTrap

Error

Database memory region (permanent or temporary) space is exhausted. This message is sent when either the permanent or temporary free space in the database is exhausted. Generally this message is preceded by the ttPartitionSpaceStateTrap warning message. See PermWarnThreshold and TempWarnThreshold in Oracle TimesTen In-Memory Database Reference.

ttPartitionSpaceStateTrap

Warning

Database memory region (permanent or temporary) space is transitioning from OK to low or vice versa. This message is sent when either the permanent memory region or the temporary memory region free space in the database reaches a threshold or transitions back below the threshold. This message is sent only when the free space has reached the threshold specified by the PermWarnThrehold or TempWarnThreshold attribute at the time of the first connection to the database. See PermWarnThreshold and TempWarnThreshold in Oracle TimesTen In-Memory Database Reference.

ttQueryThresholdWarnTrap

Warning

A SQL query exceeded the user-specified threshold. The text of the query can be found n the user log message. The Transaction ID and the Statement ID of the query can be found both in the trap and the user log message. After issuing the trap, the query continues running.

ttRepAgentClockSkewTrap

Error

Replication with a peer failed due to excessive clock skew. The skew between nodes in an active standby scheme has exceeded the allowed limit of 250ms.

ttRepAgentDiedTrap

Error

A replication agent has died anomalously. This message is sent when the main TimesTen daemon notices that a replication agent has died anomalously. This generally means that the replication agent has been terminated or has crashed.

ttRepAgentExitingTrap

Informational

Replication agent exiting ordinarily.

ttRepAgentStartingTrap

Informational

Replication agent starting.

ttRepAgentStateChange

Informational

Indicates the state of the replication agent. The possible states of the replication agent are ACTIVE, FAILED, IDLE, RECOVERING, STANDBY, and UNKNOWN. TimesTen only sends this trap when the state of the replication agent changes.

ttRepCatchupStartTrap

Warning

Indicates that TimesTen has begun to restore a master from a subscriber where bi-directional replication has been configured, after a failure.

ttRepCatchupStopTrap

Warning

Indicates that TimesTen has restored a master database from a subscriber, where bi-directional replication was configured.

ttRepConflictReportStartingTrap

Informational

Indicates that conflict reporting has been restarted because the rate of conflicts has fallen below the low water mark set in the replication scheme. This trap also indicates how many conflicts went unreported during the period in which reporting was suspended.

ttRepConflictReportStoppingTrap

Informational

Indicates that suspension of conflict reporting has occurred because the rate of conflicts has exceeded the high water mark set in the replication scheme.

ttRepReturnTransitionTrap

Warning

Replication return receipt has been enabled or disabled on the subscriber.

ttRepSubscriberFailedTrap

Error

Subscriber marked as failed because too much log accumulated on its behalf by the master.

ttRepSubscriberTCPConnectFailedTrap

Error

A replication TCP connection failed.

ttRepUpdateFailedTrap

Warning

A replication insert, update or delete operation failed.

ttUnexpectedEndOfLogTrap

Error

Premature end of log file reached during a database recovery. If your application connected with LogAutoTruncate=1 (the default), this trap represents a warning, recovery continues with error messages. If your application connected with LogAutoTruncate=0, recovery fails with error messages.

TimesTen SNMP Trap Contents

Every TimesTen SNMP trap includes the following information:

  • The GMT timestamp for when the trap was generated

  • The process ID of the process that triggered the event recorded by the trap

  • The user ID of the user whose operation triggered the event recorded by the trap

  • The name of the TimesTen instance

  • The release version of the TimesTen instance

  • A trap-specific message

In addition, most traps provide additional information specific to the trap. For example, the ttRepAgentDiedTrap also provides the replication store ID. For a list of the variables for each trap, see the TimesTen-MIB.txt file.

Example 3-1 A TimesTen SNMP Trap

A typical TimesTen SNMP trap may supply the following information:

TimesTen::ttDSTraps
Enterprise Specific Trap (TimesTen::ttDSGoingInvalidTrap) Uptime: 0:0:00:00.00
TimesTen::ttTimeStamp "2021-02-04 17:21:20 (GMT)"
TimesTen::ttPid 128974
TimesTen::ttUid "4121"
TimesTen::ttVersion "@(#)TimesTen: Release: 22.1.1.21.0 Date: 2021-02-04T16:28:39Z, instance instance1"
TimesTen::ttMesg "Data store marked invalid"
TimesTen::ttDSName "database1"
TimesTen::ttDSShmKey "0x210738f"
TimesTen::ttDSNConn 14

This specific trap, which was generated from a TimesTen daemon, is for the invalid database event. So additionally, it reports the database name, shared memory key of the database and the number of current connections to the database.