Using SNMP Traps for Diagnostics

Simple Network Management Program (SNMP) is a protocol for network management services. Even though network management software typically uses SNMP to query or control the state of network devices like routers and switches, SNMP can be used to manage almost any resource, like applications and databases. These devices or resources sometimes also generate asynchronous alerts called traps to inform the management systems of problems. TimesTen is enabled to generate such traps.

The following topics describe SNMP and traps in TimesTen:

TimesTen and SNMP

TimesTen cannot be queried nor controlled through SNMP. TimesTen only sends SNMP traps for certain critical events, to possibly facilitate some user recovery mechanisms. TimesTen can send traps for various events, including:

Assertion failure
Death of daemons
Database invalid
Replicated transaction failure
Database out of space
Autorefresh transaction failure
Replication conflict resolution
File write errors

These events also cause log entries to be written by the TimesTen daemon, but exposing them through SNMP traps allows for the possibility of having some network management software take immediate action.

Topics include:

TimesTen MIB

A Management Information Base (MIB) is similar to a database schema. It describes the structure of the SNMP data.

The MIB extension file, installation_dir/network/admin/samples/TimesTen-MIB.txt, describes the structure of the TimesTen SNMP information.

The TimesTen OID is rooted at Private Enterprise 5549. The complete path to root is iso.org.dod.internet.private.enterprise.TimesTen.* or numerically, 1.3.6.1.4.1.5549.*.

Data Types in TimesTen SNMP Traps

The TimesTen MIB defines more than 60 variables that are either INTEGER or TEXT data type.

Table 2-8 describes the variables that are ASN_INTEGER type:

Table 2-8 ASN_INTEGER Variables in the TimesTen MIB

Variable	Description
`ttPid`	Process ID
`ttTrapTruncated`	1 if the trap is too large and has been truncated
`ttDSNConn`	Connection number, starting with 0
`ttDSMaxSize`	Maximum size of database, in KB
`ttDSCurSize`	Current size of database, in KB
`ttDSReqSize`	Requested increase of database size, in KB
`ttDSError`	Error number
`ttTxnId`	Transaction ID
`ttTxnCounter`	Transaction counter
`ttStmId`	Statement ID
`ttQueryThreshold`	Query threshold time, in seconds
`ttClientPid`	Process ID of the client that generated the query
`ttViewId`	View identifier
`ttFileErrorNum`	Error number when file access encounters an error
`ttDaePid`	Process ID of the daemon
`ttDaeInst`	Daemon number, starting with 0
`ttRepPid`	Process ID of replication agent
`ttRepMasterPort`	Port number of master
`ttRepSubscriberPort`	Port number of subscriber
`ttRepConflictCount`	Replication conflict counter
`ttCacheAgentPid`	Process ID of cache agent

The rest of the variables are ASN_OCTET_STRING type.

TimesTen SNMP Trap Names and Severity Levels

TimesTen SNMP traps can be categorized by severity level. The information in the trap can be of the type:

Error
Informational
Warning

Table 2-9 describes each trap and its severity level.

Table 2-9 TimesTen SNMP Trap Description and Severity Levels

Trap Name	Severity Level	Description
`ttAssertFailTrap`	Error	TimesTen Assertion Failure.
`ttAsyncMVFailed`	Warning	A refresh of Asynchronous materialized view failed. The SNMP trap includes dsname, daemon PID and viewid. If the error is due to a transient error, such as locking, the refresh may succeed in the next refresh.
`ttCacheAgentDiedTrap`	Error	The cache agent died.
`ttCacheAgentFailoverTrap`	Warning	The cache agent detected that a connection to the Oracle database had been lost and has begun to recover the connection.
`ttCacheAutoRefFailedTrap`	Error	TimesTen Cache incremental autorefresh failed.
`ttCacheAutoRefQueFullTrap`	Warning	TimesTen Cache incremental autorefresh queue is full.
`ttCacheAutorefreshDsMarkedDeadTrap`	Warning	TimesTen Cache incremental autorefresh failed. The cache agent for a remote datastore has stopped or is no longer responding. Autorefresh for the remote datastore has been disabled.
`ttCacheAutorefreshLogSpaceDeFragDetectedTrap`	Warning	Some of the autorefresh change log tables on the Oracle database are fragmented. See Fragmented Autorefresh Change Log Table Space in Oracle TimesTen In-Memory Database Monitoring and Troubleshooting Guide.
`ttCacheAwtRtReadFailedTrap`	Error	For asynchronous writethrough cache groups, runtime information is stored on the Oracle database instance. While reading this information from the Oracle database, replication either could not find the runtime data table (`tt_version_reppeers`) or could not find the information within the table.
`ttCacheAwtRtUpdateFailedTrap`	Error	For asynchronous writethrough cache groups, runtime information is stored on the Oracle database instance. While updating this information replication either could not find the runtime data table (`tt_version_reppeers`) or could not find the information within the table.
`ttCacheCgNotAutorefreshedTrap`	Warning	The cache group will not be autorefreshed. Instead, it must be manually recovered by performing manual load or refresh cache group.
`ttCacheLowOracleTblSpace`	Warning	The tablespace the cache admin user is using is below the minimum threshold.
`ttCacheRecoveryAutorefreshTrap`	Warning	The cache agent is performing a full autorefresh. This may be needed when a change log table on the Oracle database was truncated because of lack of tablespace for the cache administration user.
`ttCacheValidationAbortedTrap`	Error	The cache agent terminated cache group validation because of an irrecoverable error. Please refer to the user error log for details.
`ttCacheValidationErrorTrap`	Error	The cache agent has detected irrecoverable anomalies with cache group cache-group-name that will prevent it from properly refreshing the cache group, or it has detected irrecoverable anomalies within the refresh interval time-in-ms. Please refer to the user error log for details.
`ttCacheValidationWarningTrap`	Warning	The cache agent has detected anomalies with cache group cache-group-name that may prevent it from properly refreshing the cache group. Please refer to the user error log for details.
`ttDaemonOutOfMemoryTrap`	Error	Call to `malloc` failed in TimesTen daemon.
`ttDSCkptFailedTrap`	Error	A checkpoint has failed. Check the user error log and get view the checkpoint history using either the `SYS.GV$CKPT_HISTORY` or `SYS.V$CKPT_HISTORY` system views or the `ttCkptHistory` built-in procedure.
`ttDSDataCorruptionTrap`	Error	Database corruption error has occurred.
`ttDSGoingInvalidTrap`	Error	Setting database to invalid state. Database invalidation usually happens when an application that is connected to the database is terminated or exits abruptly without first disconnecting from the database. If TimesTen encounters an irrecoverable internal error during a database operation, it may also invalidate the database. You must commit or rollback and recover the database.
`ttDSThreadCreateFailedTrap`	Error	A process (typically multithreaded) having multiple connections to a database exits anomalously. The subdaemon assigned to clean up the connections creates a separate thread for each connection. If creation of one of these threads fails, this trap is thrown. Thread creation may fail due to memory limitations or having too many threads in the system. After the trap is thrown, the thread creation is attempted four more times, with an increasingly longer pause between each attempt. The total time between the first and last attempt is approximately 30 seconds. If the fifth attempt fails, the database is invalidated.
`ttFileWriteErrorTrap`	Error	Error encountered during file I/O write.
`ttMainDaemonDiedTrap`	Error	Main or sub daemons died anomalously. This message is sent by a subdaemon when it notices that the main daemon has died. It suggests that the main daemon has been terminated or has crashed. You must restart the main daemon.
`ttMainDaemonExitingTrap`	Informational	Main or sub daemons exiting ordinarily.
`ttMainDaemonReadyTrap`	Informational	Main daemon has started.
`ttMsgLogOpenFailedTrap`	Error	The message log could not be opened, possibly because of a lack of privileges on the file. Check the file location and privileges.
`ttPartitionSpaceExhaustedTrap`	Error	Database memory region (permanent or temporary) space is exhausted. This message is sent when either the permanent or temporary free space in the database is exhausted. Generally this message is preceded by the `ttPartitionSpaceStateTrap` warning message. See PermWarnThreshold and TempWarnThreshold in Oracle TimesTen In-Memory Database Reference.
`ttPartitionSpaceStateTrap`	Warning	Database memory region (permanent or temporary) space is transitioning from OK to low or vice versa. This message is sent when either the permanent memory region or the temporary memory region free space in the database reaches a threshold or transitions back below the threshold. This message is sent only when the free space has reached the threshold specified by the `PermWarnThrehold` or `TempWarnThreshold` attribute at the time of the first connection to the database. See PermWarnThreshold and TempWarnThreshold in Oracle TimesTen In-Memory Database Reference.
`ttQueryThresholdWarnTrap`	Warning	A SQL query exceeded the user-specified threshold. The text of the query can be found n the user log message. The Transaction ID and the Statement ID of the query can be found both in the trap and the user log message. After issuing the trap, the query continues running.
`ttRepAgentClockSkewTrap`	Error	Replication with a peer failed due to excessive clock skew. The skew between nodes in an active standby scheme has exceeded the allowed limit of 250ms.
`ttRepAgentDiedTrap`	Error	A replication agent has died anomalously. This message is sent when the main TimesTen daemon notices that a replication agent has died anomalously. This generally means that the replication agent has been terminated or has crashed.
`ttRepAgentExitingTrap`	Informational	Replication agent exiting ordinarily.
`ttRepAgentStartingTrap`	Informational	Replication agent starting.
`ttRepAgentStateChange`	Informational	Indicates the state of the replication agent. The possible states of the replication agent are `ACTIVE`, `FAILED`, `IDLE`, `RECOVERING`, `STANDBY`, and `UNKNOWN`. TimesTen only sends this trap when the state of the replication agent changes.
`ttRepCatchupStartTrap`	Warning	Indicates that TimesTen has begun to restore a master from a subscriber where bi-directional replication has been configured, after a failure.
`ttRepCatchupStopTrap`	Warning	Indicates that TimesTen has restored a master database from a subscriber, where bi-directional replication was configured.
`ttRepConflictReportStartingTrap`	Informational	Indicates that conflict reporting has been restarted because the rate of conflicts has fallen below the low water mark set in the replication scheme. This trap also indicates how many conflicts went unreported during the period in which reporting was suspended.
`ttRepConflictReportStoppingTrap`	Informational	Indicates that suspension of conflict reporting has occurred because the rate of conflicts has exceeded the high water mark set in the replication scheme.
`ttRepReturnTransitionTrap`	Warning	Replication return receipt has been enabled or disabled on the subscriber.
`ttRepSubscriberFailedTrap`	Error	Subscriber marked as failed because too much log accumulated on its behalf by the master.
`ttRepSubscriberTCPConnectFailedTrap`	Error	A replication TCP connection failed.
`ttRepUpdateFailedTrap`	Warning	A replication insert, update or delete operation failed.
`ttUnexpectedEndOfLogTrap`	Error	Premature end of log file reached during a database recovery. If your application connected with `LogAutoTruncate=1` (the default), this trap represents a warning, recovery continues with error messages. If your application connected with `LogAutoTruncate=0`, recovery fails with error messages.

TimesTen SNMP Trap Contents

Every TimesTen SNMP trap includes the following information:

The GMT timestamp for when the trap was generated
The process ID of the process that triggered the event recorded by the trap
The user ID of the user whose operation triggered the event recorded by the trap
The name of the TimesTen instance
The release version of the TimesTen instance
A trap-specific message

In addition, most traps provide additional information specific to the trap. For example, the ttRepAgentDiedTrap also provides the replication store ID. For a list of the variables for each trap, see the TimesTen-MIB.txt file.

Example 2-1 A TimesTen SNMP Trap

A typical TimesTen SNMP trap may supply the following information:

TimesTen::ttDSTraps
Enterprise Specific Trap (TimesTen::ttDSGoingInvalidTrap) Uptime: 0:0:00:00.00
TimesTen::ttTimeStamp "2021-02-04 17:21:20 (GMT)"
TimesTen::ttPid 128974
TimesTen::ttUid "4121"
TimesTen::ttVersion "@(#)TimesTen: Release: 26.1.1.1.0 Date: 2021-02-04T16:28:39Z, instance instance1"
TimesTen::ttMesg "Data store marked invalid"
TimesTen::ttDSName "database1"
TimesTen::ttDSShmKey "0x210738f"
TimesTen::ttDSNConn 14

This specific trap, which was generated from a TimesTen daemon, is for the invalid database event. So additionally, it reports the database name, shared memory key of the database and the number of current connections to the database.

How TimesTen Sends SNMP Traps

SNMP traps can use either UDP/IP or TCP/IP as communication protocol. TimesTen supports SNMP version 1, 2, and 3 traps.

Topics include:

Generating and Receiving TimesTen SNMP Traps

TimesTen does not send SNMP traps by itself. To generate and receive SNMP traps, you need an installation of Net-SNMP, including the snmptrap utility, and to enable SNMP trap generation in the TimesTen instance configuration file, timesten.conf.

Note:

TimesTen 18.1 (or earlier) used to have its own embedded implementation for SNMP traps generation (SNMPv1 only), which required the TimesTen SNMP configuration file, snmp.ini, to enable them. TimesTen 22.1 (or later) uses a different implementation and no longer supports the snmp.ini file. However, the events triggering a trap and the contents of the trap have not changed from previous releases.

On Linux platforms, consider using SNMPv3 over (D)TLS (DTLS/UDP) for encrypted and authenticated SNMP traps. SNMPv3 over (D)TLS has these prerequisites:

Net-SNMP version 5.9 (or later) with the Transport Security Model (TSM) enabled
The snmptrap utility
OpenSSL version 1.1.1 (or later)

Note:

Neither the Net-SNMP packages nor the OpenSSL library is included with TimesTen. If the appropriate net-snmp package is available in the repository for your operating system, you must also acquire and install the net-snmp-utils package for the snmptrap utility.

Alternatively, the Net-SNMP agent, library, and tools can be downloaded from SourceForge: https://sourceforge.net/projects/net-snmp/.
(D)TLS uses X.509 certificates to authenticate client/server connections. Both the SNMP server and client need to have such certificates to use (D)TLS. In this case, the system receiving the SNMP traps and the one hosting the TimesTen instance and the snmptrap utility must have properly configured X.509 certificates.

For other supported platforms, such as Solaris or AIX, consider SNMPv3 over TLS (TLS/TCP) instead.

SNMP traps are only useful if you have a SNMP monitoring tool to receive and log SNMP trap messages. If you do not have a SNMP monitoring tool available, Net-SNMP includes the snmptrapd utility for this purpose.

For information on the Net-SNMP project, see http://www.net-snmp.org/.

Configuring the timesten.conf File

The TimesTen instance configuration file, timesten_home/conf/timesten.conf on UNIX systems, uses several attributes to enable SNMP trap generation.

Attribute	Description
`snmp_conf_path`	Colon-separated list of locations to search for the Net-SNMP configuration files, `snmp.conf` and `snmp.local.conf`.
`snmp_trap`	The location of the utility used to send SNMP traps, `snmptrap`.
`snmp_trap_dests`	Space-separated list of destinations for the SNMP traps.
`snmp_trap_opts`	List of options to pass to the `snmptrap` utility.
`snmp_version`	The SNMP version to use.
`openssl_path`	The location of the OpenSSL library used for encrypted SNMP traps.

Note:

The snmp.conf and snmp.local.conf files are the configuration files for Net-SNMP utilities. TimesTen needs the snmptrap utility to be able to read either of these files to send SNMP traps.
Only set the openssl_path attribute if you need TimesTen to use a different OpenSSL library from the one used by the system by default, or if the snmptrap utility specified in the snmp_trap attribute requires a specific OpenSSL library.

For more information on the timesten.conf file, see TimesTen Instance Configuration File in Oracle TimesTen In-Memory Database Reference.

Example 2-2 Enabling SNMPv3 over (D)TLS traps

This example provides snippets of the timesten.conf and snmp.conf files, which showcase samples of the attributes or directives required for TimesTen to generate SNMPv3 over (D)TLS traps.

To enable SNMPv3 over (D)TLS traps, ensure that the timesten.conf file in your TimesTen instance includes the attributes shown next based on your own environment:

#SNMP traps settings
snmp_conf_path=/usr/local/etc/snmp
snmp_trap=/usr/local/bin/snmptrap
snmp_trap_dests=dtlsudp:localhost:9161 dtlsudp:myhost:10161
snmp_version=3

Also, ensure that you configure the snmp.conf file for SNMPv3 over (D)TLS traps. The file should include at least the following settings:

defCommunity public
defSecurityModel tsm
defVersion 3
mibdirs /usr/local/share/snmp/mibs
mibs +TimesTen-MIB

Note:

Ensure that the TimesTen MIB is added to a location used to find and load MIBs. In this example, the /usr/local/share/snmp/mibs directory is one such location. See TimesTen MIB for where to find the TimesTen MIB.

For more information on the SNMP-related attributes for the timesten.conf file, see TimesTen Instance Configuration File in Oracle TimesTen In-Memory Database Reference.

For more information on the directives for the snmp.conf file, see the man page for snmp.conf.

Example 2-3 Enabling SNMPv1 traps

This example provides snippets of the timesten.conf and snmp.conf files, which showcase samples of the attributes or directives required for TimesTen to generate SNMPv1 traps.

WARNING:

SNMPv1 does not support encryption or authentication. It is highly recommended that you use SNMPv3 over (D)TLS or TLS for TimesTen SNMP traps.

To enable SNMPv1 traps, ensure that the timesten.conf file in your TimesTen instance includes the attributes shown next based on your own environment:

#SNMP traps settings
snmp_conf_path=/usr/local/etc/snmp
snmp_trap=/usr/local/bin/snmptrap
snmp_trap_dests=localhost:9161 myhost:10161
snmp_version=1

Also, ensure that you configure the snmp.conf file for SNMPv1 traps. The file should include at least the following settings:

defCommunity public
defVersion 1
mibdirs /usr/local/share/snmp/mibs
mibs +TimesTen-MIB

Note:

For more information on the SNMP-related attributes for the timesten.conf file, see TimesTen Instance Configuration File in Oracle TimesTen In-Memory Database Reference.

For more information on the directives for the snmp.conf file, see the man page for snmp.conf.

Trap Truncation on Overflow

The maximum packet size of a single trap is 1024 bytes. If there is more data than can fit into the 1024 byte limit, the trap is truncated to fit. In this case, the trap contains a ttTrapTruncated OID set to 1.

Trapping Out-of-Space Messages

By default, TimesTen records that database space is low based on the region space thresholds of PermWarnThreshold and TempWarnThreshold attributes. For example, if the total memory in use for the permanent memory region surpasses the value set for the PermWarnThreshold attribute, TimesTen records a message indicating that the permanent memory region is out of space. Once the used space for the permanent memory region is once again 10% below the threshold, TimesTen records a second message indicating that the permanent memory region is no longer low on space.

When connecting to a database, you can change the out-of-space threshold by setting the PermWarnThreshold and TempWarnThreshold attributes. See PermWarnThreshold and TempWarnThreshold in Oracle TimesTen In-Memory Database Reference.