Configuring Traps

The following sections describe configuring traps via NET-SNMP.

Configuring Traps for Recommended OIDs

To configure sending traps for recommended OIDs, you must install the NET-SNMP utilities package.

$ sudo yum install net-snmp-utils

SNMPv2 Traps

For SNMPv2 traps to be sent from the ISR host, edit the "/etc/snmp/snmpd.local.conf" file and add the following configuration settings for the default NET-SNMP network interface and host resource notifications. Then restart the snmpd service.

# default NET-SNMP trap configs
# receiving IP address
trap2sink <NMS_IP_address> 
# SNMPv3 user for internal queries
iquerySecName isrsnmp
agentSecName isrsnmp
# enable default monitoring
defaultMonitors yes
linkUpDownNotifications yes

Note:

For more information on the default network and resource coverage with NET-SNMP, execute the man snmpd.conf command and read "DisMan Event MIB".

SNMPv3 Traps

SNMPv3 traps require a slight change in the /etc/snmp/snmpd.local.conf file only and the remaining configuration values remain unchanged.

Note:

When you enable SNMPv3 traps, you must restart the snmpd service.

# default NET-SNMP trap configs
# receiving IP address
trapsess -l authPriv -u isrsnmp -a MD5 -A <authPassword> -x DES -X <PrivPassword><NMS_IP_address>
# SNMPv3 user for internal queries
iquerySecName isrsnmp
agentSecName isrsnmp
# enable default monitoring
defaultMonitors yes
linkUpDownNotifications yes

Monitoring MySQL Database Replication On the Index Hosts

The following sections offer examples to configure NET-SNMP for monitoring the MySQL Server error log and sending traps for certain slave database events. Specifically, slave IO errors such as a lost connection with master and slave SQL errors from failed statements are rolled into a trap with a "MySQL replication error" description along with the "/var/log/mysqld.log" file that posted the error.

Configuring NET-SNMP For Monitoring MySQL Server Error Logs

Add the following lines to the "snmpd.local.conf file on the Index hosts:

logmatch mysqlReplicationNoSlave /var/log/mysqld.log 60 Error reading relay log event for channel
logmatch mysqlReplicationNoServer /var/log/mysqld.log 60 \[ERROR\] Slave I\/O for channel
logmatch mysqlReplicationSqlError /var/log/mysqld.log 60 \[ERROR\] Slave SQL for channel
monitor -r 60 -o logMatchFileName "MySQL replication error" != logMatchCurrentCount -u isrsnmp

Restart the snmpd process.

$ systemctl restart snmpd

Configuring SNMPD to Read MySQL Server Error Log

By default, SELinux does not allow NET-SNMP's "snmpd" process to read the MySQL Server default "/var/log/mysqld.log" file. There are some tools available with snmpd to add the proper configuration, specifically the "audit2allow" command-line tool. This tool adds an SELinux "allow" rule based on the logged event for the "snmpd" process being refused "read" and "open" access.

1. Confirm the following log is entered in the "/var/log/audit.log" log files of each Index host:

   type=AVC msg=audit(1502798331.404:75558): avc: denied { read } for pid=6060 comm="snmpd" name="mysqld.log" dev="dm-0" ino=17655111 scontext=system_u:system_r:snmpd_t:s0 tcontext=system_u:object_r:mysqld_log_t:s0 tclass=file

2. Execute the following commands on each Index host:

   sudo grep mysqld_log_t /var/log/audit/audit.log | audit2allow -M mymysqldlog
   sudo semodule -i mymysqldlog.pp

3. Restart "snmpd".

   $ systemctl restart snmpd

4. Confirm the following new log entry has been entered in the "audit.log" files:

   type=AVC msg=audit(1502800379.221:75814): avc: denied { open } for pid=6134 comm="snmpd" path="/var/log/mysqld.log" dev="dm-0" ino=17655111 scontext=system_u:system_r:snmpd_t:s0 tcontext=system_u:object_r:mysqld_log_t:s0 tclass=file

5. Execute the following commands on each Index host:

   $ sudo grep mysqld_log_t /var/log/audit/audit.log | audit2allow -M mymysqldlogopen
   $ sudo semodule -i mymysqldlogopen.pp

6. Confirm traps are received during a (very invasive) stop of one of the MySQL Servers:

   $ systemctl stop mysqld

   Note:

   Stopping the MySQL Server service brings down the database used by ISR applications during this testing window.
7. Confirm the trap is received. You may also issue an snmpwalk command from a Linux host with the "net-snmp-utils RPM package. Following the above SNMP example user and password, execute the following command:

   $ snmpwalk -v3 -u isrsnmp -A n3wf0und -a MD5 -l AuthnoPriv <Index_IP> logMatchTable

   to highlight certain lines from the result:

   ...
   UCD-SNMP-MIB::logMatchFilename.1 = STRING: /var/log/mysqld.log
   ...
   UCD-SNMP-MIB::logMatchRegEx.1 = STRING: Error reading relay log event for channel - first string to match
   UCD-SNMP-MIB::logMatchRegEx.2 = STRING: \\[ERROR\\] Slave I\\/O for channel
   UCD-SNMP-MIB::logMatchRegEx.3 = STRING: \\[ERROR\\] Slave SQL for channel
   UCD-SNMP-MIB::logMatchGlobalCounter.1 = Counter32: 6 - total matches
   ...
   UCD-SNMP-MIB::logMatchCurrentCounter.1 = Counter32: 5 - matches on current log file
   ...
   UCD-SNMP-MIB::logMatchCounter.1 = Counter32: 0 - matches on current read
   ...

   For more information, see the NET-SNMP online documentation.

Monitoring the ISR with Recording Capacity Exceeded

To configure monitoring when the ISR's recording capacity has been exceeded, add the following lines to the "snmpd.local.conf" file on the RSS hosts:

logmatch recorderCapacityExceeded /opt/isr/logs/recorder/recorder.log 120 no channels available to take this call
logmatch systemCapacityExceeded /opt/isr/logs/recorder/recorder.log 120 no channels available to take this call
monitor -r 60 -o logMatchName -o logMatchFileName "RSS application error" != logMatchCurrentCount -u isrsnmp

Note:

Add the last line specifically to send a trap when the ISR exceeds capacity, or the Recorder and Converter processes have restarted. For more information, see Recommended ISR RSS SNMP Configuration.