Syslog

The SBC can be configured to send system event logs to logging servers [1]. It is recommended to configure as few logging servers as required to reduce impact on SBC performance. Monitoring via SNMP is the preferred option over using syslog. The syslog messages aren’t as efficient since they may contain many extraneous informational messages that need to be filtered out or parsed. SNMP on the other hand has the advantage of sending clearly defined trap notifications only in the event of a problem, and the system-config and trap-receiver settings can be configured to filter on specific SNMP traps to send.

If a syslog parser is used to escalate SBC issues, it is easy to classify syslog events preceded with a MAJOR or CRITICAL designation as issues that require further investigation. However, be cautious of writing any parsing rules for events that are classified as GENERAL, REDUNDANCY, CONFIG WARNING, ERROR, or MINOR (among others). Some of these may be important to escalate, but others may be strictly informational in nature.

The table below represents a sample of some of the common syslog messages that may be seen. Note that IDS_LOG examples given require the IDS Reporting Feature Group license discussed in Appendix F. Some of the examples may seem redundant. This is because in some cases more than one message may be written to syslog as a result of an event.

A failed login attempt was detected on the console port.

May  3 17:06:10 172.41.3.90 CSE-4500-20 acliConsole[31ac9b6c] AUTH authenticate_locally: Authentication failed for user user
OR
Mar 20 10:27:24.119 acliConsole@: AUTH[53] authenticate_locally: Authentication failed for user user
OR
May  3 17:06:10 172.41.3.90 CSE-4500-20 acliConsole[31ac9b6c] WARNING login authentication failure from acliConsole

A failed login attempt was detected over SSH or FTP. See the process information for further details.

Dec 18 13:39:44.121 acliTelnet0@SBC1: WARNING authentication failure for admin from acliTelnet0

An endpoint exceeded a defined constraint and was blacklisted. This is the result of DoS configuration with the IDS license.

Nov 28 17:53:47 172.41.3.41 CSE-4500-6 sipd[2dcc32a4] ERROR [IDS_LOG] SigAddr[access:192.168.101.120:0=low:DENY] ttl=86400 exp=30 Demoted to Black-List (Too many messages) last msg rcvd=REGISTER sip:192.168.66.2 SIP/2.0
Nov 28 17:53:47 172.41.3.41 CSE-4500-6 sipd[2dcc32a4] ERROR Via: SIP/2.0/UDP 192.168.190.144:20928;branch=z9hG4bKdeadb33f
Nov 28 17:53:47 172.41.3.41 CSE-4500-6 sipd[2dcc32a4] ERROR From: hacker <sip:47097@192.168.190.144:20928>
Nov 28 17:53:47 172.41.3.41 CSE-4500-6 sipd[2dcc32a4] ERROR To: <sip:47097@192.168.66.2:5060>
Nov 28 17:53:47 172.41.3.41 CSE-4500-6 sipd[2dcc32a4] ERROR Call-ID: f9844fbe7dec140ca36500a0c9119870@192.168.66.2
Nov 28 17:53:47 172.41.3.41 CSE-4500-6 sipd[2dcc32a4] ERROR CSeq: 1 REGISTER
Nov 28 17:53:47 172.41.3.41 CSE-4500-6 sipd[2dcc32a4] ERROR Contact: <sip:47097@192.168.190.144>
Nov 28 17:53:47 172.41.3.41 CSE-4500-6 sipd[2dcc32a4] ERROR User-agent: Flooder_script
Nov 28 17:53:47 172.41.3.41 CSE-4500-6 sipd[2dcc32a4] ERROR Max-Forwards: 5
Nov 28 17:53:47 172.41.3.41 CSE-4500-6 sipd[2dcc32a4] ERROR Content-Length: 0

An endpoint exceeded a defined constraint and was blacklisted. This message is a result of DoS configuration without the IDS license.

Jan 15 16:29:46.289 sipd@SBC1: FLOW[15] SigAddr[Access:192.168.135.29:0=low:DENY] ttl=86400 guard=50 exp=30 Demoted to Black-List; send SNMP trap

An endpoint exceeded a defined constraint and was demoted from trusted to untrusted.

Apr  1 11:36:53.377 sipd@CSE-4500-6: WARNING SigAddr[access:172.41.0.3:5060=medium:PERMIT] ttl=64 exp=57 Demoted to Grey-List (errors)

The sipShield SPL plug-in (v1.3) detected a message from a known SIP scanner and dropped it

Mar 28 15:05:42.500 sipd@CSE-4500-6: WARNING Scanner or attack field detected! Src IP: 172.41.0.3, User-Agent: smap 0.6.0
OR
Mar 28 15:05:42.500 sipd@CSE-4500-6: WARNING Scanner or attack field detected! Src IP: 172.41.0.3, To: [e-mail]
OR
Mar 28 15:05:42.500 sipd@CSE-4500-6: WARNING Scanner or attack field detected! Src IP: 172.41.0.3, From: [e-mail]
OR
Mar 28 15:05:42.500 sipd@CSE-4500-6: WARNING Scanner or attack field detected! Src IP: 172.41.0.3, Subject: SiVuS

A message was rejected by the SD. The status code and reason given in parenthesis will change based on the type of malformation. Examples given here include:

An INVITE received from a forbidden endpoint. In this case, allow-anonymous on the SIP interface was set to agents-only, and the INVITE was not from an agent.

An INVITE had a Max-Forwards parameter that had decremented to zero, and the SBC could not forward it further

Four examples of malformed messages that were generated from a Protos attack (too large, missing header, bad request URI, unsupported URI).

Apr  1 11:26:27.603 sipd@CSE-4500-6: IDS[64] [IDS_LOG]INVITE from source 172.41.0.3:5060 to dest 172.41.0.2:5060[UDP] realm=access; From=sipp <sip:sipp@127.0.1.1:5060>;tag=10387SIPpTag001; target=sip:service@172.41.0.2:5060 rejected!; status=403 (Forbidden)
OR
Nov 28 19:52:40 172.41.3.41 CSE-4500-6 sipd[2dcc32a4] ERROR [IDS_LOG]INVITE from source 192.168.66.54:5060 to dest 192.168.66.2:5060[UDP] realm=access; From="hacker"<sip:666@192.168.66.54:30000>; target=sip:9195551212@192.168.66.2 rejected!; status=483 (Too Many Hops); error=invalid message
OR
IDS_LOG]INVITE from source 192.168.222.1:5060 to dest 192.168.222.50:5060[UDP] realm=access; From=227 <sip:evil@127.0.1.1>;tag=227; target=sip <omitted message> rejected!; status=513 (Message Too Big)
OR
May 22 14:40:39.033 sipd@: IDS[64] [IDS_LOG]INVITE from source 192.168.222.1:5060 to dest 192.168.222.50:5060[UDP] realm=access; From=389 <sip:evil@127.0.1.1>;tag=389; target=sip:1111@192.168.222.50 rejected!; status=400 (Invalid/Missing Via Header)
OR
May 22 15:08:02.015 sipd@: IDS[64] [IDS_LOG]INVITE from source 192.168.222.1:5060 to dest 192.168.222.50:5060[UDP] realm=access; From=206 <sip:evil@127.0.1.1>;tag=206; target=%s%s%s%s%s:[e-mail] rejected!; status=400 (Bad Request-URI)
OR
May 22 15:08:01.088 sipd@: IDS[64] [IDS_LOG]INVITE from source 192.168.222.1:5060 to dest 192.168.222.50:5060[UDP] realm=access; From=197 <sip:evil@127.0.1.1>;tag=197; target=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:[e-mail] rejected!; status=416 (Unsupported URI Scheme)

A user entered enable mode (administrator level). This is not necessarily an issue, but may be an interesting event.

May  3 17:06:37 172.41.3.90 CSE-4500-20 acliConsole[31ac9b6c]  raised privileges on session from acliConsole

A user enabled SIP debugging traces. This can use large amounts of CPU if run on a production network or potentially reveal sensitive information. This is not necessarily an issue, but may be an interesting event.

May  3 17:09:26 172.41.3.90 CSE-4500-20 sipd[2fa7cc00] SIP enable SIP Debugging

The configuration file was updated. This should be investigated if changes were not authorized.

Dec 19 13:28:27.060 lemd@SBC1: CONFIG[32] Save Config has completed successfully

A new configuration was activated. This should be investigated if changes were not authorized.

Dec 19 13:28:29.863 lemd@SBC1: CONFIG[32] Configuration successfully activated
OR
Dec 19 13:28:31.864 lemd@SBC1: CONFIG[32] Activate Config Successfully Complete
OR
Mar 20 10:11:02.919 acliSSH0@: CONFIG[34] ACTIVATE-CONFIG done

One or more licenses has expired and unit functionality may be impacted

Apr  1 00:00:10.523 brokerd@CSE-4500-6: MINOR ALARM[00050016] Task[0615c064] 1 license has expired! 

One or more licenses is nearing expiration

Mar 31 00:00:10.521 sysmand@CSE-4500-6: MINOR License will expire in less than 7 days.

The number of sessions is approaching licensed capacity

Jan  1 00:02:57.480 brokerd@SBC1: MAJOR ALARM[00050004] Task[0cf72188] total number of sessions (1977) is approaching licensed capacity (2000)

The unit was powered on. This may be an indication that a power failure occurred.

Jan  8 11:33:06.545 bootstrap@SBC1: GENERAL[0] Bringing up box...

The SIP protocol stack is now active. This may be an indication that a power failure occurred or that the SIP process crashed and restarted.

May  3 17:30:08 172.41.3.90 CSE-4500-20 sipd[2fa7cc00] SIP Change to In-Service state and Start accepting messages...

Unit CPU usage has reached a critical threshold

Oct  8 19:02:02.381 brokerd@SBC1: CRITICAL ALARM[0002001b] Task[0578324c] cpu usage 93 percent is over critical threshold of 90 percent.

Unit CPU usage has reached a major threshold

Oct  8 19:02:12.708 brokerd@SBC1: MAJOR ALARM[0002001b] Task[0578324c] cpu usage 87 percent is over major threshold of 80 percent.

Unit CPU usage has reached a minor threshold

Oct  8 19:06:57.062 brokerd@SBC1: MINOR ALARM[0002001b] Task[0578324c] cpu usage 74 percent is over minor threshold of 70 percent.

A high-availability switchover was detected from the active unit. If this was not an administrative failover then it is likely that a port or process failed

Dec  3 17:30:46.275 berpd@SBC1: CRITICAL ALARM[00020021] Task[2834f658] Switchover, Active to RelinquishingActive
		

The standby unit has become the active unit. If this was not a result of an administrative action then a port or process on the active unit likely failed.

Jan  8 11:34:41.652 berpd@SBC1: CRITICAL ALARM[00020020] Task[03c3a840] Switchover, Standby to BecomingActive, active peer SBC2 has timed out
		

The standby unit is having difficulty reaching the active unit. Verify that all wancom ports are operational.

Dec  3 17:33:46.384 berpd@SBC1: CRITICAL ALARM[00020023] Task[2834f658] Unable to synchronize with Active redundant peer within BecomingStandby timeout, going OutOfService

A ethernet port used for management has gone down

Jan  8 11:34:42.171 brokerd@SBC1: MAJOR ALARM[00020009] Task[0e723a98] wancom1 link down

A ethernet port used for management has recovered from failure

Jan  8 11:34:44.788 brokerd@SBC1: MINOR ALARM[00020006] Task[0e723a98] wancom1 link up

An ethernet port used for services has gone down. Note that slot and port numbers will vary.

Mar 20 21:56:29.504 brokerd@: MAJOR ALARM[00020027] Task[00000003] Slot 1 Port 0 DOWN

All servers that can receive accounting files (CDR) are not available

May  3 17:20:11 172.41.3.90 CSE-4500-20 brokerd[10661b38] CRITICAL All of collector's push receivers are down

Transfer of an HDR file failed because the key used for authentication is incorrect

May  3 17:20:11 172.41.3.90 CSE-4500-20 collect[2eb37454] WARNING Error: HDR push failed due to bad host key.

An error occurred when attempting to transfer accounting logs

Dec 31 07:47:53.192 collect@SBC1: MINOR Error pushing collected data to 172.17.5.24 for group: system

Transfer of an HDR file failed due to invalid authentication

May  3 17:20:11 172.41.3.90 CSE-4500-20 collect[2eb37454] ERROR Error: Could not login to host '172.41.1.118
		

Media port usage is exceeding capacity. Calls may fail or experience audio issues. The severity is based on the percentage of failures to allocate a steering port. Jan 17 12:14:26.513 mbcd@SBC1: MINOR ALARM[00040006] Task[1b963548] out of steering ports for realm 'CORE'; 296 of 592 failed (50%)

OR
Jan 17 12:18:14.865 mbcd@SBC1: WARNING ALARM[00040006] Task[1b963548] out of steering ports for realm 'CORE'; 80 of 310 failed (25%)

A session agent (SIP server) has failed a health check and has been taken out of service

Jan 15 16:28:19.901 sipd@SBC1: SIP[13] SA 192.168.136.69[PBX1]PING TRANSACTION TIMEOUT to 192.168.136.69
Jan 15 16:28:19.902 sipd@SBC1: SIP[13]   was 'In Service'; set to 'Out of Service' status

A session agent (SIP server) has failed a health check and has been taken out of service

Jan 15 16:28:22.969 sipd@SBC1: SIP[13] SA 192.168.135.29[PBX2]Non-Ping TRANSACTION TIMEOUT to 192.168.135.29
Jan 15 16:28:22.970 sipd@SBC1: SIP[13]   was 'In Service'; set to 'Out of Service' status

There were no routes found for an incoming session. This may mean that the called destination is out of service, the destination address is incorrect, or that the routing table is not sufficient.

Mar 30 15:02:27.307 sipd@CSE-4500-6: IDS[64] [IDS_LOG]INVITE from source 192.168.60.10:5061 to dest 192.168.60.2:5060[UDP] realm=core; From=sipp <sip:sipp@127.0.0.1:5061>;tag=9165SIPpTag00143; target=sip:service@192.168.60.2:5060 rejected!; status=480 (No Routes Found)