15 Configuring SNMP

This chapter describes how to configure and manage SNMP services with Oracle Communications Converged Application Server:

• Overview of Converged Application Server SNMP

• Browsing the MIB

• Configuring SNMP

• Understanding and Responding to SNMP Traps

Overview of Converged Application Server SNMP

Converged Application Server includes a dedicated SNMP MIB to monitor activity on engine tier and SIP data tier server instances. The Converged Application Server MIB is available on both Managed Servers and the Administration Server of a domain. However, Converged Application Server engine and SIP data tier traps are generated only by the Managed Server instances that make up each tier. If your Administration Server is not a target for the sipserver custom resource, it will generate only WebLogic Server SNMP traps (for example, when a server in a cluster fails). Administrators should monitor both WebLogic Server and Converged Application Server traps to evaluate the behavior of the entire domain.

Note:

Converged Application Server MIB objects are read-only. You cannot modify a Converged Application Server configuration using SNMP.

Browsing the MIB

The Converged Application Server MIB file is installed in WLSS_HOME/server/lib/WLSS-MIB.asn1. Use an available SNMP management tool or MIB browser to view the contents of this file. See also "Trap Descriptions" for a description of common SNMP traps.

Configuring SNMP

To enable SNMP monitoring for the entire Converged Application Server domain, follow these steps:

1. Login to the Administration Console for the Converged Application Server domain.

2. In the left pane, select the Diagnostics > SNMP node.

3. In the Server SNMP Agents table, click the New button to create a new agent.

   Note:

   Ensure that you create a new Server SNMP agent, rather than a Domain-Scoped agent.

4. Enter a unique name for the new SNMP agent (for example, "engine1snmp") and click OK.

5. Select the newly-created SNMP agent from the Server SNMP Agents table.

6. On the Configuration > General tab:

   1. Select the Enabled check box to enable the agent.

   2. Enter an unused port number in the SNMP UDP Port field.

      Note:

      If you run multiple Managed Server instances on the same machine, each server instance must use a dedicated SNMP agent with a unique SNMP port number.

   3. Click Save.

7. Repeat the above steps to generate a unique SNMP agent for each server in your deployment (SIP data tier server, engine tier server, and Administration Server).

Understanding and Responding to SNMP Traps

The following sections describe the Converged Application Server SNMP traps in more detail. Recovery procedures for responding to individual traps are also included where applicable.

Files for Troubleshooting

The following Converged Application Server log and configuration files are frequently helpful for troubleshooting problems, and may be required by your technical support contact:

• $DOMAIN_HOME/config/config.xml

• $DOMAIN_HOME/config/custom/sipserver.xml

• $DOMAIN_HOME/servername/*.log (server and message logs)

• sip.xml (in the /WEB-INF subdirectory of the application)

• web.xml (in the /WEB-INF subdirectory of the application)

General information that can help the technical support team includes:

• The specific versions of:

  • Converged Application Server

  • Java SDK

  • Operating System

• Thread dumps for hung Converged Application Server processes

• Network analyzer logs

Trap Descriptions

Table 15-1 lists the Converged Application Server SNMP traps and indicates whether the trap is generated by servers in the engine tier or SIP data tier. Each trap is described in the sections that follow.

Table 15-1 Converged Application Server SNMP Traps

Server Node in which Trap is Generated	Trap Name
Engine Tier Servers	connectionLostToPeer
Engine Tier Servers	connectionReestablishedToPeer
Engine Tier Servers	overloadControlActivated, overloadControlDeactivated
Engine Tier Servers	sipAppDeployed
Engine Tier Servers	sipAppUndeployed
Engine Tier Servers	sipAppFailedToDeploy
Engine and SIP Data Tier Servers, if servers are members of a cluster	serverStopped
SIP Data Tier Servers	dataTierServerStopped
SIP Data Tier Servers	replicaAddedToPartition
SIP Data Tier Servers	replicaRemovedEnginesRegistration
SIP Data Tier Servers	replicaRemovedFromPartition

connectionLostToPeer

This trap is generated by an engine tier server instance when it loses its connection to a replica in the SIP data tier. It may indicate a network connection problem between the engine and SIP data tiers, or may be generated with additional traps if a SIP data tier server fails.

Recovery Procedure

If this trap occurs in isolation from other traps indicating a server failure, it generally indicates a network failure. Verify or repair the network connection between the affected engine tier server and the SIP data tier server.

If the trap is accompanied by additional traps indicating a SIP data tier server failure (for example, dataTierServerStopped), follow the recovery procedures for the associated traps.

connectionReestablishedToPeer

This trap is generated by an engine tier server instance when it successfully reconnects to a SIP data tier server after a prior failure (after a connectionLostToPeer trap was generated). Repeated instances of this trap may indicate an intermittent network failure between the engine and SIP data tiers.

Recovery Procedure

See "connectionLostToPeer".

dataTierServerStopped

Converged Application Server SIP data tier nodes generate this alarm when an unrecoverable error occurs in a WebLogic Server instance that is part of the SIP data tier. Note that this trap may be generated by the server that is shutting down, by another replica in the same partition, or in some cases by both servers (network outages can sometimes trigger both servers to generate the same trap).

Recovery Procedure

See the Recovery Procedure for "serverStopped".

overloadControlActivated, overloadControlDeactivated

Converged Application Server engine tier nodes use a configurable throttling mechanism that helps you control the number of new SIP requests that are processed. After a configured overload condition is observed, Converged Application Server destroys new SIP requests by responding with "503 Service Unavailable" to the caller. The servers continues to destroy new requests until the overload condition is resolved according to a configured threshold control value. This alarm is generated when the throttling mechanism is activated. The throttling behavior should eventually return the server to a non-overloaded state, and further action may be unnecessary. See "overload" in Chapter 30, "Engine Tier Configuration Reference (sipserver.xml)" for more information.

Recovery Procedure

Follow this recovery procedure:

1. Check other servers to see if they are nearly overloaded.

2. Check to see if the load balancer is correctly balancing load across the application servers, or if it is overloading one or more servers. If additional servers are nearly overloaded, Notify Tier 4 support immediately.

3. If the issue is limited to one server, notify Tier 4 support within one hour.

Additional Overload Information

If you set the queue length as an incoming call overload control, you can monitor the length of the queue using the Administration Console. If you specify a session rate control, you cannot monitor the session rate using the Administration Console. (The Administration Console only displays the current number of SIP sessions, not the rate of new sessions generated.)

replicaAddedToPartition

Converged Application Server SIP data tier nodes generate this alarm when a server instance is added to a partition in the SIP data tier.

Recovery Procedure

This trap is generated during normal startup procedures when SIP data tier servers are booted.

replicaRemovedEnginesRegistration

SIP data tier nodes generate this alarm if an engine server client that was not registered (or was removed from the list of registered engines) attempts to communicate with the SIP data tier. This trap is generally followed by a serverStopped trap indicating that the engine tier server was shut down to preserve SIP data tier consistency.

Recovery Procedure

Restart the engine tier server. Repeated occurrences of this trap may indicate a network problem between the engine tier server and one or more replicas.

replicaRemovedFromPartition

Converged Application Server SIP data tier nodes generate this alarm when a server is removed from the SIP data tier, either as a result of a normal shutdown operation or because of a failure. There must be at least one replica remaining in a partition to generate this trap; if a partition has only a single replica and that replica fails, the trap cannot be generated. In addition, because engine tier nodes determine when a replica has failed, an engine tier node must be running in order for this trap to be generated.

Recovery Procedure

If this trap is generated as a result of a server instance failure, additional traps will be generated to indicate the exception. See the recovery procedures for traps generated in addition to replicaRemovedFromPartition.

serverStopped

This trap indicates that the WebLogic Server instance is now down. This trap applies to both engine tier and SIP data tier server instances, but only when the servers are members of a named WebLogic Server cluster. If this trap is received spontaneously and not as a result of a controlled shutdown, follow the steps below.

Recovery Procedure

Follow this recovery procedure:

1. Use the following command to identify the hung process:

   ps –ef | grep java
   

   There should be only one PID for each WebLogic Server instance running on the machine.

2. After identifying the affected PID, use the following command to kill the process:

   kill -3 [pid]
   

3. This command generates the actual thread dump. If the process is not immediately killed, repeat the command several times, spaced 5-10 seconds apart, to help diagnose potential deadlock problems, until the process is killed.

4. Attempt to restart Converged Application Server immediately.

5. Make a backup copy of all SIP logs on the affected server to aid in troubleshooting. The location of the logs varies based on the server configuration.

6. Copy each log to assist Tier 4 support with troubleshooting the problem.

   Note:

   Converged Application Server logs are truncated according to your system configuration. Make backup logs immediately to avoid losing critical troubleshooting information.

7. Notify Tier 4 support and include the log files with the trouble ticket.

8. Monitor the server closely over next 24 hours. If the source of the problem cannot be identified in the log files, there may be a hardware or network issue that will reappear over time.

Additional Shutdown Information

The Administration Console generates SNMP messages for managed WebLogic Server instances only until the ServerShutDown message is received. Afterwards, no additional messages are generated.

sipAppDeployed

Converged Application Server engine tier nodes generate this alarm when a SIP Servlet is deployed to the container.

Recovery Procedure

This trap is generated during normal deployment operations and does not indicate an exception.

sipAppUndeployed

Converged Application Server engine tier nodes generate this alarm when a SIP application shuts down, or if a SIP application is undeployed. This generally occurs when Converged Application Server is shutdown while active requests still exist.

Recovery Procedure

During normal shutdown procedures this alarm should be filtered out and should not reach operations. If the alarm occurs during the course of normal operations, it indicates that someone has shutdown the application or server unexpectedly, or there is a problem with the application. Notify Tier 4 support immediately.

sipAppFailedToDeploy

Converged Application Server engine tier nodes generate this trap when an application deploys successfully as a Web Application but fails to deploy as a SIP application.

Recovery Procedure

The typical failure is caused by an invalid sip.xml configuration file and should occur only during software installation or upgrade procedures. When it occurs, undeploy the application, validate the sip.xml file, and retry the deployment.

Note:

This alarm should never occur during normal operations. If it does, contact Tier 4 support immediately.