17 Implementing Overload Protection

This chapter describes how to implement overload protection for Oracle Communications Online Mediation Controller.

About Overload Protection

In some cases, such as unanticipated traffic peaks or failure of a network hardware or software component, the load on Service Broker modules can increase significantly. This can cause a situation known as system overload in which Online Mediation Controller modules have insufficient resources to handle new sessions. If overload is not handled correctly, the system can malfunction and critical data can be lost.

To handle increased amounts of traffic without damaging operations of the entire system, Service Broker provides an overload protection mechanism. This mechanism operates in Processing Domains where you can define criteria for overload detection.

By default, too many number of 1) active sessions or 2) initial requests triggers an overload condition. The system rejects initial requests while the overload lasts. In addition to rejecting initial requests, Online Mediation Controller provides the capability to customize how the system behaves if overload occurs.

Using Gauges and Counters as Key Overload Indicators

When you configure gauges and counters as key overload indicators, Online Mediation Controller triggers overload protection if threshold values are crossed as measured by those indicators. You can select any of the counters and gauges provided by Online Mediation Controller to serve as key overload indicators. See "Monitoring Online Mediation Controller" for information about Online Mediation Controller counters.

Note:

Consult with Oracle Technical Support if you have questions about which runtime MBeans are best suited to implement overload protection in your network environment.

When you configure overload protection, your settings are applied uniformly across all managed servers in the domain. Usually, server load balancing allocates traffic "fairly" across servers. However, it is possible for a single managed server in a domain to enter an overload condition while the other servers are functioning normally.

Understanding System and Module Levels of Overload Protection

You configure Online Mediation Controller overload protection at the system and the module level:

System counters and gauges: These two indicators can detect and trigger an overload condition that might occur across any number of clustered managed servers or when deploying any combination of Service Broker products.
- SystemCountersRuntimeMBean.SessionGauge
- SystemCountersRuntimeMBean.InitialRequestCount
The SessionGauge gauge represents the total number of active sessions on a single managed server (JVM). Sessions are application specific, for example there are separate Online Mediation Controller, Service Controller, and Policy Controller sessions.

The InitialRequestCount counter represents the session creation rate. For example, the number of new sessions created per second on a single managed server.

You configure the SessionGauge and InitialRequestCount indicators for the managed servers in the Administration Console. All managed servers share those configuration settings.

There is no global overload protection status. However, if any managed server goes into an overload state, as defined by the shared configuration, the system stops accepting new sessions until the overload condition ceases.
Module-level counters and gauges: These indicators are for specific modules. Using the Administration Console, you can configure these indicators for overload protection under the monitoring tabs here: Expand Platform, then OCSB, then Processing Tier, and then Interworking and Supplementary Modules.

See "Counters and Gauges For Online Mediation Controller", for descriptions of the Online Mediation Controller counters and gauges.

Counters and Gauges For Online Mediation Controller

"Monitoring Online Mediation Controller" describes a broad range of counters and gauges that can be used to monitor Online Mediation Controller. You can use any of the counters or gauges to implement overload protection.

Table 17-1 lists several counters that are particularly useful for implementing overload protection for the following SOAP Web services:

Subscriber Provisioning API including StoreSubscriber, UpdateSubscriber, GetSubscriber, and DeleteSubscriber.
TopUp services including including Authenticate, TopUp, GetBalance,and VoucherTopup.

To configure counters for Subscriber Provisioning and TopUp use a JMX-client such as JConsole to access these MBeans:

Subscriber Provisioning API MBeans are in this bundle:
- oracle.ocsb.app.rcc.service.subscriber_store.provisioning.soap_ws
TopUp MBeans are in this bundle:
- oracle.ocsb.app.rcc.feature.topup.ws.ws_app

Note:

In a multi-domain topology, the counters for Subscriber Provisioning are not valid. A workaround is to deploy a unified domain.

Table 17-1 describes the counters and gauges for Subscriber Provisioning overload protection and Table 17-2 describes the counters and gauges for TopUp services overload protection.

There are no A) threshold crossed and B) threshold ceased default values for Subscriber Provisioning of for TopUp services counters.

The most important counters for implementing overload protection are:

TotalSubscriberCount for Subscriber provisioning services
TopupTotalRequestCount for TopUp services

Table 17-1 Counters and Gauges for Subscriber Provisioning Services

Attribute	Description
StoreSubscriberCount	Number of StoreSubscriber requests during the counter interval.
UpdateSubscriberCount	Number of UpdateSubscriber requests during the counter interval.
GetSubscriberCount	Number of GetSubscriber requests during the counter interval.
DeleteSubscriberCount	Number of DeleteSubscriber requests during the counter interval.
TotalSubscriberCount	The total number of the above four counters for Subscriber provisioning related requests during the counter interval.

Table 17-2 Counters and Gauges for TopUp Services

Attribute	Description
AuthenticateCount	Number of Authenticate requests during the counter interval.
TopupCount	Number of Topup requests during the counter interval.
GetBalanceCount	Number of GetBalance request.s during the counter interval.
VoucherTopupCount	Number of VoucherTopup requests during the counter interval.
TopupTotalRequestCount	The total number of the above four counters for Topup related requests during the counter interval.

Understanding the Essential Steps for Configuring Overload Protection

These steps must be followed to configure overload protection.

By default, the following gauge and counter are defined as your key overload indicators:
- SystemCountersRuntimeMBean.SessionGauge: A gauge that measures the number of active sessions handled by a single managed server. The number of active sessions includes Online Mediation Controller sessions, and if installed, sessions from other installed products: Service Controller, and Policy Controller.
- SystemCountersRuntimeMBean.InitialRequestCount: A counter that measures the rate at which a managed server receives new sessions.
  
  The SessionGauge and InitialRequestCount measurements are per managed server but all of the managed servers share a common configuration. If any managed server goes into an overload condition, the system stops accepting new sessions until the overload condition ceases.
Identify Module level counters and gauges you want to use as key overload indicators.
Configure Threshold Crossed Notifications details for your module-level counters and gauges.

For each counter and gauge you want to use as a key overload indicator, define an upper threshold and ceased value threshold. For example, if the upper threshold value is 100 and the ceased value is 90 then if 100 is crossed the system remains overloaded until the value goes below 90.

Specify a threshold name for each module-level counter or gauge. If you use either sessionGauge or initialRequestCount as the threshold name value you do not have to add these indicators to the Key Overload Indicators pane.
Configure Key Overload Indicators.

After you have configured the module-level counters and gauges you want to use for overload protection you need to specify that they are to be used by Online Mediation Controller as Key Overload Indicators.

You do this by using the Administration Console. Expand Tier Management, and then use the Key Overload pane to define your key overload indicators.

Important: Service Broker activates overload protection when any of your key overload indicator crosses its upper threshold.
Customize overload protection behavior.

The behavior of Online Mediation Controller is that if an overload condition occurs, the system continues to handle all active sessions but rejects initial requests until the overload condition ceases.

In addition to the default protection behavior, you can customize how Service Broker responds to SIP and Diameter network entities that attempt to establish sessions during a system overload.

Example: You can define the type of error and value of the SIP Retry-After header field that Service Broker uses to respond to newly established SIP sessions.

You can customize overload protection behavior by using the Administration Console. Expand Tier Management, then Overload and Tracing, and then the Overload Protection Methods pane.

Configuring Key Overload Indicators

The following sections describe in detail how to configure Key Overload Indicators.

Configuring Threshold Crossed Notifications Rules

This section describes how to create Threshold Crossed Notifications rules for overload protection. The components of these rules specify MBean type, threshold name, crossed and ceased threshold values, and other fields.The following steps are applicable to both systemwide and module-level counters and gauges. There are only two systemwide overload indicators: sessionGauge and InitialRequestCount. The default settings for these indicators should usually not be changed.

To transform any of the counters or gauges you configure in this section to be a key overload indicator you must match the threshold name value you set under the Monitoring tab with the threshold name value in the Key Overload indicators pane.

For example, the default key overload indicator threshold name sessionGauge matches the threshold name value in the default threshold crossed notifications rule also sessionGauge.

To configure Threshold Crossed Notification Rules do the following:

In the navigation tree, expand the OCSB node.
Expand Processing Tier.
Do any of the following:
- To configure System-level Counters and Gauges: Expand Tier Management, then Monitoring, and then Monitoring. Note: You cannot add more counters and gauges in addition to the default sessionGauge and InitialRequestCount indicators. However, if required you can modify details such as crossed and ceased threshold values.
- To configure Module-level Counters and Gauges for OMC:
  
  Expand Interworking or Supplementary Modules, and then expand the module for which you want to configure a counter or gauge as an overload indicator.
In the Monitoring tab, select Threshold Crossed Notifications.
Be sure you have selected Lock & Edit and then click New.
In the Threshold Name field, enter a string that names the threshold. This value is referenced by the key overload indicators.
For the Enable threshold field, select True or False. Only enabled thresholds are considered for overload protection.
In the MBean Type field, enter the type of MBean.
For the Counting Type field, select the Counting method. For gauges use CurrentGeneralValue and for counters use CurrentIntervalDeltaValue.
In the MBean Attribute field, enter an MBean attribute. For SystemGaugeRuntime use SessionGauge and for SystemCountRuntime use InitialRequestCount.
In the Threshold class field, enter High. Crossing a low threshold does not cause an overload state.
In the Threshold Value field, enter an integer value which when crossed triggers an overload state.
In the Threshold ceased value field, enter an integer value which when crossed the triggered threshold ceases. This value is applicable only to gauges.
In the Threshold crossed message field, enter a message included in the threshold notification.
In the Threshold ceased message field, enter a message included in the threshold ceased notification.
In the Server filter field, leave it empty or use a regular expression to filter on a managed server. For example "managed_1" or "server."
In the Resource filter field, enter a unique name for the indicator.
Click Apply.

Specifying Your Key Overload Indicators

Identify the module-level counters and gauges you want to use for overload protection. Use the Administration Console to list the names and threshold names for these indicators.

The Overload Protection pane is pre-populated with these two systemwide indicators:

SystemCountersRuntimeMBean.SessionGauge
SystemCountersRuntimeMBean.InitialRequestCount

To specify module-level Key Overload Indicators do the following:

In the navigation tree, expand the OCSB node.
Expand Processing Tier.
Expand Tier Management.
Select Overload Protection. The Key Overload Indicators pane appears.
Be sure you have selected Lock & Edit and then click New.
In the Name field, enter a unique name for the indicator.
In the Threshold Name field, enter a unique string that references the threshold.

Multiple indicators at the system or module-level can use the same Threshold Name. In this situation, all matching crossed thresholds will be considered to indicate an overload state.

Example: If any module-level counter or gauge use either sessionGauge or initialRequestCount as the threshold name value, you do not have to add these module-level indicators to the Key Overload Indicators pane. However, the module-level settings (e.g. crossed threshold value) will override the platform level settings for that individual module only.
Click Apply.

Configuring General Monitoring Parameters

This section describes how to configure general attributes for overload protection notifications.

Configure these attributes at both the system and module levels. Expand Tier Management, then Monitoring, then select the General tab. To configure IMs expand Interworking Modules, then the IM you want to configure, then Monitoring, and then select the General tab.

At the Tier level these parameters affect only SessionGauge and InitialRequestCount. At the module level the parameters affect all runtime MBeans in the module. Table 17-3 describes the configuration parameters in the Monitoring General tab.

Table 17-3 General Overload Configuration Parameters

Name	Description
Enable runtime MBeans	Disables runtime MBeans so you can neither poll them for values or get notifications.
Enable Notifications	Disables only notifications, so you can still poll values from the MBean.
Counter Interval (sec)	This parameter specifies the length of the interval in seconds. Note: This parameter is not configurable at the module level.
Notification trigger interval (sec)	Sampling interval in seconds for checking notifications. For example, if the Counter Interval is set to 10 seconds and the Notification trigger interval is set to 2 seconds for each counter interval the system will determine 5 times whether the threshold value has been crossed.

Configuring the Overload Protection Mechanism

When system overload occurs, Service Broker rejects new sessions and sends response messages to the network entities that attempted to establish the new sessions.

In the Overload Protection Methods pane you can configure how Service Broker responds to attempts by SIP and Diameter network entities to establish new sessions.

Table 17-4 describes configuration parameters you can access by doing the following: Expand Platform, then Processing Tier, then Overload and Tracing, and then the Overload Protection Methods subtab.

Table 17-4 Overload Protection Methods

Name	Type	Description
Enabled	BOOL	Specifies whether the overload protection methods specified in this table are enabled. Possible values: TRUE FALSE
SIP Response Status Code	STRING	Specifies a SIP error that Service Broker returns to a SIP network entity when Service Broker declines an attempt to establish a session. Default value: 503
SIP Retry-After	STRING	Specifies the value that Service Broker sets in the Retry-After header of the error response sent to the network entity. This value defines how long the network entity waits before it retries to establish a session. Default value: 300
Diameter Response Result Code	STRING	Specifies a response Result Code AVP that Service Broker returns to a Diameter network entity when Service Broker declines the attempt to establish a session. Default value: 5012
Web Service Response Status Code	INT	Specifies an error code that Online Mediation Controller returns to a Web service network entity when Service Broker declines the attempt to establish a session. Default value: 503
SAL Response Status Code	INT	Specifies an error code that Online Mediation Controller returns to a SAL application when Service Broker declines the attempt to establish a session. Default value: 503