4 Service Logic Controller (SLC)

This chapter explains how and why the SLC is used.

Service Logic Controller Overview

The SLC is used to handle calls using compiled control plan logic which is initially defined on the SMS and replicated to each SLC node.

All the main processing takes place inside the SLEE. On the SLC, this processing is primarily handled by slee_acs. Depending on the protocols involved on the network, a number of Network Connectivity Agent processes will also be running.

Service Logic Execution Environment

The main process working to handle requests between the network and the SLEE is slee_acs. slee_acs is part of the Oracle Communications Convergent Charging Controller, and is located in /IN/service_packages/ACS/

For more information on configuration options, refer to the appropriate product technical guide.

SLEE.cfg

Configuration for the SLEE in general is contained in /IN/service_packages/SLEE/etc/SLEE.cfg. This contains configuration for the resources allocated to the SLEE, and the applications, services and servicekeys of applications running on the SLEE.

  • MAX<Parameter>: Contains the maximum resources allocated to the SLEE, such as:
    • Applications
    • Services
    • Dialogs
    • Events
    • Calls
  • APPLICATION: Contains the location of the application startup scripts, and how many instances to run.
  • INTERFACE: Contains the definitions for interfaces on the SLEE, their startup scripts and interface type.
  • SERVICE: Defines the applications/interfaces as a service (can be done multiple times for each application/interface).
  • SERVICEKEY: Defines the service to be triggered for received service keys.

ACS.conf

Configuration for other ACS components are contained in /IN/service_packages/ACS/etc/acs.conf. This contains general configuration options for the following processes:

  • acsStatisticsDBInserter
  • acsStatsMaster
  • acsStatsLocal
  • acsCompilerDaemon

It also contains configuration for acsChassis, which specifies:

  • Plug-ins
  • Services
  • Normalization
  • ACS EDR generation
  • Some other specific call-handling scenario configuration options

eserv.config

Remaining configuration is primarily found inside /IN/service_packages/eserv.config. Each product has a top level section, (for example "ACS {}" for ACS) and the underlying processes for each product are configured in sub-sections of eserv.config.

There are a few exceptions to this. Most notably SUA and M3UA Interface configuration, which is found in /IN/service_packages/SLEE/etc/

SLEE Watchdog

The SLEE watchdog is responsible for keeping track of all the processes running inside the SLEE. Upon SLEE startup, all the processes are registered with the watchdog. The watchdog periodically checks each process to make sure it is processing events correctly.

If not, the watchdog marks the process as suspect and sends the process a management event. During the next watchdog cycle (default 30 seconds) the watchdog will check that the event has been processed. If the event was not actioned, the process will be aborted and restarted.

Abort information

Whenever a process is terminated or restarted by the watchdog, there are appropriate records in SLEE.log/syslog, for example: 

watchdog(18186) WARNING: Interface beVWARS3 does not exist at PID 18169, presuming dead. 
Jul 18 00:53:16.091448 watchdog(18186) WARNING: Sending SIGABRT to interface beVWARS3, process 
18169. 
Jul 18 00:53:42.250416 watchdog(18186) WARNING: Interface beVWARS3 does not exist at PID 18169, 
presuming dead. 
Jul 18 00:53:42.250844 watchdog(18186) WARNING: Restarting interface beVWARS3 (was process 
18169).
Jul 18 00:53:42 watchdog(18186) SleeInterfaceInstance::start()=3 sleeInterfaceInstance.cc@156: 
Starting Interface beVWARS3.sh: PID: 8237

The watchdog also has built in deadlock prevention. A timer is set before beginning a check loop to ensure that it does not get deadlocked on a semaphore. If the timer expires, the watchdog believes the SLEE is having serious issues and will restart the entire SLEE.

Note:

Some interactions with a process can stop it from responding to watchdog management events, one common example of this is a gcore. In this situation, the watchdog should be terminated or sent a SIGUSR1 signal to stop operating.

The SLEE will need to be restarted in order for the watchdog to become operational again.

Update loader

The updateLoader process on the SLC is the client-side of IN Replication. It is responsible for receiving and applying updates from the smsMaster process on the SMS.

The updateLoader is run from inittab, and runs in run-level 2.

While not a traffic handling application, this process is absolutely crucial to the heath of the platform; non-replicated changes can cause major subscriber issues and revenue loss on a solution.

Checking replication status

On the SLC side, the updateLoader process and log file can be checked to make sure the process is up and running, and not presenting any errors.

See the SMS Replication section for information on the SQL queries required to check replication directly from the SMS database.

Full replication

In the event of a replication issue, it may be required to instruct the updateLoader to perform a full resync.

This performs a full collection of data from the SMS database. Depending on the amount of data, and what information is configured to replicate (for example, replicating subscriber data will mean information for every subscriber on the platform is replicated) replication can take a number of hours to complete.

Performing a Full Resync

To perform a full resync, open the updateLoader startup script, and add the -resync argument to the command line (highlighted), and restart the updateLoader process. Enter:
$ vi /IN/service_packages/SMS/bin/updateLoaderStartup.sh

Edit the result as shown:

. /IN/service_packages/SMS/.profile-scp 
echo "`date` - Waiting for DB SCP" 
pid="" 
while [ -z "$pid" ] ; do 
pid=`ps -ef|grep "ora_pmon_SCP"|awk '$8!="grep"{print$2}'` 
if [ -z "$pid" ] ; then 
echo "..." 
sleep 30 
fi 
done 
echo "`date` - DB SCP is ready" 
echo "`date` - Waiting for Replication Cfg" 
while [ ! -f "/IN/service_packages/SMS/etc/replication.config" ] ; do 
echo "..." 
sleep 30 
done 
echo "`date` - Replication Cfg is ready" 
exec /IN/service_packages/SMS/bin/updateLoader -nodeid 301 –resync
Enter: 
$ pkill updateLoader

Resync progress

The progress of a Full Resync can be monitored through /IN/service_packages/SMS/tmp/updateLoader.log: 

Thu Nov 11 01:09:41 GMT 2010 - Waiting for DB SCP 
Thu Nov 11 01:09:41 GMT 2010 - DB SCP is ready 
Thu Nov 11 01:09:41 GMT 2010 - Waiting for Replication Cfg 
Thu Nov 11 01:09:41 GMT 2010 - Replication Cfg is ready 
initialiseNode: Reading '/IN/service_packages/SMS/etc/replication.def' 
… 
Node 301 SMS comparison/resync client ready. 
Oct 11 01:09:44.966814 updateLoader(21378) NOTICE: Update Loader replication process started 
(node 301) 
Cancelling any current client action. 
Oct 11 01:09:44.979322 updateLoader(21378) NOTICE: Reached master node 1 at `<IP Address>' 
RES: Thu Oct 11 01:10:08 2010: Node 301, started processing X SMS and Y SCP records. 
RES: Thu Oct 11 01:10:08 2010: Node 301, resynchronisation pass 1, started processing of X SMS 
and Y SCP records. 
Oct 11 01:10:08.291659 smsCompareResyncClient(21515) NOTICE: Beginning resynchronisation for 
node 301. 
RES: Thu Oct 11 01:10:18 2010: Node 301, table NP_DN_RANGE, group NP_DN_RANGE_0, processed A of 
X SMS and B of Y SCP records. 
Nov 11 01:12:05.308062 updateLoader(21378) NOTICE: Resynchronization Finished.  Processing 
Queued Updates 
Node 301 SMS comparison/resync client ready. 
Nov 11 01:12:05.353114 updateLoader(21378) NOTICE: Finished Processing Queued Updates

The process will periodically report how far through the resync it is, including number of rows (out of total rows - highlighted example).

Once complete, updateLoader will return to regular operations. It is recommended to remove the -resync flag as soon as the resync has finished and restart the process.

Network Connectivity Agents

Network connectivity agents (NCAs) exist to interface the various protocols running on the network with slee_acs and the rest of the SLEE.

For most NCAs, there is an associated process running to translate the incoming protocol to the internal protocol used by slee_acs (INAP).

Example NCAs

This table lists some examples of NCAs:

Protocol Package Process Log File Location
Diameter (Northbound) DCD diameterBeClient /IN/service_packages/DCD/tmp
Diameter (Southbound) DCA diameterControlAgent /IN/service_packages/DCA/tmp
MAP MMX xmsTrigger (via adapter) /IN/service_packages/XMS/tmp
SIP SCA sca_if /IN/service_packages/SLEE/tmp
SMPP MMX xmsTrigger (via adapter) /IN/service_packages/XMS/tmp
SIGTRAN SIGTRAN sua_if / m3ua_if /IN/service_packages/SLEE/tmp
XML/TCAP TCAP_IF xmlTcapInterface /IN/service_packages/SLEE/tmp
USSD UIS ussd_gw /IN/service_packages/UIS/tmp
SOAP/XML OSD osdInterface /IN/service_packages/OSD/tmp
IS41 IS41 cdmagw (m3ua/sua) /IN/service_packages/IDS41/tmp

Information logging

Each NCA generally writes to its own log file and to the syslog. This should help identify what the program was doing right before experiencing an issue. This information is a key requirement when raising SRs with Oracle support.

Checking Services

The SLEE service on the SLC can be checked using the interactive slee-ctrl interface. Slee-ctrl is part of the slee-ctrl package, and is generally available on all machines that run a SLEE.

Note:

The Monitoring and Managing chapter covers checking of services in more detail.

Interactive interface

To enter the interactive interface, run slee-ctrl with no arguments. Enter:

$ slee-ctrl

Result: You are then presented with some environment information, and a slee-ctrl prompt: 

SLEE Control: v,1.0.11: script v,1.21: functions v,1.48: pslist v,1.118 
User acs_oper; session [5570]; terminal pts/3; started Thu Oct 28 03:53:48 GMT 2010 
The following variables are set and will be used to run the SLEE. 
SLEE_USER    acs_oper 
SLEE_SCRIPT  /IN/service_packages/SLEE/bin/slee.sh 
SLEE_CONFIG  /IN/service_packages/SLEE/etc/SLEE.cfg 
SLEE_LOG     /IN/service_packages/SLEE/tmp/SLEE.log 
[acs_oper] slee-ctrl>

Tip: Entering help at prompt lists the valid commands.

Example status reporting

From the prompt, you can issue a series of different commands to interact with the SLEE, or get information about resources, and other things.

Some examples are shown below:

Status

To check the current status of the SLEE, including processes, enter:

[acs_oper] slee-ctrl> status 

------------------------ Thu Oct 25 03:56:23 GMT 2010 -------------------------- 
C APP  USER      PID PPID    STIME COMMAND 
6 SLEE acs_oper 1402    1 00:12:35 /IN/service_packages/ACS/bin/slee_acs 
1 SLEE acs_oper 1404    1 00:12:35 /IN/service_packages/SLEE/bin/capgw 
1 SLEE acs_oper 1405    1 00:12:35 /IN/service_packages/RAP/bin/rap 
1 SLEE acs_oper 1406    1 00:12:35 /IN/service_packages/LCP/bin/locApp 
1 SLEE acs_oper 1407    1 00:12:35 /IN/service_packages/ACSUSC/bin/slee_acs 
1 SLEE acs_oper 1408    1 00:12:35 /IN/service_packages/UIS/bin/ussdgw 
1 SLEE acs_oper 1409    1 00:12:35 /IN/service_packages/SLEE/bin/timerIF 
1 SLEE acs_oper 1410    1 00:12:35 /IN/service_packages/SLEE/bin/alarmIF 
1 SLEE acs_oper 1411    1 00:12:35 IN/service_packages/ACS/bin/acsStatsLocalSLEE 
1 SLEE acs_oper 1412    1 00:12:35 /IN/service_packages/SLEE/bin/replicationIF 
1 SLEE acs_oper 1413    1 00:12:35 /IN/service_packages/E2BE/bin/BeClient 
1 SLEE acs_oper 1414    1 00:12:35 /IN/service_packages/OSD/bin/osdInterface 
1 SLEE acs_oper 1415    1 00:12:35 /IN/service_packages/SCA/bin/sca 
1 SLEE acs_oper 1416    1 00:12:35 /IN/service_packages/RIMS/bin/rims 
1 SLEE acs_oper 1417    1 00:12:35 /IN/service_packages/XMS/bin/xmsTrigger 
4 SLEE acs_oper 1419    1 00:12:35 /IN/service_packages/SLEE/bin/m3ua_if 
1 SLEE acs_oper 1427    1 00:12:36 /IN/service_packages/SLEE/bin/mapGenIF 
1 SLEE acs_oper 1430    1 00:12:36 IN/service_packages/SLEE/bin/xmlTcapInterface 
1 SLEE acs_oper 1432    1 00:12:36 /IN/service_packages/SLEE/bin//watchdog 
total processes found = 27 [ 27 expected ] 
================================= run-level 3 ==================================

Resources

To check the status of SLEE resources, in particular memory/CPU usage, enter:

[acs_oper] slee-ctrl> resources 

------------------------ Fri Oct 29 01:28:31 GMT 2010 -------------------------- 
APP  USER       PID PPID S %CPU %MEM    VSZ    RSS  TIME  ELAPSED COMMAND 
SLEE acs_oper 10267    1 S  0.0  4.0 633584 314720 00:02 04:45:10 slee_acs 
SLEE acs_oper 10268    1 S  0.0  4.0 633584 314720 00:02 04:45:10 slee_acs 
SLEE acs_oper 10269    1 S  0.0  4.0 633584 314728 00:02 04:45:10 slee_acs 
SLEE acs_oper 10270    1 S  0.0  4.0 633584 314712 00:02 04:45:10 slee_acs 
SLEE acs_oper 10271    1 S  0.0  4.0 633584 314728 00:02 04:45:10 slee_acs 
SLEE acs_oper 10272    1 S  0.0  4.0 633584 314720 00:02 04:45:10 slee_acs 
SLEE acs_oper 10273    1 S  0.1  3.4 527728 271768 00:43 04:45:10 timerIF 
SLEE acs_oper 10274    1 S  0.0  3.4 527760 271800 00:00 04:45:10 alarmIF 
SLEE acs_oper 10275    1 S  0.0  3.4 528040 272296 00:07 04:45:10 acsStatsLocalSLEE 
SLEE acs_oper 10276    1 S  0.1  3.5 539712 276120 00:44 04:45:10 replicationIF 
SLEE acs_oper 10277    1 S  0.1  3.7 552960 290896 00:53 04:45:10 BeClient 
SLEE acs_oper 10278    1 S  0.0  3.6 544856 281864 00:00 04:45:10 xmlTcapInterface 
SLEE acs_oper 10279    1 S  0.1  3.4 527784 271824 01:08 04:45:10 watchdog 
total processes found = 13 
================================= run-level 3 ================================== 
Memory: total 7.9G, used 6.1G (77.6%) + 128.0K (0.0%) /tmp, free 1.8G (22.4%) OK 
Swap: total 4.4G, used 2.1G (48.9%), free 2.2G (51.1%) OK

Call resources

To check the status of SLEE call resources, in particular free calls and events, enter:

[acs_oper] slee-ctrl> check 1 

SLEE: Using shared memory offset: 0xc0000000 
01:29:03        Dialogs Apps    AppIns  Servs   Events   Calls 
                [70000] [30]    [296]   [30]    [207152] [25000] 
01:29:03        70000   29      290     24      207146    25000 
01:29:04        70000   29      290     24      207146    25000 
01:29:05        70000   29      290     24      207146    25000 
01:29:06        70000   29      290     24      207146    25000 
01:29:07        70000   29      290     24      207146    25000 
01:29:08        70000   29      290     24      207146    25000 
01:29:09        70000   29      290     24      207146    25000

A dwindling number of free calls indicates a call leak (this means the number of available calls that the SLC is decreasing in such a manner that eventually it will no longer be able to serve traffic). In this situation the only real solution is to restart the SLEE and reset the available resources.

If the problem continues to happen, it will be prudent to investigate the type of traffic hitting the platform and attempt to determine what is triggering the leak. Slower response times from other network elements can also cause a decrease in free SLEE resources.

Stop and start

To stop, start, or restart the SLEE, enter as required at the prompt:

[acs_oper] slee-ctrl> stop

[acs_oper] slee-ctrl> start

[acs_oper] slee-ctrl> restart

Handling Database Connection Reset

If the database connection on SLC node is reset, restart the SLEE processes on it. A full resync has to been done before restarting the SLEE processes.

When the database goes down, or the connection gets reset on SLC, slee_acs is not automatically restarted because of the following reasons:

  • For all of the active calls in SLC, the required session data such as call context, control plan, subscriber details etc. are cached during the call initiation. If the SLEE processes are restarted because of database reconnection, all the ongoing session data are lost, and all of the ongoing calls will end abruptly. In order to avoid this, all the processes are kept running to be able to serve all the subsequent request of active calls.
  • When database goes down, a full resync is required to get the database synched for any profile updates and other database changes. Reconnection to database, without the resync would cause a lot of inconsistent data and configuration related issues.

In such scenarios, any new call will fail because slee_acs will not be able to fetch initial configuration data for the subscriber.