9 Troubleshooting

This chapter explains the important processes on each of the server components in Convergent Charging Controller, and describes a number of example troubleshooting methods that can help aid the troubleshooting process before you raise a support ticket.

Common Troubleshooting Procedures

Refer to System Administrator's Guide for troubleshooting procedures common to all Convergent Charging Controller components.

Checking Current Processes

You can check which processes are running using the standard UNIX command: ps. To find processes being run by Oracle software, you can grep for the string 'oper', which will display all processes being run by the application operator accounts (for example, acs_oper, ccs_oper and smf_oper).

Note:

Some processes which are required for proper functioning may be run by other users, including root or the user which runs the webserver.

Example command: 
ps -ef | grep oper

For more information about the ps command, see the system documentation for the ps command. You can also check how much of the processor a process is using by running the standard UNIX tool: top. If you have some baseline measurements, you will be able to compare it with the current load.

Example command: 
top

Tip:

Some processes should only have one instance. If there are two or more instances, this may indicate a problem. For example, there will usually only be one timerIF running on each SLC.

For more information about which processes should be running on each node, check the Process List for each node in Installation Guide.

Restarting Running Processes Using Kill

Follow these steps to restart a running process.

Note:

Restarting some processes can cause system instability or data loss. Some processes must be restarted using specific tools. Check the documentation for the process before restarting.

  1. Find the Process ID for the process you want to restart. Example command:
    ps -ef | grep smsAlarmRelay

    Note:

    The second column of the results returned is the Process ID and the third column gives the Parent Process ID.

  2. Terminate the process using the kill command. Example command:
    kill -TERM 123

    Result: The process is terminated and will be restarted by the inittab process.

Checking Configuration Files

One of the significant areas where faults can occur and be remedied is in the configuration of processes. Configuration files can be edited by any standard text editor. A backup of the existing configuration file should always be taken before editing a configuration file.

For more information about the configuration files used in this application, see Configuration User's Guide.

For more information about the configuration file for a specific program or tool, see the section named after the binary in question.

Possible Problems

This topic lists common problems and actions you can take to investigate or solve them. This list enables you to check for alarms based on the overall behavior you are experiencing.

SMS Java Screens Will Not Start

Follow these steps to resolve JavaClient problems.

  1. Ensure that the HTTPD daemon (on the SMS) is running and that it is correctly configured.

  2. If you can start the SMS screens, but unable to login:

    • Ensure that the smsGui.bat/smsGui.sh files are correctly configured, and permissions are proper.

    • Ensure that the SMS console can resolve host names into IP addresses.

Java Help Screen Grayed Out

This is caused by Java Runtime Environment (jre) running out of memory for the run time heap cache. Under the default Java settings this may happen after 10 to 15 help screen accesses. Follow these steps to extend the number of Help accesses.

  1. Close the SMS screens.

  2. From the Windows system, open the Control Panel.

  3. Switch to Classic View to see the complete list of installed applications.

  4. Double click Java icon to open java Control Panel.

  5. Select the Java tab.

  6. Click View in the Java Applet Runtime Settings panel.

  7. Click the Java Runtime Parameter field.

    Note:

    This is the fourth field along, pop-up may require expanding to see this field.

  8. Type -Xms10M -Xmx512M in the Java Runtime Parameter field.

    Note:

    If other parameters are there, add these to the end.

  9. Click OK.

  10. Click Apply.

  11. Click OK.

  12. Close the Control Panel.

  13. Restart the browser and start the SMS screen.

Note:

Using Xmx512M may cause issues with starting jre. If the browser jre cannot start up, try -Xmx180M.

Replication is Failing

This table describes possible problems with replication.

Table 9-1 Replication Problems

Alarm Reason Remedy
Cannot connect to Oracle – exiting There is a problem with the replication.config files in the system.

Use smsDumpRepConfig to check that the content of replication.config is correct.

Generate a new replication.config file and check is it correctly copied to each machine.

For more information, see "unresolvable-reference.html#GUID-74FDDECE-E835-4B53-BDCF-47A6A630CD73".

Could not make fifo f – exiting A connection is being dropped because the heartbeat settings on each end of a connection are different. Check that the heartbeat settings for both ends of the connection are the same. The heartbeat settings are in replication.def, though they can be overridden at the command line for any process.

comparisonServer is Failing

This table describes possible problems with comparisonServer.

Table 9-2 comparisonServer Problems

Alarm Reason Remedy
The replication.config file is not available to inetBootstrap, so smsCompareResync is not starting up. Check that replication.config is in the correct directory and is readable by smf_oper.

Index Defragmentation

The automatic defragmentation facility provided by SMS is intended to prevent fragmentation of the replication tables which frequently use insert, delete and update functions.

In order to enable this defragmentation facility, the script fragmentation_install.sh must first be installed. This will install the stored procedure sms_defrag_rep_iot, and schedule a job to run it every 10 minutes.

The following tables are affected:

  • REP_ORA_EVENT

  • REP_ORA_RENUMBERED

Before You Begin

The process for installing the defragmentation script varies depending on the Oracle configuration available on the SMS. For most clustered environments, Oracle configuration is stored in the service parameter file (SPFILE), which permits configuration parameters to be modified at runtime. If this is the case, then the there is no need to manually alter the Oracle configuration.

However, if SPFILEs are not in use (that is the traditional PFILEs are used to manage Oracle configuration), then it is important to first modify the cache and block sizings in theinitSMF.ora file. The cache size for the 32 KB block size should be set to 32 MB or another suitably large value.

Note:

It is recommended that this activity is performed by an experienced DBA.

Enabling Defragmentation

To enable the defragmentation facility, run the following script:
fragmentation_install.sh
This script is located in:
/IN/service_packages/SMS/db/defragmentation

Disabling Defragmentation

To disable the defragmentation facility, run the uninstallation script as the oracle SMF user:
fragmentation_uninstall.sh
This script is located in:
/IN/service_packages/SMS/db/defragmentation

Oracle Configuration Restriction

While editing the parameter files, it must be noted that the following sets of parameters are mutually exclusive and cannot be used in combination with each other.

Example: You cannot use one or more of:
{db_cache_size, 
db_recycle_cache_size, 
db_keep_cache_size, 
db_nk_cache_size (where n is one of 2,4,8,16,32), 
db_cache_advice }
AND one or more of the following in your configuration:
{db_block_buffers
buffer_pool_keep
buffer_pool_recycle}