Skip Headers
Oracle® Communications Billing and Revenue Management System Administrator's Guide
Release 7.5

E16719-12
Go to Documentation Home
Home
Go to Table of Contents
Contents
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
PDF · Mobi · ePub

14 Optimizing Pipeline Manager Performance

This chapter describes tools and techniques you can use to optimize Oracle Communications Billing and Revenue Management (BRM) Pipeline Manager performance.

Before reading this chapter, you should be familiar with the following topics:

Pipeline Manager Optimization Overview

When you optimize Pipeline Manager performance, your objective is to increase the percentage of CPU time spent on user processes and to decrease the percentage of time spent idle or on system processes.

Complete performance tuning requires much testing. Due to the complexity of most Pipeline Manager configurations, optimization is a highly iterative process. You cannot configure options formulaically, but you must test many configurations and then implement the optimal configuration. This chapter describes optimization methods to guide your testing for a given set of hardware resources.

Software optimization techniques can include modifying the following:

  • The number and type of function modules.

  • The design of custom iScripts and iRules.

  • The number of system threads used by a pipeline.

  • The number of call data record (CDR) files configured for a transaction.

  • The number of pipelines configured for the system.

    Note:

    Available hardware resources can constrain the usefulness of some optimization techniques. For example, if your system has only a few CPUs, you probably will not see performance gains by using multithreaded mode.

Key Metrics for Measuring Performance

When evaluating performance improvement, the primary metrics to monitor are:

  • The ratio of CPU time spent on system processes to CPU time spent on user processes. This ratio should be about 1 to 2 or lower.

  • The percentage of idle CPU time. This percentage should be 20 percent or less.

  • The results of performance tests using sample CDR files.

About Measuring Pipeline Manager Performance

You use the Pipeline Manager instrumentation feature as the primary tool for measuring Pipeline Manager performance. See "Measuring System Latencies with Instrumentation" for more information. When instrumentation is enabled, information about how much time in microseconds each function module uses to process a certain number of files is written to the pipeline log file (pipeline.log). You then use this information when you apply some optimization techniques.

Note:

For more information on the pipeline log, see "LOG" in BRM Configuring Pipeline Rating and Discounting.

Other Pipeline Manager performance monitoring tools are:

Information Requirements

Before you optimize Pipeline Manager, be familiar with your existing system configuration, such as:

  • Total system memory.

  • Other (non-pipeline) processes running on the Pipeline Manager system that will share system resources.

  • The number and types of pipelines required for your business logic or planned load balancing.

  • The expected load for each pipeline.

  • Whether your business logic is more CPU intensive or I/O intensive. (For example, if you use the FCT_Discount module, your business logic is likely to be more CPU intensive.)

Testing Requirements

Before you optimize Pipeline Manager, you should have a set of error-free sample CDRs that resemble those used in your production system.

Optimizing Pipeline Manager

To optimize Pipeline Manager, consider the following actions:

Troubleshooting Pipeline Performance

Use the following checklist to troubleshoot drops in performance.

  • If you installed a patch, find out if the patch changed operating system functions, such as threading or memory management, or made any changes to Pipeline Manager framework modules.

  • Check recent customizations, such as iScripts. Look for customizations that might impact database access or hash usage.

  • Use database monitoring tools to monitor the Pipeline Manager database to see if there is a lot of activity. If so, check which queries are used and which indexes are used. This might point to the data involved, which might point to the module processing that data.

  • Use a monitoring command such as iostat to check I/O activity.

  • Use a memory monitoring command such as prstat, vmstat, or sar to check if the Pipeline Manager memory usage has changed. If Pipeline Manager uses an unexpected amount of memory, check for duplicate keys related to buffers and call assembly.

  • Check for large numbers of files in the following directories:

    • in

    • err

    • done

    • dupl

    • assembl

    • rej

    Delete old files that are no longer needed.

  • Look for bottlenecks in the pipeline by using the prstat command and the thread ID in the process.log file to identify slow threads. Check for:

    • icx (involuntary context switch)

    • vcx (voluntary context switch)

    • scl (system call)

    • slp (sleep)

  • Check the pipeline.log file for records of a large amount of rollbacks.

Optimizing Function Modules

Slow function modules can be very detrimental to overall Pipeline Manager performance. To optimize individual function modules:

  1. Identify the high latency modules by using instrumentation. See "Measuring System Latencies with Instrumentation" for more information.

  2. Check if the high latency modules can be optimized. For example, you might discover that the business logic used in high latency iScripts or iRules can be redesigned to improve performance.

Configuring Single-Threaded or Multithreaded Operation

You configure pipelines to run in single-threaded or multithreaded mode by using the MultiThreaded registry entry in the registry file.

  • Single-threaded mode: Use this mode if you are using a system with just a few CPUs and limited RAM.

    In a single-threaded environment, pipelines use a single thread to run all modules and only one CPU is used for each pipeline.

    If the MultiThreaded registry entry is not included in the registry file, pipelines will by default run in multithreaded mode.

    Note:

    Business logic can prevent the setup of multiple pipelines.
  • Multithreaded mode: Use this mode if your system has many CPUs.

    In a multithreaded environment, pipelines use three or more threads to process each transaction. By default, one thread is used for the input module and one for the output module. An additional thread is used for each function pool that you configure to process function modules.

    For information on optimizing pipelines when using multithreaded mode, see:

For information about the MultiThreaded registry entry, see "Pipeline Controller" in BRM Configuring Pipeline Rating and Discounting.

To configure single-threaded or multithreaded operation:

  1. Open the registry file in a text editor.

  2. Set the input controller MultiThreaded registry entry to the appropriate value:

    • True to configure the pipeline for multithreaded processing.

    • False to configure the pipeline for single-threaded processing.

    Pipelines
    {
        PipelineName
        {
            MultiThreaded = value
            ...
        }
    }
      
    
  3. Restart the pipeline. See "Starting and Stopping Individual Pipelines" for more information.

    Note:

    For more information on all registry entries pertaining to individual pipelines, see "Pipeline Controller" in BRM Configuring Pipeline Rating and Discounting.

Reducing Startup Times with Parallel Loading

You can reduce your startup times by configuring Pipeline Manager to:

  • Load all pipelines in parallel.

  • Load data modules in parallel.

  • Load function modules in parallel.

By default, Pipeline Manager loads pipelines, data modules, and function modules sequentially.

To enable parallel loading, use the Parallel Load Manager module:

  1. Open the registry file in a text editor.

  2. Configure the ifw.ParallelLoadManager section of the registry file:

    • Set the Active registry entry to True.

    • Set the NumberOfThreads registry entry to the number of threads you want Pipeline Manager to use for loading your pipelines, data modules, and function modules.

    For example:

    ifw
    {
       ...
       ParallelLoadManager
       {
          Active = True
          NumberOfThreads = 4
       }
       …
    }
      
    
  3. Restart the pipeline. See "Starting and Stopping Individual Pipelines" for more information.

Assigning Multiple Threads to Process Function Modules

If a pipeline is configured for multithreaded processing and you have idle CPU resources, you might be able to increase performance by grouping function modules into two or more function pools. The pipeline runs each function pool in a separate thread.

Important:

Adding too many function pools to a pipeline can decrease performance because the buffers between the threads consume system CPU overhead and RAM. (Typically, two to six function pools is optimal.)

Tip:

If you are using a high-latency module such as FCT_AccountBatch or FCT_Discount and have sufficient hardware resources, assign the module to its own function pool and test for performance improvement.

To create a separate thread for an individual function module or a group of function modules, you use the FunctionPool registry entry.

Important:

Before you perform this procedure, read "Optimizing a Pipeline by Using Function Pools" for more information.
  1. Submit some sample CDRs to the pipeline with instrumentation enabled. See "Measuring System Latencies with Instrumentation" for more information.

  2. Locate the instrumentation results in the pipeline.log file.

  3. Open the registry file in a text editor.

  4. Using the instrumentation data, reduce the processing time required by the slowest function pool by:

    • (Optional) Adding an additional function pool to the Functions section of the registry file.

    • Shifting one or more modules from a function pool to an adjacent function pool.

    The objective is to make the processing times of all function pools as similar as possible.

  5. Save the registry file.

  6. Restart the pipeline. See "Starting and Stopping Individual Pipelines" for more information.

  7. Measure pipeline performance with the sample CDRs by measuring transaction start times and end times in the pipeline.log file.

  8. Go to Step 3 and repeat testing until optimal results are achieved.

Configuring the DAT_AccountBatch Module Database Connections and Threads

To improve performance, you can configure multiple DAT_AccountBatch connections to the BRM database. See "DAT_AccountBatch" in BRM Configuring Pipeline Rating and Discounting. Configure the following registry entries:

  • Use the Threads registry entry to specify the number of threads. Set this value to at least the number of CPUs in the system. Increasing the number of threads increases performance, up to a point. Specifying too many threads decreases performance.

    The default is 4.

  • Use the Connections registry entry to specify the number of connections to the database. This value must be at least one more than the number of threads.

    The default is 5.

  • Use the LoadPercentage registry entry to specify the percentage of account POIDs to store locally when determining the account blocks for which each thread is responsible.

    Values must be greater than 0.000000 and less than or equal to 100.0.

    The default is 10.

Setting the Hash Map Size for Threads

You can use the following DAT_AccountBatch registry entries to set the temporary hash map size built for each thread. Each entry controls the hash map size for a different type of data; for example, accounts, logins, and services.

In general, larger maps perform better but consume more memory. Smaller maps save memory but can slow down Pipeline Manager startup. Very low numbers can dramatically slow down Pipeline Manager startup.

The default system-calculated value uses the following formula:

((number of accounts/number of threads) * 2).

The registry entries are:

  • ThreadAccountHashMapSize: Used for account data.

    Important:

    Changing the default system-calculated values for this entry is not recommended. Replacing this entry with one larger than the default wastes memory. Replacing this entry with one smaller than the default slows Pipeline Manager startup.
  • ThreadGroupSharingChargesHashMapSize: Used for charge sharing group data. The system-calculated default value might not be appropriate.

    If your accounts average fewer than two or more than four GroupSharingCharges per account, use the following formula as a guideline to calculate an entry:

    (((number of accounts * average number of GroupSharingCharges per account) / number of threads) * 75%).

  • ThreadGroupSharingDiscountsHashMapSize: Used for discount sharing group data. The system-calculated default value might not be appropriate.

    If your accounts average fewer than two or more than four GroupSharingDiscounts per account, use the following formula as a guideline to calculate an entry:

    (((number of accounts * average number of GroupSharingDiscounts per account) / number of threads) * 75%).

  • ThreadGroupSharingProfilesHashMapSizes: Used for profile sharing group data. The system-calculated default value might not be appropriate.

    If your accounts average fewer than two or more than four profile sharing groups per account, use the following formula as a guideline to calculate an entry:

    (((number of accounts * average number of GroupSharingProfiles per account) / number of threads) * 75%).

  • ThreadLoginHashMapSize: Used for login data. The system-calculated default value is appropriate for most implementations.

    If your accounts average more than four logins per account, use the following formula as a guideline to calculate an entry:

    (((number of accounts * average number of logins per account) / number of threads) * 75%).

  • ThreadServiceHashMapSize: Used for service data. The system-calculated default value is appropriate for most implementations.

    If your accounts average more than four services per account, use the following formula as a guideline to calculate an entry:

    (((number of accounts * average number of services per account) / number of threads) * 75%).

Locking Objects during DAT_AccountBatch Processing

You can set the number of pre-allocated mutex objects that are used to lock individual objects during processing to prevent multiple threads from contending for access to the same object. You can use different settings for account, login, and service objects by setting the following DAT_AccountBatch registry entries:

  • AccountLocks

  • LoginLocks

  • ServiceLocks

The default value for these entries should be appropriate in most cases. If you use a larger value, less allocation is needed for additional mutex objects during processing, but more memory is used.

The default for all entries is 10.

Configuring Threads for DAT_BalanceBatch Connections

Use the following DAT_BalanceBatch registry entry to configure connections to the BRM database:

  • Threads: Specifies the number of threads for loading the balance data from the BRM database. The number of threads must be smaller than or equal to the number of connections.

    The default is 4.

  • ThreadHashMapSize: Specifies the size of the hash map in each thread used for loading balance data from the BRM database.

    The default is 1024.

Improving Pipeline Manager Startup Performance

For information about improving Pipeline Manager startup performance, see:

Improving DAT_BalanceBatch Loading Performance

The DAT_BalanceBatch module uses the non-currency resource validity to select the non-currency sub-balances to load from the BRM database into pipeline memory. If the non-currency resource validity is not configured, at Pipeline Manager startup, DAT_BalanceBatch selects the sub-balances that were valid for 366 days by default. When the BRM database contains a large number of non-currency sub-balances, loading them leads to increased Pipeline Manager startup times.

To improve Pipeline Manager startup performance, you can set the non-currency resource validity to specify the sub-balances to load. See "Specifying Which Non-Currency Sub-Balances to Load on Startup" in BRM Setting Up Pricing and Rating for more information.

Improving DAT_AccountBatch and DAT_BalanceBatch Load Balancing

The DAT_AccountBatch and DAT_BalanceBatch modules use multithreaded framework to load account and balance data from the BRM database into Pipeline Manager memory. The modules group the accounts and balances into batches or jobs. Multiple worker threads run in parallel to process the jobs. When a thread completes processing, it is assigned another job from the jobs pool, which improves load balancing between the threads and increases Pipeline Manager startup performance.

By default, the number of jobs per thread is 3, which is appropriate in most installations to achieve load balancing. However, if thread loading times vary greatly, you can use the PerThreadJobsCount entry in the DAT_AccountBatch registry and the BalancesPerThreadJobsCount entry in the DAT_BalanceBatch registry to adjust the number of jobs per thread.

Important:

Setting the number of jobs per thread to a large number can outweigh the performance gain because of the system overhead associated with creating too many jobs. (Typically, three to eight jobs per thread is optimal). If you want to adjust the number of accounts or balances per job, you can do this by increasing or decreasing the number of threads. However, when the number of accounts or balances is too small, the data modules use one thread to optimize performance.

Breaking Up Large Nested Subsections in Registry Files

Pipeline Manager can encounter parser stack overflow errors when a pipeline registry section contains a large number of nested subsections.

You can break up large nested subsections and prevent parser stack overflow errors by using anonymous blocks in your registry file. An anonymous block consists of a nested subsection with braces { } and no subsection name, as shown below:

#------------------------------------------------------------------------
# Input section
#------------------------------------------------------------------------
Input
{
   UnitsPerTransaction = 1
   
   InputModule
   {
      { # <-- Beginning of Anonymous Block
         ModuleName  = INP_GenericStream
         Module
         {
            Grammar = ./formatDesc/Formats/Solution42/SOL42_V670_REL_InGrammar.dsc
            DefaultOutput = TELOutput
  
            InputStream
            {
               ModuleName = EXT_InFileManager
               Module
               {
                  InputPath   = ./data/incollect/reprice/in
                  InputPrefix = test_
                  InputSuffix = .edr
                  ...
               }
            } # end of InputStream
         } # end of InputModule
      } # --> End of Anonymous Block
   } # end of InputDataPool 
} 
  

You can place anonymous blocks in any location and at any hierarchy level of the registry file. For the best effect, divide large sections by placing an anonymous block around a group of smaller subsections. This breaks up the section without affecting the hierarchy of the subsections enclosed within the anonymous block.

Optimizing a Pipeline by Using Function Pools

In general, the performance of a multithreaded pipeline varies directly with its slowest thread. The objective of optimizing a multithreaded pipeline is to group the function modules into function pools so that the slowest function pool is as fast as possible. In this environment, faster threads wait a minimum amount of time for data to be delivered or processed by slower threads.

Important:

Adding too many function pools to a pipeline can decrease performance because the buffers between the threads consume system CPU overhead and RAM. (Typically, two to six function pools is optimal.)

You use instrumentation results to guide function pool configuration. Instrumentation results indicate how many microseconds are required by each module to process a given number of requests. You use this information to add function pools or regroup the modules in existing function pools.

Tip:

You cannot improve performance by adding function pools or shifting function modules to adjacent function pools if your slowest function pool:
  • Has one function module in it. (or)

  • Is faster than the input or output module.

You might be able to improve performance by reducing the number of function pools as long as the slowest function pool is faster than the output module. (Any performance gain comes from the reduced number of buffers. Fewer buffers require less system process overhead.)

Adding Additional Function Pools

You might improve system performance by adding one or more additional function pools.

The following example shows a high-level schema of a portion of a registry file for a pipeline called ALL_RATE:

Note:

For information on the buffers between the function pools, see "Configuring Buffers" for more information.
input {...}

Functions
{
    PreProcessing
    {
        FunctionPool
        {
            module_1 {}
            module_2 {}
            module_3 {}
        }
    }

    Buffer1 {...}

    Rating
    {
        FunctionPool
        {
            module_4 {}
            module_5 {}
            module_6 {}
        }
    }

    Buffer2 {...}

    PostRating
    {
        FunctionPool
        {
            module_7 {}
            module_8 {}
        }
    }

output {...}

The instrumentation output in the pipeline.log file reveals the following latencies for each module for processing a fixed set of test transactions:

Note:

For simplicity, the sample latencies have been rounded to the nearest 5,000,000 microseconds.
15.03.2004 13:25:07 testserver ifw IFW NORMAL   00516 - 
(ifw.Pipelines.ALL_RATE.Functions.PreProcessing)
Plugin processing time statistics: '
ifw.Pipelines.ALL_RATE.Functions.PreProcessing.FunctionPool.module_1.Module, 40000000
ifw.Pipelines.ALL_RATE.Functions.PreProcessing.FunctionPool.module_2.Module, 15000000
ifw.Pipelines.ALL_RATE.Functions.PreProcessing.FunctionPool.module_3.Module, 45000000

15.03.2004 13:25:07 testserver ifw IFW NORMAL 00516 - 
(ifw.Pipelines.ALL_RATE.Functions.Rating) 
Plugin processing time statistics: '
ifw.Pipelines.ALL_RATE.Functions.Rating.FunctionPool.module_4.Module, 65000000
ifw.Pipelines.ALL_RATE.Functions.Rating.FunctionPool.module_5.Module, 30000000
ifw.Pipelines.ALL_RATE.Functions.Rating.FunctionPool.module_6.Module, 90000000

15.03.2004 13:25:07 testserver ifw IFW NORMAL 00516 - 
(ifw.Pipelines.ALL_RATE.Functions.PostRating)
Plugin processing time statistics: '
ifw.Pipelines.ALL_RATE.Functions.Postrating.FunctionPool.module_7.Module, 35000000
ifw.Pipelines.ALL_RATE.Functions.Postrating.FunctionPool.module_8.Module, 50000000

This output is summarized in Table 14-1:

Table 14-1 Example 1 Module Latencies Summary

Module Module Latency (Microseconds) Function Pool Function Pool Latency (Microseconds)

module_1

40,000,000

PreProcessing

100,000,000

module_2

15,000,000

PreProcessing

100,000,000

module_3

45,000,000

PreProcessing

100,000,000

module_4

65,000,000

Rating

185,000,000

module_5

30,000,000

Rating

185,000,000

module_6

90,000,000

Rating

185,000,000

module_7

35,000,000

PostRating

85,000,000

module_8

50,000,000

PostRating

85,000,000


The total latency in this configuration is 185,000,000; this represents the microseconds used by the slowest function pool.

Figure 14-1 shows that about a third of the CPU cycles used by the function pool threads are idle:

Figure 14-1 Unused CPU Cycles Example 1

Description of Figure 14-1 follows
Description of "Figure 14-1 Unused CPU Cycles Example 1"

In this example, the pipeline can be optimized if module_6 is assigned to its own function pool, as in this revised sample:

input {...}

Functions
{
    PreProcessing
    {
        FunctionPool
        {
            module_1 {}
            module_2 {}
            module_3 {}
        }
    }

    Buffer1 {...}

    Rating
    {
        FunctionPool
        {
            module_4 {}
            module_5 {}
         }

    Buffer2 {...}

    Discounting
    {
        functionpool
        {
            module_6 {}
        }
    }

    Buffer3 {...}

    PostRating
    {
        FunctionPool
        {
            module_7 {}
            module_8 {}
        }
    }

output {...}

The latency table now appears as shown in Table 14-2:

Table 14-2 Example 2 Modules Latencies Summary

Module Module Latency (Microseconds) Function Pool Function Pool Latency (Microseconds)

module_1

40,000,000

PreProcessing

100,000,000

module_2

15,000,000

PreProcessing

100,000,000

module_3

45,000,000

PreProcessing

100,000,000

module_4

65,000,000

Rating

95,000,000

module_5

30,000,000

Rating

95,000,000

module_6

90,000,000

Discounting

90,000,000

module_7

35,000,000

PostRating

85,000,000

module_8

50,000,000

PostRating

85,000,000


Total function module latency in the new configuration is 100,000,000 microseconds, equivalent to the latency of the PreProcessing function pool. Less than eight percent of function pool CPU cycles are now idle as shown by the gray cycles in Figure 14-2:

Figure 14-2 Unused CPU Cycles Example 2

Description of Figure 14-2 follows
Description of "Figure 14-2 Unused CPU Cycles Example 2"

Shifting Modules between Function Pools

Adding an additional function pool can decrease performance in some situations (see "Adding Additional Function Pools" for more information). This can occur if the system overhead for the additional buffer more than offsets the performance gains from a faster highest-latency function pool. When this occurs, you might be able to improve performance by keeping the number of function pools constant and shifting modules to adjoining function pools.

In the sample above, if adding an additional function pool decreased performance, you could return to using three function pools and then move module 4 to the end of the PreProcessing function pool as shown in Table 14-3:

Table 14-3 Example 3 Modules Latencies Summary

Module Module Latency (Microseconds) Function Pool Function Pool Latency (Microseconds)

module_1

40,000,000

PreProcessing

165,000,000

module_2

15,000,000

PreProcessing

165,000,000

module_3

45,000,000

PreProcessing

165,000,000

module_4

65,000,000

PreProcessing

165,000,000

module_5

30,000,000

Rating

120,000,000

module_6

90,000,000

Rating

120,000,000

module_7

35,000,000

PostRating

85,000,000

module_8

50,000,000

PostRating

85,000,000


Total function module latency in the new configuration is 165,000,000 microseconds. This is equivalent to the latency of the PreProcessing function pool. Though performance gains might be more modest than in the first scenario (where a new function pool was added), the performance gain is more certain because no additional buffer overhead was added.

Configuring Buffers

In a multithreaded pipeline, each pair of consecutive threads communicates through a buffer. Because each function pool is assigned a thread, you must configure a buffer between consecutive function pools.

You configure the buffers between function pool sections in the pipeline registry file. Normally, each buffer can be configured as follows:

Buffer1
        {
          Size = 100
        }

Important:

On Solaris systems, you should configure block transfer mode. See "Block Transfer Mode" in BRM Installation Guide.

Combining Multiple CDR Files into One Transaction

Pipeline Manager is generally more efficient when it processes large CDR files. If a pipeline receives and processes small CDR files, you can improve processing performance by combining multiple CDR input files into one pipeline transaction. You use the UnitsPerTransaction registry entry in the input controller to implement this functionality. See "Input Controller" in BRM Configuring Pipeline Rating and Discounting.

The UnitsPerTransaction entry specifies the number of CDR input files that make up a transaction. By default, each CDR file forms its own transaction.

Note:

The optimal transaction size depends on your system configuration and pricing model. In general, most system configurations perform best when the total number of CDRs, which is the average number of CDRs per input file multiplied by the number of input files in the transaction, is greater than 10,000.

If the UnitsPerTransaction value is greater than 1, you can use the SequenceGeneration registry entry in the output controller to specify whether the pipeline generates one output file per CDR input file or one output file for the entire transaction (see "Output Controller" in BRM Configuring Pipeline Rating and Discounting). Pipeline Manager performance is generally faster when one output file is generated for the entire (multi-CDR) transaction.

To combine multiple CDR files into one transaction:

  1. In the Input section of the registry file, set the UnitsPerTransaction entry to the number of CDR input files that make up one transaction. For example, set UnitsPerTransaction to 100 to combine 100 CDR input files into one transaction.

    Note:

    The default UnitsPerTransaction value is 1.
    Input 
    { 
        ...
        UnitsPerTransaction = 100 
        ...
    }
      
    
  2. (Optional) In the Output section of the registry file, set the SequenceGeneration entry to Transaction. This configures the pipeline to generate one output file for the entire transaction.

    Note:

    The default SequenceGeneration value is Units, which configures the pipeline to generate one output file per CDR input file.
    Output 
    { 
        ...
        SequenceGeneration = Transaction 
        ...
    }
      
    
  3. Stop and restart the pipeline. See "Starting and Stopping Individual Pipelines".

Increasing Pipeline Manager Throughput When an EDR Is Associated with Multiple Output Streams

You can enhance Pipeline Manager throughput by configuring multithreading in the Output Controller. This allows Pipeline Manager to write multiple EDRs in parallel when the EDRs are associated with multiple output streams.

Important:

Enable multithreading in the Output Controller only if the EDRs are associated with multiple output streams.

Enabling multithreading may cause an increase in the overall memory usage of the Output Controller. However, the memory usage becomes constant after processing EDRs for some time.

To configure multithreading in the Output Controller:

  1. Open the registry file (for example, Pipeline_Home/conf/wireless.reg) in a text editor.

  2. In the MultiThreading section, do the following:

    • Set the Active registry entry to True.

    • Set the NumberOfThreads registry entry to the number of threads you want the Output Controller to create for Pipeline Manager to write multiple EDRs in parallel.

    • Set the BatchSize registry entry to the appropriate value:

      • 0 indicates that the Output Controller does not run in batch mode.

      • A value greater than 0 indicates that the Output Controller operates in batch mode with the batch size equal to the specified value.

    For example:

    Output
    {
       ...
       ...
       MultiThreading
       {
           Active = True
           NumberOfThreads = 5
           BatchSize = 500
        }
    }
      
    
  3. Save and close the file.

  4. Restart the pipeline. See "Starting and Stopping Individual Pipelines".

For information about the MultiThreading registry entry, see "Output Controller" in BRM Configuring Pipeline Rating and Discounting.

Configuring Multiple Pipelines

If you have high transaction throughput requirements and additional system resources, you might improve system performance by running multiple pipelines that perform the same function.

In general, consider running multiple pipelines if:

  • Your system has a relatively large number of CPUs.

  • The order of the input streams is not important.

    Note:

    When you use the FCT_CallAssembling or the FCT_DuplicateCheck module, you have to process the EDRs for the same account in the same pipeline. See "Using Duplicate Check with Multiple Pipelines" and "Assembling Calls with Multiple Pipelines" in BRM Configuring Pipeline Rating and Discounting.

    Tip:

    If you configure multiple pipelines and your system is running at near full capacity on a limited number of CPUs, test running the pipelines in single-threaded mode. This configuration reduces the buffer memory allocation requirement and thread-handling overhead. To enable single-threaded operation, set the MultiThreaded entry to False. See "Assigning Multiple Threads to Process Function Modules" for more information.

Customizing Flists Sent to a Real-Time Pipeline

You can configure the fields included in flists sent to a real-time pipeline by using the load_pin_rtp_trim_flist utility. See "load_pin_rtp_trim_flist" in BRM Developer's Guide. This utility is useful for:

  • Improving system efficiency by removing (trimming) fields that the pipeline does not use.

  • Supporting custom iScripts and iRules in the pipeline by adding fields to default flists that are not included in the flists by default.

To optimize the set of fields sent to a real-time pipeline:

  1. Determine which fields are required by the real-time pipeline.

  2. Create an XML file that describes the fields to be sent to the real-time pipeline based on one or more event types.

  3. Load the XML file using the load_pin_rtp_trim_flist utility.

Configuration Object Dot Notation

The load_pin_rtp_trim_flist utility creates a configuration object (/config/rtp/trim_flist). This object is used to create the trimmed flists.

The configuration object uses dot notation. For example, the PIN_FLD_STATUS_FLAGS field at the end of this portion of a sample flist:

0 PIN_FLD_INHERITED_INFO SUBSTRUCT [0] allocated 32, used 32
1     PIN_FLD_POID           POID [0] 0.0.0.1 /account 10243 13
1     PIN_FLD_MOD_T        TSTAMP [0] (1063218065) Wed Sep 10 11:21:05 2003
1     PIN_FLD_ACCOUNT_NO      STR [0] "0.0.0.1-10243"
1     PIN_FLD_CURRENCY        INT [0] 840
1     PIN_FLD_BILL_WHEN       INT [0] 1
1     PIN_FLD_LAST_BILL_T  TSTAMP [0] (1063217469) Wed Sep 10 11:11:09 2003
1     PIN_FLD_BAL_GRP_OBJ    POID [0] 0.0.0.1 /balance_group 8323 4
1     PIN_FLD_SERVICE_INFO   SUBSTRUCT [0] allocated 32, used 32
2         PIN_FLD_STATUS         ENUM [0] 10100
2         PIN_FLD_STATUS_FLAGS    INT [0] 0
  

is represented as:

PIN_FLD_INHERITED_INFO.PIN_FLD_SERVICE_INFO.PIN_FLD_STATUS_FLAGS
  

in the configuration object.

About the field_list.xml File

The field_list.xml file specifies the fields from the /account and /service objects that are included in the flist that is sent to Pipeline Manager. You can define conditions in <EventMap> sections in the XML file that indicate which fields should be included in the flist depending on the event type.

The following example shows the XML file structure with session and provisioning event filters:

  <EventMapList>

    <!--* The following event map specifies fields sent 
        * when the event type is exactly /event/session -->

    <EventMap>
      <Event>
        <Type>/event/session</Type>
        <Flags>0</Flags>
      </Event>
      <RequiredField>

          <!-- List of fields sent put here. -->

      </RequiredField>
    </EventMap>

    <!--* The following event map specifies fields sent 
        * when the event type starts with /event/session/ -->

    <EventMap>
      <Event>
        <Type>/event/session/</Type>
        <Flags>1</Flags>
      </Event>
      <RequiredField>

          <!-- List of fields sent put here. -->

      </RequiredField>
    </EventMap>

    <!--* The following event map specifies fields sent 
        * when when a provisioning event matches any of three conditions. -->

    <EventMap>
      <Event>
        <Type>/event/provisioning</Type>
        <Flags>0</Flags>
      </Event>
      <Event>
        <Type>/event/provisioning/session</Type>
        <Flags>0</Flags>
      </Event>
      <Event>
        <Type>/event/provisioning/</Type>
        <Flags>1</Flags>
      </Event>
      <RequiredField>

          <!-- List of fields sent put here. -->

      </RequiredField>
    </EventMap>

    <!--* The following event map specifies fields sent when none of the 
        * above conditions are true. -->

    <EventMap>
      <Event>
        <Type>*</Type>
        <Flags>1</Flags>
      </Event>
      <RequiredField>

          <!-- List of fields sent put here. -->

      </RequiredField>
    </EventMap>
  </EventMapList>
  

The Flags tag in the XML file specifies event matching criteria.

  • A Flags value of 0 specifies that an exact match is required.

  • A Flags value of 1 specifies that the event type must start with the string specified in the Type tag. The value 1 is also used when indicating Type value asterisk (*). This value matches all event types.

    Important:

    Search order is important. The fields included with the flist are the fields specified in the first event map section of the XML file where the event type matches the string in the Type field.

You can use the sample XML fields list (BRM_Home/sys/data/config/pin_config_rtp_trim_flist.xml) as a base for your custom XML file.

For a detailed example using session event filters, see "Usage Example".

Mapping Events to Flists

Because one flist can be used by more than one event, you can specify the relationship between an event and the flist.

For example, the following section is of an event map XML file:

<EventMap>
      <Event>
        <Type>/event/session</Type>
        <Flags>0</Flags>
      </Event>
      <Event>
        <Type>/event/session/</Type>
        <Flags>1</Flags>
      </Event>
  

is mapped to an flist as follows:

0 PIN_FLD_EVENT_MAP               ARRAY [0] allocated 20, used 8
1     PIN_FLD_EVENTS              ARRAY [0] allocated 20, used 8
2         PIN_FLD_EVENT_TYPE            STR [0] "/event/session"
2         PIN_FLD_FLAGS                 INT [0] 0
1     PIN_FLD_EVENTS              ARRAY [1] allocated 20, used 8
2         PIN_FLD_EVENT_TYPE            STR [0] "/event/session/"
2         PIN_FLD_FLAGS                 INT [0] 1

Usage Example

An unmodified flist might look like the sample shown in "Sample Unmodified Flist". However, in this example, Pipeline Manager only requires subsets of fields listed in "Sample Fields Required by Pipeline Manager" depending on the event type.

In this example, to implement the trimmed flist:

  1. Create the XML file shown in "sample.xml File" to modify the default list of fields ("Sample Unmodified Flist") included in the flist.

  2. Load the XML file using the utility:

    load_pin_rtp_trim_flist -f sample.xml [-v] [-d]
    

Sample Unmodified Flist

The following is the default (untrimmed) list of fields sent to Pipeline Manager.

0 PIN_FLD_POID           POID [0] 0.0.0.1 /event/session -1 0
0 PIN_FLD_EVENT        SUBSTRUCT [0] allocated 25, used 25
1     PIN_FLD_POID           POID [0] 0.0.0.1 /event/session -1 0
1     PIN_FLD_NAME            STR [0] "Activity Session Log"
1     PIN_FLD_USERID         POID [0] 0.0.0.1 /service/pcm_client 1 0
1     PIN_FLD_ACCOUNT_OBJ    POID [0] 0.0.0.1 /account 10243 0
1     PIN_FLD_PROGRAM_NAME    STR [0] "testnap"
1     PIN_FLD_START_T      TSTAMP [0] (1065785673) Fri Oct 10 04:34:33 2003
1     PIN_FLD_END_T        TSTAMP [0] (1065785683) Fri Oct 10 04:34:43 2003
1     PIN_FLD_SERVICE_OBJ    POID [0] 0.0.0.1 /service/ip 11907 1
1     PIN_FLD_SYS_DESCR       STR [0] "Session: generic"
1     PIN_FLD_RUM_NAME        STR [0] "Duration"
1     PIN_FLD_UNIT           ENUM [0] 1
1     PIN_FLD_TOD_MODE       ENUM [0] 2
1     PIN_FLD_NET_QUANTITY DECIMAL [0] 60.000000000000000
1     PIN_FLD_MIN_QUANTITY DECIMAL [0] 60.000000000000000
1     PIN_FLD_INCR_QUANTITY DECIMAL [0] 60.000000000000000
1     PIN_FLD_MIN_UNIT       ENUM [0] 2
1     PIN_FLD_INCR_UNIT      ENUM [0] 2
1     PIN_FLD_ROUNDING_MODE   ENUM [0] 1
1     PIN_FLD_TIMEZONE_MODE   ENUM [0] 1
1     PIN_FLD_RATED_TIMEZONE_ID    STR [0] "GMT-08:00"
1     PIN_FLD_TIMEZONE_ADJ_START_T TSTAMP [0] (1065760473) Thu Oct 09 21:34:33 2003
1     PIN_FLD_TIMEZONE_ADJ_END_T TSTAMP [0] (1065760483) Thu Oct 09 21:34:43 2003
1     PIN_FLD_TOTAL         ARRAY [840] allocated 20, used 1
2         PIN_FLD_AMOUNT       DECIMAL [0] 0.0166667
1     PIN_FLD_BAL_IMPACTS   ARRAY [0] allocated 20, used 17
2         PIN_FLD_ACCOUNT_OBJ    POID [0] 0.0.0.1 /account 10243 13
2         PIN_FLD_AMOUNT       DECIMAL [0] 0.0166667
2         PIN_FLD_RESOURCE_ID     INT [0] 840
2         PIN_FLD_PRODUCT_OBJ    POID [0] 0.0.0.1 /product 10030 0
2         PIN_FLD_RATE_OBJ       POID [0] 0.0.0.1 /rate 9390 1
2         PIN_FLD_DISCOUNT     DECIMAL [0] 0
2         PIN_FLD_AMOUNT_DEFERRED DECIMAL [0] 0
2         PIN_FLD_GL_ID           INT [0] 104
2         PIN_FLD_IMPACT_TYPE    ENUM [0] 1
2         PIN_FLD_QUANTITY     DECIMAL [0] 60.00000000
2         PIN_FLD_RATE_TAG        STR [0] "$1 per hour"
2         PIN_FLD_TAX_CODE        STR [0] ""
2         PIN_FLD_IMPACT_CATEGORY    STR [0] "default"
2         PIN_FLD_PACKAGE_ID    INT [0] "20030910"
2         PIN_FLD_LINEAGE         STR [0] ""
2         PIN_FLD_PERCENT      DECIMAL [0] 1
2         PIN_FLD_BAL_GRP_OBJ    POID [0] 0.0.0.1 /balance_group 8323 4
1     PIN_FLD_UNRATED_QUANTITY DECIMAL [0] 0
0 PIN_FLD_DISCOUNTS     ARRAY [0] allocated 20, used 8
1     PIN_FLD_ACCOUNT_OBJ    POID [0] 0.0.0.1 /account 10243 0
1     PIN_FLD_OWNER_OBJ      POID [0] 0.0.0.1 /service/ip 11907 1
1     PIN_FLD_BAL_GRP_OBJ    POID [0] 0.0.0.1 /balance_group 8323 4
1     PIN_FLD_DISCOUNT_LIST  ARRAY [0] allocated 20, used 19
2         PIN_FLD_CREATED_T    TSTAMP [0] (1063218065) Wed Sep 10 11:21:05 2003
2         PIN_FLD_CYCLE_END_T  TSTAMP [0] (0) <null>
2         PIN_FLD_CYCLE_START_T TSTAMP [0] (1052871608) Tue May 13 17:20:08 2003
2         PIN_FLD_DEAL_OBJ       POID [0] 0.0.0.0  0 0
2         PIN_FLD_DESCR           STR [0] ""
2         PIN_FLD_DISCOUNT_OBJ   POID [0] 0.0.0.1 /discount 8273 0
2         PIN_FLD_LAST_MODIFIED_T TSTAMP [0] (1063218065) Wed Sep 10 11:21:05 2003
2         PIN_FLD_PACKAGE_ID    INT [0] "12222"
2         PIN_FLD_PLAN_OBJ       POID [0] 0.0.0.0  0 0
2         PIN_FLD_PURCHASE_END_T TSTAMP [0] (0) <null>
2         PIN_FLD_PURCHASE_START_T TSTAMP [0] (1052871608) Tue May 13 17:20:08 2003
2         PIN_FLD_QUANTITY     DECIMAL [0] 1
2         PIN_FLD_SERVICE_OBJ    POID [0] 0.0.0.0  0 0
2         PIN_FLD_STATUS         ENUM [0] 1
2         PIN_FLD_STATUS_FLAGS    INT [0] 1
2         PIN_FLD_USAGE_END_T  TSTAMP [0] (0) <null>
2         PIN_FLD_USAGE_START_T TSTAMP [0] (1052871608) Tue May 13 17:20:08 2003
2         PIN_FLD_FLAGS           INT [0] 1
2         PIN_FLD_TYPE           ENUM [0] 602
1     PIN_FLD_DISCOUNT_LIST  ARRAY [1] allocated 20, used 19
2         PIN_FLD_CREATED_T    TSTAMP [0] (1063218065) Wed Sep 10 11:21:05 2003
2         PIN_FLD_CYCLE_END_T  TSTAMP [0] (1071385462) Sat Dec 13 23:04:22 2003
2         PIN_FLD_CYCLE_START_T TSTAMP [0] (1052895862) Wed May 14 00:04:22 2003
2         PIN_FLD_DEAL_OBJ       POID [0] 0.0.0.0  0 0
2         PIN_FLD_DESCR           STR [0] ""
2         PIN_FLD_DISCOUNT_OBJ   POID [0] 0.0.0.1 /discount 11345 0
2         PIN_FLD_LAST_MODIFIED_T TSTAMP [0] (1063218065) Wed Sep 10 11:21:05 2003
2         PIN_FLD_PACKAGE_ID    INT [0] "22222"
2         PIN_FLD_PLAN_OBJ       POID [0] 0.0.0.0  0 0
2         PIN_FLD_PURCHASE_END_T TSTAMP [0] (1068793462) Thu Nov 13 23:04:22 2003
2         PIN_FLD_PURCHASE_START_T TSTAMP [0] (1052871608) Tue May 13 17:20:08 2003
2         PIN_FLD_QUANTITY     DECIMAL [0] 1
2         PIN_FLD_SERVICE_OBJ    POID [0] 0.0.0.1 /service/ip 11907 1
2         PIN_FLD_STATUS         ENUM [0] 1
2         PIN_FLD_STATUS_FLAGS    INT [0] 1
2         PIN_FLD_USAGE_END_T  TSTAMP [0] (1068793462) Thu Nov 13 23:04:22 2003
2         PIN_FLD_USAGE_START_T TSTAMP [0] (1052871608) Tue May 13 17:20:08 2003
2         PIN_FLD_FLAGS           INT [0] 1
2         PIN_FLD_TYPE           ENUM [0] 602
1     PIN_FLD_DISCOUNT_LIST  ARRAY [2] allocated 28, used 28
2         PIN_FLD_POID           POID [0] 0.0.0.1 /discount 8219 1
2         PIN_FLD_CREATED_T    TSTAMP [0] (1064333980) Tue Sep 23 09:19:40 2003
2         PIN_FLD_MOD_T        TSTAMP [0] (1061399955) Wed Aug 20 10:19:15 2003
2         PIN_FLD_READ_ACCESS     STR [0] "B"
2         PIN_FLD_WRITE_ACCESS    STR [0] "S"
2         PIN_FLD_ACCOUNT_OBJ    POID [0] 0.0.0.1 /account 1 1
2         PIN_FLD_DESCR           STR [0] ""
2         PIN_FLD_END_T        TSTAMP [0] (1069333980) Thu Nov 20 05:13:00 2003
2         PIN_FLD_MODE           ENUM [0] 801
2         PIN_FLD_NAME            STR [0] "System discount 1"
2         PIN_FLD_OWN_MAX      DECIMAL [0] 0
2         PIN_FLD_OWN_MIN      DECIMAL [0] 0
2         PIN_FLD_PERMITTED       STR [0] ""
2         PIN_FLD_PRIORITY     DECIMAL [0] 1
2         PIN_FLD_PURCHASE_MAX DECIMAL [0] 0
2         PIN_FLD_PURCHASE_MIN DECIMAL [0] 0
2         PIN_FLD_START_T      TSTAMP [0] (1061399955) Wed Aug 20 10:19:15 2003
2         PIN_FLD_TYPE           ENUM [0] 603
2         PIN_FLD_USAGE_MAP     ARRAY [0] allocated 20, used 4
3             PIN_FLD_DISCOUNT_MODEL    STR [0] "DMStandard"
3             PIN_FLD_EVENT_TYPE      STR [0] "/event"
3             PIN_FLD_FLAGS           INT [0] 0
3             PIN_FLD_SNOWBALL_FLAG    INT [0] 0
2         PIN_FLD_DISCOUNT_OBJ   POID [0] 0.0.0.1 /discount 8219 1
2         PIN_FLD_SERVICE_OBJ    POID [0] NULL poid pointer
2         PIN_FLD_PACKAGE_ID    INT [0] ""
2         PIN_FLD_PURCHASE_START_T TSTAMP [0] (1061399955) Wed Aug 20 10:19:15 2003
2         PIN_FLD_USAGE_START_T TSTAMP [0] (1061399955) Wed Aug 20 10:19:15 2003
2         PIN_FLD_PURCHASE_END_T TSTAMP [0] (1069333980) Thu Nov 20 05:13:00 2003
2         PIN_FLD_USAGE_END_T  TSTAMP [0] (1069333980) Thu Nov 20 05:13:00 2003
2         PIN_FLD_STATUS         ENUM [0] 1
2         PIN_FLD_FLAGS           INT [0] 1
1     PIN_FLD_DISCOUNT_LIST  ARRAY [3] allocated 28, used 28
2         PIN_FLD_POID           POID [0] 0.0.0.1 /discount 9755 1
2         PIN_FLD_CREATED_T    TSTAMP [0] (1064334036) Tue Sep 23 09:20:36 2003
2         PIN_FLD_MOD_T        TSTAMP [0] (1061399955) Wed Aug 20 10:19:15 2003
2         PIN_FLD_READ_ACCESS     STR [0] "B"
2         PIN_FLD_WRITE_ACCESS    STR [0] "S"
2         PIN_FLD_ACCOUNT_OBJ    POID [0] 0.0.0.1 /account 1 1
2         PIN_FLD_DESCR           STR [0] ""
2         PIN_FLD_END_T        TSTAMP [0] (1069334036) Thu Nov 20 05:13:56 2003
2         PIN_FLD_MODE           ENUM [0] 801
2         PIN_FLD_NAME            STR [0] "Sys discount 3"
2         PIN_FLD_OWN_MAX      DECIMAL [0] 0
2         PIN_FLD_OWN_MIN      DECIMAL [0] 0
2         PIN_FLD_PERMITTED       STR [0] ""
2         PIN_FLD_PRIORITY     DECIMAL [0] 14
2         PIN_FLD_PURCHASE_MAX DECIMAL [0] 0
2         PIN_FLD_PURCHASE_MIN DECIMAL [0] 0
2         PIN_FLD_START_T      TSTAMP [0] (1061399955) Wed Aug 20 10:19:15 2003
2         PIN_FLD_TYPE           ENUM [0] 603
2         PIN_FLD_USAGE_MAP     ARRAY [0] allocated 20, used 4
3             PIN_FLD_DISCOUNT_MODEL    STR [0] "DMStandard"
3             PIN_FLD_EVENT_TYPE      STR [0] "/event/session"
3             PIN_FLD_FLAGS           INT [0] 1
3             PIN_FLD_SNOWBALL_FLAG    INT [0] 0
2         PIN_FLD_DISCOUNT_OBJ   POID [0] 0.0.0.1 /discount 9755 1
2         PIN_FLD_SERVICE_OBJ    POID [0] NULL poid pointer
2         PIN_FLD_PACKAGE_ID    INT [0] ""
2         PIN_FLD_PURCHASE_START_T TSTAMP [0] (1061399955) Wed Aug 20 10:19:15 2003
2         PIN_FLD_USAGE_START_T TSTAMP [0] (1061399955) Wed Aug 20 10:19:15 2003
2         PIN_FLD_PURCHASE_END_T TSTAMP [0] (1069334036) Thu Nov 20 05:13:56 2003
2         PIN_FLD_USAGE_END_T  TSTAMP [0] (1069334036) Thu Nov 20 05:13:56 2003
2         PIN_FLD_STATUS         ENUM [0] 1
2         PIN_FLD_FLAGS           INT [0] 1
1     PIN_FLD_DISCOUNT_LIST  ARRAY [4] allocated 28, used 28
2         PIN_FLD_POID           POID [0] 0.0.0.1 /discount 11291 1
2         PIN_FLD_CREATED_T    TSTAMP [0] (1064334029) Tue Sep 23 09:20:29 2003
2         PIN_FLD_MOD_T        TSTAMP [0] (1061399955) Wed Aug 20 10:19:15 2003
2         PIN_FLD_READ_ACCESS     STR [0] "B"
2         PIN_FLD_WRITE_ACCESS    STR [0] "S"
2         PIN_FLD_ACCOUNT_OBJ    POID [0] 0.0.0.1 /account 1 1
2         PIN_FLD_DESCR           STR [0] ""
2         PIN_FLD_END_T        TSTAMP [0] (1069334029) Thu Nov 20 05:13:49 2003
2         PIN_FLD_MODE           ENUM [0] 801
2         PIN_FLD_NAME            STR [0] "Sys discount 2"
2         PIN_FLD_OWN_MAX      DECIMAL [0] 0
2         PIN_FLD_OWN_MIN      DECIMAL [0] 0
2         PIN_FLD_PERMITTED       STR [0] ""
2         PIN_FLD_PRIORITY     DECIMAL [0] 200
2         PIN_FLD_PURCHASE_MAX DECIMAL [0] 0
2         PIN_FLD_PURCHASE_MIN DECIMAL [0] 0
2         PIN_FLD_START_T      TSTAMP [0] (1061399955) Wed Aug 20 10:19:15 2003
2         PIN_FLD_TYPE           ENUM [0] 603
2         PIN_FLD_USAGE_MAP     ARRAY [0] allocated 20, used 4
3             PIN_FLD_DISCOUNT_MODEL    STR [0] "DMStandard"
3             PIN_FLD_EVENT_TYPE      STR [0] "/event/session"
3             PIN_FLD_FLAGS           INT [0] 1
3             PIN_FLD_SNOWBALL_FLAG    INT [0] 0
2         PIN_FLD_DISCOUNT_OBJ   POID [0] 0.0.0.1 /discount 11291 1
2         PIN_FLD_SERVICE_OBJ    POID [0] NULL poid pointer
2         PIN_FLD_PACKAGE_ID    STR [0] ""
2         PIN_FLD_PURCHASE_START_T TSTAMP [0] (1061399955) Wed Aug 20 10:19:15 2003
2         PIN_FLD_USAGE_START_T TSTAMP [0] (1061399955) Wed Aug 20 10:19:15 2003
2         PIN_FLD_PURCHASE_END_T TSTAMP [0] (1069334029) Thu Nov 20 05:13:49 2003
2         PIN_FLD_USAGE_END_T  TSTAMP [0] (1069334029) Thu Nov 20 05:13:49 2003
2         PIN_FLD_STATUS         ENUM [0] 1
2         PIN_FLD_FLAGS           INT [0] 1
0 PIN_FLD_BAL_INFO      ARRAY [0] allocated 20, used 3
1     PIN_FLD_BAL_GRP_OBJ    POID [0] 0.0.0.1 /balance_group 8323 4
1     PIN_FLD_BALANCES      ARRAY [840] allocated 11, used 6
2         PIN_FLD_NEXT_BAL     DECIMAL [0] 0
2         PIN_FLD_RESERVED_AMOUNT DECIMAL [0] 0
2         PIN_FLD_CURRENT_BAL  DECIMAL [0] 19.590836
2         PIN_FLD_CREDIT_LIMIT DECIMAL [0] 100
2         PIN_FLD_CREDIT_FLOOR DECIMAL [0] 0
2         PIN_FLD_CREDIT_THRESHOLDS    INT [0] 0
1     PIN_FLD_BALANCES      ARRAY [1000001] allocated 7, used 6
2         PIN_FLD_NEXT_BAL     DECIMAL [0] 0
2         PIN_FLD_RESERVED_AMOUNT DECIMAL [0] 0
2         PIN_FLD_CURRENT_BAL  DECIMAL [0] 0
2         PIN_FLD_CREDIT_LIMIT DECIMAL [0] 100
2         PIN_FLD_CREDIT_FLOOR DECIMAL [0] 0
2         PIN_FLD_CREDIT_THRESHOLDS    INT [0] 0
0 PIN_FLD_INHERITED_INFO SUBSTRUCT [0] allocated 32, used 32
1     PIN_FLD_POID           POID [0] 0.0.0.1 /account 10243 13
1     PIN_FLD_MOD_T        TSTAMP [0] (1063218065) Wed Sep 10 11:21:05 2003
1     PIN_FLD_ACCOUNT_NO      STR [0] "0.0.0.1-10243"
1     PIN_FLD_BRAND_OBJ      POID [0] 0.0.0.1 /account 1 0
1     PIN_FLD_TIMEZONE_ID     STR [0] ""
1     PIN_FLD_STATUS         ENUM [0] 10100
1     PIN_FLD_STATUS_FLAGS    INT [0] 0
1     PIN_FLD_CURRENCY        INT [0] 840
1     PIN_FLD_CURRENCY_SECONDARY    INT [0] 0
1     PIN_FLD_GROUP_OBJ      POID [0] 0.0.0.0  0 0
1     PIN_FLD_CLOSE_WHEN_T TSTAMP [0] (0) <null>
1     PIN_FLD_ITEM_POID_LIST    STR [0] "0.0.0.1|/item/misc 8835 0"
1     PIN_FLD_NEXT_ITEM_POID_LIST    STR [0] ""
1     PIN_FLD_ACTG_TYPE      ENUM [0] 2
1     PIN_FLD_LAST_STATUS_T TSTAMP [0] (1063217469) Wed Sep 10 11:11:09 2003
1     PIN_FLD_GL_SEGMENT      STR [0] "."
1     PIN_FLD_BILL_WHEN       INT [0] 1
1     PIN_FLD_PAY_TYPE       ENUM [0] 10001
1     PIN_FLD_AR_BILLINFO_OBJ   POID [0] 0.0.0.1 /billinfo 8451 0
1     PIN_FLD_NEXT_BILL_OBJ   POID [0] 0.0.0.0  0 0
1     PIN_FLD_NEXT_BILL_T  TSTAMP [0] (1065769200) Fri Oct 10 00:00:00 2003
1     PIN_FLD_LAST_BILL_T  TSTAMP [0] (1063217469) Wed Sep 10 11:11:09 2003
1     PIN_FLD_ACTG_LAST_T  TSTAMP [0] (1063217469) Wed Sep 10 11:11:09 2003
1     PIN_FLD_ACTG_FUTURE_T TSTAMP [0] (1068451200) Mon Nov 10 00:00:00 2003
1     PIN_FLD_BILL_ACTGCYCLES_LEFT    INT [0] 1
1     PIN_FLD_PAYINFO_OBJ    POID [0] 0.0.0.1 /payinfo/invoice 11267 0
1     PIN_FLD_ACTG_NEXT_T  TSTAMP [0] (1065769200) Fri Oct 10 00:00:00 2003
1     PIN_FLD_LAST_BILL_OBJ   POID [0] 0.0.0.0  0 0
1     PIN_FLD_BILL_OBJ       POID [0] 0.0.0.1 /bill 10499 0
1     PIN_FLD_PENDING_RECV DECIMAL [0] 0
1     PIN_FLD_BAL_GRP_OBJ    POID [0] 0.0.0.1 /balance_group 8323 4
1     PIN_FLD_SERVICE_INFO SUBSTRUCT [0] allocated 51, used 26
2         PIN_FLD_POID           POID [0] 0.0.0.1 /service/ip 11907 5
2         PIN_FLD_CREATED_T    TSTAMP [0] (1063217471) Wed Sep 10 11:11:11 2003
2         PIN_FLD_MOD_T        TSTAMP [0] (1063217473) Wed Sep 10 11:11:13 2003
2         PIN_FLD_READ_ACCESS     STR [0] "L"
2         PIN_FLD_WRITE_ACCESS    STR [0] "L"
2         PIN_FLD_AAC_ACCESS      STR [0] ""
2         PIN_FLD_AAC_PACKAGE     STR [0] ""
2         PIN_FLD_AAC_PROMO_CODE    STR [0] ""
2         PIN_FLD_AAC_SERIAL_NUM    STR [0] ""
2         PIN_FLD_AAC_SOURCE      STR [0] ""
2         PIN_FLD_AAC_VENDOR      STR [0] ""
2         PIN_FLD_ACCOUNT_OBJ    POID [0] 0.0.0.1 /account 10243 0
2         PIN_FLD_CLOSE_WHEN_T TSTAMP [0] (0) <null>
2         PIN_FLD_EFFECTIVE_T  TSTAMP [0] (1063217469) Wed Sep 10 11:11:09 2003
2         PIN_FLD_ITEM_POID_LIST    STR [0] "0.0.0.1|/item/cycle_forward 11651 0"
2         PIN_FLD_LASTSTAT_CMNT    STR [0] ""
2         PIN_FLD_LAST_STATUS_T TSTAMP [0] (1063217469) Wed Sep 10 11:11:09 2003
2         PIN_FLD_LOGIN           STR [0] "00491732411"
2         PIN_FLD_NAME            STR [0] "PIN Service Object"
2         PIN_FLD_NEXT_ITEM_POID_LIST    STR [0] ""
2         PIN_FLD_PASSWD          STR [0] "clear|00491732411"
2         PIN_FLD_PROFILE_OBJ    POID [0] 0.0.0.0  0 0
2         PIN_FLD_STATUS         ENUM [0] 10100
2         PIN_FLD_STATUS_FLAGS    INT [0] 0
2         PIN_FLD_SERVICE_IP   SUBSTRUCT [0] allocated 20, used 3
3             PIN_FLD_COMPRESSION    ENUM [0] 0
3             PIN_FLD_IPADDR       BINSTR [0] 1 00
3             PIN_FLD_PROTOCOL       ENUM [0] 0
2         PIN_FLD_BAL_GRP_OBJ    POID [0] 0.0.0.1 /balance_group 8323 4

Sample Fields Required by Pipeline Manager

The following are sample fields, in flist format, required by Pipeline Manager when the event type is /event/session:

Important:

You cannot trim the default fields for the PIN_FLD_INHERITED_INFO substruct listed in "Sample Unmodified Flist". However, you can specify additional /account and /service fields. In the text below, the /account field PIN_FLD_RESIDENCE_FLAG is specified at the end of the list. It is added to the default PIN_FLD_INHERITED_INFO fields sent to Pipeline Manager.
0 PIN_FLD_POID           POID [0] 0.0.0.1 /event/session -1 0
0 PIN_FLD_EVENT        SUBSTRUCT [0] allocated 25, used 25
1     PIN_FLD_POID           POID [0] 0.0.0.1 /event/session -1 0
1     PIN_FLD_START_T      TSTAMP [0] (1065785673) Fri Oct 10 04:34:33 2003
1     PIN_FLD_END_T        TSTAMP [0] (1065785683) Fri Oct 10 04:34:43 2003
1     PIN_FLD_BAL_IMPACTS   ARRAY [0] allocated 20, used 17 and other array elements
2         PIN_FLD_AMOUNT       DECIMAL [0] 0.0166667
2         PIN_FLD_AMOUNT_DEFERRED DECIMAL [0] 0
2         PIN_FLD_RESOURCE_ID     INT [0] 840
2         PIN_FLD_GL_ID           INT [0] 104
2         PIN_FLD_IMPACT_TYPE    ENUM [0] 1
2         PIN_FLD_QUANTITY     DECIMAL [0] 60.00000000
2         PIN_FLD_RATE_TAG        STR [0] "$1 per hour"
2         PIN_FLD_TAX_CODE        STR [0] ""
0 PIN_FLD_DISCOUNTS     ARRAY [0] allocated 20, used 8 and other array elements
1     PIN_FLD_ACCOUNT_OBJ    POID [0] 0.0.0.1 /account 10243 0
1     PIN_FLD_OWNER_OBJ      POID [0] 0.0.0.1 /service/ip 11907 1
1     PIN_FLD_BAL_GRP_OBJ    POID [0] 0.0.0.1 /balance_group 8323 4
1     PIN_FLD_DISCOUNT_LIST  ARRAY [0] allocated 20, used 19 and other array elements
2         PIN_FLD_DISCOUNT_OBJ   POID [0] 0.0.0.1 /discount 8273 0
2         PIN_FLD_PACKAGE_ID    INT [0] "12222"
2         PIN_FLD_PURCHASE_END_T TSTAMP [0] (0) <null>
2         PIN_FLD_PURCHASE_START_T TSTAMP [0] (1052871608) Tue May 13 17:20:08 2003
2         PIN_FLD_QUANTITY     DECIMAL [0] 1
2         PIN_FLD_STATUS         ENUM [0] 1
2         PIN_FLD_USAGE_END_T  TSTAMP [0] (0) <null>
2         PIN_FLD_USAGE_START_T TSTAMP [0] (1052871608) Tue May 13 17:20:08 2003
2         PIN_FLD_FLAGS           INT [0] 1
2         PIN_FLD_TYPE           ENUM [0] 602
0 PIN_FLD_BAL_INFO      ARRAY [0] allocated 20, used 3 and other array elements
1     PIN_FLD_BAL_GRP_OBJ    POID [0] 0.0.0.1 /balance_group 8323 4
1     PIN_FLD_BALANCES      ARRAY [840] allocated 11, used 6 and other array elements
2         PIN_FLD_CURRENT_BAL  DECIMAL [0] 19.590836
1     PIN_FLD_BALANCES      ARRAY [1000001] allocated 7, used 6 and other array elements
2         PIN_FLD_CURRENT_BAL  DECIMAL [0] 0
0 PIN_FLD_INHERITED_INFO SUBSTRUCT [0] allocated 32, used 32
1     PIN_FLD_RESIDENCE_FLAG ENUM [0] 1
  

A different set of fields is required when the event type is /event/session/ (including the final forward slash), and another set of fields is sent for any other type of event.

To implement the trimmed flist in the example, create the following XML file (sample.xml). When this XML file is loaded with load_pin_rtp_trim_flist, the flist sent to Pipeline Manager is constructed as follows:

  • If the event type is exactly /event/session, the PIN_FLD_RESIDENCE_FLAG field is included with the trimmed flist as shown in the flist sample above.

  • If the event type starts with /event/session/ (including the last forward slash), the PIN_FLD_RESIDENCE_FLAG field is not included with the trimmed flist.

  • If the event type is any other value (which matches the section specified by Type value * with Flags value 1), then neither the PIN_FLD_RESIDENCE_FLAG field nor the PIN_FLD_BAL_IMPACTS array is included with the trimmed flist.

    Important:

    You cannot trim the default fields for the PIN_FLD_INHERITED_INFO substruct listed in "Sample Unmodified Flist". However, you can specify additional /account and /service fields. In the text below, the /account field PIN_FLD_RESIDENCE_FLAG is specified at the end of the list. It is added to the default PIN_FLD_INHERITED_INFO fields sent to Pipeline Manager.

sample.xml File

<?xml version="1.0" encoding="UTF-8" ?>
<!--
================================================================
Copyright (c) 2004 Portal Software, Inc. All rights reserved.
This material is the confidential property of Portal Software, Inc.
or its Subsidiaries or licensors and may be used, reproduced, stored
or transmitted only in accordance with a valid Portal license or
sublicense agreement.
================================================================
-->

<RTPTrimFlistConfiguration xmlns="http://www.portal.com/InfranetXMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.portal.com/InfranetXMLSchema pin_config_rtp_trim_flist.xsd">
  <EventMapList>
    <EventMap>

<!-- Section which specifies fields sent when the event type is exactly /event/session -->

      <Event>
        <Type>/event/session</Type>
        <Flags>0</Flags>
      </Event>
      <RequiredField>
        <Name>PIN_FLD_POID</Name>
      </RequiredField>
      <RequiredField>
        <Name>PIN_FLD_EVENT</Name>
        <RequiredField>
          <Name>PIN_FLD_POID</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_START_T</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_END_T</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_BAL_IMPACTS</Name>
          <RequiredField>
            <Name>PIN_FLD_AMOUNT</Name>
          </RequiredField>
            <RequiredField>
            <Name>PIN_FLD_AMOUNT_DEFERRED</Name> 
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_RESOURCE_ID</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_GL_ID</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_IMPACT_TYPE</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_QUANTITY</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_RATE_TAG</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_TAX_CODE</Name>
          </RequiredField>
        </RequiredField>
      </RequiredField>
      <RequiredField>
        <Name>PIN_FLD_DISCOUNTS</Name>
        <RequiredField>
          <Name>PIN_FLD_ACCOUNT_OBJ</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_OWNER_OBJ</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_BAL_GRP_OBJ</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_DISCOUNT_LIST</Name>
          <RequiredField>
            <Name>PIN_FLD_DISCOUNT_OBJ</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_PACKAGE_ID</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_PURCHASE_END_T</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_PURCHASE_START_T</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_QUANTITY</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_STATUS</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_USAGE_END_T</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_USAGE_START_T</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_FLAGS</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_TYPE</Name>
          </RequiredField>
        </RequiredField>
      </RequiredField>
      <RequiredField>
        <Name>PIN_FLD_BAL_INFO</Name>
        <RequiredField>
          <Name>PIN_FLD_BAL_GRP_OBJ</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_BALANCES</Name>
          <RequiredField>
            <Name>PIN_FLD_CURRENT_BAL</Name>
          </RequiredField>
        </RequiredField>
      </RequiredField>
      <RequiredField>
        <Name>PIN_FLD_INHERITED_INFO</Name>
        <RequiredField>
          <Name>PIN_FLD_RESIDENCE_FLAG</Name>
        </RequiredField>
      </RequiredField>
    </EventMap>

<!-- Section which specifies fields sent when the event type starts with /event/session/ -->

    </EventMap>
      <Event>
        <Type>/event/session/</Type>
        <Flags>1</Flags>
      </Event>
      <RequiredField>
        <Name>PIN_FLD_POID</Name>
      </RequiredField>
      <RequiredField>
        <Name>PIN_FLD_EVENT</Name>
        <RequiredField>
          <Name>PIN_FLD_POID</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_START_T</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_END_T</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_BAL_IMPACTS</Name>
          <RequiredField>
            <Name>PIN_FLD_AMOUNT</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_RESOURCE_ID</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_GL_ID</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_IMPACT_TYPE</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_QUANTITY</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_RATE_TAG</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_TAX_CODE</Name>
          </RequiredField>
        </RequiredField>
      </RequiredField>
      <RequiredField>
        <Name>PIN_FLD_DISCOUNTS</Name>
        <RequiredField>
          <Name>PIN_FLD_ACCOUNT_OBJ</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_OWNER_OBJ</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_BAL_GRP_OBJ</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_DISCOUNT_LIST</Name>
          <RequiredField>
            <Name>PIN_FLD_DISCOUNT_OBJ</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_PACKAGE_ID</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_PURCHASE_END_T</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_PURCHASE_START_T</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_QUANTITY</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_STATUS</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_USAGE_END_T</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_USAGE_START_T</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_FLAGS</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_TYPE</Name>
          </RequiredField>
        </RequiredField>
      </RequiredField>
      <RequiredField>
        <Name>PIN_FLD_BAL_INFO</Name>
        <RequiredField>
          <Name>PIN_FLD_BAL_GRP_OBJ</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_BALANCES</Name>
          <RequiredField>
            <Name>PIN_FLD_CURRENT_BAL</Name>
          </RequiredField>
        </RequiredField>
      </RequiredField>
    </EventMap>

<!--* Section which specifies fields sent when the event type is
    * any other value.-->

    <EventMap>
      <Event>
        <Type>*</Type>
        <Flags>1</Flags>
      </Event>
      <RequiredField>
        <Name>PIN_FLD_POID</Name>
      </RequiredField>
      <RequiredField>
        <Name>PIN_FLD_EVENT</Name>
        <RequiredField>
          <Name>PIN_FLD_POID</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_START_T</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_END_T</Name>
        </RequiredField>
      </RequiredField>
      <RequiredField>
        <Name>PIN_FLD_DISCOUNTS</Name>
        <RequiredField>
          <Name>PIN_FLD_ACCOUNT_OBJ</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_OWNER_OBJ</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_BAL_GRP_OBJ</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_DISCOUNT_LIST</Name>
          <RequiredField>
            <Name>PIN_FLD_DISCOUNT_OBJ</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_PACKAGE_ID</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_PURCHASE_END_T</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_PURCHASE_START_T</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_QUANTITY</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_STATUS</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_USAGE_END_T</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_USAGE_START_T</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_FLAGS</Name>
          </RequiredField>
          <RequiredField>
            <Name>PIN_FLD_TYPE</Name>
          </RequiredField>
        </RequiredField>
      </RequiredField>
      <RequiredField>
        <Name>PIN_FLD_BAL_INFO</Name>
        <RequiredField>
          <Name>PIN_FLD_BAL_GRP_OBJ</Name>
        </RequiredField>
        <RequiredField>
          <Name>PIN_FLD_BALANCES</Name>
          <RequiredField>
            <Name>PIN_FLD_CURRENT_BAL</Name>
          </RequiredField>
        </RequiredField>
      </RequiredField>
    </EventMap>
  </EventMapList>
</RTPTrimFlistConfiguration>

Measuring System Latencies with Instrumentation

You use the Pipeline Manager instrumentation feature to determine how much processing time each Pipeline Manager component (function modules, iScripts, and iRules) is consuming in microseconds. This information allows you to:

  • Determine system benchmarks.

  • Identify performance bottlenecks at the function module level.

  • Add or reconfigure function pools to optimize CPU utilization.

Instrumentation collects statistics for the following components:

  • The input module.

  • Each function module.

  • The output module.

After each transaction, the statistics for each pipeline tested are written to the pipeline.log file.

Using Instrumentation to Collect Module Performance Statistics

To enable instrumentation:

  1. Start the pipeline.

  2. Send a signal to the pipeline to toggle instrumentation on and off. Use the following commands to toggle the instrumentation state:

    Solaris, Linux, and AIX:

    kill -s USR1 ifw_process_pid 
    

    HP-UX IA64:

    kill -USR1 ifw_process_pid 
    

    AIX:

    kill -s USR1 ifw_process_pid 
    

    At the end of each transaction, the statistics are logged to the pipeline.log file and the statistics counters are reset.

    Note:

    By default, Pipeline Manager instrumentation is disabled on startup. When Pipeline Manager is running, you can toggle between the disabled and enabled modes.

    Important:

    Pipeline Manager begins gathering statistics immediately after receiving the signal. To assure accurate measurements, be sure that Pipeline Manager is not processing transactions when the signal is sent.
  3. Process a sample CDR file.

  4. Check the pipeline log files for processing time statistics. See "Viewing Instrumentation Testing Results" for more information.

  5. When testing is complete, stop the instrumentation process by sending another signal. See step 2.

Viewing Instrumentation Testing Results

Each log file record consists of the fully qualified module name and the accumulated processing time spent in the module.

Note:

Pipeline processing time statistics are not cumulative. The output module writes data to a file whereas a function module processes EDRs in a different thread.

Sample Log File

The following sample log file shows instrumentation data:

15.03.2004 13:25:07 test         ifw          IFW       NORMAL   00516 - (ifw.Pipelines.ALL_RATE.Functions.PreProcessing) Plugin processing time statistics: 'ifw.Pipelines.ALL_RATE.Functions.PreProcessing.FunctionPool.APNMap
.Module, 7676390
ifw.Pipelines.ALL_RATE.Functions.PreProcessing.FunctionPool.CustomerRating.Module, 22629863
ifw.Pipelines.ALL_RATE.Functions.PreProcessing.FunctionPool.CustomerSearch.Module, 239523272
ifw.Pipelines.ALL_RATE.Functions.PreProcessing.FunctionPool.EventDiscarding.Module, 19874050
ifw.Pipelines.ALL_RATE.Functions.PreProcessing.FunctionPool.PreRatingZone.Module, 18751824
ifw.Pipelines.ALL_RATE.Functions.PreProcessing.FunctionPool.PreRecycle.Module, 1916139
ifw.Pipelines.ALL_RATE.Functions.PreProcessing.FunctionPool.PrefixDesc.Module, 8001348
ifw.Pipelines.ALL_RATE.Functions.PreProcessing.FunctionPool.ServiceCodeMap.Module, 4543899
ifw.Pipelines.ALL_RATE.Functions.PreProcessing.FunctionPool.UsageClassMap.Module, 6083775
ifw.Pipelines.ALL_RATE.Functions.PreProcessing.FunctionPool.UsageScenarioMap.Module, 9786078
ifw.Pipelines.ALL_RATE.Functions.PreProcessing.FunctionPool.UsageType.Module, 57114053
'
15.03.2004 13:25:07 test         ifw          IFW       NORMAL   00516 - (ifw.Pipelines.ALL_RATE.Functions.Rating) Plugin processing time statistics: 'ifw.Pipelines.ALL_RATE.Functions.Rating.FunctionPool.AddInfranetBillingRe
cord.Module, 44927730
ifw.Pipelines.ALL_RATE.Functions.Rating.FunctionPool.MainRating.Module, 78250224
ifw.Pipelines.ALL_RATE.Functions.Rating.FunctionPool.RateAdjustment.Module, 2358093
ifw.Pipelines.ALL_RATE.Functions.Rating.FunctionPool.Recycle.Module, 1225628
ifw.Pipelines.ALL_RATE.Functions.Rating.FunctionPool.Rejection.Module, 1785748
ifw.Pipelines.ALL_RATE.Functions.Rating.FunctionPool.ServiceOutputSplit.Module, 6480466
ifw.Pipelines.ALL_RATE.Functions.Rating.FunctionPool.SpecialDayRate.Module, 8109825
'
15.03.2004 13:25:07 test         ifw          IFW       NORMAL   00516 - (ifw.Pipelines.ALL_RATE.Output.OutputCollection) Plugin processing time statistics: 'ifw.Pipelines.ALL_RATE.Output.OutputCollection.DevNull.Module, 67
ifw.Pipelines.ALL_RATE.Output.OutputCollection.DuplicateOutput.Module, 23
ifw.Pipelines.ALL_RATE.Output.OutputCollection.FAXOutput.Module, 561
ifw.Pipelines.ALL_RATE.Output.OutputCollection.GPRSOutput.Module, 728
ifw.Pipelines.ALL_RATE.Output.OutputCollection.RejectOutput.Module, 30
ifw.Pipelines.ALL_RATE.Output.OutputCollection.SMSOutput.Module, 552
ifw.Pipelines.ALL_RATE.Output.OutputCollection.TELOutput.Module, 178434585
ifw.Pipelines.ALL_RATE.Output.OutputCollection.WAPOutput.Module, 550
Note: To aggregate the counters into a nice report, please take a look at TracePerformanceReporting

Optimizing the DAT_USC_Map Module

The DAT_USC_Map module uses the Pipeline Manager framework component (FSM) to compile data mapping rules, which are stored in the database as regular expressions. The FSM compiles the data mapping structures only during Pipeline Manager startup because the rules can contain many comparisons of mapping patterns; this impacts startup performance. You can optimize the DAT_USC_Map module to enable Pipeline Manager to serialize the data structures and restore them from the serialized format.

About Precompiling USC Mapping Rules

Not all USC mapping data is stored in a compiled format: for example, rules used to define zone models. When the DAT_USC_Map module loads event data, it reorganizes it according to zone models to enable faster searching of the data structures during run time. This increases load time and memory requirements. To reduce the impact, you can configure Pipeline Manager to serialize the data structures the first time they are loaded and then reuse the serialized version during subsequent startup operations.

When Pipeline Manager begins processing data for a given zone model, it checks to see if a precompiled data file exists for that zone model. If so, it prepares the complex data structure by using the serialized format rather than by recompiling the structure from the USC map data.

If you enable the precompiling functionality, the following data is serialized:

  • USC group

  • Usage class and usage type

  • Service code and service class

  • Wholesale zone and retail zone

    Note:

    Data that is not in the precompiled format is read from the database or file system, depending on your DAT_USC_Map module configuration. See "DAT_USC_Map" in BRM Configuring Pipeline Rating and Discounting.

For more information, see "Precompiling Usage Scenario Mapping Data".

About Filtering Mapping Rules

You use USC groups to assemble the rules that define which services and service configurations are available to the pipeline; they contain the rules for mapping the service EDR attributes to each usage class.

You can configure your system to filter mapping rules based on USC groups so only the rules in the USC groups you specify are compiled and loaded into the DAT_USC_Map module. All other rules are ignored. This is more efficient than having one zone model that uses a large number of rules.

Note:

This is necessary only when your USC mapping rules are stored in the database; if they are read from a file, the data is already organized according to USC groups.

Generally you define USC Groups to contain the mapping rules for a specific type of EDR processing. For example, say you rate telephony services and process EDRs by using three USC groups (GSM, SMS, and GPRS), each of which contains mapping rules to determine domestic standard charges, domestic roaming charges, and international charges.

To increase performance, you can define the mapping rules for each set of charges in a separate zone model. Then, when an EDR is processed, based on the USC group specified, only the rules used in those zone models are compiled and loaded. This increases startup performance.

For more information, see "Filtering the Mapping Data to Compile and Load".

For information on USC groups, see "About Usage Scenario Mapping" in BRM Setting Up Pricing and Rating.

Configuring the DAT_USC_Map Module for Startup Performance

You improve startup performance of the DAT_USC_Map module by:

  • Increasing the number of threads used to load mapping data.

  • Precompiling usage scenario mapping data.

  • Filtering the mapping data to compile and load.

You define these configurations in the Pipeline Manager registry file. For more information, see "About Configuring Pipeline Manager".

Increasing the Number of Threads Used to Load Mapping Data

The DAT_USC_Map module loads mapping rules for each zone model in a USC group by using a separate thread; therefore, it is only necessary to increase the number of threads when your USC groups contain multiple zone models.

To use multiple threads, set the NumberOfThreads registry entry to the desired number of threads. This enables Pipeline Manager to compile data in parallel and to restore it from the precompiled data files.

For example:

NumberOfThreads = 4
  

The default is 1.

Note:

You can use this entry as a semaphore.

Precompiling Usage Scenario Mapping Data

To enable precompiling of USC mapping data, set the PreCompiledDataDir registry entry. This entry both enables the precompile functionality and defines the location of the compiled data files. By default, compiled data files are saved in the ./compiled_usc_data directory.

Pipeline Manager saves them with the following naming convention:

USCzoneModelID.pc

For example, GSM.pc, GSM_DOMESTIC.pc, and GSM_ROAMING.pc.

If this entry is set, compiled files are created the next time the pipeline starts. For each subsequent run, the data files are validated against the data structures in the database and, if necessary, recompiled and resaved to the file system.

Note:

You can use this entry as a semaphore.

Filtering the Mapping Data to Compile and Load

If the source for your USC mapping rules is the database rather than a file, you can filter which rules are compiled and loaded into the DAT_USC_Map module when a pipeline starts by setting the UscGroupsregistry entry to one or more USC groups. For example:

UscGroups {GSM GSM_ROAMING}

Important:

You can specify only one USC group per each pipeline running in your system. If you use multiple USC groups, you must configure Pipeline Manager to run multiple pipeline instances. To do this, configure the Pipeline Manager registry file so the FCT_USC_Map in each pipeline instance refers to the appropriate DAT_USC_Map module reference and UscGroups entry. For more information, see "About Configuring Pipeline Manager".

By default, all mapping rules are loaded into the pipeline. see "About Filtering Mapping Rules" for more information.

Note:

You can use this entry as a semaphore.

Using Semaphores

You can use the new NumberOfThreads, PreCompiledDataDir, and UscGroups registry entries as semaphores to configure and control Pipeline Manager during pipeline startup. These semaphores perform the same tasks that the Reload semaphore performs, as specified in the startup registry or last-processed semaphore:

  1. Load mapping data from the source (Database or File).

  2. Create the USC zone model (from data in PreCompiledDataDir or USCMapFile).

  3. Compile or precompile each USC zone model.

When you change the values of these semaphores after startup, they are not updated automatically in your system; you must use the Reload semaphore to update them during run time.

For example:

  • To use multiple threads to load data, edit the NumberOfThreads semaphore and then call the Reload semaphore. Each thread processes a different zone model when loading the USC data.

  • To reload USC data using a different set of files in the PreCompiledDataDir directory, edit the PreCompiledDataDir semaphore and then call the Reload semaphore.

  • To filter a different set of mapping rules, edit the UscGroups semaphore and then call the Reload semaphore.

For more information on the DAT_USC_Map semaphore entries, see "Semaphore File Entries" in BRM Configuring Pipeline Rating and Discounting.

Other Pipeline Manager Monitoring Tools

This section describes additional Pipeline Manager performance-monitoring tools.

Viewing the Pipeline Log

You can see the results of tests for each pipeline in that pipeline's pipeline.log file.

Tip:

Open each log file in a terminal windows and run tail -f on the logs.

After each batch stream is processed, the pipeline writes the following information to the pipeline.log files:

  • The number of processed EDRs.

  • The number of errors that occurred during EDR processing.

  • The number of EDRs processed per second for a stream.

  • If instrumentation is on, the instrumentation results. See "Viewing Instrumentation Testing Results" for more information.

Tuning Tips

  • Let the system process a few files before you measure performance. This assures that any additional memory needed (for example, for the buffers) has been allocated.

  • Use the system monitor tool to monitor system utilization.

Configuring Buffer Size Polling

Use the QueueRequestTimeout Controller entry in the registry to specify the interval in seconds that each queue's fill status is written to the log. For example:

  ifw 
  { 
    Active = TRUE 
    ProcessLoopTimeout = 10 
    QueueRequestTimeout = 10 # Optional, 0 disables
    ...
  

The default is 0 (no polling).

Buffer fill status information can indicate which function pool is the slowest. Over time, buffers in front of the slowest function pool fill up, and those that occur later in the stream are empty.

Note:

Instrumentation is the recommended tool for identifying the slowest function pool. See "Measuring System Latencies with Instrumentation".

OS-Specific Pipeline Manager Monitoring Tools

This section describes OS-specific tools that you can use to monitor and maintain your Pipeline Manager system.

Solaris Monitoring Tools

This section describes Solaris monitoring tools.

Displaying Thread Details

To display details of the threads within a process, use the prstat command:

prstat -Lmv -p
  

Example output:

prstat -Lmv -p 22376
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID 
 22376 integ     86  13 0.0 0.0 0.0 0.0 0.0 0.9  12  3K .1M   0 ifw/4
 22376 integ     61  34 0.0 0.0 0.0 0.5 2.8 2.0 298  1K 64K   0 ifw/16
 22376 integ     52 0.8 0.0 0.0 0.0  42 0.0 4.9  56 11K 11K   0 ifw/117
 22376 integ     43 3.6 0.0 0.0 0.0  52 0.0 1.8 158  1K  7K   0 ifw/5
 22376 integ     22 0.1 0.0 0.0 0.0  75 0.0 2.6 393 125 463   0 ifw/116
 22376 integ     21 0.1 0.0 0.0 0.0  77 0.0 2.6  89 357 412   0 ifw/115
...
Total: 1 processes, 48 lwps, load averages: 4.18, 1.94, 2.38
  

In the pipeline process.log file, you can see when a thread is created, its name, and corresponding OS number:

09.12.2003 19:54:38 igscoll1     ifw          IFW       NORMAL   00000 - 
(ifw.ProcessLog.Module) Thread instance ID '2'; and 
Name 'ifw.ProcessLog.Module'.
...
09.12.2003 20:01:31 igscoll1     ifw          IFW       NORMAL   00000 - 
(ifw.Pipelines.GSM.Input) Thread instance ID '16'; and 
Name 'ifw.Pipelines.GSM.Input'.
...
09.12.2003 21:38:40 igscoll1     ifw          IFW       NORMAL   00000 - 
(ifw.Pipelines.GSM.Functions.PreProcessing2) Thread instance ID '135'; and 
Name 'ifw.Pipelines.GSM.Functions.PreProcessing2'.
...

Dumping the Call Stack of an Active Process

To dump the call stack of an active process, use the vmstat and mpstat pstack commands.

Identifying Thread Functions

To identify what each thread is used for, such as the pipeline framework, input, output, or function modules, use the mpstat command:

mpstat pstack

Tip:

To determine the correlation between pipeline thread names and thread numbers, see the pipeline.log file.

HP-UX IA64 Monitoring Tools

Tools useful for monitoring Pipeline Manager on HP-UX IA64 systems include:

  • glance

  • sar -AM

  • top

  • vmstat

  • iostat

For more information on these and other tools, see the HP documentation.

Linux Monitoring Tools

Tools useful for monitoring Pipeline Manager on Linux systems include:

  • vmstat

  • sar

  • top

  • pmap

  • gnome-system-monitor

  • sysstat package (iostat, mpstat, sadc, and sar)

AIX Monitoring Tools

Tools useful for monitoring Pipeline Manager on AIX systems include:

  • sar

  • vmstat

  • iostat

  • filemon

  • topas

  • trace

  • svmon

  • netpmon