5 Configuring EDR Preprocessing

This chapter describes how to configure the modules used for preprocessing event data records (EDRs) in the Oracle Communications Billing and Revenue Management (BRM) Pipeline Manager; for example, it describes how to discard duplicate EDRs.

Before reading this document, you should understand how Pipeline Manager works and how to configure it. See the following:

For information about recycling EDRs, see "About the EDR Recycling Features".

Handling Duplicate EDRs

Use the FCT_DuplicateCheck module to find EDRs that have already been processed and send them to a special output stream. This prevents you from charging a customer twice for the same usage.

As a pipeline processes EDRs, the following occurs:

  1. The FCT_DuplicateCheck module keeps a record of EDRs that have already been processed.

  2. As new EDRs are processed, the module checks for duplicate EDRs by comparing data in the incoming EDRs with data in the EDRs that have already been processed.

    The FCT_DuplicateCheck module uses the following criteria to check for duplicate EDRs:

    • The date of an EDR. If an EDR is older than a certain date, it is not processed and no further checking is performed. It is unlikely that a duplicate EDR will be processed much later than an original EDR.

    • The data in an EDR. You configure which data to use when comparing a new EDR against EDRs that have already been processed.

    • A search key. An EDR is considered a duplicate if it has the same search key and the same values contained in the fields used for data comparison.

  3. If an EDR is a duplicate, it is flagged with an error so other modules do not need to process it, and it is moved to a separate output stream.

Configuring Duplicate EDR Checking

To enable duplicate EDR checking, configure the FCT_DuplicateCheck module. See "FCT_DuplicateCheck" and the following topics:

To define the output stream for duplicate EDRs, use the Output configuration in the registry. You typically use the OUT_Reject module. See "Sample Output Configuration" and "OUT_Reject" for more information.

Important:

When you enable duplicate checking, you must also set the transaction manager RedoEnabled entry to True. See "About Cancelling Transactions When a Rollback Occurs" in BRM System Administrator's Guide and "Transaction Manager" for more information.

Setting Date Parameters for Storing Processed EDRs

To check for duplicate EDRs, you need to store a record of the EDRs that have already been processed. The FCT_DuplicateCheck module then checks incoming EDRs against the previously processed EDRs. If you receive a high volume of EDRs, you cannot store a record of all EDRs. Therefore, to limit the number of EDRs you store, you specify date criteria. If an EDR is older than the specified date, it is not processed.

Note:

The date of an EDR is derived from its DETAIL.CHARGING_START_TIMESTAMP field.

To specify date parameters for EDR storage, you use the following date settings, specified in the FCT_DuplicateCheck registry. Depending on an EDR's date relative to these settings and on whether the FCT_DuplicateCheck module is connected to the database, the EDR is ignored, stored in a file, stored in the database, or stored in memory.

  • StoreLimit: Specifies the oldest date that previously-processed EDRs can be stored. EDRs dated earlier than the StoreLimit date are ignored and not processed by the FCT_DuplicateCheck module.

  • BufferLimit: Specifies the oldest date that previously-processed EDRs can be stored in memory. All EDRs whose date is equal to or later than the BufferLimit date are stored in memory. The FCT_DuplicateCheck module searches the memory directly, thus improving performance.

The StoreLimit date must be equal to or earlier than the BufferLimit date. For example, if the StoreLimit is June 1, the BufferLimit date can be June 1 or later.

Important:

Because StoreLimit and BufferLimit are specified as absolute dates (for example, August 2, 2004), you need to change them daily. You change them by using an FCT_DuplicateCheck semaphore file entry. See "FCT_DuplicateCheck" for more information.

Specifying the Fields to Use for Duplicate Check

You configure which data to use when comparing a new EDR against EDRs that have already been processed. A typical duplicate check for a phone call compares the A number, B number, and start time in a new and processed EDR. If the data in all fields match, the new EDR is flagged as a duplicate.

To specify which fields to use, use the FCT_DuplicateCheck Fields registry entry. See "FCT_DuplicateCheck" for more information.

The following example shows a typical configuration:

Fields
{
    1  =  DETAIL.BASIC_SERVICE
    2  =  DETAIL.B_NUMBER
}

Important:

Do not use the DETAIL.CHARGING_START_TIMESTAMP field for duplicate checking.

Specifying a Search Key for Duplicate Check

The duplicate check search key identifies duplicate EDRs. An EDR is considered a duplicate if it has the same time, the same search key value, and the same values for fields listed in the IFW_DUPLICATECHECK table or in the Fields lists stored as a record in memory.

The search key is used as a key to the internal data structure. For example, if the search key is A_NUMBER, then A_NUMBER is the hash key used to find the data in memory or in a file for the EDR being checked.

Managing FCT_DuplicateCheck Data Files

The FCT_DuplicateCheck module uses data files to store EDR data while the EDRs are processed. The file name syntax is:

File_Name_Transaction Id .dat

File_Name is the value entered in the FileName registry entry:

FileName = duplicateData

The file contains the data from the fields specified in the registry and the EDR date. At the end of each transaction, the FCT_DuplicateCheck module saves the data from memory to disk in a new transaction file.

Note:

  • To store EDRs in files for duplicate checking, configure unique FileName settings, Path registry settings, or both for each module.

  • To store EDRs in the database for duplicate checking, use the FCT_DuplicateChecking module's DataConnection registry entry to connect the module to the database. See "About Storing EDRs in a Database Instead of Files".

The duplicate check transaction files should be backed up routinely when the pipeline is not processing EDRs. Temporary files, however, should not be backed up.

To restore transaction files from a backup, shut down the pipeline and restart after restoring backed up transactions files. You can restore duplicate check data without shutting down provided that the pipeline is not processing any EDRs. There is no semaphore to reload the data file, but resetting the StoreLimit or BufferLimit settings results in a reload. The values of these settings do not need to be changed from their original startup registry settings.

After an abnormal termination, temporary files may be left behind (there should be only one because file mode should only be used with a single pipeline). These files correspond to transactions that were never committed, and the input files associated with these transactions will be reprocessed upon restarting. You should delete these temporary files before restarting. Temporary files use the suffix .tmp.

For information about managing transaction rollback files, see "About Pipeline Manager Transactions" in BRM System Administrator's Guide.

About Storing EDRs in a Database Instead of Files

If you use the FCT_DuplicateCheck module's DataConnection registry entry to connect the module to the database (see "About Storing EDRs in a Database Instead of Files"), the module handles EDRs as follows:

  • EDRs whose date is equal to or later than the StoreLimit date and earlier than the BufferLimit date are stored in the database instead of in files. If an EDR is stored in the database, files are not created.

  • EDRs whose date is equal to or later than the BufferLimit date are stored in memory and in files.

    Tip:

    To avoid using excessive disk space when checking for duplicate EDRs, use the FCT_DuplicateCheck module's DataConnection registry entry to connect the module to the database. (The StoreLimit and BufferLimit date settings, and connecting the module to the database, are designed to help maintain performance without overloading memory.)

When the FCT_DuplicateCheck module is connected to the database, an entry is added to the IFW_DUPLICATECHECK database table for each unique EDR. For each duplicate EDR, the error INF_DUPLICATE_EDR is reported and no entry is added to the table.

If the pipeline transaction is cancelled, all the rows with the current transaction ID are removed.

If you do not use the FCT_DuplicateCheck module's DataConnection registry entry to connect the module to the database (the default), the module handles EDRs as follows:

  • EDRs whose date is equal to or later than the StoreLimit date and earlier than the BufferLimit date are stored in files instead of the database. (Each EDR is assigned a value that is stored in memory. If the EDR is stored in files, the module checks the memory for the value, which points to the EDR in the file. The module then reads the EDR data from the file.)

  • EDRs whose date is equal to or later than the BufferLimit date are stored in memory and in files.

    Note:

    Although not connecting the module to the database enables faster checking for duplicate EDRs, it uses a large amount of disk space.

Create Database Tables for Duplicate Check Data

Use the FCT_DuplicateCheck TableSuffix registry entry to create multiple IFW_DUPLICATECHECK tables when you run multiple pipelines. You typically do this when you run a separate pipeline for each type of service.

For example, if the pipeline processes GSM EDRs, you can use GSM as the table suffix to use a table named IFW_DUPLICATECHECK_GSM.

You need to manually create the tables and the indexes. For example:

  • IFW_DUPLICATECHECK_GSM

  • BIDX_DUPCHK_DATA_GSM

Using Duplicate Check with Multiple Pipelines

When you use the FCT_DuplicateCheck module, you must process the EDRs for each account in the same pipeline. If you use duplicate check in multiple pipelines for the same account, duplicate calls may get processed by different pipelines and will not be recognized as being duplicate.

Suspending Duplicate EDRs

By design, a duplicate EDR is not considered as a suspended EDR and is sent to a different output stream than suspense handling stream. Therefore, duplicate EDRs are not handled by suspense handling.

If your business requires to process duplicate EDRs as suspended EDRs, you can do the following:

  1. Change the entry StreamName = DuplicateOutput to StreamName = SuspenseCreateOutput in the FCT_DuplicateCheck registry.

    This will cause the duplicate check reject file to be routed to the suspense handling stream.

  2. Add an iScript in the pipeline after duplicate check processing, to check for EDR error. If the error is duplicate check error, set the error to some other error and set error status as major.

    This will cause the duplicate EDR to be processed by FCT_Reject and FCT_Suspense plugins.

Note:

You should ensure that duplicate EDRs are associated with a specific error code to be able to manage and monitor the EDRs in Suspense Manager Center.

Assembling EDRs

Use the FCT_CallAssembling module to assemble multiple CDRs into a single EDR that pipeline modules can rate. You typically need to assemble CDRs for long phone calls or GPRS sessions that have been recorded in multiple CDRs.

The default behavior of FCT_CallAssembling is designed to assemble time duration calls. This is appropriate for wireless voice calls that are rated based on how long the call lasts. You can also configure this module to assemble calls in a manner appropriate for EDRs rated based on the volume of data sent. This is appropriate for a long data transfer session, such as downloading a movie. You can choose to collect both time duration and data volume from multiple CDRs, and a number of other matrixes, such as:

  • Volume sent

  • Volume received

  • Number of units

  • Retail charge amount

  • Wholesale charge amount

For example, a GPRS session might last for 24 hours. The network might be configured to generate an intermediate CDR every 30 minutes. This GPRS data session is recorded by several partial CDRs. If you rate by volume per session, you use the FCT_CallAssembling module to assemble the partial CDRs into one EDR before rating.

However, remember that the more metrics that you collect, the more system resources you will use.

How FCT_CallAssembling Classifies EDRs

FCT_CallAssembling uses the LONG_DURATION_INDICATOR EDR field to classify assembled calls.

The purpose of LONG_DURATION_INDICATOR changes when calls are assembled. CDRs arrive with this field set to one of these values, which identify the type of call segment:

  • F: The first segment of the call.

  • L: The last segment of the call.

  • I: An intermediate segment of the call.

After FCT_CallAssembling assembles these call segments into an EDR, it gives the EDR one of these long duration indicators to specify its status.

  • C: Complete call. The first and last segments have both arrived, enabling the call's duration to be calculated. If intermediate call segments arrive after the call is complete, they are given a long duration indicator of XC, XO, or XP and may not be rated.

  • S: A single CDR containing the entire call.

  • SL: Slice of a call. Used if KeepCallOpen = True or MaxDuration = True. Composed of the first call segment and any intermediate call segments that have arrived when the call is flushed. This part of the call is rated. When any other intermediate segments and the last segment arrive, the call is given a long duration indicator of C (complete), and these segments are rated.

  • P: Partially assembled call. Used if FlushLimit = True or KeepCallOpen = false. Partially assembled calls are rated with the information in whatever segments have arrived. Subsequent call segments are given a long duration indicator of XP and are not rated.

  • XC: Late intermediate EDR. This long duration indicator is for intermediate call segments that arrive after the call is marked complete and rated. Late intermediate call segments are not rated; instead they are used for auditing.

  • XO: Late overlap EDR. Used if time duration rating is used, and DropLateCDRs=False. This status indicates that the call segment was flushed before this CDR arrived, and it represents a time duration period already rated. These late segments are not rated; instead they are used for auditing.

  • XP: Late timeout EDR. Used if DropLateCDRs = False. This status indicates that the call timed out before this CDR arrived. These late segments are not rated; instead they are used for auditing.

Managing the Call Assembling Data Files

Open EDRs are stored in a data file. You configure the path and file name in the Path and FileName entries in the FCT_CallAssembling registry. The syntax for the file name is:

File_Name_Transaction_ID.dat

This file is read at startup and reloaded after rollback. While processing, data is stored in a temporary file.

The FCT_CallAssembling module processes the EDRs in a single transaction. You should backup the work files routinely when the pipeline is not processing any EDRs.

To restore from a backup, shut down the pipeline and restart after restoring backed up work files. Work files use the suffix .dat.

After abnormal termination, temporary files may be left behind. Temporary work files use the suffix .tmp. These correspond to transactions that never committed. Therefore, the input files associated with these transactions will be reprocessed upon restarting. Delete these temporary files before restarting.

Important:

Never delete the most recent .dat file.

For information about managing transaction rollback files, see "About Pipeline Manager Transactions" in BRM System Administrator's Guide.

Configuring Call Assembling

To configure call assembling, see "FCT_CallAssembling" and the following topics:

In addition to specifying how to perform call assembly, you can configure FCT_CallAssembling to do the following:

If you are upgrading, see "Migrating Call Assembling Data Between Releases and Pipelines".

Rating Calls by Time Duration

By default, the FCT_CallAssembling module assembles the time duration of a call so it can be rated.

To assemble EDRs for their time duration, the FCT_CallAssembling module identifies partial EDRs by using the following EDR attributes:

  • The chain reference. This ID identifies which event the partial EDR belongs to. Multiple partial EDRs that belong to the same event all have the same chain reference.

    Important:

    Chain reference must be unique for each call instance. Oracle does not support identical chain references across several call events.

  • The long duration indicator. For a list of these indicators, see "How FCT_CallAssembling Classifies EDRs".

Parts of EDRs can be processed out of order; for example, the Last segment might arrive before the First segment. The FCT_CallAssembling module manages EDRs by tracking their status:

  • As a soon as a first or intermediate call segment record arrives, the EDR is stored in a data file and the state is set to Open. See "Managing the Call Assembling Data Files".

  • By default, if a First or Last segment is already stored in a file, and the matching Last segment or First segments arrive, the record state is changed to Closed. The EDR is moved back into the pipeline. Any Intermediate segment that belong to the assembled EDR that arrive after the assembled EDR is closed are ignored.

    You can also direct this module to wait for all CDRs before closing a call, or close it after you send in a semaphore. This allows you to rate incomplete calls or calls that never receive a first or last record.

The time duration for a call is calculated as follows:

Call Duration = [(Start_Time_Of_Last_Segment) - (Start_Time_Of_First_Segment)] + Duration of the Last Segment

For example, if there is a call which starts at Jan 01, 2009 12:00:00 PM and ends at Jan 01, 2009 at 12:30:00 PM. The EDR will have the following information:

  • First Segments TimeStamp as Jan 01, 2009 12:00:00 PM and Duration

  • Last Segment TimeStamp would be Jan 01, 2009 12:25:00 PM + 300(Seconds)

In this case, the call duration will be calculated as follows:

[(12:25:00 PM) - (12:00:00 PM)] + 300 = 1800 Seconds

Rating Incomplete Time Duration Calls

Some EDRs can never be completely assembled because the First, Intermediate, or Last segment never arrives. In this situation, you can choose to close these calls using either the FlushLimit semaphore entry or the MaxDuration startup registry entry. Both of these entries force calls to be flushed and rated after the time limit you set.

Here is a comparison of the two entries:

  • MaxDuration is a startup registry entry. It takes effect when you start the pipeline. Each time a new call segment arrives, it recalculates the total time duration for that call. If the new time duration exceeds the limit you set, the call is flushed and it remains open for more CDRs. You set the time duration in seconds. To change the setting, you must restart the pipeline.

    For details, see "Using MaxDuration to Rate Incomplete Calls".

  • FlushLimit is a semaphore registry entry used with KeepCallOpen=True. It sets a maximum age a call can have before being flushed. When you send in the semaphore, the pipeline calculates whether a call exceeds the maximum age. If a call exceeds the FlushLimit setting, FCT_CallAssembling creates and rates the EDR and then closes the call. You set the time limit in days. A setting of 0 flushes all calls. A setting of 1 flushes all calls that have been opened more than 1 day. You can change the FlushLimit setting every time you send in a semaphore.

    For details, see "Using FlushLimit to Rate Incomplete Calls".

Using MaxDuration to Rate Incomplete Calls

MaxDuration is a startup registry entry that directs FCT_CallAssembling to rate segments of a wireless call periodically. This entry specifies the total amount of time duration (in seconds) that a call can have before the call segments that have arrived are rated. FCT_CallAssembling recalculates the call duration for every call each time a new call segment arrives and compares it to the MaxDuration setting. If the new time duration equals or exceeds the setting for MaxDuration, FCT_CallAssembling creates an EDR to rate the existing portion of the call.

To use the MaxDuration entry, add it to the startup registry and start (or restart) the pipeline.

Table 5-1 illustrates FCT_CallAssembling behavior with MaxDuration set. In the example, the CDRs did not arrive in chronological order.

Example call with KeepCallOpen=True and MaxDuration=28800 (8 hours)

Table 5-1 FCT_CallAssembling and CDRs

CDR Start time End time Duration (hours) FCT_CallAssembling takes this action...

F

1:00

4:00

3

Waits. MaxDuration setting not reached.

I1

4:00

7:00

3

Waits. MaxDuration setting not reached.

I2

7:00

10:00

3

Creates EDR for 9 hours of call duration. MaxDuration setting exceeded.

I3

10:00

13:00

3

Waits. MaxDuration setting not reached.

I5

16:00

19:00

3

Creates EDR for 9 hours of call duration, even though a CDR is missing. MaxDuration setting exceeded.

I4

13:00

16:00

3

Not rated; call duration already rated. This EDR is either emitted as XO or dropped, depending on the DropLateCDRs setting.

In this example, if MaxDuration is set to 28800 seconds (8 hours), FCT_CallAssembling rates the EDRs of a call with a total time duration of more than 8 hours. This call arrives in 3-hour segments and will be rated after the third segment arrives, and FCT_CallAssembling calculates that the 9-hour call duration exceeds the 8-hour limit. When the MaxDuration setting is reached, the call segments are flushed and rated, the long duration indicator for the call is set to SL, and the call is left open for more call segments.

If a CDR is missing, FCT_CallAssembling adds the missing call time represented by the EDR if it can. In the example above, the I4 call segment arrived last, but the time duration it represented had already been rated when the I5 segment arrived. FCT_CallAssembling calculated the time duration by subtracting the start time from I3 and the end time from I5. The difference was 9 hours of time duration, which exceeds the 8-hour setting, so an EDR was created to rate that duration.

Using FlushLimit to Rate Incomplete Calls

FlushLimit assembles the calls and emits them into the pipeline for rating. The call has a long duration indicator set to P for "Partial."

In this example, FlushLimit flushes all calls the have a CHARGING_START_TIMESTAMP older than five days from today.

FlushLimit=5

To flush all incomplete calls, use 0; for example:

FlushLimit=0

The flush operation does not happen immediately at the time of semaphore execution. Instead, it happens at the arrival of the next pipeline transaction. This is done to ensure that Flush operations happen within the context of a transaction.

Removing Incomplete Time Duration Calls

When you flush EDRs, they re-enter the pipeline as part of the current transaction. The EDRs are still stored in the work files (.dat and .EDR), with a state of Timeout. This prevents a late-arriving call segment from re-opening a call that has already been flushed. If you are sure that no further segments will arrive, use the RemoveLimit semaphore entry to remove calls from the work files.

The RemoveLimit entry removes all calls with the Closed or Timeout status, but leaves calls with a state of Closed_Rejected or Timeout_Rejected alone. Closed_Rejected or Timeout_Rejected calls will be recycled. However, if you are sure that calls with the Closed_Rejected or Timeout_Rejected status will not be recycled, use a RemoveRejectedLimit semaphore entry to remove these calls from the work files.

Dropping Late Calls

You can drop late EDRs from the pipeline entirely, or send them through as non-valid EDRs. If you send them through the pipeline, you can use them for auditing.

Use the DropLateCDR registry entry to specify how to handle the output of late EDRs:

  • True (Default) = Drop late EDRs from the pipeline. The EDRs are counted in the report of late-arriving EDRs.

  • False = Send late EDRs through the pipeline as non-valid. The EDRs are not counted in the report of late-arriving EDRs. They are not rated.

Rating Calls by Implied Time Duration

By default, FCT_CallAssembling calculates a call's time duration by using the difference between the first EDR's start time and the last EDR's end time. This includes the time duration for any CDRs between the first and last that have not yet arrived. The time duration for these missing CDRs is included in the EDR and rated.

If KeepCallOpen=True calls are rated in segments. When a semaphore with KeepCallOpen=True is sent in:

  • All the CDRs that have arrived for a call are assembled.

  • An EDR is emitted and rated

  • The call is kept open for more CDRs.

When the next semaphore is sent in, all CDRs that have arrived since the last semaphore are assembled as an EDR and rated.

By default, FCT_CallAssembling calculates a call's time duration by subtracting the end time of the last CDR that has arrived from the start time of the first. Because the CDRs between the first and last are not used in the calculation, it does not matter if they have arrived when the EDR is created. If a CDR between the first and last CDRs is missing when the time duration is calculated, the missing CDR is dropped when it finally does arrive.

Example time duration registry

CallAssembling
 {
  ModuleName = FCT_CallAssembling 
  Module 
   {
      Active            = True
      AssembleVolume    = False
      AssembleSGSN      = False
      SplitAtGaps       = False
      MaxDuration       = 900
      Path              = ./data/assy
      FileName          = calls
      Mode              = Normal
      RejectMissingChain = True
      CallDurationTolerance = 59
      DropLateCDRs          = False

      AssembledEDR  {
          1 = Detail.custom_fields_from_last_edr1
          2 = Detail.custom_field_from_last_edr2...
      }
   }
 }

Note:

If you use both the FCT_CallAssembling and FCT_Reject in the same pipeline, use the FCT_Reject module CallAssemblingModule registry entry to ensure that complete EDRs are recycled. See "FCT_Reject".

Rating Calls by Volume of Data Sent

Use the AssembleVolume and/or AssembleSGSN registry entries to direct FCT_CallAssembling to rate calls by the volume of data sent. Both AssembleVolume and AssembleSGSN ensure that you capture the entire volume of data sent for a single call, and they both direct FCT_CallAssembling to rate a call only after all of its call records have arrived.

If you do not plan to rate calls by volume of data sent, leave AssembleVolume set to False. This saves system resources by disabling the registry entries which rate by volume.

The volume-based rating entries protect against lost revenue. Call records can arrive in any order, so it is not unusual for an intermediate segment to arrive after the first and last segments have arrived and been rated. In this case, any intermediate segments that arrive after the call is closed are dropped, and all the volume they contain are lost (and so is the revenue).

By default, FCT_CallAssembling calculates the time duration for each CDR individually. FCT_CallAssembling subtracts the end times from the start times of each CDR to calculate the time duration. This module then adds the time durations of all CDRs as they arrive to create a grand total for the call.

If your business requires that all non-contiguous CDRs be rated as separate EDRs (TAP requires this) set the SplitAtGaps registry entry to True. If not, then set this entry to False, and non-contiguous CDRs will be collected into a single EDR, saving system resources.

However, if you are rating calls by volume of data sent, lost intermediate call records will cause calls to remain open indefinitely. Send in a Flushlimit entry frequently to avoid this.

For information on data calls that you want to keep open indefinitely and rate periodically, see "Rating Continuous Data Calls by Segment".

Example volume of data call registry

CallAssembling
 {
  ModuleName = FCT_CallAssembling 
  Module 
   {
      Active            = True
      AssembleVolume    = True
      AssembleSGSN      = True
      SplitAtGaps       = False
      Path              = ./data/assy
      FileName          = calls
      RejectMissingChain = True
      CallDurationTolerance = 59
      DropLateCDRs          = False

      AssembledEDR  {
          1 = Detail.custom_fields_from_last_edr1
          2 = Detail.custom_field_from_last_edr2...
      }
   }
 }

Example TAP volume of data call registry

This example adheres to the TAP 3.10 standard. Volume and SGSN data are recorded precisely at the start and end times of each call segment. Because SplitAtGaps = True, if any call segments are missing, the segments before and after it are emitted as separate EDRs.

CallAssembling
 {
  ModuleName = FCT_CallAssembling 
  Module 
   {
      Active            = True
      AssembleVolume    = True
      AssembleSGSN      = True
      SplitAtGaps       = True
      Path              = ./data/assy
      FileName          = calls
      RejectMissingChain = True
      CallDurationTolerance = 59
      DropLateCDRs          = False

      AssembledEDR  {
          1 = Detail.custom_fields_from_last_edr1
          2 = Detail.custom_field_from_last_edr2...
      }
   }
 }

Specifying a Time Error

Use the CallDurationTolerance startup registry entry to specify an amount of time (in seconds) that is an acceptable amount of time error for each call.

Important:

The default value for this entry correctly handles most calls, so you do not need to change this entry unless you notice a problem.

Mobile phone calls are commonly split into multiple call records. Each call record should start at the same second that the previous one ended. For example, a call record with an end time of 12:01:30 should be followed by the next call record with a start time of 12:01:30. Unfortunately, it is common for these start and end times to either not quite match, or to overlap slightly (time error). The reason may be that the call records come from different routing switches with clocks that have not been synchronized, or the switches themselves have difference time tolerances.

The CallDurationTolerance default value is 60 seconds of tolerance to compensate for this time error. If a call has less than 60 seconds of error, a call is considered complete and sent for rating. Otherwise the call is left open and must be closed with a semaphore entry. A single call with 10 call records, each overlapping by 3 seconds, creates a call with 30 seconds of time error. This 30-second time error is less than the CallDurationTolerance 60-second time limit, so the call is considered complete and sent down the pipeline for rating.

Important:

Setting this entry too low causes an inordinate number of calls to be left open indefinitely. Setting this entry too high can cause the pipeline to rate calls with missing call segments. The 60-second default value is appropriate for most BRM implementations.

Rating Continuous Data Calls by Segment

Use the KeepCallOpen semaphore registry entry together with FlushLimit, to keep calls open indefinitely, and rate them in segments periodically. This feature is designed for data calls that are kept open continuously (days at a time). These long data calls are usually rated periodically to capture revenue.

KeepCallOpen is an update registry entry sent in the FlushLimit semaphore.

For example, a bank might keep a continuous call open to each of its ATMs to pass data back and forth. Using the default behavior, the call would not be rated for days. You will probably want to capture the revenue for these types of calls periodically, perhaps every 12 or 24 hours. Setting KeepCallOpen to True keeps, these calls open. You then rate these calls in segments by sending in a FlushLimit semaphore entry.

If you set KeepCallOpen to True and send it in with the FlushLimit semaphore every 12 hours, an EDR is created twice a day, each EDR rating the previous 12 hours of call time and volume.

KeepCallOpen is an entry in the FlushLimit update registry entry.

By default KeepCallOpen is set to False. The default behavior directs this module to rate the call when the first and last segments have arrived, or when the FlushLimit is sent, whichever comes first. Any subsequent records are ignored.

Rating Partial Calls by Service

Use the FlushServiceCode semaphore registry entry to limit the effect of FlushLimit to calls with specific service codes. FlushServiceCode is sent as an entry in the FlushLimit update registry entry. If the FlushServiceCode entry matches the value of the DETAIL.INTERNAL_SERVICE_CODE field of the call segment, the call is flushed. All other partial calls are ignored.

For example, the following FlushServiceCode entry directs the pipeline to rate only calls with the dat service code:

{

  FlushLimit=0
  FlushServiceCode = dat

}

Figure 5-1 shows how the FlushServiceCode and KeepCallOpen semaphore registry entries interact.

Figure 5-1 FlushServiceCode and KeepCallOpen Semaphore Interaction

Description of Figure 5-1 follows
Description of ''Figure 5-1 FlushServiceCode and KeepCallOpen Semaphore Interaction''

Capturing Fields From the Last Call Record

Use the AssembledEDR startup registry entry to capture data from Basic Detail Record fields for the L call segment.

The final record of a call has a long duration indicator of L (last), however the L call record may not be the last to arrive at the pipeline. In some cases it is important that information be captured from the L call record. For example, the number that terminated the call can only come from the L call record.

If AssembledEDR is used, the pipeline captures Basic Detail Record data from the L call record and adds it to the EDR emitted, regardless of whether it was actually the last call record received.

List any Basic Detail Record fields to capture data from in the AssembledEDR entry, including any custom fields that your business requires. The pipeline will then include the data from those fields on the EDR that is compiled and processed. See "Sample Registry" for an example.

Tracking the Status of Assembled Calls

You can use a semaphore command to receive a report about the status of calls currently being assembled. The report provides the following information:

  • The number of partially assembled calls.

  • The number of EDRs currently waiting to be assembled.

  • The number of late EDRs that are parts of calls that are no longer being assembled, for one of the following reasons:

    • The call was assembled and sent to the pipeline. In this case, the first and last portion of a call was processed, and an intermediate part arrived after the EDR had been sent into the pipeline.

    • The call was flushed by a FlushLimit semaphore, and another portion arrived after the call was flushed.

    The report includes the following data about late EDRs:

    • The number of late-arriving EDRs for flushed calls.

    • The number of late-arriving EDRs for assembled calls.

    • The total number of late-arriving EDRs.

In addition to creating the report, each assembled EDR includes a NUMBER_OF_CDRS field that stores the number of CDRs that were included in the EDR. You can use this data in an aggregation scenario to gather additional data about assembled calls.

Note:

For EDRs that are not part of split calls, the FCT_CallAssembling module enters 1 in the NUMBER_OF_CDRS field.

Migrating Call Assembling Data Between Releases and Pipelines

When you upgrade BRM from one release to another or apply patches, you need to migrate the call data in your .dat files to the new format by using the XML support provided with the FCT_CallAssembling module.

For information on migrating EDR files, see "Upgrading Incomplete Calls to the New Container Description" in BRM System Administrator's Guide.

You use the following semaphore registry entries to change the format of the data from one release or pipeline to another:

  • ExportDataToXml

  • ImportDataFromXml

For more information, see the semaphore file entries in "FCT_CallAssembling".

To migrate call assembly data, perform the following steps:

  1. Export the data from your existing data files to an XML file by using a semaphore registry file with the ExportDataToXml entry:

    ExportDataToXml
{
   CallsPerFile = Value
}

  2. Import the data from the XML file into the data file with the new format by using the ImportDataFromXml entry:

    ImportDataFromXml
{
   FileName = filename.xml
}

Assembling Calls with Multiple Pipelines

When you use the FCT_CallAssembling module, you must process the EDRs for each account in the same pipeline. If you assemble calls in multiple pipelines for the same account, the call segments may get processed by different pipelines and cannot be assembled.

Discarding and Skipping EDRs

You can use the FCT_Discard module to skip or discard EDRs:

  • Skipping an EDR removes it from the pipeline.

  • Discarding an EDR sends it to a different output stream. You will probably want to audit the information in EDRs with a LONG_DURATION_INDICATOR of XC, XO, or XP before you discard them.

In both the cases the state of the EDR becomes invalid. (To indicate a discarded EDR, a value is entered in the DETAIL.DISCARDING field.)

For example, you can filter the following EDRs:

  • Discard EDRs that are older than three days and have a B_NUMBER that begins with 0049.

  • Discard EDRs that have a RECORD_TYPE that begins with a 9 followed by two digits, an INTERN_SERVICE_CODE that ends with a 2, and a WHOLESALE_CHARGED_AMOUNT_VALUE of 0.

For information on the regular expressions you can use, see "About Using Regular Expressions when Specifying the Data to Extract".

To create a valid mapping, the data in the EDR must match with all of the mapping data.

Configuring EDR Discarding

To configure EDR discarding:

  1. Specify which EDRs to discard or skip. See "About Configuring Discard and Skip Expressions".

  2. Configure the OUT_DevNull module as the output stream for discarded EDRs. See "Configuring Output of Discarded EDRs".

  3. Configure the FCT_Discard module. See "FCT_Discard".

    • Use the FCT_Discard module StreamName registry entry to specify the output stream.

    • To send EDRs to different output streams without making them invalid, use the FCT_EnhancedSplitting module. To process EDRs that belong to accounts in different BRM database schemas, see "FCT_AccountRouter".

About Configuring Discard and Skip Expressions

To specify which EDRs to discard or skip, you create a set of regular expressions using a set of data fields. These fields are not mandatory so if they do not exist in the EDR description it does not matter. If all expressions match the available fields, the EDR will be removed.

You can create different discard rules for separate pipelines.

You define discard rules in Pricing Center. The discard rules are stored in the IFW_DISCARDING table.

Use the Reload semaphore file entry to reload regular expression patterns after you change them.

You can discard or skip EDRs based on the following:

  • Rank. This specifies the order in which to evaluate the rules.

  • Record type.

  • Source network.

  • Destination network.

  • Call complete indicator.

  • Long duration indicator.

  • Usage class.

  • Internal service code.

  • GSM switch or GPRS switch.

  • Tariff class.

  • Tariff subclass.

  • Connection type.

  • Connection subtype.

  • B number.

  • The age of the EDR.

  • If the wholesale charge should be zero.

For information on the regular expressions you can use, see "About Using Regular Expressions when Specifying the Data to Extract".

To configure the FCT_Discard module. See "FCT_Discard".

Configuring Output of Discarded EDRs

Use the OUT_DevNull module to handle EDRs that should be discarded from a pipeline.

To discard EDRs, do the following:

OUT_DevNull is a submodule of the Output Collection module. All registry parameters and error messages are handled by the Output Collection module. See "Output Collection".

Generating Multiple TAP MOC and MTC Records

When you process TAP files, you use the ISC_TapSplitting iScript to splits mobile originating and terminating EDRs into multiple EDRs when the CDR contains more than one basic service. The ISC_TapSplitting creates a new EDR for each additional basic service.

Splitting mobile originating and terminating EDRs enables CDR rejection when a basic service record associated with a CDR is in error. It also permits custom validations to be added prior to splitting.

Service information for all secondary services (supplementary, VAS, CAMEL), which are part of a basic service EDR, are added to the last basic service EDR. Charge information for these secondary services are added to the last charge breakdown record of the last basic service EDR.

EDR splitting is performed after TAP validation. TAP files and EDRs that are rejected due to errors are not split. After the new EDRs are created, the original EDR is deleted.

To configure ISC_TapSplitting, see "ISC_TapSplitting".

Using Rules to Send EDRs to Different Output Streams

Note:

To send EDRs to different output streams for the Rated Event (RE) Loader, use the IRL_EventTypeSplitting iScript. See "Sending EDRs to Pipeline Output Streams".

Use the FCT_EnhancedSplitting module to specify different output streams for EDRs based on rules. For example:

  • You can split EDRs for different service types into different output streams.

  • You can split EDRs from roaming outcollects and incollects into different streams. See "About Rating Roaming Events" in BRM Configuring Roaming in Pipeline Manager.

To send EDRs to different output streams, you define a set of rules. For example, you can send all telephony EDRs from a specific network to a different output stream.

You can use the following data in the rules:

  • Record type.

  • Service code.

  • Usage class.

  • Source and destination network.

  • Switch.

  • Trunk in/out.

  • A number and B number area code.

  • Normalized C number area code (forwarded or routed number).

This example assigns all telephony EDRs with an A number starting with 49 to the Out49 system brand:

Service code :   TEL
A number:        0049.* 
System brand:    Out49

In addition you can specify the order in which to evaluate the rules, and the dates when the rule is valid.

The FCT_EnhancedSplitting module evaluates each EDR against the rules. The first rule that matches the criteria defines the system brand to use. The system brand is identified by a code. You use that code to map the system brand to an output stream in the registry. In this example, system brand Out49 is mapped to the EdrOutputOut49 output:

SystemBrands 
{ 
Out49 = EdrOutputOut49 
}

You can create separate splitting rules for different pipelines.

You can use a semaphore file entry to reload a new set of rules.

Configuring Enhanced Splitting

To configure enhanced splitting, you do the following:

  1. Configure system brands. Each system brand is mapped to an output stream. See "Creating Brands" in BRM Managing Customers.

  2. Use Pricing Center to configure the splitting rules. Each rule is associated with a system brand. If an EDR matches a rule, it uses the system brand defined in the rule.

    To create a valid mapping, the data in the EDR must match with all of the mapping data.

    For the data fields, you use regular expressions. For information on the regular expressions you can use, see "About Using Regular Expressions when Specifying the Data to Extract".

  3. Configure an output stream for each system brand. See "Configuring EDR Output Processing".

  4. Configure the FCT_EnhancedSplitting module. This defines the output stream for each system brand. See "FCT_EnhancedSplitting".

Sending EDRs to Pipeline Output Streams

The way you configure the pipeline to split EDRs into separate output streams depends on your system configuration:

Sending EDRs to an Output Stream Based on Service Code

When you load rated events in the BRM database, RE Loader loads events for separate services from separate directories. Therefore, your pipeline needs to send EDRs to separate output streams for each internal service code. To do so, you use the IRL_EventTypeSplitting iScript.

To specify the output stream, you edit the IRL_EventTypeSplitting.data file. This file maps service codes to output streams. For example, this entry maps the TEL service code to the TelOutput stream:

TEL;TelOutput

Note:

You can also use the FCT_EnhancedSplitting module to send EDRs to different output streams based on EDR content. See "Using Rules to Send EDRs to Different Output Streams".

The IRL_EventTypeSplitting iScript must run after the FCT_ServiceCodeMap module and before FCT_Reject module.

To configure the IRL_EventTypeSplitting iScript, you configure it as part of the FCT_IRules module. See "FCT_IRules".

For more information, see "IRL_EventTypeSplitting".

Sending EDRs to an Output Stream Based on Logical Partition and Service Code

In an IMDB Cache system with multiple logical partitions, RE Loader must load pre-rated and re-rated events into the correct logical partition. However, RE Loader does not process logical partition information. To ensure that events are loaded into the correct logical partition, you must configure your pipelines to send EDRs to separate output streams based on the logical partition and service type.

For example, if your IMDB Cache system contains two logical partitions that each support the GSM and GPRS services, the pipeline must split EDRs into four output streams:

  • GSM services from logical partition 0 to GSM output stream 1

  • GSM services from logical partition 1 to GSM output stream 2

  • GPRS services from logical partition 0 to GPRS output stream 1

  • GPRS services from logical partition 1 to GPRS output stream 2

To do so, you must configure the FCT_AccountLPRouter module and IRL_EventTypeSplitting_tt iScript.

To specify the output stream, you must:

  • Set the FCT_AccountLPRouter Active registry entry to True.

  • Edit the IRL_EventTypeSplitting_tt.data file. This file maps logical partitions and service codes to output streams.

    For example, the following entries map the TEL service code from logical partition 0 to the TELOutput1 stream, and the TEL service code from logical partition 1 to the TELOutput2 stream:

    TEL;0;TELOutput1
TEL;1;TELOutput2

Note:

You can also use the FCT_EnhancedSplitting module to send EDRs to different output streams based on EDR content. See "Using Rules to Send EDRs to Different Output Streams".

The IRL_EventTypeSplitting_tt iScript must run after the FCT_ServiceCodeMap module and before the FCT_Reject module.

To configure the IRL_EventTypeSplitting_tt iScript, you configure it as part of the FCT_IRules module. See "FCT_IRules".

For more information, see "IRL_EventTypeSplitting_tt".

Using Pipeline Manager with Multiple Database Schemas

In a multischema environment, you start a separate instance of Pipeline Manager for each BRM database schema.

  • Use the FCT_AccountRouter module to find the account and send the EDR to the correct instance of Pipeline Manager. The FCT_AccountRouter runs in its own instance of Pipeline Manager.

  • Use the DAT_AccountBatch module to supply data to the FCT_AccountRouter module. It runs in the same instance of Pipeline Manager as the FCT_AccountRouter module. See "DAT_AccountBatch".

To find the A number customer and the B number customer, the FCT_AccountRouter module does the following:

  1. The module looks for the following data in the EDR:

    • The internal service code that indicates which data can be used to identify the account. For example, if the internal service code is a telephony service, the identifying data is the A number. A different service might use the IMSI as the identifier.

      You identify which data to use by using the Pricing Center. See "Specifying Which Data is Used for Identifying Accounts" in BRM Setting Up Pricing and Rating.

    • The timestamp for the EDR. The timestamp is important because telephone numbers can be used by different accounts at different times.

  2. The module uses the DAT_AccountBatch module to look up the account. See "DAT_AccountBatch".

    Note:

    • If no A customer is found, the EDR is rejected. If the B customer is missing, no error is generated.

    • Because phone numbers can be recycled, the search is made on data from BRM audit objects.

  3. The DAT_AccountBatch module returns the database number.

  4. The FCT_AccountRouter sends the EDR to the correct pipeline, using the configuration defined in the registry.

Setting Up Account Identification in Multischema Systems

To set up account identification in a multischema system, do the following:

  1. Configure service and account data in the Pipeline Manager database:

    • Configure internal service codes. The DAT_AccountBatch module retrieves account information based on which services are rated. See "About Mapping Services" in BRM Setting Up Pricing and Rating.

    • Configure how the FCT_AccountRouter module looks up accounts (for example, by telephone number or IMSI). You specify which data is used for identifying accounts when you configure the FCT_Account module. See "Specifying Which Data is Used for Identifying Accounts" in BRM Setting Up Pricing and Rating.

    • (Optional) Configure account ID prefixes to use for handling duplicate telephone numbers. You configure this when you configure the FCT_Account module. See "Configuring Account ID Prefixes" in BRM Setting Up Pricing and Rating.

  2. Configure the FCT_AccountRouter. See "FCT_AccountRouter".

    Note:

    Since the FCT_AccountRouter module needs the internal service code, the FCT_AccountRouter module must be placed after the FCT_ServiceCodeMap module in the pipeline.

  3. Configure the DAT_AccountBatch module UseAsRouter registry entry.

    If set to True, the module is used by the FCT_AccountRouter module to route EDRs to separate Pipeline Manager instances. See "FCT_AccountRouter".

    If set to False (the default), the module is used by the FCT_Account module to get account data.

    See "DAT_AccountBatch".