54 Understanding Rated Event Loader

This chapter describes Oracle Communications Billing and Revenue Management (BRM) Rated Event (RE) Loader and how it imports pipeline-rated events into the BRM database.

About RE Loader

RE Loader is an optional BRM application that loads pipeline-rated events directly into the BRM database, bypassing the Connection Manager (CM) and Data Manager (DM). RE Loader then updates account balances, billing items, and journals in the BRM database. After the events are loaded, you can run BRM applications such as billing and reports on the rated data.

About the Database Schema

RE Loader uses a partitioned database and inserts prerated events into separate partitions.

Note:

You must partition your database when loading prerated events.

About RE Loader Event Types

By default, RE Loader loads the wireless services and corresponding BRM event types shown in Table 54-1.

However, you can configure RE Loader to load custom events. For information, see "Configuring the RE Loader Infranet.properties File".

Table 54-1 Service and Event Types Loaded by Default

Service Type	Event Type
GPRS (General Packet Radio Service)	/event/delayed/session/gprs
GSM (Global System for Mobile Communication)	/event/delayed/session/telco/gsm

These events are loaded into separate BRM database partitions allocated for delayed events. The event types are called "delayed" because they are rated before they are loaded, and there is a delay between the two actions. This is unlike events loaded in real time or by Universal Event (UE) Loader.

Note:

You should not load the same event types by using RE Loader and another method such as an optional manager or UE Loader.

About Loading Prerated Events

Prerated events are events that have been rated by Pipeline Manager prior to being loaded into the BRM database. Basic steps of pipeline rating and event loading include:

1. Pipeline Manager rates events associated with call detail records (CDRs).

   For information on how Pipeline Manager prerates events, see "How Events Are Rated by Using Pipeline Manager".

2. Pipeline Manager creates an output file for each service type and places them in one or more output directories.

   You configure the number and location of your pipeline output directories by using the pipeline EXT_OutFileManager module.

3. RE Loader loads the prerated events.

   For information, see "RE Loader Process Overview".

About Loading Rerated Events

It is possible to discover pricing or rating configuration errors after events have been rated by Pipeline Manager and loaded into the BRM database. When this occurs, you rerate any incorrectly rated events and reload them into the BRM database.

When you need to rerate and reload pipeline-rated events, you must:

1. Extract events that need rerating from the BRM database by using the Event Extraction Tool.

   Note:

   Event Extraction Manager is included in the RE Loader installation.

2. Rerate those events by using Pipeline Manager. Pipeline Manager backs out the previous rating changes and then rerates the events.

3. Reload the rerated events by using RE Loader.

   For information, see "RE Loader Process Overview".

RE Loader Process Overview

RE Loader processes output files generated from Pipeline Manager. You send these files to RE Loader manually through a command-line utility or automatically by the Batch Controller.

After RE Loader receives a pipeline output file, it:

1. Checks the event header to determine the storable class type and whether the file contains prerated, rerated, or discount events.

2. Parses the event data record (EDR) data into multiple temporary files, one for each BRM database table to be loaded, and places the files in a temporary directory.

3. Loads events from each temporary file into the BRM database by using multiple Oracle SQL Loader utility sqlldr processes.

4. Calls stored procedures to update the account balances, bill items, and journals.

5. Logs the session information in its log file (rel.pinlog).

About Running RE Loader

You can run RE Loader in one of the following ways:

• Manually from a command line. See "About Running RE Loader Manually".

• Automatically by using Batch Controller and the RE Loader batch handler. See "About Running RE Loader Automatically".

• As a daemon. See "About Running the RE Loader Daemon".

About Running RE Loader Manually

When you run RE Loader manually from a command line, you specify the location of the pipeline output file in the command line.

Figure 54-1 shows the RE Loader work flow when you run it manually from a command line:

Figure 54-1 Work Flow of Manual Execution of RE Loader

Description of "Figure 54-1 Work Flow of Manual Execution of RE Loader"

About Running RE Loader Automatically

To schedule RE Loader to run automatically, you must set up the following components:

• Batch Controller, which detects when an EDR file is present in the pipeline output directory and starts the RE Loader batch handler.

• RE Loader batch handler, which moves the EDR file from the pipeline output directory to the RE Loader processing directory and starts the RE Loader utility (pin_rel).

Figure 54-2 shows the RE Loader work flow when you schedule loading of events:

Figure 54-2 Work Flow of Scheduled Execution of RE Loader

Description of "Figure 54-2 Work Flow of Scheduled Execution of RE Loader"

The following actions are performed when RE Loader is scheduled to run automatically:

1. Batch Controller detects an EDR file in a pipeline output directory and starts the RE Loader batch handler.

2. The RE Loader batch handler moves the EDR file from the pipeline output directory to the RE Loader processing directory and starts the RE Loader utility (pin_rel).

3. RE Loader processes the file, loads the events into the BRM database, and updates the account balances, billing items, and journals. For more information, see "RE Loader Process Overview".

4. RE Loader batch handler moves the original file to an archive directory if the records are successfully loaded or to a reject directory if the records are not successfully loaded.

   Note:

   The RE Loader batch handler loads one file at a time. You can load more than one file at a time by configuring Batch Controller to call several RE Loader batch handler processes. For more information, see "About Running Multiple RE Loader Processes".

About Running the RE Loader Daemon

When you run the RE Loader daemon, the daemon creates RE Loader threads to load EDR files. Each RE Loader thread loads one EDR file. The number of threads that the RE Loader daemon creates depends on the maximum number of threads that you configure to run simultaneously at a particular time. Because you can configure multiple threads that can run at a time, you can load more than one EDR file simultaneously. This helps in increasing loading performance.

For example, if you configure 10 threads to run simultaneously at peak time, the RE Loader daemon creates 10 threads, which means that 10 EDR files are loaded at the same time.

Note:

Running the RE Loader daemon is the preferable method if most of the EDR files that you want to load are of small size.

To run the RE Loader daemon, you must set up the following components:

• The RE Loader Infranet.properties file, which contains the entries for running the RE Loader daemon. See Table 56-5 for more information.

• The RE Loader daemon start and stop scripts (start_rel_daemon and stop_rel_daemon). See "Running the RE Loader Daemon" for more information.

You can also run the RE Loader daemon by using the pin_ctl utility.

Figure 54-3 shows the RE Loader work flow when you run the RE Loader daemon:

Figure 54-3 Work Flow of RE Loader Daemon

Description of "Figure 54-3 Work Flow of RE Loader Daemon"

The following actions are performed when you run the RE Loader daemon:

1. The RE Loader daemon detects an EDR file in a pipeline output directory.

2. The RE Loader daemon creates RE Loader threads up to the maximum number of threads configured.

3. The RE Loader threads process the files, load the events into the BRM database, and update the account balances, billing items, and journals.

4. The RE Loader threads move the original file to an archive directory if the records are successfully loaded or to a reject directory if the records are not successfully loaded. These directories are configured in the RE Loader Infranet.properties file.

About Running Multiple RE Loader Processes

To achieve better loading performance, the sample Batch Controller configuration file (SampleBatchControllerInfranet.properties) is set to run three RE Loader processes in parallel. This means that when you schedule RE Loader to run automatically, Batch Controller starts up to three instances of the RE Loader batch handler. Each instance of the RE Loader batch handler starts an instance of the RE Loader utility.

If you configure RE Loader to run manually, you can start multiple processes from the command line.

Note:

If you run multiple RE Loader processes in parallel, configure Pipeline Manager to delete empty output streams. For more information, see "Configuring Pipeline Manager to Delete Empty Output Streams".

Figure 54-4 shows an example of three RE Loader batch handler and pin_rel utility processes running to load EDR files. Each utility loads one file into the BRM database:

Figure 54-4 Example of Three Batch Handlers and pin_rel Utilities

Description of "Figure 54-4 Example of Three Batch Handlers and pin_rel Utilities"

The following actions are performed when multiple RE Loader processes are configured:

1. Batch Controller starts an RE Loader batch handler for each new file it detects in the pipeline output directory, up to the maximum number of RE Loader batch handler processes configured.

2. Each RE Loader batch handler process starts an RE Loader utility process.

3. The RE Loader utility processes the events in the file in three phases:

   • Preprocessing

   • Loading

   • Account balance, bill item, and journal updates

   During the loading phase, the database tables are locked so that only one RE Loader process can load events at one time.

4. Each RE Loader process polls the other processes to see whether they are currently in the loading phase and waits its turn to load events. When the first process is loading, the second process performs prepreprocessing tasks while waiting its turn. When the first process finishes loading, the second process loads while the first process performs account balance, bill item, and journal updates for the events it loaded:

   
   Description of the illustration rlo_multi_processes.gif

5. When an RE Loader process has completed all three phases, it is free to process another file.

The files are loaded in sequence, one directory at a time. If the number of RE Loader processes is set to 3, only three handler processes can run at one time for all directories.

Setting the Optimal Number of RE Loader Processes

Because there are three phases to processing EDR files, the optimal number of RE Loader processes to configure is at least three. If you want to configure more than three processes, you should test your RE Loader performance. Because only one RE Loader process can load events at a time, having more than three means those that have finished the preprocessing phase wait their turn to load events. Depending on the size of your EDR files and the time it takes to load events, configuring four or five processes might save time.

You configure the number of RE Loader processes to run in parallel by setting the number of RE Loader batch handlers to start in the Batch Controller configuration file. For more information, see "Configuring Batch Controller".

Configuring Pipeline Manager to Delete Empty Output Streams

If you run multiple RE Loader processes, you should configure Pipeline Manager to delete empty output streams.

Pipeline Manager generates files based on the number of output streams that are running. If some of the output streams are empty, the pipeline can produce empty files, which causes an error when RE Loader attempts to load the empty files.

To produce only one output stream, set the DeleteEmptyStream pipeline registry entry to True. This is the default. For more information, see "Configuring Output for Rated Events" and "OUT_GenericStream".

About Backing Up RE Loader Files

By default, RE Loader skips redo generation when loading files into the BRM database. This optimizes loading performance, but it can cause you to lose data if your system shuts down ungracefully.

To prevent data loss when your system shuts down:

• Make full backups of the BRM database on a regular basis.

• Archive all successfully loaded files until you make a full database backup.

You can re-enable redo generation, at the cost of loading performance, by modifying the RE Loader control files. For information, see "Configuring Whether to Perform Redo Generation".

About Handling Errors

If any errors occur during event loading, all events loaded in that session are deleted from the database. After events have been successfully loaded, if any errors occur during the update procedure, you can correct the errors and then update the relevant events by rerunning the RE Loader utility. The utility detects that the events loaded correctly and performs only the update procedure. For more information, see "Troubleshooting Event Loading".

About Using RE Loader in a Multischema System

If you use a multischema system, you must set up the following for each BRM database schema in your system:

• Pipeline Manager instance. Each instance of Pipeline Manager must also have its own set of output files.

• RE Loader instance. Each instance of RE Loader must also have its own set of RE Loader processing directories.

• (Running RE Loader automatically only) An instance of Batch Controller and the RE Loader batch handler.

You must also install and configure a separate instance of Pipeline Manager. This Pipeline Manager multischema account router routes EDRs to the appropriate schema Pipeline Manager instance based on the account's database number.

Figure 54-5 shows the flow of data in a multischema system:

Figure 54-5 Multischema System Data Flow

Description of "Figure 54-5 Multischema System Data Flow"