21 About Rating Events Created by External Sources

This chapter presents an overview of loading events from event log files into the Oracle Communications Billing and Revenue Management (BRM) database for rating.

Table 21-1 lists additional BRM documents that contain information about loading events from event log files:

Table 21-1 Links to Information on Loading Events from Event Log File

Documentation	Topic
About Importing Events from External Sources	Prerequisites for creating an event import template
Online help for the Universal Event (UE) Mapper application (start Developer Center and see UE Mapper online help)	Instructions for creating an event import template
Loading Events from External Sources	Instructions for loading events and handling errors
"Universal Event Loader" utility page	Syntax of utility you use to load events
"pin_uei_deploy" utility page in BRM Developer's Guide.	Syntax of utility you use to migrate event import templates from one database to another
"Creating custom bill items" in BRM Configuring and Running Billing.	Considerations for using custom bill items

About Importing and Rating Events

BRM can rate service usage in real time. To rate usage in real time, you need a BRM client application such as RADIUS Manager to capture billable events. For some services, it is easier to import and rate events recorded in event log files than to create a BRM client application that captures and rates the events in real time. You can import events from event log files that include data from Web servers and other sources.

UE Importer consists of UE Loader, a command-line event loader that parses the event log file and loads the event into BRM, and UE Mapper, a client GUI application that you use to create event import templates that specify the file format. UE Loader uses the template to parse and load the event log file.

Figure 21-1 shows that:

UE Mapper:
1. Reads the event log file when you create an event import template for the log file.
2. Saves the event import template to the BRM database.
3. Reads the template from the database when you modify the template.
UE Loader:
1. Reads the event log file.
2. Uses the event import template for instructions on loading the events from the log file into the database.
3. Loads the events into the BRM database.

Figure 21-1 BRM and UE Importer

Description of ''Figure 21-1 BRM and UE Importer''

About Universal Event Mapper

To capture the events you want to import from event log files, use Universal Event (UE) Mapper to create an event import template. You can create multiple templates for your needs. For example, you might need one template for IP telephony data records and another for Internet dialup usage records. You can also create a single template to process both of these record types if they occur in the same event log file. Event import templates are stored in the BRM database in XML format.

For more information on creating event import templates, see "About Importing Events from External Sources". For step-by-step procedures on creating event import templates, see UE Mapper and see the Help.

About Universal Event Loader

To import and rate events from event log files, you use Universal Event (UE) Loader. UE Loader reads the event import template to load data from event log files into BRM as billable events. When the events are loaded, BRM rates the events and updates customer account balances. UE Loader is included in the BRM server software and is located in BRM_Home/apps/uel.

About Setting Up Event Importing

To import and rate events from event log files, you must do the following:

If you have not already done so, create a price list that includes the products and rate plans used for rating the events. You might need to define new event and service subclasses. For more information, see "About Creating a Price List".
Use UE Mapper to create a template that translates the event log file entries into BRM event fields. UE Loader uses the template when loading events. See "About Importing Events from External Sources".
Run UE Loader to import and rate the events. See "Loading Events from External Sources". You can load events manually or set up automatic loading by using Batch Controller. For information on automatic loading, see "Controlling batch operations" in BRM System Administrator's Guide.

You might need to create custom opcodes to import events if BRM opcodes cannot handle the type of data you must import. See "About Creating Custom Opcodes".

About Supported Event Log Files

This section describes the data log files you can import (load event data) into the BRM database.

You can import ASCII or binary data from event log files. If your event log file is a binary hex file, you can import records that use a "fixed width" record format. If your event log file is a text file, you can import records that use a "fixed width", "delimited", or "attribute value" record format.

Event Log File Terminology

The event log file can contain a header record (optional), data records, and a trailer record (optional). The header record and the trailer record can be defined by a unique record type locator. The data records can be more than one record type as specified by their record type locators. Figure 21-2 shows a sample event log containing the different record types.

Figure 21-2 Sample Event Log with Records

Description of ''Figure 21-2 Sample Event Log with Records''

You specify whether your event log file has a header and trailer record when you create the event import template. If you do not specify that your event log file has a header and trailer record, UE Loader assumes all records in the event log file are data records.

Each of these three types of records (header, data, and trailer) might be identified by a unique record type. If you have a header and trailer record, and you do not define a record type for each, UE Loader assumes the first record that is imported is the header and the last record is the trailer.

Important:

There can be only one header record and one trailer record in an event log file.

For more information on record types, see "About Loading Event Records Based on Record Type".

Supported File Types

You can import event log files with these file types:

text file type

Consists of data in ASCII text format. The record format for records of this file type must be delimited, attribute value, or fixed width.

binary hex file type

Consists of data as BCD (Binary-Coded Decimal) integers and EBCDIC (Extended Binary-Coded Decimal Interchange Code) strings. The record format for records of this file type must be fixed width.

Supported Record Formats

You can import the following record formats into BRM. All records in a log file must have the same record format.

attribute value

Fields within the records are specified by attribute-value pairs in this format: <name><attribute-value separator><value><delimiter>. For example, login=bob, is an attribute-value pair that uses an equal sign as the separator and a comma as the delimiter. Attribute-value pairs can occur in any order in the event log file. Records can be in one line or span multiple lines.

delimited

Records and record fields are delimited by ASCII characters. The delimiter consists of only one character, such as a comma.

fixed width

Fixed width data can be one of the following:

Continuous: Each record has a fixed length
Delimited: The fields have a fixed length

Within their specified length, fields can contain padding characters and can be left or right justified.

Note:

If there are multiple record types, the records may or may not be uniform in length. Record structure can also vary between record types. For example, you can define columns 1 through 8 as the start time for IP telephony record types, and columns 1 through 5 as the start time for Internet dialup record types.

Supported Trailer and Header Records

Typically, there are three kinds of records in an event log file: header records, data records, and trailer records. A file can include one header record, several data records, and one trailer record. Data records can have any number of data record types, such as data records pertaining to email services, dialup IP services, IP telephony services, and so on. You specify record types in UE Mapper by specifying a record type locator, a field that identifies each record's type.

You specify that your event log file has a header and trailer record in the event import template so UE Loader does not load them into the database. The optional header and trailer records may or may not have a record type. If a header and trailer record is specified without a record type, the first record is treated as the header and the last record is treated as the trailer.

Note:

Event log files are not required to have header and trailer records.

Header Records

There can be only one header record in a data log file. The header record must precede all data records and must use the same record format as all other records (delimited, fixed-width, or attribute-value format).

Header records usually contain information relevant to all events in the data records. A header record can also contain information that validates the status of the data log file, such as the number of expected data records in the file. You can set up a header check in UE Mapper to use this data to validate the number of records in the event log file.

Trailer Records

There can be only one trailer record in an event log file. It must use the same record format as all other records (delimited, fixed-width, or attribute-value format), and it must follow all data records.

Trailer records usually contain information to verify the status of the event log file. If your trailer records include this information, you can set up trailer checks by using UE Mapper to validate the event log file.

Figure 21-3 shows an event log file with a header record, data records of several record types, and a trailer record:

Figure 21-3 Sample Event Log with Records

Description of ''Figure 21-3 Sample Event Log with Records''

About Creating Custom Opcodes

If BRM opcodes cannot handle your event log files, you might need to create custom opcodes to load events.

About Custom Opcodes to Map Event Data to Accounts

You might need to create a custom opcode to find the customer accounts associated with the events in your event log files. Usually, the data in the event log file includes login names that existing BRM opcodes can use to find the account for each event. If the event log file does not include login names, you must write a custom opcode that finds the account based on other data in the event log file, such as the customer's IP address. See "Creating Opcodes for Finding Accounts Associated with Event Data".

About Custom Opcodes to Load Event Data

You might need to create a custom opcode to load the events into the BRM database. In many cases, you can use existing BRM opcodes to load event data, but to preprocess the data in the event log file either before rating or before storing the event data in the BRM database, you must create a custom opcode.

For example, if your event log files include a value for downloaded kilobytes, and you need the value represented in bytes, you can create an opcode to preprocess the data before loading the event. Also, if the event log files include area codes and phone numbers in different fields, you may want to concatenate the area code and the phone number in the opcode to store it as a single number in the database. For more information, see "Creating Opcodes for Loading Event Data".

About Loading Event Records Based on Record Type

Data records can have any number of data record types, such as data records pertaining to email services, dialup IP services, IP telephony services, and so on. You specify record types in UE Mapper by specifying a record type locator, a field that identifies each record's type.

One event log file can contain data records of several record types. When you create an event import template for log files with multiple record types, you specify the following items for each record type:

The BRM event type associated with the data
The opcode that loads the data
The opcode that finds the account associated with the data
The filters that are applied to load or log the data

About Ignoring Record Types When Loading

An event import template specifies which record types are loaded into your BRM database.

You might not want to load all event records from your event log files into BRM. For example, you specify a record type locator so you can use filters to prevent unnecessary record types from being loaded into the database. A record type locator is a unique field that identifies the different record types.

Using filters can improve loading performance. For example, in many IP telephony CDR files, there are start accounting event records and start stop accounting event records. The start stop accounting event records contain the necessary information to rate IP telephony calls, so it is not necessary to load the start accounting event records. Using filters to prevent loading start accounting records into the database significantly increases loading performance.

About Using Filters for Loading Events

You can set up filters in the event import template so that events in your event log file that satisfy specified criteria are handled as follows:

Discarded (not loaded). See "About Discarding Event Records Based on Record Content".
Logged to a separate file. You can log events that are loaded or discarded to a separate file. See "About Logging Event Records Based on Record Content".

Event filtering occurs after UE Loader does the following:

Reads the event log file.
Parses the contents of the event log file.

This task includes verifying whether each event in the log file is associated with a customer account in the BRM database. Thus, account verification is done for some events that might later be filtered out of the event log file.
Creates a cache file that contains the results of the parsing.

After creating the cache file, UE Loader applies any filters in the event import template to the events in the cache file, removing events that satisfy the filter-out criteria.

For instructions on setting up filters, start Developer Center and see the UE Mapper Help.

About Discarding Event Records Based on Record Content

Use filters to prevent loading certain event log file records into the database. Records that have particular data field values can be marked and rejected without causing errors in the loading process.

For example, if you have a list of accounts with an inactive account status, you can reject event records that have login fields associated with those accounts.

Another reason for rejecting records based on record content is when you have a minimum amount of usage for which you want to charge users. For example, you can reject records where the dialup session is less than fifteen seconds.

About Logging Event Records Based on Record Content

Use filters to log specific event log file records to a file that you then load into the database. The filter marks records that have particular data field values and logs them to a file.

For example, if you notice accounts with high volume usage, you might want to offer these customers a pricing plan for high volume users. A filter can mark and log the accounts that have a usage value above your filter criteria to create a list of customers to contact. You can also set up a filter to mark and log accounts that show signs of fraudulent activity.

You set up filters when you create the event import template. For instructions, start Developer Center and see the UE Mapper Help.

About Validating the Event Log File

If your event log file includes header and trailer records, you can set up checks to verify that the complete number or size of records is in the log file.

You set up header and trailer checks by using UE Mapper when you create the event import template. For instructions, start Developer Center and see the UE Mapper Help.

About Importing International Log Files

UE Loader supports log files in English, French, German, and Japanese. For instructions on setting up UE Loader for these languages, see "Configuring UE Loader for International Log Files". UE Loader supports UTF8 and Unicode encoding formats.

Importing and Rating Events from External Sources

You can rate service usage by importing usage events from log files, such as Web server log files and IP telephony call detail records (CDR) files, into BRM. This section provides a brief technical overview of how UE Loader imports event data from event log files into the BRM database for rating. It also explains how to prepare for creating an event import template.

For an overview of importing events, see "About Rating Events Created by External Sources". For instructions on loading data log file event data into the BRM database, see "Loading Events from External Sources".

About Importing Events from External Sources

To import usage events from log files, you create event import templates that specify the usage fields to import and how to import them. You use UE Mapper, a component of Developer Center, to create event import templates. After you create the templates, you use the UE Loader, a command-line utility installed with BRM, to test the templates and load events from your event log files into BRM. UE Loader reads the event import templates in the database for instructions on loading the events.

Overview of Preparing for Loading Events

There are seven steps to prepare for loading external event data into the BRM database:

If necessary, create an opcode to find the accounts associated with the event data to load. See "Creating Opcodes for Finding Accounts Associated with Event Data".
If necessary, create an opcode to load the event data. See "Creating Opcodes for Loading Event Data".
Configure UE Mapper to customize delimiter options. See "Adding Options to UE Mapper Drop-Down Lists".
If you use custom events and fields, understand the procedures for importing them. See "Making Custom Fields Available for Mapping" and "Displaying the Description of Custom Events".
Create an event import template using UE Mapper and save it to a test BRM database. See the UE Mapper online Help.

The event import template drives the event import process. It specifies:
- The event data to load from the event log file records.
- The BRM storable object and storable object fields into which the event data is loaded.
- The BRM opcode that finds the accounts associated with the event data (optional in some cases).
- The BRM opcode that loads the event data.
Test the event import template by running UE Loader to ensure that event data can be loaded successfully from a sample event log file.
Save the event import template to your BRM production database so that UE Loader can load external event data as needed.

How UE Loader Works

UE Loader first loads the event log file records into an object queue. It then reads data from the event import template to construct an flist with the account search criteria and pass the flist to an opcode that finds the account associated with the events. After UE Loader obtains this account, it reads data from the event import template to construct an flist with the event data and passes the flist to an opcode that loads the event data into the account. If UE Loader is set up to use multiple threads, it can load events into several accounts at once. For more information on multiple threads, see "Setting the Number of Threads for Performance".

Constructing an Flist by Using UE Mapper

For UE Loader to construct an flist, it reads the event import template for the flist instructions you specified. You can specify arrays in the template if the flist should include arrays: for example, to import multiple contacts. You use UE Mapper to populate the fields of an flist with the event data to import into BRM.

To see a valid input flist that was created to load event data from the log file into a specific BRM event object:

Load the sample event import template (BRM_Home/apps/sample_handler/sample_template.xml) into the BRM database:
```
pin_uei_deploy -t SampleTemplate -c -i sample_template.xml 
```

Create a sample event log file named SampleEvent in a text editor and add the following contents:

Field1Field2Field3Field4Field5 
v01/jun/2003:01:04:00 am PST01/jun/2003:01:07:00 am PST0812 
v01/jun/2004:01:04:00 am PST01/jun/2004:01:07:00 am PST2013 
v01/jun/2005:01:04:00 am PST01/jun/2005:01:07:00 am PST3015 
Field1Field2Field3Field4Field5

Restart Developer Center.
Open UE Mapper in Developer Center.
Choose Modify Template from the File menu.
In the Modify Template dialog box, select SampleTemplate from the Templates list and click Modify.
In the Open Event Log File dialog box, specify the SampleEvent event log file and click Open.

Creating an Event Import Template

Before you can import events, you use UE Mapper to create an event import template. This section describes tasks you might need to do before you can create the template.

UE Mapper is a Developer Center application. UE Mapper includes detailed help for creating event import templates.

Before You Begin

To create an event import template:

You should be familiar with the contents of the event log file you are importing, including:
- What each record and field in the event log file means. Each record corresponds to an event in BRM, and each piece of data corresponds to a field in an event.
- How the event log file is formatted, including which character signifies the end of a record and which character separates data fields in a record. For example, records might be separated by line ends, fields might be separated by commas, or records might contain continuous data in a fixed-width format.
You must understand BRM events, in particular, which type of BRM events are associated with the log file events you are loading.
You must understand BRM opcodes, including required flist fields and the data types of those fields.

Before you create an event import template, you might need to perform one or more of the following tasks:

Preparing Event Log File Data
Creating Opcodes for Finding Accounts Associated with Event Data
Creating Opcodes for Loading Event Data
Making Custom Fields Available for Mapping
Displaying the Description of Custom Events
Adding Options to UE Mapper Drop-Down Lists
Migrating Event Import Templates from One BRM Database to Another

Preparing Event Log File Data

In some cases, you must preprocess the records in your event log file before you can use the file to create an event import template. You might need to inspect the data in the input records for errors and manipulate the event data before it can be imported into BRM.

Creating Opcodes for Finding Accounts Associated with Event Data

The BRM opcode commonly used to find customer accounts associated with event data is PCM_OP_ACT_FIND. This opcode uses the customer login name in the event record to search the account table in the database associated with the event. If your event records do not contain the customer login name, you must create a custom opcode that uses another field to find the accounts.

Important:

The custom opcode must return the account Portal object ID (POID). It can also return the service, but it is not required to.

For example, if you have IP telephony call detail records (CDRs) that include the call origination number or call ID number of the user instead of the login name, you could create a custom opcode to map the call ID in the event to the account associated with that call ID.

Creating Opcodes for Loading Event Data

In some cases, you must create custom opcodes to load event data.

The BRM opcodes that are commonly used to load data from event log files into BRM are PCM_OP_ACT_USAGE and PCM_OP_ACT_LOAD_SESSION. If you use a method for rating events that is more complex than these opcodes can handle (the data you need to load is not in the format these opcodes expect), you cannot map data to them. You must create a custom opcode that can load your event data.

For example, when importing IP telephony call detail records (CDRs), you probably want to perform duplicate session checking and handle start and stop accounting events. PCM_OP_ACT_USAGE and PCM_OP_ACT_LOAD_SESSION do not handle that data, so you create custom opcodes to handle these transactions. For other examples for creating a custom opcode to load event data, see "About Custom Opcodes to Load Event Data".

Making Custom Fields Available for Mapping

If you create custom BRM fields, you must make these fields available to UE Mapper before they can be displayed for mapping. You use Storable Class Editor to create the fields. For instructions about making custom fields available to Java applications such as UE Mapper, see BRM Developer's Guide.

Displaying the Description of Custom Events

If you load events using PCM_OP_ACT_USAGE, include the field that describes your custom events in the event mapping by giving a default value for the field. By doing this, the description of your custom events is displayed in the Event Browser.

To display the description of custom events in the Event Browser, follow these steps:

Using UE Mapper, open the event import template used to import log files with custom usage events.
When creating the flist for event mapping, map the PIN_FLD_SYS_DESCR field of the custom event to a default value, where the value represents the description of the custom event.

If you load events using PCM_OP_LOAD_SESSION, the opcode fills in the PIN_FLD_SYS_DESCR field with a default value automatically. PCM_OP_LOAD_SESSION uses the default value Session:event type, where event type is the event type you specify in the Event Properties tab. If you do not want this description, map the field to a default value of your choice.

Adding Options to UE Mapper Drop-Down Lists

You might need to add items, such as delimiters, to some drop-down lists in the Data Definition tab of UE Mapper. The drop-down lists include popular options, but you can edit the lists in the UEMapper.properties file to include other options you need. Follow the instructions in the UEMapper.properties file in /opt/portal/release/DevCenter/UEMapper.properties.

Migrating Event Import Templates from One BRM Database to Another

Use the pin_uei_deploy utility to migrate an event import template from one database to another. For example, use the utility when you move a template from a test database to a production database. Using the pin_uei_deploy utility, you read an event import template from one database, save it as an output file on a local system, and then load it into another database.

Important:

Do not modify the event import template when it is saved as a file on the local system. To modify an event import template, use UE Mapper in Developer Center.

To migrate an event import template from one database to another:

Ensure that BRM is running on both servers.
Ensure that you have a pin.conf file in the same directory as pin_uei_deploy.
Ensure that you have write privileges for the directory where you intend to save the output file.
Check the output directory to ensure that you do not overwrite output files you want to save. The pin_uei_deploy utility overwrites an output file of the same name without displaying a warning.
In the pin_uei_deploy utility pin.conf file, set the connection information to point to the database you are migrating the event import template from.
List all the event import templates stored on that database.
```
pin_uei_deploy -l 
```
Note the name of the template you are migrating.
Download the template from the database to a local file. You can use any name for the output file. The output file is saved to the current directory unless you specify a path.
```
pin_uei_deploy -t template_name -r -o output_file_name
```
In the pin_uei_deploy utility pin.conf file, set the connection information to point to the database you are migrating the event import template to.
List all the event import templates stored in that database.
```
pin_uei_deploy -l 
```
If the name of the template you are migrating exists in that database, ensure that you want to delete or overwrite the existing template as described in the next step.
Load the template using these options:

If the name of the template you are migrating does not already exist on the database, load the template using the create (-c) mode:
```
pin_uei_deploy -t template_name -c -i output_file_name
```
If the name of the template you are migrating already exists in the database, do one of the following:
- Delete the existing template in the database before loading using the -c mode:
```
pin_uei_deploy -t template_name -d 
```
- Load the template using the modify (-m) mode of operation to overwrite the existing template:
```
pin_uei_deploy -t template_name -m -i output_file_name 
```
Verify that the template has been loaded by listing all templates in the database:
```
pin_uei_deploy -l 
```