This chapter describes how to set up output processing for the Oracle Communications Billing and Revenue Management (BRM) Pipeline Manager. The input can be CDRs for telco rating.
For background information about Pipeline Manager, see "About Pipeline Rating".
To set up EDR input processing, see "Configuring EDR Input Processing".
Pipeline Manager can generate data for various purposes:
To provide rated events for the Rated Event (RE) Loader to load into the BRM database.
To collect rejected EDRs for recycling.
To collect duplicate EDRs.
To send data to the Pipeline Manager database for additional processing in another pipeline.
To handle discarded EDRs.
The output process works as follows:
The completed EDRs are moved to the output buffer.
The output module reads data from the output buffer and processes the data. The processing performed depends on the output configuration, for example:
If the output consists of rated EDRs, the output module uses grammar and description files to convert the EDRs to a format that the RE Loader uses.
If the output consists of rejected EDRs, the output module creates reject files.
If the output consists of AAA EDRs, the output module uses grammar and description files to convert the EDRs to a format that the external prepaid network understands, such as flist, Diameter, or MBI.
The output module writes the data to the specified output stream. The destination of an output stream can be a file directory, a location in the database, or an external network. You can configure different directories, file names, and network destinations for each output stream.
Output processing is managed by the output controller and the output collection module.
The output controller manages the overall output process. You can configure output properties that apply to all streams. See "Output Controller".
The Output controller performs the following functions:
Manages the output stream's trailer information.
Rejects input streams when they exceed a specified maximum number of errors.
Checks for duplicate CDR files.
Notifies the Transaction Manager when a transaction ends.
Stops the pipeline when critical errors occur.
The input stream's trailer information can become invalid when a pipeline discards or rejects individual EDRs or when a pipeline splits EDRs into multiple output streams. For example, the input stream's trailer contains a total charge amount field, which contains the charge amount for all EDRs. When a pipeline discards some EDRs in the input stream, the total charge amount is no longer valid. To correct this, the Output Controller automatically recalculates the trailer information for each output stream.
Output collection defines all the output streams for the pipeline. The Output Collection module performs the following functions:
Generates the output devices that are specified in the registry at system startup.
Passes the EDR container to the specified output device.
You configure the Output Collection module by editing a pipeline's OutputCollection section of the registry file. For information, see "Output Controller".
The following output modules are available:
OUT_GenericStream: Used to process rated events. This module converts the EDR format to a format needed for further processing, for example, RE Loader format to load rated events or TAP format for outcollect processing of roaming records.
See "Configuring Output for Rated Events and AAA Responses" and "OUT_GenericStream".
OUT_Reject: Used to process rejected or duplicate events. See "Configuring Output for Rejected or Duplicate EDRs" and "OUT_Reject".
OUT_DB: Used to send data to the database. See "Sending Output to a Database" and "OUT_DB".
OUT_DevNull: Used to discard EDRs that you don't want to recycle. See "Configuring Output of Discarded EDRs" and "OUT_DevNull".
EXT_OutFileManager: Used to manage the output files of the OUT_GenericStream module and the OUT_Reject modules. See "EXT_OutFileManager".
Output processing uses the following types of files:
Temporary data file: Stores records during processing. There is one temporary file for each external file being processed. If the entire external file is rejected, the temporary file is not used.
Output File: Stores the EDRs after processing. This file is derived from the temporary file. When processing is completed successfully, the temporary file is renamed and becomes the output file.
Temporary stream file list: Stores the names of temporary data files. This is required when the Replace registry entry is enabled. In that case, the output file name is appended with a sequence number. Each output stream uses a separate temporary stream file list. For example, the telephony output stream registry includes these entries:
TempDataPath = ./samples/wireless/data/telout TempDataPrefix = tel.tmp. TempDataSuffix = .data
Note:
These registry entries are used for internal pipeline processing only and should not be changed.These output processing files are managed by the "EXT_OutFileManager" module.
ASN.1 objects are composed of a triplet TLV (Tag/LengthOfValue/Value). Therefore, before writing ASN.1 to the output, you must calculate the LengthOfValue field for every element of the ASN.1 tree that you are building. To do this, use the EXT_AsnTree module.
This extension has functions to perform the following tasks:
Builds a tree of ASN.1 objects.
Updates the LengthOfValue fields of the ASN.1 objects.
Flushes the resulting ASN.1 data block to an output stream.
For more information, see "ASN.1 Functions" in BRM Developer's Reference.
To configure output processing, do the following:
Create the directories for the output streams.
For output of rated events and AAA responses:
Set up an output stream format description file.
Set up a mapping file.
Set up a grammar file.
In most cases, you need to modify only the sample stream format description and output mapping files.
Configure these output sections in the registry:
The output mapping in the DataDescription section. This specifies the output mapping for rated events. See "Configuring the Output DataDescription Registry Section".
The OutputBuffer section in the Pipeline section.
The Output section. This section includes the configuration for the pipeline output controller and for output modules such as the OUT_GenericStream module. See "About the Output Modules".
Configure the output stream for the function modules that create the output. For example, configure the FCT_Reject module to send rejected EDRs to the reject output stream.
The output section in a pipeline has this hierarchy:
Pipeline
Output
Output controller
Output collection
Output streams
The output controller manages the overall output process. You can configure output properties that apply to all streams. See "Output Controller".
Output collection defines all the output streams for the pipeline. You configure the Output Collection module by editing the OutputCollection section of the registry file. For information, see "Output Collection".
Output streams send EDRs to the appropriate output location. See "About the Output Modules".
The Statistic subgroup controls the statistics related to Pipeline Manager's EDR processing rate. You can view these statistics in the output logs, HTTP browser, and the console output from the SNMP binaries.
For more information regarding the use of the SNMP binaries to view performance statistics, see "SNMP Utilities" in BRM System Administrator's Guide.
For more information regarding the use of HTTP browser to view performance statistics, see "Monitoring Pipeline Manager EDR Throughput" in BRM System Administrator's Guide.
The Statistic subgroup has one registry option, EDRCountCriteria. This option can have two possible values:
INPUT: The Pipeline Manager considers only the detail CDRs that are passed through it as input for calculating the EDR statistics.
ALL: This is the default option. The Pipeline Manager considers all the EDRs that are passed through it for calculating the EDR statistics. This includes the duplicate EDRs created and the EDRs directed to an additional output stream.
Note:
The Statistic subgroup is optional. If this subgroup is not present, the behavior of the Pipeline Manager is similar to that when EDRCountCriteria is set to ALL.This sample shows the output hierarchy:
Pipelines
{
W_SAMPLE
{
Output
{
WriteDefaultEdr = False
MaxErrorRates
{
}
#The following subgroup is optional
Statistic
{
EdrCountCriteria = ALL
}
OutputCollection
{
Output stream 1
{
ModuleName = OUT_Module_1
...
}
Output stream 2
{
ModuleName = OUT_Module_2
...
}
} # end of Output
} # END W_SAMPLE
} # END Pipelines
To send rated events from pipeline rating and responses from AAA request processing, you configure the OUT_GenericStream module.
For pipeline rating, this module converts EDRs to the output format used by RE Loader. The output is a file that is loaded by RE Loader.
To convert data from EDR format to a file, the module uses the following files:
Stream format description
Output grammar: Specifies which records to include in the output.
Output mapping: Specifies how to map the data in EDRs to data in the format defined by the output grammar file.
The sample registry files include stream format description, output grammar, and output mapping files that convert data from the BRM EDR, TAP, and CIBER formats.
To configure the OUT_GenericStream module, you specify the following:
The output grammar file.
The BRM event type that the output file contains, such as /event/delayed/session/gsm.
The type of pipeline, for example, rating or backout.
Whether to delete empty output streams. See "Configuring Pipeline Manager to Delete Empty Output Streams".
The output module, either "EXT_OutFileManager".
See "OUT_GenericStream".
Events from different services (GSM, SMS, and so forth) must be delivered to the RE Loader or a prepaid network in separate files. To do this:
Configure the IRL_EventTypeSplitting iScript to split EDRs by service code. See "Sending EDRs to Pipeline Output Streams".
In the registry, configure an instance of the OUT_GenericStream module for each service.
Each EDR must have a default output stream and may also contain additional output streams. Use the following iScript functions to manipulate multiple output streams in a single registry:
"edrAddAdditionalStream" in BRM Developer's Reference.
"edrRemoveAdditionalStream" in BRM Developer's Reference.
"edrGetAdditionalStream" in BRM Developer's Reference.
"edrContainsAdditionalStream" in BRM Developer's Reference.
To add output streams to the default stream in the sample registry:
Create an iScript file that adds the stream.
The edrAddAdditionalStream iScript document contains an example addoutmod.isc iScript file that shows how to add two additional output module streams.
Reference this iScript file in the pipeline FunctionPool registry section.
The edrAddAdditionalStream iScript document contains an example function registry section that shows how to use the addoutmod.isc file to add an output stream in the function registry.
Define the stream in the pipeline output registry section.
The edrAddAdditionalStream iScript document contains an example output registry section that shows how to configure additional output streams by using OUT_GenericStream.
See edrRemoveAdditionalStream for an example that shows how to remove an EDR output stream.
You configure a DataDescription section in the registry for each pipeline. The DataDescription section specifies the stream format description and input mapping files (see "About the Input Process") and the output mapping file.
DataDescription
{
Standard
{
ModuleName = Standard
Module
{
StreamFormats
{
Format1 = ./formatDesc/Formats/Flist/Flist_v01.dsc
}
InputMapping
{
Mapping1 = ./formatDesc/Formats/Flist/Flist_v01_InMap.dsc
}
OutputMapping
{
Mapping1 = ./formatDesc/Formats/Flist/Flist_v01_OutMap.dsc
}
}
}
}
Pipeline module instances prioritize the order in which formats are considered for parsing in the order that the stream format description files are listed in the StreamFormats section of the registry.
Parsing errors can occur in a pipeline if the stream format description files are listed in the incorrect order. For example, the stream format description file that applies to your custom input stream would be listed in the StreamFormats section before the output stream format description file.
The OUT_Reject module writes rejected EDRs to a reject file. A rejected EDR is identical to the EDR input.
To configure output for rejected EDRs, do the following:
For rejected EDRs, configure the FCT_Reject module.
See:
For duplicate EDRs, configure the FCT_DuplicateCheck module. See "Handling Duplicate EDRs".
In the registry, configure an instance of the OUT_Reject module for rejected EDRs and an instance for duplicate EDRs. See "OUT_Reject".
You can use the same output directory with different file suffixes and/or prefixes for rejected or duplicate EDRs.
The OUT_Reject module is a submodule of the Output Collection module. All registry parameters and error messages are handled by the Output Collection module. See "Output Collection".
File handling for rejected and duplicate EDRs is performed by the EXT_OutFileManager module. See "Sending Output to a File".
To send output to a file, use the "EXT_OutFileManager" module. The EXT_OutFileManager module handles file prefixes, suffixes, and paths for the OUT_GenericStream and OUT_Reject modules.
The EXT_OutFileManager module writes the output data to a temporary data file. When the TAM reports a successful completion of a transaction, the file is renamed to an output file and the temporary data file is removed.
If a transaction is rolled back, the original input is restored, and the temporary data file is moved to an error directory and renamed with an error prefix and/or suffix.
By default, the EXT_OutFileManager module is configured to delete empty output files. There is a short period of time during which empty output files are renamed and deleted. If another process, such as one that manages output files, tries to manipulate the file during this period, the pipeline may experience errors. To avoid this problem, configure any output file management processes to wait approximately one minute before attempting to manipulate files.
Use the TempPrefix registry entry to specify the prefix for temporary data files.
Important:
Do not change the TempDataPrefix, TempDataSuffix, and TempDataPath registry entries. These entries are used by the pipeline for internal data processing only.See "EXT_OutFileManager".
Use the OutputPath, OutputPrefix, and OutputSuffix registry entries to manage the output files for each stream.
Important:
To ensure output file integrity, specify a unique combination of OutputPath, OutputSuffix, and OutputPrefix values for each output stream defined in the registry.See "EXT_OutFileManager".
Use the UseInputStreamName registry entry to specify to use the input file name to build the output file name.
For example, when UseInputStreamName = [2,4;4,6;8,&], the following characters from the input file name are used to build the output file name:
Characters 2 to 4.
Characters 4 to 6.
Characters 8 to end of string, where the '&' symbol indicates the end of the string.
For instance, if the name of the input file is test12345678, Outputprefix is test, and Outputsuffix is .edr, the output file name will be testestt1245678.edr
See "EXT_OutFileManager".
Use the SequencerPrefix registry entry to specify a prefix to the sequence number before it gets appended to the generated output file name.
Note:
This entry is used only when AppendSequencerNumber is set to True.For example, when SequencerPrefix is "+", Outputprefix is test, Outputsuffix is .edr, the output file name will be test+000002.edr.
By default, the SequencerPrefix is "_" and if no SequencerPrefix is needed, it has to be specified as SequencerPrefix = ""
Important:
Do not use the characters #, $, =, /, or \ to specify SequencerPrefix.If you want to use the output of one pipeline as the input for another pipeline, you must move the output file to another directory before it can be used by the next pipeline. You move the files by using Event Handler and an external script. Event Handler would run your external script when prompted by a specified internal event. The external script would move the output file from one directory to another. For more information, see "Using Events to Start External Programs" in BRM System Administrator's Guide.
For example, if you want to use the output file from pipeline A as the input file to pipeline B:
Configure pipeline A to generate output files in directory 1.
Configure pipeline B to process input files from directory 2.
Create a simple script (ExternalScript) that moves the output files from directory 1 to directory 2.
Configure Event Handler to run ExternalScript whenever pipeline A generates an output file. For example, you can configure Event Handler to run ExternalScript when it receives an EVT_OUTPUT_FILE_READY event from the "EXT_OutFileManager" module.
Use the OUT_DB module to send data to a database. You configure the following:
A SQL command parameter file
Files that define the following:
SqlBeginStream statement
HEADER records
DETAIL records
TRAILER records
SqlEndStream statement
The OUT_DB module
The OUT_DB module reads the parameter file for each new output stream.
When you configure the OUT_DB module, you configure the following:
The database to load data into.
Path and file names for the configuration files.
Aliases for the stream name and row number.
Source and destination for the header record.
File management options.
For more information about the OUT_DB module registry entries, see "OUT_DB".
The parameter file contains key and value pairs. The keys are used by other configuration files (SqlBeginStream, HeaderTableDefinition, DetailTableDefinition, TrailerTableDefinition, SqlEndStream). The syntax to recognize keys is ${KEY}. When the other files are processed, all found keys are replaced by the corresponding values. The files are not deleted.
It is possible to change the database tables in a running system if there are no open streams.
The SqlBeginStream file and the SqlEndStream file are used to define SQL statements that are passed to the database. The begin stream statement is executed before the first EDR arrives, and the end stream statement is executed after the last EDR is processed.
The HeaderTableDefinition, DetailTableDefinition (only if the NumberOfRows registry entry is set to 1), and TrailerTableDefinition files are used to describe the table in which the EDRs must be stored. For more information, see "HEADER, TRAILER, and DETAIL Table Definitions"
First the table name is defined with ${TABLE}, which is replaced by the definition in the parameter file. Then the column names of the table are defined. Each line holds a column of the table and its EDR container field, which is mapped into this column. The value is the EDR container field with external and alias names delimited by a comma ( , ).
${HEADER_TABLE}
;RECORD_TYPE = HEADER.RECORD_TYPE , HDR_RECORD_TYPE
Each column starts with the FieldDelimiter after the table name.
Because schema definition is not supported by the RWDBBulkInserter, when the NumberOfRows registry entry is set to 1, the columns are defined like the example above. For bulk insertion into the database when NumberOfRows is greater than 1, the column definition is deleted from the table definition file and the schema of the table definition in the database is used to create the BulkInsert. The following example shows the DETAIL table definition file for BulkInsert:
${DETAILTABLE}
;DETAIL.RECORD_TYPE , RECORD_TYPE
;DETAIL.RECORD_NUMBER , RECORD_NUMBER
...
In this case, each sequence of the EDR container field definition in the table definition file must be in the same order as the column definition in the database table.
Each logical block must end with two number signs ( ## ) except within the table definition file. A comment line must start with two slashes ( // ).
All these files result in one final configuration file for each stream with replaced parameter keys within the SQL statements, except optional registry keys. The registry keys (StreamNameAlias and RowNumAlias) are replaced immediately before the SQL statement is passed to the database.
The name of the final configuration file is the stream name. It is located in the ControlPath. If the SaveConfigurationFile registry entry is enabled, a suffix is added. If processing is successful, the suffix is .done; otherwise, it is .err.
The destination value can be applied to an appropriate field in the HEADER record of an output module. In the BRM format, this entry is used to fill the recipient field in the HEADER record.
The source value can be applied to an appropriate field in the HEADER record of an output module. In the BRM format, this entry is used to fill the sender field in the HEADER record.
Because of splitting and rejecting, there is a possibility that an output stream may contain only the HEADER and TRAILER records. In this case, the production of a default DETAIL record can be forced. If the WriteDefaultEdr registry entry is set to True, every DETAIL table contains at least one DETAIL record.
If the DeleteWithoutDetails registry entry is set to True, all insert and update operations will be rolled back and the default record will be deleted.
These settings are ignored if the output is for a reject stream.
Items in bold text are parameter keys.
DETAIL_TABLE sol42_out ## HEADER_TABLE sol42_out_header ## TRAILER_TABLE sol42_out_trailer ##
The format of HeaderTableDefinition does not depend on the NumberOfRows registry entry. The format of HeaderTableDefinition is as follows:
${HEADER_TABLE}
;RECORD_TYPE = HEADER.RECORD_TYPE , HDR_RECORD_TYPE
The format of TrailerTableDefinition does not depend on the NumberOfRows registry entry. The format of TrailerTableDefinition is as follows:
${TRAILER_TABLE}
;RECORD_TYPE = TRAILER.RECORD_TYPE , TRR_RECORD_TYPE
The format of DetailTableDefinition depends on the NumberOfRows registry entry. When NumberOfRows is set to 1, the format of DetailTableDefinition is as follows:
${DETAIL_TABLE}
;record_type = DETAIL.RECORD_TYPE , RECORD_TYPE
;record_number = DETAIL.RECORD_NUMBER , RECORD_NUMBER
When NumberOfRows is greater than 1, the format of DetailTableDefinition is as follows:
${DETAIL_TABLE}
;DETAIL.RECORD_TYPE , RECORD_TYPE
;DETAIL.RECORD_NUMBER , RECORD_NUMBER
Identifiers in bold text are registry entries. They are replaced before the statement is passed to the database.
One statement:
insert into stream_process (streamname, startdate,runmode,numberofrow,destination,source,info)
values
// this is a comment line
(__StreamName__, sysdate,'Debug',500,'${TABLE}','sol42_detailin','testing')
More statements:
begin insert into tab1 (col1, col2) values (val1, val2); insert into tab2 (col1, col2, col3) values (val1, val2, val3); // if an SQL block is used there must be a semicolon at the end end;
Identifiers in bold text are registry entries. They are replaced before the statement is passed to the database.
update stream_process set enddate = sysdate, edrnum = __RowNum__, // duration only valid, if startdate and sysdate within one day duration= (3600*to_char(sysdate,'HH24')+60*to_char(sysdate,'MI')+to_char(sysdate,'SS'))- (3600*to_char(startdate,'HH24')+60*to_char(startdate,'MI')+to_char(startdate,'SS')) where streamname = __StreamName__
SqlBeginStream insert into stream_process (streamname, startdate,runmode,numberofrow,destination,source,info) values (__StreamName__, sysdate,'Debug',500,'sol42_out','sol42_detailin','testing') ## HeaderTableDefinition sol42_out_header ## DetailTableDefinition sol42_out ## TrailerTableDefinition sol42_out_trailer ## SqlEndStream update stream_process set enddate = sysdate, edrnum = __RowNum__, duration= (3600*to_char(sysdate,'HH24')+60*to_char(sysdate,'MI')+to_char(sysdate,'SS'))- (3600*to_char(startdate,'HH24')+60*to_char(startdate,'MI')+to_char(startdate,'SS')) where streamname = __StreamName__ ##