5 Configuring the Rated Event Infranet.properties Files

Learn how to configure the Oracle Communications Billing and Revenue Management (BRM) Rated Event Loader (RE Loader) and the Kafka Event Streaming Infranet.properties files.

The RE Loader Infranet.properties file applies to the original processing method for either Oracle Communications Elastic Charging Engine (ECE) or Pipeline Manager, while the Event Stream Processor (ESP) Infranet.properties file applies to ECE REF JSONFILE events, ECE Composable Services events, and supported external events consumed from Kafka. See "Methods of Transferring Rated Events to BRM" for information about the processing methods.

Topics in this chapter:

Configuring the RE Loader Infranet.properties File

To configure RE Loader, you edit the RE Loader Infranet.properties file.

The Infranet.properties file contains configuration information for processing CDR files, such as the location of the RE Loader processing directory, how to connect to the BRM database, and how to process specific events.

Note:

In ECE, some parameters have values that are used by the code if the corresponding values are removed or commented out from the infranet.properties file. If these values are different from the default values in the file, they are mentioned in the tables below.

The tables in this chapter provide information for configuring the infranet.properties file for both ECE and Pipeline Manager.

To configure your RE Loader Infranet.properties file:

  1. Open the BRM_home/apps/pin_rel/Infranet.properties file in a text editor.

  2. Edit the file.

  3. Save and close the file.

  4. If you are using the RE Loader daemon, use the pin_ctl utility to run the BRM_home/bin/start_rel_daemon script. See "Running the pin_ctl Utility" in BRM System Administrator's Guide.

Topics in this section:

BRM Database Connection

Table 5-1 contains the RE Loader Infranet.properties file entries for connecting to the BRM database.

Table 5-1 Database Connection Entries

Entry Description

infranet.rel.dbtype

Specifies the BRM database type.

If not included in the file, ECE uses the value ORACLE.

The default value is oracle.

infranet.rel.dbname

Specifies the BRM database name.

Note: Your database name is the TNSNAMES alias in the Oracle_home/network/admin/tnsnames.ora file.

Ensure that this value is correct for your system.

The default value is pindb.

infranet.rel.userid

Specifies the user ID for connecting to the BRM database.

Ensure that this value is correct for your system.

The default value is pin.

infranet.rel.dbhost

Specifies the database machine's host name.

The default value is localhost.

infranet.rel.dbport

Specifies the database port number.

The default value is 1433.

infranet.rel.partition_set_number

Specifies the partition set number, from 1 through 7. This entry applies only to BRM databases with multiple delayed partition sets.

  • 1 uses delayed partition set P_1D to P_12D.

  • 2 uses delayed partition set P_1D to P_12D2.

  • 3 uses delayed partition set P_1D to P_12D3.

  • 4 uses delayed partition set P_1D to P_12D4.

  • 5 uses delayed partition set P_1D to P_12D5.

  • 6 uses delayed partition set P_1D to P_12D6.

  • 7 uses delayed partition set P_1D to P_12D7.

The default value is 1.

BRM Server Connection

Table 5-2 contains the RE Loader Infranet.properties file entries for connecting to the BRM server.

Table 5-2 BRM Server Connection Entries

Entry Description

infranet.login.type

Specifies whether RE Loader requires a login name and password to log in to BRM.

  • 0 specifies that a login name and password are not required.

  • 1 specifies that a login name and password are required.

The default value is 1.

infranet.failover

In high-availability systems, specifies the secondary CM connection. For example:

infranet.failover.1 = pcp://root.0.0.0.db_no:password@failover_host:failover_port/service/pcm_client

There is no default value for this parameter.

infranet.rel.polling_interval

Specifies the interval, in milliseconds, that RE Loader checks the database to see whether another process is loading.

The polling interval depends on the number and size of your input files. If you have very large files, make the polling interval longer. If you have many small files, make the interval shorter.

The default value is 1000.

infranet.rel.polling_time_out

Specifies the time, in milliseconds, that RE Loader waits to load events before exiting.

The time-out period depends on the number and size of your input files and how many parallel RE Loader processes are running. If you have very large files or many processes, make the time-out period longer.

The default value is 600000.

infranet.rel.updater_threads

Specifies the number of threads dedicated to the update and preupdate stored procedures. You can specify a fixed number of threads or configure RE Loader to adjust the number of threads based on the number of database objects to update.

To specify a fixed number of threads, set the entry equal to the desired number of threads.

To configure RE Loader to automatically adjust the number of threads, set the entry to 0. In this case, RE Loader spawns the following number of threads:

  • Less than 1,000 objects: 2 threads

  • Between 1,000 and 200,000 objects: 4 threads

  • More than 200,000 objects: 8 threads

Note: Specifying a number of threads that exceeds the number of CPUs in your system may cause deadlock due to a lack of system resources. If you set the infranet.rel.updater_threads entry to a value greater than 8, RE Loader returns a warning message and continues processing.

If not included in the file, ECE uses the value 1.

The default value is 4.

infranet.rel.validate_dbnumber

Specifies whether RE Loader performs an extra validation step to ensure that it is loading a call detail record (CDR) file into the correct database schema.

Important: Use this option only for debugging. In a production environment, set this to false. Setting it to true degrades performance while loading data into the database.

See "Turning Off Database Verification to Improve Processing Performance".

The default value is true.

infranet.rel.validate_indexes

Specifies whether RE Loader verifies that the database indexes are correct before loading data into the database.

Important: Use this option only for debugging. In a production environment, set this to false. Setting it to true degrades performance while loading data into the database. See "Turning Off Index Verification to Improve Database Loading Performance".

If not included in the file, ECE uses the value true.

The default value is false.

infranet.rel.max_increment_by

Specifies the number of database schemas in your system. This value is used by the POID generation algorithm to ensure that POIDs are unique across all database schemas in your system.

See "Preventing POID Errors in Multischema Systems".

The default value is 20.

infranet.rel.sort.limit

Defines the maximum number of CDRs that the preprocessing script can sort by account ID. This improves performance later during the balance updating process.

If the number of CDRs in the input file is greater than the infranet.rel.sort.limit value, the preprocessing script does not sort the CDRs.

If not included in the file, ECE uses the value 500000.

The default value is 100000.

RE Loader Daemon

Table 5-3 contains the RE Loader Infranet.properties file entries for running the RE Loader daemon.

Table 5-3 RE Loader Daemon Entries

Entry Description

batch.check.interval

Specifies the time interval, in seconds, to monitor files from the output directory.

There is no default value for this parameter.

batch.file.rename.extension

Specifies the file name extension that the RE Loader daemon uses to rename the interim files before processing them.

The default value is .bc.

batch.start.highload.time

Specifies the start time of your system's busiest period. Specify the hour, minute, and second, in hhmmss format, using the 24-hour clock.

There is no default value for this parameter.

batch.end.highload.time

Specifies the start time of your system's slowest period. Specify the hour, minute, and second, in hhmmss format, using the 24-hour clock.

There is no default value for this parameter.

batch.lock.socket.addr

Specifies the port address of the process.

The default value is 24507.

batch.rel.archiveDir

Specifies the full path to the directory where a successfully processed file is archived. This is the default archive directory for all the event handlers.

The default value is BRM_home/apps/pin_rel/gsm/tel/archive.

batch.rel.rejectDir

Specifies the full path to the directory where an unsuccessfully processed file is stored. This is the default reject directory for all the event handlers.

The default value is BRM_home/apps/pin_rel/gsm/tel/archive.

batch.random.events

Specifies the name of the event type to process. If you have two or more types of events, separate each with a comma, but no blank space. For example, TEL,SMS,GPRS.

There is no default value for this parameter.

event.max.at.highload.time

Specifies the highest number of the RE Loader threads permitted to run simultaneously for this event during the high-load time (that is, from batch.start.highload.time to batch.end.highload.time).

For example, if event.max.at.highload.time is 2, two threads are permitted to run simultaneously for this event during the high-load time.

There is no default value for this parameter.

event.max.at.lowload.time

Specifies the highest number of the RE Loader threads permitted to run simultaneously for this event during the low-load time (that is, from batch.end.highload.time to batch.start.highload.time).

For example, if event.max.at.lowload.time is 2, two threads are permitted to run simultaneously for this event during the low-load time.

There is no default value for this parameter.

event.file.location

Specifies the full path name of the directory to monitor for the arrival of new files that match the pattern in event.file.pattern.

There is no default value for this parameter.

event.file.pattern

Specifies the file name pattern to look for. You can use an asterisk (*) to represent zero or more characters in the file name. No other wildcards are supported.

There is no default value for this parameter.

event.archiveDir

(Optional) Specifies the full path to the directory where a successfully processed file is archived for a particular event handler. When multiple event handlers are configured, configure this entry to specify the directory that archives files from a particular event handler.

There is no default value for this parameter. If it is not provided, the files are placed in the BRM_home/apps/pin_rel directory.

event.rejectDir

(Optional) Specifies the full path to the directory where an unsuccessfully processed file is stored for a particular event handler. When multiple event handlers are configured, configure this entry to specify the directory that stores files from a particular event handler.

There is no default value for this parameter. If it is not provided, the files are placed in the BRM_home/apps/pin_rel directory.

event.file.type

Specifies the type of input CDR file.

  • ECE_PRE_SPLIT specifies that RE Loader uses ECE generated preprocessed control files and data files.

  • STANDARD specifies that RE Loader uses pipeline generated input files.

The default is STANDARD.

Always use STANDARD for Pipeline Manager.

Note: You cannot specify both ECE_PRE_SPLIT and STANDARD in the same Infranet.properties file.

Default RE Loader Processing

Table 5-4 contains the RE Loader Infranet.properties file entries for configuring the default processing of CDRs. The configuration information in this section applies to all events except for those defined in the storable class-specific section.

Table 5-4 Default Configuration Entries

Entry Description

infranet.rel.default.interim_directory

Specifies the RE Loader processing directory. This is the location where RE Formatter loads the CDR files.

If not included in the file, ECE uses the value ./.

The default value is BRM_home/apps/pin_rel.

infranet.rel.default.supported_creation_processes

Specifies which creation processes are supported. As initially configured, RE Loader supports all creation processes:

  • PIN_REL_TRANSFORM_CDR specifies that the file was last processed by the pin_rel_transform_cdr.pl script and therefore contains discount events.

  • SUSPENSE_CREATE specifies that the RE Loader process creates new suspense records in the suspended usage table.

  • SUSPENSE_UPDATE specifies that the RE Loader process updates existing suspense records in the suspended usage table.

  • RATING_PIPELINE specifies that the file was last processed by the rating pipeline and therefore contains prerated events. This option applies only to Pipeline Manager.

  • RERATING_PIPELINE specifies that the file was last processed by the rerating pipeline and therefore contains rerated events. This option applies only to Pipeline Manager.

By default, RE Loader supports all creation processes.

infranet.rel.default.failure_script

Specifies the script called when RE Loader attempts to reload events that previously failed to load into the database.

The default value is pin_rel_handle_interim_files.pl.

infranet.rel.default.failure_flags

Specifies the flag passed to the failure script.

You can specify the following flags in the default pin_rel_handle_interim_files.pl failure script:

  • 0 to do nothing.

  • 1 to rename the interim files by appending .saved.timestamp to the file name.

  • 2 to delete the temporary files.

  • 3 to move the unsuccessfully processed data files generated by ECE to the reject directory. This option is only available for ECE.

The initial value and the default are 1.

infranet.rel.default.preprocess_script

Specifies the name of the preprocessing script.

The default value is pin_rel_preprocess_cdr.pl.

infranet.rel.default.preprocess_flags

Specifies the flag passed to the preprocessing script.

The default value is 0.

infranet.rel.default.load_util

Specifies the name of the load utility.

For Oracle's SQL Loader, it also specifies whether the utility uses direct-path loading or conventional-path loading:

  • Direct-path loading: This is the fastest way to load events into the database. It can be 10% to 30% faster than conventional-path loading, depending on the file size, memory size, storage configuration, and storage performance. However, direct-path loading has limits for concurrent system activities. When an event is loaded in direct-path mode, the load utility locks the event's entire partition and some of the table's indexes. This prevents other operations from updating or reading the event table. Direct-path mode is recommended when the event table has limited concurrent usage.

  • Conventional-path loading: This is the recommended loading mode if BRM performs many concurrent operations on the event table. For example, use conventional-path loading if BRM is rerating events, performing billing-time taxation, or generating detailed invoices concurrently with RE Loader. Conventional mode is also recommended if you have small source files for RE Loader because the performance gained by using direct-path loading is surpassed by the mode's preprocessing and file-handling overhead.

    Important: If you use conventional-path loading, use the APPEND option in your RE Loader control files. Do not use the TRUNCATE option.

To specify the load utility name and loading mode:

  • UtilityName direct=true unrecoverable specifies to use direct-path loading. This is the default.

  • UtilityName direct=false specifies to use conventional-path loading.

The default is sqlldr direct=true streamsize=5000000 readsize=10000000 unrecoverable.

infranet.rel.default.preupdater_sproc

Specifies the name of the preupdate stored procedure.

The default value is pin_rel.pin_rel_pre_updater_sp.

infranet.rel.default.preupdater_batch_size

Specifies the size of the preupdate batch.

The default value is 5.

infranet.rel.default.preupdater_flags

Specifies the flag passed to the preupdate stored procedure.

The default value is 1.

infranet.rel.default.updater_sproc

Specifies the name of the update stored procedure.

The default value is pin_rel.pin_rel_updater_sp.

infranet.rel.default.updater_batch_size

Specifies the size of the update batch.

The default value is 5.

infranet.rel.default.updater_flags

Specifies the flag passed to the update stored procedure. The default value is 1.

infranet.rel.default.success_script

Specifies the script called when RE Loader successfully loads a batch of events into the BRM database.

The default value is pin_rel_handle_interim_files.pl.

infranet.rel.default.success_flags

Specifies the flag passed to the success script.

You can specify the following flags in the default pin_rel_handle_interim_files.pl script:

  • 0 to do nothing.

  • 1 to rename the interim files by appending .saved.timestamp to the file name.

  • 2 to delete the temporary files.

The default value is 2.

infranet.rel.default.storable_class

Specifies the storable class you are loading.

Important: If you use Oracle SQL Loader, use the APPEND option in your RE Loader control files. Do not use the TRUNCATE option.

If not included in the file, ECE uses the value /event/delayed/session/gprs.

There is no default value for this parameter.

infranet.rel.default.creation_process

Specifies whether the file contains prerated, rerated, or discount events:

  • PIN_REL_TRANSFORM_CDR specifies that the file was last processed by the pin_rel_transform_cdr.pl script and therefore contains discount events.

  • RATING_PIPELINE specifies that the file was last processed by the rating pipeline and therefore contains prerated events. This option applies only to Pipeline Manager.

  • RERATING_PIPELINE specifies that the file was last processed by the rerating pipeline and therefore contains rerated events. This option applies only to Pipeline Manager.

Important: RE Loader can dynamically source the creation process from the event record header file. Remove the comment from this entry only if all of your event record files come from the same creation process.

infranet.rel.default.ece_control_file_directory

Specifies the location of the control files generated by BRM Elastic Charging Engine (ECE). The default is BRM_home/apps/pin_rel.

This parameter is not applicable to Pipeline Manager.

infranet.rel.default.ece_data_file_directory

Specifies the location of the data files generated by ECE. The default is BRM_home/apps/pin_rel.

This parameter is not applicable to Pipeline Manager.

infranet.rel.ece_preprocessed

Specifies whether RE Loader uses ECE generated preprocessed control files and data files:

  • TRUE specifies that RE Loader uses ECE generated preprocessed control files and data files.

  • FALSE specifies that RE Loader uses pipeline generated input files.

The default is FALSE.

This parameter should always be FALSE for Pipeline Manager.

Loading Event Records

Table 5-5 contains the RE Loader Infranet.properties file entries for configuring how RE Loader handles event records.

Table 5-5 RE Loader Event Handling Configuration Entries

Entry Description

infranet.rel.use_end_time

Specifies whether RE Loader uses the start time or end time of the rated event for deciding the billing cycle.

  • 1 specifies that RE Loader uses the end time of the rated event for deciding the billing cycle. The initial value and the default are 1.

  • 0 specifies that RE Loader uses the start time of the rated event for deciding the billing cycle.

infranet.rel.default.header.record_type

Specifies the header record type. The initial value and the default are 010.

infranet.rel.default.detail.record_type

Specifies the detail record type. The initial value and the default are 020.

infranet.rel.default.trailer.record_type

Specifies the trailer record type. The initial value and the default are 090.

infranet.rel.field.delimiter

Specifies the delimiter symbol. The initial value and the default are \t for tabs.

infranet.rel.header.position.storable_class

Specifies which field in the event record file contains the storable class name. The initial value and the default are 20.

Note: When you set this field to 0, RE Loader uses the default storable class specified in infranet.rel.default.storable_class.

infranet.rel.header.position.creation_process

Specifies which field in the event record file contains the name of the creation process (for example, whether the file contains prerated, rerated, or discount events). The initial value and the default are 18.

Note: You can specify 0 if you do not need this field validated.

infranet.rel.header.position.sender

Specifies which field in the event record file contains the sender. The initial value and the default are 3.

infranet.rel.header.position.recipient

Specifies which field in the event record file contains the recipient. The initial value and the default are 4.

infranet.rel.header.position.file_sequence

Specifies which field in the event record file contains the file sequence number. The initial value and the default are 5.

Note: You can specify 0 if you do not need this field validated.

infranet.rel.header.position.creation_timestamp

Specifies which field in the event record file contains the creation time stamp. The initial value and the default are 7.

Note: You can specify 0 if you do not need this field validated.

infranet.rel.trailer.position.record_count

Specifies the field position of the field that contains the total number of detail records in the output file.

The initial value and the default are 7. The field position starts with 1.

infranet.rel.file_extension.disc.transform_script

By default, this entry is commented out.

The initial value is pin_rel_transform_cdr.pl.

No default exists.

infranet.rel.file_extension.disc.transform_flags

By default, this entry is commented out.

The initial value is 0.

No default exists.

infranet.rel.header.position.object_cache_type

Set this value to 0. The default value is 23.

This parameter is not applicable to ECE.

Storable-Class RE Loader Processing

Table 5-6 contains the RE Loader Infranet.properties file entries for configuring CDR processing for specific storable classes.

For each storable class, only the infranet.rel.storable_class.classname.number_of_tables and infranet.rel.storable_class.classname.table.N.name entries are mandatory. RE Loader uses the values from the related infranet.rel.default settings for any undefined storable class-specific entries.

When editing these entries:

  • Create a set of entries for each event you want to load.

  • Replace classname with the appropriate storable class name. For example, use event_delayed_session_gprs for the /event/delayed/session/gprs storable class.

  • Create a set of *.table.N.* entries for each table. For example, if the storable class contains three tables, create a set of *.table.1.* entries, a set of *.table.2.* entries, and a set of *.table.3.* entries.

Table 5-6 Storable-Class-Specific Configuration Entries

Entry Description

infranet.rel.storable_class.classname.interim_directory

RE Loader processing directory. This is the location where preprocessed events are temporarily stored before they are loaded.

infranet.rel.storable_class.classname.supported_creation_processes

Specifies whether the file contains prerated, rerated, or discount events.

infranet.rel.storable_class.classname.failure_script

Specifies the script to call when RE Loader attempts to load events that previously failed to load into the database.

infranet.rel.storable_class.classname.failure_flags

Specifies the flag to pass to the failure script.

infranet.rel.storable_class.classname.preprocess_script

Specifies the name of the preprocessing script.

infranet.rel.storable_class.classname.preprocess_flags

Specifies the flag to pass to the preprocessing script.

infranet.rel.storable_class.classname.number_of_tables

Specifies the number of tables in the storable class.

Important: This entry is mandatory for all types of events.

infranet.rel.storable_class.classname.table.N.name

Specifies the name of a storable class table.

Important: This entry is mandatory for all types of events.

infranet.rel.storable_class.classname.table.N.load_util

Specifies the name of the load utility. For Oracle's SQL Loader, it also specifies whether the utility uses direct-path loading or conventional-path loading:

  • UtilityName direct=true unrecoverable specifies to use direct-path loading.

  • UtilityName direct=false specifies to use conventional-path loading.

Important: If you use conventional-path loading, use the APPEND option in your RE Loader control files. Do not use the TRUNCATE option.

infranet.rel.storable_class.classname.table.N.control_file

Specifies the control file to use when loading the data file into the database.

infranet.rel.storable_class.classname.preupdater_sproc

Specifies the name of the preupdater stored procedure.

infranet.rel.storable_class.classname.preupdater_batch_size

Specifies the preupdater batch size.

infranet.rel.storable_class.classname.preupdater_flags

Specifies the flag to pass to the preupdater stored procedure.

infranet.rel.storable_class.classname.updater_sproc

Specifies the name of the updater stored procedure.

infranet.rel.storable_class.classname.updater_batch_size

Specifies the updater batch size.

infranet.rel.storable_class.classname.updater_flags

Specifies the flag to pass to the updater stored procedure.

infranet.rel.storable_class.classname.success_script

Specifies the script to call when RE Loader successfully loads a data file into the BRM database.

infranet.rel.storable_class.classname.success_flags

Specifies the flag to pass to the success script when RE Loader successfully loads a data file into the BRM database.

RE Loader Logging

Table 5-7 contains the RE Loader Infranet.properties file entries for configuring logging.

Table 5-7 RE Loader Logging Configuration Entries

Entry Description

infranet.log.file

Specifies the name of the RE Loader log file.

The initial value is rel.pinlog.

No default exists.

infranet.log.name

Specifies the name of the application.

The initial value is REL for RE Loader.

No default exists.

infranet.log.level

Specifies the log reporting level:

  • 1 specifies error-level reporting.

  • 2 specifies warning-level reporting.

  • 3 specifies debug-level reporting.

The initial value is 1.

The default is String.valueOf(ErrorLog.Error) (equivalent to 1).

See "Setting the Reporting Level for Logging Messages" in BRM System Administrator's Guide.

infranet.log.logallebuf

Specifies whether RE Loader automatically logs all EbufException errors.

The initial value is true.

No default exists.

infranet.rel.custom_error_codes

Specifies the name of the custom error code file.

The initial value is CustomErrorCodes.properties.

No default exists.

To move this file from its default location, you must create a symbolic link between the name of the file and its new location. To create this link, go to the BRM_home/apps/pin_rel directory and enter the following at the command prompt: 

ln -s path_to_where_file_was_moved/CustomErrorCodes.properties ./CustomErrorCodes.properties

Configuring the Event Stream Processor Infranet.properties File

This information applies to ESP settings for internal ECE Rated Event Formatter (REF) JSONFILE events, ECE Composable Services events, and supported externally rated events consumed from Kafka. ECE ZIPFILE streaming uses the RE Manager Stream Processor. For more information about these processing paths, see "Event Streaming Mode".

Table 5-8 through Table 5-22 list the shared ESP settings and source-specific entries in the ESP Infranet.properties file. Use these tables as the property reference when following "Processing Rated Events for Loading Using Kafka".

To configure the ESP Infranet.properties file:

  1. Open the BRM_home/apps/event_stream_processor/Infranet.properties file in a text editor.

  2. Edit the file.

  3. Save and close the file.

  4. Start or restart ESP.

Note:

The tables use the following labels to show which event source a setting applies to:

  • (Internal event) applies to the ECE REF JSONFILE path, including its internal-event batching settings.

  • (ECE Composable Services) applies to ECE Composable Services events.

  • (External event) applies to third-party prerated events.

  • No label means the setting applies to all three event sources.

The following topics are in this section:

Kafka Consumer Configuration

You can configure the parameters in this section to poll Kafka topics and pass the events in batches to the ESP for processing.

Properties prefixed with kafka.config. use Apache Kafka consumer configuration names. ESP removes the prefix before passing the settings to the Kafka consumer.

Table 5-8 lists the parameters for polling Kafka topics and passing events in batches.

Table 5-8 Kafka Consumer Entries

Name Default Description

kafka.config.auto.offset.reset

earliest

The value to use only when a consumer has no valid committed offset for a partition.

Available values are:

  • earliest: Starts consumption from the oldest available message in the partition.
  • latest: Starts consumption from the next new message that arrives after the consumer begins.
  • none: Returns an error if there is no valid committed offset for the partition.

kafka.config.bootstrap.servers

${HOSTNAME}:9092

A list of host/port pairs to use for establishing the initial connection to the Kafka cluster.

This list should be in the form:

host1:port1,host2:port2,...

kafka.config.enable.auto.commit

false

Specifies if the consumer's offset is periodically committed in the background.

Note: This parameter is disabled for manual offset commits.

kafka.config.key.deserializer

org.apache.kafka.common.serialization.StringDeserializer

The deserializer class for keys.

kafka.config.value.deserializer

org.apache.kafka.common.se rialization.ByteArrayDeserializer

The deserializer class for values.

kafka.config.max.poll.records

5

The maximum number of records returned in a single poll.

Note: Set max.poll.records to 1 when you enable batching in Rated Event Formatter. Also, ensure that you set event_stream_processor.ref_batching_enabled to true.

Kafka Consumer Availability and Batching Configuration

You can configure the parameters in this section to manage how the consumer connects to the Kafka broker and processes events in batches.

Table 5-9 lists the parameters for connecting to the Kafka broker and processing events in batches.

Table 5-9 Kafka Consumer Availability and Batching Entries

Name Default Description

kafka.server.check.retries

5

The number of retry attempts to connect to the Kafka server if it is unreachable.

kafka.server.check.timeout.ms

10000

The timeout, in milliseconds, to wait for a response from the Kafka server for each attempt.

kafka.server.offset.retry.delay.ms

5000

The delay, in milliseconds, before retrying offset retrieval from the database when an error occurs during an offset fetch.

kafka.server.poll.timeout.ms

1000

The maximum time, in milliseconds, to wait for messages during each poll call.

event_stream_processor.ref_batching_enabled (Internal event)

false

Specifies if streaming mode is enabled for internal event batch processing.

kafka.server.batch.timeout.ms

No default value

The optional timeout, in milliseconds, before flushing a partial batch.

Helidon Server Configuration

You can configure the parameters in this section to manage how the Helidon server binds to network interfaces and listens for incoming HTTP requests.

Table 5-10 lists the parameters for configuring the Helidon server’s network interface and port settings.

Table 5-10 Helidon Server Entries

Name Default Description

server.host

${HOSTNAME}

The network interface the server binds to.

server.port

8080

The port on which the Helidon server listens for incoming HTTP requests.

Metrics Configuration

You can configure the parameters in this section to manage how the application monitors and collects metrics during operation.

Table 5-11 lists the parameters for configuring metric monitoring intervals and thread settings.

Table 5-11 Metric Entries

Name Default Description

metrics.monitor.retry.interval

60

The interval, in seconds, for retrying metric monitoring operations.

metrics.monitor.thread.name

kafka-consumer-metrics-monitor

The name of the thread that monitors Kafka consumer metrics.

Topic Configuration

You can configure the topics for the ESP to consume events from Kafka. Configure a topic for each event source and a matching handler when the topic requires a specific handler.

ESP supports topic-specific Kafka consumer settings that override the corresponding default settings prefixed with kafka.config.. To configure a setting for a specific topic, use topic.consumer.config.topic-name.configuration-name,

where:

  • topic-name identifies the Kafka topic.

  • configuration-name identifies the Kafka consumer setting.

For example, the following entry sets max.poll.records to 100 for test-topic, overriding kafka.config.max.poll.records only for that topic: 

topic.consumer.config.test-topic.max.poll.records=100

Table 5-12 lists the parameters for configuring these event topics. Sequence determines the order in which the system consumes internal Kafka topics.

Table 5-12 Topic Entries

Name Default Description

event_stream_processor.topic.internal.sequence (Internal event)

RatedEvents

The name of the internal Kafka topic to be consumed.
event_stream_processor.topic.custom.sequence (ECE Composable Services and External event)

No default value

The name of the custom Kafka topic to be consumed.

topic.consumer.count.topic-name

1

The number of consumers to start for the specified Kafka topic.

topic.consumer.config.topic-name.max.poll.records

100

The maximum number of records returned in a poll for the specified Kafka topic.

topic.consumer.config.topic-name.value.deserializer (ECE Composable Services)

io.cloudevents.kafka.CloudEventDeserializer

Reads CloudEvents from the specified ECE Composable Services Kafka topic.

Handler Configuration

You can configure handler classes to process internal and custom events.

Table 5-13 lists the parameters for configuring event handler classes responsible for handling various events. Sequence determines the order in which the system handles events.

Table 5-13 Handler Entries

Name Default Description

event_stream_processor.handler.default (Required)

com.oracle.brm.esp.core.event_stream_processor.handler.EventHandlerImpl

The default event handler implementation.

event_stream_processor.handler.internal.sequence (Internal event)

com.oracle.brm.esp.core.event_stream_processor.handler.InternalEventHandlerImpl

The class for handling the Rated Event Formatter (REF) generated events.

event_stream_processor.handler.custom.sequence (ECE Composable Services and External event)

com.oracle.brm.esp.core.event_stream_processor.handler.CustomEventHandlerImpl

The class for handling events from the topic with the matching custom.sequence suffix.

Note:

If a handler entry is already configured for the ECE Composable Services topic, set it to the same fully qualified EventHandlerImpl class listed for ECE Composable Services events in Table 5-13. A matching handler entry takes precedence over event_stream_processor.handler.default.

JDBC Connection Pool Configuration

You can configure the parameters to define how the event stream processor connects to the BRM database.

Table 5-14 lists the parameters available for configuring the JDBC connection pool for database connectivity.

Table 5-14 JDBC Connection Pool Entries

Name Default Description

event_stream_processor.jdbc_pool.connectionURL

jdbc:oracle:thin:@//HOST:{PORT}/PINDB

The connection URL for the BRM database.

event_stream_processor.jdbc_pool.connectionWaitTimeout

10

The maximum time, in seconds, to wait for a connection.

event_stream_processor.jdbc_pool.fastConnectionFailoverEnabled

false

Specifies whether Fast Connection Failover (FCF) is enabled.

event_stream_processor.jdbc_pool.initialPoolSize

1

The initial number of connections to create in the pool.

event_stream_processor.jdbc_pool.jmxEnabled

true

Specifies if UCP JMX metrics are enabled.

event_stream_processor.jdbc_pool.maxPoolSize

50

The maximum number of connections permitted in the pool at once.

event_stream_processor.jdbc_pool.maxStatements

50

The maximum number of database statements that can be cached.

event_stream_processor.jdbc_pool.metricUpdateInterval

10

The frequency, in seconds, for updating UCP metrics.

event_stream_processor.jdbc_pool.minPoolSize

1

The minimum number of connections maintained when idle.

event_stream_processor.jdbc_pool.nonTransientErrorCodes

1,1450,12899,1722,20010,1489,6550

The nontransient error codes. These identify persistent errors that are not retried.

event_stream_processor.jdbc_pool.onsConfiguration

No default value

The Oracle Notification Services configuration for FCF.

event_stream_processor.jdbc_pool.poolLogLevel

INFO

The initial log level for UCP logging.

event_stream_processor.jdbc_pool.sqlForValidateConnection

SELECT SYSDATE FROM DUAL

The SQL command for validating database connections if enabled.

event_stream_processor.jdbc_pool.user

pin

The BRM database user name.

event_stream_processor.jdbc_pool.validateConnectionOnBorrow

false

Specifies if each connection is validated when retrieving from the pool.

event_stream_processor.jdbc_pool.validateSchema

true

Specifies if the BRM schema is validated on pool creation.

event_stream_processor.jdbc_pool.wallet_entry_name

infranet.rel.password

The wallet entry name for the database password.

event_stream_processor.jdbc_pool.wallet_location

${BRM_CONF_WALLET}

The path to the BRM wallet for the database.
event_stream_processor.jdbc_pool.user.dbno pindbno The user name for the specified BRM database schema.
event_stream_processor.jdbc_pool.wallet_entry_name.dbno infranet.rel.password.1 The wallet entry containing the password for the specified BRM database schema.

SSL DB Configuration

You can configure the parameters to specify how the event stream processor manages SSL settings for secure connections to the BRM database.

Table 5-15 lists the parameters for configuring SSL options in the JDBC connection pool.

Table 5-15 SSL DB Entries

Name Default Description

event_stream_processor.jdbc_pool.dbSSLClientAuth

false

Specifies if SSL client authentication is enabled.

event_stream_processor.jdbc_pool.dbSSLEnabled

false

Specifies if SSL is enabled for database connections.

event_stream_processor.jdbc_pool.sslKeyStoreLocation

DB_SSL_KEYSTORE_LOC/cwallet.sso

The path to the SSL keystore.

event_stream_processor.jdbc_pool.sslTrustStoreLocation

DB_SSL_TRUSTSTORE_LOC/cwallet.sso

The path to the SSL trust store.

event_stream_processor.jdbc_pool.sslServerCertDN

No default value

The name of the SSL server certificate.

event_stream_processor.dbno

1

Specifies the BRM database schema number.

Note: If event_stream_processor.dbno is omitted, ESP assumes schema 0.

event_stream_processor.jdbc_pool.connectionURL.dbno jdbc:oracle:thin:@//${HOSTNAME}:1521/PINDB1 The JDBC URL for the specified BRM database schema.

Mapper Configuration

You can configure a mapper when ECE Composable Services or external event records require mapping to the BRM event structure.

Table 5-16 lists the mapper configuration entries. The supplied file specifies a mapper configuration path and sets the mapper thread-pool capacity to 0.

Table 5-16 Mapper Entries

Name Default Description
mapper.configuration.path ${PIN_HOME}/apps/event_stream_processor/eem/scenarios/application.yaml

The path to the mapper application.yaml file.

event_stream_processor.mapper_thread_pool_capacity 0

The maximum mapper thread-pool capacity.

Note:When you set this entry to 0, ESP sizes the pool based on the processors available to the JVM.

The application.yaml file specifies the location of mapping specification files and the routing rules that ESP uses to select them. For non-JSON input, you can also configure a converter in this file to convert incoming records to JSON before mapping.

Table 5-17 describes the configuration entries and their nested entries in application.yaml.

Table 5-17 Mapper Application Configuration Entries

Entry or Subpath Description
baseMappingDir Specifies the directory that contains the mapping specification files.
mapperSpecificationKeys Specifies routing rules that associate fields and values in the incoming JSON event with mapping specifications.
JSON-field-path under mapperSpecificationKeys Identifies the incoming field that ESP evaluates to select a mapping specification, such as serviceType.
matching-value under JSON-field-path Associates an incoming field value with a mapping specification name. Specify the name without the .yaml extension. ESP resolves the specification file in baseMappingDir.
jsonConverters Specifies optional converter configurations for non-JSON input. Omit this entry when the incoming format does not require conversion to JSON.
converter-name under jsonConverters

Identifies a converter configuration, such as partnerCsv or partner-delimited. Its child entries define the supported record types, converter class, and converter-specific settings. For more information, see the examples in "Configuring Mapper Scenarios in application.yaml" and "Configuring a Custom Format Adapter".

supportedRecordTypes under converter-name Lists the incoming record types that the converter handles, such as text/csv.
converterClass under converter-name Specifies the fully qualified name of the converter implementation class.
configuration under converter-name Specifies settings that the converter implementation defines.
setting-name under configuration Specifies an individual converter setting. You can specify any setting supported by your converter implementation. For example, metadataPath can identify a metadata file if the converter defines that setting. Setting names and requirements depend on the converter implementation. For more information, see the example in "Configuring Mapper Scenarios in application.yaml".

For a configuration procedure and an example showing these entries, see "Configuring Mapper Scenarios in application.yaml".

For information about defining converter settings, see "Configuring a Custom Format Adapter".

For converter implementation requirements, see "About the Format Adapter Interface".

For deployment and restart instructions, see "Compiling and Deploying a Custom Format Adapter".

The selected mapping specification defines how ESP maps incoming values to BRM tables and fields. For information about configuring these mappings, see "Configuring Mapper Specifications".

Event Deduplication Configuration

You can configure ESP to check event keys for duplicates before committing ECE Composable Services or external events.

Table 5-18 lists the event deduplication entries.

Table 5-18 Event Deduplication Entries

Name Default Description

event_stream_processor.dedup.enabled

true

Enables or disables ESP duplicate-event checking.

event_stream_processor.dedup.retention_days

30

Number of days to retain accepted event keys before expired partitions are deleted.

event_stream_processor.dedup.sproc_logging.enabled

-1

Specifies the buffer size for troubleshooting messages from the deduplication stored procedure. Set it to -1 to disable the messages, 0 to allow unlimited output, or a positive number to set a limit. ESP writes the messages to its DEBUG log.

BRM Business Time Configuration

You can configure ESP to use the JVM clock or get the current BRM private virtual time from CM for each batch of ECE Composable Services or external events.

Table 5-19 lists the parameters for selecting the business time source and configuring the CM connection.

Table 5-19 BRM Business Time Entries

Name Default Description

event_stream_processor.time.source

system

Specifies the source and cm-pvt of business time.

event_stream_processor.time.cm.login_type

1

The CM login type used when ESP retrieves business time from CM.

event_stream_processor.time.cm.ssl.enabled

false

Specifies whether SSL/TLS is enabled for ESP’s connection to CM.

event_stream_processor.time.cm.ssl.wallet.location

${PIN_HOME}/wallet/client

The path to the wallet ESP uses to connect securely to CM.

event_stream_processor.time.cm.ssl.wallet.filename

cwallet.sso

The name of the wallet file ESP uses to connect securely to CM.

Enrichment Module Configuration

You can configure batch enrichment classes to process external events before ESP commits them to BRM. ESP runs the classes in the configured sequence. ESP also supports these entries for ECE Composable Services TMF events, but the TMF path does not currently require batch enrichment.

Table 5-20 lists the parameters for configuring batch enrichment classes and selector rules.

Table 5-20 Enrichment Entries

Name Default Description

event_stream_processor.batch.enrichment.class.sequence (External event)

No default value

Specifies an enrichment class.

event_stream_processor.selector.config_path (External event)

${PIN_HOME}/apps/event_stream_processor/generic_selector.yaml

The path to the YAML file containing rules and conditions for the configured selector classes.

Kafka SSL and SASL Authentication Configuration

You can configure ESP to use SSL and SASL authentication when it connects to Kafka. Configure the security protocol and authentication mechanism required by the Kafka broker.

Table 5-21 lists the entries for configuring Kafka SSL and SASL authentication.

Table 5-21 Kafka SSL and SASL Authentication Entries

Name Default Description

kafka.config.security.protocol

SASL_SSL

Specifies the security protocol used to connect to Kafka.

kafka.config.sasl.mechanism

PLAIN

Specifies the SASL authentication mechanism.

kafka.config.ssl.truststore.location

/path/to/client.truststore.jks

The absolute path to the trust store used to validate the Kafka broker certificate.

kafka.config.ssl.truststore.password

No default value

The BRM wallet entry reference for the trust store password.

kafka.config.ssl.keystore.location

/path/to/client.keystore.jks

The absolute path to the client keystore. Specify this entry when the Kafka broker requires mutual TLS authentication.

kafka.config.ssl.keystore.password

No default value

The BRM wallet entry reference for the client keystore password. Specify this entry when the Kafka broker requires mutual TLS authentication.

kafka.config.ssl.key.password

No default value

The BRM wallet entry reference for the client private key password. Specify this entry when the Kafka broker requires mutual TLS authentication.

kafka.config.sasl.jaas.config

org.apache.kafka.common.security.plain.PlainLoginModule required username="espkafka" password="{Password}";

Specifies the JAAS login configuration for the selected SASL mechanism.

kafka.config.sasl.oauthbearer.token.endpoint.url

http://<oauth-server-host>:<port>/realms/<realm-name>/protocol/openid-connect/token

The token endpoint URL used for OAUTHBEARER authentication. Include this URL in the Kafka client’s allowed URLs configuration.

kafka.config.sasl.login.callback.handler.class

org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginCallbackHandler

The login callback handler class used for OAUTHBEARER authentication.

oauth.allowed.urls

No default value

Specifies the allowed OAuth token endpoint URLs. Separate multiple URLs with commas.

Additional ESP Configuration

You can configure schema-specific database entries when ESP connects to more than one BRM database schema.

Table 5-22 lists the parameters for configuring connections to additional BRM schemas.

Table 5-22 Additional ESP Entries

Name Default Description

event_stream_processor.partition_set

1

The partition set number to use for delayed events.

event_stream_processor.poid_increment_by

5

The maximum allowed increment value for POID ranges.