22 Loading Events from External Sources
This chapter describes how to load event data from event log files into the Oracle Communications Billing and Revenue Management (BRM) database to be rated. It also provides troubleshooting information for loading events.
For general information about importing events, see "About Rating Events Created by External Sources".
About Loading Events
You load data from an event log file into the BRM database by creating an event import template with UE Mapper. You then load the event data manually by using UE Loader or automatically by using Batch Controller and a batch handler.
UE Loader is a command-line utility that reads data about service usage from the event log file, reads the event import template for instructions on loading the data, such as which opcodes to call to load the data, and loads the data into the BRM database. When the events are loaded, BRM rates the events and changes customer account balances. You can set Batch Controller and create a UE Loader batch handler to run UE Loader automatically (to load event data on a regular basis). For information about Batch Controller, see "Controlling batch operations" in BRM System Administrator's Guide.
For information on creating an event import template and a technical overview of the event import process, see "About Rating Events Created by External Sources".
Figure 22-1 shows how UE Mapper and UE Loader work to load events into the BRM database:
Figure 22-1 Loading Events into BRM Using UE Mapper and UE Loader
Description of ''Figure 22-1 Loading Events into BRM Using UE Mapper and UE Loader''
Configuring UE Loader
To configure UE Loader, edit the UE Loader properties file (BRM_Home/apps/uel/Infranet.properties). Follow these procedures:
Configuring Connection Information for UE Loader
To specify connection information:
-
Open UE Loader properties file (BRM_Home/apps/uel/Infranet.properties).
-
Follow the instructions in the file to set the standard connection information, such as the Connection Manager (CM) name and port number.
-
Save and close the file.
Enabling UE Loader to Convert Flists to Strings
Important:
This is a mandatory configuration.Ensure that UE Loader is configured to convert flists to strings:
-
Open UE Loader properties file (BRM_Home/apps/uel/Infranet.properties).
-
Ensure that the infranet.flist.tostring.enable entry is set to true:
infranet.flist.tostring.enable = trueIf this entry is not present or is set to false, UE Loader might load records into the wrong accounts.
Specifying the Event Log File Location
When you run UE Loader, you specify only the name of the event log file, not the file location. You specify the file location in the Infranet.properties file.
For information about UE Loader log files, see "Troubleshooting Event Loading".
-
Open the properties file (BRM_Home/apps/uel/Infranet.properties).
-
Enter the directory name in the infranet.uel.event_log_file_location entry. This is the directory that stores event log files for loading.
When specifying the path, always use forward slashes (/), and type a slash after the last directory name.
infranet.uel.event_log_file_location=/var/portal/7.3.1/apps/uel/ -
Save and close the file.
Specifying Whether to Log Duplicate Event Log Files
By default, UE Loader does not notify you when a duplicate event log file is loaded into the database.
You can configure UE Loader to log an error when a duplicate event log file is loaded by using the infranet.uel.duplicate_check entry. When this entry is set to True, UE Loader creates a DuplicateFiles.txt file and adds the following for each duplicate file:
-
The duplicate event log file's name
-
The timestamp of when the duplicate event log file was processed
To log an error when duplicate event log files are loaded into the database:
-
Open UE Loader properties file (BRM_Home/apps/uel/Infranet.properties).
-
Change the infranet.uel.duplicate_check entry to True:
parameter infranet.uel.duplicate_check=True -
Save and close the file.
Setting the Number of Threads for Performance
You can improve UE Loader performance by changing the maximum number of threads. Running multiple threads loads multiple events simultaneously, which gives faster performance but puts a greater load on the BRM system. The number of threads that you specify depends on your BRM system configuration, for example, the number and speed of CPUs.
Note:
The number of threads used when processing also depends on the event import template that you use to load the events. For example, if you load events based on their occurrence in the event log file, they are processed one at a time, so using multiple threads does not improve performance. For more information, see "Specifying How to Load Events".-
Open the UE Loader properties file (BRM_Home/apps/uel/Infranet.properties).
-
Change the infranet.uel.max_threads entry, for example:
infranet.uel.max_threads=5 -
Save and close the file.
Tip:
You can run multiple UE Loaders to improve performance. Each UE Loader can run multiple threads. If you have only one CM and one DM, however, there is a limit to how much you can improve performance by using multiple UE Loaders.
Setting Memory Options
Avoid memory errors with these guidelines:
-
UE Loader can process large log files, such as files with 200,000 records. The memory required to load events depends on the number of records in your event log file.
-
If you have a large number of records and UE Loader reports an out-of-memory error (error code -2), change the infranet.uel.queue_size entry in the Infranet.properties file. Setting this entry causes UE Loader to read records into memory in sizable chunks rather than reading all records into memory at once.
Important:
If the infranet.uel.queue_size entry in the Infranet.properties file is set to 0, all records are read into memory at once. -
You can configure the memory available to UE Loader by modifying the -mx setting in the uel script.
Specifying How to Load Events
You can load events based on their occurrence in the event log file, or you can load them based on their account identifier. Specify the loading method when you create the event import template by using the Group events by account identifier option in UE Mapper. For more information, start Developer Center and see the UE Mapper Help.
Important:
If a custom bill item is rated cumulatively, UE Loader must group events by account identification for loading. For more information, see UE Mapper Help.Configuring Log File Locations for UE Loader
To specify locations of log files:
-
Open the UE Loader properties file (BRM_Home/apps/uel/Infranet.properties).
-
Enter directory names in the following entries:
-
To set the location of the error file that records errors found in event loading mode, use the infranet.uel.load_error_file_location entry.
-
To set the location for the record of successfully loaded events, use the infranet.uel.load_success_file_location entry.
-
To set the location for the standard BRM client error log file, use the infranet.uel.error_log_file_location entry.
-
To set the location for the file that records events you filter by event field values, use the infranet.uel.filter_log_file_location entry.
When specifying the path, always use forward slashes (/), and type a slash after the last directory name.
infranet.uel.error_log_file_location=/var/portal/7.3.1/apps/uel/ -
-
Save and close the file.
Specifying the Maximum Load Errors to Process before Quitting
The maximum number of errors applies to errors generated from all threads, not errors per thread.
-
Open the properties file (BRM_Home/apps/uel/Infranet.properties).
-
Set the infranet.uel.max_errors entry, for example:
infranet.uel.max_errors=100 -
Save and close the file.
Specifying the Directory to Store Logged Event Records
You can set up filters so that particular records are logged to a file, whether you load them or not. See "About Discarding Event Records Based on Record Content" and "About Logging Event Records Based on Record Content".
To specify the directory to store the files that log filtered records:
-
Open the UE Loader properties file (BRM_Home/apps/uel/Infranet.properties).
-
Set the infranet.uel.filter_log_file_location entry.
-
Save and close the file.
Specifying the Default Date/Time Format for Timestamps
You can specify a timestamp format to support loading events from different countries that format dates and times in different ways. The timestamp format must match the date/time format in the event log file that you are importing. The timestamp format is used by fields with the data type PIN_FLDT_TSTAMP.
You can specify the date/time format for each timestamp field (fields with the data type PIN_FLDT_TSTAMP) in your event log file when you create the event import template. This is necessary if you have records that use different timestamp formats in the same event log file.
If you do not specify the date/time format for a record on a per field basis when you create the event import template, UE Loader uses the date/time format specified in its Infranet.properties file as the default format.
To set the default date/time format:
-
Determine the date format by looking at the event log file you are importing.
-
Open the properties file (BRM_Home/apps/uel/Infranet.properties).
-
Change the infranet.uel.date_pattern entry to match the date format in the event log file you are importing. For example. if your records use a time format specified in a.m./p.m. hours (01 - 12), you specify hours by using a lowercase h and including the a parameter to indicate that the time is in a.m./p.m. format:
infranet.uel.date_pattern=dd/MMM/yyyy:hh:mm:ss a zzzz -
Save and close the file.
Specifying Time Format Symbols
Table 22-1 shows examples of date formats:
Table 22-1 Example Date/Time Formats
| Date in Event Log File | Date/time Format in Infranet.properties File |
|---|---|
|
26/October/2005:10:55:49 Pacific Standard Time |
dd/MMM/yyyy:hh:mm:ss a zzzz |
|
12:08 PM |
h:mm a |
|
12 o'clock PM, Pacific Daylight Time |
hh 'o''clock' a, zzzz |
|
2005.07.10 AD at 15:08:56 PDT |
yyyy.MM.dd G 'at' hh:mm:ss z |
|
Wed, July 10, '05 |
EEE, MMM d, ''yy |
|
0:00 PM, PST |
K:mm a, z |
|
2005.July.10 AD 12:08 PM |
yyyyy.MMMMM.dd GGG hh:mm aaa |
Specifying Time Format Syntax
Each element in the time format syntax is represented by a character. For example, the year is represented by the "y" character. You control how each element is matched by the number of characters you use. For example, with the year 2001:
-
yy matches 01
-
yyyy matches 2001
There are also different symbol types, such as Text and Number. The number of characters has a different meaning for different symbol types as shown in Table 22-2:
| Symbol Type | Rules for Number of Characters |
|---|---|
|
Text |
4 characters or more uses the full word. Fewer than 4 characters presents an abbreviation. For example, for E (day of the week):
|
|
Number |
Any number of characters uses the minimum required number of digits, except for years. With years, 2 characters uses the short version, for example: yy = 99 |
|
Text and number |
3 characters or more uses text, 1 or 2 characters uses numbers. For example:
|
In addition, you can use an escape character to match text as shown in Table 22-3:
Table 22-3 Use of Escape Character to Match Text
| Log File | Date/Time Symbols |
|---|---|
|
July 2 at 12:01 |
MMM d 'at' hh:mm |
To match a single quotation mark, use double-single quotation marks as a literal, as shown in Table 22-4:
Any characters outside the range of a-z and A-Z, such as colon (:), semicolon (;), and at symbol (@), are treated as quoted text, even if they are not enclosed in single quotation marks.
Important:
Characters within the ranges of a-z and A-Z that are not used as symbols, such as C, and are not enclosed in quotes, cause an error.Allowing for Leniency in Time References
When a CDR contains a date value that uses a +1300 time reference, the system cannot read the date properly and returns an error. For example, the date entry might look like this: 20061202000000+1300.
To enable the system to process CDRs that use the +1300 time zone reference, set the Infranet.uel.lenient value to true in the UE Loader properties file (BRM_Home/apps/uel/Infranet.properties) The default value is false:
Infranet.uel.lenient = true
Configuring UE Loader for International Log Files
This section provides instructions for configuring UE Loader to use international log files.
Configuring the UE Loader Infranet.properties File
UE Loader supports English, French, German, and Japanese log files. To support these languages, set the following parameters in the UE Loader Infranet.properties file (BRM_Home/apps/uel/Infranet.properties) shown in Table 22-5:
Table 22-5 Parameters in UE Loader File
| Infranet.properties File Entry | Description |
|---|---|
|
infranet.uel.event_log_file_lang_code |
Specify the language code. See "Language, Country, and Encoding Codes". |
|
infranet.uel.event_log_file_country_code |
Specify the country code. UE Loader supports all country codes that apply to each language (English, French, German, and Japanese). |
|
infranet.uel.event_log_file_variant_code |
Specify the variant code. |
|
infranet.uel.event_log_file_encoding |
Specify the encoding code. See "Language, Country, and Encoding Codes". |
|
infranet.uel.date_pattern |
Specify the date pattern for the locale. See "Specifying the Default Date/Time Format for Timestamps". |
Language, Country, and Encoding Codes
Listed below are the language, country, and encoding codes for languages supported by each operating system.
Note:
UE Loader supports all country codes for each language.Table 22-6 lists the system encoding support for English.
Table 22-6 System Encoding Support for English
| Operating System | Language Code | Country Codes | Encoding Code |
|---|---|---|---|
|
Solaris |
en |
AU, CA, IE, NZ, ZA, GB, and US |
8859-1 |
|
Linux |
en |
AU, CA, IE, NZ, ZA, GB, and US |
None |
|
HP-UX IA64 |
en |
AU, CA, IE, NZ, ZA, GB, and US |
ISO81 |
|
AIX |
en |
AU, CA, IE, NZ, ZA, GB, and US |
ISO8859-1 |
Table 22-7 lists the system encoding for Japanese.
Table 22-7 System Encoding Support for Japanese
| Operating System | Language Code | Country Codes | Encoding Code |
|---|---|---|---|
|
Solaris |
ja |
JP |
SJIS |
|
Linux |
ja |
JP |
None |
|
HP-UX IA64 |
ja |
JP |
sjis |
|
AIX |
ja |
JP |
IBM-943 |
Table 22-8 lists the system encoding support for German.
Table 22-8 System Encoding Support for German
| Operating System | Language Code | Country Codes | Encoding Code |
|---|---|---|---|
|
Solaris |
de |
AT, DE, LU, and CH |
8859-1 |
|
Linux |
de |
AT, DE, LU, and CH |
None |
|
HP-UX IA64 |
de |
AT, DE, LU, and CH |
ISO81 |
|
AIX |
de |
AT, DE, LU, and CH |
ISO8859-1 |
Table 22-9 lists the system encoding for French.
Loading Events
You can load events manually by running UE Loader from a command line or you can schedule UE Loader to run automatically. See:
About the Order of Events
If any event balance impacts are configured to start immediately and never expire, consider the order of events when you load them.
When an event is loaded, BRM checks the validity periods of the account's resource sub-balances. The sub-balance that has a start time earlier than the event's start time is the one that receives the impact. If there is no sub-balance that has an earlier start time, BRM creates a new sub-balance, sets its start time to the event's start time, and impacts that sub-balance. This can cause the creation of multiple sub-balances for the same resource when events are loaded out of order. Having multiple sub-balances for the same resource can impact performance.
For more information, see "How Resources in Validity-Based Sub-Balances Are Updated".
To prevent the creation of multiple account sub-balances for resources that start immediately and never expire, be sure to sort the events in ascending order based on their timestamp before running UE Loader.
Loading Events Manually
Use UE Loader to manually load events from data log files into the BRM database. UE Loader is a command-line utility installed with BRM, and its executable file is in BRM_Home/apps/uel. For information on UE Loader command-line syntax, see "Universal Event Loader".
For information on loading events from data log files automatically, see "Loading Events Automatically".
Before You Begin
-
You must configure UE Loader before you can load events. See "Configuring UE Loader".
-
To avoid memory errors, follow the guidelines in "Setting Memory Options".
-
An event import template is required to load event data. You must know the exact name of the event import template that was created for the event log file you are loading.
-
Before loading events, you must specify the date/time format used in the event log file. You can specify the date/time format for all records in your log file by editing the UE Loader Infranet.properties file. See "Specifying the Default Date/Time Format for Timestamps". The date/time format for each record can also be specified on a per field basis when the event import template is created. If the date/time format is not specified on a per field basis, the date/time format specified in the Infranet.properties file is used.
-
Before loading events, run the UE Loader program on a test database to test the event import template and any custom opcodes that were created for loading your log file events.
To load events manually, run UE Loader by using this syntax:
uel -t template_name [-v] [-m parse|load] [-test] log_file_name
The optional -v verbose parameter causes UE Loader to output error messages or debug messages on the console according to the debug level you set in the Infranet.properties file.
The -m and -test modes are used more during testing of event loading. When you are loading events into a production database, you do not need to use these options.
For example, if the template name is WebServer and the event log file name is webserver.log, use this command to load events:
uel -t WebServer webserver.log
-
Do not enter the path of the event log file. You specify the path in the UE Loader Infranet.properties file. See "Specifying the Event Log File Location".
-
The log file can use any file extension or no extension.
For information on UE Loader command-line syntax, see "Universal Event Loader".
Loading Events Automatically
You can load event log files automatically and manually. Some BRM service integration components provide a batch handler that executes UE Loader to process event log files automatically, while BRM system software provides Batch Controller that controls the handler. You can also write a custom handler. You can load files of event records automatically on a periodic basis by configuring UE Loader, Batch Controller, and a handler.
To set up automatic event loading:
-
Configure UE Loader. See "Configuring UE Loader".
-
Set up a handler to launch UE Loader.
See the documentation of the service integration component you use for instructions on setting up the handler for that component.
-
Set up Batch Controller to do these tasks:
-
Scan a configured directory for record files from an external source.
-
Move the files to the appropriate handler for processing.
See also "Controlling Batch Operations" in BRM System Administrator's Guide.
-
Troubleshooting Event Loading
This section explains UE Loader log files and error codes, and provides tips on troubleshooting event loading failures.
Understanding UE Loader Log Files
UE Loader records errors and successes in the appropriate log files in BRM_Home/apps/uel. Table 22-10 describes the UE Loader log files.
Table 22-10 UE Loader Log Files
| Type of File | Log File Name | Description |
|---|---|---|
|
Cache file |
event_log_file_name_cache.xml |
The cache file is an intermediate file generated by UE Loader after it parses the event log file and before it loads events into the database. UE Loader stores the results of event log parsing in the cache file, which includes:
The record number appears in the error log so you know which record to fix in the cache file if there are errors during event loading. |
|
Filter log |
event_log_file_name_filtered.xml |
The filter log file is generated when events are loaded. It lists records you have specified to archive. Records are archived based on filters you set up in the event import template. See "About Logging Event Records Based on Record Content". Note: The event import template specifies whether filtered records are loaded into the BRM database. |
|
Error log |
event_log_file_name_lerr.xml |
The error log file is generated when events are loaded. It reports any loading errors and specifies the records in the event log file that caused the errors. |
|
Success log |
event_log_file_name_lsucc.xml |
The success log file is generated when events are loaded. It lists all records that load into the database successfully. |
|
Application log |
uel.pinlog |
The standard application log file contains complete descriptions of the errors generated in the other log files. To log all errors, set the log level to 3. |
When errors occur, you typically check the log files in this order:
-
UE Loader pinlog
-
Error file
-
Success file
-
Cache file
Check the cache file for NULL account attributes if an account could not be found during event loading.
-
Filter file
Check the filter file if you have set up filters for event loading to see that events were loaded and logged according to the filter.
Note:
The name that UE Loader generates for its log files includes the extension of the event log file. For example, when the event log file is named CDR3.txt, the cache, filter, error, and success files are called:-
CDR3.txt_cache.xml
-
CDR3.txt_filtered.xml
-
CDR3.txt_lerr.xml
-
CDR3.txt_lsucc.xml.
-
Understanding UE Loader Error Codes
Table 22-11 describes the UE Loader error codes that appear in the uel.pinlog, event_log_file_name_lerr.xml file, and event_log_file_name_cache.xml file:
Table 22-11 UE Loader Error Codes
| Error Code | Brief Description | Error Description | Possible Required Actions |
|---|---|---|---|
|
0 |
NO_ERROR |
All records loaded successfully. |
No action needed. |
|
-1 |
FAILED_HEADER_TRAILER_CHECK |
The header record or trailer record check failed. |
|
|
-2 |
NOT_ENOUGH_MEMORY |
UE Loader ran out of memory. |
|
|
-3 |
OUT_OF_DISK_SPACE |
UE Loader ran out of disk space while creating log files. For example, there was insufficient disk space in the specified directory to store UE Loader success, error, or filter log files. |
Specify a location for log files that contains more disk space. See "Configuring Log File Locations for UE Loader". |
|
-4 |
TEMPLATE_ERROR |
UE Loader detected a problem with the event import template. For example, you used the wrong version of a template (a template that was created with an earlier version of UE Mapper). |
To convert an event import template to the latest version, migrate the template in UE Mapper. See the UE Mapper Help for instructions. |
|
-100 |
BAD_INF_CONNECTION |
The BRM system is not available. For example, the CM is down. |
|
|
1 |
FIELD_PARSE_ERROR |
An error occurred when parsing the fields in the event log file. For example, UE Loader detected no matching string delimiter when parsing the event record data. |
|
|
2 |
ACCT_FORMAT_ERROR |
When UE Loader constructed the input flist to pass to the opcode that finds the account, it detected an error in the data format. Possible causes:
|
|
|
3 |
LOAD_FORMAT_ERROR |
When UE Loader constructed the input flist to pass to the opcode that loads the event data into BRM, it detected an error in the data format. Possible causes are similar to error code 2. |
|
|
4 |
PREV_PARSE_ERROR |
When loading events, a parse error occurred. Note: You do not receive this error if you choose to continue loading events when parse errors occur. To configure this, set the infranet.uel.load_on_parse_errors entry in the UE Loader Infranet.properties file. You can also configure a maximum number of parse errors before UE Loader aborts. |
Correct the record in the event log file. Note: This type of error cannot be corrected in the cache file (for reloading with the -m load option) because the order of events is incorrect. The error must be corrected in the event log file. |
|
5 |
TOO_MANY_LOAD_ERRORS |
The number of load errors exceeded the number of errors you allow. This error is generated when a number of load format errors (code 3) or load opcode errors (code 101) occur. Note: To set the maximum number of errors, use the infranet.uel.max_load_errors entry in the UE Loader Infranet.properties file. |
See the possible actions for error codes 3 and 101. |
|
10 |
COMMAND_LINE_ARG_ERROR |
The command line syntax you used to run UE Loader is incorrect. |
See the "Universal Event Loader" utility page for the correct syntax. |
|
11 |
FILE_NOT_FOUND_ERROR |
The event log file you specified in the UE Loader command line cannot be found. |
|
|
100 |
ACCT_NOT_FOUND |
The opcode that finds accounts associated with event data could not find an account. Possible causes:
|
|
|
101 |
LOAD_OPCODE_ERROR |
The opcode that loads event data into BRM could not load the data. |
|
|
111 |
The template61.dtd file is missing. |
The default UE Loader template file (template61.dtd) is not present in the directory from which the UE Loader utility is run. |
Copy the template file to the directory from which you run UE Loader. The default template, template61.dtd, is in BRM_Home/apps/uel. |
Troubleshooting Tips
Some common causes of failure to load event records into the BRM database:
-
Incorrect connection settings in your UE Loader configuration file.
See the UE Loader properties file (BRM_Home/apps/uel/Infranet.properties) for instructions on setting each entry.
-
Incorrect format of the event data in your event log files.
UE Loader expects the records in your event log file to use the format defined in the event import template.
-
Incorrect setup of the event import template used to import the event data.
UE Loader reads the event import template to obtain instructions for loading the events, so the event import template must specify the correct mapping for the records in your log file.
-
Incorrect time format of the event data.
Specifying the Maximum Load Errors to Process Before Quitting
If you choose to continue loading events when parse errors occur, you can specify the number of errors to allow before UE Loader quits. To set this maximum number of load errors, use the infranet.uel.max_load_errors entry in the UE Loader properties file (BRM_Home/apps/uel/Infranet.properties).
Troubleshooting Connectivity Errors
If you have trouble connecting to the database, ensure that you are using the correct host name, and that the port number, database number, and host name are specified correctly in your UE Loader properties file (BRM_Home/apps/uel/Infranet.properties).
Troubleshooting Preprocessing Errors
UE Loader reports two types of preprocessing errors:
-
Errors found when reading the event log file.
These errors include the record number, line number, and error description. These errors are typically the result of UE Loader finding an unexpected type of data, such as unmatched literal string indicators.
-
Errors in the custom opcode that identifies the account that generated the event.
Some preprocessing errors can be fixed by editing the event import template. Errors in the template are typically fixed by a developer by using the UE Mapper application in Developer Center.
In some cases, an event log file includes a record that has not been generated correctly by the event log file source. In this case, edit the event log file manually to fix the record before loading it.
Troubleshooting Event Loading Errors
Event loading errors include the record number and contents of the record as read from the intermediate cache file. To troubleshoot loading errors, open the uel.pinlog file and read the contents of the error buffer returned by the BRM opcode.
Loading errors are usually the result of the following:
-
Errors in the BRM system, such as an offline Data Manager (DM).
-
Errors in the event import template. Errors in the template are typically fixed by a developer familiar with BRM programming concepts and techniques.
Testing Event Importing
During testing of event importing, note that:
-
UE Loader does not use the time set by the pin_virtual_time utility. If the event start and end times are not obtained from the log file, UE Loader uses the system time.
-
You may want to load a specific event log file more than once.
If you load events from an event log file successfully, UE Loader generates the log files for it, such as the success log file and the error log file. These log files must be deleted before you can reload events from the same event log file.
Reloading Events That Failed to Load
If events fail to load into the BRM database, UE Loader logs the records in an error log file called event_log_file_name_lerr.xml. Errors are also recorded in the event_log_file_name_cache.xml.
-
Find the record number of the event record(s) that failed to load by looking in the event_log_file_name_lerr.xml file.
-
Find the event records that failed to load in the event_log_file_name_cache.xml file by using the record number(s) you found in the error log file.
-
Fix the errors in the event records.
-
Reload the events by running UE Loader with the -m load parameter:
uel -t template_name -m load event_log_file_name
UE Loader reads the files generated in the previous run, and ignores previously loaded events.
For more information on the cache file and error file, see "Understanding UE Loader Log Files".
Improving UE Loader Performance
You can improve UE Loader performance by adjusting the number of UE Loader threads and processes and by modifying entries in the UE Loader Infranet.properties file.
Important:
Optimal performance configuration for UE Loader depends on your system configuration such as the number and speed of CPUs and the size of your hard drive and cache memory. To find the UE Loader settings that work best for your system, you must test different combinations. For example, vary the number of threads and processes and test with different UE Mapper templates, record distribution, and queue sizes.You can improve UE Loader performance by doing one or more of the following:
-
Increase the number of threads running in UE Loader.
Increase the number of threads in the infranet.uei.max_threads entry in the Infranet.properties file. For more information, see "Setting the Number of Threads for Performance".
Important:
If the loading order of records is important, for example, when loading session events, use multiple threads only if you configure UE Mapper to load CDRs grouped by account ID. -
Increase the number of UE Loader processes.
You can run several UE Loader processes at the same time and load a file with each process. Do this by splitting the input file into several files before loading.
Even better performance can be achieved by running several UE Loader processes using multiple threads in each process.
-
Limit the number of records loaded into memory.
Increase the queue size in the infranet.uel.queue_size entry in the Infranet.properties file. The queue_size entry specifies the number of records to load simultaneously. The queue size is shared by multiple threads, which have concurrent access to the queue. For more information, see "Setting Memory Options".
-
Configure UE Loader to load CDRs grouped by account ID.
Select Group events by account identifier when setting up your template in UE Mapper. For more information, start Developer Center and see UE Mapper Help.
-
Load CDRs in the order they occur when using multiple threads.
Select Group events by account identifier in UE Mapper and ensure that all CDRs from the same account are loaded in order (ordered by time).