9 Troubleshooting Rated Event Loading

Learn how to troubleshoot the Oracle Communications Billing and Revenue Management (BRM) Rated Event Loader (RE Loader).

Topics in this chapter:

See also:

About Troubleshooting Rated Event Loading

There are two distinct error-handling actions that RE Loader takes, depending on when the error occurs:

  • If an error occurs while events are being loaded, the process is canceled and all events loaded in the session are deleted from the BRM database. The SQL loader errors are logged in a file (BRM_home/apps/pin_rel/file_name.bad) and a fatal error is recorded in the RE Loader log file (Processing_directory/rel.pinlog).

  • If an error occurs while RE Loader is updating account balances, bill items, or journals, the loaded events are left in the database and an error is recorded in the RE Loader log file (Processing_directory/rel.pinlog). If RE Loader stops due to errors while updating account balances, bill items, or journals, correct the problem and run RE Loader again.

Some error messages are sent to standard BRM error handling. Check the rel.pinlog log file. See "Checking the RE Loader Log Files for Error Codes".

RE Loader checks for status in two places:

  • The /batch/rel session status object.

    This object stores the status of the last RE Loader process. When you start RE Loader, it checks that status. If you try to reload a file that RE Loader has already successfully updated, the file is rejected because the session status indicates that the update for that file is complete.

  • The REL_SUB_PROCESSES_T table.

    This tables stores information about loading errors that occurred during the preupdating stage. See "Checking for Errors that Occurred during the PreUpdate Process".

Checking the RE Loader Log Files for Error Codes

RE Loader uses the SQL Loader utility, sqlldr, to load events into the BRM database. The sqlldr process creates a new log file for each input file so that log files from a previous process are not overwritten.

The log files and the temporary files created during preprocessing incorporate the name of the input file in their file names, making it easier to debug if an error occurs.

Error codes follow the fully qualified error code (FQEC) scheme, which consists of a major code that represents the component and a minor code that represents the error number. All BRM-defined errors use a minor code from 0 through 99, and all custom errors use minor codes 100 and above.

For information on how to create custom error codes for RE Loader scripts and utilities, see "Creating Custom Error Codes".

Note:

Because modifying a stored procedure can corrupt data and cause maintenance and upgrade problems, custom error codes cannot be created for stored procedures.

The major and minor error codes for each RE Loader component are shown in Table 9-1.

Table 9-1 RE Loader Major and Minor Error Codes

Component Description Major Code BRM Reserved Minor Codes Customer Reserved Minor Codes

All

Universal code for success.

0

N/A

N/A

RE Loader driver

pin_rel script and Java driver code.

1000

0 - 999

N/A

Failure script

Script called when RE Loader attempts to load a data file that previously failed to load into the BRM database.

2000

0 - 99

100 - 255

Transform script

pin_rel_transform_cdr.pl script, which converts discount files into event record format.

3000

0 - 99

100 - 255

Preprocess script

pin_rel_preprocess_cdr.pl script, which preprocesses the data files and creates bulk-loadable (.blk) files.

4000

0 - 99

100 - 255

Load utility

sqlldr utility, which loads data into the BRM database.

5000

0

1 - 999

Preupdate stored procedure

Stored procedure for updating the loaded data before releasing the partition to other RE Loader sessions.

7000

0 - 99

Not available

Update stored procedure

Stored procedure for updating account balances, bill items, and journals.

8000

0 - 99

Not available

Success script

Script that runs automatically when RE Loader successfully loads a data file into the BRM database.

9000

0 - 99

100 - 255

Database consistency check stored procedure

Stored procedure for verifying that the database indexes are correct before loading data into the database.

10000

0 - 99

Not available

Table 9-2 shows the BRM-defined error codes and messages, where value is the value returned in the error message:

Table 9-2 BRM-Defined Error Codes

RE Loader Error Number Error Message

1000

REL encountered an error.

1002

The infranet.rel.dbtype properties value found is not supported: value

Supported values are: value

1003

The infranet.rel.partition_set_number properties value found is not valid: value

Valid values are between value and value.

1004

A table name properties value is missing for the given storable class: value

1005

A duplicate table name properties value was found: value

1006

The load_util properties value is missing for the given storable class: value

1007

A control file properties value is missing for the given storable class: value

1008

The control file name could not be found in the command line.

1009

REL cannot be executed until the Event Extraction Manager is complete.

1010

An unexpected SQL exception has occurred.

1011

An error occurred while attempting to connect to the BRM database.

1012

An error occurred while attempting to connect to the CM.

Please validate the infranet.connection property value and ensure the CM is running.

1013

An error occurred while attempting to perform an opcode call.

1014

An interrupt has occurred and caused an error.

1015

The following file was not found: value

1016

An unexpected I/O error was encountered.

1017

The POID selected from the database sequence exceeds the maximum supported range of 244: value

1018

REL failed to select the partition name from the database.

1019

The poid_db could not be found in the input file.

1020

The poid_db found in the input file does not match the BRM database number for this CM connection.

Found: value

Expected: value

1021

The header record could not be found in the input file.

1022

The storable class was not defined, or was not found in the header record.

1023

The time format found in the header record is not valid: value

1024

The creation process found in the header record is not supported: value

Valid values are: value

1026

An invalid command-line was provided.

1027

The CM and JDBC BRM database connections are not configured to the same database schema.

1028

The REL session has timed out waiting for another REL session to complete.

1029

The file has previously completed successfully so it will not be loaded again: value

1030

The file is currently being processed by another REL session: value

1031

The value key is missing from the properties file.

1032

The value value is missing from the properties file.

1033

The configured number of tables for this storable class does not match the configured tables: value

1034

A number formatting error was encountered in the properties value for: value

1035

The infranet.rel.updater_threads properties value found is not valid: value

Valid values are between value and value.

To have REL auto-choose an appropriate number of threads, use the value: value

1036

An error occurred while attempting to parse a number for: value

1038

Cannot have control file with 'TRUNCATE' option when running REL in parallel loading mode between multiple REL processes.

Table 9-3 shows the BRM-defined failure script error codes.

Table 9-3 Failure Script Error Messages

Failure Script Error Number Error Message

2000

The failure script encountered an error.

The given command-line was: value

2001

The failure script command-line given arguments are not supported.

The given command-line was: value

2002

The failure script command-line given flags value provided is not supported.

The given command-line was: value

2003

The failure script command-line given directory could not be read.

The given command-line was: value

Table 9-4 shows the BRM-defined transform script error codes.

Table 9-4 Transform Script Error Messages

Transform Script Error Number Error Message

3000

The transform script encountered an error.

3001

The transform script command-line given arguments are not supported.

The given command-line was: value

3002

The transform script command-line given input file could not be read.

The given command-line was: value

3003

The transform script command-line given output file could not be created.

3004

The transform script command-line given negative discount carry over value is invalid.

The given command-line was: value

Table 9-5 shows the BRM-defined preprocess script error codes.

Table 9-5 Preprocess Script Error Messages

Preprocess Script Error Number Error Message

4000

The preprocess script encountered an error.

The given command-line was: value

4001

The preprocess script command-line given arguments are not supported.

The given command-line was: value

4002

The preprocess script failed to open a file.

4003

The preprocess script found the input file to be missing a balance record.

The given command-line was: value

4004

The preprocess script found the input file to be missing a detail record.

The given command-line was: value

4005

The preprocess script command-line given tables are not supported.

The given command-line was: value

4006

The preprocess script command-line given increment_by value is not valid.

The given command-line was: value

4007

The preprocess script did not find the expected number of records in the input file.

The given command-line was: value

4008

The preprocess script found the input file to be missing an event record.

The given command line was: value

Used by SE Loader.

4009

The preprocess script did not find the expected size for an event record.

The given command line was: value

Used by SE Loader.

4010

The preprocess script failed to parse fields mapping data for generating the control file.

The given command line was: value

Used by SE Loader.

Table 9-6 shows the BRM-defined load utility error codes.

Table 9-6 Load Utility Error Messages

Load Utility Error Number Error Message

5000

The database load utility encountered an error.

Table 9-7 shows the BRM-defined insert stored procedure error codes.

Table 9-7 Insert Stored Procedure Error Messages

Insert Stored Procedure Error Number Error Message

6000

The insert stored procedure encountered an error.

Table 9-8 shows the BRM-defined preupdate stored procedure error codes.

Table 9-8 Preupdate Stored Procedure Error Messages

Preupdate Stored Procedure Error Number Error Message

7000

The preupdate stored procedure encountered an error.

7001

The preupdate stored procedure encountered an error on a select statement.

7002

The preupdate stored procedure encountered an error on an insert statement.

7003

The preupdate stored procedure encountered an error on an update statement.

7004

The preupdate stored procedure encountered an error on a delete statement.

7008

The preupdate stored procedure encountered a parsing error.

7010

The preupdate stored procedure could not find an item for an account.

7011

The preupdate stored procedure encountered an unexpected error.

Table 9-9 shows the BRM-defined update stored procedure error codes.

Table 9-9 Update Stored Procedure Error Messages

Update Stored Procedure Error Number Error Message

8000

The update stored procedure encountered an error.

8001

The update stored procedure encountered an error on a select statement.

8002

The update stored procedure encountered an error on an insert statement.

8003

The update stored procedure encountered an error on an update statement.

8004

The update stored procedure encountered an error on a delete statement.

8008

The update stored procedure encountered a parsing error.

8009

The update stored procedure found its record is already being processed.

8010

The update stored procedure could not find an item for an account.

8011

The update stored procedure encountered an unexpected error.

8012

The update stored procedure encountered an invalid record count error.

8013

The update stored procedure encountered an error when updating the account balances.

8014

The update stored procedure encountered an error when updating the item balances.

8015

The update stored procedure encountered an error at TREL precommit.

8016

The update stored procedure encountered an error at TREL postcommit.

Table 9-10 shows the BRM-defined success script error codes.

Table 9-10 Success Script Error Messages

Success Script Error Number Error Message

9000

The success script encountered an error.

The given command-line was: value

9001

The success script command-line given arguments are not supported.

The given command-line was: value

9002

The success script command-line given flags value provided is not supported.

The given command-line was: value

9003

The success script command-line given directory could not be read.

The given command-line was: value

Table 9-11 shows the BRM-defined database consistency check error codes.

Table 9-11 Database Consistency Check Error Messages

Database Consistency Check Error Number Error Message

10000

The database consistency check encountered an error.

10005

The database consistency check found an unpartitioned index.

10006

The database consistency check found an incorrectly partitioned index.

10007

The database consistency check found an unusable index.

Checking for Errors that Occurred during the PreUpdate Process

Errors that occur during the preupdate stage of the loading process are stored in the REL_SUB_PROCESSES_T table. To check for values in the table, run SQL*Plus.

Table 9-12 shows the error codes stored in the REL_SUB_PROCESSES_T table:

Table 9-12 Error Codes Stored in the REL_SUB_PROCESSES_T Table

Status Code Status Number Description

ERROR_SELECTING

–20001

An error occurred when selecting data from a table or tables.

ERROR_INSERTING

–20002

An error occurred during the insert process.

ERROR_UPDATING

–20003

An error occurred during the update process.

ERROR_DELETING

–20004

An error occurred during the delete process.

ERROR_UNPARTITIONED_INDEX

–20005

An error occurred because the index is not partitioned.

ERROR_INCORRECT_PART_INDEX

–20006

An error occurred because the index is global partitioned.

ERROR_UNUSABLE_INDEX

–20007

The index partitions are unusable.

ERROR_PARSING

–20008

An error occurred during the data parsing process.

ERROR_ALREADY_BEING_PROCESSED

–20009

An error occurred because the record is being processed by another thread.

ERROR_ITEM_NOT_IN_ACCOUNT

–20010

An error occurred because the item is already billed and pre_updater_flag is not enabled.

ERROR_UNEXPECTED

–20011

An unexpected error occurred in the pre-update procedure.

ERROR_UPDATE_ACCT_BALANCES

–20013

An error occurred while updating account balances.

ERROR_UPDATE_ITEM_BALANCES

–20014

An error occurred while updating item balances.

Fixing Event Loading Errors

To troubleshoot event loading errors, check the RE Loader log file BRM_home/apps/pin_rel/rel.pinlog, where BRM_home is the directory in which you installed BRM components. See "Checking the RE Loader Log Files for Error Codes".

At times, when RE Loader fails, the rel.pinlog file does not list the error. If this occurs, check the status column in the BATCH_T table in the BRM database for the status of the REL process. Table 9-13 lists the status entries (and the corresponding code attributes).

Table 9-13 Status Entries in the BATCH_T Table

Value Decimal Value Hex Value Description
UPDATE_COMPLETE 0 0x0000 The load and update completed successfully. This is a terminal state: further action will not take place without intervention.
LOAD_ERROR 1 0x0001 An error was detected during the SQL Loader phase. This is a terminal state: further action will not take place without intervention.
UPDATE_ERROR 2 0x0002 An error was detected during the updater phase. This is a terminal state: further action will not take place without intervention.
PREUPDATE_ERROR 8 0x0008 An error was detected during the pre-updater phase. This is a terminal state: further action will not take place without intervention.
CREATED 9 0x0009 (ZIP_DB mode only) The row has been created, but not yet processed. The data has been loaded from REF, but not yet processed by the BRM REM.
REL_START 16 0x0010 The REL process has started - initial registration.
PRE_PROCESS 48 0x0030 REL is doing file pre-processing (using the Perl script).
START_LOAD 64 0x0040 REL is setting up SQL Loader control files/threads.
LOADING 80 0x0050 REL is waiting for SQL Loader threads to complete.
LOAD_COMPLETE 96 0x0060 Load has completed successfully (all SQL Loader processes OK).
START_PREUPDATE 1024 0x0400 REL is setting up pre-processing procedure threads.
PREUPDATING 1280 0x0500 REL is waiting for the pre-updater procedures to finish.
PREUPDATE_COMPLETE 1536 0x0600 Pre-updater phase has completed.
START_UPDATE 4096 0x1000 REL is setting up the updater threads.
UPDATING 8192 0x2000 REL is waiting for updater procedures to complete.

The correct troubleshooting effort for an event loading error depends upon the error scenario:

  • RE Loader fails to start:

    The RE Loader log file (rel.pinlog) displays the error code 107.

    The error occurs if REL is not running.

    Start REL using the following command:

    rel<rated event file>
  • Load failure:

    The RE Loader log file (rel.pinlog) displays the error code 5000. The status entry for the REL process in the BATCH_T table in the BRM database displays 1 (see Table 9-13).

    In this error scenario, RE Loader failed either before or during the loading of the events in the event file. The events are deleted from the event tables.

    To troubleshoot this error, reload the events normally by using the same command to process the original event file.

  • RE Loader fails during the loading:

    The RE Loader log file (rel.pinlog) does not display any error. The status entry for the REL process in the BATCH_T table in the BRM database displays 80 (see Table 9-13).

    In this error scenario, RE Loader failed during the loading of the events and RE Loader was unable to update the session status or execute the cleanup process.

    Use the -override option to start a new process to reload the events. For example:

    pin_rel -override event_file_name

    where event_file_name is the event file.

  • Error occurs during the pre-update stored procedure:

    The RE Loader log file (rel.pinlog) displays pre-update stored procedure error codes starting at 7000 and below 8000. The status entry for the REL process in the BATCH_T table in the BRM database displays 8 (see Table 9-13).

    In this error scenario, RE Loader crashed during the execution of the pre-update stored procedure.

    To troubleshoot this error, reload the events normally by using the same command to process the original event file.

  • RE Loader fails during the updating of events:

    The RE Loader log file (rel.pinlog) does not display any error. The status entry for the REL process in the BATCH_T table in the BRM database displays 8192 (see Table 9-13).

    In this error scenario, RE Loader crashed during the updating of the events and RE Loader was unable to update the session status or execute the cleanup process.

    To troubleshoot this error, reload the events normally by using the same command to process the original event file.

  • Error occurs during the update stored procedure:

    The RE Loader log file (rel.pinlog) displays update stored procedure error codes starting at 8000 and below 9000. The status entry for the REL process in the BATCH_T table in the BRM database displays 2.

    In this error scenario, RE Loader successfully loaded the events but failed during the execution of the update stored procedure. The BATCH_REL_SUB_PROCESSES_T table lists the last commit, indicating the point at which the database update failed.

    Reload the events normally. The update starts from this point.

Debugging Mismatches between Data Files and Control Files

RE Loader customizations can sometimes cause data files and control files to become unsynchronized, resulting in SQL Loader failures. To help you debug these situations, use the pin_rel_enum_blk.pl script, which enumerates fields in your bulk-loadable files. You can then manually compare the data file entries to the control file.

To debug mismatches between your data files and control files, enter the following commands:

% cd BRM_home/apps/pin_rel
% pin_rel_enum_blk.pl file_name [Line_num]

where:

  • file_name specifies the name of the bulk-loadable file. For example, test2.blk.

  • Line_num specifies the line number of the bulk-loadable file that you want to enumerate. The default is 1.

Retrieving Data About Events You Load

BRM stores information about events loaded by RE Loader in a /batch/rel object. This object contains the input file name, number of records loaded, and other session information.

Note:

If you use multiple database schemas, the /batch/rel object is created in the schema specified in the RE Loader Infranet.properties file.

Troubleshooting ZIP File Processing

You can configure BRM to process call detail records (CDRs) as ZIP files. The Rated Event Loader Manager (REL Manager) provides tools you can use to find, correct, and reload ZIP files that have errors. See "Rated Event Loader Manager Utility" for more information about how to use this utility.

The general process for handling ZIP files is:

  1. Find the files that have errors using zip_detail.
  2. Write the failed files to the file system using zip_file_write. Each ZIP file will be extracted to individual CDR files in a directory specific to the ZIP file.
  3. If needed, find out what files you have extracted with zip_file_write using zip_file_search. You can also use zip_detail again to see which files have been extracted.
  4. Correct the data as needed.
  5. Load the modified ZIP file back into the BRM database using zip_file_reload.
  6. Depending on the settings you used in zip_file_reload, the data is either reprocessed automatically, or you can reprocess the data using retry_session.

Table 9-14 provides the list of ZIP file error codes.

Table 9-14 ZIP File Status Codes

Code Status Name Description
0 COMPLETED The ZIP file has been successfully loaded into the database.
1 INSERTING The ZIP File is being inserted into the database; the transaction is not complete.
2 READY The ZIP file has been committed to the database and is ready to be processed by RE Manager.
3 PROCESSING The ZIP file is being processed by RE Manager.
4 DATA_ERROR The ZIP file contains data which cannot be loaded into BRM. You must correct the data manually. Consult the RE Manager of RE Formatter logs for more information about the error.
5 PROCESS_ERROR An unknown transient processing error occurred while the ZIP file data was being loaded. You can retry using REL Manager.

Process Example

The following is a simple example of the process described above.

First, you use the zip_detail command to see the status of the ZIP files that have been sent for processing:

RELManager> zip_detail 2022-06-10
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+------------+--------------------+--------------+------------+
|         Poid | Mod Time            | Input Filename                                               | Status | State              |  Records | Zip Status | Zip State          | Zip Size(Mb) | Extracted? |
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+------------+--------------------+--------------+------------+
|    112539709 | 2022-06-10 08:47:18 | BRMCDR_GSM_2022-06-10T15:41:02Z_2022-06-10T15:41:12Z         |      0 | COMPLETE           |        4 |          0 | COMPLETE           |        0.006 | NO         |
|    112531503 | 2022-06-10 08:47:18 | BRMCDR_GSM_2022-06-10T15:40:52Z_2022-06-10T15:41:02Z         |      2 | LOAD_ERROR         |       12 |          0 | COMPLETE           |        0.007 | NO         |
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+------------+--------------------+--------------+------------+
|              |                     |                                                              |        |                    |       16 |            |                    |        0.013 |            |
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+------------+--------------------+--------------+------------+
2 rows processed.

You can see that the ZIP file with the POID 112531503 has a State of LOAD_ERROR.

Next, extract the contents of the ZIP file to the file system using zip_file_write:

RELManager> zip_file_write 112531503
Successfully written ZIP data for POID '112531503'
Wrote 1 of 1 POIDs provided
Files can be found in directory: /home/brmuser/data/unzipped

The directory that contains the directory for the POID is displayed in the result of zip_file_write.

You can see that you have extracted the file using zip_file_search, which only displays extracted files:

RELManager> zip_file_search
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+---------------------+---------------------+--------+
|         Poid | Mod Time            | Input Filename                                               | Status | State              |  Records | Start Time          | End Time            | Is ZIP?|
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+---------------------+---------------------+--------+
|    112531503 | 2022-06-10 08:47:18 | BRMCDR_GSM_2022-06-10T15:40:52Z_2022-06-10T15:41:02Z         |      1 | LOAD_ERROR         |       12 | 2022-06-10 08:41:11 | 2022-06-10 08:41:11 | YES    |
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+---------------------+---------------------+--------+
1 rows processed.

Another way you can see that the file has been extracted is using zip_detail again:

RELManager> zip_detail 2022-06-10
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+------------+--------------------+--------------+------------+
|         Poid | Mod Time            | Input Filename                                               | Status | State              |  Records | Zip Status | Zip State          | Zip Size(Mb) | Extracted? |
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+------------+--------------------+--------------+------------+
|    112539709 | 2022-06-10 08:47:18 | BRMCDR_GSM_2022-06-10T15:41:02Z_2022-06-10T15:41:12Z         |      0 | COMPLETE           |        4 |          0 | COMPLETE           |        0.006 | NO         |
|    112531503 | 2022-06-10 08:47:18 | BRMCDR_GSM_2022-06-10T15:40:52Z_2022-06-10T15:41:02Z         |      1 | LOAD_ERROR         |       12 |          0 | COMPLETE           |        0.007 | YES        |
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+------------+--------------------+--------------+------------+
|              |                     |                                                              |        |                    |       16 |            |                    |        0.013 |            |
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+------------+--------------------+--------------+------------+
2 rows processed.

You can see in the last column that the file for POID 112531503 has been extracted.

Next, go to the directory that contains the uncompressed files and correct whatever data caused the error.

When the data has been corrected, you can reload the data into the BRM database using zip_file_reload:

RELManager> zip_file_reload false 112531503
Successfully updated ZIP data for POID '112531503'
Updated 1 of 1 POIDs provided
Successfully loaded files have been renamed with suffix '.reloaded_<timestamp>' in directory: /scratch/ri-user-1/data/unzipped

Because the example used "false," the ZIP file was reloaded but not reprocessed. To reprocess the file, you run retry_session:

RELManager> retry_session 112531503
Successfully submitted Poid[112531503] File[BRMCDR_GSM_2022-06-10T15:40:52Z_2022-06-10T15:41:02Z] for retry
Submitted 1 sessions for retry (0 failures)

You can check that the file was processed successfully using zip_detail:

RELManager> zip_detail 2022-06-10
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+------------+--------------------+--------------+------------+
|         Poid | Mod Time            | Input Filename                                               | Status | State              |  Records | Zip Status | Zip State          | Zip Size(Mb) | Extracted? |
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+------------+--------------------+--------------+------------+
|    112539709 | 2022-06-10 08:47:18 | BRMCDR_GSM_2022-06-10T15:41:02Z_2022-06-10T15:41:12Z         |      0 | COMPLETE           |        4 |          0 | COMPLETE           |        0.006 | NO         |
|    112531503 | 2022-06-10 09:02:58 | BRMCDR_GSM_2022-06-10T15:40:52Z_2022-06-10T15:41:02Z         |      0 | COMPLETE           |       12 |          5 | PROCESS_ERROR      |        0.007 | NO         |
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+------------+--------------------+--------------+------------+
|              |                     |                                                              |        |                    |       16 |            |                    |        0.013 |            |
+--------------+---------------------+--------------------------------------------------------------+--------+--------------------+----------+------------+--------------------+--------------+------------+
2 rows processed.

The State of POID 112531503 is now shown to be COMPLETE.