Initial Measurement Data Header Export

Export of header data from the Initial Measurement Data maintenance object is supported via "Specialized" export.  See  Specialized Data Export  in the Application Framework Administrative User Guide for more information about specialized data export.

The Initial Measurement Data maintenance object is configured to support data export via the following options:

Option Type

Option Value

Data Export Class

F1SE (Specialized)

Export Batch Control

D1-SIMDC (Specialized IMD Header Extract Current)

Initial vs. Current vs. Historical Extract

There are three types of extracts used with the Initial Measurement Data maintenance object:

  • Initial Extracts gather an initial extract of initial measurement data headers, controlled by providing a start and end date time of the initial measurement data headers that should be extracted. This process should be used to perform the initial extract of the data prior to using the daily current and daily historical extracts.
  • Daily Current Extracts gather measurement data for measurements within the last 30 days that have been either added or changed. This extract is based on the Last Update Timestamp field on the Initial Measurement Data table, as well as the Export Through Date/Time field on the Data Export Control record (see below) to include only changes since the last extract.
  • Daily Historical Extracts gather data for initial measurements older than the last 30 days that have either been added or changed. To avoid having to scan the entire Initial Measurement Data table, this extract is based on Change Data Capture.  During initial measurement data processing if an initial measurement has measurements that are older than 30 days, then a Change Data Capture record is created in the General Process table.
    Enabling historical change data capture requires configuration of the following algorithm parameters:
    • Capture Historical Measurement Changes: true
    • Historical IMD Extract Batch Control: D1-SMDEH (Specialized IMD Header Extract - Historical)
    These parameters should be configured as described on the following algorithms:
    • Update Latest Measurement Date/Time on MC with Consumption Sync (D1-UPD-DTMC)
    • Update Latest Measurement Date/Time on Scalar MC with Consumption Sync (D1-UDTSCMCRE)
    • Update Latest Date/Time on MC from a Future Measurement (D1-UPD-DTMCF)
    • Update Latest Date/Time on Scalar Measuring Component (D1-UPDDTSCMC)
Note: This use of Change Data Capture is an exception to the general rule described in the  Specialized Data Export  section of the Application Framework Administrative User Guide.

Data Export Control Records

Data export from the Initial Measurement Data maintenance object requires a Data Export Control record. This record designates that the Initial Measurement Data maintenance object should be exported, and tracks the date and time through which data has been exported so far, allowing subsequent export processes to only export incremental changes.

See  Defining Data Export Control in the Application Framework Administrative User Guide for more information about creating Data Export Control records.

Batch Controls

Export of data and creation of export files is performed by the following batch controls: 

  • Specialized IMD Header Initial Extract (D1-SIMDI): Used for initial extract of initial measurement data. Unlike the current extract this batch is controlled directly by providing a start and end date time of the IMD headers that should be extracted. This process should be used to perform the initial extract of the IMD header data prior to using the current extract (D1-SIMDC) and the historical extract (D1-SIMDH). For best performance it is recommended to break up the initial extract period into multiple smaller periods and to use multiple threads for each period. The extract will retrieve all IMD headers with a IMD to date/time that is greater than the input start date/time and less than or equal to the input end date/time.
  • Specialized IMD Header Data Extract - Current (D1-SIMDC): Used for daily extracts of current (new) initial measurement data. This process extracts data that has changed since the last extract process has run (based on the Export Through Date/Time field on the Data Export Control record). This process updates the Export Through Date/Time field at the start of batch processing to eliminate the possibility of not setting an initial measurement as historical by Change Data Capture during the processing window.
  • Specialized IMD Header Extract - Historical (D1-SMDEH): Used for daily extracts of historical initial measurement data. This process runs against Change Data Capture records in the General Process table.

Batch processing is threaded by on Initial Measurement Data ID, and creates one or more files based on the number of threads processed and batch control parameters.

File Creation and Naming

The extract process uses two controls to determine the number of files to be created:

  1. Each thread will create at least 1 file (more threads result in smaller files)
  2. The File Size Limit parameter can be used to specify set the number of initial measurement data rows to include in any given file. This can result in a single thread producing multiple files.

The data in each file is not ordered, each thread will retrieve initial measurement data for a set of Initial Measurement Data IDs, but beyond that there will be no control for how the data is loaded into the file.

File Naming

Since each thread can produce multiple files and the number of files per thread cannot be predicted the file names will also include an indicator for the first file in a thread and the last file in a thread.

  • IMD_​EXPORT_​<file prefix batch parameter>_​<batch number>_​<batch thread number>_​<total thread count>_​<file number>_​<last file indicator>_​<process date>.csv.gz

where:

  • <file prefix batch parameter> is an optional file prefix defined using the File Name Prefix batch control parameter
  • <batch number> is the batch number for the process
  • <batch thread number> is the thread number that created the file
  • <total thread count> is the total number of threads
  • <file number> is the number of the file
  • <last file indicator>: identifies the last files created by a thread (E)
  • <process date time>is the date and time at which the file was created, in YYYYMMDDhhmmss format (YYYY = year, MM = month, DD = day, hh = hour, mm = minute, ss = second)

Examples: For batch number 135, comprising 10 threads:

  • First file of 3rd out of 10 threads: IMD_​EXPORT_​SOME_​PREFIX_​135_​3_​10_​1_​20191104173233.csv.gz
  • All files before the last file of 3rd out of 10 threads: IMD_​EXPORT_​SOME_​PREFIX_​135_​3_​10_​2_​20191104173233.csv.gz
  • Last file of 3rd out of 10 threads: IMD_​EXPORT_​SOME_​PREFIX_​135_​3_​10_​E_​20191104173233.csv.gz

File Format

The extract process produces comma separated values (CSV) files with the data described in the following table. The field names in the Field column are included in the output file as a header row if the Include Header parameter on the batch controls is set to “Y”.
Note: All date/times will be converted from OUAF format to the appropriate ISO format in UTC with the time zone provided (as "Z").

Applicability

Field

Data Type

Sample Content

Description

Interval

Scalar

Subtractive Interval

INIT_​MSRMT_​DATA_​ID

CHAR(12)

52482245116604

Primary Key of the initial measurement

X

X

X

MEASR_​COMP_​ID

CHAR(12)

000004769213

Primary Key of the measuring component for the initial measurement

X

X

X

IMD_​EXT_​ID

CHAR(12)

external_​file

External ID for the initial measurement. Usually the name of the file that the initial measurement data was received in.

X

X

X

BUS_​OBJ_​CD

CHAR(12)

D1-ManualIMDInterval

The business object that governs the initial measurement

X

X

X

BO_​STATUS_​CD

VARCHAR(30)

COMPLETED

The status of the initial measurement

X

X

X

BO_​STATUS_​REASON_​CD

VARCHAR(30)

FORCE COMPLETE

A reason code for the status. Generally populated when there was a VEE exception that was overridden and the initial measurement was completed without fixing the error.

X

X

X

D1_​FROM_​DTTM

DATE/TIME

2019-01-01T00:00:00Z

Start of the time period the initial measurement covers.

For subtractive scalar data this represents the prior measurement that will be used as the start reading when calculating consumption.

For interval data this represents the start of the first interval for the initial measurement. Note: no measurements will be created for this date/time as MDM represents intervals as the end of the interval. So the first interval of the initial measurement will be this date/time plus the interval size (e.g. if the interval size is 15 minutes then the first interval would be 15 minutes after this date/time).

X

X

X

D1_​TO_​DTTM

DATE/TIME

2019-01-02T00:00:00Z

End of the time period the initial measurement covers.

For scalar data this date/time will match the measurement.

For interval data this will match the date/time of the last interval measurement.

X

X

X

CRE_​DTTM

DATE/TIME

2019-01-03T00:00:00Z

Date and time when the initial measurement was initially created

X

X

X

STATUS_​UPD_​DTTM

DATE/TIME

2019-01-03T00:00:00Z

Date and time when the initial measurement's status was last updated

X

X

X

LAST_​UPDATE_​DTTM

DATE/TIME

2019-01-03T00:00:00Z

Date and time when the initial measurement was last updated.

Note: updates can happen outside of a status change so this can be more recent than the STATUS_​UPD_​DTTM.

X

X

X

Sample File

The following is a sample export file containing initial measurement header records. Note that this does not include labels or column headers, but the data is in the same order as the column descriptions above in the File Format section above.

55394295586154,553275345711,external file1,D1-ManualIMDScalar,ERROR,,2020-01-02T08:00:00Z,2022-01-01T08:00:00Z,2020-08-17T18:47:41Z,2020-08-17T18:47:41Z,2020-08-17T18:48:54Z
55392270285377,553886220134,external file2,D1-ManualIMDInterval,ERROR,,2022-01-01T08:00:00Z,2022-01-02T08:00:00Z,2020-08-15T04:18:43Z,2020-08-15T04:25:57Z,2020-08-15T04:25:57Z
55383485376359,553886220134,external file3,D1-ManualIMDInterval,ERROR,,2022-01-01T08:00:00Z,2022-01-02T08:00:00Z,2020-08-15T04:47:36Z,2020-08-15T04:55:03Z,2020-08-15T04:55:03Z