Initial Measurement Data Header Export
Export of header data from the Initial Measurement Datamaintenance object is supported via "Specialized" export. SeeSpecializedData Exportin theApplication Framework Administrative User Guidefor more information about specialized data export.
The Initial Measurement Datamaintenance object is configured to support data export via the following options:
Option Type
Option Value
Data Export Class
F1SE (Specialized)
Export Batch Control
D1-SIMDC (Specialized IMD Header Extract Current)
Current vs. Historical Extract
There are two types of extracts used with the Initial Measurement Datamaintenance object:
• Daily Current Extractsgather measurement data for measurements within the last 30 days that have been either added or changed. This extract is based on the Last Update Timestampfield on the Initial Measurement Datatable, as well as the Export Through Date/Time field on the Data Export Control record (see below) to include only changes since the last extract.
• Daily Historical Extractsgather data for initial measurementsolder than the last 30 days that have either been added or changed.To avoid having to scan the entire Initial Measurement Data table, this extract is based on Change Data Capture. During initial measurement data processing if an initial measurement has measurements that are older than 30 days, then a Change Data Capture record is created in the General Process table.
Enabling historical change data capture requires configuration of the following algorithm parameters:
◦ Capture Historical Measurement Changes: true
◦ Historical IMD Extract Batch Control:D1-SMDEH (Specialized IMD Header Extract - Historical)
These parameters should be configured as described on the following algorithms:
◦ Update Latest Measurement Date/Time on MC with Consumption Sync (D1-UPD-DTMC)
◦ Update Latest Measurement Date/Time on Scalar MC with Consumption Sync (D1-UDTSCMCRE)
◦ Update Latest Date/Time on MC from a Future Measurement (D1-UPD-DTMCF)
◦ Update Latest Date/Time on Scalar Measuring Component (D1-UPDDTSCMC)
Note: This use of Change Data Capture is an exception to the general rule described in theSpecializedData Exportsection of theApplication Framework Administrative User Guide.
Data Export Control Records
Data export from the Initial Measurement Data maintenance object requires a Data Export Control record. This record designates that the Initial Measurement Data maintenance object should be exported, andtracks the date and time through which data has been exported so far, allowing subsequent export processes to only export incremental changes.
SeeDefiningData Export Control in theApplication Framework Administrative User Guide for more information about creating Data Export Control records.
Batch Controls
Export of data and creation of export files is performed by the following batch controls:
• Specialized IMD Header Data Extract - Current (D1-SIMDC): Used for daily extracts of current (new) initial measurement data. This process extracts data that has changed since the last extract process has run (based on the Export Through Date/Time field on the Data Export Control record). This process updates the Export Through Date/Time field at the start of batch processing to eliminate the possibility of not setting an initial measurement as historical by Change Data Capture during the processing window.
• Specialized IMD Header Extract - Historical (D1-SMDEH): Used for daily extracts of historical initial measurement data. This process runs against Change Data Capture records in the General Process table.
Batch processing is threaded by on Initial Measurement Data ID, and creates one or more files based on the number of threads processed and batch control parameters.
File Creation and Naming
The extract process uses two controls to determine the number of files to be created:
1. Each thread will create at least 1 file (more threads result in smaller files)
2. TheFile Size Limitparameter can be used to specify set the number of initial measurement data rows to include in any given file. This can result in a single thread producing multiple files.
The data in each file is not ordered, each thread will retrieve initial measurement data for a set of Initial Measurement Data IDs, but beyond that there will be no control for how the data is loaded into the file.
File Naming
Since each thread can produce multiple files and the number of files per thread cannot be predicted the file names will also include an indicator for the first file in a thread and the last file in a thread:
• IMD_​EXPORT_​<file prefix batch parameter>_​<batch number>_​<batch thread number>_​<total thread count>_​<file number>_​<last file indicator>_​<process date>.csv.gz
where:
• <file prefix batch parameter> is an optional file prefix defined using theFile Name Prefixbatch control parameter.
• <batch number> is the batch number for the process
• <batch thread number> is the thread number that created the file
• <total thread count> is the total number of threads
• <file number> is the number of the file
• <last file indicator>: identifies the last files created by a thread (E)
• <process date time>is the date and time at which the file was created, in YYYYMMDDhhmmss format (YYYY = year, MM = month, DD = day, hh = hour, mm = minute, ss = second)
Examples: For batch number 135, comprising 10 threads,
• First file of 3rd out of 10 threads: IMD_​EXPORT_​SOME_​PREFIX_​135_​3_​10_​1_​20191104173233.csv.gz
• All files before the last file of 3rd out of 10 threads: IMD_​EXPORT_​SOME_​PREFIX_​135_​3_​10_​2_​20191104173233.csv.gz
• Last file of 3rd out of 10 threads:IMD_​EXPORT_​SOME_​PREFIX_​135_​3_​10_​E_​20191104173233.csv.gz
File Format
The extract process produces comma separated values (CSV) files with the data described in the following table. The field names in the Field column are included in the output file as a header row if the Include Header parameter on the batch controls is set to “Y”.
Note:
Note: all date/times will be converted from OUAF format to the appropriate ISO format in UTCwith the time zone provided (as "Z").
Applicability
Field
Data Type
Sample Content
Description
Interval
Scalar
Subtractive Interval
INIT_​MSRMT_​DATA_​ID
CHAR(12)
52482245116604
Primary Key of theinitial measurement
X
X
X
MEASR_​COMP_​ID
CHAR(12)
000004769213
Primary Key of the measuring component for the initial measurement
X
X
X
IMD_​EXT_​ID
CHAR(12)
external_​file
External ID for the initial measurement. Usually the name of the file that the initial measurement data was received in
X
X
X
BUS_​OBJ_​CD
CHAR(12)
D1-ManualIMDInterval
The business object that governs the initial measurement
X
X
X
BO_​STATUS_​CD
VARCHAR(30)
COMPLETED
The status of the initial measurement
X
X
X
BO_​STATUS_​REASON_​CD
VARCHAR(30)
FORCE COMPLETE
A reason code for the status. Generally populated when there was a VEE exception that was overridden and the initial measurement was completed without fixing the error.
X
X
X
D1_​FROM_​DTTM
DATE/TIME
2019-01-01T00:00:00Z
Start of the time period the initial measurement covers.
For subtractive scalar data this represents the prior measurement that will be used as the start reading when calculating consumption.
For interval data this represents the start of the first interval for theinitial measurement. Note: no measurements will be created for this date/time as MDM represents intervals as the end of the interval. So the first interval of theinitial measurementwill be this date/time plus the interval size (e.g. if the interval size is 15 minutes then the first interval would be 15 minutes after this date/time).
X
X
X
D1_​TO_​DTTM
DATE/TIME
2019-01-02T00:00:00Z
End of the time period the initial measurement covers.
For scalar data this date/time will match the measurement.
For interval data this will match the date/time of the last interval measurement.
X
X
X
CRE_​DTTM
DATE/TIME
2019-01-03T00:00:00Z
Date and time when the initial measurement was initially created.
X
X
X
STATUS_​UPD_​DTTM
DATE/TIME
2019-01-03T00:00:00Z
Date and time when the initial measurement's status was last updated.
X
X
X
LAST_​UPDATE_​DTTM
DATE/TIME
2019-01-03T00:00:00Z
Date and time when the initial measurement was last updated.
Note: updates can happen outside of a status change so this can be more recent than the STATUS_​UPD_​DTTM.
X
X
X
Sample File
The following is a sample export file containing initial measurement header records. Note that this does not include labels or column headers, but the data is in the same order as the column descriptions above in the File Format section above.
55394295586154,553275345711,externalfile1,D1-ManualIMDScalar,ERROR,,2020-01-02T08:00:00Z,2022-01-01T08:00:00Z,2020-08-17T18:47:41Z,2020-08-17T18:47:41Z,2020-08-17T18:48:54Z
55392270285377,553886220134,externalfile2,D1-ManualIMDInterval,ERROR,,2022-01-01T08:00:00Z,2022-01-02T08:00:00Z,2020-08-15T04:18:43Z,2020-08-15T04:25:57Z,2020-08-15T04:25:57Z
55383485376359,553886220134,externalfile3,D1-ManualIMDInterval,ERROR,,2022-01-01T08:00:00Z,2022-01-02T08:00:00Z,2020-08-15T04:47:36Z,2020-08-15T04:55:03Z,2020-08-15T04:55:03Z