About Periodic Estimation

Periodic estimation is a process that scans eligible devices and their measuring components for missing measurements that have not been received within a required amount of time. Once missing measurements are identified it then generates estimation initial measurements to fill in those missing measurements. It is important to note that the periodic estimation itself does not actually perform any estimations. It relies on the estimation initial measurement to generate the estimate through the estimation VEE rules that are executed.

The periodic estimation process is as follows:

Determine if measuring component is eligible for estimation.
Scan the eligible period for missing measurements.
Confirm missing measurements do not fall within a period where the device was not installed or was disconnected.
- For scalar measuring components, initial measurements are NOT created for disconnected periods.
- For interval measuring components, initial measurements will be created for disconnected periods, but the intervals will be marked as 'No Read - Disconnected' (Measurement Condition code 102000) and will be ignored for estimation by VEE processing.
Confirm there are no in progress initial measurements that cover the period of the missing measurements.
Either generate an initial measurement or a To Do for each measurement gap that is found.

This process relies on an important date that is maintained on each measuring component:

Latest Contiguous Measurement Date/Time: this represents the last time the measuring component had a complete history of measurements without any gaps. By nature it implies that the is a gap immediately after it.

This date allows the periodic estimation process to efficiently identify those measuring components that are missing measurements and need to be estimated.

Note: It is possible to disable periodic estimation for a given device by updating the periodic estimation eligibility on the service point it is associated to. This is useful if there are one-off service points that should either never be eligible for estimation or should temporarily have estimation turned off.

Periodic Estimation Feature Configuration

The estimation begin date for interval and scalar periodic estimation can be influenced by a feature configuration option under the feature type Measurement Data Options (D1MD).

When a No of Hours in Past to Retrieve Last Useable Measurement option is defined it will have two impacts on periodic estimation, both of which are only applicable when the measuring component’s type does not have a value configured for the Maximum Days to Estimate.

When periodic estimation is run for a measuring component that does not yet have a Last Contiguous Measurement Date Time then this option will be used to compute the beginning of the search range to find a useable measurement. That date is then used to begin estimation from.
When running for a measuring component where the Last Contiguous Measurement Date Time is already set then this option will be used to compute the oldest date that estimation can begin from.

For both scenarios the begin date time calculation will be performed by subtracting the value configured from the current Process Date Time. For example, if this option were set to 744 hours (31 days) and the process date time were 02/01 12:00:00AM then the begin date time would be calculated as 01/01 12:00:00AM.

When implemented this option will also instruct the Update Latest Date/Time on Scalar Measuring Component algorithm type (D1-UPDDTSCMC), which is responsible for maintaining the latest measurement date time information for scalar measuring components, to skip initializing the latest measurement date/time information when an initial measurement is received for a measuring component where there is existing measurement data history and this feature configuration option is configured. For those scenarios the initialization of the latest measurement date/time information should be deferred to the Auto-Read Scalar Period Estimation process. Note: this only applies for situations when the latest measurement date/time information has not been set previously and the initial measurement is not the first measurement received for the measuring component.

Note: This does not apply to the analogous logic for an interval initial measurement.

Interval Periodic Estimation

When missing measurements have been identified by the interval periodic estimation functionality an estimation initial measurement will be created for each contiguous set of missing measurements in the estimation period.

There are two methodologies available for estimating interval measuring components:

This diagram illustrates the methodologies for estimating interval measuring components.

Rolling Delay: In this method the execution time of the periodic estimation process is an important factor in which measuring components will estimate how far each measuring component estimate. This method supports a minimum number of hours that will be estimated (as defined by the "Hours to Estimate" on the measuring component type).
Predictable Cut-off Time: In this method all measuring components for a measuring component type will estimate until a specified point in time (as defined by the "Cut-off Time" on the measuring component type). With this method the cut-off time determines how far estimations are made rather than the execution time of the periodic estimation process. There is no minimum estimation size for this method it will always estimate up until the cut-off date/time even if that is only a single interval.

The cut-off time takes into consideration daylight savings time (DST) when the device is configured as sending data in legal time (meaning the Incoming Data Shift has been set to Always in Local). In these circumstances the cut-off time will be considered to be adjusted for DST. This is important because measurement data is stored in standard. For example:


Cut-off Time: 12:00AM	Measurement Time in Standard
01/15/2010 12:00AM (DST not in effect)	01/15/2010 12:00AM
04/15/2010 12:00AM (DST in effect)	04/14/2010 11:00PM

Furthermore, the cut-off time defined on the measuring component type does not have a time zone associated to it and as such it is considered to be defined in the time zone of the particular measuring component being processed. For example, a cut-off time of 12:00AM would be considered to be 12:00AM UTC -08:00 and 12:00AM UTC -05:00 based on the time zones of the respective measuring components. This allows a single measuring component to apply to many time zones and behave uniformly.

Algorithm Type – Create Interval IMD and To Do Based Upon Install History (D1-CIITBIH)

Algorithms based on this algorithm type determine if a measuring component is missing final measurements during the estimation horizon for any period where the associated device configuration was installed. If final measurements are missing it will either create an estimate initial measurement, create a To Do Entry, or both.

There are a few scenarios that will result in this algorithm being used:

If the measuring component’s service point has an Estimation Eligibility of Not Eligible
If the measuring component’s service point has open Estimation Suppression activities that overlap the System Date/Time and the MDM Master Configuration field Fill Missing Data with Zero is not set to “Yes”
If the measuring component’s Latest Contiguous Measurement Date Time is greater than the process date time less the measuring component type’s Number of Hours to Wait

The estimation horizon begin date is determined by first determining the initial horizon start date time:

If the Measuring Component Type’s Maximum Days to Estimate is configured it will be the more recent of the following two dates:
- The process date time less the Maximum Days to Estimate
- The Measuring Component’s Latest Contiguous Measurement Date Time
Otherwise, if the Measurement Data Option feature configuration type is configured with a value set for the No of Hours in Past to Retrieve Last Useable Measurement option then the start will be calculated as follows:
- If the Latest Contiguous Measurement Date Time is populated then take the more recent of:
  - The process date time less the No of Hours in Past to Retrieve Last Useable Measurement
  - The Measuring Component’s Latest Contiguous Measurement Date Time
- Otherwise, if the Latest Contiguous Measurement Date Time is not populated then:
  - Search for a measurement between the following two dates:
    - Start: the process date time less the No of Hours in Past to Retrieve Last Useable Measurement
    - End: the process date time
  - If a measurement is found that date is used. Otherwise the process date time less the No of Hours in Past to Retrieve Last useable Measurement is used.
Otherwise, the Measuring Component’s Latest Contiguous Measurement Date Time is used.
Otherwise, if the device is installed it will be set to the Installation Date Time of the Installation Event (IE).

The estimation horizon end date will be calculated based on the measuring component type’s Periodic Estimation Execution Method.

Rolling Delay:

The more recent of the following dates will be used:

The Latest Measurement Date Time plus the Measuring Component Type (MC Type) - Number of Hours to Estimate defined.
The Process Date Time less the MC Type - Number of Hours to Wait
Estimations are limited by either a device configuration change or removal. Therefore, the date determined above will be limited to the earlier of the device configuration Effective End Date Time or the Removal Date Time on Install Event.

Predictable Cut-off:

In predictable cut-off the intent is to only estimate up to the last cut-off time that has passed based on the Number of Hours to Wait from the measuring component type and no further.

First the process date time will be adjusted based on the Number of Hours to Wait configured on the measuring component type. For example, if the process date/time is 01/15/2010 06:00PM and the Number of Hours to Wait is 48 then the adjusted process date/time is: 01/13/2010 06:00PM.

Next, the last cut-off time to have passed will be identified and that will become the estimation end date time. In our above example this would be 01/13/2010 12:00AM.

Once the estimation horizon begin and end date times have been identified they may be further split into smaller periods based upon the Measuring Component’s device configuration installation history.

For Example:

Estimation horizon is April 10th at 12AM through April 20th at 12AM
The Measuring Component’s device configuration was installed at one service point on April 1st at 12AM and removed on April 15th at 12AM
The Measuring Component’s device configuration was then installed at a new service point on April 18th at 12AM
Given the above information the estimation horizon would be broken up into the following two periods:
- April 10th 12AM through April 15th 12AM (this is for the first installation)
- April 18th 12AM through April 20th 12AM (this is for the second installation)

Each of the estimation horizon periods determined based upon the installation history will be further examined to determine contiguous periods where final measurements are missing.

For Example:

Processing April 10th through April 15th
One final measurement exists for April 12th for 8AM through 9AM
Yields the following sub-periods:
- April 10th 12AM through April 12th 8AM
- April 12th 9AM through April 15th 12AM

-For every missing period found,
- Find Initial Measurements that are in a non Final state and the initial measurement’s period has an overlap with the missing period.
  - For each initial measurement found,
    - If the initial measurement's period falls exactly within the boundaries of the missing period, skip the creation of an estimate initial measurement for the missing period.
    - If the initial measurement's end date/time is before the start of the missing period, the initial measurement’s end date/time is now the start of the missing period. Proceed to create an estimate initial measurement for the adjusted missing period.
    - If the initial measurement's start date/time is after the start of the missing period, the initial measurement’s start date/time is now the end of the missing period. Proceed to create an estimate initial measurement for the adjusted missing period.

Each time Periodic Estimation is executed the Latest Contiguous Measurement will be validated and potentially updated. For example, if it is before the install event’sInstallation Date Time, it will be updated to be the install event’s Installation Date Time.

Note: If the Latest Contiguous Measurement Date Time falls within the installation window of a prior installation it will not be adjusted to the current install event’s Installation Date Time.

Scalar Periodic Estimation

Scalar periodic estimation is only available for automatically read scalar measuring components since it requires that scalar measuring components be read on at least a daily basis to be eligible.

The scalar estimation process relies heaving on details from the measuring component type to identify when a measurement is missing. This is because scalar measurements are not required to fall on specific boundaries like interval data. For example, a scalar measuring component read on a daily basis around midnight could have measurements at 12:00AM, 11:55PM, 12:01PM or even at various other points during the day if a field reading or on demand read is performed.

Auto-read scalar measuring component types are configured to provide information that will:

Identify at what point or points during the day the measuring component will be read. This is a combination of the Hours Between Measurements and the First Daily Measurement Time on the measuring component type.
Identify the allowable tolerance for measurements that do not fall on the specific expected measurement date/times (e.g. 11:55PM and 12:05AM are OK but 9AM is not). This is defined by the Early and Late Measurement Threshold on the measuring component type.

This allows the scalar periodic estimation process to identify the points during the day that measurements are expected and if no measurement are found it will generate an estimation initial measurement.

Unlike interval periodic estimation each missing measurement will be created as a separate estimation initial measurement.

The Early Measurement Threshold and Late Measurement Threshold fields support situations where a head end system is not be able to report their readings at a consistent time each day. For example a head end system might send reads as follows:

3/1 11:55PM
3/3 12:01AM
3/4 12:00AM
3/5 12:02AM
3/6 11:59 PM

These fields create a window prior to and after an expected reading where anything received within that window would satisfy the expected reading. For example:

For a measuring component with an Early Measurement Threshold of 00:10:00 (10 minutes), a reading for 3/1 11:55PM would satisfy the early threshold for the 3/2 12:00AM expected reading.
For a measuring component with an Late Measurement Threshold of 00:10:00 (10 minutes), a reading for 3/3 12:01AM would satisfy the late threshold for the 3/3 12:00AM expected reading.

To determine if a measuring component is eligible for estimation the process will evaluate if the Last Contiguous Measurement Date/Time far enough in the past. Specifically the number of hours elapsed since the Last Contiguous Measurement Date/Time must be greater than or equal to the Hours Before Estimation from the measuring component type.

Note: The first daily measurement time takes into consideration daylight savings time (DST) when the device is configured as sending data in legal time (meaning the Incoming Data Shift has been set to Always in Local). In these circumstances the first daily measurement time will be considered to be adjusted for DST. This is important because measurement data is stored in standard. For example:


First Daily Measurement Time: 12:00AM	Measurement Time in Standard
01/15/2010 12:00AM (DST not in effect)	01/15/2010 12:00AM
04/15/2010 12:00AM (DST in effect)	04/14/2010 11:00PM

Furthermore, the first daily measurement time defined on the measuring component type does not have a time zone associated to it and as such it is considered to be defined in the time zone of the particular measuring component being processed. For example, a first daily measurement time of 12:00AM would be considered to be 12:00AM UTC -08:00 and 12:00AM UTC -05:00 based on the time zones of the respective measuring components. This allows a single measuring component to apply to many time zones and behave uniformly.

Algorithm Type – Auto-Read Scalar Periodic Estimation (D1-ARSPE)

Algorithm based on this algorithm type determine if a measuring component is missing final measurements during the estimation horizon for any period where the associated device configuration was installed. If final measurements are missing it will either create an estimate initial measurement, create a To Do Entry, or both.

There are a few scenarios that will result in this algorithm being used:

If the measuring component’s service point has an Estimation Eligibility of Not Eligible
If the measuring component’s service point has open Estimation Suppression activities that overlap the System Date/Time and the MDM Master Configuration field Fill Missing Data with Zero is not set to “Yes”
If the measuring component’s Latest Contiguous Measurement Date Time is greater than the process date time less the measuring component type’s Number of Hours to Wait

The estimation horizon begin date is determined by first determining the initial horizon start date time:

If the Measuring Component Type’s Maximum Days to Estimate is configured it will be the more recent of the following two dates:
- The process date time less the Maximum Days to Estimate
- The Measuring Component’s Latest Contiguous Measurement Date Time
Otherwise, if the Measurement Data Option feature configuration type is configured with a value set for the No of Hours in Past to Retrieve Last Useable Measurement option then the start will be calculated as follows:
- If the Latest Contiguous Measurement Date Time is populated then take the more recent of:
  - The process date time less the No of Hours in Past to Retrieve Last Useable Measurement
  - The Measuring Component’s Latest Contiguous Measurement Date Time
- Otherwise, if the Latest Contiguous Measurement Date Time is not populated then:
  - Search for a measurement between the following two dates:
    - Start: the process date time less the No of Hours in Past to Retrieve Last useable Measurement
    - End: the process date time
  - If a measurement is found that date is used. Otherwise the process date time less the No of Hours in Past to Retrieve Last Useable Measurement is used.
Otherwise, the Measuring Component’s Latest Contiguous Measurement Date Time is used.
Otherwise, if the device is installed it will be set to the Installation Date Time of the Installation Event (IE).

The estimation horizon end date will be whichever of the following dates that is furthest in the past:

Process Date/Time - Hours Before Estimation + Number of Hours to Estimate
The device configuration’s effective end date/time
The Off date/time of the device configuration's installation event.

Second, the algorithm identifies any expected measurement date/time without a final measurement or an in progress initial measurement. For each missing measurement an Estimate initial measurement will be created. Estimations will only be made for periods where the device is installed at a service point. If the Evaluate Installation Event Off Period parameter has been set to 'N' or left blank then the device must also be turned on.

To determine if there are any missing final measurements during the estimation horizon the algorithm uses the concept of expected measurement date/times as explained above. If there are any expected measurements not covered by a final measurements as dictated by the early and late thresholds then an initial measurement will be generated.