Configuring and Running Historical Evaluations

You can use historical evaluations to score existing initial measurement data (IMDs) originally processed using VEE using ML, compare the ML results with original VEE results, and determine an appropriate anomaly score threshold.

This process use batch processes to create and extract historical "verification" data that can be evaluated by ML and then compared to the original VEE data, Another batch process captures the results in a statistics control record, which can be viewed using the Meter ML and VEE Dashboard portal.

Note:

Historical verification data is used only for coordinating the scoring by the ML model and evaluation. This data does NOT create or update any measurement data.

Note:

Historical evaluation should be used in non-production environments (such as DEV or TEST) ONLY.

Configuring Historical Evaluations

Configuring historical evaluation involves the following:

Create ML VEE Exception Type

Define a VEE Exception Type that specifies the VEE exception to be created when the machine learning anomaly score is higher than the Anomaly Score Threshold or when there was an error during the scoring process when used with historical evaluation. This may be a different VEE exception Type than used for ongoing ML processing. See MDM Master Configuration in Configuring Machine Learning Anomaly Scoring for more information.

Update MDM Master Configuration

Update the Machine Learning Configuration section of the MDM Master Configuration as follows:

  • Exception Type: Specify the ML VEE Exception Type created for ML historical evaluation (above).
  • Anomaly Score Threshold: 0 (this ensures that the system retains the ML prediction and anomaly score for all IMDs selected for the evaluation)

See MDM Master Configuration in Configuring Machine Learning Anomaly Scoring for more information.

Configure Consumption Extract

Configure a Consumption Extraction Type for each ML model to be tested based on the ML Historical Verification Extract Type (D1-MLVerificationExtractType) business object. This construction extract type is used to define what is going to be scored as well as how VEE results should be interpreted as compared to ML results.

This should include the following attributes that define the model:

  • Interval Scalar: A flag that indicates whether Interval or Scalar data is to be evaluated. Together with Measurement Frequency and Service Type, this helps define eligible measuring-component types.

  • Measurement Frequency: The frequency at which measurements are captured by the measuring component, represented as hours:minutes:seconds (00:00:00). Together with Interval Scalar and Service Type, this helps define eligible measuring-component types.

    • For interval data: This is the interval size.

    • For scalar data: This should align with the expected hours between measurements defined on the scalar measuring component type (such as 24:00:00 for daily, 08:00:00 for 3 times per day).

  • Service Type: The Service Type for the measurement data being evaluated (such as Electric, Gas, and so on). Together with Interval Scalar and Measurement Frequency, this helps define eligible measuring component types.

  • Service Point Types: One or more service point types and UOM/TOU/SQI combination eligible for ML verification. These combinations determine which historical measuring components are evaluated.

  • VEE Exception Type Inclusions: One or more VEE exception types that represent consumption quality issues ML is expected to detect. When this list is populated, any matching exception qualifies the IMD as having a VEE detected issue, regardless of severity. When empty, any non-informational VEE exception qualifies. You should exclude conditions that ML cannot evaluate so as to ensure ML to VEE comparisons remain meaningful.

  • UT Exception Type Inclusions: One or more Usage Transaction exception types that indicate user intervention for an IMD with a qualifying VEE exception. The process compares the IMD with Usage Transactions for its Usage Subscription whose periods overlap the IMD period. A qualifying exception is interpreted as evidence of user action to resolve the IMD's data quality issue. When the list is populated, any matching exception counts regardless of severity. When empty, any non-informational Usage Transaction exception counts. Include only exception types that represent data quality issues so that comparisons between ML results and inferred user actions remain meaningful.

See About VEE and Understanding Exception Types for more information.

Configure Statistics Control

Configure a Statistic Control for each ML model to be tested based on the ML Historical Verification Control (D1-MLVerificationControl) business object. Snapshots created based on this statistics control compare the results from ML scoring with the previous VEE results, identifying when they agreed/disagreed. The process attempts to include user actions performed on VEE exceptions to determine if VEE exceptions were true issues or false positives.

These statistic controls are defined by the following:

  • Retention Mode: Defines how to retain data for snapshots created from this statistic control
  • Snapshot Start Date: The date on which snapshots created from this statistic control start.

    • Interval Scalar: A flag that indicates whether Interval or Scalar data is to be evaluated. Together with Measurement Frequency and Service Type, this helps define eligible measuring-component types.

    • Measurement Frequency: The size of the intervals measured by the measuring component, represented as hours:minutes:seconds (00:00:00). Together with Interval Scalar and Service Type, this helps define eligible measuring-component types.

      • For interval data: This is the interval size.

      • For scalar data: This should align with the expected hours between measurements defined on the scalar measuring component type (such as 24:00:00 for daily, 08:00:00 for 3 times per day).

    • Service Type: The Service Type for the measurement data being evaluated (such as Electric, Gas, and so on). Together with Interval Scalar and Measurement Frequency, this helps define eligible measuring-component types.

  • ML Threshold Override: Use to override the Anomaly Score Threshold on the Master Configuration if applicable. This is not used with this historical evaluations.
  • Snapshot Business Object: The business object to use when creating snapshots based on this statistic control. This should be ML Verification Statistics (D1-MLVerificationStatistics)

Configure SGG ML Payload Processing

Configure ML payload processing as described in Configure Payload Processing Extendable Lookups in Configuring Machine Learning Anomaly Scoring to process results returned from the ML model.

Running Historical Evaluations

Running historical evaluations involves the following:

Create Historical Evaluation IMDs

Run the following batch processes in sequence to create "historical validation" initial measurement data that can be used for evaluation by running the following batch processes in sequence:

  • Generate ML Historical Verification IMDs (D1-GMLVI) to select and create IMDs for evaluation. The IMDs are placed in the Waiting for ML state.

  • Extract ML Historical Validation IMDs (D1-EXHVI) to export the selected IMDs for ML scoring.

Historical Verification IMDs:

  • Are created only from "initial load" initial measurements,
  • Duplicate the preVEE data which is sent to ML,
  • Duplicate the VEE exceptions for statitsics computation
  • Link to the original IMD for traceability

ML Processing

The machine learning model scores the data and sends the results to an object storage location (similar to the process used with Machine Learning Anomaly Scoring

Run ML Payload Processing and Analysis

Once the ML scoring results have been returned to object storage, run the following processes in sequence to process the payloads and capture information about the evaluation:

  • SGG ML Payload Processing Monitor Template (D1-PLMLP) to process the returned ML results.

  • IMD Monitor - Waiting for ML IMDs (D1-IMDML) to capture evaluation information and move the IMDs to the Analysis in Progress state.

Capture Statistics

Open the Historical Statistics Control, and click Capture. When the dialog opens, enter the Start Date and the Number of Days to process and click OK. This creates a historical ML Verification Statistics Snapshot for each requested day using the statistics control's ML Threshold Override or the Anomaly Score Threshold defined on the MDM Master Configuration.

Process ML Historical Statistics Snapshots

Run the ML Verification Statistic Snapshot Monitor (D1MLVSSM) batch process to calculate daily ML Verification Statistics Snapshots. The process selects ML Historical Validation IMDs based on service type, interval/scalar mode, and frequency, then stores ML, VEE, user comparisons and anomaly-score distributions for dashboard analysis.

Re-evaluating Historical Results

You can make changes to the Consumption Extract Type or Statistics Control to fine tine the results and identify what works best for your implementation.

If you change the VEE Exception Type or Usage Transaction Exception Type configuration in the Consumption Extract Type:

  • Run the Transition ML Historical Validation IMDs (D1-TMHVI) batch with the Transition Condition parameter set to "D1RC"
  • Run the ML Verification Statistic Snapshot Monitor (D1MLVSSM) batch process again and review the updated results.

If you only change the ML Threshold Override on the Statistic Control:

  • Perform the Capture action again
  • Run the ML Verification Statistic Snapshot Monitor (D1MLVSSM) batch process again and review the updated results.

This process can be run in stages:

  • Stage1: ML scoring, run once
  • Stage 2: Capture statistical inputs, which involves running D1-TMHVI), generating new snapshots, and running D1MLVSSM to calculate statistics
  • Stage 3: Anomaly score evaluation, which involved creating snapshots and running statistics collection to see how the statistics change and to identify the right threshold

Removing Historical Evaluation Data

After completing a historical evaluation, run Transition ML Historical Validation IMDs (D1-TMHVI) batch processing with the Transition Condition parameter set to "D1CN" to complete the IMDs and remove their IMD control records.

To remove historical evaluation IMDs that were created in error, run the Remove ML Historical Validation IMDs (D1-RMHVI). batch process.

Note:

Removing historical evaluation data is not mandatory, but if historical verification is being run on large amounts of data over a long period of time removing periods where research is complete help ensures that performance of the various processes does not degrade.

Parent topic: Machine Learning Comparative Reporting