Configuring and Running Ongoing Evaluations

You can use ongoing evaluation to collect comparative reporting statistics during normal ML anomaly scoring processing.

Note:

It is not expected that historical and ongoing evaluations would be configured and active for the same time period. Ongoing is intended for tracking live production ML results, whereas historical evaluations are intended for data that has already been processed using VEE.

Configuring Ongoing Evaluation

Configuring ongoing evaluations involves the following:

Create ML VEE Exception Type

Define a VEE Exception Type using the ML Exception with Statistics business object (D1-MLExceptionWithStats). This specifies the VEE exception to be created when the machine learning anomaly score is higher than the Anomaly Score Threshold or when there was an error during the scoring process when used with ongoing evaluation. See MDM Master Configuration in Configuring Machine Learning Anomaly Scoring for more information.

Note:

The ML Exception with Statistics business object (D1-MLExceptionWithStats) lifecycle delays capturing of statistics until after the production usage loading batch window so it can be done in off times during the day. The VEE exception itself is what is capturing the key inputs to the statistics calculations

Update MDM Master Configuration

Update the Machine Learning Configuration section of the MDM Master Configuration as follows:

  • Exception Type: Specify the ML VEE Exception Type created for ML ongoing evaluation (above).
  • Anomaly Score Threshold: The fallback threshold used when a threshold is not defined for the measuring component or measuring component type.

See MDM Master Configuration in Configuring Machine Learning Anomaly Scoring for more information.

Configure Consumption Extract

Configure a Consumption Extraction Type for each ML model to be tested based on the ML Historical Verification Extract Type (D1-MLVerificationExtractType) business object. Extracts created based on this type are used by the VEE Exception to capture the statistic inputs (such as which VEE exceptions to consider).

This should include the following attributes that define the model:

  • Interval Scalar: A flag that indicates whether Interval or Scalar data is to be evaluated. Together with Measurement Frequency and Service Type, this helps define eligible measuring-component types.

  • Measurement Frequency: The frequency at which measurements are captured by the measuring component, represented as hours:minutes:seconds (00:00:00). Together with Interval Scalar and Service Type, this helps define eligible measuring-component types.

    • For interval data: This is the interval size.

    • For scalar data: This should align with the expected hours between measurements defined on the scalar measuring component type (such as 24:00:00 for daily, 08:00:00 for 3 times per day).

  • Service Type: The Service Type for the measurement data being evaluated (such as Electric, Gas, and so on). Together with Interval Scalar and Measurement Frequency, this helps define eligible measuring-component types.

  • Service Point Types: One or more service point types and UOM/TOU/SQI combination eligible for ML verification. These combinations determine which historical measuring components are evaluated.

  • VEE Exception Type Inclusions: One or more VEE exception types that represent consumption quality issues ML is expected to detect. When this list is populated, any matching exception qualifies the IMD as having a VEE detected issue, regardless of severity. When empty, any non-informational VEE exception qualifies. You should exclude conditions that ML cannot evaluate so as to ensure ML to VEE comparisons remain meaningful.

  • UT Exception Type Inclusions: One or more Usage Transaction exception types that indicate user intervention for an IMD with a qualifying VEE exception. The process compares the IMD with Usage Transactions for its Usage Subscription whose periods overlap the IMD period. A qualifying exception is interpreted as evidence of user action to resolve the IMD's data quality issue. When the list is populated, any matching exception counts regardless of severity. When empty, any non-informational Usage Transaction exception counts. Include only exception types that represent data quality issues so that comparisons between ML results and inferred user actions remain meaningful.

See About VEE and Understanding Exception Types for more information.

Configure Statistics Control

Configure a Statistic Control for each ML model to be tested based on the ML Ongoing Verification Control (D1-MLOngoingVerificationCntrl) business object. This is similar to the statistics control created for historical evaluation, except that it uses the VEE exception as the source of the statistical inputs.

These statistic controls are defined by the following:

  • Snapshot Frequency Hours: The frequency (in hours) at
  • Retention Mode: Defines how to retain data for snapshots created from this statistic control
  • Interval Scalar: A flag that indicates whether Interval or Scalar data is to be evaluated. Together with Measurement Frequency and Service Type, this helps define eligible measuring-component types.

  • Measurement Frequency: The size of the intervals measured by the measuring component, represented as hours:minutes:seconds (00:00:00). Together with Interval Scalar and Service Type, this helps define eligible measuring-component types.

    • For interval data: This is the interval size.

    • For scalar data: This should align with the expected hours between measurements defined on the scalar measuring component type (such as 24:00:00 for daily, 08:00:00 for 3 times per day).\

  • Service Type: The Service Type for the measurement data being evaluated (such as Electric, Gas, and so on). Together with Interval Scalar and Measurement Frequency, this helps define eligible measuring-component types.

  • Snapshot Business Object: The business object to use when creating snapshots based on this statistic control. This should be ML Verification Statistics (D1-MLVerificationStatistics)

Configure SGG ML Payload Processing

Configure ML payload processing as described in Configure Payload Processing Extendable Lookups in Configuring Machine Learning Anomaly Scoring to process results returned from the ML model.

Running Ongoing Evaluation

Running ongoing evaluations involves the following:

Batch Processing

Schedule the following batch processes to run on a regular basis:

  • Payload Processing Monitor Template with Export (D1-PLPSO) to process incoming measurement data and sends it to ML for scoring.

  • SGG ML Payload Processing Monitor Template (D1-PLMLP) to process the returned ML results

  • Process IMD Control Stage Data (D1-IMDCS) to process records in the IMD Control table, including:

    • Running appropriate VEE processing based on the ML result for each IMD with measuring component ordering.

    • The results are either IMDs created in error, a header IMDs created for exceptions, or published measurements

    • This is when the VEE exception will be created for ML

  • VEE Exception Monitor (D2-VEEDF) to transition all pending VEE exceptions and create the appropriate statistic characteristics against the IMD associated to the VEE exception

  • Statistics Control Monitor (F1-STATS) to create snapshots that are available for review on the Meter ML and VEE Dashboard.

Parent topic: Machine Learning Comparative Reporting