8 Get Started with Model Monitoring

Model monitoring allows you to monitor the quality of model predictions over time and helps you with insights on the causes of model quality issues.

The Model Monitor page allows you to create, run, and track model monitors and results. This page lists the model monitors. You can preview the model drift by selecting a single model monitor of interest, as shown in the screenshot below. In this screenshot, the monitor Power Consumption is selected. On the lower pane of the Model Monitors page, the Model Drift for the selected monitor is displayed. The X axis depicts the analysis period, and the Y axis depicts the model drift values. The horizontal red dotted line is the threshold value. The line depicts the drift value for each point in time for the analysis period. Hover your mouse over the line to view the drift values.

Figure 8-1 Model Monitors page

You can perform the following tasks on the Model Monitors page:

• Create: Create a model monitor.
• Edit: Select a model monitor and click Edit to edit the monitor.
• Duplicate: Select a model monitor and click Duplicate to create a copy of the monitor.
• Delete: Select a model monitor and click Delete to delete the model monitor.
• History: Select a model monitor and click History to view the runtime details of the model monitor. If there is a data monitor job associated with the model monitor, the data monitor runtime details are displayed on the lower pane. Click Back to Monitors to go back to the Model Monitoring page.
• Start: Start a model monitor.
• Stop: Stop a model monitor that is running.
• More: Select a model monitor, click More and then click:
  • Enable: Enable a model monitor schedule. By default, the model monitor is enabled. The status is displayed as SCHEDULED.
  • Disable: Disable a model monitor schedule. The status is displayed as DISABLED.
  • Show Managed Monitors: Click this option to view the models created and managed by OML Services REST API.

The Model Monitors page displays the following information about the model monitors:

• Name: This is the name of the model monitor.
• Data Monitor: The check mark in this column indicates that there is a data monitor job associated with the model monitor. Click on the three dots to view:

  Figure 8-2 View Results and Settings of Data Monitor

  
  • Results: Displays the data monitoring results computed by the data monitor job. The data monitor name is auto-generated along with the prefix OML. The data drift and monitored features along with the computed statistics are displayed in view-only mode on a separate page. Click Monitors on the top left corner to return to the Model Monitors page. Click Details on the top right corner to view the settings of the data monitor.

    Figure 8-3 Data Monitor Results

    
  • Settings: Displays the data monitor settings, additional settings, and the monitored features in view-only mode. Click Back on the top right corner of the page to return to the Model Monitors page.

    Figure 8-4 Data Monitor Settings

    
• Baseline Data: This is a table or view that contains baseline data to monitor.
• New Data: This is a table or view with new data to be compared against the baseline data.
• Last Start Date: This is the date on which the model monitor was last started.
• Last Status: This is the latest status of the model monitor. The statuses are SUCCEEDED, FAILED, RUNNING, SCHEDULED.
• Next Run Date: This is the date on which the next run is scheduled.
• Status: This is the current status of the model monitor.
• Schedule: This is the frequency of the monitoring job, that is, how frequently the model monitor is set to run on the New Data.

• Create a Model Monitor
  A model monitor helps you monitor several compatible models, and compute the model drift chart. Compatible models refer to those models that are trained on the same target and mining function. The model drift chart consists of multiple series of data drift points, one for each monitored model.
• View Model Monitor Results
  The Model Monitor Results page displays the monitoring results of each model that are being monitored. By clicking on any model that has run successfully, you can view the detailed analysis of each model such as model drift, model metrics, prediction statistics, feature impact, prediction distribution and predictive versus drift importance for each feature. The predictive impact versus drift importance for each monitored feature is computed only if data monitoring is enabled.
• View Model History
  The History page displays the runtime details of the model monitors.