domino logo
Latest (5.5)
  • Tech Ecosystem
  • Get Started
  • Domino Cloud
  • Collaborate
  • Projects
  • Work with Data
  • Workspaces
  • Environments
  • Executions
  • Deploy Models and Apps
  • Model Monitoring
  • Organizations
  • Security and Credentials
  • Notifications
  • Download the Audit Log
  • Data Planes
  • Search
  • Domino CLI
  • Troubleshooting
  • Get Help
domino logo
About Domino
Domino Data LabKnowledge BaseData Science BlogTraining
User Guide
>
Model Monitoring
>
Set Up Monitoring for Model APIs
>
Set Scheduled Checks

Set Scheduled Checks

Use Scheduled Checks to ensure that you are notified if data drift or model quality metrics degrade beyond the threshold for any period. For data drift, Domino uses the timestamps of the predictions to select data for the scheduled checks. For model quality, Domino uses the timestamp of the prediction data to select the prediction and ground truth data on which to run periodic model quality checks. You can specify how often to repeat the checks and the time range of the data to be used for calculations for those checks. When a check fails, emails are sent. See Set up Notifications.

Prerequisites

You must have ground truth data available for the date ranges selected for the Schedule Checks. To do this, set up the data source and notify Domino of new ground truth data.

Create a scheduled check

  1. From the navigation pane, click Model APIs.

  2. Click the name of the model for which you want to set up scheduled checks.

  3. Click Monitoring.

  4. Go to Configure Monitoring > Schedule.

    configure schedule

  5. Type a name for the check.

  6. Set up the frequency at which the check must run.

  7. In the Select Data to Check area, select one of the following:

    Caution
    • Use new data since last check time

      • For data drift, this checks predictions with timestamps later than the last scheduled check.

      • For model quality, this checks only ground truth labels ingested into the Model Monitor after the last scheduled check ran and matches them with historical predictions made by the model.

    • Data since last x <time period>

      • For data drift, this checks predictions whose timestamps are within the last specified interval (for example, the last three days). For model quality, this checks only the ground truth labels ingested within the last specified interval (for example, within the last three days), and matches them with historical predictions made by the model.

  8. Click Save.

Domino automatically creates data in a Domino Dataset named prediction_data for every project. When you reproduce a workspace, this data is included. Predictions are in Parquet format and are updated hourly as the Model API processes inputs. All the data captured is stored in the file and populated in a workspace so you can diagnose flaws in the model and retrain the model as needed.

Domino Data LabKnowledge BaseData Science BlogTraining
Copyright © 2022 Domino Data Lab. All rights reserved.