Use Scheduled Checks to ensure that you are notified if data drift or model quality metrics degrade beyond the threshold for any period. For data drift, Domino uses the timestamps of the predictions to select data for the scheduled checks. For model quality, Domino uses the timestamp of the prediction data to select the prediction and ground truth data on which to run periodic model quality checks. You can specify how often to repeat the checks and the time range of the data to be used for calculations for those checks. When a check fails, emails are sent. See Set up Notifications.
You must have ground truth data available for the date ranges selected for the Schedule Checks. To do this, set up the data source and notify Domino of new ground truth data.
-
From the navigation pane, click Endpoints.
-
Click the name of the model for which you want to set up scheduled checks.
-
Click Grafana Monitoring.
-
Go to Configure Monitoring > Schedule.
-
Type a name for the check.
-
Set up the frequency at which the check must run.
-
In the Select Data to Check area, select one of the following:
Caution-
When you make a selection, remember that you must update the data manually and ensure that there is enough time for the data to be ingested. For example, if you set the check to repeat every day at 11 PM but you load new ground truth data at 10:45 PM, this might not be enough time for the data to be ingested.
-
If you select Use new data since last check, the system assumes new data was ingested. If none was ingested, you will see an empty dashboard for the specified date and time on the Analyze tab.
-
Use new data since last check time
-
For data drift, this checks predictions with timestamps later than the last scheduled check.
-
For model quality, this checks only ground truth labels ingested into the Model Monitor after the last scheduled check ran and matches them with historical predictions made by the model.
-
-
Data since last x <time period>
-
For data drift, this checks predictions whose timestamps are within the last specified interval (for example, the last three days). For model quality, this checks only the ground truth labels ingested within the last specified interval (for example, within the last three days), and matches them with historical predictions made by the model.
-
-
-
Click Save.
Domino automatically creates data in a Domino Dataset named prediction_data for every project. When you reproduce a workspace, this data is included. Predictions are in Parquet format and are updated hourly as the Domino endpoint processes inputs. All the data captured is stored in the file and populated in a workspace so you can diagnose flaws in the model and retrain the model as needed.