In a production setting, APIs typically push prediction and ground truth data into the Model Monitor.
You can use Scheduled Checks to ensure that you are notified if data drift or model quality metrics degrade beyond the threshold for any period. For data drift, Domino uses the timestamps of the predictions to select data for the scheduled checks. For model quality, Domino uses the ingestion time of the ground truth labels to select data. You can specify how often to repeat the checks and the time range of the data to be used for calculations for those checks. When a check fails, email notifications are sent. See Set up Notifications.
For scheduled checks on data drift and model quality, prediction and ground truth data must be available periodically.
-
From the navigation pane, click Model Monitor.
-
Click the name of the model for which you want to set up scheduled checks.
-
Click Data Drift or Model Quality and then click Scheduled Checks.
-
Type a name for the check.
-
Set up the frequency at which the check must run.
-
In the Select Data to Check area, select one of the following:
Caution-
When you make a selection, remember that you must update the data manually and ensure that there is enough time for the data to be ingested. For example, if you set the check to repeat every day at 11 PM but you load new ground truth data at 10:45 PM, this might not be enough time for the data to be ingested.
-
If you select Use new data since last check, the system assumes new data was ingested. If none was ingested, you will see an empty dashboard for the specified date and time on the Analyze tab.
-
Use new data since last check time
-
For data drift, this checks predictions with timestamps later than the last scheduled check.
-
For model quality, this checks only ground truth labels ingested into the Model Monitor after the last scheduled check ran and matches them with historical predictions made by the model.
-
-
Data since last x <time period>
-
For data drift, this checks predictions whose timestamps are within the last specified interval (for example, the last three days). For model quality, this checks only the ground truth labels ingested within the last specified interval (for example, within the last three days), and matches them with historical predictions made by the model.
-
-
-
Click Save.
In the Model Monitor, you can click Checks History to see historical data about scheduled checks.