Model Monitoring and Remediation

Domino’s Model Monitor periodically analyzes your models in production and alerts you when a model’s performance falls outside of the desired range, using the model’s training data, prediction data, and ground truth data. The monitor can read this data from multiple supported data stores.

Models are not monitored by default. Once you have configured model monitoring, there are two ways to access monitoring data:

  • Click Model Monitor in the Domino navigation bar to go to the Model Monitor landing page.

    Here you can see all monitored models, whether they are deployed as Model APIs or in another environment.

  • For Model APIs, click the Monitoring tab on a model’s API page to view the model’s latest metrics.

Domino monitors two facets of your models’ performance:

  • Data drift monitoring compares live predictions with the model’s training data, then sends an alert when live predictions are too divergent from the training data. See Monitoring Data Drift.

  • Model quality monitoring compares live predictions with your ground truth data, then sends an alert when live predictions are too divergent from the ground truth predictions. See Model Quality.

../../_images/model-monitoring-approach.png

When data drift or model quality issues are detected, the Cohort Analysis feature identifies underperforming data cohorts so you can take action. See Cohort Analysis.