About Domino
Domino Data LabKnowledge BaseData Science BlogTraining

Data Drift and Quality Monitoring

Domino monitors data drift and model quality.

Data drift metrics compare training data and prediction data

Data drift

Data drift occurs when production data diverges from the model’s original training data. Data drift can happen for many reasons, including a changing business environment, evolving user behavior and interest, modifications to data from third-party data sources, data quality issues, and even issues in upstream data processing pipelines. Data drift monitoring compares live predictions with the model’s training data, and then sends an alert when live predictions diverge too much from the training data. See Analyze Data Drift.

Model quality

Model quality monitoring compares the model’s predicted values against the actual results (or labels for the predictions) using ground truth data to generate quality metrics. For classification models, Domino reports the following metrics:

  1. Accuracy

  2. Precision

  3. Recall

  4. F1

  5. AUC ROC

  6. Log Loss

  7. Gini (Normalized)

Domino also provides a confusion matrix and classification report for further quality evaluation.

For regression models, Domino reports the following metrics.

  1. Mean Square Error (MSE)

  2. Mean Absolute Error (MAE)

  3. Mean Absolute Percentage Error (MAPE)

  4. R-Squared (R2)

  5. Gini (Normalized)

See Metrics for classification and regression models for details.

See Analyze Model Quality.

Remediation

When data drift or model quality issues are detected, the Cohort Analysis feature identifies underperforming cohorts of data so you can take remedial action. See Cohort Analysis.

Use Domino to recreate the development environment originally used to train the model. Then, you can use this environment to access the prediction data to diagnose issues, update code, and retrain your model with the latest production data.