Troubleshoot the Model Monitor

This topic provides solutions for common error messages you might see in the model monitoring interface.

Data ingestion taking longer than expected

Data is ingested sequentially. When multiple users and/or models ingest data at the same time, the ingest jobs are queued and executed sequentially. This can cause ingest jobs to take longer than expected, occasionally delaying ingesting a small dataset if a larger dataset is ahead of it in the queue.

Model not running

If your model is still building or starting, wait for its status to change to Running.

If your model is stopped, use the following steps to start the model:

  1. In the Domino endpoint, click Versions.

  2. In the Actions column for the version you want to start, click the three dots to open the menu.

  3. Click Start Version.

  4. Wait for the model status to change to Running.

Monitoring is not enabled

This happens for the following reasons:

  • Your model’s prediction data might not be configured.

  • Your model’s predict() function might not include the domino.log() function needed to enable monitoring. See Set up Prediction Capture for instructions.

  • Your model’s training data might not be configured. Training data is required for monitoring data drift. See Set up Drift Detection.

  • Your model’s ground truth data might not be configured. Ground truth data is required for monitoring model quality. See Set up Model Quality Monitoring.

  • The first scheduled check might not have occurred yet. Check the monitoring schedule to see when the next monitoring data check will occur.

    Note

    Prediction Data is analyzed through 23:59 of the previous day. Data from the current day is not included. Domino reads the timestamps in the dataset, if they are present; if not, then it uses the ingestion timestamp.

Waiting for data

The Model Monitor is waiting for sufficient data to start rendering monitoring results.

  1. From the Domino endpoint, click Grafana Monitoring.

  2. Click Configure monitoring > Target Ranges. Go to Date Filter > Today to view the data that has been analyzed so far.

Model quality monitoring is not enabled

Model quality is based on ground truth data. If you see this message on your model’s monitoring page, then your model’s ground truth data might not be configured. See Set up Model Quality Monitoring.

Drift monitoring is not enabled

If you see this message on your model’s monitoring page, then your model’s training data might not be set up. See Set up Drift Detection.

Selected training set version cannot be used

  • The selected Training Set Version cannot currently be used for monitoring because it doesn’t contain a schema definition.

  • The Training Set version you selected might be empty, you might have selected the wrong model type, or you might need to also add Target to the Categorical feature while creating the Training set. See Set up Drift Detection.

Error in library(“DominoDataCapture”)

Error in library(“DominoDataCapture”): there is no package called ‘DominoDataCapture

Your project’s environment does not include the prediction data capture library. To add it, do one of the following:

Notifications not being sent

See Configure Notifications for Model Monitoring.