Domino’s monitoring feature uses the training data to calculate the probability distributions of all features and prediction columns to detect data drift for input features and output predictions of your model. It approximates these columns by creating bins, and then counting the frequency for each bin. This acts as the reference pattern.
The monitor feature uses prediction data to calculate the probability distribution, using the same bins, and then applies the specified statistical divergence or distance test to quantify the dissimilarity between the training and prediction distributions for each column.
By default, drift checks are scheduled to occur daily at 12 PM UTC.
Out-of-the-box, the Model Monitor supports several statistical tests. Each feature can have a different test. When a new model is registered, it inherits the global default test settings. However, you can change the test types and thresholds to values suitable for each model. If you save these changes, they will be used for subsequent automated checks, such as when new predication data is uploaded or when Scheduled Checks run for the model.
If you are using the monitoring feature for a Model API, go to Configure monitoring > Target Ranges to access the Configure Tests and Thresholds page.
If you are using the Model Monitor, in the navigation pane, click Model Monitor. Select the model that you want to monitor and then click Data Drift.
A - Use the Date Filter to refine the date range and review the Drift and its trends for specific periods.
B - To reduce alert noise, click the bell icon next to features to disable alerts to exclude them from the Scheduled Checks.
C - Select the Test Type, the Acceptable Range, and the Threshold. See Data drift statistical tests for more information about test types. Click Recalculate to see the results.
D - If you made changes to the tests and want to set them as the defaults for the model in subsequent automated checks, go to Other Actions > Save Checks Config to save the changes.
E - If you experimented with changes to the tests and wanted to load the model’s default configuration to reset the table, go to Other Actions > Restore Checks Config.
If you are using the monitoring feature for a Model API, click the X in the top right corner to close the page when you are finished.
Out-of-the-box, the Model Monitor supports several statistical tests.
- Kullback–Leibler Divergence
Kullback–Leibler divergence (also called relative entropy) is a measure of how one probability distribution is different from a second, reference probability distribution. Kullback–Leibler divergence of 0 indicates that the two distributions in question are identical. It is a robust test that works for different distributions and therefore is most commonly used to detect drift.
- Chi-square Statistic
Chi-square test in another popular divergence test well-suited for categorical data.
The chi square statistic is a statistical hypothesis testing technique to test how two distributions of categorical variables are related to each other. Specifically, the chi-square statistic is a single number that quantifies the difference between the observed counts versus the counts that are expected if there was no relationship between the variables at all. A low value of chi-square means there is a high correlation between the two sets of data being compared.
- Kolmogorov–Smirnov Test
The two-sample K–S test is a useful and general nonparametric method to compare two samples. This is because it is sensitive to differences in both location and shape of the empirical cumulative distribution functions of the two samples. It is well-suited for numerical data.
- Wasserstein distance
Wasserstein distance, also popularly known as the Earth Mover’s distance, is the minimum amount of work to transform a reference distribution into the target distribution. It gives a natural measure of the distance between two distributions. In contrast to several of the common divergences on distributions such as Kullback-Leibler, it is (weakly) continuous, and therefore ideal for analyzing corrupted data.
- Energy Distance
Energy distance is a statistical distance between the distributions of random vectors, which characterizes the equality of distributions. Energy distance is zero, if and only if the distributions are identical.
- Population Stability Index
Population Stability Index (PSI) is a popular metric in the finance industry to measure changes in distribution for two datasets. It produces less noise and has the advantage of a generally accepted threshold of 0.2-0.25.