Custom Model Monitoring metrics

Use Domino’s Custom Model Monitoring Metrics SDK to define custom metrics and use them alongside out-of-the-box drift and model quality metrics that are monitored in Domino Model Monitor. With this SDK, you can register new metrics and define the logic to compute them. You can author this logic and evaluate it from within a Domino project.

For every model that you register for monitoring, you can select a registered metric, associate the data sources from which the metric is computed, and set up the execution environment to compute this metric on a periodic basis. You are notified by email when a metric behaves abnormally based on threshold definitions.

For end-to-end working code with a description of the workflow, see the /custom metrics example folder in the quick-start project.

Use custom metrics

Use the following library functions from the Custom Metrics SDK to build and deploy your own monitoring metrics.

Instantiate the client. Enter your DMM model ID:

import domino
dmm_model_id = "XXXXXXXXXXXXX"
d = domino.Domino("integration-test/quick-start")
metrics_client = d.custom_metrics_client()

Log the custom metrics:

metrics_client.log_metric(dmm_model_id, "accuracy", 7.1234, "2022-10-08T00:00:00Z", { "example_tag1" : "value1", "example_tag2" : "value2" })

metrics_client.log_metrics([
{ "modelMonitoringId" : dmm_model_id, "metric" : "accuracy", "value" : 7.1234,
"timestamp" : "2022-10-08T00:00:00Z",
"tags" : { "example_tag1" : "value1", "example_tag2" : "value2" }
]
},
{ "modelMonitoringId" : dmm_model_id, "metric" : "accuracy", "value" : 8.4567,
"timestamp" : "2022-10-09T00:00:00Z" }
])

modelMonitoringId: ID of the monitored model to send metric alerts for
metric: Name of the metric to send alert for
value: Value of the metric
timestamp: Timezone is in UTC in ISO 8601 format.
tags: Custom metadata for metric represented as key-value string pairs

Send a custom metrics alert:
```
metrics_client.trigger_alert(dmm_model_id, "metric_name", 3.14,
condition = metrics_client.BETWEEN,
lower_limit=2.0, upper_limit=3.0,
description = "Breached 2.0-3.0 range." )
```
- modelMonitoringId: ID of the monitored model for which to send metric alerts.
- metric: Name of the metric for which to send the alert.
- value: Value of the metric.
- condition: Target range for the metric defined by lower and upper limit bounds.
  The following are potential values for the condition argument:
  - metrics_client.LESS_THAN = "lessThan"
  - metrics_client.LESS_THAN_EQUAL = "lessThanEqual"
  - metrics_client.GREATER_THAN = "greaterThan"
  - metrics_client.GREATER_THAN_EQUAL = "greaterThanEqual"
  - metrics_client.BETWEEN = "between"
- lower_limit: The lower limit for the condition.
- upper_limit: The upper limit for the condition.
- description: Optional message included in the alert.
Retrieve the custom metrics:
```
res = metrics_client.read_metrics(dmm_model_id, "accuracy",
"2022-10-08T00:00:00Z", "2022-10-10T00:00:00Z" )
```
- modelMonitoringId: ID of the monitored model for which to retrieve the metric values.
- metric: Name of the metric for which to retrieve the metric values.
- start_timestamp: The start timestamp of the range when the metrics were logged. The timezone is in UTC in ISO 8601 format.
- end_timestamp: The end timestamp of the range when the metrics were logged. The timezone is in UTC in ISO 8601 format.

Sample output

{
  'metricValues': [
      {'timestamp': '2022-10-08T00:00:00Z', 'value': 3.123, 'tags': []},
      {'timestamp': '2022-10-08T00:00:00Z', 'value': 3.123, 'tags': []},
      {'timestamp': '2022-10-08T00:00:00Z', 'value': 3.123, 'tags': []},
      {'timestamp': '2022-10-08T00:00:00Z', 'value': 3.123, 'tags': []},
{'timestamp': '2022-10-08T00:00:00Z', 'value': 7.1234, 'tags': {'example_tag1': 'value1', 'example_tag2': 'value2'}},
      {'timestamp': '2022-10-09T00:00:00Z', 'value': 8.4567, 'tags': []}
   ],
   'metadata': {'requestId': '97e61f63-3be5-4766-a07a-06c931d91268', 'notices': []}
}

Results:

timestamp: The time this metric was logged.
value: The value of the metric.
tags: Custom metadata for metric represented as key-value string pairs.