Understanding the performance of your experiments can involve analyzing many outputs and results. It’s often useful to see key metrics at a glance across all your runs, to allow you to quickly identify which experiments are worth investigating further.
Domino’s diagnostic statistics functionality allows you to do just that.
To use this feature, write a file named dominostats.json to the root
of your project directory. Use keys in this JSON file to identify the
outputs you’re interested in, and then add the corresponding values.
Here is some example R code that writes three key/value pairs to
dominostats.json:
diagnostics = list("R^2" = 0.99, "p-value" = 0.05, "sse" = 10.49)
library(jsonlite)
fileConn<-file("dominostats.json")
writeLines(toJSON(diagnostics), fileConn)
close(fileConn)Here is the same data being written to dominostats.json by Python
code:
import json
with open('dominostats.json', 'w') as f:
f.write(json.dumps({"R^2": 0.99, "p-value": 0.05, "sse": 10.49}))The resulting dominostats.json file from these code examples looks
like this:
{
"sse": 10.49,
"R^2": 0.99,
"p-value": 0.05
}The dominostats.json file is deleted before each run automatically by
Domino. Therefore, past dominostats.json files will not pollute new
Jobs on your Jobs dashboard. If Domino detects that this file has been
written to the project root by a Job, it will parse the values out and
show them as columns on the Jobs dashboard.
You can see the keys represented as available columns in the dashboard,
and each row contains the corresponding value from a Job.
You can also click Jobs Timeline at the top of the dashboard to expand a
line chart of dominostats.json values over time. This chart shows all
Jobs displayed in the current dashboard. To filter to a specific set of
related Jobs, use tagging to create a separate dashboard view.
The x-axis ticks on the timeline represent individual Jobs, and the y-axis represents the values for the statistics in those jobs. Hover along the chart to see individual data points as tooltips, and click at a point to open the details for the Job that produced that value. You can also click and drag on the chart to zoom in, and you can click on each stat in the legend at upper right to toggle its line on and off.