Track and monitor experiments

Domino’s Experiment Manager uses MLflow Tracking to log experiment details, such as parameters, metrics, and artifacts. Domino stores all your experiment metadata, data, and results within the project, making it easy to reproduce your work. This allows you to:

  • Monitor experiments with both automatic and manual logging.

  • Compare and review experiment runs effectively.

  • Collaborate securely with colleagues.

Experiment Manager dashboard

Experiment types

You can log two types of experiments in Domino:

Traditional ML experiments track parameters, metrics, and artifacts from model training workflows. When you train a model, Domino logs the configuration settings, performance metrics, and output files so you can compare different approaches.

You use traditional ML experiments when you train machine learning models with scikit-learn, TensorFlow, PyTorch, or similar frameworks.

Agentic system experiments capture traces from LLM calls, agent interactions, and prompt chains. A trace shows the sequence of calls your system makes, including inputs, outputs, and downstream tool calls. You can attach evaluations to traces to measure quality, accuracy, or safety. You use agentic system experiments when you build applications with LLMs or AI agents.

How experiments work in Domino

When you click Experiments in the Domino left navigation, you’ll see an empty dashboard unless at least one run has been logged. Domino doesn’t create experiments through the UI.

Instead

  • You create an MLFlow experiment by calling mlflow.set_experiment() in code.

  • The experiment appears in the UI only after you log at least one run.

For example, to set a unique experiment name:

# create a new experiment
import mlflow
import os
starting_domino_user = os.environ["DOMINO_STARTING_USERNAME"]
experiment_name = f"Domino_Experiment_{starting_domino_user}"
# Replace <your_experiment_name> with the name of your experiment
mlflow.set_experiment(experiment_name="<your_experiment_name>")

Tip: Experiment names need to be unique across your Domino Deployment. A good practice is to use a unique identifier (such as username) as part of your experiment name.

Step 1: Create and log an experiment run

Launch a workspace or job to run your experiment code and log results to the Experiment Manager. Jobs provide better reproducibility and version control.

  1. From your project, click Workspaces in the left navigation.

  2. Launch a workspace using any environment based on the Domino Standard Environment. These environments already have the mlflow package installed.

  3. Open a Python script or notebook to run your experiment code.

You can log experiments using auto-logging or manual logging. Auto-logging records parameters, metrics, and artifacts automatically for supported libraries. Manual logging gives you full control over what parameters, metrics, and artifacts are recorded.

This example shows you how to auto-log a scikit-learn experiment.

# import MLflow library import mlflow import os from sklearn.model_selection import train_test_split from sklearn.datasets import load_diabetes from sklearn.ensemble import RandomForestRegressor # create and set a new experiment starting_domino_user = os.environ["DOMINO_STARTING_USERNAME"] experiment_name = f"Domino_Experiment_{starting_domino_user}" mlflow.set_experiment(experiment_name=experiment_name) # enable auto-logging mlflow.autolog() # start the run with mlflow.start_run(): db = load_diabetes() X_train, X_test, y_train, y_test = train_test_split(db.data, db.target) rf = RandomForestRegressor(n_estimators = 100, max_depth = 6, max_features = 3) rf.fit(X_train, y_train) rf.score(X_test, y_test) # end the run mlflow.end_run()
# import MLflow library
import mlflow
import os

from sklearn.model_selection import train_test_split
from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestRegressor

# create and set a new experiment
starting_domino_user = os.environ["DOMINO_STARTING_USERNAME"]
experiment_name = f"Domino_Experiment_{starting_domino_user}"
mlflow.set_experiment(experiment_name=experiment_name)


# enable auto-logging
mlflow.autolog()

# start the run
with mlflow.start_run():
    db = load_diabetes()
    X_train, X_test, y_train, y_test = train_test_split(db.data, db.target)
    rf = RandomForestRegressor(n_estimators = 100, max_depth = 6, max_features = 3)
    rf.fit(X_train, y_train)
    rf.score(X_test, y_test)

# end the run
mlflow.end_run()

Once your workspace is running, you’re ready to write and execute code that will create and log an experiment.

Step 2: Monitor and evaluate experiments

Once you have logged at least one run in an experiment, you can view and analyze it in Domino.

  1. Click Experiments in Domino’s left navigation pane.

  2. Find and click the experiment to evaluate and show its associated runs.

  3. Click a run to analyze the results in detail.

The run details show different information depending on your experiment type.

Traditional ML experiments

When you log traditional ML experiments, Domino captures parameters, metrics, and artifacts from your model training. Here’s how to view and compare your results:

View metrics and artifacts

The run details show parameters you configured, metrics that measure performance, and artifacts like model files or visualizations logged during training.

You can:

  • Review parameter values to understand your configuration

  • Track metrics over time to measure model performance

  • Download artifacts like trained models, plots, or data files

    Compare runs

    After reviewing individual runs, you can compare multiple runs to see how different configurations affect outcomes. Select up to four runs from the table view and click Compare.

Compare experiments

You can compare runs to see how parameters affect important metrics like model accuracy or training speed.

Compare runs

Agentic system experiments

When you log agentic system experiments, Domino captures traces from LLM calls and agent interactions. Here’s how to view and compare your traces:

View traces

When you open a run from an agentic system experiment, select the Traces tab to see the sequence of calls made during execution.

View Traces

You can:

  • See evaluations that were logged in code

  • Add metrics (float values) or labels (string values) to traces by clicking Add Metric/Label

  • Click any metric or label cell to open the detailed trace view

    Compare traces

    When comparing agentic system runs, open the Traces comparison view to see how configurations perform on the same inputs.

Compare traces

Review results across runs to spot patterns, compare traces and evaluations side-by-side, and click any metric or label cell to open the detailed trace view.

Step 3: Export experiment results

You can export your single experiment results to a CSV or compare experiments and download to a PDF.

To export single experiment results:

  1. From the Experiments view, open the experiment you want to export.

  2. Click the three dots in the upper right and choose Download CSV.

To export comparison results:

  1. From the Experiments view, select two to four runs and click Compare.

  2. Click the three dots in the upper right and choose Export as PDF.

Step 4: Share your projects and experiments

You can set permissions for project assets, including MLflow logs, on a project level. Use these methods on your projects to control access:

  • Choose the visibility for your project. This will help you control who can see your project.

    • Searchable: Discoverable by other Domino users.

    • Private: Only viewable or discoverable by your project collaborators.

  • Invite collaborators and set their permissions based on project roles. This gives you detailed control over what they can access.

Troubleshooting

  • Experiment names must be unique across all projects within the same Domino instance.

  • Child runs aren’t deleted when you delete a run that has children. Delete the child runs separately.

  • You can’t stop a run from the UI. To stop a run, execute mlflow.end_run() from your workspace or job.

  • When you trigger an experiment from a workspace, the manager shows the file name. But if you are running that and editing it, it doesn’t rename the experiment automatically. After completing an experiment in a workspace, trigger a job to manage version control.

  • Best practice is to give your runs a name - otherwise it automatically creates one like gifted-slug-123.

Advanced functionality

You can upload large artifact files directly to blob storage without going through the MLflow proxy server.

This feature must be enabled inside the user notebook code by setting the environment variable MLFLOW_ENABLE_PROXY_MULTIPART_UPLOAD to true.

import os
os.environ['MLFLOW_ENABLE_PROXY_MULTIPART_UPLOAD'] = "true"

This is helpful for both log_artifact calls and registering large language models. It is currently supported only in AWS and GCP environments. There are two additional settings available for configuration:

  • MLFLOW_MULTIPART_UPLOAD_MINIMUM_FILE_SIZE - the minimum file size required to initiate multipart uploads.

  • MLFLOW_MULTIPART_UPLOAD_CHUNK_SIZE - the size of each chunk of the multipart upload. Note that a file may be divided into a maximum of 1000 chunks.

Multipart upload for proxied artifact access in the MLflow documentation has more information on using this feature. Registering Hugging Face LLMs with MLflow has directions specific to Domino.

Next steps